.Net Core 3 Preview SequenceReader Length Delimited Parsing - .net-core

I'm trying to use SequenceReader<T> in .Net Core Preview 8 to parse Guacamole Protocol network traffic.
The traffic might look as follows:
5.error,14.some text here,1.0;
This is a single error instruction. There are 3 fields:
OpCode = error
Reason = some text here
Status = 0 (see Status Codes)
The fields are comma delimited (semi-colon terminated), but they also have the length prefixed on each field. I presume that's so that you could parse something like:
5.error,24.some, text, with, commas,1.0;
To produce Reason = some, text, with, commas.
Simple comma delimited parsing is simple enough to do (with or without SequenceReader). However, to utilise the length I've tried the following:
public static bool TryGetNextElement(this ref SerializationContext context, out ReadOnlySequence<byte> element)
{
element = default;
var start = context.Reader.Position;
if (!context.Reader.TryReadTo(out ReadOnlySequence<byte> lengthSlice, Utf8Bytes.Period, advancePastDelimiter: true))
return false;
if (!lengthSlice.TryGetInt(out var length))
return false;
context.Reader.Advance(length);
element = context.Reader.Sequence.Slice(start, context.Reader.Position);
return true;
}
Based on my understanding of the initial proposal, this should work, though also could be simplified I think because some of the methods in the proposal make life a bit easier than that which is available in .Net Core Preview 8.
However, the problem with this code is that the SequenceReader does not seem to Advance as I would expect. It's Position and Consumed properties remain unchanged when advancing, so the element I slice at the end is always an empty sequence.
What do I need to do in order to parse this protocol correctly?

I'm guessing that .Reader here is a property; this is important because SequenceReader<T> is a mutable struct, but every time you access .SomeProperty you are working with an isolated copy of the reader. It is fine to hide it behind a property, but you'd need to make sure you work with a local and then push back when complete, i.e.
var reader = context.Reader;
var start = reader.Position;
if (!reader.TryReadTo(out ReadOnlySequence<byte> lengthSlice,
Utf8Bytes.Period, advancePastDelimiter: true))
return false;
if (!lengthSlice.TryGetInt(out var length))
return false;
reader.Advance(length);
element = reader.Sequence.Slice(start, reader.Position);
context.Reader = reader; // update position
return true;
Note that a nice feature of this is that in the failure cases (return false), you won't have changed the state yet, because you've only been mutating your local standalone clone.
You could also consider a ref-return property for .Reader.

Related

issue with paginated reading datastore

I have a weired issue, I can't believe such a common feature could be broken (the error is certainely on my side), but I can't find how to make it work. I want to use the cursor from datastore to get paginated results, I keep getting all of them whatever i do
FetchOptions fetchOptions = FetchOptions.Builder.withChunkSize(5).prefetchSize(6);
String datastoreCursor = filter.getDatastoreCursor();
if (datastoreCursor != null) {
fetchOptions = fetchOptions.startCursor(Cursor.fromWebSafeString(datastoreCursor));
}
QueryResultList<Entity> result = preparedQuery.asQueryResultList(fetchOptions);
ArrayList<Product> productList = new ArrayList<Product>();
// int count = 0;
for (Entity entity : result) {
// if (++count == PRODUCTS_PER_PAGE)
// break;
Key key = entity.getKey();
productList.add(populateProduct(key.getId(), true, entity));
}
toReturn.setDatastoreCursor(result.getCursor());
Also if I don't read the rows (uncomment the lines with counter) and get the cursor the resulting cursor is the same. I thought it might bring me back to the last read element under the datastabase cursor (thinking result.getCursor() reflects the state of the db cursor)
I'm getting a cursor with this value E-ABAOsB8gEQbW9kaWZpY2F0aW9uRGF0ZfoBCQiIjsfAmKm_AuwBggIhagljaGF0YW1vamVyFAsSB1Byb2R1Y3QYgICAgICosgsMFA that points to no more elements (I have 23 elements for my test that I all receive from the first query)
When you use a QueryResultList, the requested cursor will always point to the end of the list. As specified by the javadoc of QueryResultList#getCursor:
Gets a Cursor that points to the result immediately after the last one in this list.
Even though you provide a prefetch and chunk size, The entire result list will still have all of your results since you have not specified a limit. Thus, the expected cursor is the cursor after the final element.
If you only want a specific number of entities per page, you should set a limit on the FetchOptions using the limit method. Then when you call getCursor(), you'll get a cursor at the end of your page, as opposed to the end of your dataset.
Instead, you could also use a QueryResultIterator. Unlike the QueryResultList, calling getCursor on a QueryResultIterator will result in the cursor that points after the last entity retrieved by calling .next() (javadoc).

Lasso 9 Hangs on Inserting Pair with Map Value into Array?

EDIT: I accidentally misrepresented the problem when trying to pare-down the example code. A key part of my code is that I am attempting to sort the array after adding elements to it. The hang appears on sort, not insert. The following abstracted code will consistently hang:
<?=
local('a' = array)
#a->insert('test1' = map('a'='1'))
#a->insert('test2' = map('b'='2')) // comment-out to make work
#a->sort
#a
?>
I have a result set for which I want to insert a pair of values into an array for each unique key, as follows:
resultset(2) => {
records => {
if(!$logTypeClasses->contains(field('logTypeClass'))) => {
local(i) = pair(field('logTypeClass'), map('title' = field('logType'), 'class' = field('logTypeClass')))
log_critical(#i)
$logTypeClasses->insert(#i) // Lasso hangs on this line, will return if commented-out
}
}
}
Strangely, I cannot insert the #i local variable into thread variable without Lasso hanging. I never receive an error, and the page never returns. It just hangs indefinitely.
I do see the pairs logged correctly, which leads me to believe that the pair-generating syntax is correct.
I can make the code work as long as the value side of the pair is not a map with values. In other words, it works when the value side of the pair is a string, or even an empty map. As soon as I add key=value parameters to the map, it fails.
I must be missing something obvious. Any pointers? Thanks in advance for your time and consideration.
I can verify the bug with the basic code you sent with sorting. The question does arise how exactly one sorts pairs. I'm betting you want them sorted by the first element in the pair, but I could also see the claim that they should be sorted by last element in the pair (by values instead of by keys)
One thing that might work better is to keep it as a map of maps. If you need the sorted data for some reason, you could do map->keys->asArray->sort
Ex:
local(data) = map('test1' = map('a'=2,'b'=3))
#data->insert('test2' = map('c'=33, 'd'=42))
local(keys) = #data->keys->asArray
#keys->sort
#keys
Even better, if you're going to just iterate through a sorted set, you can just use a query expression:
local(data) = map('test1' = map('a'=2,'b'=3))
#data->insert('test2' = map('c'=33, 'd'=42))
with elm in #data->eachPair
let key = #elm->first
let value = #elm->second
order by #key
do { ... }
I doubt you problem is the pair with map construct per se.
This test code works as expected:
var(testcontainer = array)
inline(-database = 'mysql', -table = 'help_topic', -findall) => {
resultset(1) => {
records => {
if(!$testcontainer->contains(field('name'))) => {
local(i) = pair(field('name'), map('description' = field('description'), 'name' = field('name')))
$testcontainer->insert(#i)
}
}
}
}
$testcontainer
When Lasso hangs like that with no feedback and no immediate crash it is usually trapped in some kind of infinite loop. I'm speculating that it might have to do with Lasso using references whenever possible. Maybe some part of your code is using a reference that references itself. Or something.

I am having trouble adding elements to a referenced vector of a map

EDIT: Forgot to mention that my programming language is C++, and my IDE is Qt Creator.
I don't understand why I am having trouble adding elements to a referenced vector of a map (i.e. sometimes it works and sometimes it fails).
For example:
class C_html4_tags {
public:
C_html4_tags();
//.......
typedef std::vector<S_html_attr_value> TYPE_attr_values;
//.....
};
//...
C_html4_tags::C_html4_tags() {
//........
map<S_browser, TYPE_attr_values> attr_supported_attr_values_map;
S_browser dummy_browser; //a fake browser key for referencing a fully Html 4.01-compliant supported attr values list
dummy_browser.name = "Dummy";
//.....
TYPE_attr_values& attr_supported_attr_values = attr_supported_attr_values_map[dummy_browser];
//......
**attr_supported_attr_values = attr_supported_attr_values_map[dummy_browser];**
//...
**attr_supported_attr_values.clear();
attr_supported_attr_values_map.clear();**
//......
**attr_supported_attr_values = attr_supported_attr_values_map[dummy_browser];**
//...
**attr_supported_attr_values.clear();
attr_supported_attr_values_map.clear();**
}
I do the bolded lines multiple times with a bunch of different attributes, with very little difference between them, without a problem until reaching this one attribute (a_tabindex_attr), in which the IDE doesn't report anything wrong when running it normally (except "The program has unexpectedly finished." However, when debugging it, it now reports:
Signal received
The inferior stopped because it received a signal from the Operating
System.
Signal name : SIGSEGV Signal meaning : Segmentation fault
And the backtrace points to the attribute I mentioned above, on the line of code which does this:
attr_supported_attr_values.clear();
Now, after adding some debugging cout lines to the code, I learned that what's happening is, for some reason, even after doing:
attr_supported_attr_values.push_back(attr_value);
the vector object which is the returned mapped value of:
attr_supported_attr_values = attr_supported_attr_values_map[dummy_browser];
is not actually being changed when I push_back an S_html_attr_value object to the referenced vector. Now, I don't why that is, since I do pretty much the exact same thing in a bunch of other attributes' code before this one, without any problem, but for some reason now, on this one, it ain't adding the object to attr_supported_attr_values (which is a reference to the returned mapped value of 'dummy_browser'). And I know this for a fact, because I outputted the size of the internal mapped value object of the map (using an iterator for the map), and it was 0 after the call to push_back(). However, what is even more odd is, I ALSO outputted attr_supported_attr_values.size() after the call to push_back(), and that was 1! Now how could that be since they're both supposed to be the same object??
Note:
The full code of what I'm talking about is below (or at least the attribute code, minus the debugging statements...):
//a_tabindex_attr:
attr_desc = "Specifies the position of an <a> element in the\n"
"tabbing order for the current document.\n"
"The tabbing order defines the order in which\n"
"elements will receive focus when navigated by\n"
"the user via the keyboard. The tabbing order\n"
"may include elements nested within other elements.\n"
"Elements that may receive focus based on tabindex\n"
"adhere to the following rules:\n\n"
"1. Those elements that support this attribute and\n"
"assign a positive value to it are navigated first.\n"
"Navigation proceeds from the element with the\n"
"lowest tabindex value to the element with the\n"
"highest value. Values need not be sequential\n"
"nor must they begin with any particular value.\n"
"Elements that have identical tabindex values\n"
"should be navigated in the order they appear\n"
"in the character stream.\n"
"2. Those elements that do not support this\n"
"attribute or support it and assign it a value\n"
"of \"0\" are navigated next. These elements are\n"
"navigated in the order they appear in the\n"
"character stream.\n"
"3. Elements that are disabled do not participate\n"
"in the tabbing order.";
attr_supported_attr_values = attr_supported_attr_values_map[dummy_browser];
attr_value_desc = "A number you specify for the tab index/order.\n"
"It must be a value between 0 and 32767.";
attr_value_content = "<i>number</i>";
attr_value = C_html4_attributes_obj.getAttrValue(attr_value_desc, attr_value_content,
attr_value_supported_browsers,
attr_value_required, attr_value_deprecated,
attr_value_notes, attr_value_tips);
attr_value.is_relative = true;
attr_supported_attr_values.push_back(attr_value);
try {
N_init_class_members::initAttr(C_html4_attributes::attr_tabindex, a_tabindex_attr, attr_desc,
attr_supported_browsers, attr_supported_attr_values_map, attr_required,
attr_deprecated, attr_notes, attr_tips);
}
catch (const char* exc) {
string exc_str = "Error! Call to N_init_class_members::initAttr() from\n"
"constructor of C_html4_tags class. That function\n"
"reported the following exception:\n\n";
exc_str += exc;
throw (exc_str.c_str()); //re-throw exception
}
a_tabindex_attr.is_standard_attr = true;
a_tabindex_attr.is_optional_attr = true;
//cleanup from attribute operations so we can begin working on the next attribute:
C_html4_attributes_obj.clear(); //wipes the slate clean for all the supported attributes' properties except content
attr_desc.clear();
attr_supported_attr_values.clear();
attr_supported_attr_values_map.clear();
attr_value_desc.clear();
attr_value_content.clear();

AX 2009: Adjusting User Group Length

We're looking into refining our User Groups in Dynamics AX 2009 into more precise and fine-tuned groupings due to the wide range of variability between specific people within the same department. With this plan, it wouldn't be uncommon for majority of our users to fall user 5+ user groups.
Part of this would involve us expanding the default length of the User Group ID from 10 to 40 (as per Best Practice for naming conventions) since 10 characters don't give us enough room to adequately name each group as we would like (again, based on Best Practice Naming Conventions).
We have found that the main information seems to be obtained from the UserGroupInfo table, but that table isn't present under the Data Dictionary (it's under the System Documentation, so unavailable to be changed that way by my understanding). We've also found the UserGroupName EDT, but that is already set at 40 characters. The form itself doesn't seem to restricting the length of the field either. We've discussed changing the field on the SQL directly, but again my understanding is that if we do a full synchronization it would overwrite this change.
Where can we go to change this particular setting, or is it possible to change?
The size of the user group id is defined as as system extended data type (here \System Documentation\Types\userGroupId) and you cannot change any of the properties including the size 10 length.
You should live with that, don't try to fake the system using direct SQL changes. Even if you did that, AX would still believe that length is 10.
You could change the SysUserInfo form to show the group name only. The groupId might as well be assigned by a number sequence in your context.
I wrote a job to change the string size via X++ and it works for EDTs, but it can't seem to find the "userGroupId". From the general feel of AX I get, I'd be willing to guess that they just have it in a different location, but maybe not. I wonder if this could be tweaked to work:
static void Job9(Args _args)
{
#AOT
TreeNode treeNode;
Struct propertiesExt;
Map mapNewPropertyValues;
void setTreeNodePropertyExt(
Struct _propertiesExt,
Map _newProperties
)
{
Counter propertiesCount;
Array propertyInfoArray;
Struct propertyInfo;
str propertyValue;
int i;
;
_newProperties.insert('IsDefault', '0');
propertiesCount = _propertiesExt.value('Entries');
propertyInfoArray = _propertiesExt.value('PropertyInfo');
for (i = 1; i <= propertiesCount; i++)
{
propertyInfo = propertyInfoArray.value(i);
if (_newProperties.exists(propertyInfo.value('Name')))
{
propertyValue = _newProperties.lookup(propertyInfo.value('Name'));
propertyInfo.value('Value', propertyValue);
}
}
}
;
treeNode = TreeNode::findNode(#ExtendedDataTypesPath);
// This doesn't seem to be able to find the system type
//treeNode = treeNode.AOTfindChild('userGroupId');
treeNode = treeNode.AOTfindChild('AccountCategory');
propertiesExt = treeNode.AOTgetPropertiesExt();
mapNewPropertyValues = new Map(Types::String, Types::String);
mapNewPropertyValues.insert('StringSize', '30');
setTreeNodePropertyExt(propertiesExt, mapNewPropertyValues);
treeNode.AOTsetPropertiesExt(propertiesExt);
treeNode.AOTsave();
info("Done");
}

Full text search on a mobile device?

We'll soon be embarking on the development of a new mobile application. This particular app will be used for heavy searching of text based fields. Any suggestions from the group at large for what sort of database engine is best suited to allowing these types of searches on a mobile platform?
Specifics include Windows Mobile 6 and we'll be using the .Net CF. Also some of the text based fields will be anywhere between 35 and 500 characters. The device will operate in two different methods, batch and WiFi. Of course for WiFi we can just submit requests to a full blown DB engine and just fetch results back. This question centres around the "batch" version which will house a database loaded with information on the devices flash/removable storage card.
At any rate, I know SQLCE has some basic indexing but you don't get into the real fancy "full text" style indexes until you've got the full blown version which of course isn't available on a mobile platform.
An example of what the data would look like:
"apron carpenter adjustable leather container pocket waist hardware belt" etc. etc.
I haven't gotten into the evaluation of any other specific options yet as I figure I'd leverage the experience of this group in order to first point me down some specific avenues.
Any suggestions/tips?
Just recently I had the same issue. Here is what I did:
I created a class to hold just an id and the text for each object (in my case I called it a sku (item number) and a description). This creates a smaller object that uses less memory since it is only used for searching. I'll still grab the full-blown objects from the database after I find matches.
public class SmallItem
{
private int _sku;
public int Sku
{
get { return _sku; }
set { _sku = value; }
}
// Size of max description size + 1 for null terminator.
private char[] _description = new char[36];
public char[] Description
{
get { return _description; }
set { _description = value; }
}
public SmallItem()
{
}
}
After this class is created, you can then create an array (I actually used a List in my case) of these objects and use it for searching throughout your application. The initialization of this list takes a bit of time, but you only need to worry about this at start up. Basically just run a query on your database and grab the data you need to create this list.
Once you have a list, you can quickly go through it searching for any words you want. Since it's a contains, it must also find words within words (e.g. drill would return drill, drillbit, drills etc.). To do this, we wrote a home-grown, unmanaged c# contains function. It takes in a string array of words (so you can search for more than one word... we use it for "AND" searches... the description must contain all words passed in... "OR" is not currently supported in this example). As it searches through the list of words it builds a list of IDs, which are then passed back to the calling function. Once you have a list of IDs, you can easily run a fast query in your database to return the full-blown objects based on a fast indexed ID number. I should mention that we also limit the maximum number of results returned. This could be taken out. It's just handy if someone types in something like "e" as their search term. That's going to return a lot of results.
Here's the example of custom Contains function:
public static int[] Contains(string[] descriptionTerms, int maxResults, List<SmallItem> itemList)
{
// Don't allow more than the maximum allowable results constant.
int[] matchingSkus = new int[maxResults];
// Indexes and counters.
int matchNumber = 0;
int currentWord = 0;
int totalWords = descriptionTerms.Count() - 1; // - 1 because it will be used with 0 based array indexes
bool matchedWord;
try
{
/* Character array of character arrays. Each array is a word we want to match.
* We need the + 1 because totalWords had - 1 (We are setting a size/length here,
* so it is not 0 based... we used - 1 on totalWords because it is used for 0
* based index referencing.)
* */
char[][] allWordsToMatch = new char[totalWords + 1][];
// Character array to hold the current word to match.
char[] wordToMatch = new char[36]; // Max allowable word size + null terminator... I just picked 36 to be consistent with max description size.
// Loop through the original string array or words to match and create the character arrays.
for (currentWord = 0; currentWord <= totalWords; currentWord++)
{
char[] desc = new char[descriptionTerms[currentWord].Length + 1];
Array.Copy(descriptionTerms[currentWord].ToUpper().ToCharArray(), desc, descriptionTerms[currentWord].Length);
allWordsToMatch[currentWord] = desc;
}
// Offsets for description and filter(word to match) pointers.
int descriptionOffset = 0, filterOffset = 0;
// Loop through the list of items trying to find matching words.
foreach (SmallItem i in itemList)
{
// If we have reached our maximum allowable matches, we should stop searching and just return the results.
if (matchNumber == maxResults)
break;
// Loop through the "words to match" filter list.
for (currentWord = 0; currentWord <= totalWords; currentWord++)
{
// Reset our match flag and current word to match.
matchedWord = false;
wordToMatch = allWordsToMatch[currentWord];
// Delving into unmanaged code for SCREAMING performance ;)
unsafe
{
// Pointer to the description of the current item on the list (starting at first char).
fixed (char* pdesc = &i.Description[0])
{
// Pointer to the current word we are trying to match (starting at first char).
fixed (char* pfilter = &wordToMatch[0])
{
// Reset the description offset.
descriptionOffset = 0;
// Continue our search on the current word until we hit a null terminator for the char array.
while (*(pdesc + descriptionOffset) != '\0')
{
// We've matched the first character of the word we're trying to match.
if (*(pdesc + descriptionOffset) == *pfilter)
{
// Reset the filter offset.
filterOffset = 0;
/* Keep moving the offsets together while we have consecutive character matches. Once we hit a non-match
* or a null terminator, we need to jump out of this loop.
* */
while (*(pfilter + filterOffset) != '\0' && *(pfilter + filterOffset) == *(pdesc + descriptionOffset))
{
// Increase the offsets together to the next character.
++filterOffset;
++descriptionOffset;
}
// We hit matches all the way to the null terminator. The entire word was a match.
if (*(pfilter + filterOffset) == '\0')
{
// If our current word matched is the last word on the match list, we have matched all words.
if (currentWord == totalWords)
{
// Add the sku as a match.
matchingSkus[matchNumber] = i.Sku.ToString();
matchNumber++;
/* Break out of this item description. We have matched all needed words and can move to
* the next item.
* */
break;
}
/* We've matched a word, but still have more words left in our list of words to match.
* Set our match flag to true, which will mean we continue continue to search for the
* next word on the list.
* */
matchedWord = true;
}
}
// No match on the current character. Move to next one.
descriptionOffset++;
}
/* The current word had no match, so no sense in looking for the rest of the words. Break to the
* next item description.
* */
if (!matchedWord)
break;
}
}
}
}
};
// We have our list of matching skus. We'll resize the array and pass it back.
Array.Resize(ref matchingSkus, matchNumber);
return matchingSkus;
}
catch (Exception ex)
{
// Handle the exception
}
}
Once you have the list of matching skus, you can iterate through the array and build a query command that only returns the matching skus.
For an idea of performance, here's what we have found (doing the following steps):
Search ~171,000 items
Create list of all matching items
Query the database, returning only the matching items
Build full-blown items (similar to SmallItem class, but a lot more fields)
Populate a datagrid with the full-blow item objects.
On our mobile units, the entire process takes 2-4 seconds (takes 2 if we hit our match limit before we have searched all items... takes 4 seconds if we have to scan every item).
I've also tried doing this without unmanaged code and using String.IndexOf (and tried String.Contains... had same performance as IndexOf as it should). That way was much slower... about 25 seconds.
I've also tried using a StreamReader and a file containing lines of [Sku Number]|[Description]. The code was similar to the unmanaged code example. This way took about 15 seconds for an entire scan. Not too bad for speed, but not great. The file and StreamReader method has one advantage over the way I showed you though. The file can be created ahead of time. The way I showed you requires the memory and the initial time to load the List when the application starts up. For our 171,000 items, this takes about 2 minutes. If you can afford to wait for that initial load each time the app starts up (which can be done on a separate thread of course), then searching this way is the fastest way (that I've found at least).
Hope that helps.
PS - Thanks to Dolch for helping with some of the unmanaged code.
You could try Lucene.Net. I'm not sure how well it's suited to mobile devices, but it is billed as a "high-performance, full-featured text search engine library".
http://incubator.apache.org/lucene.net/
http://lucene.apache.org/java/docs/

Resources