Redis Scan Count in production - count

I am in the process of replacing a redis KEYS command in favor of SCAN. However, the keyspace is about 3 Million Keys. What would be a good COUNT parameter to use without affecting I/O performance?

I am also kind of facing the same problem. But I tried this. Hope, this will help...
getKeyList (pattern) {
console.log('pattern', pattern)
let found = []
let cursor = '0'
while(true){
const getAsync = promisify(this.client.scan).bind(this.client)
const reply = getAsync(cursor, 'MATCH', pattern)
cursor = reply[0];
if(reply[1] != false){
found.push(reply[1])
}
if(cursor == 0){
break
}
}
return found
}
*Note that you have to manipulate the return as you want.

Related

AppSpreadsheet (GAS): avoid some problems with sistematic tested data

In my current job with spreadsheet, all inserted data passes through a test, checking if the same value is found on the same index in other sheets. Failing, a caution message is put in the current cell.
//mimimalist algorithm
function safeInsertion(data, row_, col_)
{
let rrow = row_ - 1; //range row
let rcol = col_ - 1; // range col
const active_sheet_name = getActiveSheetName(); // do as the its name suggest
const all_sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets();
//test to evaluate the value to be inserted in the sheet
for (let sh of all_sheets)
{
if (sh.getName() === active_sheet_name)
continue;
//getSheetValues do as its name suggest.
if( getSheetValues(sh)[rrow][rcol] === data )
return "prohibited insertion"
}
return data;
}
// usage (in cell): =safeInsertion("A scarce data", ROW(), COLUMN())
The problems are:
cached values confuse me sometimes. The script or data is changed but not perceived by the sheet itself until renewing manually the cell's content or refreshing all table. Is there any relevant configuration available to this issue?
Sometimes, at loading, a messing result appears. Almost all data are prohibited, for example (originally, all was fine!).
What can I do to obtain a stable sheet using this approach?
PS: The original function does more testing on each data insertion. Those tests consist on counting the frequency in the actual sheet and in all sheets.
EDIT:
In fact, I can't create a stable sheet. For test, a let you a copy of my code with minimal adaptations.
function safelyPut(data, max_onesheet, max_allsheet, row, col)
{
// general initialization
const data_regex = "\^\s*"+data+"\s*$"
const spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
const activesheet = spreadsheet.getActiveSheet();
const active_text_finder = activesheet.createTextFinder(data_regex)
.useRegularExpression(true)
.matchEntireCell(true);
const all_text_finder = spreadsheet.createTextFinder(data_regex)
.useRegularExpression(true)
.matchEntireCell(true);
const all_occurrences = all_text_finder.findAll();
//test general data's environment
const active_freq = active_text_finder.findAll().length;
if (max_onesheet <= active_freq)
return "Too much in a sheet";
const all_freq = all_occurrences.length;
if (max_allsheet <= all_freq)
return "Too much in the work";
//test unicity in a position
const active_sname = activesheet.getName();
for (occurrence of all_occurrences)
{
const sname = occurrence.getSheet().getName();
//if (SYSTEM_SHEETS.includes(sname))
//continue;
if (sname != active_sname)
if (occurrence.getRow() == row && occurrence.getColumn() == col)
if (occurrence.getValue() == data)
{
return `${sname} contains same data with the same indexes.`;
};
}
return data;
}
Create two or three cells and put randomly in a short range short range a value following the usage
=safeInsertion("Scarce Data", 3; 5; ROW(), COLUMN())
Do it, probably you will get a unstable sheet.
About cached values confuse me sometimes. The script is changed but not perceived by the sheet until renewing manually the cell's content or refreshing all table. No relevant configuration available to this issue?, when you want to refresh your custom function of safeInsertion, I thought that this thread might be useful.
About Sometimes, at loading, a messing result appears. Almost all data are prohibited, for example (originally, all was fine!). and What can I do to obtain a stable sheet using this approach?, in this case, for example, how about reducing the process cost of your script? I thought that by reducing the process cost of the script, your situation might be a bit stable.
When your script is modified by reducing the process cost, how about the following modification?
Modified script:
function safeInsertion(data, row_, col_) {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const range = ss.createTextFinder(data).matchEntireCell(true).findNext();
return range && range.getRow() == row_ && range.getColumn() == col_ && range.getSheet().getSheetName() != ss.getActiveSheet().getSheetName() ? "prohibited insertion" : data;
}
The usage of this is the same with your current script like =safeInsertion("A scarce data", ROW(), COLUMN()).
In this modification, TextFinder is used. Because I thought that when the value is searched from all sheets in a Google Spreadsheet, TextFinder is suitable for reducing the process cost.
References:
createTextFinder(findText) of Class Spreadsheet
findNext()

Modified function not working as intended without recursion

I have a recursive function which iterates though directory trees listing the file names located in them.
Here is the function:
void WINAPI SearchFile(PSTR Directory)
{
HANDLE hFind;
WIN32_FIND_DATA FindData;
char SearchName[1024],FullPath[1024];
memset(SearchName,0,sizeof(SearchName));
memset(&FindData,0,sizeof(WIN32_FIND_DATA));
sprintf(SearchName,"%s\\*",Directory);
hFind=FindFirstFile(SearchName,&FindData);
if(hFind!=INVALID_HANDLE_VALUE)
{
while(FindNextFile(hFind,&FindData))
{
if(FindData.cFileName[0]=='.')
{
continue;
}
memset(FullPath,0,sizeof(FullPath));
sprintf(FullPath,"%s\\%s",Directory,FindData.cFileName);
if(FindData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
{
MessageBoxA(NULL, FullPath, "Directory", MB_OK);
SearchFile(FullPath);
}
else
{
MessageBoxA(NULL, FullPath, "File", MB_OK);
}
}
FindClose(hFind);
}
}
There are obviously differences between both functions but I don't understand what's making them act differently. Does anyone know why I am having this problem?
for fast understand error need look for line
goto label;
//SearchFile(FullPath);
at this point hFind containing valid data and FindClose(hFind); need be called for it. but after goto label; executed - your overwrite hFind with hFind = FindFirstFile(SearchName, &FindData); - so you already never close original hFind, never can return to iterate folder after such go to sub-folder. this is key point - need save original hFind before go to sub directory and restore it after. when you do recursive function call - this is done auto - because every sub directory in this case enumerated in self stack frame, which have separate hFind. this is native solution use recursion here.
but possible convert recursion to loop here because we call self always from the single place and as result to this single place. so we can not save return address in stack but do unconditional jump (goto) to known place.
then code have some extra errors, you never check for string buffers overflow, why 1024 as max length is hard-coded when file path can be up to 32768 chars, you not check for reparse point as result can enter to infinite loop, use FindFirstFile instead FindFirstFileEx, etc.
correct code for enumerate sub-folder in loop can be next
void DoEnum(PCWSTR pcszRoot)
{
SIZE_T FileNameLength = wcslen(pcszRoot);
// initial check for . and ..
switch (FileNameLength)
{
case 2:
if (pcszRoot[1] != '.') break;
case 1:
if (pcszRoot[0] == '.') return;
}
static const WCHAR mask[] = L"\\*";
WCHAR FileName[MAXSHORT + 1];
if (_countof(FileName) < FileNameLength + _countof(mask))
{
return;
}
ULONG dwError;
HANDLE hFindFile = 0;
WIN32_FIND_DATA FindData{};
enum { MaxDeep = 0x200 };
//++ stack
HANDLE hFindFileV[MaxDeep];
PWSTR pszV[MaxDeep];
char prefix[MaxDeep+1];
//--stack
ULONG Level = MaxDeep;
memset(prefix, '\t', MaxDeep);
prefix[MaxDeep] = 0;
PWSTR psz = FileName;
goto __enter;
__loop:
hFindFile = FindFirstFileEx(FileName, FindExInfoBasic, &FindData, FindExSearchNameMatch, 0, FIND_FIRST_EX_LARGE_FETCH);
if (hFindFile != INVALID_HANDLE_VALUE)
{
do
{
pcszRoot = FindData.cFileName;
// skip . and ..
switch (FileNameLength = wcslen(pcszRoot))
{
case 2:
if (pcszRoot[1] != '.') break;
case 1:
if (pcszRoot[0] == '.') continue;
}
if (FindData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
{
if ((SIZE_T)(FileName + _countof(FileName) - psz) < FileNameLength + _countof(mask))
{
continue;
}
__enter:
memcpy(psz, pcszRoot, (1 + FileNameLength) * sizeof(WCHAR));
if (FindData.dwFileAttributes & FILE_ATTRIBUTE_REPARSE_POINT)
{
DbgPrint("%sreparse point: <%S>\n", prefix + Level, pcszRoot);
}
else
{
if (Level)
{
DbgPrint("%s<%S>\n", prefix + Level, psz);
hFindFileV[--Level] = hFindFile;
pszV[Level] = psz;
memcpy(psz += FileNameLength, mask, sizeof(mask));
psz++;
goto __loop;
__return:
*--psz = 0;
psz = pszV[Level];
hFindFile = hFindFileV[Level++];
DbgPrint("%s</%S>\n", prefix + Level, psz);
}
}
}
else
{
DbgPrint("%s[%u%u] %S\n", prefix + Level, FindData.nFileSizeLow, FindData.nFileSizeHigh, pcszRoot);
}
if (!hFindFile)
{
// top level exit
return ;
}
} while (FindNextFile(hFindFile, &FindData));
if ((dwError = GetLastError()) == ERROR_NO_MORE_FILES)
{
dwError = NOERROR;
}
FindClose(hFindFile);
}
else
{
dwError = GetLastError();
}
if (dwError)
{
DbgPrint("<%S> err = %u\n", FileName, dwError);
}
goto __return;
}
The reason for the difference is actually the confusion brought to you by goto label.If you are using the recursive version, after the recursive execution is completed, it will return to the recursive place to continue execution.
In your code, you continue to execute while (FindNextFile(hFind, &FindData)), but when you use goto label, it will jump out of the original loop and restart the program from the label, which leads to what you said list a single directory tree before ending.
If you modify the modified code to the following iterative version, you can understand why there is such a problem.
void fun()
{
char* Directory = "D:\\test";
HANDLE hFind;
WIN32_FIND_DATA FindData;
char SearchName[1024], FullPath[1024];
char LastName[1024] = "";
while (1)
{
memset(SearchName, 0, sizeof(SearchName));
memset(&FindData, 0, sizeof(WIN32_FIND_DATA));
sprintf(SearchName, "%s\\*", Directory);
if (strcmp(SearchName, LastName) == 0)
{
return;
}
strcpy(LastName, SearchName);
hFind = FindFirstFile(SearchName, &FindData);
if (hFind != INVALID_HANDLE_VALUE)
{
while (FindNextFile(hFind, &FindData))
{
if (FindData.cFileName[0] == '.')
{
continue;
}
memset(FullPath, 0, sizeof(FullPath));
sprintf(FullPath, "%s\\%s", Directory, FindData.cFileName);
if (FindData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
{
MessageBoxA(NULL, Directory, "Directory", MB_OK);
char cArray[1024];
memset(cArray, 0, sizeof(cArray));
sprintf(cArray, "%s", FullPath);
Directory = cArray;
break;
}
else
{
MessageBoxA(NULL, FullPath, "File", MB_OK);
}
}
FindClose(hFind);
}
}
}
So you cannot achieve the same purpose as recursion by using goto, here you can only use recursion. Of course, I have provided a way to traverse directories non-recursively using queues, which is a more scientific way.
One of the key things that you obtain from recursion is a separate set of local variables for each call to the recursive function. When a function calls itself, and in the recursive call modifies local variables, those local-variable changes do not (directly) affect the local variables of the caller. In your original program, this applies to variables hFind, FindData, SearchName, and FullPath.
If you want similar behavior in a non-recursive version of the function then you need to manually preserve the state of your traversal of one level of the tree when you descend to another level. The goto statement doesn't do any such thing -- it just redirects the control flow of your program. Although there are a few good use cases for goto in C, they are uncommon, and yours is not one of them.
There are several ways to implement manually preserving state, but I would suggest
creating a structure type in which to store those data that characterize the state of your traversal of a particular level. Those appear to be only hFind and FindData -- it looks like the other locals don't need to be preserved. Maybe something like this, then:
struct dir_state {
HANDLE hFind;
WIN32_FIND_DATA FindData;
};
Dynamically allocating an array of structures of that type.
unsigned depth_limit = DEFAULT_DEPTH_LIMIT;
struct dir_state *traversal_states
= malloc(depth_limit * sizeof(*traversal_states));
if (traversal_states == NULL) // ... handle allocation error ...
Tracking the depth of your tree traversal, and for each directory you process, using the array element whose index is the relative depth of that directory.
// For example:
traversal_states[depth].hFind
= FindFirstFile(SearchName, &traversal_states[depth].FindData);
// etc.
Remembering the size of the array, so as to be able to reallocate it larger if the traversal descends too deep for its current size.
// For example:
if (depth >= depth_limit) {
depth_limit = depth_limit * 3 / 2;
struct dir_state *temp
= realloc(traversal_states, depth_limit * sizeof(*traversal_states));
if (temp == NULL) {
// handle error, discontinuing traversal
}
traversal_states = temp;
}
Also, use an ordinary for, while, or do loop instead of a backward-jumping goto. There will be a few details to work out to track when to use FindFirstFile and when FindNextFile (which you would still have with goto), but I'm sure you can sort it out.
Details are left as an exercise.
Unless necessary due to memory or processing constraints or infinite recursion tail conditions that would be complication to introduce there really isn't much need to not use recursion here, since it leads to a readable and elegant solution.
I also want to point out that in "modern" C, any solution using a GOTO is likely not a solution you want since they are so often confusing to use and leads to memory issues (we have loops now to make all of that so much simpler).
Instead of the GOTOs I would suggest implementing a stack of the directories. Wrap the printing logic a while or do-while, and as you are iterating over the files add any directories to the stack. At every new iteration pop and walk the directory at the head of the stack. The loop condition just needs to check if the directory stack is empty, before continuing its block.

How can I perform SQL Count with ESP32 SQLite3?

I have spent ages searching for an answer to this, but i can't quite get my head around it.
Basically.I have a sqlite db on a sd card connected to a esp32. SQLite works perfectly and will return when using the standard sample code. For example:
rc = db_exec(db2, "Select * from domain_rank where domain between 'google.com' and 'google.com.z'");
if (rc != SQLITE_OK) {
sqlite3_close(db1);
sqlite3_close(db2);
return;
}`
this looks at a "db_exec" function in the sketch then passes it to a "callback" function. Is there a way to do without this? All I want to do is count the number of records in a select statement.
Thanks
Andrew
Ok I think I may have found a solution
Putting it here just in case anyone else is interested.
String sql = "Select count(*) from surnames where name = 'MICHELLE'";
rc = sqlite3_prepare_v2(db1, sql.c_str(), 1000, &res, &tail);
if (rc != SQLITE_OK) {
String resp = "Failed to fetch data: ";
Serial.println(resp.c_str());
return;
}
while (sqlite3_step(res) == SQLITE_ROW) {
rec_count = sqlite3_column_int(res, 0);
}
Serial.println(rec_count);
can't quite get my head around what the while loop is doing counting the cells, i thought the sql count* query would just return an int value I could use directly. So I'm still a little confused.
Comments would be welcomed! If anyone has any better ideas?
Thanks
Andrew

Cannot get Realm result for objects filtered by the latest (nsdate) value of a property of a collection property swift (the example is clearer)

I Have the following model
class Process: Object {
#objc dynamic var processID:Int = 1
let steps = List<Step>()
}
class Step: Object {
#objc private dynamic var stepCode: Int = 0
#objc dynamic var stepDateUTC: Date? = nil
var stepType: ProcessStepType {
get {
return ProcessStepType(rawValue: stepCode) ?? .created
}
set {
stepCode = newValue.rawValue
}
}
}
enum ProcessStepType: Int { // to review - real value
case created = 0
case scheduled = 1
case processing = 2
case paused = 3
case finished = 4
}
A process can start, processing , paused , resume (to be in step processing again), pause , resume again,etc. the current step is the one with the latest stepDateUTC
I am trying to get all Processes, having for last step ,a step of stepType processing "processing ", ie. where for the last stepDate, stepCode is 2 .
I came with the following predicate... which doesn't work. Any idea of the right perform to perform such query ?
my best trial is the one. Is it possible to get to this result via one realm query .
let processes = realm.objects(Process.self).filter(NSPredicate(format: "ANY steps.stepCode = 2 AND NOT (ANY steps.stepCode = 4)")
let ongoingprocesses = processes.filter(){$0.steps.sorted(byKeyPath: "stepDateUTC", ascending: false).first!.stepType == .processing}
what I hoped would work
NSPredicate(format: "steps[LAST].stepCode = \(TicketStepType.processing.rawValue)")
I understand [LAST] is not supported by realm (as per the cheatsheet). but is there anyway around I could achieve my goal through a realm query?
There are a few ways to approach this and it doesn't appear the date property is relevant because lists are stored in sequential order (as long as they are not altered), so the last element in the List was added last.
This first piece of code will filter for processes where the last element is 'processing'. I coded this long-handed so the flow is more understandable.
let results = realm.objects(Process.self).filter { p in
let lastIndex = p.steps.count - 1
let step = p.steps[lastIndex]
let type = step.stepType
if type == .processing {
return true
}
return false
}
Note that Realm objects are lazily loaded - which means thousands of objects have a low memory impact. By filtering using Swift, the objects are filtered in memory so the impact is more significant.
The second piece of code is what I would suggest as it makes filtering much simpler, but would require a slight change to the Process model.
class Process: Object {
#objc dynamic var processID:Int = 1
let stepHistory = List<Step>() //RENAMED: the history of the steps
#objc dynamic var name = ""
//ADDED: new property tracks current step
#objc dynamic var current_step = ProcessStepType.created.index
}
My thought here is that the Process model keeps a 'history' of steps that have occurred so far, and then what the current_step is.
I also modified the ProcessStepType enum to make it more filterable friendly.
enum ProcessStepType: Int { // to review - real value
case created = 0
case scheduled = 1
case processing = 2
case paused = 3
case finished = 4
//this is used when filtering
var index: Int {
switch self {
case .created:
return 0
case .scheduled:
return 1
case .processing:
return 2
case .paused:
return 3
case .finished:
return 4
}
}
}
Then to return all processes where the last step in the list is 'processing' here's the filter
let results2 = realm.objects(Process.self).filter("current_step == %#", ProcessStepType.processing.index)
The final thought is to add some code to the Process model so when a step is added to the list, the current_step var is also updated. Coding that is left to the OP.

How do I translate LR(1) Parse into a Abstract syntax tree?

I have coded a table driven LR(1) parser and it is working very well however I am having a bit of a disconnect on the stage of turing a parse into a syntax tree/abstract syntax tree. This is a project that I m very passionate about but I have really just hit a dead end here. Thank you for your help in advance.
Edit: Also my parser just uses a 2d array and an action object that tells it where to go next or if its a reduction where to go and how many items to pop. I noticed that many people use the visitor pattern. Im not sure how they know what type of node to make.
Here is the pushdown automata for context
while (lexer.hasNext() || parseStack.size() > 0) {
Action topOfStack = parseStack.peek();
token = parseStack.size() > 0 ? lexer.nextToken() : new Token(TokenType.EOF, "EOF");
topOfStack.setToken(token);
int row = topOfStack.getTransitionIndex();
int column = getTerminalIndex(token.getLexeme());
column = token.getType() == TokenType.IDENTIFIER
&& !terminalsContain(token.getLexeme()) ? 0 : column;
Action action = actionTable[row][column];
if (action instanceof Accept) {
System.out.println("valid parse!!!!!!");
} else if (action instanceof Reduction) {
Reduction reduction = (Reduction) action;
popStack(parseStack, reduction.getNumberOfItemsToPop());
column = reduction.getTransitionIndex();
row = parseStack.peek().getTransitionIndex();
parseStack.push(new Action(gotoTable[row][column]));
lexer.backupTokenStream();
} else if (action != null) {
parseStack.push(actionTable[row][column]);
} else {
System.out.println("Parse error");
System.out.println("On token: " + token.getLexeme());
break;
}
Each reduction in the LR parsing process corresponds to an internal node in the parse tree. The rule being reduced is the internal AST node, and the items popped off the stack correspond to the children of that internal node. The item pushed for the goto corresponds to the internal node, while those pushed by shift actions correspond to leaves (tokens) of the AST.
Putting all that together, you can easily build an AST by createing a new internal node every time you do a reduction and wiring everything together appropriately.

Resources