I have an AppleScript script that runs a recursive function that counts instances of inserted plugins.
After each plugin that is inserted the function checks the CPU and decides whether there's a CPU overload or not.
If a CPU overload was found, it starts to remove plugins until it reaches a point that is satisfactory.
Then it returns the number of instances that was loaded on the computer.
The problem is that I get a stack overflow after a certain amount of runs. Does AppleScript has a limit of internal recursive threads?
on plugin_recurse(mode, plugin_name, component, track_count, instance_count, has_ref, min_instances, last_max)
try
log "mode - " & mode
if mode is "ret" then return {track_count, instance_count, last_max}
if mode is "add" then
if (instance_count - (10 * track_count) = 0) then
create_track(component)
set track_count to track_count + 1
end if
set instance_count to instance_count + 1
insert_plugin(plugin_name, component, track_count, instance_count)
if has_ref then
set CPUover to false
if min_instances = 1 then
set mode to "ret"
else
set min_instances to min_instances - 1
end if
else
set {CPUover, last_max} to check_cpu(last_max)
end if
if CPUover then
set mode to "sub"
end if
end if
if mode is "sub" then
if instance_count > 1 then
remove_plugin(plugin_name, component, track_count, instance_count)
set instance_count to instance_count - 1
if ((10 * track_count) - instance_count = 10) then
remove_track(track_count)
set track_count to track_count - 1
end if
set {CPUover, last_max} to check_cpu(last_max)
if not CPUover then
set mode to "ret"
end if
else
set mode to "ret"
end if
end if
plugin_recurse(mode, plugin_name, component, track_count, instance_count, has_ref, min_instances, last_max)
on error err
error err
end try
end plugin_recurse
The problem is that I get a stack overflow after a certain amount of
runs. Does AppleScript has a limit of internal recursive threads?
Yes, AppleScript is limited to many boundaries. Your error is caused by the limit of depth of handler (functions) calls. On my machine the level limit is set to 577. Such limitations are quite common for OOP languages because the 'virtual machine' that runs your code needs it's limits. Objective-C, for example, has also an limitation in recursion. If you need more, your code is considered as bad coding and you should try using a normal loop. However, I have to admit that 577 is not a very high number compared with other OOP's limits.
For such code, where it's uncertain how many recursion there will be, it's normally better to use loops than recursion.
Related
Anybody have any tips for looping, and continue? For example, I placed about 2500 pnrs on a queue, and I need to add a remark to each of them. Is it possible for a script to add the remark then move to the next pnr?
For example, I placed about 2500 pnrs on a queue, and I need to add a remark to each of them. Is it possible for a script to add the remark then move to the next pnr?
Loops are supported in Scribe but have to be built manually by creating your own iteration variables and breaking the loop manually when you know the work is complete.
What you are describing is definitely possible, but working in queues can be difficult as there are many possible responses when you try to end the PNRs. You'll have to capture the response to detect whether you need to do something else to get out of the error condition (e.g. if a PNR warning indicates you have to double-end the record).
If possible, its likely simpler to work off the queue by collecting all PNR locators and then looping through that list, adding your remarks, and then ending the PNRs. You'll still have to capture the response to determine if the PNR is actually ended properly, but you won't have to deal with the buggy queue behavior. A basic Scribe loop sample is below. Note that I haven't been a Scribe developer for a while and I did this in Notepad so there might be some errors in here, but hopefully it's a good starting point.
DEFINE [ROW=N:8] ;iteration variable/counter
DEFINE [LOCATOR_FILE=*:60] ;File Path
DEFINE [TEMP_LOCATOR=*:6] ;pnr locator variable, read from the temp file
DEFINE [BREAK=*:1] ;loop breaking variable
OPEN F=[TEMP_LOCATOR] L=0 ;open the file of locators
[BREAK] = ""
[ROW] = 0
REPEAT
[ROW] = [ROW] + 1
[TEMP_LOCATOR] = "" ;Reset temp locator variable, this will break our loop
READ F=[LOCATOR_FILE] R=[ROW] C=1 [TEMP_LOCATOR]
IF $[TEMP_LOCATOR] = 6 THEN ;test length of locator, if this is 6 chars, you have a good one, pull it up and add your remark
»"5YOUR REMARK HERE"{ENTER}«
»ER{ENTER}«
;trap errors
READ F="EMUFIND:" R=0 C=0 [TEMP_LOCATOR] ;read for the locator being present on this screen, which should indicate that the ER was successful - you'll have to trap other errors here though
IF [#SYSTEM_ERROR] = 0 THEN ;this locator was found, ER appears successful
»I{ENTER}« ;Ignore this PNR and move to the next one
ELSE
[BREAK] = "Y" ;error found afeter ER, break loop. Maybe show a popup box or something, up to you
ENDIF
ELSE ;No locator found in file, break the loop
[BREAK] = "Y"
ENDIF
UNTIL [BREAK] = "Y"
CLOSE [LOCATOR_FILE]
(1) The test suite with installation & running instructions:
https://github.com/TeddyTeddy/robot-fw-rest-instance-library-tests-v2
(2) The test suite is testing a locally running JSON RESTFUL API server:
https://github.com/typicode/json-server
The DB file for the server (i.e. db.json) is at the root of (1). The server reads the file and creates the API endpoints based on it.
An example run of the server:
(base) ~/Python/Robot/robot-fw-rest-instance-library-tests-v2$ json-server --watch db.json
\{^_^}/ hi!
Loading db.json
Done
Resources
http://localhost:3000/posts
http://localhost:3000/comments
http://localhost:3000/albums
http://localhost:3000/photos
http://localhost:3000/users
http://localhost:3000/todos
(3) With the given db.json, you can make the following request to the server:
GET /posts?_start=<start_index>&_end=<end_index>
where _start is inclusive and _end is exclusive. Note that start_index starts from 0 just like in Array.Slice method.
(4) To be able to comprehensively test (3), i wrote the following Robot Test Case, which is provided in (1):
Slicing Posts With All Possible Start And End Combinations
[Documentation] Referring to the API documentation:
... GET /posts?_start=20&_end=30
... where _start is inclusive and _end is exclusive
... This test case make the above API call with all possible combinations of _start and _end values.
... For each call, the test case fetches expected_posts from database for the same _start and _end.
... It then compares the expected_posts with observed_posts. It also calculates the expected length
... of observed_posts and compares that with the observed length of observed_posts
[Tags] read-tested slicing run-me-only
FOR ${start_index} IN RANGE ${0} ${NUMBER_OF_POSTS+10}
FOR ${end_index} IN RANGE ${start_index+1} ${NUMBER_OF_POSTS + 10 +1}
Log To Console start_index:${start_index}
Log To Console end_index:${end_index}
# note that start_index starts from zero when posts are fetched from database
${expected_posts} = Fetch Posts From Database ${start_index} ${end_index}
# note that start_index starts from 0 too when posts are fetched via API call
# test call
${observed_posts} = Get Sliced Posts ${start_index} ${end_index}
Should Be Equal ${expected_posts} ${observed_posts}
# note that start_index is between [0, NUMBER_OF_POSTS-1]
# and end_index is between [start_index+1, start_index+NUMBER_OF_POSTS]
# we expect observed_posts to be a non-empty list at least containing 1 item
${observed_length} = Get Length ${observed_posts}
# calculate expected_length of the observed_posts list
IF ${end_index} < ${NUMBER_OF_POSTS}
${expected_length} = Evaluate $end_index-$start_index
ELSE IF ${end_index} >= ${NUMBER_OF_POSTS} and ${start_index} < ${NUMBER_OF_POSTS}
${expected_length} = Evaluate $NUMBER_OF_POSTS-$start_index
ELSE
${expected_length} = Set Variable ${0}
END
Should Be Equal ${expected_length} ${observed_length}
Free Memory ${expected_posts} # (*)
Free Memory ${observed_posts} # (*)
END
Reload Library REST # (**)
END
Note that when you follow the instruction to run the test suite via ./run, you will only execute this test case (because of --include run-me-only tag in run command).
The problem
As the test case runs, the amount of memory Robot & RESTInstance use grows to gigabyte levels in a few minutes.
The Question
How can I prevent this from happening?
How can I free the memory used in inner loop's iteration?
My failed attempts to fix the problem
I added the codes marked with (*) into the test case with the following custom keyword:
#keyword
def free_memory(reference):
del reference
Note also that I use RESTInstance library to make the GET call:
Get Sliced Posts
[Documentation] start_index starts from 1 as we fetch from the API now
[Arguments] ${start_index} ${end_index}
GET /posts?_start=${start_index}&_end=${end_index}
${posts} = Output response body
[Return] ${posts}
AFAIK, RESTInstance library keeps a list of instance objects:
https://asyrjasalo.github.io/RESTinstance/#Rest%20Instances
So, this list is growing by adding an instance object per API call. So, I tried:
Reload Library REST # (**)
in the test case, once the iteration with the outermost FOR loop ended. I thought the list would be destroyed & re-created as we reload the library, but the memory consumption kept rising still.
I wrote a query which contains multiple for each statements. The query is taking more than 20 minutes to fetch the data. Is there a way to check what time each loop started and ended. (How much time does each loop takes to execute and also the total time taken to complete the program).
You could do as you ask (just follow JensD's suggestsions) but you would likely be better served to use the profiler. You can easily add profiling for a code snippet:
assign
profiler:enabled = yes
profiler:description = "description of this test"
profiler:profiling = yes
profiler:file-name = "filename.prf"
.
/* this is deliberately awful code that should take a long time to run */
for each orderline no-lock:
for each order no-lock:
for each customer no-lock:
if customer.custNum = order.custNum and orderLine.orderNum = orderLine.orderNum then
. /* do something */
end.
end.
end.
/* end of test snippet */
assign
profiler:enabled = no
profiler:profiling = no
.
profiler:write-data().
You can then load that prf file into an analysis tool. The specifics depend on your development environment - if you are using an up to date version of PSDOE there is a Profiler analyzer included, if not you might want to download ProTop
https://demo.wss.com/download.php and use the simple report included in lib/zprof_topx.p.
Ultimately what you are going to discover is that one or more of your FOR EACH statements is almost certainly using a WHERE clause that is a poor match for your available indexes.
To fix that you will need to determine which indexes are actually being selected and review the index selection rules. Some excellent material on that topic can be found here: http://pugchallenge.org/downloads2019/303_FindingData.pdf
If you don't want to go to the trouble of reading that then you should at least take a look at the actual index selection as shown by:
compile program.p xref program.xref
Do the selected indexes match your expectation? Did WHOLE-INDEX (aka "table scan") show up?
Using ETIME you can initiate a counter of milliseconds. It could be called once or several times to tell how much time has passed since reset.
ETIME(TRUE).
/*
Loop is here but instead I'll insert a small pause.
*/
PAUSE 0.5.
MESSAGE "This took" ETIME "milliseconds" VIEW-AS ALERT-BOX.
Milliseconds might not be useful when dealing with several minutes. Then you can use TIME to keep track of seconds but you need to handle start time yourself then.
DEFINE VARIABLE iStart AS INTEGER NO-UNDO.
iStart = TIME.
/*
Loop is here but instead I'll insert a slightly longer pause.
*/
PAUSE 2.
MESSAGE "This took" TIME - iStart "seconds" VIEW-AS ALERT-BOX.
If you want to keep track of several times then it might be better to output to a log file instead of using a MESSAGE-box that will stop execution until it's clicked.
DEFINE VARIABLE i AS INTEGER NO-UNDO.
DEFINE STREAM str.
OUTPUT STREAM str TO c:\temp\timing.txt.
ETIME(TRUE).
/*
Fake loop
*/
DO i = 1 TO 20:
PAUSE 0.1.
PUT STREAM str UNFORMATTED "Timing no " i " " ETIME "ms" SKIP.
END.
OUTPUT CLOSE.
I have written a recursive code in R.
Before invoking R, I set the stack size to 96 MB at the shell with:
ulimit -s 96000
I invoked R with maximum protection pointer stack size of 500000 with:
R --max-ppsize 500000
And I changed the maximum recursion depth to 500000:
options(expression = 500000)
I both used the binary R package at Arch Linux repositories (without memory profiling) and also a binary compiled by me with memory profiling option. Both are of version 3.4.2
I used two versions of the code with and without gc().
The problem is that R exits the code with "node stack overflow" error while only 16 MB of the 93 MB of total available stack is used and depth is just below one percent of the expressions option of 5e5:
size current direction eval_depth
93388800 16284704 1 4958
Error: node stack overflow
The current stack usage change between the last two iterations were around 10K. The only passed and saved object is a numeric vector of 19 items.
The recursive portion of the code is below:
network_recursive <- function(called)
{
print(Cstack_info())
callers <- list_caller[[called + 1]] # get the callers of the called
callers <- callers[!bool[callers + 1]] # subset for nofriends - new friends
new_friend_no <- length(callers) # number of new friends
print(list(called, callers) )
if (new_friend_no > 0) # if1 still new friends
{
friends <<- friends + new_friend_no # increment friend no
print(friends)
bool[callers + 1] <<- T # toggle friends
sapply(callers, network_recursive) # recurse network control
} # close if1
print("end of recursion")
}
What may be the reason for this stack overflow?
Some notes on the R source code, related to the issue.
The portion of the code that triggers the error is lines 5987-5988 from src/main/eval.c:
5975 #ifdef USE_BINDING_CACHE
5976 if (useCache) {
5977 R_len_t n = LENGTH(constants);
5978 # ifdef CACHE_MAX
5979 if (n > CACHE_MAX) {
5980 n = CACHE_MAX;
5981 smallcache = FALSE;
5982 }
5983 # endif
5984 # ifdef CACHE_ON_STACK
5985 /* initialize binding cache on the stack */
5986 vcache = R_BCNodeStackTop;
5987 if (R_BCNodeStackTop + n > R_BCNodeStackEnd)
5988 nodeStackOverflow();
5989 while (n > 0) {
5990 SETSTACK(0, R_NilValue);
5991 R_BCNodeStackTop++;
5992 n--;
5993 }
5994 # else
5995 /* allocate binding cache and protect on stack */
5996 vcache = allocVector(VECSXP, n);
5997 BCNPUSH(vcache);
5998 # endif
5999 }
6000 #endif
Off the top of my head, I see that you used options(expression = 500000), but the field in the list returned by "options()" is called 'expressions' (with an s). If you typed it in the way you described in your question, then the 'expressions' field remained at 5000, not the 500000 you intended to set it as. So this might be why you maxed out while only using what you thought was 1% of the stack depth.
The node stack has its own limit, which is fixed (defined in Defn.h, R_BCNODESTACKSIZE). If you have a real example where the limit is too small, please submit a bug report, we could increase it or also add a command line option for it. The "node stack" is used by the byte-code interpreter, which interprets byte-code produced by the byte-code compiler. Cstack_info() does not display the node stack usage. The node stack is not allocated on the C stack.
Programs based on deep recursion will be very slow in R anyway as function calls are quite expensive. For practical purposes, when a limit related to recursion depth is hit, it might be better to rewrite the program to avoid recursion rather then increasing the limits.
Just as an experiment one might disable the just-in-time compiler and by that reduce the stress on the node stack. It won't be completely eliminated, because some packages are already compiled at installation by default, including base and recommended packages, so e.g. sapply is compiled. Also, this might on the other hand increase the stress on the recursively eliminated expressions, and the program will run even slower.
When an out-of-memory error is raised in a parfor, is there any way to kill only one Matlab slave to free some memory instead of having the entire script terminate?
Here is what happens by default when an out-of-memory error occurs in a parfor: the script terminated, as shown in the screenshot below.
I wish there was a way to just kill one slave (i.e. removing a worker from parpool) or stop using it to release as much memory as possible from it:
If you get a out of memory in the master process there is no chance to fix this. For out of memory on the slave, this should do it:
The simple idea of the code: Restart the parfor again and again with the missing data until you get all results. If one iteration fails, a flag (file) is written which let's all iterations throw an error as soon as the first error occurred. This way we get "out of the loop" without wasting time producing other out of memory.
%Your intended iterator
iterator=1:10;
%flags which indicate what succeeded
succeeded=false(size(iterator));
%result array
result=nan(size(iterator));
FLAG='ANY_WORKER_CRASHED';
while ~all(succeeded)
fprintf('Another try\n')
%determine which iterations should be done
todo=iterator(~succeeded);
%initialize array for the remaining results
partresult=nan(size(todo));
%initialize flags which indicate which iterations succeeded (we can not
%throw erros, it throws aray results)
partsucceeded=false(size(todo));
%flag indicates that any worker crashed. Have to use file based
%solution, don't know a better one. #'
delete(FLAG);
try
parfor falseindex=1:sum(~succeeded)
realindex=todo(falseindex);
try
% The flag is used to let all other workers jump out of the
% loop as soon as one calculation has crashed.
if exist(FLAG,'file')
error('some other worker crashed');
end
% insert your code here
%dummy code which randomly trowsexpection
if rand<.5
error('hit out of memory')
end
partresult(falseindex)=realindex*2
% End of user code
partsucceeded(falseindex)=true;
fprintf('trying to run %d and succeeded\n',realindex)
catch ME
% catch errors within workers to preserve work
partresult(falseindex)=nan
partsucceeded(falseindex)=false;
fprintf('trying to run %d but it failed\n',realindex)
fclose(fopen(FLAG,'w'));
end
end
catch
%reduce poolsize by 1
newsize = matlabpool('size')-1;
matlabpool close
matlabpool(newsize)
end
%put the result of the current iteration into the full result
result(~succeeded)=partresult;
succeeded(~succeeded)=partsucceeded;
end
After quite bit of research, and a lot of trial and error, I think I may have a decent, compact answer. What you're going to do is:
Declare some max memory value. You can set it dynamically using the MATLAB function memory, but I like to set it directly.
Call memory inside your parfor loop, which returns the memory information for that particular worker.
If the memory used by the worker exceeds the threshold, cancel the task that worker was working on. Now, here it get's a bit tricky. Depending on the way you're using parfor, you'll either need to delete or cancel either the task or worker. I've verified that it works with the code below when there is one task per worker, on a remote cluster.
Insert the following code at the beginning of your parfor contents. Tweak as necessary.
memLimit = 280000000; %// This doesn't have to be in parfor. Everything else does.
memData = memory;
if memData.MemUsedMATLAB > memLimit
task = getCurrentTask();
cancel(task);
end
Enjoy! (Fun question, by the way.)
One other option to consider is that since R2013b, you can open a parallel pool with 'SpmdEnabled' set to false - this allows MATLAB worker processes to die without the whole pool being shut down - see the doc here http://www.mathworks.co.uk/help/distcomp/parpool.html . Of course, you still need to arrange somehow to shutdown the workers.