I want to make a VDP scheduler job in Denodo 8 wait for a certain amount of time. The wait function in the job creation process is not working as expected so I figured I'd write it into the VQL. However when i try the suggested function from the documentation (https://community.denodo.com/docs/html/browse/8.0/en/vdp/vql/stored_procedures/predefined_stored_procedures/wait) the Denodo 8 VQL shell doesn't recognize the function.
--Not working
SELECT WAIT('10000');
Returns the following error:
Function 'wait' with arity 1 not found
--Not working
WAIT('10000');
Returns the following error:
Error parsing command 'WAIT('10000')'
Any suggestions would be much appreciated.
There are two ways of invoking WAIT:
Option #1
-- Wait for one minute
CALL WAIT(60000);
Option #2:
-- Wait for ten seconds
SELECT timeinmillis
FROM WAIT()
WHERE timeinmillis = 10000;
Related
I have the requirement to run dummy jobs for 30 minutes and 60 minutes respectively.
I have tried with --delay 30 in command line jobs, but I did not get the expected delay.
Designating a job as type ‘dummy’ will bypass anything contained within the command line field.
You have two options to create a 30/60minute timer job.
Option a:
Make the job a command line type job and put sleep 1800 or sleep 3600 in the command line field.
Option b:
Make the job a dummy type job and put sleep 1800 or sleep 3600 in either the pre-execution or post-execution fields.
By default the sleep command operates on seconds. For windows you may want to look into using the power shell version which would be powershell.exe -command start-sleep 1800
Use _sleep over sleep instead
Another way to enable a waiting time, either before or after an OS-type Job is by using the pre-execution or post-execution command options, as appropriate.
The use of _sleep is more convenient because it is operating system independent and is provided by the Control-M/Agent, which means that you do not require an extra deployment for that functionality.
There are 2 cyclic jobs A and B such that A is predecessor to B and A runs after every 2 minutes from end while B runs after every 1 minute from end. Problem is job B keeps on re running and failing even after one failure. I thought on adding 'ON STATEMENT * CODE NOTOK DO STOP CYCLIC ' in steps of job B . Will this work? If not,What could be the workaround?
Cheers,
Gourav
CODE = NOTOK is possibly an issue.
ON/DO using the STATEMENT/CODE combination is usually something like -
ON
STATEMENT = *
CODE = your literal error string in here
DO
STOP CYCLIC
e.g. -
ON
STATEMENT = *
CODE = requested file not file found
DO
STOP CYCLIC
the CODE = field should be surrounded by asterix.
If you just want to stop the job once it failed use the below statement in your control-M xml file and reload it or else directly add it to your control-m job. If you have other requirement please let us know.
<ON STMT="*" CODE="NOTOK">
<DOACTION ACTION="SPCYC"/>
</ON>
In MyJob, how do you limit the number of task attempts (if a task fails)?
I have long running tasks (have increased the timeout, accordingly), but I want the job to end after 2 failed attempts at the same task, rather than 4-5.
I couldn't find anything like this in the docs:
http://mrjob.readthedocs.org/en/latest//en/latest/guides/configs-reference.html
For map jobs, you can set mapreduce.map.maxattempts in Hadoop 2. For reduce jobs, set mapreduce.reduce.maxattempts (source).
The equivalents in Hadoop 1 are: mapred.map.max.attempts and mapred.reduce.max.attempts.
If you are using a conf file in MRJob, you can set this as:
runners:
emr:
jobconf:
mapreduce.map.maxattempts: 2
When an out-of-memory error is raised in a parfor, is there any way to kill only one Matlab slave to free some memory instead of having the entire script terminate?
Here is what happens by default when an out-of-memory error occurs in a parfor: the script terminated, as shown in the screenshot below.
I wish there was a way to just kill one slave (i.e. removing a worker from parpool) or stop using it to release as much memory as possible from it:
If you get a out of memory in the master process there is no chance to fix this. For out of memory on the slave, this should do it:
The simple idea of the code: Restart the parfor again and again with the missing data until you get all results. If one iteration fails, a flag (file) is written which let's all iterations throw an error as soon as the first error occurred. This way we get "out of the loop" without wasting time producing other out of memory.
%Your intended iterator
iterator=1:10;
%flags which indicate what succeeded
succeeded=false(size(iterator));
%result array
result=nan(size(iterator));
FLAG='ANY_WORKER_CRASHED';
while ~all(succeeded)
fprintf('Another try\n')
%determine which iterations should be done
todo=iterator(~succeeded);
%initialize array for the remaining results
partresult=nan(size(todo));
%initialize flags which indicate which iterations succeeded (we can not
%throw erros, it throws aray results)
partsucceeded=false(size(todo));
%flag indicates that any worker crashed. Have to use file based
%solution, don't know a better one. #'
delete(FLAG);
try
parfor falseindex=1:sum(~succeeded)
realindex=todo(falseindex);
try
% The flag is used to let all other workers jump out of the
% loop as soon as one calculation has crashed.
if exist(FLAG,'file')
error('some other worker crashed');
end
% insert your code here
%dummy code which randomly trowsexpection
if rand<.5
error('hit out of memory')
end
partresult(falseindex)=realindex*2
% End of user code
partsucceeded(falseindex)=true;
fprintf('trying to run %d and succeeded\n',realindex)
catch ME
% catch errors within workers to preserve work
partresult(falseindex)=nan
partsucceeded(falseindex)=false;
fprintf('trying to run %d but it failed\n',realindex)
fclose(fopen(FLAG,'w'));
end
end
catch
%reduce poolsize by 1
newsize = matlabpool('size')-1;
matlabpool close
matlabpool(newsize)
end
%put the result of the current iteration into the full result
result(~succeeded)=partresult;
succeeded(~succeeded)=partsucceeded;
end
After quite bit of research, and a lot of trial and error, I think I may have a decent, compact answer. What you're going to do is:
Declare some max memory value. You can set it dynamically using the MATLAB function memory, but I like to set it directly.
Call memory inside your parfor loop, which returns the memory information for that particular worker.
If the memory used by the worker exceeds the threshold, cancel the task that worker was working on. Now, here it get's a bit tricky. Depending on the way you're using parfor, you'll either need to delete or cancel either the task or worker. I've verified that it works with the code below when there is one task per worker, on a remote cluster.
Insert the following code at the beginning of your parfor contents. Tweak as necessary.
memLimit = 280000000; %// This doesn't have to be in parfor. Everything else does.
memData = memory;
if memData.MemUsedMATLAB > memLimit
task = getCurrentTask();
cancel(task);
end
Enjoy! (Fun question, by the way.)
One other option to consider is that since R2013b, you can open a parallel pool with 'SpmdEnabled' set to false - this allows MATLAB worker processes to die without the whole pool being shut down - see the doc here http://www.mathworks.co.uk/help/distcomp/parpool.html . Of course, you still need to arrange somehow to shutdown the workers.
I have a script which logs certain data when a benchmark (some c code like matrix multiplication) runs.
I want to first start the log script when benchmark starts, this is easy since I can just start the binary from the log script and then proceed to log the info.
But the real question is when do I stop it? The benchmark can stop at anytime (The log script shouldn't stop the benchmark). How do I get the info/variable which can be used in the log script to stop it when benchmark program stops?
I was thinking if I can use PID of the benchmark, but then thought there should be a better solution than searching and using the PID.
Thanks!
Perhaps you could try something like this:
#!/bin/bash
#
# Your main script
#
# Run your log program in background
your_log_program &
# The PID of last background program
LOGPROGRAMPID=$!
# Install EXIT trap (EXIT is a bash's special event)
trap 'kill -15 $LOGPROGRAMPID; exit 0' EXIT
# In foreground launch your benchmark program
run_your_benchmark_program
# When benchmark program ends your trap will be launched and it will kill your log
# program.