I have file watcher which will be getting terminated as set by term_run_time. But the waiting job (waiting for file watcher to success) still be in activated state, I need to terminate waiting jobs also.
Text_Box
Job_A
File_watcher_A
As defined above box name is Text_Box , job is Job_A, File watcher is file watcher. Job_A has condition Job_A -> Condition (Success(File_watcher_A))
Once file_Watcher_A terminated Job_A also should terminate, can you please suggest how to achieve it.
Set the box_terminator attribute in your File Watcher job definition to true. This will terminate the box when the File Watcher terminates.
Next, set the job_terminator attribute in Job_A to true. This will terminate Job_A when the box terminates.
Related
I run a python script through Airflow.
The script gets a source file from S3. I want to set the script to mark the task as 'skipped' when there is no file in the bucket.
from airflow.exceptions import AirflowSkipException
if len(file_list) == 0 : #This means there's no file in the bucket. I omitted some codes.
print('The target file does not exist. Skipping the task...')
raise AirflowSkipException
However, Airflow still marks this task 'failure' when the file doesn't exist.
Am I missing anything? Should I add something to DAG too?
I think you should call raise AirflowSkipException() with the round parentheses at the end, otherwise you are not raising an instance of the AirflowSkipException class, but the class itself which I guess is creating the error that sets the task to failed.
I am trying to understand how, for the above command, the shell arranges for output re-direction when the process associated with the shell itself is replaced by that of echo? Would very much appreciate any help.
Regards
Rupam
Output redirection of an external command is achieved by manipulating file descriptor, typically involving dup2, a system call that assigns a file descriptor to an existing open file. In this case, the standard output of the process that executes echo is instructed to point to the target output file.
The shell does a close equivalent of the following steps:
create a copy of the current process by invoking fork(); the subsequent steps happen in the child process. (This step is omitted when the command is invoked with exec, as in your example.)
open test for writing and remember the file descriptor
make file descriptor 1 equal to the descriptor, by calling dup2(fd, 1)
call execlp or similar to replace current process execution with execution of echo
The echo command simply writes to standard output, a fancy name for file descriptor number 1. It doesn't care if that ends up writing to a TTY, to a file, or to another program. The above steps make sure that file descriptor 1 really points to the open file test.
There is a couple of options while re-running a workflow via Oozie command line.
oozie.wf.rerun.failnodes
oozie.wf.rerun.skip.nodes
Option 1 works fine, however, while re-running workflow with option 2, it throws error E0404.
oozie job -oozie http://<url>/oozie -Doozie.wf.rerun.skip.nodes=node1,node2 -rerun WFID
Error: E0404 : E0404: Only one of the properties are allowed [oozie.wf.rerun.skip.nodes OR oozie.wf.rerun.failnodes]
However, below works fine.
oozie job -oozie http://<url>/oozie -Doozie.wf.rerun.failnodes=true -rerun WFID
Everytime an oozie job is executed in a rerun mode, it will try to reuse the previous run's conifg file. you can however pass additional properties to it using -D option and thats how we pass oozie.wf.rerun.failnodes and oozie.wf.rerun.skip.nodes.
If you have execueted your job in rerun mode already once with oozie.wf.rerun.failnodes=true once, then in your next run you cannot use
oozie job -oozie http://<url>/oozie -Doozie.wf.rerun.skip.nodes=node1,node2 -rerun WFID
because when its trying to reuse config file, oozie.wf.rerun.failnodes property is already existing in its properties and that's when oozie tries to throw an error like you have faced.
you could start the workflow from the beginning by giving oozie.wf.rerun.failnodes=false property...thats what i do when i reran a job already, this is similar to skip node oozie.wf.rerun.skip.nodes=, but we cant skip anything
For some reason phpagi script randomly stop's in the middle. This only happens once every 20-50 calls. I was able to notice several such 'fails' realtime in asterisk CLI. No error's were displayed. Script sent several verbose messages until just stopped.
Script is used for billing, it has several while's and sql queries. max execution time set to 30seconds.
I didn't find any errors in the /var/log/messages or /var/log/asterisk/messages
Asterisk 1.6.2.24
PHP 5.1.6 (cli) (built: Jun 27 2012 12:21:13)
[context-x]
exten => 1,1,Dial(SIP/XXXXXX)
exten => h,1,AGI(script.php)
Any ideas why it could just stop for random calls?
Thanks.
UPDATE:
I've just noticed that when the problem occurs AGI returns 4:
-- <SIP/xxxxxx-00000185>AGI Script script.php completed, returning 4
What's wrong?
From what I'm hearing from your symptoms, it doesn't sound like it's specifically an Asterisk issue, but, could be an issue with your script. Here's a number of tips regarding how I would go about debugging an application without any knowledge of what's happening behind the scenes of the AGI script itself...
Firstly, AGI Script script.php completed, returning 4 means that your script is exiting without a clean exit status code. returning 0 is what you'd like to see. You can see the last exit status code at your bash prompt by running a script and then checking the status code with the $? variable. Like so:
[user#host ~]$ ./script.php
[user#host ~]$ echo $?
0
That's what Asterisk is telling you what happened to your script. Anything non-zero is "something's wrong here". Generally, you can customize these to your liking, so specifically 4, I'm not sure.
One thing you'll want to do is turn on agi debugging like so:
host*CLI> agi set debug on
Then run your script, and see if you can find that your php script is spitting out any errors.
Another recommendation that I would have is to make sure your php is logging to your syslog so that you can find errors in /var/log/messages. You can do this by setting in your /etc/php.ini this line:
error_log = syslog
Lastly, to try to replicate the error, I would suggest using a development box, and originate a bunch of calls for yourself, and build a script to create a bunch of "call files".
Here's a call file to get you started:
Channel: LOCAL/100#mycontext
MaxRetries: 2
RetryTime: 60
WaitTime: 30
Application: Wait
Data: 30
When you've created a file, move it to /var/spool/asterisk/outgoing (Moving is important, you want the pointer move, because asterisk may pick up the file too soon if you're writing to it in that directory first).
You can also originate a call on an extension from the CLI:
host*CLI> channel originate LOCAL/100#mycontext application Wait 5
You may also want to use other options in the call file, such as CallerID: John Doe <8005551212> in order to feed interesting data to your AGI application while you create tests to replicate the issue.
I have written an R script that pulls some data from a database, performs several operations on it and post the output to a new database.
I would like this script to run every day at a specific time but I can not find any way to do this effectively.
Can anyone recommend a resource I could look at to solve this issue? I am running this script on a Windows machine.
Actually under Windows you do not even have to create a batch file first to use the Scheduler.
Open the scheduler: START -> All Programs -> Accesories -> System Tools -> Scheduler
Create a new Task
under tab Action, create a new action
choose Start Program
browse to Rscript.exe which should be placed e.g. here:
"C:\Program Files\R\R-3.0.2\bin\x64\Rscript.exe"
input the name of your file in the parameters field
input the path where the script is to be found in the Start in field
go to the Triggers tab
create new trigger
choose that task should be done each day, month, ... repeated several times, or whatever you like
Supposing your R script is mytest.r, located in D:\mydocuments\, you can create a batch file including the following command:
C:\R\R-2.10.1\bin\Rcmd.exe BATCH D:\mydocuments\mytest.r
Then add it, as a new task, to windows task scheduler, setting there the triggering conditions.
You could also omit the batch file. Set C:\R\R-2.10.1\bin\Rcmd.exe in the program/script textbox in task scheduler, and give as Arguments the rest of the initial command: BATCH D:\mydocuments\mytest.r
Scheduling R Tasks via Windows Task Scheduler (Posted on February 11, 2015)
taskscheduleR: R package to schedule R scripts with the Windows task manager (Posted on March 17, 2016)
EDIT
I recently adopted the use of batch files again, because I wanted the cmd window to be minimized (I couldn't find another way).
Specifically, I fill the windows task scheduler Actions tab as follows:
Program/script:
cmd.exe
Add arguments (optional):
/c start /min D:\mydocuments\mytest.bat ^& exit
Contents of mytest.bat:
C:\R\R-3.5.2\bin\x64\Rscript.exe D:\mydocuments\mytest.r params
Now there is built in option in RStudio to do this, to run scheduler first install below packages
install.packages('data.table')
install.packages('knitr')
install.packages('miniUI')
install.packages('shiny')
install.packages("taskscheduleR", repos = "http://www.datatailor.be/rcube", type =
"source")
After installing go to
**TOOLS -> ADDINS ->BROWSE ADDINS ->taskscheduleR -> Select it and execute it.**
Setting up the task scheduler
Step 1) Open the task scheduler (Start > search Task Scheduler)
Step 2) Click "Action" > "Create Task"
Step 3) Select "Run only when the user is logged on", uncheck "Run with highest priveledges", name your task,
configure for "Windows Vista/Windows Server 2008"
Step 4) Under the "Triggers" tab, set when you would like the script to run
Step 5) Under the "Actions" tab, put the full location of the Rscript.exe file, i.e.
"C:\Program Files\R\R-3.6.2\bin\Rscript.exe" (include the quotes)
Put the name of your script with with -e and source() in arguments wrapping it like this:
-e "source('C:/location_of_my_script/test.R')"
Troubleshooting a Rscript scheduled in the Task Scheduler
When you run a script using the Task Scheduler, it is difficult to troubleshoot any issues because you don't get any error messages.
This can be resolved by using the sink() function in R which will allow you to output all error messages to a file that you specify. Here is how you can do this:
# Set up error log ------------------------------------------------------------
error_log <- file("C:/location_of_my_script/error_log.Rout", open="wt")
sink(error_log, type="message")
try({
# insert your code here
})
The other thing that you will have to change to make your Rscript work is to specify the full file path of any file paths in your script.
This will not work in task scheduler:
source("./functions/import_function.R")
You will need to specify the full file path of any scripts you are sourcing within your Rscript:
source("C:/location_of_my_script/functions/import_function.R")
Additionally, I would remove any special characters from any file paths that you are referencing in your R script. For example:
df <- fread("C:/location_of_my_data/file#2342.csv")
may not run. Instead, try:
df <- fread("C:/location_of_my_data/file_2342.csv")
Changing windows passwords
Beware: Changing windows passwords will pause your task scheduler script(s). You will need to log back into the task scheduler and enter your password to get them started again.
I set up my tasks via the SCHTASKS program. For running scripts on startup, you would write something along the lines of
SCHTASKS /Create /SC ONSTART /TN MyProgram /TR "R CMD BATCH --vanilla d:\path\to\script.R"
See this website for more details on SCHTASKS. More details at Microsoft's website.
You can use Windows Task Scheduler.
After following any combination of these steps and you receive the "Argument Batch Ignored" error after R.exe runs, try this, it worked for me.
In Windows Task Scheduler:
Replace BATCH "C:\Users\desktop\yourscript.R"in the arguments field
with
CMD BATCH --vanilla --slave "C:\Users\desktop\yourscript.R"