JIL script allows the passing of envvars to a command as below.
My question is how to specify the envars in a file on the file system rather than in the JIL script?
insert_job: unix_job
job_type: CMD
machine: unixprod
command: /home/scripts/pay
envvars: NAME="user 1"
envvars: JOB=PAYROLL
envvars: PWD=/usr/scripts/dailyrun
Set a profile attribute in your JIL that points to a script file which loads your environment variables.
profile: /your/path/here/config.ksh
Inside config.ksh:
#!/usr/bin/ksh
export NAME="user 1"
export JOB=PAYROLL
export PWD=/usr/scripts/dailyrun
Related
In my airflow task, I am creating a file using open() method in airflow dag and writing records into it. Then sending it with a mail within same task. Will it get deleted automatically or will exists into the dag?
filename = to_report_name(context)+'_'+currentNextRunTime.strftime('%m.%d.%Y_%H-%M')+'_'+currentNextRunTime.tzname()+'.'+extension.lower()
with open(filename, "w+b") as file:
file.write(download_response.content)
print(file.name)
send_report(context,file)
The file will not be deleted automaticly. The code you execute is pure Python if you want the file to be deleted once the operation is done then use tempfile module which gurentee the file will be deleted once it's closed. Example:
import tempfile, os
with tempfile.NamedTemporaryFile() as file:
os.rename(file.name, '/tmp/my_custom_name.txt') # use this if you want to rename the file
file.write(...)
I'm running a command to (re)create a remote file. I'd like to see a difference between the old and the new versions of the file -- and for the task to set changed: false, if there are no differences.
Is there a way to do this without doing it all by hand -- with multiple tasks (creating a backup, running the command, diff-ing the two, etc.)?
Ok, apparently, this is not currently possible. The best I could come up with was:
Run the command with output sent simply to stdout, rather than directly to a file. This is possible with many commands (such as with -o /dev/stdout), allows to declare it with changed_when: False and check_mode: ansible_check_mode. Save the results:
- name: Run the command to obtain content
command: foo -
register: foo
changed_when: False
check_mode: ansible_check_mode
...
Use the copy module to (re)write the file with the captured output:
- name: Write the content into file
copy:
dest: /where/ever/it/was
content: '{{ foo.stdout }}'
...
This is Ok for when your files aren't too large -- requiring "only" one additional task... One per file.
For everything else, I created this enhancement-request.
Input:
I want to write a script name nginx-conf-update.sh.
when is run
$./nginx-conf-update.sh 3
Output:
I'm a newbie in Oozie and I've read some Oozie shell action examples but this got me confused about certain things.
There are examples I've seen where there is no <file> tag.
Some example, like in Cloudera here, repeats the shell script in file tag:
<shell xmlns="uri:oozie:shell-action:0.2">
<exec>check-hour.sh</exec>
<argument>${earthquakeMinThreshold}</argument>
<file>check-hour.sh</file>
</shell>
While in Oozie's website, writes the shell script (the reference ${EXEC} from job.properties, which points to script.sh file) twice, separated by #.
<shell xmlns="uri:oozie:shell-action:0.1">
...
<exec>${EXEC}</exec>
<argument>A</argument>
<argument>B</argument>
<file>${EXEC}#${EXEC}</file>
</shell>
There are also examples I've seen where the path (HDFS or local?) is prepended before the script.sh#script.sh within the <file> tag.
<shell xmlns="uri:oozie:shell-action:0.1">
...
<exec>script.sh</exec>
<argument>A</argument>
<argument>B</argument>
<file>/path/script.sh#script.sh</file>
</shell>
As I understand, any shell script file can be included in the workflow HDFS path (same path where workflow.xml resides).
Can someone explain the differences in these examples and how <exec>, <file>, script.sh#script.sh, and the /path/script.sh#script.sh are used?
<file>hdfs:///apps/duh/mystuff/check-hour.sh</file> means "download that HDFS file into the Current Working Dir of the YARN container that runs the Oozie Launcher for the Shell action, using the same file name by default, so that I can reference it as ./check-hour.sh or simply check-hour.sh in the <exec> element".
<file>check-hour.sh</file> means "download that HDFS file -- from my user's home dir e.g. hdfs:///user/borat/check-hour.sh -- into etc. etc.".
<file>hdfs:///apps/duh/mystuff/check-hour.sh#youpi</file> means "download that HDFS file etc. etc., renaming it as youpi, so that I can reference it as ./youpi or simply youpi in the element".
Note that the Hue UI often inserts unnecessary # stuff with no actual name change. That's why you will see it so often.
I have the following at the end of my script:
export PATH=/usr/openwin/bin:/opt/plat/AUTOSYS/4.0/autosys/bin:/usr/kerberos/bin::/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/opt/netezzaClient/bin:/xenv/ant/X/1.8.0/bin:/export/opt/jdk/1.6.0_16/bin:$PATH
export JAVA_HOME=/export/opt/jdk/1.6.0_16
echo "END PATH - $PATH"
which prints this.
END PATH - /usr/openwin/bin:/opt/plat/AUTOSYS/4.0/autosys/bin:/usr/kerberos/bin::/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/opt/netezzaClient/bin:/xenv/ant/X/1.8.0/bin:/export/opt/jdk/1.6.0_16/bin:/opt/edtsesn/share/bin:/xenv/cvs/sun4/5.6p4/1.10/bin:/xenv/rationalrose/sun4/5.x/6.0.9242/rose/bin:/opt/netscape/4.70_B2/bin:/opt/SCssh/3.4_C0/bin:/opt/PDolvwm/bin:/usr/kerberos/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin:/usr/ucb:/usr/bin:/usr/local/etc:/bin:/usr/local/bin:/etc:/software/scripts:/usr/5bin:/usr/demo:/usr/openwin/bin:/usr/tran/sparc/lib:/usr/ccs/bin:/opt/sybase/1192/bin:/tmp/wm40824:/opt/edtsdba/bin:/xenv/scripts/bin:/xenv/workshop/sun4/5.8mu4/6.1a/bin:/home/pj03962/1192/bin:/home/pj03962/1192/bin:/xenv/java/X/1.6.0_11/bin:/xenv/cvs/:/xenv/java/X/1.6.0_11/bin:/xenv/cvs/sun4/5.6p4/1.10/bin:/xenv/ant/sun4/5.x/1.6.2/bin:/opt/SCssh/3.7.1_C0/bin:/opt/xemacs/bin:/home/pj03962/125/OCS-12_5/bin:/home/pj03962/125/125/bin:/opt/perforce/bin:/opt/netezzaClient/bin:/opt/netezzaClient/bin
yet
bash-3.00$ env $PATH
env: /usr/kerberos/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/X11R6/bin: No such file or directory
No such file comes because a /home/usr/ folder does not exist for my account. But this still does not shed any light as to why it has not added the other values to the path variable.
I guess you've called your script without sourcing it.
For example, if your script is named "myscript.sh", you may have called "./myscript.sh" or "bash myscript.sh". Your modifications of env var inside the script won't leak out the script, you need to source it (call it with 'source' or '.' first).
eg:
. ./myscript.sh
The changes in myscript.sh will modify your current environment.
For the "env $PATH" : I think it's an error, since you're trying to run the command in the "PATH" variable. Which doesn't exist (/usr/kerberos/bin:/bin:... isn't the name of an existing file on your system !).
You should use instead: echo $PATH