I understand that in drafting a autosys jil file, I can use "update_job: job_name" to change the value of one or more job's attribute. For example:
update_job: abc_job
start_time: some_new_time
my question is, now I'd like to add a new attribute,
run_calendar
to abc_job, can I just do:
update_job: abc_job
run_calendar: my_cal
I know I could have used a combine of:
delete_job: abc_job
insert_job: abc_job
with new attributes added, but since I have quite a few updates to do and each job has a certain size of body, I prefer to a more succinct solution.
You can still use update on your abc_job as follows:
update_job: abc_job
run_calendar: my_cal
However if you already used days_of_week you need to add that to your JIL as well so it would look like:
update_job: abc_job
run_calendar: my_cal
days_of_week:
Leaving the days_of_week empty.
Dave
Related
I'm adding liquibase to an existing project where tables and data already exist. I would like to know how I might limit the scope of changelogSync[SQL] to a subset of available changes.
Background
I've run liquibase generateChangeLog to capture the current state and placed this into say src/main/resources/db/changelog/changes/V2021.04.13.00.00.00__init01.yaml.
I've also added another changeset to cover some new requirements in a new file. Let's call it src/main/resources/db/changelog/changes/V2021.04.13.00.00.00__new-feature.yaml.
I've added a main changelog file src/main/resources/db/changelog/db.changelog-master.yaml with the following contents:
databaseChangeLog:
- includeAll:
path: changes
relativeToChangelogFile: true
I now want to ensure that when I run liquibase changelogSync[SQL] against a particular version of the db that the scope is limited to the first changelog init01, thereby allowing from that point on a liquibase update or updateToTag et al, to continue with changes following init01.
I'm surprised to see that the changelogSync[SQL] commands don't seem to offer some way (that I can see from the docs for how to do this.
Besides printing the SQL and manually changing it, is there something I've missed? Any suggested approaches welcome. Thanks!
What about changelogSyncToTagSQL ?
Wouldn't it cover your needs?
Or maybe you could try changelogSyncSQL with additional parameters "label" or and "context" ?
changelogSyncToTagSQL
context
labels
As it stands, the only solution I've found is to generate the SQL and then manually edit its contents to filter the change sets which don't correspond to your current schema.
Then you can apply the sql to the db.
I have a DAG with one DataflowTemplateOperator that can deal with different json files. When I trigger the dag I pass some parameters via {{dag_run.conf['param1']}} and works fine.
The issue I have is trying to rename the task_id based on param1.
i.e. task_id="df_operator_read_object_json_file_{{dag_run.conf['param1']}}",
it complains about only alphanumeric characters
or
task_id="df_operator_read_object_json_file_{}".format(dag_run.conf['param1']),
it does not recognise dag_run plus the alpha issue.
The whole idea behind this is that when I see at the dataflow jobs console and job has failed I know who the offender is based on param1. Dataflow Job names are based on task_id like this:
df-operator-read-object-json-file-8b9eecec
and what I need is this:
df-operator-read-object-param1-json-file-8b9eecec
Any ideas if this is possible?
There is no need to generate new operator per file.
DataflowTemplatedJobStartOperator has job_name parameter which is also templated so can be used with Jinja.
I didn't test it but this should work:
from airflow.providers.google.cloud.operators.dataflow import DataflowTemplatedJobStartOperator
op = DataflowTemplatedJobStartOperator(
task_id="df_operator_read_object_json_file",
job_name= "df_operator_read_object_json_file_{{dag_run.conf['param1']}}"
template='gs://dataflow-templates/your_template',
location='europe-west3',
)
From the documentation, config-default.xml must be presented in the workflow workspace.
- /workflow.xml
- /config-default.xml
|
- /lib/ (*.jar;*.so)
The problem
I've created a custom Oozie action and try to add default values for retry-max and retry-interval to all the custom actions.
So my workflow.xml will look like this:
<workflow-app xmlns="uri:oozie:workflow:0.3" name="wf-name">
<action name="custom-action" retry-max="${default_retry_max}" retry-interval="${default_retry_interval}">
</action>
config-default.xml file contains the values of default_retry_max and default_retry_interval.
What I've tried
Putting config-default.xml to every workflow workspace. This works, but the problem is there will be this file everywhere.
Setting oozie.service.LiteWorkflowStoreService.user.retry.max and oozie.service.LiteWorkflowStoreService.user.retry.inteval also works, but it would affect all action types.
I've also looked at Global Configurations, but it doesn't solve this problem.
I think there should be a way to put config-default.xml to oozie.libpath and only those workflows that use this libpath will be affected.
AFAIK, there is unfortunately no clean way to do it.
You might be interested in this recently created feature request: https://issues.apache.org/jira/browse/OOZIE-3179
The only thing that worked for me was to begin the workflow with a shell step that uses a script stored in hdfs. This script holds the centralized configuration. The script would look like this:
#!/bin/sh
echo "oozie.use.system.libpath=true"
echo "hbase_zookeeper_quorum=localhost"
.. etc other system or custom variables ..
Yes, the script simply prints the variables to the stdout.
Let's say the shell step action is called "global_config". All following steps are able to get the variables using following syntax:
${wf:actionData('global_config')['hbase_zookeeper_quorum']}
HTH...
It looks like salt.states.file doesn't like wildcards in the ID or name. I'm trying to manage permissions of a consistent subdirectory within a variable parent, e.g., I want to manage permissions on 'poo'. 'poo' is consistent (with lots of fiber), but its parent directory can be variable:
/massive/poo
/lotso/poo
/runny/poo
/Manny Pacquiao vs Floyd Mayweather/poo
You get the idea.
It's okay for the parent to have the default permissions, but I want to manage the subdirectory. Something that
chmod 775 /*/poo
would take care of.
Is there a way to do this with salt states?
You can run specify a cmd.run for this.
Test_command:
cmd.run:
- name: chmod 755 /*/poo
- unless: (insert some logic here)
I include the insert statement because it's "best" to always check if the command needs to run unless you want it to run EVERY TIME, unless you want slow performance...
Hope this helps.
I am trying to have the template of Phabricator's arc diff something similar to this:
Some title
Summary:
Reviewers:
TaskID: (← missing!)
Subscribers:
By default, Phabricator doesn't come with a "TaskID field". Is there any buildin support, or how do I achieve this?
Basically I want to associate a Task ID field with the actual Manifest Task's ID.
Use Ref T123 (links) or Fixes T123 (links, and closes when pushed) in the summary, or add Maniphest Tasks: T123 on its own line. This field is builtin, but the field is not shown by default because most users use the Ref/Fixes shorthand.
You could implement shouldAppearInCommitMessageTemplate() on DifferentialManiphestTasksField to force it to appear on the template. This may be a configurable option after https://secure.phabricator.com/T6030