I'm trying to write an autosys JIL which creates the job if it doesn't exist but updates it otherwise. I would've sworn there was a command for that, like 'update_or_create' or something but I can't find it anywhere.
there are three distinct commands: insert, update and delete _job. no "insert-or-update_job"
You would best do this writing a Python script with something like SQLAlchemy/SQLACodeGen - then interrogate the AutoSys database to use the required verb prefix.
Related
I just started looking into Flyway using the command line route, and was wondering if it is possible to reuse the .sql file?
Example:
I have a file called V1__Create_user.sql which has
CREATE USER ${user_name} WITH PASSWORD '${pass}';
GRANT readaccess TO ${user_name};
It looks like I can only use this sql file once, that is when I run below command
flyway -placeholders.userName=test_user -placeholders.pass=test migrate
When I run the above command again with different user_name and password for the placeholders, no changes was made.
So, I was wondering if there's a way to reuse that sql file instead of generating new sql files containing the same sql query over and over ?
The way Flyway works is that it runs the scripts you provide it in the order in which you provide it. It puts a marker in the database showing which scripts it has already run. Then, when you rerun it, it will only run the scripts that it has not yet: run V2__Whatever, V2.2__Something, etc..
You can't go back & modify an existing script and expect the tool to pick up that it has changed and then rerun it. Even if that script is using placeholders.
That doesn't include the repeatable scripts (stuff like view definitions) that run every time you deploy using Flyway. If you had to, you could make that a repeatable script and run it that way. BUT, that will work as long as every deployment is incremental, 1 to 2, 2 to 3, 3 to 4. As soon as you need to deploy from version 1 to 4, you can't pass multiple commands in.
Since every instance of SQL in relational data stores I'm aware of is a declarative language, the best way to deal with it, is to use it as such. Yes, that means stuff like CREATE USER is used over & over. However, since each USER created is unique, that's just how it works.
We are evaluating Airflow for scheduling and data pipeline design. However we are not able to find out how to achieve the following two task:
(1) How to change the DAG schedule through the GUI?
(2) How to achieve the incremental update when the data source is Oracle or MySQl.
This is what we have tried:
(1) We tried changing the schedule of the DAG in the GUI, but looks like that only changes the schedule of that particular instance.
(2) We tried to handle the incremental update programatically by storing the last column value. Is there any other better way of doing incremental update?
1) You can't change the DAG schedule in the GUI, you have to do this in python code when you write the DAG
2) How you do incremental updates is entirely up to you, however I would use a combination of Airflow macros https://airflow.apache.org/code.html#macros and SQL files with JINJA templates https://airflow.apache.org/concepts.html#jinja-templating
Might be worth having a look through the Airflow documentation as it sounds like you're not entirely familiar with its concepts.
I know that i can do that with CRON but there are a lot of issues with these method. I have a Entity "Events" and when $eventDate <= Date (NOW) it will be deleted dynamically. I'm using PHP Framework Symfony2.
Why don't you want to use cron? Another approach is using queues that allow to use delayed messages (something like Beanstalkd, it has a lot of bundles for Symfony) -- but obviously it's more complicated than cron.
What issues are with a cron? You can create a symfony command that do what you want and call it from a event listener on the request or from a cron.
You can make a symfony command that delete the Event when $eventDate <= Date (NOW) and you can add a cron to run the command every X time
Our common workflow when creating a new sql migration script is
to write and execute every single statement in the developers local datatase schema. When finished, it's checked into the source control system.
Problem is: at the database scheme of the creating developer, the script is already "executed". For scripts not beeing reentrant - it would be convenient to have s.th. like Dbmaintain's maven task "markDatabaseAsUpToDatemaven".
Does Flyway have s.th. equivalent?
P.S.: Our current workflow (as a workaround) is as follows:
"mvn flyway:migrate" this file as an empty file (without content - so it never fails).
put the sql statments in, save & "migrate" again.
"mvn flyway:repair"
Thanks
While the workflow you describe sounds like it can do the job, you can achieve the same in a simpler and fully automated way: set cleanOnValidationError to true (on for dev!) and everytime the script changes, the DB gets recreated.
More info: http://flywaydb.org/documentation/maven/migrate.html#cleanOnValidationError
Is it possible to instruct command line Flyway to write down the needed SQL update commands in a file, instead of executing them in the Database?
This is not supported yet. Please +1 https://github.com/flyway/flyway/issues/87