The Apache Airflow project has seemingly forced the TriggerDagRunOperator conf to allow JSON only. This is different than how it was previously where I was allowed to pass class instances without any special handling.
https://github.com/apache/airflow/blob/master/airflow/operators/dagrun_operator.py#L94
https://github.com/apache/airflow/blob/master/airflow/api/common/experimental/trigger_dag.py#L65
These are the 2 snippets that enforce this.
Is anyone facing this issue as well? If so, how are you working around this limitation?
I'm currently bypassing this issue by creating a custom operator based on the operator before these changes.
Related
I'm using airflow_docker and it seems it does not come with an s3KeySensor. It does come with a Sensor Class. How can I create my own custom s3KeySensor? Do I just have to inherit from Sensor and overwrite the poke method? Can I literally just copy the source code from the s3KeySensor? Here's the source code for the s3KeySensor
The reason I am using airflow docker is that it can run in a container and I can pass in aws role creds to the task container so that it has the proper and exact permissions to do an action other than using the worker container's role permissions.
I would recommend upgrading to the latest version Airflow or at least start at Airflow 2.0.
In Airflow 2.0, a majority of the operators, sensors, and hooks that are not considered core functionality are categorized and separated into providers. Amazon has a provider, apache-airflow-providers-amazon, that includes the S3KeySensor.
You can also install backport providers so the one you would need is apache-airflow-backport-providers-amazon if you want to stay on Airflow 1.10.5 (based on the links you sent).
We want to avoid users to manually editing or adding new variables/connections from Airflow GUI in Production. Currently, we are using JSON files which loads all the connections and variables for Airflow.
Can experts guide me on how to achieve this?
Sergiy's right, you will need to start looking at Airflow's Role Based Access (RBAC) documentation and implement from there. Here's a place to start: https://airflow.readthedocs.io/en/latest/howto/add-new-role.html
I understand that Incremental strategy is defined at Schema level.
I have an incremental load already set-up and working fine, a rule in the scheduler launches it every hour. (Rule 1)
Now, I want to generate a full reload of the schema, but based on another trigger data. Here are the rules I want to manage :
Is it possible to setup the Rule 2? and how to do that?
This is currently not supported. You would need to setup an external process for that and perform a full load using a command as described in this page: www.
Symfony introduced a new Dotenv component since Symfony 3 which allows us to handle environment variables as application parameters. This looks really nice and it's the best practice to follow according to 12factor app manifesto.
Now, regarding Symfony 4 they went further by pushing forward this practice and that's why I started using environment variables via the .env file.
And then I wanted to deploy and I realized that the .env file must not be persisted on the server as it would be the same as having a parameters.yml file.
So I've been digging into the documentation a bit and I found this article which explains that we can directly create environment variables via some webserver directives. That's great for code being executed via FPM but it does not tell us how to handle environment variables when running a command via the CLI for instance.
How can I achieve this ?
Should there be an equivalent of a .env file stored somewhere? But then parameters would be duplicated ?
I'm welcoming any help ;)
Finally had the time to check the link Neodan posted and everything is in there!
So for those of you wondering what to do, simply edit the /etc/environment file and add your variables. Then reboot your server and all your processes will have access to these variables.
I guess that's the simplest solution. The only drawback of this method is that these variables are available by any process / users but that's ok as far as I'm concerned.
If you want a more secure solution I suppose that you could, as I stated before, configure your webserver to add environment variables and export them via your .bash_profile or .bashrc file but be careful about how you start your shell (when deploying your application for instance). It's more complicated to maintain and prone to errors I'd say.
N.B.: You also might want to be careful about how you name your variables to prevent collisions.
I'm trying to set up one of my AX environments to have an XPO imported whenever the server process is started up. Because the environment is updated from production on a regular basis, and the code needs to be unique to this (non-production) environment, it seems the best option would be to use the AOTImport command on startup. However, I'm having trouble identifying the exact syntax/setup to get this to work.
Looking through the system code, it seems the syntax should be aotimport_[path to file]. Is this correct? The server does not seem to complain about this command, but the file in question does not get imported. I have also tried variations on this command, but have not seen it work.
I supose you're trying to execute a command on SysStartupCmd classes. If so, this methods are fired when AX client starts, not the AOS. It's docummented on this page:
http://msdn.microsoft.com/en-us/library/aa569641(v=ax.50).aspx
If you want to automate this import it can be done scheduling an execution of the AX client (ax32.exe) on your build workflow that run the import (it's recommended to run a full compilation after importing). This is discussed on other questions here on SO.