Just like it is possible to set connections via an environment variable following the name AIRFLOW_CONN_{conn_id}, is there a way to set pools?
This is so I can set up a local Docker test environment with all configurations populated.
Related
I'm using AWS to start up an Apache+Wordpress task. The WP username and password values are stored in AWS ParameterStore and passed to the Fargate container as environment variables. I can shell into the Fargate container (as root) via the aws cli utility and verify the variables from ParameterStore are in the root environment and correct.
The Apache/WP process (www-data user) is configured to retrieve the WP_PASSWORD variable from it's environment via wp-config.php and getenv('WP_DB_PASSWORD') PHP function. Previously (on-prem), this was achieved by using /etc/apache2/envvars however I'd rather not have the cleartext DB pass laying on disk like this.
How do I get the values from ParameterStore passed to the www-data environment without writing a file somewhere?
I am trying to connect to mysql database using MySqlHook. Under Admin -> Connections I have defined a new connection of the type mysql with the name myappname_db. I have used this in my code as drupalHook = MySqlHook(conn_name_attr='myappname_db')
However, when I run the dag in local, I see that it picks up the built in default credential from Admin -> Credentials i.e. mysql_default instead of myappname_db
To rectify this do I need to update any setting in airflow.cfg or any other config.
Thanks.
The MySqlHook uses an attribute mysql_conn_id to store the connection id.
conn_name_attr refers to the name of the attribute (mysql_conn_id in this case), which is fetched dynamically in the MySqlHook.
I am trying to get my environment variables in one of my pages, but it is always returning undefined. I have no issues with accessing the variables in api folder but in pages/page.tsx it doesn't return the variables.
I access my variables using
const SECRET = process.env.SECRET from my .env file.
How can I fix this issue?
I believe the preferred way to implement what you're trying to do is to use env variables on the server side within getServerSideProps()/getStaticProps() methods. This should work as expected without any tricks.
But if you want to access env variables on the client you have to prefix your variable with NEXT_PUBLIC_
Please refer to official docs:
By default all environment variables loaded through .env.local are only available in the Node.js environment, meaning they won't be exposed to the browser.
In order to expose a variable to the browser you have to prefix the variable with NEXT_PUBLIC_. For example:
NEXT_PUBLIC_ANALYTICS_ID=abcdefghijk
Environment
ASP.NET WebForms app over IIS
Docker container host
AWS ECS hosting platform
Each client hosting its own copy of the app with private database connection string
Background
In the non-docker environment, each copy is a virtual directory under IIS, and thus have their own individual web.config pointing to dedicated databases. The underlying codebase is the same for each client, with no client-specific customization involved. The route becomes / here.
In the docker environment (one container per client), each copy goes over as a central root application.
Challange
Since the root image is going to be the same, how to have the web.config overridden for each client deployment.
We shouldn't create multiple images (one per client) as that will mean having extra deployment jobs and losing out on centralization. The connection strings should ideally be stored in some kind of dictionary storage applicable at ECS level which can provide client-specific values upon loading of corresponding containers.
Presenting the approach we used to solve this issue. Hope it may help others struck in similar cases.
With the problem statement tied to having a single root image and having any customization being applied at runtime, we knew that there needs to be a transformation of web.config at time of loading of the corresponding containers.
The solution was to use a PowerShell script that will read the web.config and get replace the specific values which were having a custom prefix embedded to the key. The values got passed from custom environmental variables within ECS and the web.config also got updated to have the keys with the prefix added.
Now since the docker container can have only a single entry point, a new base image was created which instantiated an IIS server and called a PowerShell script as startup. The called script called this transformation script and then set the ServiceMonitor on the w3cwp.
Thanks a lot for this article https://anthonychu.ca/post/overriding-web-config-settings-environment-variables-containerized-aspnet-apps/
I would use environment variables as the OP suggests for this with a start up transform, however I want to make the point that you do not want sensitive information in ENV variables, like DB passwords, in your ECS task definition.
For that protected information, you should use ECS secrets coupled with Parameter Store in Systems Manager. These values can be stored encrypted in the Parameter Store (using a KMS key) and the ECS Agent will 'inject' them as ENV variables on task startup.
For me, to simplify matters, I simply use secrets for everything although you can choose to only encrypt the sensitive information and leave the others clear.
I dynamically add the secrets for the given application into my task definitions at deploy time by looking up the 'secrets' for the given app by 'namespace' (something that Parameter Store supports). Then, if I need to add a new parameter, I can just add a new secret to the store in the given namespace and re-deploy the app. It will pick up and inject into the task definition any newly defined secrets automatically (or remove ones that have been retired).
Sample ruby code for creating task definition:
params = ssm_client.get_parameters_by_path(path: '/production/my_app/').parameters
secrets = params.map{ |p| { name: p.name.split("/")[-1], value_from: p.arn } }
task_def.container_definitions[0].secrets = secrets
This last transform injects the secrets such that the secret 'name' is the ENV variable name... which ends up looking like this:
"secrets": [
{
"valueFrom": "arn:aws:ssm:us-east-1:578610029524:parameter/production/my_app/DB_HOSTNAME",
"name": "DB_HOSTNAME"
},
{
"valueFrom": "arn:aws:ssm:us-east-1:578610029524:parameter/production/my_app/DB_PASSWORD",
"name": "DB_PASSWORD"
}
You can see there are no values now in the task definition. They are retrieved and injected when ECS starts up your task.
More information:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html
I'м newbie on Docker.
In the dashboard, I deploy the Wordpress and the Mariadb in different layers. In a container with Wordpress, I made a connection with the Mariadb.
What variables should I edit in the WordPress container, what would it be initialized with the Mariadb database?
The Links section is intended for establishing connection between your Docker containers (obviously, they should be placed at different layers inside a single environment for that).
After such a connection is set, a container will be able to work with environment variables of the linked template (herewith, the imported properties will have a special prefix to be easily separated from this container’s native ones).
To set a new link, click the Add button and fill in the appeared fields:
Node - select the layer with the required image using the drop-down list of ones, available within the current environment
Alias - type a connection alias (DB in our case). Subsequently, it will be used as a prefix for the chosen container’s variables, imported to the currently configured one.
After that click Save to confirm linking settings. You can link as many different nodes to a single container as you require.
You always can Edit or Remove the unnecessary link with the corresponding buttons at the top pane of the Docker layer settings frame.
docker layer
After the new settings are applied, you can check the results by switching to the Variables section (where the newly imported parameters will be listed).
Tip: Upon linking Docker containers, Jelastic also adds the corresponding DNS record (with the identical to the used alias name)
to Jelastic DB. In such a way, you can refer to a particular container
from inside of these two environment layers not just over its IP
address or NodeID, but also specifying the assigned alias with
counter, i.e. {alias_name}_N.
For example, after linking with DB alias (as it’s shown above), you
can ping specific containers at the appropriate layer as “db_1”,
“db_2”, etc while working with Platform internal network via Jelastic
SSH Gateway. Herewith, if using common layer alias (i.e. without
counter, “db” in our case), the system will use Round-Robin algorithm
to choose any container within the defined node group.
https://docs.jelastic.com/docker-links
UPD1
In order to initialize database add these variables to MariaDB:
MYSQL_ROOT_PASSWORD
This variable is mandatory and specifies the password that will be set for the MariaDB root superuser account. In the above example, it was set to my-secret-pw.
MYSQL_DATABASE
This variable is optional and allows you to specify the name of a database to be created on image startup. If a user/password was supplied (see below) then that user will be granted superuser access (corresponding to GRANT ALL) to this database.
MYSQL_USER, MYSQL_PASSWORD
These variables are optional, used in conjunction to create a new user and to set that user's password. This user will be granted superuser permissions (see above) for the database specified by the MYSQL_DATABASE variable. Both variables are required for a user to be created.