Is there a way to encrypt the airflow config file sql_alchemy_conn string , the password shown in example is plaintext . What options are there to secure it. Also if the password has special chars how it must be escaped in the config file
Trying to install airflow using airflow role.
# See: https://www.sqlalchemy.org/
sql_alchemy_conn:
value: "postgresql+psycopg2://pgclusteradm#servername:PLAINTEXTPASSWORD#server.postgres.database.azure.com/airflow2"
Way to encrypt password, couldn't find how to encrypt this.
You can provide the database URI through environment variables instead of the config file. This doesn't encrypt it or necessarily make it more secure, but it at least isn't plainly sitting in a permanent file.
In your airflow.cfg you can put a placeholder:
[core]
...
sql_alchemy_conn = override_me
...
Then set AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://... in an environment variable when you bring up Airflow components. This way of setting and overriding configuration options through environment variables is detailed in the docs, but the basic format is AIRFLOW__{SECTION}__{KEY}=<value>.
There are 2 ways of securing this as mentioned in docs:
1) Environment Variable:
You can override the setting in airflow.cfg by setting the following environment variable:
AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_string
This way you can keep the setting in airflow.cfg as empty so no one can view the password.
2) Get string by running command:
You can also derive the connection string at run time by appending _cmd to the key like this:
[core]
sql_alchemy_conn_cmd = bash_command_to_run
Related
I am trying to create a database for my Flask application in the main directory of my project. This is my code for initializing a database:
app.config["SQLALCHEMY_DATABASE_URI"] = 'sqlite:///users.db'
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app)
Flask requires application context, so this is how I create the database:
$ flask shell
>>> db.create_all()
I also tried doing it with:
$ python
>>> from app import app, db
>>> app.app_context().push()
>>> db.create_all()
Both of these options create the database in the /instance directory. Is there any way to get around this and create it in the main directory of the project?
The instance path is the preferred and default location for the database. I recommend you to use this one for security reasons. However, it is also possible to choose an alternative solution in which the full length of the path is specified in the configuration.
The following configuration corresponds to an outdated variant, where the database is created in the current working directory. Please don't use this anymore.
app.config['SQLALCHEMY_DATABASE_URI'] ='sqlite:///' + os.path.join(os.getcwd(), 'users.db')
This corresponds to the current solution.
app.config['SQLALCHEMY_DATABASE_URI'] ='sqlite:///' + os.path.join(app.instance_path, 'users.db')
i would like cloud-init to not use 50-cloud-init.yaml. I have prepared my own file.
Do you know how to do this
You can add
network:
config: disabled
to /etc/cloud/cloud.cfg or a file in /etc/cloud/cloud.cfg.d.
Another option is to add
network-config=disabled
to the kernel command line.
While the network config yaml technically works as userdata, the network configuration will have already been written out before userdata is read.
One other option is to write your netplan configuration into /etc/netplan/99-some-name.yaml. If you have configuration there that overlaps with what is in 50-cloud-init.yaml, your configuration will override what is in the default configuration.
See https://cloudinit.readthedocs.io/en/latest/topics/network-config.html#disabling-network-configuration .
Documentation for MariaDB says clearly how to set up server defaults like charset and collation. So I opened my Mariadb console and run:
SET character_set_server = 'utf8mb4';
SET collation_server = 'utf8mb4_slovak_ci';
Consoles wrote OK.
Then I restart the server, but as I tried to create new database there are still the same latin2 charset and swedish collation.
I do it automatically via Symfony cosole command
php bin/console doctrine:database:create
What is wrong with that? I did it like documentation says.
SET character_set_server changes the character set for the current connection.
SET global character_set_server changes the character set globally for each new connection.
However if you restart your server the default character sets for server will be read from the configuration file. If the configuration file doesn't specify character set, then defaults will be used. So for making your settings permanent, you have to change the character sets in your configuration file (https://mariadb.com/kb/en/configuring-mariadb-with-option-files/)
First, run the SHOW VARIABLES like "%collation%"; to show you the current configs.
To change collation_server setting, you have to use the keyword GLOBAL and therefore your statement will be SET GLOBAL collation_server = 'utf8mb4_slovak_ci';
I need the names of the keys for a build pipeline in GitHub actions, the values of the keys encrypted in the repo but decrypted in my local environment.
Example:
I need the .env file like this (decrypted) in my local environment:
ACCESS_KEY = mykeysecret123
ANOTHER_KEY = key2secret
.env-file
And like this (encrypted) in the GitHub repo:
ACCESS_KEY = f4d5dfa6f5da4f6as5dfsdhgzxcvr4
ANOTHER_KEY = dhjdujfk5f64f8ghjdj5j56jhjh4jhf
.env-file-encrypted
Very similar to what transcrypt does, but transcypt encrypts the whole file, and I need the key names of the file to be not-encrypted, so when the build runs in Github actions it can find the names of the variables. The build does not need the actual secret values. It's failing becase I have a config file with:
import { ACCESS_KEY, ANOTHER_KEY} from 'react-native-dotenv'
And when the build looks for those names in the .env file, it can't find them if the file is encrypted with transcrypt or git-crypt or any other encrypting tool that encrypts the whole file.
There is a nice tool by Mozilla that encrypts only the values and not the keys of your secret file
https://github.com/mozilla/sops
Follow the detailed tutorial video here : https://www.youtube.com/watch?v=V2PRhxphH2w
react-native-dotenv maintainer here! I took a look at your screenshots and I'm noticing a couple issues:
Make sure to remove all spaces in the .env file around the =, e.g. ACCESS_KEY=f4d5dfa6f5da4f6as5dfsdhgzxcvr4.
you need to do import { ACCESS_KEY, ANOTHER_KEY } from '#env'
I would like to set the configuration of my symfony2 project using environment variables.
In the server I have defined:
SYMFONY__DATABASE__USER
SYMFONY__DATABASE__PASSWORD
SYMFONY__DATABASE__NAME
SYMFONY__DATABASE__HOST
SYMFONY__DATABASE__DRIVER
My parameters.yml.dist looks like this:
#app/config/parameters.yml.dist
parameters:
database_host: "%database.host%"
database_port: ~
database_name: "%database.name%"
database_user: "%database.user%"
database_password: "%database.password%"
database_driver: "%database.driver%"
when I run composer I get an exception
composer install --dev --no-interaction --prefer-source
[Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException]
You have requested a non-existent parameter "database.driver". Did you mean one of these: "database_user", "database_driver"?
These variables are defined in the server so I can modify the parameters.yml.dist to define these values. But this does not seams the right way, because wat I really want to use are the environment variables.
Note: I want to read this environment variables in travis, heroku and my vagrant machine. I only want to have in the repository the vagrant machine variables.
Which is the proper way to do this?
How should look my parameters.yml.dist?
Looks you are doing everything okay.
Here is the complete documentation for Setting Environment Variables which I believe you already read.
What is important to note is this:
Also, in order for your console to work (which does not use Apache),
you must export these as shell variables. On a Unix system, you can
run the following:
$ export SYMFONY__DATABASE__USER=user
$ export SYMFONY__DATABASE__PASSWORD=secret
I remember once I have a similar issue, I was setting everything on APACHE, but when running commands it wasn't working because I forgot to EXPORT the variables on the system.
Be aware that using export is a temp solution, if you reset your server those values will be lost, you will need to setup in a permanent way according to your OS.
I think you solved this long time ago, but the problem is actually that you have 2 _ between DATABASE and USER and the parser for this have a string replace function that replaces every __ with a . .
For your example to work you should have written like this:
SYMFONY__DATABASE_USER -> database_user
SYMFONY__DATABASE__USER -> database.user
You can try this bundle if your system version is >= 2.6.2:
This bundle provides a way to read parameters from environment
variables at runtime. The value defined in the container parameter is
used as fallback when the environment variable is not available.