I am using bitnami/airflow helm chart for my kubernetes environment. I would like to pull in the dag and plugin folders from my bitbucket git repository. How can I pass cert or credentials for my git to the image
https://hub.kubeapps.com/charts/bitnami/airflow
Without passing any credentials it gives my cannot read password error
I use this to set my git repo
airflow.clonePluginsFromGit.repository
They mention the following in their github repo
If you use a private repository from GitHub, a possible option to
clone the files is using a Personal Access Token and using it as part
of the URL:
https://USERNAME:PERSONAL_ACCESS_TOKEN#github.com/USERNAME/REPOSITORY
This should work for bitbucket too, just replace personal_access_token with one acquired from bitbucket. It might also be called an app_password depending on if you're using bitbucket cloud.
You can configure the following parameters:
airflow.auth.forcePassword to force users to specify a password
airflow.auth.username is the username to access web UI
airflow.auth.password is the password to access web UI
airflow.auth.fernetKey to set the fernet key to secure connection
airflow.auth.existingSecret is the name of an existing secret containing airflow password and fernet key
Related
The Cloud Composer doesn't support SMTP password storage in environment, according to the tutorial we can specify a command return SMTP password, I have tried bash command to export env var from airflow.cfg but failed to store password in airflow.cfg.
Our Gsuit credential is stored in LastPass, so the ideal way is to connect composer with lastpass to retrieve LDAP credential. Some applicable solution I can think of right now: 1) move credential from LastPass to Vault, enable vault in composer via ssh and use vault kv get airflow/connections/smtp_default in composer smtp_password_cmd; 2) create connections in airflow with LastPass and retrieve with airflow.hooks.base in code base (might need excessive config for security reason) 3) easiest way, hide hardcoded gmail password somewhere (not sure where thou)
Can someone point out a direction how to retrieve LDAP credential from third-party storage ( we don't want to use google secret at this moment) in cloud composer?
You can use existing secrets backend or add your own custom secret backend if none of the secret backends is good for you. The existing backends you can use are vault, Google secrets, Amazon secrets, but rolling your own backend is super simple if none of the existing ones suit you.
https://airflow.apache.org/docs/apache-airflow/2.2.0/security/secrets/secrets-backend/index.html
In case of SMTP you need to define a connection (any type - http will do) which you have to specify via connection id (and the connection should have user/password for SMTP)
I have a group with two different private repositories and I want to read a file .Rdata from one of the repos.
I tried this. But is not working
library(repmis)
source_data("https://github.com/opetchey/RREEBES/raw/Beninca_development/Beninca_etal_2008_Nature/data/GLE_estimate.Rdata?raw=True")
//The url of the code above is an example, I put the url of my private repo.
If the repository is private, you will need some kind of authentication to access it.
Accessing the https url would probably require using a cookie, or some additional header. Look at the docs for your git library for http/https authentication.
You can also try to access it through ssh, which will most probably go through your system ssh settings, and for example use a configured ssh key for authentication.
Here are the github help pages for setting up ssh for github :
Connecting to Github with SSH
I have a docker repository in Artifactory which I want to replicate:
Created a new user for it, put it in a group and gave that group permission to read, annotate, deploy/cache and delete/overwrite on the repository.
Logged in to the Artifactory UI using the new user's credentials, added the user credentials to the repository using Set Me Up and generated an access token.
However, when I configure replication for the repository using the new user's credentials I get a forbidden 403 error. I can access the repository via the UI, but when I try to use the REST API I also get a forbidden 403 error, so I figured that this was the root of my problem. Using the admin account works (curl -X GET -u admin:pass artifactory-url/artifactory/api/repositories/my-repo).
The REST API method for setting repository replication configuration requires a user with admin permissions. You can see the required permission in the API documentation.
I'm using salt-cloud with Rackspace OpenStack cloud severs.
I've setup the provider and a profile without explicitly setting any key or password settings. When I issue the command to create a new vm e.g.
salt-cloud -p profile vmname
the machine gets created quite happily and salt-cloud appears to be logging in repeatedly to perform the initial configuration and bootstrapping the minion.
My questions is how is it logging in? With which details? And how I could I find them and login to minion manually myself after it's been created by salt-cloud?
So I worked it out myself. The answer is it used password authentication.
I created a vm with debug level logging and the generated password is output.
salt-cloud --log-level debug -p profile vmname &> outputfile
Searching the output file gave this line:
...
221 [DEBUG ] Using ************ as the password
...
(221 is the line number added by vim)
Using that password I was able to login to the VM.
Salt-cloud queries the openstack api for the root password of the new vm and uses that to log in.
You can then use Salt to set a new password.
I'm currently trying to transisiton a maven build to a Gradle build. The show stopper hurdle I've hit is that our internal artifacts are deployed to an internal repository that requires authentication to read.
And you know what authentication means... usernames and passwords. The problem is I don't want to require developers to store their password as plaintext on their hard drives. Maven supports password encryption but I'm not seeing how to make Gradle do it.
Is there some magic access to the Ivy CredentialStore that will support encrypted passwords? or will I have to wait for a new version?
The gradle-credentials plugin will let you encrypt the contents of a gradle.encrypted.properties file, which will then be available in the build as properties of the credentials object.
The gradle maven settings plugin will let you use Maven's system for encrypting/decrypting credentials.
We use Artifactory and Ivy's CredentialsStore works fine.
repositories {
org.apache.ivy.util.url.CredentialsStore.INSTANCE.addCredentials(REALM, HOST, USER, PASSWORD);
mavenRepo urls: [ "http://repo.mycompany.com/repo" ]
}
To have each developer use a different username create a gradle.properties file that contains
HOST=repo.mycompany.com
REALM=My Company Realm
USER=theusername
PASSWORD={DESede}xyz123abc
The {DESede} encrypted password prefix may be a Artifactory specific convention.