How to encrypt variables when using airflow EmrAddStepsOperator? - airflow

Question

I have an Airflow DAG that starts an AWS EMR cluster to run steps. On the DAG we pass some variables that are set on Airflow Variables. But some of these variables are encrypted at Airflow, but when passing to EMR, we can see then clearly at EMR console. Is there any way to hide this?
Here is how we are defining the step. The airflow variable db_pass must be encrypted or hidden somehow
{
"Name": "EMR JOB",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"{{var.value.job_script}}",
"--database_user={{var.value.db_user}}",
"--database_pass={{var.value.db_pass}}"
]
}
}
]
This SAMPLE_STEP_DEFINITION is then passed as to the EmrAddStepsOperator:
...
sample_task = EmrAddStepsOperator(
steps=SAMPLE_STEP_DEFINITION,
...

Answer 1

There are several ways to do this. First I would suggest to encrypt passwords with KMS. Here is the code how to do it:
def encryptString(plainText: String, keyArn: String): String = {
val req = new EncryptRequest().withKeyId(keyArn).withPlaintext(ByteBuffer.wrap(plainText.getBytes))
Base64.getEncoder.encodeToString(kmsClient.encrypt(req).getCiphertextBlob.array())
}
def decryptString(encryptedText: String, keyArn: String): String = {
val req = new DecryptRequest().withCiphertextBlob(ByteBuffer.wrap(Base64.getDecoder.decode(encryptedText)))
new String(kmsClient.decrypt(req).getPlaintext.array())
}
Your just need to attach decrypt permission to EMR_EC2_DefaultRole.
Another way is to pass a config file stored on S3 with password.