Delete Terraform resource aws_secretsmanager_secret_version does not delete Secrets Manager secret entry - terraform-provider-aws

I created an AWS secrets manager and a secret key-value entry using Terraform as below. However, After I comment out below aws_secretsmanager_secret_version resource and terraform apply, terraform shows it deletes the secret key-value entry, but I can still see the entry in AWS console and I can still use CLI to get the secret key-value using aws secretsmanager get-secret-value --secret-id myTestName.
Is this entry really deleted? Why I still see it in AWS console? or maybe it is deleted but the one shown in console and cli is an old version? at least Terraform deleted it from its state file.
resource "aws_secretsmanager_secret" "test" {
name = "myTestName"
}
# I deleted secret key-value entry by
# commenting out below and apply terraform again
resource "aws_secretsmanager_secret_version" "test" {
secret_id = aws_secretsmanager_secret.test.id
secret_string = <<EOF
{
"test-key": "test-value"
}
EOF
}

According to AWS documentation:
...Secrets Manager does not immediately delete secrets. Instead,
Secrets Manager immediately makes the secrets inaccessible and
scheduled for deletion after a recovery window of a minimum of
seven days...
Due to critical nature of the secrets, this functionality is there for a reason - to prevent you from accidentally deleting important production-grade secret, which would cause serious problems with accessing services.
If you still want to delete a secret, you can do it with force:
aws secretsmanager delete-secret --secret-id your-secret --force-delete-without-recovery --region your-region
You may need to delete it with force if you want to immediately create new secret with the same name, to avoid name conflict.
Update: As you clarified, for you specific case - where you wish to delete the version of the secret, it cannot be done while you have only one version of the secret with the AWSCURRENT label:
aws secretsmanager get-secret-value --secret-id myTestName
...
"Name": "myTestName",
"SecretString": " ...
"VersionStages": [
"AWSCURRENT"
]
...
From the terraform documentation:
If the AWSCURRENT staging label is present on this version during
resource deletion, that label cannot be removed and will be skipped to
prevent errors when fully deleting the secret. That label will leave
this secret version active even after the resource is deleted from
Terraform unless the secret itself is deleted. Move the AWSCURRENT
staging label before or after deleting this resource from Terraform to
fully trigger version deprecation if necessary.

Related

VAULT - How to unwrap Secret ID with HTTP request

I want to integrate vault with a Java application. I follow this blog to do it.
The question is when I have wrapping token, I want to Unwrap it in step number 9 in the picture above with HTTP request and get the secret_id. I see the API document in here, but it require X-Vault-Token which can not store in my JAVA application. Without it the API response permission denied.
But when I use vault command: VAULT_TOKEN=xxxxxxxxxx vault unwrap -field=secret_id, it response a secret that what I want (I do not login to Vault).
Any have experience about this please help. Thank you.
In your linked diagram, step 10 (the next step) is literally "Login with Role ID and Secret ID". If you want to wrap a different secret, then you can change the pattern completely, but the blog post you're referencing wants you to use the Secret ID from the wrapped token response to then log into Vault with that role and get your final secrets.
So, take the output of SECRET_ID=$(VAULT_TOKEN=xxxxxxxxxx vault unwrap -field=secret_id), export it, and then run a resp=$(vault write auth/approle/login role_id="${ROLE_ID}" secret_id="${SECRET_ID}"); VAULT_TOKEN=$(echo "${resp}" | jq -r .auth.client_token), export the VAULT_TOKEN, then call Vault to get the secret you really want (vault kv get secret/path/to/foobar) and do something with it.
#!/usr/bin/env bash
wrap_token=$(cat ./wrapped_token.txt)
role_id=$(cat ./approle_role_id.txt)
secret_id=$(VAULT_TOKEN="${wrap_token}" vault unwrap -field=secret_id)
resp=$(vault write -format=json auth/approle/login role_id="${role_id}" secret_id="${secret_id}")
VAULT_TOKEN=$(echo "${resp}" | jq -r '.auth.client_token')
export VAULT_TOKEN
# Put a secret in a file
# Best to ensure that the fs permissions are suitably restricted
UMASK=0077 vault kv get -format=json path/to/secret > ./secret_sink.json
# Put a secret in an environment variable
SECRET=$(vault kv get -format=json path/to/secret)
export SECRET
In case you want to reduce the security of your pattern, you can read below...
If you want to avoid logging into Vault and simply give the app a secret, you can avoid many of these steps by having your trusted CI solution request the secret directly, i.e. vault kv get -wrap_ttl=24h secret/path/to/secret, and then the unwrapping step you're doing will actually contain a secret you want to use, instead of the intermediary secret that would allow you to log into Vault and establish an application identity. However, this is not recommended as it would make your CI solution want to access more secrets, which is far from least privilege, and it makes it incredibly difficult to audit where the secrets are actually being leveraged from a Vault perspective, which is one of the primary benefits of implementing a central secrets management solution like Vault.

VPC creation problem in aws via terraform

I have been trying to create vpc infrastructure in AWS through terraform I am unable to perform the "Terraform apply" command anyone has a similar problem while using a free trial account.
Error: Error creating VPC: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 4HZVo3-eWCS-YLhRy55P_0T13F_fPtA29TYrJrSe5_dyPxIcqRbh7_wCcrCZr2cpmb-B5--_fxVaOngBfHD_7yfnPH7NLf1rrqpb7ge1mvQrK8P0Ltfpgpm37nZXezZUoYf1t4peB25aCxnbfeboHpgJjcFnHvqvf5so5G2PufnGZSB4FUZMfdaqppnJ-sNT7b36TonHUDNbLhBVUl5Fwd8d02R-6ZraRYvDx-o4lDfP9xSWs6PMUFXNr1qzruYaeMYMxIe-9kGOQptgBLYZXsxr966ajor-p6aLJAKlIwPGN7Iz7v893oGpGgz_8wxTv4oEb5GnfYOuPOqSyEMLKI69b2JUvVU1m4tCcjKBaHJARP5sIiFSGhh4lb_E0_cKkmmFfKzyET2h8YkSD8U9Lm4rRtGbAEJvIoDZYDkNxlW7W2XvsccmLnQFeSxpLolVhguExkP7DT9uXffJzFEjQn-VkhqKnWlwv0vxIcOcoLP04Li5WAqRRr3l7yK2bYznfg
│ status code: 403, request id: 5c297a4d-7bcf-4bb4-b311-37480e1f26b8
make sure you have properly setup aws credentials and permissions.
check these two files
~/.aws/credentials
~/.aws/config
this docs can help you.
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
Did you configure your access keys?
provider "aws" {
region = "us-west-2"
access_key = "my-access-key"
secret_key = "my-secret-key"
}
There are multiple ways to do it (described here).
My above example can be a good start but you don't want to commit those so I recommend to configure the keys in ~/.aws/credentials (like you need them for AWS CLI). The aws provider will pick them up automatically and so you don't need to define them somewhere in your terraform code.

Web.Config transforms for Multi-Tenant deployment of WebForms app in docker over AWS ECS

Environment
ASP.NET WebForms app over IIS
Docker container host
AWS ECS hosting platform
Each client hosting its own copy of the app with private database connection string
Background
In the non-docker environment, each copy is a virtual directory under IIS, and thus have their own individual web.config pointing to dedicated databases. The underlying codebase is the same for each client, with no client-specific customization involved. The route becomes / here.
In the docker environment (one container per client), each copy goes over as a central root application.
Challange
Since the root image is going to be the same, how to have the web.config overridden for each client deployment.
We shouldn't create multiple images (one per client) as that will mean having extra deployment jobs and losing out on centralization. The connection strings should ideally be stored in some kind of dictionary storage applicable at ECS level which can provide client-specific values upon loading of corresponding containers.
Presenting the approach we used to solve this issue. Hope it may help others struck in similar cases.
With the problem statement tied to having a single root image and having any customization being applied at runtime, we knew that there needs to be a transformation of web.config at time of loading of the corresponding containers.
The solution was to use a PowerShell script that will read the web.config and get replace the specific values which were having a custom prefix embedded to the key. The values got passed from custom environmental variables within ECS and the web.config also got updated to have the keys with the prefix added.
Now since the docker container can have only a single entry point, a new base image was created which instantiated an IIS server and called a PowerShell script as startup. The called script called this transformation script and then set the ServiceMonitor on the w3cwp.
Thanks a lot for this article https://anthonychu.ca/post/overriding-web-config-settings-environment-variables-containerized-aspnet-apps/
I would use environment variables as the OP suggests for this with a start up transform, however I want to make the point that you do not want sensitive information in ENV variables, like DB passwords, in your ECS task definition.
For that protected information, you should use ECS secrets coupled with Parameter Store in Systems Manager. These values can be stored encrypted in the Parameter Store (using a KMS key) and the ECS Agent will 'inject' them as ENV variables on task startup.
For me, to simplify matters, I simply use secrets for everything although you can choose to only encrypt the sensitive information and leave the others clear.
I dynamically add the secrets for the given application into my task definitions at deploy time by looking up the 'secrets' for the given app by 'namespace' (something that Parameter Store supports). Then, if I need to add a new parameter, I can just add a new secret to the store in the given namespace and re-deploy the app. It will pick up and inject into the task definition any newly defined secrets automatically (or remove ones that have been retired).
Sample ruby code for creating task definition:
params = ssm_client.get_parameters_by_path(path: '/production/my_app/').parameters
secrets = params.map{ |p| { name: p.name.split("/")[-1], value_from: p.arn } }
task_def.container_definitions[0].secrets = secrets
This last transform injects the secrets such that the secret 'name' is the ENV variable name... which ends up looking like this:
"secrets": [
{
"valueFrom": "arn:aws:ssm:us-east-1:578610029524:parameter/production/my_app/DB_HOSTNAME",
"name": "DB_HOSTNAME"
},
{
"valueFrom": "arn:aws:ssm:us-east-1:578610029524:parameter/production/my_app/DB_PASSWORD",
"name": "DB_PASSWORD"
}
You can see there are no values now in the task definition. They are retrieved and injected when ECS starts up your task.
More information:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html

BAD_GATEWAY when connecting Google Cloud Endpoints to Cloud SQL

I am trying to connect from GCP endpoints to a Cloud SQL (PostgreSQL) database in a different project. My endpoints backend is an app engine in the flexible environment using Python.
The endpoints API works fine for non-db requests and for db requests when run locally. But the deployed API produces this result when requiring DB access:
{
"code": 13,
"message": "BAD_GATEWAY",
"details": [
{
"#type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "application"
}
]
}
I've followed this link (https://cloud.google.com/endpoints/docs/openapi/get-started-app-engine) to create the endpoints project, and this (https://cloud.google.com/appengine/docs/flexible/python/using-cloud-sql-postgres) to link to Cloud SQL from a different project.
The one difference is that I don't use the SQLALCHEMY_DATABASE_URI env variable to connect, but take the connection string from a config file to use with psycopg2 SQL strings. This code works on CE servers in the same project.
Also double checked that the project with the PostgreSQL db was given Cloud SQL Editor access to the service account of the Endpoints project. And, the db connection string works fine if the app engine is in the same project as the Cloud SQL db (not coming from endpoints project).
Not sure what else to try. How can I get more details on the BAD_GATEWAY? That's all that's in the endpoints logfile and there's nothing in the Cloud SQL logfile.
Many thanks --
Dan
Here's my app.yaml:
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
python_version: 3
env_variables:
SQLALCHEMY_DATABASE_URI: >-
postgresql+psycopg2://postgres:password#/postgres?host=/cloudsql/cloudsql-project-id:us-east1:instance-id
beta_settings:
cloud_sql_instances: cloudsql-project-id:us-east1:instance-id
endpoints_api_service:
name: api-project-id.appspot.com
rollout_strategy: managed
And requirements.txt:
Flask==0.12.2
Flask-SQLAlchemy==2.3.2
flask-cors==3.0.3
gunicorn==19.7.1
six==1.11.0
pyyaml==3.12
requests==2.18.4
google-auth==1.4.1
google-auth-oauthlib==0.2.0
psycopg2==2.7.4
(This should be a comment but formatting really worsen the reading, I will update on here)
I am trying to reproduce your error and I come up with some questions:
How are you handling the environment variables in the tutorials? Have you hard-coded them or are you using environment variables? They are reset with the Cloud Shell (if you are using Cloud Shell).
This is not clear for me: do you see any kind of log file in CloudSQL (without errors) or you don't see even logs?
CloudSQL, app.yaml and requirements.txt configurations are related. Could you provide more information on this? If you update the post, be careful and do not post username, passwords or other sensitive information.
Are both projects in the same region/zone? Sometimes this is a requisite, but I don't see anything pointing this in the documentation.
My intuition points to a credentials issue, but it would be useful if you add more information to the post to better understand where the issue cames from.

SSH key injection in openstack

During vm creation in openstack, one can specify a keypair name, so that the specified public key get injected to the newly created vm.
I would like to know in which state of machine the key injection is done, completely? Given the machine is in ACTIVE state, does that guarantee that the key injection is completed?
Details:
I have a limited quota for the key pairs and I would like to delete each keypair from openstack immediately after they get injected to the target machine. I have only access to openstack ReST API and NOT to the target vm.
UPDATE
Looking at nova instances table, I can see that "key name" and "key data" are existing there too. I think the key is copied to this table and then the original key is not referenced any more. So deleting the key shouldn't cause any issue. am I wrong?
What you can do is try a ssh connection and once that succeeds, proceed to delete the keypair.
To answer your question directly, the key is added via the cloud-init. You can grep for ssh in /var/log/cloud-init.log to see when exactly it happens. (It happens pretty early in the cloud-init process).
I don't think there is any API way of figuring out when exactly the key injection happens. Machine in ACTIVE state is not a guarantee that cloud-init part of key injection is done (though for practical purposes, it does happen pretty early).
You could try checking it via nova console-log. Though the output of console-log has limited buffer, so it may overshoot the key addition part and hence you may not see it in console log.
So, I think checking via actual ssh connection is the only sure shot way.

Resources