I am in the middle of migrating our self managed Airflow environment to Cloud Composer 2 and was wondering how I can migrate the DAG run history from the external environment to the new Cloud Composer environment. Has anyone does this before? Is there documentation?
Here you can find an explanation of how to access airflow database on cloud composer.
What you need to do, is migrate you self managed Airflow DB to the same version used in Cloud Composer:
airflow db upgrade -r "<current version>:<cloud composer version>"
# ex:
airflow db upgrade -r "2.0.0:2.2.0"
Then you can export your DB and import it in the Cloud Composer DB:
Ex for postgres DB:
# in your db
pg_dump -U <your db user> -d <your db name (airflow)> -f airflow_dump.sql
# in cloud composer db
psql -U <cloud composer db user> -d <cloud composer db name> -f airflow_dump.sql
Related
Each minute, I have a K8S Cronjob who run Symfony command. My problem is the huge time my pod warmup Symfony cache before execute the command : 56s ...
I search a solution for stock the cache in docker container but i can't execute cache:warmup during docker build, because commande need database and redis up.
Do you have a solution for me ?
Thanks !
All I want to do is essentially take the exact dynamdb tables with their data that exist in a remote instance ( e.g. amplify staging environment/api) and import those locally.
I looked at datasync but that seemed to be FE only. I want to take the exact data from staging and sync that data to my local amplify instance - is this even possible? I can't find any information that is helping right now.
Very used to using mongo/postgres etc. and literally being able to take a DB dump and just import that...I may be missing something here?
How about using dynamodump
Download the data from AWS to your local machine:
python dynamodump.py -m backup -r REGION_NAME -s TABLE_NAME
Then import to Local DynamoDB:
dynamodump -m restore -r local -s SOURCE_TABLE_NAME -d LOCAL_TABLE_NAME --host localhost --port 8000
You have to build a custom script that reads from the online DynamoDB and then populate the Local DynamoDB. I found the docker image be just perfect to have an instance, make sure to give the jar file name to prevent the image to be ephemeral and have persistence of data.
Sort of macro instructions:
Download Docker Desktop (if you want)
Start docker desktop and in a terminal ask for Dynamo DB official image:
https://hub.docker.com/r/amazon/dynamodb-local/
docker pull amazon/dynamodb-local
And then run the docker container:
docker run --name dynamodb -p 8000:8000 -d amazon/dynamodb-local -jar DB.jar
Now you can start a python script that get the data from the online DB and copy in the local dynamoDB, as in official docs:
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/copy-amazon-dynamodb-tables-across-accounts-using-a-custom-implementation.html
Working out the connections to the local container (localhost:8000) you shall be able to copy all the data.
I'm not too well versed on local Amplify instances, but for DynamoDB there is a product which you can use locally called DynamoDB Local.
Full list of download instructions for DynamoDB Local are available here: Downloading And Running DynamoDB Local
If you have docker installed in your local machine, you can easily download and start the DynamoDB Local service with a few commands:
Download
docker pull amazon/dynamodb-local
Run
docker run --name dynamodb -p 8000:8000 -d amazon/dynamodb-local -jar DB.jar
This will allow you to avail of 90% of DynamoDB features locally. However, migrating data from DynamoDB Web Service to DynamoDB local is not something that is provided out of the box. For that, you would need to create a small script which you run locally, read data from your existing table and write to your local instance.
An example of reading from one table and writing to a second can be found in the docs here: Copy Amazon Dynamodb Tables Across Accounts
One change you will have make is manually setting the endpoint_url for DynamoDB Local:
dynamodb_client = boto3.Session(
aws_access_key_id=args['AWS_ACCESS_KEY_ID'],
aws_secret_access_key=args['AWS_SECRET_ACCESS_KEY'],
aws_session_token=args['TEMPORARY_SESSION_TOKEN'],
endpoint_url='YOUR_DDB_LOCAL_ENDPOINT_URL'
).client('dynamodb')
I want to get the latest migrations from my hasura endpoint to my local filesystem.
Command I'm trying to run
hasura init config --endpoint someendpoint.cloudfront.net/ --admin-secret mysecret
Output
INFO Metadata exported
INFO Creating migrations for source: default
ERRO creating migrations failed:
1 error occurred:
* applying migrations on source: default: cannot fetch schema dump: pg_dump request: 500
{
"path": "$",
"error": "internal exception while executing pg_dump",
"code": "unexpected"
}
run `hasura migrate create --from-server` from your project directory to retry
INFO directory created. execute the following commands to continue:
cd /home/kanhaya/Documents/clineage/study-container/src/conifg
hasura console
The Haura is deployed using a custom Docker image:
FROM hasura/graphql-engine:v2.3.0 as base
FROM ubuntu:focal-20220302
ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true
ENV HASURA_GRAPHQL_DEV_MODE=true
# ENV HASURA_GRAPHQL_PG_CONNECTIONS=15
ENV HASURA_GRAPHQL_DATABASE_URL=somedatabaseURL
ENV HASURA_GRAPHQL_ADMIN_SECRET=mysecret
ENV EVENT_TRIGGER=google.com
ENV STUDY_CONFIG_ID=1
COPY ./Caddyfile ./Caddyfile
COPY ./install-packages.sh ./install-packages.sh
USER root
RUN ./install-packages.sh
RUN apt-get -y update \
&& apt-get install -y libpq-dev
COPY --from=base /bin/graphql-engine /bin/graphql-engine
EXPOSE 8080
CMD caddy start && graphql-engine --database-url $HASURA_GRAPHQL_DATABASE_URL serve --admin-secret $HASURA_GRAPHQL_ADMIN_SECRET
Which version of Postgres are you currently running?
There's a conversation over here with a couple of workarounds for supporting 14 in the Hasura container (looks like version mis-match in one of the libraries):
https://github.com/hasura/graphql-engine/issues/7676
I have successfully deployed my meteor application using mup deploy. As stated in documentation to access database we need to run this command:
docker exec -it mongodb mongo <appName>
How can I use mongodump command with this setup? I have tried running
docker exec -it mongodb mongodump --db appName --archive=baza.gz --gzip
Command runs successfully, but I can not find baza.gz archive
As I found dump gets saved in docker container. To access backup from local filesystem we need to copy it from docker container.
To dump:
docker exec -it mongodb mongodump --db appName --archive=/root/baza.gz --gzip
To copy from docker container to local filesystem:
docker cp mongodb:/root/backup.gz /home/local_user
I want to copy my heroku production db (postgres) to my development (sqlite).
Copying a postgres db into another postgres db is easy using heroku pg:pull. Does anyone know how to use this command to copy postgres into sqlite?
Heroku docs on pg:pull do not say how to use different types of dbs. This old article implied that it used to be possible. Setting up a local postgres db is something I'd like to avoid.
You will need do a pg_restore locally then dump the data using the -a option to dump data only.
It should look something like this:
Download a data dump.
heroku addons:add pgbackups
heroku pgbackups:capture
curl -o latest.dump `heroku pgbackups:url`
Create a temporary database.
sudo -u postgres createdb tempdb
Restore the dump to your temporary database.
sudo -u postgres pg_restore --verbose --clean --no-acl --no-owner -h localhost -d tempdb latest.dump
Dump the data in the correct format.
sudo -u postgres pg_dump --inserts -a -b tempdb > data.sql
Read dump in sqlite3.
sqlite3
> .read data.sql
This is an approximate solution. You will most likely need to make some small adjustments.
I agree with Craig Ringer that it might be worth getting postgres running locally. Hopefully this process will do the trick though!