Apache Airflow Docker : sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:mysqldb
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.3.3}
# build: .
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://xxx:xxx#xxxxx:xxxx/airflow
# For backward compatibility, with Airflow <2.3
AIRFLOW__CORE__SQL_ALCHEMY_CONN: mysql+mysqldb://root:xxxx#xxxxxx:xxxxx/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+mysqldb://root:xxxxxx#xxxxxx:xxxxxx/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#xxxxxx:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- airflow:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
user: "${AIRFLOW_UID:-50000}:0"
I have the same problem,How should this problem be solved.
I think that the problem is with the celery result backend, where celery uses databases drivers different from airflow drivers, you can try:
AIRFLOW__CELERY__RESULT_BACKEND: db+mysql://root:xxxxxx#xxxxxx:xxxxxx/airflow
Here is the celery configuration doc.
Related
Here i created a User model using sqlmodel.
from sqlmodel import SQLModel, Field, Index
from typing import Optional
class User(SQLModel, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
name: str = Field(max_length=30)
surname: str = Field(max_length=50)
docker-compose with postgres:
version: "3.4"
services:
db:
image: postgres:14.0-alpine
restart: always
container_name: test_db
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=testdb
ports:
- "5432:5432"
volumes:
- db:/var/lib/postgresql/data
volumes:
db:
Now i am trying to create migrations with alembic revision
with "alembic revision --autogenerate -m "msg""
But it falls with
File "C:\Python3.10\lib\socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed
The error indicates that the hostname cannot be resolved. Is your database running locally? Has your python file that holds the SQLAlchemy engine access to it?
I also see that you are running albemic from your global python installation (File "C:\Python3.10\), does that have the same dependencies as your python application? In any case, it is very advisable to use virtual environments to ensure you are developing with the same modules as you are running albemic migrations with.
I'm trying to use the WordPress local development environment, which sets up a bunch of docker containers (and I'm new to Docker). One of them is a WordPress CLI, which I presume is running some scripts to do some configuration, but that particular container doesn't stay running (I believe this is intentional). I'm guessing that a script it's running is failing, and that's happening because of some configuration error, but I can't figure out how to tell what it's doing when it executes (what scripts it's running, what environment variables are set, etc...).
Is there any way to somehow "trace" what the container is trying to do?
$docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
wordpressdevelop/phpunit latest 8ebb6f73d762 2 days ago 732MB
wordpress latest 6c2c086d9173 2 days ago 554MB
wordpress <none> 408627ce79b1 2 days ago 551MB
composer latest ff854871a595 9 days ago 173MB
wordpress cli ee6de7f71aa0 9 days ago 137MB // I want to see what this does
mariadb latest e27cf5bc24fe 12 days ago 401MB
$docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
19007b991e08 wordpress "docker-entrypoint.s…" 48 minutes ago Up 48 minutes 0.0.0.0:8888->80/tcp e6bc9159b910bda3d9b0dae2e230eabd_wordpress_1
26ac5c7ec782 wordpress "docker-entrypoint.s…" 48 minutes ago Up 48 minutes 0.0.0.0:8889->80/tcp e6bc9159b910bda3d9b0dae2e230eabd_tests-wordpress_1
8ae0a4dc4f77 mariadb "docker-entrypoint.s…" 48 minutes ago Up 48 minutes 0.0.0.0:54989->3306/tcp e6bc9159b910bda3d9b0dae2e230eabd_mysql_1
This is on MacOS 11.2.2 running Docker Desktop, Docker version 20.10.5.
I'm not sure if it's relevant, but for completeness' sake, I've included the docker-compose.yml which wp-env generates, below.
Thanks!
Background
This used to Just Work, but I think I broke something in my environment, and am trying to diagnose that. I initially asked about that on WordPress StackExchange, but have since dug deeper. The Docker compose step fails thusly:
...
⠏ Configuring WordPress.Creating e6bc9159b910bda3d9b0dae2e230eabd_cli_run ... done
⠹ Configuring WordPress.mysqlcheck: Got error: 1045: Access denied for user 'username_here'#'172.19.0.5' (using password: YES) when trying to connect
The database container is left up and running, and if I go look in the database, I see it's configured to have root connect with no password:
MariaDB [(none)]> select user, host, password from mysql.user;
+-------------+-----------+----------+
| User | Host | Password |
+-------------+-----------+----------+
| mariadb.sys | localhost | |
| root | localhost | |
| root | % | |
+-------------+-----------+----------+
3 rows in set (0.003 sec)
In the main WordPress container, the wp-config.php file contains this little snippet:
...
// a helper function to lookup "env_FILE", "env", then fallback
function getenv_docker($env, $default) {
if ($fileEnv = getenv($env . '_FILE')) {
return file_get_contents($fileEnv);
}
else if ($val = getenv($env)) {
return $val;
}
else {
return $default;
}
}
...
/** MySQL database username */
define( 'DB_USER', getenv_docker('WORDPRESS_DB_USER', 'username_here') );
/** MySQL database password */
define( 'DB_PASSWORD', getenv_docker('WORDPRESS_DB_PASSWORD', 'password_here') )
Given the error message, I assume the CLI container is trying to do something similar, but the environment variables WORDPRESS_* aren't set, and so it's using the defaults, which ... aren't working. What I think I need to do is track down something that's failing to set those variables earlier in the run process.
docker-compose.yml
version: '3.7'
services:
mysql:
image: mariadb
ports:
- '3306'
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
- 'mysql:/var/lib/mysql'
wordpress:
build: .
depends_on:
- mysql
image: wordpress
ports:
- '${WP_ENV_PORT:-8888}:80'
environment:
WORDPRESS_DB_NAME: wordpress
volumes: &ref_0
- 'wordpress:/var/www/html'
- '/Users/cwr/src/cwra/foo:/var/www/html/wp-content/plugins/foo'
tests-wordpress:
depends_on:
- mysql
image: wordpress
ports:
- '${WP_ENV_TESTS_PORT:-8889}:80'
environment:
WORDPRESS_DB_NAME: tests-wordpress
volumes: &ref_1
- 'tests-wordpress:/var/www/html'
- '/Users/cwr/src/cwra/foo:/var/www/html/wp-content/plugins/foo'
cli:
depends_on:
- wordpress
image: 'wordpress:cli'
volumes: *ref_0
user: '33:33'
tests-cli:
depends_on:
- tests-wordpress
image: 'wordpress:cli'
volumes: *ref_1
user: '33:33'
composer:
image: composer
volumes:
- '/Users/cwr/src/cwra/foo:/app'
phpunit:
image: 'wordpressdevelop/phpunit:latest'
depends_on:
- tests-wordpress
volumes:
- 'tests-wordpress:/var/www/html'
- '/Users/cwr/src/cwra/foo:/var/www/html/wp-content/plugins/foo'
- 'phpunit-uploads:/var/www/html/wp-content/uploads'
environment:
LOCAL_DIR: html
WP_PHPUNIT__TESTS_CONFIG: /var/www/html/phpunit-wp-config.php
volumes:
wordpress: {}
tests-wordpress: {}
mysql: {}
phpunit-uploads: {}
docker logs could help you here as it is also showing logs of exited containers (Source).
UPDATE: Acc. to the OP, the actual issue is still unclear but starting with a fresh docker & node installation did the trick & got wp-env running.
I'm totally new to Flyway but I'm trying to migrate a number of identical test databases using the docker-compose flyway+mysql arrangement described in https://github.com/flyway/flyway-docker
As far as I can tell, the migrate command can take multiple schemas in its -schemas argument but it only seems to apply the actual SQL migration to the first schema in the list.
For example, when I run the migrate with schemas=test_1,test_2,test_3, flyway creates all three databases but only creates the tables specified in the migration file on the first test_1 database.
Is there a way to apply the SQL migration file to all the schemas in the list?
I'm going to leave this question up in case someone can still answer how multiple schemas is useful if the migration file isn't applied to all databases in the list. But, I was able to handle multiple databases in a docker-compose by overriding the flyway entrypoint and command.
So now my docker-compose service looks like:
services:
flyway:
image: flyway/flyway:6.1.4
volumes:
- ./migrations:/flyway/sql
depends_on:
- db
entrypoint: ["bash"]
command: > -c "/flyway/flyway -url=jdbc:mysql://db -schemas=test1 migrate;
/flyway/flyway -url=jdbc:mysql://db -schemas=test2 migrate"
For me what worked was breaking up my migrations into separate executions in my docker-compose file along with docker-postgresql-multiple-databases as follows:
version: '3.8'
services:
postgres-db:
image: 'postgres:13.3'
environment:
POSTGRES_MULTIPLE_DATABASES: 'customers,addresses'
POSTGRES_USER: 'pocketlaundry'
POSTGRES_PASSWORD: 'iceprism'
volumes:
- ./docker-postgresql-multiple-databases:/docker-entrypoint-initdb.d
expose:
- '5432' # Publishes 5432 to other containers (addresses-flyway, customers-flyway) but NOT to host machine
ports:
- '5432:5432'
addresses-flyway:
image: flyway/flyway:7.12.0
command: -url=jdbc:postgresql://postgres-db:5432/addresses -schemas=public -user=pocketlaundry -password=iceprism -connectRetries=60 migrate
volumes:
- ./sports-ball-project/src/test/resources/db/addresses/migrations:/flyway/sql
depends_on:
- postgres-db
links:
- postgres-db
customers-flyway:
image: flyway/flyway:7.12.0
command: -url=jdbc:postgresql://postgres-db:5432/customers -schemas=public -user=pocketlaundry -password=iceprism -connectRetries=60 migrate
volumes:
- ./sports-ball-project/src/test/resources/db/customers/migrations:/flyway/sql
depends_on:
- postgres-db
links:
- postgres-db
I am working on the .NET Core solution below:
and I am using the Gitlab CI definition below:
.gitlab-ci.yml:
image: docker:stable
services:
- docker:dind
stages:
- test
- build
- deploy-db
- deploy
variables:
DOCKER_REGISTRY: "our.registry.eu"
IMAGE_NAME: "${DOCKER_REGISTRY}/membercreditor"
# secrets are on GitLab environment variables (settings > ci/cd)
RANCHER_ENV: 1a10
RANCHER_STACK: api
RANCHER_SVC: membercreditor
RANCHER_DEV_KEY: a dev key
RANCHER_DEV_URL: http://our.dev.rancher.io:8080
RANCHER_PREPROD_KEY: a preprod key
RANCHER_PREPROD_URL: http://our.preprod.rancher.io:8080
RANCHER_PROD_KEY: a prod key
RANCHER_PROD_URL: http://our.prod.rancher.io:8080
.test:
stage: test
image: mcr.microsoft.com/dotnet/core/sdk:3.1-alpine
script:
- dotnet test ./Rm.MemberCreditor.Api.Tests/ -l:junit /p:CollectCoverage=true /p:Include="[Rm.*]*" /p:CoverletOutput=./TestResults/
- dotnet test ./Rm.MemberCreditor.Domain.Tests/ -l:junit /p:CollectCoverage=true /p:Include="[Rm.*]*" /p:CoverletOutput=./TestResults/ /p:MergeWith=../Rm.MemberCreditor.Api.Tests/TestResults/coverage.json
- dotnet test ./Rm.MemberCreditor.IntegrationTests/ -l:junit /p:CollectCoverage=true /p:Include="[Rm.*]*" /p:CoverletOutput=./TestResults/ /p:MergeWith=../Rm.MemberCreditor.Domain.Tests/TestResults/coverage.json
coverage: '/Total[ |]+(\d+[,.]\d+)%/'
artifacts:
reports:
junit: ./*.Tests/TestResults/*.xml
.build:
stage: build
tags:
- dind
script:
- docker build -t $IMAGE_NAME:$APP_VERSION -f Dockerfile .
- docker login $DOCKER_REGISTRY -u svc_finance_revmgt -p $SVC_FINANCE_REVMGT_TOKEN
- docker push $IMAGE_NAME:$APP_VERSION
.deploy-db:
stage: deploy-db
image: mcr.microsoft.com/dotnet/core/sdk:3.1-alpine
script:
- export CONNECTIONSTRINGS__MEMBERCREDITOR=$CONNECTIONSTRINGS__MEMBERCREDITOR
- export SECRETS__CS_MEMBERCREDITOR=$SECRETS__CS_MEMBERCREDITOR
- dotnet tool restore
- dotnet ef database update -p ./Rm.MemberCreditor.Data.Migrations -s ./Rm.MemberCreditor.Api -v
.deploy:
stage: deploy
image: cdrx/rancher-gitlab-deploy
script:
- upgrade --rancher-key $RANCHER_KEY
--rancher-secret $RANCHER_SECRET
--rancher-url $RANCHER_URL
--environment $RANCHER_ENV
--stack $RANCHER_STACK
--service $RANCHER_SVC
--new-image $IMAGE_NAME:$APP_VERSION
###########################
### Merge Requests
###########################
test-merge-request:
extends: .test
only:
refs:
- merge_requests
###########################
### Integration
###########################
test-integration:
extends: .test
environment:
name: integration
only:
- develop
build-integration:
extends: .build
environment:
name: integration
dependencies:
- test-integration
before_script:
- APP_VERSION=latest-integration
only:
- develop
deploy-db-integration:
extends: .deploy-db
environment:
name: integration
dependencies:
- build-integration
before_script:
- export ASPNETCORE_ENVIRONMENT=Integration
only:
- develop
deploy-integration:
extends: .deploy
environment:
name: integration
dependencies:
- deploy-db-integration
before_script:
- APP_VERSION=latest-integration
- RANCHER_KEY=$RANCHER_DEV_KEY
- RANCHER_SECRET=$RANCHER_DEV_SECRET
- RANCHER_URL=$RANCHER_DEV_URL
only:
- develop
###########################
### UAT
###########################
test-uat:
extends: .test
environment:
name: uat
only:
- master
build-uat:
extends: .build
environment:
name: uat
dependencies:
- test-uat
before_script:
- APP_VERSION=latest-uat
only:
- master
deploy-db-uat:
extends: .deploy-db
environment:
name: uat
dependencies:
- build-uat
before_script:
- export ASPNETCORE_ENVIRONMENT=Uat
only:
- master
deploy-uat:
extends: .deploy
environment:
name: uat
dependencies:
- deploy-db-uat
before_script:
- APP_VERSION=latest-uat
- RANCHER_KEY=$RANCHER_PREPROD_KEY
- RANCHER_SECRET=$RANCHER_PREPROD_SECRET
- RANCHER_URL=$RANCHER_PREPROD_URL
only:
- master
###########################
### Production
###########################
build-prod:
extends: .build
environment:
name: production
before_script:
- APP_VERSION=$CI_COMMIT_TAG
only:
- /^\d+.\d+.\d+$/
except:
- /^(?:[^m]|m[^a]|ma[^s]|mas[^t]|mast[^e]|maste[^r]).*#/
deploy-db-prod:
extends: .deploy-db
environment:
name: production
dependencies:
- build-prod
before_script:
- export ASPNETCORE_ENVIRONMENT=Production
only:
- /^\d+.\d+.\d+$/
except:
- /^(?:[^m]|m[^a]|ma[^s]|mas[^t]|mast[^e]|maste[^r]).*#/
deploy-prod:
extends: .deploy
environment:
name: production
when: manual
dependencies:
- deploy-db-prod
before_script:
- APP_VERSION=$CI_COMMIT_TAG
- RANCHER_KEY=$RANCHER_PROD_KEY
- RANCHER_SECRET=$RANCHER_PROD_SECRET
- RANCHER_URL=$RANCHER_PROD_URL
only:
- /^\d+.\d+.\d+$/
except:
- /^(?:[^m]|m[^a]|ma[^s]|mas[^t]|mast[^e]|maste[^r]).*#/
When the Gitlab is running the test-merge-request, I am getting the error below:
Fetching changes with git depth set to 50...
00:02
Initialized empty Git repository in /builds/finance/products/revenuemgmt/modules/membercreditor/.git/
Created fresh repository.
From https://our.org/finance/products/revenuemgmt/modules/membercreditor
* [new ref] refs/merge-requests/38/head -> refs/merge-requests/38/head
* [new ref] refs/pipelines/355543 -> refs/pipelines/355543
Checking out 603e9a2c as refs/merge-requests/38/head...
Skipping Git submodules setup
$ dotnet test ./Rm.MemberCreditor.Api.Tests/ -l:junit /p:CollectCoverage=true /p:Include="[Rm.*]*" /p:CoverletOutput=./TestResults/
00:12
FSC : error FS0078: Unable to find the file 'Legacy/Queries/GetOrderRelatedInfoForAccountingEntryCreation.sql' in any of /builds/finance/products/revenuemgmt/modules/membercreditor/Rm.MemberCreditor.Data [/builds/finance/products/revenuemgmt/modules/membercreditor/Rm.MemberCreditor.Data/Rm.MemberCreditor.Data.fsproj]
Uploading artifacts...
00:02
WARNING: ./*.Tests/TestResults/*.xml: no matching files
ERROR: No files to upload
ERROR: Job failed: exit code 1
GetOrderRelatedInfoForAccountingEntryCreation.sql is defined as an embedded resource, I tried to force to copy the file to the output but I still am getting the same very same error.
I am not sure why this post has been down-voted, but fun fact, seems there is a bug in Rider 2019.3.1.
The issue arose because even though the file was showing up as GetOrderRelatedInfoForAccountingEntryCreation.sql in Rider aka pascal case it was actually written as a camel case (i.e. getOrderRelatedInfoForAccountingEntryCreation.sql) on the disk, go figure.
Once the name adjusted to pascal for the actual file on the disk, everything worked fine.
I'm usin REDIS in my Symfony app and today Gitlab CI stopped working with error (after composer install and cache clear):
!! In AbstractConnection.php line 155:
!!
!! php_network_getaddresses: getaddrinfo failed: Name does not resolve [tcp://
!! redis:6379]
Here is my gitlab-ci.yml:
image: docker:latest
services:
- docker:dind
- name: redis:5-alpine
alias: redis
stages:
- dependencies
- build-docker
variables:
#REDIS_PORT: 6379 - doesn't help
#REDIS_HOST: redis - doesn't help
dependencies:
stage: dependencies
image: docker
cache:
paths:
- bin
- vendor
- config/packages
- public/bundles
policy: push
script:
- sh ./run composer install --ignore-platform-reqs --prefer-dist --optimize-autoloader
except:
- tags
Gitlab CI works fine until today's morning... Can anyone help? Thanks a lot.