How to use placeholders prefix in flyway command line - flyway

Originally posted in https://github.com/flyway/flyway/issues/2429
I have an issue (probably a wrong configuration) using flyway placeholders; I can use placeholders for my own variables; but it fails cos one value in an sql query has a similar syntax as flyway placeholder syntax.
Which version and edition of Flyway are you using?
5.2.4 using official docker image
If this is not the latest version, can you reproduce the issue with the latest one as well? (Many bugs are fixed in newer releases and upgrading will often resolve the issue)
5.2.4 tag is the latest version in docker hub (https://hub.docker.com/r/boxfuse/flyway/)
Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)
Command line thru the docker image
Which database are you using (type & version)?
MySQL Server version: 5.7.26 - MySQL Community Server (GPL) - This is a legacy project
Which operating system are you using?
Linux CentOS 7 x64 (uname -r = 3.10.0-957.5.1.el7.x86_64)
What did you do?
(Please include the content causing the issue, any relevant configuration settings, the SQL statement that failed (if relevant) and the command you ran.)
I apply flyway to initialize/update a MySQL database; here are a couple of SQL commands.
Here I use placeholders with xxx prefixes:
CREATE USER IF NOT EXISTS '${xxxdbuser}'#'${xxxdbclip}' IDENTIFIED WITH mysql_native_password BY '${xxxdbpass}';
GRANT ALL PRIVILEGES ON ${xxxdbbase}.* TO '${xxxdbuser}'#'${xxxdbclip}';
FLUSH PRIVILEGES;
... then in another SQL script, from a thirdparty app, I insert a content with ${row}. I don't want Flyway to interpret ${row} as a placeholder, only my own vars starting by ${xxx such as ${xxxdbuser}
INSERT INTO `xxx_xxx` (`name`, `template`, `lang`, `group`, `version`, `data`, `size`, `style`, `modified`) VALUES
... ('addressbook.email.rows', '', '', 0, '1.3.001', 'a:1:{i:0;a:6:{ ... \"label\";s:21:\"$row_cont[type_label]\";s:4:\"name\";s:12:\"${row}[type]\";s:5:\"align\";... :{i:0;s:4:\"100%\";}}}', '100%', '', 1150326789), ...
I guess the placeholderPrefix parameter described into https://flywaydb.org/documentation/commandline/info or FLYWAY_PLACEHOLDER_PREFIX env var described into https://flywaydb.org/documentation/envvars#FLYWAY_PLACEHOLDER_PREFIX is for that purpose; but I didn't succeed in using them!
Here is my command using docker:
docker run --rm --network="$(docker network ls --filter name=app_mysql_dev --filter "label=type=app" --format '{{.ID}}')" \
-v `pwd`/code/Admin/install:/flyway/sql \
-e FLYWAY_URL=jdbc:mysql://${host}:${port}?useSSL=false \
-e FLYWAY_SCHEMAS=${base} \
-e FLYWAY_USER=root \
-e FLYWAY_PASSWORD=${root_pwd} \
-e FLYWAY_PLACEHOLDERS_PREFIX="\${xxx" \
-e FLYWAY_PLACEHOLDERS_XXXDBBASE=${base} \
-e FLYWAY_PLACEHOLDERS_XXXDBUSER=${user} \
-e FLYWAY_PLACEHOLDERS_XXXDBPASS=${pass} \
-e FLYWAY_PLACEHOLDERS_XXXDBCLIP=${clip} \
-e FLYWAY_PLACEHOLDERS_XXXVHOST=${vhost} \
-e FLYWAY_PLACEHOLDERS_XXXSCHEME=${scheme} \
-e FLYWAY_CONNECT_RETRIES=5 \
boxfuse/flyway:5.2.4 -locations=filesystem:/flyway/sql/custom/ \
migrate
What did you expect to see?
All ${xxx placeholders should be replaced by their corresponding ENV values; and the ${row} chain in SQL code stay unchanged.
What did you see instead?
Flyway error:
Flyway Community Edition 5.2.4 by Boxfuse
Database: jdbc:mysql://tasks.atlas-mysql:3306 (MySQL 5.7)
ERROR: No value provided for placeholder expressions: ${row}. Check your configuration!
I guess I did not configure my command correctly... any help, advise and/or command line example would help.
Regrads,
Chris

I think there are a couple of problems in your command:
-e FLYWAY_PLACEHOLDERS_PREFIX="\${xxx"
should be FLYWAY_PLACEHOLDER_PREFIX (no S), and
-e FLYWAY_PLACEHOLDERS_XXXDBBASE=${base}
should be FLYWAY_PLACEHOLDERS_DBBASE (as XXX is part of the prefix, it's not included in the placeholder name; and analogously for following lines).

Related

Create Airflow connections on Cloud Composer using gcloud CLI

I am trying to create airflow connections on Cloud Composer using gcloud CLI.
I follow the document and run the following comment.
https://cloud.google.com/composer/docs/how-to/managing/connections#creating_new_airflow_connections
gcloud composer environments run my-env \
--project my-project \
--location asia-northeast1 \
connections -- --add \
--conn_id=my-conn-id \
--conn_type=google_cloud_platform \
--conn_extra '{"extra\__google\_cloud\_platform\__project": "my-project", \
"extra\__google\_cloud\_platform\__key_path": "/home/airflow/gcs/data/keys/my-key.json", \
"extra\__google\_cloud\_platform\__scope": "https://www.googleapis.com/auth/cloud-platform"}'
kubeconfig entry generated for asia-northeast1-my-env-44718514-gke.
Executing within the following kubectl namespace: default
W0525 22:51:11.244104 93234 flags.go:39] conn_extra is DEPRECATED and will be removed in
a future version. Use conn-extra instead.
W0525 22:51:11.244246 93234 flags.go:39] conn_extra is DEPRECATED and will be removed in
a future version. Use conn-extra instead.
W0525 22:51:11.244256 93234 flags.go:39] conn_extra is DEPRECATED and will be removed in
a future version. Use conn-extra instead.
[2019-05-25 14:51:13,215] {settings.py:176} INFO - setting.configure_orm():
Using pool settings. pool_size=5, pool_recycle=1800
[2019-05-25 14:51:13,598] {default_celery.py:80} WARNING - You have configured a
result_backend of redis://airflow-redis-service:6379/0, it is highly recommended to use an
alternative result_backend (i.e. a database).
[2019-05-25 14:51:13,600] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-05-25 14:51:13,680] {app.py:51} WARNING - Using default Composer Environment Variables.
Overrides have not been applied.
[2019-05-25 14:51:13,688] {configuration.py:516} INFO - Reading the config from
/etc/airflow/airflow.cfg
[2019-05-25 14:51:13,698] {configuration.py:516} INFO - Reading the config from
/etc/airflow/airflow.cfg
The connection is successfully created but project ID, key file path and scope are empty. So the connection is invalid.
When I manually created, those properties are not empty. Am I missing something?
Composer image: composer-1.5.0-airflow-1.10.1
I am not able to replicate it. When I run the following command the connection does get added with the extras field:
gcloud composer environments run my-env \
--project my-project \
--location europe-west1 \
connections -- --add \
--conn_id=my-conn-id \
--conn_type=google_cloud_platform \
--conn_extra='{"extra__google_cloud_platform__project": "my-project", "extra__google_cloud_platform__key_path":"/tesf"}
Found some syntax error on escaping quotation. This works fine.
$ CONNECTION_CREATE_COMMAND="gcloud composer environments run $COMPOSER_ENVIRONMENT \
--project $COMPOSER_PROJECT \
--location ${COMPOSER_LOCATION} \
connections -- --add \
--conn_id=${CONN_ID_BASE}_${app}_${c} \
--conn_type=google_cloud_platform \
--conn_extra '{\"extra__google_cloud_platform__project\": \"${BQ_PROJECT}\", \
\"extra__google_cloud_platform__key_path\": \"${KEY_JSON_FILE_PATH}\", \
\"extra__google_cloud_platform__scope\": \"https://www.googleapis.com/auth/cloud-platform\"}'"
$ eval $CONNECTION_CREATE_COMMAND

MUP (meteor) deploy deletes files

Sadly although mup deploy works perfectly, I have a folder called ".uploads", which users can upload files into.
Each deploy deletes the files in the directory. I would like to exclude or protect the file from the deploy deleting the files, any ideas?
Currently I filed an issue: https://github.com/arunoda/meteor-up/issues/1022
Not sure if its an issue with mup or my system setup. I use tomitrescak:meteor-uploads and also cfs:file-collection. They both have the same issue.
But from what I see it shall be easy to do, you need to modify the script at https://github.com/arunoda/meteor-up/blob/mupx/templates/linux/start.sh#L26
And add a new mapping. You can map multiple volumes such as following: Mounting multiple volumes on a docker container?
So your script would look like (mounting the folder on the host machine /opt/uploads/myapp to the folder /opt/uploads in the container):
docker run \
-d \
--restart=always \
--publish=$PORT:80 \
--volume=$BUNDLE_PATH:/bundle \
--volume=/opt/uploads/myapp:/opt/uploads/ \ .... this is the new line
--env-file=$ENV_FILE \
--link=mongodb:mongodb \
--hostname="$HOSTNAME-$APPNAME" \
--env=MONGO_URL=mongodb://mongodb:27017/$APPNAME \
--name=$APPNAME \
meteorhacks/meteord:base
This can be found in "start.sh" in MUP until this issue is resolved and you can mount volumes via the options.
Also see discussion here: https://github.com/tomitrescak/meteor-uploads/issues/235#issuecomment-228618130

Meteor Up Docker and Graphicsmagick

I'm looking for how to install Graphicsmagick at Meteor Up Docker.
I found this solution (Access binaries inside docker) but I couldn't make work, where do I put those lines at start.sh?
meteorDockerId=docker ps | grep meteorhacks/meteord:base | awk '{print $1}'
docker exec $meteorDockerId apt-get install graphicsmagick -y
That's my start.sh:
#!/bin/bash
APPNAME=instagatas
APP_PATH=/opt/$APPNAME
BUNDLE_PATH=$APP_PATH/current
ENV_FILE=$APP_PATH/config/env.list
PORT=80
USE_LOCAL_MONGO=0
# remove previous version of the app, if exists
docker rm -f $APPNAME
# remove frontend container if exists
docker rm -f $APPNAME-frontend
set -e
docker pull meteorhacks/meteord:base
if [ "$USE_LOCAL_MONGO" == "1" ]; then
docker run \
-d \
--restart=always \
--publish=$PORT:80 \
--volume=$BUNDLE_PATH:/bundle \
--env-file=$ENV_FILE \
--link=mongodb:mongodb \
--hostname="$HOSTNAME-$APPNAME" \
--env=MONGO_URL=mongodb://mongodb:27017/$APPNAME \
--name=$APPNAME \
meteorhacks/meteord:base
else
docker run \
-d \
--restart=always \
--publish=$PORT:80 \
--volume=$BUNDLE_PATH:/bundle \
--hostname="$HOSTNAME-$APPNAME" \
--env-file=$ENV_FILE \
--name=$APPNAME \
meteorhacks/meteord:base
fi
docker pull meteorhacks/mup-frontend-server:latest
docker run \
-d \
--restart=always \
--volume=/opt/$APPNAME/config/bundle.crt:/bundle.crt \
--volume=/opt/$APPNAME/config/private.key:/private.key \
--link=$APPNAME:backend \
--publish=443:443 \
--name=$APPNAME-frontend \
meteorhacks/mup-frontend-server /start.sh
Re-installing the graphicsmagick package every time you re-start the containers seems like a hack I wouldn't want to do.
If you're modifying the start script already, might as well use a Dockerfile:
FROM meteorhacks/meteord:base
RUN apt-get install graphicsmagick -y
Then modify start.sh template to build a new docker image with graphicsmagick, tag it and use that image instead:
see: https://gist.github.com/so0k/7d4be21c5e2d9abd3743/revisions
EDIT: Where to put Dockerfile?
start.sh template is copied to /opt/<appName>/config/, currently the Dockerfile would need to be in that same directory (/opt/<appName>/config/Dockerfile)
see Linux init Task
Alternatively, you can specify specific Dockerfile with the -f flag for the docker build
Or your third option is to pipe Dockerfile to docker build using a here document
I've updated the start.sh gist, we no longer pull the meteord:base image and build it instead:
docker build -t meteorhacks/meteord:app - << EOF
FROM meteorhacks/meteord:base
RUN apt-get install graphicsmagick -y
EOF
The docker build will run every time, but as long as the requirements aren't changing, docker will use the docker images it cached.
The development Version of Meteor Up at Kadirahq allows specification of a custom Docker Image in the config file (mup.js).
MeteorD-Images with Graphicsmagick installed are available on Docker Hub.
This got me a working deployment (Meteor 1.3.2.4, Meter Up 309cefb, Node v5.4.1):
mup.js:
module.exports = {
…
meteor: {
dockerImage: 'ianmartorell/meteord-graphicsmagick',
…
},
};
I couldn't get the docker image that #bskp mentioned to work, so I figured out how to write one that uses abernix/meteord:base and then has graphicsmagick installed. Very simple, but it seems to be working for me on Meteor 1.4.1.1
I just did this in my mup.js file
docker: {
image: "joshjoe/meteor-graphicsmagick",
},
This was a huge pain to get working, so I'd be happy to help anyone who is struggling with this.
https://github.com/c316/meteor-graphicsmagick
If the if statement successes, you should be able to see a running container corresponding to the image you are grepping. In my opinion you can add the two lines after the fi to obtain the environment variable.
Build an image for get things right, but you can do temporary:
docker exec -it MeteorAppName apt-get install imagemagick -y
docker restart MeteorAppName
Check imagemagick: docker exec -it MeteorAppName convert -version
Why don't you add the following package meteor add cfs:graphicsmagick
https://atmospherejs.com/cfs/graphicsmagick
It tries to make sure Graphicsmagick is available. It worked for my use case i think it will work with docker too.

5125cc8e register_matching_rule: could not locate associated matching rule generalizedTimeMatch

After compiling openldap-2.4.33 on Centos 6.3 with the following options, I am unable to understand what this error is telling me:
Server was installed as a 'minimal' install, with the following addons:
yum install ntp autofs gcc make perl strace nmap tree rpm-build rpm-devel rpmdevtools rpm-libs rpm-python \
openssl openssl-devel perl-CPAN libtool libtool-ltdl-devel.x86_64 libtool-ltdl.x86_64 \
db4.x86_64 nss_db.x86_64 compat-db.x86_64 db4-devel.x86_64 \
tcp_wrappers.x86_64 tcp_wrappers-devel.x86_64 tcp_wrappers-libs.x86_64 \
unixODBC unixODBC-devel mysql-devel cyrus-sasl-devel.x86_64 perl-ExtUtils-Embed.x86_64 \
-y
After basic installation of the server as a VM on ESX, I ran the following ./configure to compile and install :
export CPPFLAGS="-I /usr/lib64/perl5/CORE"
export LDFLAGS="-L/usr/lib64 -L/usr/lib64/perl5/CORE"
export PERL_CPPFLAGS="`perl -MExtUtils::Embed -e ccopts -I/usr/lib64/perl5/CORE`"
ldconfig
./configure \
--prefix=/ \
--enable-shared --enable-debug --enable-dynamic --enable-syslog --enable-proctitle --enable-ipv6 \
--enable-local --enable-slapd --enable-cleartext --enable-crypt --enable-lmpasswd --enable-spasswd \
--enable-modules --enable-rewrite --enable-rlookups --enable-slapi --enable-slp --enable-wrappers \
--enable-backends --enable-bdb --enable-dnssrv --enable-hdb --enable-ldap --enable-mdb \
--enable-meta --enable-monitor --enable-null --enable-passwd --enable-perl --enable-relay \
--enable-shell --enable-sock --enable-sql --enable-overlays --enable-accesslog --enable-auditlog \
--enable-collect --enable-constraint --enable-dds --enable-deref --enable-dyngroup --enable-dynlist \
--enable-memberof --enable-ppolicy --enable-proxycache --enable-refint --enable-retcode --enable-rwm \
--enable-seqmod --enable-sssvlv --enable-syncprov --enable-translucent --enable-unique --enable-valsort \
--enable-perl --disable-ndb --with-cyrus-sasl --with-threads --with-tls --with-yielding-select \
--with-mp
I've taken the basic slapd.conf and only added my own dn.
When I run slaptest this is what I get:
slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d/
5125cefd register_matching_rule: could not locate associated matching rule generalizedTimeMatch for ( 2.5.13.28 NAME 'generalizedTimeOrderingMatch' SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 )
slap_schema_init: Error registering matching rule ( 2.5.13.28 NAME 'generalizedTimeOrderingMatch' SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 )
5125cefd slaptest: slap_schema_init failed
The only schema with some kind of clue is ppolicy.schema, but I'm at a loss as to what to do.
Those matching rules are internal, both are defined in OpenLDAP's servers/slapd/schema_init.c with no conditionals.
generalizedTimeMatch is defined first, and then generalizedTimeOrderingMatch, which refers to generalizedTimeMatch in its "associated" matching rule. The error originates in servers/slapd/mr.c as the matching rules are added.
The matching rules are built in an array of struct slap_mrule_defs_rec, and iterated over in order. There's no obvious way for that to fail.
Your list of options and overlays is quite, um, complete.
There is a chance that there's some incompatibility or dependency problem with the overlays, but I don't see it (several overlays add to the schema and use those matching rules as a side-effect of some attributes: dds, ppolicy, accesslog; as does the monitor backend).
My best guess is that there is some compile problem, possibly arising from compiler options, either optimisation/alignment and/or some stale .o file, but I'm guessing here. You don't include your actual make and install steps, there's a similarly slim chance that you have some conflict arising from an incomplete installation, or previous installation (old binaries or schema files).
I'd suggest:
make clean
make depend && make && make test
and see what happens (make test will take quite some time). It that works, then you might consider installing to /usr/local to avoid conflicting files. If that doesn't work then try a simple configure with minimal options:
./configure --with-threads --with-tls
and then add in just the modules and backend(s) you need.

symfony2 app console: no available command

After HDD crash, I had to reimport my symfony2 app into Eclipse from my SVN server.
After syncing everything, I can't use the console anymore. I only get 2 commands : list and help.
I tried:
php bin/vendors install --reinstall
At the end, I got the following message:
[InvalidArgumentException]
There are no commands defined in the "assets" namespace.
[InvalidArgumentException]
There are no commands defined in the "cache" namespace.
My configuration is pretty simple:
- ubuntu server 11.04 (64bits)
- virtualbox OSE
How can I fix it?
Here is the result of app/console list command:
oc#ubuntu-server:/var/www/projets/Simoov2/src$ app/console list
Symfony version 2.0.0-RC4 - app/dev/debug
Usage:
[options] command [arguments]
Options:
--help -h Display this help message.
--quiet -q Do not output any message.
--verbose -v Increase verbosity of messages.
--version -V Display this program version.
--ansi Force ANSI output.
--no-ansi Disable ANSI output.
--no-interaction -n Do not ask any interactive question.
--shell -s Launch the shell.
--env -e The Environment name.
--no-debug Switches off debug mode.
Available commands:
help Displays help for a command
list Lists commands
OK. As I thought, this is not related to symfony2 but related to virtualbox mounting system.

Resources