Mixing creation and migration scripts with Flyway - flyway

Can Flyway be used to mix creation and migration scripts so that:
new installations run a schema creation script
existing installations run migration scripts, and never see the creation scripts of subsequent versions
?
E.g. given:
db/create/V1/V1__schema.sql
db/create/V2/V2__schema.sql
db/create/V3/V3__schema.sql
db/migration/V1/V1.1__migrateA.sql
db/migration/V2/V2.1__migrateB.sql
db/migration/V2/V2.2__migrateC.sql
An existing V1 installation would run the following to get to V3:
db/migration/V1/V1.1__migrateA.sql
db/migration/V2/V2.1__migrateB.sql
db/migration/V2/V2.2__migrateC.sql
It would never run the following, as these represent schema-only SQL produced by mysqldump:
db/create/V2/V2__schema.sql
db/create/V3/V3__schema.sql
A new V3 installation would run:
db/create/V3/V3__schema.sql
The above conflicts with the approach recommended by Upgrade scenario when using Flyway but is required as data is populated independently of the migration.
It looks like it should be possible to use flyway.locations to support this, but installations would always need to include the path to their creation script so that Flyway can see it.
The alternative appears to be to run the creation scripts outside of Flyway and set a baseline, but it would be nice if Flyway could manage everything.

In the end, I developed a tool to do this.
The tool has the latest schema in:
db/schema/schema.sql
and the migration scripts in:
db/migration/<version>/<version>.<sequence>__<JIRA issue>.sql
e.g.:
db/migration/V1/V1.1__JIRA-15.sql
db/migration/V2/V2.1__JIRA-12.sql
db/migration/V2/V2.2__JIRA-22.sql
db/migration/V3/V3.0__JIRA-34.sql
If the database has no tables, schema.sql is executed, and then flyway is baselined with the most recent version, as reported by Flyway's
MigrationInfoService.pending() method.
i.e. the last MigrationInfo element returned by pending() determines the version to pass to Flyway.setBaselineVersion() before invoking Flway.baseline()
e.g:
DbSupport support = DbSupportFactory.createDbSupport(connection, true);
Schema schema = support.getOriginalSchema();
if (schema.allTables().length == 0) {
Resource resource = new ClassPathResource("db/schema/schema.sql", getClass().getClassLoader());
SqlScript script = new SqlScript(resource.loadAsString("UTF-8"), support);
script.execute(support.getJdbcTemplate());
MigrationInfo[] pending = flyway.info().pending();
MigrationInfo version = pending.length > 0 ? pending[pending.length - 1] : null;
if (version != null) {
flyway.setBaselineVersion(version.getVersion());
flyway.setBaselineDescription(version.getDecription());
flyway.baseline();
}
}
This ensures that none of the migration scripts are invoked for newly created databases, but does mean that schema.sql must already contain all of the changes.
If the database has tables, but no Flyway information, it is baselined according to the detected schema version.

Related

Flyway Gradle Plugin with Cloud Spanner - No database found to handle url

I'm trying to setup Flyway for Google Cloud Spanner (beta) using the flyway gradle plugin but it gets the error below when executing ./gradlew flywayinfo.
> Error occured while executing flywayInfo
No database found to handle jdbc:cloudspanner:/projects/<my-project>/instances/<my-instance>/databases/<my-db>
build.gradle
plugins {
id 'java'
id 'org.flywaydb.flyway' version '7.13.0'
}
...
dependencies {
implementation(
'org.flywaydb:flyway-gcp-spanner:7.13.0-beta'
)
}
flyway {
url = 'jdbc:cloudspanner:/projects/<my-project>/instances/<my-instance>/databases/<my-db>'
}
The values in the url correspond to my project and instance names.
I've also tried:
using a service account key in the end of the URL
adding the com.google.cloud:google-cloud-spanner-jdbc:2.3.2 JDBC driver dependency (implementation)
I'm behind a proxy but I have set it in my gradle.properties with systemProp.http.proxyHost and systemProp.http.proxyPort (also for https)
Using Flyway CLI and the API programmatically works.
It seems like the error comes from flyway implementation here. Your issue seems somewhat similar with https://github.com/flyway/flyway/issues/3028.
Consider opening a new issue here: https://github.com/flyway/flyway/issues

MarkLogic 9.0.5 and DHF 3.0 fail to load mlcp module on mlDeploy

I have a clean marklogic server installation together with gradle DHF(Data Hub Framework) 3.0 project.
In the build.gradle, I have a custom task that triggers an input flow and loads the files by using MLCP.
The code looks like the following.
task importEntity(type: com.marklogic.gradle.task.MlcpTask) {
classpath = configurations.mlcp
command = "IMPORT"
...
...
transform_module = "/com.marklogic.hub/mlcp-flow-transform.xqy"
transform_namespace = "http://marklogic.com/data-hub/mlcp-flow-transform"
... }
When the command is run, the execution fails due to a non-existing module (/com.marklogic.hub/mlcp-flow-transform.xqy) in MarkLogic modules database.
If I check modules database, I see that the only custom modules were uploaded but not a single dependency from MarkLogic on mlDeploy.
I am wondering, what could have gone wrong.
Thank in advance.
Best,
Igor

Why does Flyway ignoreMissingMigrations is not taken into account?

I have a setup in which migrations from previous scripts were removed.
The flyway configuration specifies that ignoreMissingMigrations is true.
However, Flyway fails with the following error
Validate failed: Detected applied migration not resolved locally: version_x
where version_x is the first version that was removed after baseline.
Why do I get this error although ignoreMissingMigrations is true ?
Note: Flyway version: 4.2.0
The problem comes from a special setup that Flyway is unable to handle correctly.
We have no newer applied migration, thus Flyway see this migration as a future migration instead of a missing migration. Thus the solution is to set ignoreFutureMigrations to true in addition to ignoreMissingMigrations.

Sudden syntax error after deployment

I have a working symfony project. I have it on a private bitbucked repository and locally the website works without an issue.
Today I tried to deploy the project onto an external server linuxpl.com.
Steps taken include:
Istalling composer
Adding the mysql database
Running git clone to get the data into a proper location
Running composer install on the folder to install everything and connect to the db
Cleared the cache
Set the project root as ....domain/project_name/web
However after completing all these steps, when running the website with regular server:run I'm getting this odd error:
Parse error: syntax error, unexpected '.' in /home/spirifer/domains/surowcewobiektywie.pl/konkurs/vendor/twig/twig/lib/Twig/Extension/Core.php on line 1571
Not sure if this is of any importance but the mentioned code partion looks like this in my local files:
// Some objects throw exceptions when they have __call, and the method we try
// to call is not supported. If ignoreStrictCheck is true, we should return null.
try {
$ret = $object->$method(...$arguments);
} catch (BadMethodCallException $e) {
if ($call && ($ignoreStrictCheck || !$env->isStrictVariables())) {
return;
}
throw $e;
}
The local version does not differ from the one on the server.
My local machine has PHP 7.0.9 and the remove server has PHP 7.0.14
How could I fix this issue?
PHP 5.6 adds Variadic functions, with "...". However, Twig v1.x only required the use of PHP 5.2.7 or above.
If you didn't explicitly update to Twig 2.0, it's very possible you have used the 'death star' version constraint in the composer file - '*'. which allows uncontrolled version updates to the latest version. If this is the case, you will need to either update your version of PHP, or at least require just a previous version of Twig/twig, "^1.32" would be the latest in the version 1 series of Twig.

How to skip a specific migration with flyway?

I'm using flyway with gradle, I've run one of the migrations manually inside the database console, I want to run flyway, but tell it to ignore one specific migration version in between all the others.
Can this be done?
You would have to hack it a bit to get it to work, so I don't recommend this approach, but it would work in a pinch.
I've only tested this with Maven, but I'm pretty sure it'd work with Gradle too.
Migrate up until the version before the one you applied manually
# Assuming you applied 01.002 manually
$ mvn flyway:migrate -Dflyway.target=01.001
Insert a row for the script you applied
-- Make sure these vals closely replicate those from other rows
insert into schema_version( installed_rank, version, description, type, script, checksum, installed_by, installed_on, execution_time, success)
values ( 2, '01.002', 'static_data', 'SQL', 'V01/V01.002__static_data.sql', null, 'username', current_timestamp, 0, true );
Repair the schema_version checksum
$ mvn flyway:repair
Apply the other migrations
$ mvn flyway:migrate -Dflyway.validateOnMigrate=false -Dflyway.outOfOrder=true
The two -D properties there may not be necessary, depending on whether you got the insert correct or not. Flyway may disagree with your script description, for example, even if the checksum is now correct.
Not Recommended, but if you still want to:
Run flywayMigrate, let the migration fail.
Manually, update the flyway meta table (success column) for that specific version of migration.
Run flywayMigrate again.
Done, flyway will now start with the next version of migration.
As of version 7, you can add it directly to your Maven or Grade file
Gradle - Skip
flyway {
skipExecutingMigrations = true
}
Maven - Skip
<configuration>
<skipExecutingMigrations>true</skipExecutingMigrations>
</configuration>
Documentation Reference Skip
Gradle - Cherry Pick
flyway {
cherryPick = '2.0'
}
Maven - Cherry Pick
<configuration>
<cherryPick>2.0</cherryPick>
</configuration>
Documentation Reference Cherry Pick

Resources