OUR SYSTEM
We are trying to put migrations as .sql files under version control. Developers would write a VN__*.sql file, commit to version control and a job that runs every 5 minutes would automatically migrate to a Dev and Test database. Once the change proved to not cause problems, someone else would run a manual job to run the migration on Production.
MY PROBLEM:
I had a demo migration that created a few tables. I checked V4__DemoTables.sql into version control on my PC.
On our Linux box a job that runs every 5 minutes extracted the new file from version control, then ran the flyway.sh file. It detected the file and executed it.
But the .sql file had a typo. And we are using Neteeza which has problems with flyway automatically wrapping a migration in a BEGIN TRAN ... END TRAN. So the migration created 2 tables, then aborted before the third.
No problem I thought. I dropped the 2 tables that the .sql file did create. Checked V4__ out of version control, fixed the typo and re-submitted it.
Five minutes later the update was extracted but flyway complains that the checksum does not match. So it will NOT run the updated V4__DemoTables.sql file.
How do I get flyway to accept the updated file and update the checksum in the SCHEMA_VERSION file in case of a typo?
Reading the docs it seems like the developers suggest I should have created a new V4_1_DemoTables.sql file with the fix's. But this would have collided with the commands in the V4__ file so this seemed wrong.
So here is what the docs imply I need to do:
Leave V4__ as a 'successful' migration according to the
SCHEMA_VERSION table.
Create V4_1_ to delete the tables that were created before the typo
line in V4__.
Create V4_2_ which has the typo fix from the original file to do all
the real work.
Is this correct?
If the migration completes successfully, but some of the db objects are not quite right yet (typo in column name, ...), do as you said and push a follow-up script that fixes it (rename column, ...).
If the migration fails, and it didn't run on a DB with DDL transaction, the DB must be cleaned manually. This means:
Reverting the effects of the migration on the DB
Removing the version from the SCHEMA_VERSION table and marking the previous one as current
This second step will be automated in the future with the introduction of the flyway.repair() command.
Related
I'm using EF Core and code migrations. It's my understanding that the snapshot file is supposed to be the "target database" to build future migration files off of.
Situation:
1) I did an add-migration to create a migration after some changes. This updated the modelsnapshot which I guess is the expected behavior.
2) I then realized I realized I mis-typed one of my fields. Instead of a string it was supposed to be a byte[]
3) I made the changes to the classes
4) I did add-migration again.
Now the DataSnapshot seems to be out of sync because the new add-migration does not include this type file change.
It doesn't seem like I can do a remove-migration because no migration was actually applied to any database.
I know in EF6, this was a fine way to do it because it always targeted an actual database for the changes and not this snapshot file. So just deleting the change file and remaking the change file was good enough.
This is the second time I've had to make some change to a migration file after realizing there was an issue (without actually running the update against anything) and both times were just incredible pains to try to fix the issue without wiping out all my previous migrations and starting over.
My question is how do I handle this now? Should I be deleting both the change file and the snapshot file every time I do backtrack on an add migration?
I'm really concerned that I'm missing something here because now I have to make sure that not only is the change file correct but also the current snapshot is correct as well and if that snapshot ever goes out of sync, then I have to go and manually update it to
Probably have solved this by now, but I just ran into this.
Not sure if this is exactly the same issue, but this is how I understand it.
update-database is a DB action and updates your DB to current version or target migration.
remove-migration is an EF action and updates your migration stack and updates your snapshot.
To downversion I had to update database then remove migration down to the intended target.
At this point adding a new migration gave me the expected set of changes, where as before it was giving me none.
This appeared to do what I expected from EF doing update-database and update-database -target
The only thing I think you can actually do wrong in this case is delete your migration files before doing the update, because the down steps are indicated in the migration files, so it wont be able to complete this step if you remove the files. If you have the files, you can freely target migrations to set your current DB state.
I believe the confusing part is that the snapshot was previously used on every migration, where as now there's one snapshot for "last migration"
I have run the Corda DB Migration tool and created a .jar containing .sql scripts of the tables that represent the States Objects of my CordApps as explained in https://docs.corda.r3.com/database-management.html#database-management-tool
However after putting the .jar in the /cordapps folder of my node, I got an error indicating the Checksum Verification is failing: "liquibase.exception.ValidationFailedException: Validation Failed: 2 change sets check sum..."
Do you know how could I reset the checksum of my objects so that the verification does not fail and I could start the Corda Node successfully?
Thank you
Stan
Truncate your migration related table i.e. databasechangelog & databasechangeloglock. I think truncating only databasechangelog should do the trick for you.
This can happen if you change existing migration scripts which have been run previously. Corda will attempt to check the migration scripts against what has been run and find that the scripts no longer match.
For example: "MyApp" version 1.0 has a migration "init" script that creates table FooBar. On starting the node the table is created and the databasechangelog and databasechangeloglock tables insert a record the migration.
Then "MyApp" version 1.1 comes out which has a modified version of the "init" script. This new script matches the name of the script previously run on version 1.0 but no longer matches the checksum as the content has changed.
Kid101's solution does work but you would only need to delete the relevant entry for the changed script in databasechangelog and databasechangeloglock. Also note that new Corda nodes will skip the original migration and start on version 1.1 which will bypass the issue.
So while building a new database using our database migration scripts written in a springboot flyway project, we realized we made some mistakes.
Some old scripts need to be changed to ensure that we do not face these issues when we make a new database schema again. These issues are mostly related - an info table was not populated with entries in the project and there are scripts that refer to the data in the migration project -- this data does not exist because we never included a script to include data.
How can we correct this project - the only way I can think of is to correct scripts such that all inserts are replaced by - insert if not exists or replace create statements by create if not exists.
and then delete all entries in schema version and re-run this on all the database which are using this schema.
I cannot go back and correct my script because then the migration project will fail because of checksum issues.
You are rigth, if this project and the scripts are running in some existing projects you can not modify them because the checksum would fail.
Then the cleanest way I can think would be add a file called "DB-GENERAL-FIXES" or something like that, where you can add all SQL validations to restore the DB to a stable status. For the new implementations will be extra work first build it wrongly and then clean it, but if you are sharing the same code in production right now...is the best option
We are using Flyway to migrate the database schema and we already have more than 100 migration scripts.
Once we "squashed" multiple migrations into a single first-version migration, this is ok during development, as we drop and recreate the schema. But in production this wouldn't work, as Flyway won't be able to validate the migrations.
I couldn't find any documentation or best practice of what to do in this case. The problem is that the file quantity increases constantly, I don't want to see thousands of migration files everytime, essentially if production is already in the latest version. I mean, the migration scripts that have a version number that is lower than the version in production are irrelevant to us, it would be awesome if we could squash those files into a single migration.
We are using MySQL.
How should we handle this?
Isn't that what re-baselining would do?
I'm still new to flyway, but this is how I think it would work. Please test the following first before taking my word for it.
Delete the schema_version table.
Delete your migration scripts.
Run flyway baseline
(this recreates the schema_version table and adds a baseline record as version 1)
Now you're good to go. Bear in mind that you will not be able to 'migrate' to any prior version as you've removed all of your migration scripts, but this might not be a problem for you.
Step by step solution:
drop table schema_version;
Export database structure as a script via MySQL Workbench, for example. Name this script V1__Baseline.sql
Delete all migration scripts and add V1__Baseline.sql to your scripts folder, so it is the only script available for Flyway
Run Flyway's "baseline" command
Done
We do this to allow us to compress scripts for building new DB in dev environments but also run against existing production DB without having to log on and delete the flyway_version_history table, and we can keep the scripts (mainly for reference):
Compress all the scripts to a new script e.g. V1 to V42 into a new scripts V43.
Convert V1 to V42 to text files by putting .txt on the end.
Set the baseline to 43.
Set flyway to ignore missing migrations.
In script V43 use an 'if' block to protect the create/insert statements so that they don't run for the existing production database. We use postgres so it is something like this:
DO $$
DECLARE
flywayVersion INTEGER;
BEGIN
SELECT coalesce(max(installed_rank), 0) INTO flywayVersion FROM flyway_schema_history;
RAISE NOTICE 'flywayVersion = %', flywayVersion;
IF flywayVersion = 0 THEN
RAISE NOTICE 'Creating the DB from scratch';
CREATE TABLE...
.....
END IF;
END$$;
The flyway command looks something like this:
Flyway.configure()
.dataSource(...)
.baselineVersion("43")
.ignoreMissingMigrations(true)
.load()
.migrate()
I haven't tried this, but what if you deleted all the migrations, create a new migration that creates the new starting point as version 1, set it as the baseline version -- and then modify your configuration to use a different table (e.g. flyway_schema_history_2)?
In existing databases, Flyway will see that you have a non-empty schema with no (recognized) flyway table and ignore the baseline migration. In new environments it will run the baseline migration too.
Am I missing anything?
(Of course a separate problem is how to generate the "compressed" migration. If you don't need any seed data you can just do a schema-only backup of your database and use that. If your migrations populate data too you will probably have to work that out manually.)
I think this article answers your question best:
https://medium.com/att-israel/flyway-squashing-migrations-2993d75dae96
For postgres a reusable script has been created that you could execute every so many months for instance. You can of course adapt the script to MySQL specific things instead of postgres:
https://github.com/the-serious-programmer/flyway-postgres-squash-script
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/