I use Flyway-core 5.1.4 version in my Java Spring boot project.
I know if I change something in R_ migraion I will get new record in flyway_schema_history table with new checksum. But I got random apply of repeatable migrations in the end of my flyway_schema_history table without any changes in R- migration, even without additing any V_ migrations and without re compiling or redeploying of my project.
How is it possible ?
And it seems to me that I got old version of my R- migration, something like overriding.
Do you see the checksum of the repeatable migration change each time it is run? You should do, as that's the trigger for it to be re-run; and if that is happening then it suggests something is changing the file, perhaps only whitespace but that is enough.
Related
I am currently using the Flyway Command Line tool to manage our scripts which will be invoked via our release process triggered from our CI Build server.
The issue is I have 274 repeatable migrations covering package specs, package bodies, functions, procedures, views and materialised views.
When I run migrate everything works as expected with migrations executing followed by any changed repeatable migrations but lets say in the next release we want to delete a object which one of the repeatable migrations maintains. For example, we want to remove the repeatable script that defined ProcedureOne (ie R__ProcedureOne.sql).
To do this I would have a new migration script (V3.1.5.1.01__DropProcedureOne.sql) but I would also remove the repeatable migration script so the object isn't created again and maintained.
However, executing flyway info shows the R__ProcedureOne.sql script with the status of MISSING.
While I agree it is missing it is a deliberate action to have it deleted as it is no longer required opposed to being misplaced.
I am aware of the migrate option ignoreMissingMigrations but i think using this introduces risk and could mask genuine missing files.
What is the general guidance on how best to remove a repeatable scripts?
I suggest that you simply keep the file but make it empty (ie zero bytes). Alternatively have a comment in the file that explains that the object it represents has been removed.
As for actually removing it, another option from what you suggested could be to update the repeatable migration to remove itself then update again to zero length. This has the advantage of being able to be replayed into an empty database; since repeatable migrations are applied after versioned the procedure in your example won't exist to be dropped. The disadvantage is running two migrations.
So while building a new database using our database migration scripts written in a springboot flyway project, we realized we made some mistakes.
Some old scripts need to be changed to ensure that we do not face these issues when we make a new database schema again. These issues are mostly related - an info table was not populated with entries in the project and there are scripts that refer to the data in the migration project -- this data does not exist because we never included a script to include data.
How can we correct this project - the only way I can think of is to correct scripts such that all inserts are replaced by - insert if not exists or replace create statements by create if not exists.
and then delete all entries in schema version and re-run this on all the database which are using this schema.
I cannot go back and correct my script because then the migration project will fail because of checksum issues.
You are rigth, if this project and the scripts are running in some existing projects you can not modify them because the checksum would fail.
Then the cleanest way I can think would be add a file called "DB-GENERAL-FIXES" or something like that, where you can add all SQL validations to restore the DB to a stable status. For the new implementations will be extra work first build it wrongly and then clean it, but if you are sharing the same code in production right now...is the best option
We are using Flyway to migrate the database schema and we already have more than 100 migration scripts.
Once we "squashed" multiple migrations into a single first-version migration, this is ok during development, as we drop and recreate the schema. But in production this wouldn't work, as Flyway won't be able to validate the migrations.
I couldn't find any documentation or best practice of what to do in this case. The problem is that the file quantity increases constantly, I don't want to see thousands of migration files everytime, essentially if production is already in the latest version. I mean, the migration scripts that have a version number that is lower than the version in production are irrelevant to us, it would be awesome if we could squash those files into a single migration.
We are using MySQL.
How should we handle this?
Isn't that what re-baselining would do?
I'm still new to flyway, but this is how I think it would work. Please test the following first before taking my word for it.
Delete the schema_version table.
Delete your migration scripts.
Run flyway baseline
(this recreates the schema_version table and adds a baseline record as version 1)
Now you're good to go. Bear in mind that you will not be able to 'migrate' to any prior version as you've removed all of your migration scripts, but this might not be a problem for you.
Step by step solution:
drop table schema_version;
Export database structure as a script via MySQL Workbench, for example. Name this script V1__Baseline.sql
Delete all migration scripts and add V1__Baseline.sql to your scripts folder, so it is the only script available for Flyway
Run Flyway's "baseline" command
Done
We do this to allow us to compress scripts for building new DB in dev environments but also run against existing production DB without having to log on and delete the flyway_version_history table, and we can keep the scripts (mainly for reference):
Compress all the scripts to a new script e.g. V1 to V42 into a new scripts V43.
Convert V1 to V42 to text files by putting .txt on the end.
Set the baseline to 43.
Set flyway to ignore missing migrations.
In script V43 use an 'if' block to protect the create/insert statements so that they don't run for the existing production database. We use postgres so it is something like this:
DO $$
DECLARE
flywayVersion INTEGER;
BEGIN
SELECT coalesce(max(installed_rank), 0) INTO flywayVersion FROM flyway_schema_history;
RAISE NOTICE 'flywayVersion = %', flywayVersion;
IF flywayVersion = 0 THEN
RAISE NOTICE 'Creating the DB from scratch';
CREATE TABLE...
.....
END IF;
END$$;
The flyway command looks something like this:
Flyway.configure()
.dataSource(...)
.baselineVersion("43")
.ignoreMissingMigrations(true)
.load()
.migrate()
I haven't tried this, but what if you deleted all the migrations, create a new migration that creates the new starting point as version 1, set it as the baseline version -- and then modify your configuration to use a different table (e.g. flyway_schema_history_2)?
In existing databases, Flyway will see that you have a non-empty schema with no (recognized) flyway table and ignore the baseline migration. In new environments it will run the baseline migration too.
Am I missing anything?
(Of course a separate problem is how to generate the "compressed" migration. If you don't need any seed data you can just do a schema-only backup of your database and use that. If your migrations populate data too you will probably have to work that out manually.)
I think this article answers your question best:
https://medium.com/att-israel/flyway-squashing-migrations-2993d75dae96
For postgres a reusable script has been created that you could execute every so many months for instance. You can of course adapt the script to MySQL specific things instead of postgres:
https://github.com/the-serious-programmer/flyway-postgres-squash-script
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/
db-maintain has the notion of repeatable scripts, such as stored procedures/triggers. When there is a change to such a script it needs to be rolled out again. Flyway seemingly always checks for the filename convention. So to have a repeatable script we might need to rename the file. Is there a more efficient way of doing this?
Flyway started to support repeatable migrations in version 4.0. Repeatable migrations are reapplied every time their checksum changes and can be maintained as single files in version control.
It is your responsibility to ensure the same repeatable migration can
be applied multiple times. This usually involves making use of CREATE
OR REPLACE clauses in your DDL statements.
More details here.
This issue has come up in the Issue Tracker and in this other question.
There is currently no out of the box support for this.
Personally I would
package the repeatable actions in a stored procedure or trigger and add it to the DB as part of a regular migration
make sure this procedure/trigger gets invoked once per migration after that (could be as little a one-line statement)
ensure changes to the procedure/trigger necessary after that also happen as part of regular migrations
If that doesn't do it, feel free to star the Issue and comment with details about your use case.
Update: Repeatable scripts are now fully supported as of Flyway 4.0. See https://flywaydb.org/documentation/migration/repeatable