I am running flyway migrations on a postgres DB with existing tables. In doing so flyway threw this error
Found non-empty schema(s) "public" but no schema history table. Use baseline() or set baselineOnMigrate to true to initialize the schema history table.
After reading the docs on baselineOnMigrate, all we need to do is toggle this to true and we should be able to complete the migrations.
Need clarity on enabling this feature:
do we set FLYWAY_BASELINE_ON_MIGRATE to TRUE only once and when the schema history table is created we can turn it back to FALSE? Or do we have to persist the TRUE value?
Yes, you'd set it only once. That approach is really only used where Flyway takes all its parameters from the configuration file...it is just intended for a single use (e.g. just for one specific copy of a database, out of many). Normally, you baseline from an action at the command line like this:
Flyway baseline #FlywayArgs -baselineVersion='1.3.1' -baselineDescription='Existing version of MyDatabase'
This article is using a SQL Server database rather than Postgres but might give a good idea of how flyway baselines work: https://www.red-gate.com/hub/product-learning/flyway/flyway-baselines-and-consolidations.
Related
I have delivered a Product to the customer. Now I have upgraded the Product, which includes changes to the database.
Customer wants to upgrade the Product. Now will Flyway help in the migration of Customer data from older version to newer version. Please let me know, if this is a valid use case. The flyway documentation talks about its use during development only.
Flyway allows you to change your database by running a set of scripts in a defined order. These scripts are called 'migrations' as they allow you to 'migrate' your database from one version to another.
The idea is you can start with an an empty database and each migration script will successively bring that database up from empty up to the current version. However, it's also possible to start with an existing database by creating a 'baseline' migration.
As SudhirR said, Flyway's primary use case is to define schema changes. However, it's perfectly possible to change data also. Since Flyway is just running plain SQL, in principle almost anything you can do in a SQL script you can also do in a Flyway migration.
In the case you described it should be possible to use Flyway to migrate the customer database. The steps you could take are:
Generate a sql script that includes the entire DDL (including indexes, triggers, procedures, ...) of the production database. To do this you will need to add insert statements for all the reference data present in the database.
Save this script in your Flyway project as something like 'V1__base_version.sql'
Run the flyway baseline command against your production database
This will set up your production database for use with Flyway
Add a new migration script to migrate your customer's data to the new version
e.g. create new table, copy data from old table to new table, delete old table
Run flyway migrate to upgrade production
These steps are adapted from the Flyway documentation page here.
Of course you should read the Flyway docs and manually test on a throwaway DB before you run anything against production. However I think in principle Flyway could be a good fit for your use case.
Flyway should be used for schema migrations and any reference data (basic data that is required by the system/application in order to function properly).
Putting client specific data migrations would not be a use case. However, if you can represent the data migration "generically" by not using IDs and instead use names or types than it could be a candidate. Meaning if you could write a migration in a way that could be applied to all clients, then that would be the use case to put it in as a flyway migration.
Otherwise data migrations would be applied in some other way outside of the process like requesting special access to the database or having some team that manages the database to apply the scripts.
If you are doing custom data modifications quite often then I'd say something is wrong in some other area of the SDLC and you may need to increase testing so that bugs don't mess up the data in the first place.
I have the following.
An Entity Framework model that maps against a SQLSERVER database. This has production data, but there is no __MigrationHistory table. I need to keep the data.
A new updated model including new tables, columns etc.
I would like to add migrations so that model changes can be easily managed. I would like to support both MigrateDatabaseToLatestVersion by creating all default tables, as well as being able to apply migration #2 patches on the current data structure.
I have followed Microsofts tutorial "Code First Migrations with an existing database" but this gives me the problem of the initial -ignoreChanges resulting in not being able to create the model from scratch. I need to be able to both recreate it as well as add migrations on the existing database without migration history. This since the -ignoreChanges on the initial migration gives me an empty up(), and the second migration script will only contain code for the patched tables/columns. If I don't ignore changes then the framework will try to create tables that already exist.
I was considering to not ignore changes on the initial migration and then do some kind of "CREATE TABLE IF NOT EXISTS" for all existing tables, but that seems impossible.
I bet I'm missing something right out in the open. What is it?
not tested but it should do:
use old model to create a _MigrationHistory table
copy/paste (select/insert or any etl method) the _MigrationHistory in the production database
update the model to new one
create a new migration
push the migration to the db.
DO ALL THIS FIRST IN A TEST ENVIRONMENT !
I am working with Symfony and Doctrine. In the middle of the project I need to implement the Doctrine migrations, because the DB changes were too much and I need a better way to handle it.
How is the best way to start with the migrations, when there is already data on prod and it need to stay there and not to be touched?
My plan will be:
On my test system
drop all tables
run php bin/console doctrine:migrations:diff
The new automatic migration file, which I become holds all the table structures of my current state
go live to the table "migration_versions" and add the ID of this migration, so it will be skipped by the first run of the
migrations
run the migration php bin/console doctrine:migrations:migrate
In this way, I have all the structures from my entities, but I will not destroy my live data.
What do you thing?
If there is already some data on production, then your best take is to do a make:migration or doc:mig:diff from the current schema state. That will generate only the necessary sql that will update the new changes, but nothing else. Your first migration version will countain only the sql to update from the current state, and not from the beginning of times.
And also, once you have adopted this, you have to do every database modification under migrations. For example, if you need to add a non-nullable field, you usually add the new field with not null, then fill all the rows of that field with a default or calculated one, and then alter the table to make the field not nullable. Migrations will generate some boilerplate code to make your life easier, but it also requires a lot of care from the development team. Always test them first in a database that you can get rid of. And you will run into FK constraints and lots of other issues, but basically you have to solve them doing SQL.
Old thread here, but what I do in cases like this:
Backup dev database (structure and data, but with table create statements protected with checking so they are only created if they don’t already exist)
Drop all tables so the database is empty
Generate migration (since database is empty, the generated migration will constitute all commands necessary to generate your entire schema)
Run migration you just generated to build schema
Import test data from your dump
That puts you right back where you started but with an initial migration that can build your schema from nothing.
We are using Flyway to migrate the database schema and we already have more than 100 migration scripts.
Once we "squashed" multiple migrations into a single first-version migration, this is ok during development, as we drop and recreate the schema. But in production this wouldn't work, as Flyway won't be able to validate the migrations.
I couldn't find any documentation or best practice of what to do in this case. The problem is that the file quantity increases constantly, I don't want to see thousands of migration files everytime, essentially if production is already in the latest version. I mean, the migration scripts that have a version number that is lower than the version in production are irrelevant to us, it would be awesome if we could squash those files into a single migration.
We are using MySQL.
How should we handle this?
Isn't that what re-baselining would do?
I'm still new to flyway, but this is how I think it would work. Please test the following first before taking my word for it.
Delete the schema_version table.
Delete your migration scripts.
Run flyway baseline
(this recreates the schema_version table and adds a baseline record as version 1)
Now you're good to go. Bear in mind that you will not be able to 'migrate' to any prior version as you've removed all of your migration scripts, but this might not be a problem for you.
Step by step solution:
drop table schema_version;
Export database structure as a script via MySQL Workbench, for example. Name this script V1__Baseline.sql
Delete all migration scripts and add V1__Baseline.sql to your scripts folder, so it is the only script available for Flyway
Run Flyway's "baseline" command
Done
We do this to allow us to compress scripts for building new DB in dev environments but also run against existing production DB without having to log on and delete the flyway_version_history table, and we can keep the scripts (mainly for reference):
Compress all the scripts to a new script e.g. V1 to V42 into a new scripts V43.
Convert V1 to V42 to text files by putting .txt on the end.
Set the baseline to 43.
Set flyway to ignore missing migrations.
In script V43 use an 'if' block to protect the create/insert statements so that they don't run for the existing production database. We use postgres so it is something like this:
DO $$
DECLARE
flywayVersion INTEGER;
BEGIN
SELECT coalesce(max(installed_rank), 0) INTO flywayVersion FROM flyway_schema_history;
RAISE NOTICE 'flywayVersion = %', flywayVersion;
IF flywayVersion = 0 THEN
RAISE NOTICE 'Creating the DB from scratch';
CREATE TABLE...
.....
END IF;
END$$;
The flyway command looks something like this:
Flyway.configure()
.dataSource(...)
.baselineVersion("43")
.ignoreMissingMigrations(true)
.load()
.migrate()
I haven't tried this, but what if you deleted all the migrations, create a new migration that creates the new starting point as version 1, set it as the baseline version -- and then modify your configuration to use a different table (e.g. flyway_schema_history_2)?
In existing databases, Flyway will see that you have a non-empty schema with no (recognized) flyway table and ignore the baseline migration. In new environments it will run the baseline migration too.
Am I missing anything?
(Of course a separate problem is how to generate the "compressed" migration. If you don't need any seed data you can just do a schema-only backup of your database and use that. If your migrations populate data too you will probably have to work that out manually.)
I think this article answers your question best:
https://medium.com/att-israel/flyway-squashing-migrations-2993d75dae96
For postgres a reusable script has been created that you could execute every so many months for instance. You can of course adapt the script to MySQL specific things instead of postgres:
https://github.com/the-serious-programmer/flyway-postgres-squash-script
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/