I have existing database postgres which not using Flyway and i need replicate it.
How to move existing database state to new empty database?
I don't have any migration sql.
So I am expecting command like generateChangelog in Liquibase but it seems in Flyway not existing command like that.
Flyway currently only manages the scripts that you create. It doesn't create scripts for you. So, in order to take an existing database and get it into Flyway processing, you'll need to generate the scripts for that database. You can use the methods outlined here to get the scripts for your database. Then, just rename them to the Flyway standard. You'll be off and running.
Related
I have delivered a Product to the customer. Now I have upgraded the Product, which includes changes to the database.
Customer wants to upgrade the Product. Now will Flyway help in the migration of Customer data from older version to newer version. Please let me know, if this is a valid use case. The flyway documentation talks about its use during development only.
Flyway allows you to change your database by running a set of scripts in a defined order. These scripts are called 'migrations' as they allow you to 'migrate' your database from one version to another.
The idea is you can start with an an empty database and each migration script will successively bring that database up from empty up to the current version. However, it's also possible to start with an existing database by creating a 'baseline' migration.
As SudhirR said, Flyway's primary use case is to define schema changes. However, it's perfectly possible to change data also. Since Flyway is just running plain SQL, in principle almost anything you can do in a SQL script you can also do in a Flyway migration.
In the case you described it should be possible to use Flyway to migrate the customer database. The steps you could take are:
Generate a sql script that includes the entire DDL (including indexes, triggers, procedures, ...) of the production database. To do this you will need to add insert statements for all the reference data present in the database.
Save this script in your Flyway project as something like 'V1__base_version.sql'
Run the flyway baseline command against your production database
This will set up your production database for use with Flyway
Add a new migration script to migrate your customer's data to the new version
e.g. create new table, copy data from old table to new table, delete old table
Run flyway migrate to upgrade production
These steps are adapted from the Flyway documentation page here.
Of course you should read the Flyway docs and manually test on a throwaway DB before you run anything against production. However I think in principle Flyway could be a good fit for your use case.
Flyway should be used for schema migrations and any reference data (basic data that is required by the system/application in order to function properly).
Putting client specific data migrations would not be a use case. However, if you can represent the data migration "generically" by not using IDs and instead use names or types than it could be a candidate. Meaning if you could write a migration in a way that could be applied to all clients, then that would be the use case to put it in as a flyway migration.
Otherwise data migrations would be applied in some other way outside of the process like requesting special access to the database or having some team that manages the database to apply the scripts.
If you are doing custom data modifications quite often then I'd say something is wrong in some other area of the SDLC and you may need to increase testing so that bugs don't mess up the data in the first place.
So just exploring the possibility of using flyway to maintain my DDL statements against Amazon Athena using Athena provided JDBC driver, Athena supports only CREATE statements (hive DDL) and no INSERTS.
So if database metadata table is the only one that flyway creates and updates, Is there anyway I can externalize the creating, storing into a totally different database ?
Currently this is not possible in flyway, as the schema history is read from/written to the database defined by the current jdbc connection. You can see this for yourself in the JDBCTableSchemaHistory file.
If you wish to add this support, you could create a pull request on the repo, or just add an issue detailing exactly what you want the behaviour to be.
So while building a new database using our database migration scripts written in a springboot flyway project, we realized we made some mistakes.
Some old scripts need to be changed to ensure that we do not face these issues when we make a new database schema again. These issues are mostly related - an info table was not populated with entries in the project and there are scripts that refer to the data in the migration project -- this data does not exist because we never included a script to include data.
How can we correct this project - the only way I can think of is to correct scripts such that all inserts are replaced by - insert if not exists or replace create statements by create if not exists.
and then delete all entries in schema version and re-run this on all the database which are using this schema.
I cannot go back and correct my script because then the migration project will fail because of checksum issues.
You are rigth, if this project and the scripts are running in some existing projects you can not modify them because the checksum would fail.
Then the cleanest way I can think would be add a file called "DB-GENERAL-FIXES" or something like that, where you can add all SQL validations to restore the DB to a stable status. For the new implementations will be extra work first build it wrongly and then clean it, but if you are sharing the same code in production right now...is the best option
I am working with doctrine:migrations:diff in order to prepare database evolutions.
This command creates files into app/DoctrineMigrations
Thoses files contains sql commands in order to upgrade or downgrade database scructure.
I want to store those sql commands into the database itself. In fact, i have several instances of databases. If sql commands are store into files, it is a big problem.
I have read somewhere that DoctrineMigrations bundle can create a table called "migration_versions", but i do not manage to find where i have read this...
I cannot really understand what you're trying to do.
Migrations are used when your code needs altered database structure. For example, a new table or a new column. These new requirements for a table or column comes from your newly written code, so it's only natural to place the migrations also as a code in your repository.
How and when would migrations even get to your database? How would you guarantee that migration is executed before the code changes, which use that new structure?
Generally, migrations are used in this way:
You develop your code, add new features, change existing ones. Your code needs changes to database.
You generate doctrine migration class, which contains needed SQL statements for your current database to get to the needed state.
You alter the class adding any more required SQL statements. For example, UPDATE statements to migrate your data, not only the structure.
You execute migration locally.
You test your code with database changes. If you need more changes, you either add new migration, or execute migration down, delete it and regenerate it. Never change the migration class, as you'll loose what's supposed to be in the database and what's not.
You commit your migration together with code that uses it.
Then comes the deployment part:
- For each server, upload the code, clear and warm-up cache, run other installation scripts. Then run migrations. And only then switch to the new code.
This way your database is always in-sync with current code in the server that uses that database.
migration_versions database table is created automatically by doctrine migrations. It holds only the version numbers of migration classes - it's used for keeping track which migrations were already run and which was not.
This way when you run doctrine:migrations:migrate all not-yet-ran migrations are executed. This allows to migrate few commits at once, have several migrations in a commit etc.
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/