I just started looking into Flyway using the command line route, and was wondering if it is possible to reuse the .sql file?
Example:
I have a file called V1__Create_user.sql which has
CREATE USER ${user_name} WITH PASSWORD '${pass}';
GRANT readaccess TO ${user_name};
It looks like I can only use this sql file once, that is when I run below command
flyway -placeholders.userName=test_user -placeholders.pass=test migrate
When I run the above command again with different user_name and password for the placeholders, no changes was made.
So, I was wondering if there's a way to reuse that sql file instead of generating new sql files containing the same sql query over and over ?
The way Flyway works is that it runs the scripts you provide it in the order in which you provide it. It puts a marker in the database showing which scripts it has already run. Then, when you rerun it, it will only run the scripts that it has not yet: run V2__Whatever, V2.2__Something, etc..
You can't go back & modify an existing script and expect the tool to pick up that it has changed and then rerun it. Even if that script is using placeholders.
That doesn't include the repeatable scripts (stuff like view definitions) that run every time you deploy using Flyway. If you had to, you could make that a repeatable script and run it that way. BUT, that will work as long as every deployment is incremental, 1 to 2, 2 to 3, 3 to 4. As soon as you need to deploy from version 1 to 4, you can't pass multiple commands in.
Since every instance of SQL in relational data stores I'm aware of is a declarative language, the best way to deal with it, is to use it as such. Yes, that means stuff like CREATE USER is used over & over. However, since each USER created is unique, that's just how it works.
Related
I use flyway 8.5.0 and I want that my beforeMigrate or my afterMigrate sql is reported in the history table. Is this feaseable? or is there any config to setup this?
Then an other question: my repetable only runs when they change (checksum) but for my understanding the repetible sql should run every time. not so?
The beforeMigrate and afterMigrate SQL wont appear in your history table. If you look at the tutorial example for callbacks you can see that beforeMigrate can be called before the schema history table is created which would cause issues if it was trying to add itself to it. Additionally, I'm assuming these will be mostly static executions and would not really be part of the version history.
https://flywaydb.org/documentation/tutorials/callbacks
For repeatable, no they are only applied when the checksum has changed.
Repeatable migrations are very useful for managing database objects whose definition can then simply be maintained in a single file in version control. Instead of being run just once, they are (re-)applied every time their checksum changes.
https://flywaydb.org/documentation/tutorials/repeatable
I'm using SQLite for a few small projects and I've run into an issue today that is easily solved using other SQL databases but apparently it's a major stumbling block here.
Typically, we manage schema updates using a separate file for each update...
setup.001.sql
setup.002.sql
...
setup.011.sql
etc.
Through the use of various if statements, we can check if certain schema updates need to be performed within the SQL scripts themselves such that it's simply a matter of executing each script in order to bring any version of the database to the current version.
So I've found a couple of issues with this in SQLite:
There does not appear to be an if statement
There does not appear to be a clean way to retrieve PRAGMA user_version into a local variable for checking
How then, does one execute updates dependent on this information internally within a SQL Script? I do not want to have to code a separate update script in another language just to be able to run these scripts conditionally. This seems like a pretty basic need for any database provider.
SQLite is an embedded database; it is designed to be used together with a 'real' programming language. You have to put the logic into your own application.
The output of PRAGMA user_version can be read like the output of any other query.
Our application has a number of tables containing reference data. We have been using the traditional Flyway approach of creating delta files for each change in data but with frequent changes its a bit hard to manage this way. It would be easier to have a script with a truncate followed by inserts to reload the table from scratch and when data changes the developer would edit this file as needed.
Is there a clean way to accomplish this in Flyway without generating checksum errors? Hopefully without creating a new version of the load script each time a change is needed.
Have you tried adding your reference data as a beforeMigrate callback script?
"Using the default settings, Flyway looks in its default locations (/sql) for the Command-line tool) for SQL files like beforeMigrate.sql, beforeEachMigrate.sql, afterEachMigrate.sql"
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/
Our common workflow when creating a new sql migration script is
to write and execute every single statement in the developers local datatase schema. When finished, it's checked into the source control system.
Problem is: at the database scheme of the creating developer, the script is already "executed". For scripts not beeing reentrant - it would be convenient to have s.th. like Dbmaintain's maven task "markDatabaseAsUpToDatemaven".
Does Flyway have s.th. equivalent?
P.S.: Our current workflow (as a workaround) is as follows:
"mvn flyway:migrate" this file as an empty file (without content - so it never fails).
put the sql statments in, save & "migrate" again.
"mvn flyway:repair"
Thanks
While the workflow you describe sounds like it can do the job, you can achieve the same in a simpler and fully automated way: set cleanOnValidationError to true (on for dev!) and everytime the script changes, the DB gets recreated.
More info: http://flywaydb.org/documentation/maven/migrate.html#cleanOnValidationError