Currently using flyway to version database scripts. I'm at the point where I want to baseline the database version (version 10), such that when I migrate new scripts, that it doesn't scan/ validate the previous scripts going forward.
I tried the following statement:
mvn flyway:baseline -Dflyway.url=jdbc:mysql://localhost:3306/db -Dflyway.user=username -Dflyway.baselineVersion=10.0 -Dflyway.baselineDescription=First_Baseline
However when i try and run this statement, i get the following error:
[ERROR] Failed to execute goal org.flywaydb.pro:flyway-maven-plugin:5.0.7:baseline (default-cli) on project myProject: org.flywaydb.core.api.FlywayException: Unable to baseline schema history table "public"."flyway_schema_history" as it already contains migrations -> [Help 1]
Is anyone able to provide any reason why I cannot baseline my existing database going forward (even though Flyway baseline is designed to baseline existing database - looking through the documentation)?
Any advice is appreciated
Thanks.
even though Flyway baseline is designed to baseline existing database - looking through the documentation
Not certainly in that way.
Flyway baseline is designed to baseline existing database without applied migrations (when you start using Flyway in a project with an existing database).
There is an old associated issue in the Flyway's Github - https://github.com/flyway/flyway/issues/470
Related
I´m new to Airflow after running normal terminal statements as I do everyday:
cd ../../Airflow
bash Run.sh
I got this error
[2021-12-09 17:04:04,486] {db.py:815} ERROR - Automatic migration is not available
[2021-12-09 17:04:04,486] {db.py:817} ERROR - The task_fail table has 14 rows without a corresponding dag_run row. You must manually correct this problem (possibly by deleting the problem rows).
Can you guide me how to solve this? Thanks in advance
I do not see your "run.sh" script. For some reason you got a new version of Airflow and the migration was run (probably that's part of your script).
What you see is airflow telling you, that migration to a new version failed because some of the rows in the DB got corrupted (there could be many reasons for that but likely a programmatic error in earlier version of airlfow or some manual modification of the db by you or some db operators, or - for example - shutting down Airlfow in a non-clean way.
You should do what the message says - you should remove the faulty rows and for that - unfortunately - you need to connect to your database and run appropriate SQL queries to delete the rows. You need to look at the table and delete the rows. i am sure you can google the righ SQL to run.
In case you are not familiar with how airflow DB works you can see at this chapter - https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html#post-upgrade-warnings - the message there is slightly different, but the guidance on how to approach is similar. find the right table, delete the rows that are described in the message and you should be good to go.
We have not used Flyway from the beginning of our project. We are at an advanced state of development. An expert review has suggested to use Flyway in our project.
The problem is that we have moved part of our services (microservices) into another testing environment as well.
What is the best way to properly implement Flyway? The requirements are:
In Development environment, no need to alter the schema which is already existing. But all new scripts should be done using Flyway.
In Testing environment, no need to alter the schema which is already existing. But what is not available in testing environment should be created automatically using Flyway when we do migrate project from Dev to test.
When we do migration to a totally new envrionment (UAT, Production etc) the entire schema should be created automatically using Flyway.
From the documentation, what I understood is:
Take a backup of the development schema (both DDL and DML) as SQL script files, give a file name like V1_0_1__initial.sql.
Clean the development database using "flyway clean".
Baseline the Development database "flyway baseline -baselineversion=1.0.0"
Now, execute "flyway migrate" which will apply the SQL script file V1_0_1__initial.sql.
Any new scripts should be written with higher version numbers (like V2_0_1__account_table.sql)
Is this the correct way or is there any better way to do this?
The problem is that I have a test database where we have different set of data (Data in Dev and test are different and I would like to keep the data as it is in both the environments). If so, is it good to separate the DDL and DML in different script files when we take it from the Dev environment and apply them separately in each environment? The DML can be added manually as required; but bit confused if I am doing the right thing.
Thanks in advance.
So, there are actually two questions here. Data management and Flyway management.
In terms of data management, yes, that should be a separate thing. Data grows and grows. Trying to manage data, beyond simple lookup tables, from source control quickly becomes very problematic. Not to mention that you want different data in different environments. This also makes automating deployments much more difficult (branching would be your friend if you insist on going this route, one branch for each data set, then deploy appropriately).
You can implement Flyway on an existing project, yes. The key is establishing the baseline. You don't have to do all the steps you outlined above. Let's say you have an existing database. You have to get the script that defines that database. That single script should include all appropriate DDL (and, if you want, DML). Name it following the Flyway standards. Something like V1.0__Baseline.sql.
With that in place, all you must do is run:
flyway baseline
That will establish your existing code base as the start point. From there, you just have to create scripts following the naming standard: V1.1xxx V2.0xxx V53000.1xxx. And run
flyway migrate
To deploy appropriate changes.
The only caveat to this is that, as the documentation states, you must ensure that all your databases match this V1.0 that you're creating and marking as the baseline. Any deviation will cause errors as you introduce new changes and migrate them into place. As long as you've got matching baseline points, you should be able to proceed with different data in different environments with no issues.
This is my how-to instruction on integration flyway with prod DB: https://delicious-snipe-938.notion.site/How-to-integrate-Flyway-with-existing-MySQL-DB-in-Prod-PostgreSQL-is-similar-1eabafa8a0e844e88205c2f32513bbbe.
I'd like to be able to deploy to my database after a deployment failure, because in my case I am dealing with concurrent and independent deployments.
the flyway clean command helps to remove failed migrations in the migration table. Using that command helps as my future deployments will be executed but the downside is that I lose the history of failed deployments.
Is there a way to tell flyway to repair the migration table and keep the history of failed deployments - say put the success flag to '2' for example.
In my case all deployments are independents so we create unique migration numbers at deployment time - this allows us to redeploy anything without consideration of flyway's status (we re only using the automation mechanism provided with flyway and have disabled other 'migration' safeguards).
This isn't supported natively. I agree that removing the history of failed migrations would be handy to have as an option.
Use of the callbacks may give you a workaround. Specifically you could take a copy of failed migrations in the beforeRepair callback. If using SQL files make a beforeRepair.sql (or whatever you have for your migration suffix).
Can I replicate a schema using flyway from one environment to another.
Is it possible by one by one table or whole schema to replicate from Dev to Prod?
You could certainly share a set of migrations to be applied across multiple databases.
For example, you could have a structure:
db/migration/
--V2_base_schema.sql
--V3_base_data.sql
--V4_change_table.sql
--R__function.sql
as a resource bundle and provide the applicable runtime parameters to each environment in order to have the same migrations carried out on each database. Each database maintaining it's own schema_version, of course.
If you are asking if Flyway is the tool to somehow dump and restore, there is no such functionality - look to your databases native tools for that (eg pg_dump / pg_restore for PostgreSQL).
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/