I'd like to be able to deploy to my database after a deployment failure, because in my case I am dealing with concurrent and independent deployments.
the flyway clean command helps to remove failed migrations in the migration table. Using that command helps as my future deployments will be executed but the downside is that I lose the history of failed deployments.
Is there a way to tell flyway to repair the migration table and keep the history of failed deployments - say put the success flag to '2' for example.
In my case all deployments are independents so we create unique migration numbers at deployment time - this allows us to redeploy anything without consideration of flyway's status (we re only using the automation mechanism provided with flyway and have disabled other 'migration' safeguards).
This isn't supported natively. I agree that removing the history of failed migrations would be handy to have as an option.
Use of the callbacks may give you a workaround. Specifically you could take a copy of failed migrations in the beforeRepair callback. If using SQL files make a beforeRepair.sql (or whatever you have for your migration suffix).
Related
We have a production system with a large DB (several hundred tables) and would like to begin using Flyway to manage DDL changes that occur through the dev cycle. However, the organization is setup in such a way that there are production DB changes that sometimes occur, mostly just data changes but possibly DDL, that will happen outside of a data migration tool. While this is obviously an organizational challenge, does this fact alone cripple a tool like Flyway? Or is there a workflow where Flyway could rebuild its indices on demand such that any out-of-band DB change like this is pulled in?
We'd love to use Flyway, but would need to integrate it incrementally until all teams using the system are trained/bought in.
When introducing Flyway to a DB with existing data you will need to baseline Flyway to integrate with your existing data. See baseline.
For changes made after this, Flyway will only track and version changes made from its own migration scripts and not changes made externally to it. However, this does not mean you cannot use the two together, but you would need to be more aware of your database structure to avoid conflicts between your flyway migrations and external changes.
Transactional data changes made to production shouldn't impact Flyway as these won't be versioned.
If you're referring to static data (eg lookup data) that you'd like Flyway to manage, then this isn't detected by Flyway (at least not today). If you discover that you have drift you'll need to add the changes as a new migration script using idempotent syntax to ensure that next time this runs against production it doesn't try to make the same changes again.
For out-of-band schema changes, The enterprise edition of Flyway has a drift check, so at least you'd be made aware of them. However, as for the data changes described above, you'll need to manually add these schema changes as an idempotent migration script.
I'm using EF Core Migrations in my .NET Core projects, and deploy using DevOps pipelines.
In my build pipeline, I build a migration SQL script using a Command Line task to execute a dotnet ef migrations script command (using the --idempotent option), then execute it in the release pipeline using an Azure SQL Database deployment task. All fairly standard, to judge from a simple google search (eg. here), which is how I learned of the approach in the first place.
This is my problem: To achieve certain desired results, in some of my migrations I use Custom Migration Operations (migrationBuilder.Sql("...")) to execute some hand-crafted SQL as part of a migration.
However, as the project develops, and my DB schema changes over time, older such migrations inevitably contain SQL which no longer fits the schema. One would think this isn't a problem, as any migration is meant to be applied to a database with some specific schema version only. However, as it turns out, the dotnet ef migrations script command builds a SQL script with a bunch of conditional SQL block like this:
IF NOT EXISTS(SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20190123160628_SomeMigrationId')
BEGIN
/* Generated or custom SQL here */
END;
Note: every migration is included in the script, and the IF NOT EXIST-clauses ensure that only the not-yet-applied ones are executed in the database.
However, the custom migration task SQL statements are included as-is in the script, and if they are outdated, they longer compile. Then the DB deployment task fails, as does the whole deployment.
Has anyone else had and solved this problem, or know another way to deploy migrations which doesn't have this issue? It seems to me this seriously affects the utility of the MigrationBuilder.Sql() command, which is the only way to arbitrarily "massage" data as part of a migration.
I changed a flyway script for error, and this brought my migrations to an error state (I have 5 script versions, but when I run my app it starts from the 4th, and get an error on 'reaction already exists').
I tried to use clean from flyway cli, but it didn't completely solve the problem... also, when I try pushing my branch to git, CI/CD pipeline will fail for the migration.
Since I'm in development environment, would it be a terrible idea to all delete migration scripts?
Would deleting all scripts allow me to 'start from scratch' in my development environment, or do I need to push the whole project again to avoid issues? (Project is not in production yet, I don't really need flyway migrations)
Depending on which flyway version you are using, one option is to go to the flyway_schema_history or schema_version table and just locate the row (corresponding to the failed script) and change the success column to true or 1. Then manually run the script (the correct one) against DB.
Sqitch has the ability to run Revert and Verification scripts. Does or are their plans for Flyway to support such abilities?
I'm unfamiliar with the capabilities of Sqitch, but I can briefly explain what's possible with Flyway.
Flyway does have support for revert scripts, via the Undo command. The process works by allowing you to specify an 'Undo migration' alongside a standard migration script. The Undo migration knows how to revert the changes specified in the standard migration script. Note that this is a Pro feature only.
Undo migrations do not work for Repeatable Migrations, which are used mainly for Views, procedures, and so on. These should just be amended and then migrated normally.
Flyway also allows you to verify your migrations to a certain extent with the Validate command. Validate checks that the migrations applied to the database match the migrations in your project.
From briefly looking through the Sqitch docs, it looks like their verification feature allows you to specify a script for each migration to see if it makes sense to run it. Flyway doesn't natively support such a process. However, since the migration scripts are just plain SQL, nothing stops you from manually including such verification logic in the migration scripts themselves.
As a potential alternative, you could use the Dry Run feature, which produces the script which will ultimately be used to migrate the database. You could execute the Dry Run script against a test environment to see if the deployment would succeed. Dry Run is also a Pro feature.
I hope that helps.
Thanks
Mikiel
My current project has a few Flyway migrations in place that are used to import initial data into a database. This data is convenient especially for developers to be able to quickly setup the project. Production data is imported through some batch jobs and has a newer version.
Some of these migrations are quite big (~20MB) and so everytime the application starts, Flyway takes some time to calculate the checksum of the migrations. This also is a problem for integration tests as they also take longer because of this.
I consider this approach to be a misuse of Flyway, I think migration tools should be used mainly for structural data.
I want to remove those files from our application and rather use a configuration management tool (e.g. Vagrant, Puppet, Chef) to import test data on developer environments. However, I can't just delete the migration files from the application as Flyway will complain that a migration has been recorded in the database but is not present in the application migrations.
My first thinking was to create a new migration with a high-priority version number that will
Delete the test data
Delete the migration from the schema_version table
and then remove the migration scripts. This however does not work, Flyway still complains that the removed migration script is missing.
Is there a restriction that you cannot interact with the schema_version table in migrations?
What other options do I have? If at all possible I want to do this using Flyway and not manually.
Repair is your best bet. Empty those data migrations and run the repair command to have their checksums recalculated based on the empty files.