I´m new to Airflow after running normal terminal statements as I do everyday:
cd ../../Airflow
bash Run.sh
I got this error
[2021-12-09 17:04:04,486] {db.py:815} ERROR - Automatic migration is not available
[2021-12-09 17:04:04,486] {db.py:817} ERROR - The task_fail table has 14 rows without a corresponding dag_run row. You must manually correct this problem (possibly by deleting the problem rows).
Can you guide me how to solve this? Thanks in advance
I do not see your "run.sh" script. For some reason you got a new version of Airflow and the migration was run (probably that's part of your script).
What you see is airflow telling you, that migration to a new version failed because some of the rows in the DB got corrupted (there could be many reasons for that but likely a programmatic error in earlier version of airlfow or some manual modification of the db by you or some db operators, or - for example - shutting down Airlfow in a non-clean way.
You should do what the message says - you should remove the faulty rows and for that - unfortunately - you need to connect to your database and run appropriate SQL queries to delete the rows. You need to look at the table and delete the rows. i am sure you can google the righ SQL to run.
In case you are not familiar with how airflow DB works you can see at this chapter - https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html#post-upgrade-warnings - the message there is slightly different, but the guidance on how to approach is similar. find the right table, delete the rows that are described in the message and you should be good to go.
Related
Currently using flyway to version database scripts. I'm at the point where I want to baseline the database version (version 10), such that when I migrate new scripts, that it doesn't scan/ validate the previous scripts going forward.
I tried the following statement:
mvn flyway:baseline -Dflyway.url=jdbc:mysql://localhost:3306/db -Dflyway.user=username -Dflyway.baselineVersion=10.0 -Dflyway.baselineDescription=First_Baseline
However when i try and run this statement, i get the following error:
[ERROR] Failed to execute goal org.flywaydb.pro:flyway-maven-plugin:5.0.7:baseline (default-cli) on project myProject: org.flywaydb.core.api.FlywayException: Unable to baseline schema history table "public"."flyway_schema_history" as it already contains migrations -> [Help 1]
Is anyone able to provide any reason why I cannot baseline my existing database going forward (even though Flyway baseline is designed to baseline existing database - looking through the documentation)?
Any advice is appreciated
Thanks.
even though Flyway baseline is designed to baseline existing database - looking through the documentation
Not certainly in that way.
Flyway baseline is designed to baseline existing database without applied migrations (when you start using Flyway in a project with an existing database).
There is an old associated issue in the Flyway's Github - https://github.com/flyway/flyway/issues/470
Background
I am running a flyway migration against DB2 using the command line interface. I have done a number of tests and all works fine. I added in some commands to cause a failure.
Question
Can someone confirm if alter table, drop table or create table SHOULD be rolled back if the migration fails and the database supports DDL? When i test it it looks like an alter table add column statement was not rolled back after a failure in the same flyway script.
Ok so i checked the flyway website and found a list of supported drivers and versions where DDL is supported. It seems like the version of DB2 we are using is below this threshold. Based on that i guess the execution of migrate script is not bound in a single transaction and that is why i am seeing some changes remaining that were applied before the script failure.
I am trying flyway for first time, evaluating how it will fit into our project.
Trying to understand how a Failed Scenario will work
Naturally, what i did next was, modified the sql script, and re tried running, but got checksum error
Have three Ques here
So I guess the only way out is ... need to make a 1.2 with correct format or manually modify 'schema_version' table. Right , or am i missing something?
Wondering how will such a scenario work in case if this script is called from continuous integration tools (Jenkins or Bamboo). Manual Intervention will be needed.
Not sure if some other tool like Liquibase will behave in a different (better) manner
In that situation I think you should use "flyway repair" rather the "flyway migrate"
https://flywaydb.org/documentation/command/repair
One thing from your post that is not clear is was the script you ran a single DDL statement or a number of statements, of which one or more failed?. The reason for asking is that flyway records the results of a migration, but does not itself clean up 'script errors'. Depending on the database you are using this could be done by running the DDL statements within a transaction.
Liquibased operates with a much tighter connection to the database as it directly interacts with the DDL that can be expressed in a range of different formats. As such it has a much tighter control over the management of DDL deployment.
Upstream insists on manual rolling back of failed migration and re-applying it again. There is no "skip" command.
But you can manually fix and complete failed migration and manually change "schema_version"."success" to 1.
just use flywaytest to test my application.
I would like to know if there is a method that, after importing a migration script, I make the rollback.
If I was not clear, just ask information.
thanks
Your question is not very clear but if you would like to rollback a failed migration and retry here are the steps you need to take:
If you look in your schema_version table you should see the script which failed (state column = FAILED). You need to correct this script so it doesn't fail again. Also if some of the script completed successfully you may have to manually revert these changes as running them again may cause a failure.
You then delete this row from the table e.g. delete from schema_version where state = 'FAILED';
Then set the current version back to the script before the one that failed. You do this by setting it's CURRENT_VERSION to 1 e.g. update schema_version set current_version = 1 where version = 1.XX;
You can then attempt to run the flyway migration again.
The flywaydb not support rollback in case of problems during migration the best you can user is:
http://flywaydb.org/documentation/maven/repair.html
But you will have to ensure for yourself the integrity of the data within the database.
If the database you're working to support checkpoints, always advise that before migrating, do the checkpoint.
good luck
the flyway does not support rollback with one-line command,
there's no command like mvn flyway:rollback , there are 2 ways you can do (IMO)
follow the answer from #user528827
try to use the rails's migration :)
This is explained in the FAQ:
https://flywaydb.org/documentation/faq#downgrade
https://flywaydb.org/documentation/faq#rollback
https://flywaydb.org/documentation/faq#repair
Update: Flyway 5.0 now comes with an undo command.
OUR SYSTEM
We are trying to put migrations as .sql files under version control. Developers would write a VN__*.sql file, commit to version control and a job that runs every 5 minutes would automatically migrate to a Dev and Test database. Once the change proved to not cause problems, someone else would run a manual job to run the migration on Production.
MY PROBLEM:
I had a demo migration that created a few tables. I checked V4__DemoTables.sql into version control on my PC.
On our Linux box a job that runs every 5 minutes extracted the new file from version control, then ran the flyway.sh file. It detected the file and executed it.
But the .sql file had a typo. And we are using Neteeza which has problems with flyway automatically wrapping a migration in a BEGIN TRAN ... END TRAN. So the migration created 2 tables, then aborted before the third.
No problem I thought. I dropped the 2 tables that the .sql file did create. Checked V4__ out of version control, fixed the typo and re-submitted it.
Five minutes later the update was extracted but flyway complains that the checksum does not match. So it will NOT run the updated V4__DemoTables.sql file.
How do I get flyway to accept the updated file and update the checksum in the SCHEMA_VERSION file in case of a typo?
Reading the docs it seems like the developers suggest I should have created a new V4_1_DemoTables.sql file with the fix's. But this would have collided with the commands in the V4__ file so this seemed wrong.
So here is what the docs imply I need to do:
Leave V4__ as a 'successful' migration according to the
SCHEMA_VERSION table.
Create V4_1_ to delete the tables that were created before the typo
line in V4__.
Create V4_2_ which has the typo fix from the original file to do all
the real work.
Is this correct?
If the migration completes successfully, but some of the db objects are not quite right yet (typo in column name, ...), do as you said and push a follow-up script that fixes it (rename column, ...).
If the migration fails, and it didn't run on a DB with DDL transaction, the DB must be cleaned manually. This means:
Reverting the effects of the migration on the DB
Removing the version from the SCHEMA_VERSION table and marking the previous one as current
This second step will be automated in the future with the introduction of the flyway.repair() command.