Flyway version (3.0 -> 4.2.0) updated checksums but marks migrations as Future state - flyway

I am currently working to get Flyway 3.0 version up to 4.2.0 and am having some trouble going from 3.0 -> 4.2.0. So far I've pulled in the 4.2.0 dependency and updated the renamed config fields.
When looking through the Flyway migration docs & issues it seemed that when a user runs flyway on version 4.2.0 that the checksums update automatically. I assumed that meant flyway migrate or any relevant commands would trigger this update, via the documentation:
Flyway 4.0 now ignores line endings when calculating checksums and all existing checksums in your metadata table will be automatically adjusted on first run to be compatible with this new algorithm.
However, the checksums did not update when running flyway migrate. As a work around, I found the helpful repair command which amends the migration checksum values from the old version to the new version and ran that:
flyway repair -user=user -password=password -url=jdbc:mysql://127.0.0.1:3306/db -table=db_migrations
Flyway 4.2.0 by Boxfuse
Database: jdbc:mysql://127.0.0.1:3306/db_migrations (MySQL 5.6)
Repair of failed migration in metadata table `db_migrations`.`schema_version` not necessary. No failed migration detected.
Successfully repaired metadata table `db_migrations`.`schema_version` (execution time 00:00.005s).
Manual cleanup of the remaining effects the failed migration may still be required.
Database: jdbc:mysql://127.0.0.1:3306/db_migrations (MySQL 5.6)
Repair of failed migration in metadata table `db_migrations`.`schema_version` not necessary. No failed migration detected.
Successfully repaired metadata table `db_migrations`.`schema_version` (execution time 00:00.004s).
Manual cleanup of the remaining effects the failed migration may still be required.
[1]- Done flyway repair -user=user -password=password -url=jdbc:mysql://127.0.0.1:3306/db -table=db_migrations
[2]+ Done flyway repair -user=user -password=password -url=jdbc:mysql://127.0.0.1:3306/db -table=db_migrations
The checksums updated but when I ran flyway info it shows that all the previous migrations are now marked future. Looking through the documentation future is defined:
These are migrations that were performed by a newer deployment of the application that are not yet available in this version. For example: we have migrations available on the classpath up to version 3.0. The schema history table indicates that a migration to version 4.0 (unknown to us) has already been applied.
However when checking the version I get the correct one:
flyway -v
Flyway 4.2.0 by Boxfuse
TLDR: Ran the repair command to update migration checksums and now flyway tells me the migrations have a checksum from a future version. Checked command line flyway and application flyway version and they are both at 4.2.0. Unsure how to debug / proceed. Any help would be appreciated.
Edit: Found one of the maintainers comments on another question defining future further
future -> executed, no longer found in configured locations, newer than the newest found script
So possibly the cmd line flyway can't locate the migration scripts it sees in the db?

Related

How to upgrade airflow?

There seems to be no proper documentation about upgrading airflow. The Upgrading Airflow to a newer version page only talks about upgrading the database. So what is the proper way of upgrading airflow?
Is it just upgrading the python packages to the newest versions? Or should I use the same venv and install the newer airflow version completely from scratch? Or is it something else altogether?
I'm guessing doing the database upgrade would be the final step followed by one of these steps.
I was also struggling with upgrading airflow for minor versions and didn't feel like I found a good answer in the docs. I think I have the right approach after looking back at how I installed airflow in the first place.
If you followed the guide to run airflow locally you'll want to change the value for AIRFLOW_VERSION in the commands to your desired version.
If you followed the guide to run airflow on docker, then you'll want to fetch the latest docker-compose.yaml. The command on the site always has the latest version. Then re-run docker compose up.
You can confirm you have the right version by running airflow version. I run airflow via docker so the docker steps work for me, I imagine the local steps should be about the same.
Adding to Vivian's answer -
I had installed airflow from PyPi and was upgrading from 2.2.4 to 2.3.0.
To upgrade airflow,
I installed the new version of airflow in the same virtual environment as 2.2.4 (using this).
Upgraded the database using airflow db upgrade. More details here.
You might have to manually upgrade providers using pip install packagename -U
After this, when I started Airflow, I got an error related to some missing conf. Airflow wanted the newest version of airflow.cfg, but I had the older version. To fix this,
Renamed my airflow.cfg to airflowbackup.cfg. This is done so that airflow will make a new airflow.cfg on start up when it sees that there is no config file.
Compared airflowbackup.cfg with the 2.2.4 config to find out all the fields I had changed.
Manually made those same changes in the newly made airflow.cfg

Airflow upgrade 1.10.14 failed and cannot rollback to previous version which has the 1.10.10

I upgraded docker image to use airflow 1.10.14. Airflow is deployed with helm and I have an init-container which execute script to initialize airflow. The init script contain commands
...
airflow upgradedb
alembic upgrade heads
...
The upgrade failed so I need to rollback to previous deployed release version which contains the 1.10.10 version of airflow but it is now getting the alembic error. I tried to delete the row/record in the alembic_version table based on my search.
The error in scheduler container is this:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.DuplicateColumn) column "operator" of relation "task_instance" already exists
All the other pods are running fine (webserver and workers).
Any resolution/workaround to this issue?
Unless you are ok with scrapping your entire metadata DB (connections, variables, task runs, etc) I might opt to just push to 1.10.15 and see if the bug you encountered is resolved there. From my best understanding it is not possible to downgrade the DB after the upgrade has been done.
Suggesting upgrade to 1.10.15 based on if you remember encountering an issue similar to this user here. The CLI fix can be found here. If you found another issue with your 1.10.14 upgrade besides the one I noted for the CLI, it might be worth investigating a path to resolving that instead.

Force Swift Package Manager to honour Package.resolved

I am trying to use Swift Package manager to update my dependencies locally to match those in the Package.resolved file.
For example, when I update my dependencies I run swift package update, which updates the Package.resolved file, which I then commit. When I pull this new commit on to a computer that has an existing .build directory with the old versions of the dependencies I would like to update the local versions to the one in Package.resolved.
I was expecting swift package resolve to do this, but instead it updates the Package.resolved according to the .build/dependencies-state.json.
The obvious solution is to delete the .build directory, but this means the dependencies are recompiled every time, which can be slow on CI.
I want the equivalent of bundle install, yarn install, or carthage boostrap. Is this possible?
This has been confirmed to be a bug, and I have opened a bug in the Swift JIRA.

Safe to run Airflow upgradedb command on same Airflow version?

Airflow has an upgradedb command that needs to be run when upgrading Airflow versions. I wonder if it's safe to run even if the version is the same
The way it works is in db.py they use the alembic command module to check the checked in files in the migrations directory https://github.com/apache/incubator-airflow/tree/master/airflow/migrations/versions, and only make the changes if the commit version differs. But these files only get changed/added once we change the version so the upgrade db step does nothing when it's the same version/whl.
Adding it as a default step since I've verified it's safe to do so.

Flyway DataBase initialization or update

I want to use flyway for an existing application.
There should be two different cases:
1) Application installation: DDL and initilization of the Database
2) Application update: migrate database
The problem is if i install Version 6.0 i have e.g.:
V1_0_1__CreateShema (for 6.0.0)
V1_0_2__InitData (for 6.0.0)
....
V4_0_0__Update_DDL_and_DML
V5_0_0__Update_DDL_and_DML
V6_0_0__Update_DDL_and_DML
If i set init version to V1.0.0 it will execute all updates until V6_0_0 but it should skip all updates after initialization of the Database.
To install an old shema so i can use all udpate scripts even for a fresh install would be very hard (no db version history so i have to do reverse engineering)
I could use a workaround:
1) Check if version table exist
2) If not - migrate to version 1.0.2
3) drop the version table
4) init again with version 6.0.0
There should be a better way isnt it?
I've highlighted the problem:
V1_0_1__CreateShema (for 6.0.0)
V1_0_2__InitData (for 6.0.0)
To get to 6.0 you should run all scripts up to V6_0_0__Update_DDL_and_DML
V1 should be for V1 and never change afterwards.

Resources