I'm investigating how Flyaway works to see if it's suitable for our upgrade and migration needs.
Can somebody tell me if Flyaway automatically handles precedence so that for example, if a table has a trigger, it creates the table before attempting to create the trigger, or is the user responsible to manually creating and ordering the actual SQL Statements required in each schema update?
The latter - Flyway does not keep knowledge of the dependencies between database objects (as it's a very large and gnarly problem, and it's highly db-dependent). Which database do you use? A tool like Redgate Schema Compare will generate the script with all the correct dependency orderings for you.
Related
We have not used Flyway from the beginning of our project. We are at an advanced state of development. An expert review has suggested to use Flyway in our project.
The problem is that we have moved part of our services (microservices) into another testing environment as well.
What is the best way to properly implement Flyway? The requirements are:
In Development environment, no need to alter the schema which is already existing. But all new scripts should be done using Flyway.
In Testing environment, no need to alter the schema which is already existing. But what is not available in testing environment should be created automatically using Flyway when we do migrate project from Dev to test.
When we do migration to a totally new envrionment (UAT, Production etc) the entire schema should be created automatically using Flyway.
From the documentation, what I understood is:
Take a backup of the development schema (both DDL and DML) as SQL script files, give a file name like V1_0_1__initial.sql.
Clean the development database using "flyway clean".
Baseline the Development database "flyway baseline -baselineversion=1.0.0"
Now, execute "flyway migrate" which will apply the SQL script file V1_0_1__initial.sql.
Any new scripts should be written with higher version numbers (like V2_0_1__account_table.sql)
Is this the correct way or is there any better way to do this?
The problem is that I have a test database where we have different set of data (Data in Dev and test are different and I would like to keep the data as it is in both the environments). If so, is it good to separate the DDL and DML in different script files when we take it from the Dev environment and apply them separately in each environment? The DML can be added manually as required; but bit confused if I am doing the right thing.
Thanks in advance.
So, there are actually two questions here. Data management and Flyway management.
In terms of data management, yes, that should be a separate thing. Data grows and grows. Trying to manage data, beyond simple lookup tables, from source control quickly becomes very problematic. Not to mention that you want different data in different environments. This also makes automating deployments much more difficult (branching would be your friend if you insist on going this route, one branch for each data set, then deploy appropriately).
You can implement Flyway on an existing project, yes. The key is establishing the baseline. You don't have to do all the steps you outlined above. Let's say you have an existing database. You have to get the script that defines that database. That single script should include all appropriate DDL (and, if you want, DML). Name it following the Flyway standards. Something like V1.0__Baseline.sql.
With that in place, all you must do is run:
flyway baseline
That will establish your existing code base as the start point. From there, you just have to create scripts following the naming standard: V1.1xxx V2.0xxx V53000.1xxx. And run
flyway migrate
To deploy appropriate changes.
The only caveat to this is that, as the documentation states, you must ensure that all your databases match this V1.0 that you're creating and marking as the baseline. Any deviation will cause errors as you introduce new changes and migrate them into place. As long as you've got matching baseline points, you should be able to proceed with different data in different environments with no issues.
This is my how-to instruction on integration flyway with prod DB: https://delicious-snipe-938.notion.site/How-to-integrate-Flyway-with-existing-MySQL-DB-in-Prod-PostgreSQL-is-similar-1eabafa8a0e844e88205c2f32513bbbe.
I have delivered a Product to the customer. Now I have upgraded the Product, which includes changes to the database.
Customer wants to upgrade the Product. Now will Flyway help in the migration of Customer data from older version to newer version. Please let me know, if this is a valid use case. The flyway documentation talks about its use during development only.
Flyway allows you to change your database by running a set of scripts in a defined order. These scripts are called 'migrations' as they allow you to 'migrate' your database from one version to another.
The idea is you can start with an an empty database and each migration script will successively bring that database up from empty up to the current version. However, it's also possible to start with an existing database by creating a 'baseline' migration.
As SudhirR said, Flyway's primary use case is to define schema changes. However, it's perfectly possible to change data also. Since Flyway is just running plain SQL, in principle almost anything you can do in a SQL script you can also do in a Flyway migration.
In the case you described it should be possible to use Flyway to migrate the customer database. The steps you could take are:
Generate a sql script that includes the entire DDL (including indexes, triggers, procedures, ...) of the production database. To do this you will need to add insert statements for all the reference data present in the database.
Save this script in your Flyway project as something like 'V1__base_version.sql'
Run the flyway baseline command against your production database
This will set up your production database for use with Flyway
Add a new migration script to migrate your customer's data to the new version
e.g. create new table, copy data from old table to new table, delete old table
Run flyway migrate to upgrade production
These steps are adapted from the Flyway documentation page here.
Of course you should read the Flyway docs and manually test on a throwaway DB before you run anything against production. However I think in principle Flyway could be a good fit for your use case.
Flyway should be used for schema migrations and any reference data (basic data that is required by the system/application in order to function properly).
Putting client specific data migrations would not be a use case. However, if you can represent the data migration "generically" by not using IDs and instead use names or types than it could be a candidate. Meaning if you could write a migration in a way that could be applied to all clients, then that would be the use case to put it in as a flyway migration.
Otherwise data migrations would be applied in some other way outside of the process like requesting special access to the database or having some team that manages the database to apply the scripts.
If you are doing custom data modifications quite often then I'd say something is wrong in some other area of the SDLC and you may need to increase testing so that bugs don't mess up the data in the first place.
I'm evaluating Flyway and want to know if it can check for the presence of any externally made changes? I.e. if someone makes a change directly to the database, outside of Flyway, can I catch that?
I tried validate and info but it doesn't seem to notice.
No, it can't. Flyway expects you to make all changes to what you want it to manage (structure, reference data, ...) via Flyway.
Here at Redgate we support the concept of a schema snapshot. If you use either SQL Server or Oracle (as these are the database we have best support for) then you can take a snapshot after deployment and later compare this artifact with target database the next time you deploy to make sure that it hasn't "drifted". Our tools can also output a candidate Flyway migration script that will ensure Flyway is consistent with the actual target. If you're interested in this approach I can send you sample scripts of how this is achieved. But remember, this only works with Oracle and SQL Server.
I'm using SQLite for a few small projects and I've run into an issue today that is easily solved using other SQL databases but apparently it's a major stumbling block here.
Typically, we manage schema updates using a separate file for each update...
setup.001.sql
setup.002.sql
...
setup.011.sql
etc.
Through the use of various if statements, we can check if certain schema updates need to be performed within the SQL scripts themselves such that it's simply a matter of executing each script in order to bring any version of the database to the current version.
So I've found a couple of issues with this in SQLite:
There does not appear to be an if statement
There does not appear to be a clean way to retrieve PRAGMA user_version into a local variable for checking
How then, does one execute updates dependent on this information internally within a SQL Script? I do not want to have to code a separate update script in another language just to be able to run these scripts conditionally. This seems like a pretty basic need for any database provider.
SQLite is an embedded database; it is designed to be used together with a 'real' programming language. You have to put the logic into your own application.
The output of PRAGMA user_version can be read like the output of any other query.
I am developing a small registration application for a friend zumba class, using Flask, SQLAlchemy and Flask-migrate(alembic) to deal with db update. I settled on SQlite because the application has to be self contained and runs locally on a laptop without internet access and SQLite requires no installation of a service or other, which is a must too.
Dealing with SQLite lack of support of ALTER table wasn't a problem during the initial development as I simply destroyed, recreated the DB when that problem arised. But now that my friend is actually using the application I am facing a problem.
Following a feature request a table has to be modified and once again I get the dreaded " "No support for ALTER of constraints in SQLite dialect". I foresee that this problem will probably arise in the future too.
How can I deal with this problem? I am pretty much a newbie when it comes to dealing with database. I read that a way to deal with that is to create a new table, create the new constraint and copy the data and rename the table, but I have no idea how to implement that in the alembic script.
You can set a variable (render_as_batch=True) in the env.py file created with the initial migation.
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=True
)
It requires alembic > 0.7.0
This enables generation of batch operation migrations, i.e. creates a new table with the constraint, copies the existing data over, and removes the old table. See http://alembic.zzzcomputing.com/en/latest/batch.html#batch-mode-with-autogenerate
If you still encounter issues, be advised - there is still nuance with sqlite, e.g. http://alembic.zzzcomputing.com/en/latest/batch.html#dropping-unnamed-or-named-foreign-key-constraints