I have a website solution that uses on SQLite database for each tenant account. Without going into much depth about why we chose this solution, we chose it due to SQLite support on distributed/offline systems.
All databases are manipulated using the same PHP file structure. I wish to update the database version iteratively for all accounts so that they are all at the same version number.
I have a script that loops over each, and can use either PHP(Yii) or the shell to execute queries.
I would like to wrap the manipulation to each database in a transaction. It appears as though DDL commands may already be auto-commit in nature.
Question: How to accomplish a SQLite DB upgrade which, if it fails, will report a failure? Using that report, I could prompt the system to re-attempt or report an error to an admin. Ideally, I would like to wrap the whole upgrade in a transaction to prevent inconsistencies, but I'm fairly certain that this is not possible.
One thought I had was to backup the database temporarily during upgrade, and delete it on success.
As CL stated in the comments, DDL is fully transactional in SQLite. So, for each database in our system, we wrap it in a transaction, and it executes atomically. We leverage the application to ensure that all databases upgrade successfully if any fail during upgrade.
Related
We have a production system with a large DB (several hundred tables) and would like to begin using Flyway to manage DDL changes that occur through the dev cycle. However, the organization is setup in such a way that there are production DB changes that sometimes occur, mostly just data changes but possibly DDL, that will happen outside of a data migration tool. While this is obviously an organizational challenge, does this fact alone cripple a tool like Flyway? Or is there a workflow where Flyway could rebuild its indices on demand such that any out-of-band DB change like this is pulled in?
We'd love to use Flyway, but would need to integrate it incrementally until all teams using the system are trained/bought in.
When introducing Flyway to a DB with existing data you will need to baseline Flyway to integrate with your existing data. See baseline.
For changes made after this, Flyway will only track and version changes made from its own migration scripts and not changes made externally to it. However, this does not mean you cannot use the two together, but you would need to be more aware of your database structure to avoid conflicts between your flyway migrations and external changes.
Transactional data changes made to production shouldn't impact Flyway as these won't be versioned.
If you're referring to static data (eg lookup data) that you'd like Flyway to manage, then this isn't detected by Flyway (at least not today). If you discover that you have drift you'll need to add the changes as a new migration script using idempotent syntax to ensure that next time this runs against production it doesn't try to make the same changes again.
For out-of-band schema changes, The enterprise edition of Flyway has a drift check, so at least you'd be made aware of them. However, as for the data changes described above, you'll need to manually add these schema changes as an idempotent migration script.
I have delivered a Product to the customer. Now I have upgraded the Product, which includes changes to the database.
Customer wants to upgrade the Product. Now will Flyway help in the migration of Customer data from older version to newer version. Please let me know, if this is a valid use case. The flyway documentation talks about its use during development only.
Flyway allows you to change your database by running a set of scripts in a defined order. These scripts are called 'migrations' as they allow you to 'migrate' your database from one version to another.
The idea is you can start with an an empty database and each migration script will successively bring that database up from empty up to the current version. However, it's also possible to start with an existing database by creating a 'baseline' migration.
As SudhirR said, Flyway's primary use case is to define schema changes. However, it's perfectly possible to change data also. Since Flyway is just running plain SQL, in principle almost anything you can do in a SQL script you can also do in a Flyway migration.
In the case you described it should be possible to use Flyway to migrate the customer database. The steps you could take are:
Generate a sql script that includes the entire DDL (including indexes, triggers, procedures, ...) of the production database. To do this you will need to add insert statements for all the reference data present in the database.
Save this script in your Flyway project as something like 'V1__base_version.sql'
Run the flyway baseline command against your production database
This will set up your production database for use with Flyway
Add a new migration script to migrate your customer's data to the new version
e.g. create new table, copy data from old table to new table, delete old table
Run flyway migrate to upgrade production
These steps are adapted from the Flyway documentation page here.
Of course you should read the Flyway docs and manually test on a throwaway DB before you run anything against production. However I think in principle Flyway could be a good fit for your use case.
Flyway should be used for schema migrations and any reference data (basic data that is required by the system/application in order to function properly).
Putting client specific data migrations would not be a use case. However, if you can represent the data migration "generically" by not using IDs and instead use names or types than it could be a candidate. Meaning if you could write a migration in a way that could be applied to all clients, then that would be the use case to put it in as a flyway migration.
Otherwise data migrations would be applied in some other way outside of the process like requesting special access to the database or having some team that manages the database to apply the scripts.
If you are doing custom data modifications quite often then I'd say something is wrong in some other area of the SDLC and you may need to increase testing so that bugs don't mess up the data in the first place.
I'm using SQLite for a few small projects and I've run into an issue today that is easily solved using other SQL databases but apparently it's a major stumbling block here.
Typically, we manage schema updates using a separate file for each update...
setup.001.sql
setup.002.sql
...
setup.011.sql
etc.
Through the use of various if statements, we can check if certain schema updates need to be performed within the SQL scripts themselves such that it's simply a matter of executing each script in order to bring any version of the database to the current version.
So I've found a couple of issues with this in SQLite:
There does not appear to be an if statement
There does not appear to be a clean way to retrieve PRAGMA user_version into a local variable for checking
How then, does one execute updates dependent on this information internally within a SQL Script? I do not want to have to code a separate update script in another language just to be able to run these scripts conditionally. This seems like a pretty basic need for any database provider.
SQLite is an embedded database; it is designed to be used together with a 'real' programming language. You have to put the logic into your own application.
The output of PRAGMA user_version can be read like the output of any other query.
I am developing a small registration application for a friend zumba class, using Flask, SQLAlchemy and Flask-migrate(alembic) to deal with db update. I settled on SQlite because the application has to be self contained and runs locally on a laptop without internet access and SQLite requires no installation of a service or other, which is a must too.
Dealing with SQLite lack of support of ALTER table wasn't a problem during the initial development as I simply destroyed, recreated the DB when that problem arised. But now that my friend is actually using the application I am facing a problem.
Following a feature request a table has to be modified and once again I get the dreaded " "No support for ALTER of constraints in SQLite dialect". I foresee that this problem will probably arise in the future too.
How can I deal with this problem? I am pretty much a newbie when it comes to dealing with database. I read that a way to deal with that is to create a new table, create the new constraint and copy the data and rename the table, but I have no idea how to implement that in the alembic script.
You can set a variable (render_as_batch=True) in the env.py file created with the initial migation.
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=True
)
It requires alembic > 0.7.0
This enables generation of batch operation migrations, i.e. creates a new table with the constraint, copies the existing data over, and removes the old table. See http://alembic.zzzcomputing.com/en/latest/batch.html#batch-mode-with-autogenerate
If you still encounter issues, be advised - there is still nuance with sqlite, e.g. http://alembic.zzzcomputing.com/en/latest/batch.html#dropping-unnamed-or-named-foreign-key-constraints
How to manage database changes while upgrading desktop applications?
We have a SQLite database for one of our desktop apps. The uninstaller keeps the database to be used by the next install. But what if the next install has an updated version? How to keep the data but upgrade the tables?
We use .NET and Linq2sql
Here is how I do this:
In my code I define the version of the database
#define DB_VERSION 2
This version number is incremented every time I make a change to the code that 'breaks' the database ( makes an incompatible change to the schema or the semantics of the db contents )
When the code creates a new database, it executes this SQL command
QueryFormat(L"PRAGMA SCHEMA_VERSION = %d;",DB_VERSION);
Note that this must be AFTER all CREATE TABLE commands have been executed, otherwise sqlite increments SCHEMA_VERSION.
When the code opens an existing database, the first thing it does is
Query(L"PRAGMA schema_version;")
The SCHEMA_VERSION that is returned from this query is compared with the DB_VERSION. If they do not match, then the database was created by a different software version. What happens next depends on the details of what you need. Typically:
database was created by a more recent software: inform user and exit
database was created by older software for which there is an upgrade: offer user option to run upgrade code, or re-initialize database
database was created by older software with no upgrade: offer user option to re-initialize database or exit.
The details of how the upgrade code works depends very much on what you need. In general open the old database AND open a new empty database. Read the old tables, convert the data as required, and write to the new database. Close the dbs. Delete the old db. Rename the new db to the standard db name. Open the new db.
If nothing else, the output of echo .dump | sqlite my_database.sqlite is designed to be extremely portable, even to non-SQLite databases.
Or, if I'm misreading your question, you may want alter table.