Is it possible to use a different migration framework for your relational database with hasura?
I am seeing hasura has the ability to manage migrations as noted in the documentation here.
We are using liquibase as the migration framework for all of our other projects and want to use hasura, but keep our existing migration framework (liquibase).
In the setup documentation already linked above, there's a prompt that asks if you want to initialize the project with metadata and migrations. Is it as simple as saying no here?:
? Initialize project with metadata & migrations from https://docs-demo.hasura.app ? Yes
Can this be done or do you have to use the hasura migrations if you want to use hasura?
Yes, you can manage your database migrations however you want to, and you have no obligation to use Hasura. Hasura's migrations are just a collection of .sql files that can be applied/revoked sequentially.
What is critically important is that you keep Hasura's metadata in sync with the database state.
For example, if you're tracking a database column in Hasura, and you use a SQL client to drop that column in your DB, Hasura's metadata (which describes the tables, columns, etc. that are exposed through the API) will be inconsistent with the database state. The proper way to manage a task like that is to either (1) use the Hasura console UI, (2) use the Hasura metadata HTTP API, or (3) manually edit and apply metadata with the Hasura CLI.
The task of keeping Hasura metadata in sync with DB state becomes non-trivial very quickly as you start to make use of features like "actions" and "events". You should run through some real-life migration scenarios with your current set up to get a sense of the challenges.
Related
I have delivered a Product to the customer. Now I have upgraded the Product, which includes changes to the database.
Customer wants to upgrade the Product. Now will Flyway help in the migration of Customer data from older version to newer version. Please let me know, if this is a valid use case. The flyway documentation talks about its use during development only.
Flyway allows you to change your database by running a set of scripts in a defined order. These scripts are called 'migrations' as they allow you to 'migrate' your database from one version to another.
The idea is you can start with an an empty database and each migration script will successively bring that database up from empty up to the current version. However, it's also possible to start with an existing database by creating a 'baseline' migration.
As SudhirR said, Flyway's primary use case is to define schema changes. However, it's perfectly possible to change data also. Since Flyway is just running plain SQL, in principle almost anything you can do in a SQL script you can also do in a Flyway migration.
In the case you described it should be possible to use Flyway to migrate the customer database. The steps you could take are:
Generate a sql script that includes the entire DDL (including indexes, triggers, procedures, ...) of the production database. To do this you will need to add insert statements for all the reference data present in the database.
Save this script in your Flyway project as something like 'V1__base_version.sql'
Run the flyway baseline command against your production database
This will set up your production database for use with Flyway
Add a new migration script to migrate your customer's data to the new version
e.g. create new table, copy data from old table to new table, delete old table
Run flyway migrate to upgrade production
These steps are adapted from the Flyway documentation page here.
Of course you should read the Flyway docs and manually test on a throwaway DB before you run anything against production. However I think in principle Flyway could be a good fit for your use case.
Flyway should be used for schema migrations and any reference data (basic data that is required by the system/application in order to function properly).
Putting client specific data migrations would not be a use case. However, if you can represent the data migration "generically" by not using IDs and instead use names or types than it could be a candidate. Meaning if you could write a migration in a way that could be applied to all clients, then that would be the use case to put it in as a flyway migration.
Otherwise data migrations would be applied in some other way outside of the process like requesting special access to the database or having some team that manages the database to apply the scripts.
If you are doing custom data modifications quite often then I'd say something is wrong in some other area of the SDLC and you may need to increase testing so that bugs don't mess up the data in the first place.
Is it possible lock the database from any other connections when running the migrations through Database.Migrate()?
We have multiple service instances running the same code (on AWS Lambda), and do the migrations on startup. Now we have to manually make sure that only one instance is running when we want to apply some migrations, otherwise they can both try to do it and break things bad.
Is there a database level solution to this?
ef-core 2.1
Not really sure if this is what you are looking for, but if you are willing to add plain SQL to your migration you could set database to single user mode: Read more
Can I replicate a schema using flyway from one environment to another.
Is it possible by one by one table or whole schema to replicate from Dev to Prod?
You could certainly share a set of migrations to be applied across multiple databases.
For example, you could have a structure:
db/migration/
--V2_base_schema.sql
--V3_base_data.sql
--V4_change_table.sql
--R__function.sql
as a resource bundle and provide the applicable runtime parameters to each environment in order to have the same migrations carried out on each database. Each database maintaining it's own schema_version, of course.
If you are asking if Flyway is the tool to somehow dump and restore, there is no such functionality - look to your databases native tools for that (eg pg_dump / pg_restore for PostgreSQL).
I am working with doctrine:migrations:diff in order to prepare database evolutions.
This command creates files into app/DoctrineMigrations
Thoses files contains sql commands in order to upgrade or downgrade database scructure.
I want to store those sql commands into the database itself. In fact, i have several instances of databases. If sql commands are store into files, it is a big problem.
I have read somewhere that DoctrineMigrations bundle can create a table called "migration_versions", but i do not manage to find where i have read this...
I cannot really understand what you're trying to do.
Migrations are used when your code needs altered database structure. For example, a new table or a new column. These new requirements for a table or column comes from your newly written code, so it's only natural to place the migrations also as a code in your repository.
How and when would migrations even get to your database? How would you guarantee that migration is executed before the code changes, which use that new structure?
Generally, migrations are used in this way:
You develop your code, add new features, change existing ones. Your code needs changes to database.
You generate doctrine migration class, which contains needed SQL statements for your current database to get to the needed state.
You alter the class adding any more required SQL statements. For example, UPDATE statements to migrate your data, not only the structure.
You execute migration locally.
You test your code with database changes. If you need more changes, you either add new migration, or execute migration down, delete it and regenerate it. Never change the migration class, as you'll loose what's supposed to be in the database and what's not.
You commit your migration together with code that uses it.
Then comes the deployment part:
- For each server, upload the code, clear and warm-up cache, run other installation scripts. Then run migrations. And only then switch to the new code.
This way your database is always in-sync with current code in the server that uses that database.
migration_versions database table is created automatically by doctrine migrations. It holds only the version numbers of migration classes - it's used for keeping track which migrations were already run and which was not.
This way when you run doctrine:migrations:migrate all not-yet-ran migrations are executed. This allows to migrate few commits at once, have several migrations in a commit etc.
IMHO using maven for migrations is fine only for development machines.
On servers you don't usually have maven available (and it might be impossible to have it installed there).
So: How do I init a database without maven?
Do I just call flyway.init()?
What if the db is already init-ed?
Can I execute sql statements to init the db?
My foreign keys and indexes are different/messed up in between different databases instances, so I already made a complete schema script and tested it with data export, schema drop, schema re-create and data restore. I am going to do that on all databases to ensure that they are exactly the same.
Yes, you can simply call flyway.init()
You can use flyway.status() to check if the DB has been inited.
This process will become easier with Flyway 1.8, where a new property called initOnMigrate has been introduced. The first time it runs, it will then init an existing non-empty database (PROD) when you run migrate or just migrate on an empty one (DEV).