I have a project where I need to deploy my corda nodes multiple times going from development to pre-production to production environments. Each of these environments has there own data which are stored on the node but of course this data must be manually re-entered by an admin. Delivery is around the corner and we are now trying to figure out how to essentially create some way of allowing us to do CD type situation. That is to say we would like to provide some updates to the cordapps but on a "live" situation where data is in use.
I was wondering is there anyway to do a data migration to facilitate these updates? There would unlikely be corda version updates in between these deployments but, there could be schema and state changes etc.
There is probably a situation I am missing but even a solution such as dumping out the database and then running a schema script or something and then importing into database is viable.
We are running this on Corda OS 4.4, with the H2 database.
Edit: if the answer could be given in the context of how one might go about in the case of a non-backwards compatible schema change for instance deletion of data, that would be appreciated.
Manish has already broken this down in the comments, I'd say to just take a look at what the database looks like, and make sure to understand schema migrations.
Also, be aware that you have a param you should add to your cordapps for schema migrations
nodeDefaults {
projectCordapp {
deploy = false
}
cordapp project(':contracts')
cordapp project(':workflows')
runSchemaMigration = true
rpcUsers = [[user: "user1", "password": "test", "permissions": ["ALL"]]]
}
source: https://github.com/corda/samples-java/blob/master/Features/postgres-cordapp/build.gradle#L99-L109
links on schema migrations:
https://docs.corda.net/docs/corda-enterprise/4.6/node-database-migration-logging.html
https://www.corda.net/blog/upgrade-to-corda-os-46the-database-migration/
links on database tables:
https://docs.corda.net/docs/corda-enterprise/4.6/node/operating/node-database-tables.html#database-tables
Related
Is it possible to use a different migration framework for your relational database with hasura?
I am seeing hasura has the ability to manage migrations as noted in the documentation here.
We are using liquibase as the migration framework for all of our other projects and want to use hasura, but keep our existing migration framework (liquibase).
In the setup documentation already linked above, there's a prompt that asks if you want to initialize the project with metadata and migrations. Is it as simple as saying no here?:
? Initialize project with metadata & migrations from https://docs-demo.hasura.app ? Yes
Can this be done or do you have to use the hasura migrations if you want to use hasura?
Yes, you can manage your database migrations however you want to, and you have no obligation to use Hasura. Hasura's migrations are just a collection of .sql files that can be applied/revoked sequentially.
What is critically important is that you keep Hasura's metadata in sync with the database state.
For example, if you're tracking a database column in Hasura, and you use a SQL client to drop that column in your DB, Hasura's metadata (which describes the tables, columns, etc. that are exposed through the API) will be inconsistent with the database state. The proper way to manage a task like that is to either (1) use the Hasura console UI, (2) use the Hasura metadata HTTP API, or (3) manually edit and apply metadata with the Hasura CLI.
The task of keeping Hasura metadata in sync with DB state becomes non-trivial very quickly as you start to make use of features like "actions" and "events". You should run through some real-life migration scenarios with your current set up to get a sense of the challenges.
I have delivered a Product to the customer. Now I have upgraded the Product, which includes changes to the database.
Customer wants to upgrade the Product. Now will Flyway help in the migration of Customer data from older version to newer version. Please let me know, if this is a valid use case. The flyway documentation talks about its use during development only.
Flyway allows you to change your database by running a set of scripts in a defined order. These scripts are called 'migrations' as they allow you to 'migrate' your database from one version to another.
The idea is you can start with an an empty database and each migration script will successively bring that database up from empty up to the current version. However, it's also possible to start with an existing database by creating a 'baseline' migration.
As SudhirR said, Flyway's primary use case is to define schema changes. However, it's perfectly possible to change data also. Since Flyway is just running plain SQL, in principle almost anything you can do in a SQL script you can also do in a Flyway migration.
In the case you described it should be possible to use Flyway to migrate the customer database. The steps you could take are:
Generate a sql script that includes the entire DDL (including indexes, triggers, procedures, ...) of the production database. To do this you will need to add insert statements for all the reference data present in the database.
Save this script in your Flyway project as something like 'V1__base_version.sql'
Run the flyway baseline command against your production database
This will set up your production database for use with Flyway
Add a new migration script to migrate your customer's data to the new version
e.g. create new table, copy data from old table to new table, delete old table
Run flyway migrate to upgrade production
These steps are adapted from the Flyway documentation page here.
Of course you should read the Flyway docs and manually test on a throwaway DB before you run anything against production. However I think in principle Flyway could be a good fit for your use case.
Flyway should be used for schema migrations and any reference data (basic data that is required by the system/application in order to function properly).
Putting client specific data migrations would not be a use case. However, if you can represent the data migration "generically" by not using IDs and instead use names or types than it could be a candidate. Meaning if you could write a migration in a way that could be applied to all clients, then that would be the use case to put it in as a flyway migration.
Otherwise data migrations would be applied in some other way outside of the process like requesting special access to the database or having some team that manages the database to apply the scripts.
If you are doing custom data modifications quite often then I'd say something is wrong in some other area of the SDLC and you may need to increase testing so that bugs don't mess up the data in the first place.
Is it possible lock the database from any other connections when running the migrations through Database.Migrate()?
We have multiple service instances running the same code (on AWS Lambda), and do the migrations on startup. Now we have to manually make sure that only one instance is running when we want to apply some migrations, otherwise they can both try to do it and break things bad.
Is there a database level solution to this?
ef-core 2.1
Not really sure if this is what you are looking for, but if you are willing to add plain SQL to your migration you could set database to single user mode: Read more
Can I replicate a schema using flyway from one environment to another.
Is it possible by one by one table or whole schema to replicate from Dev to Prod?
You could certainly share a set of migrations to be applied across multiple databases.
For example, you could have a structure:
db/migration/
--V2_base_schema.sql
--V3_base_data.sql
--V4_change_table.sql
--R__function.sql
as a resource bundle and provide the applicable runtime parameters to each environment in order to have the same migrations carried out on each database. Each database maintaining it's own schema_version, of course.
If you are asking if Flyway is the tool to somehow dump and restore, there is no such functionality - look to your databases native tools for that (eg pg_dump / pg_restore for PostgreSQL).
IMHO using maven for migrations is fine only for development machines.
On servers you don't usually have maven available (and it might be impossible to have it installed there).
So: How do I init a database without maven?
Do I just call flyway.init()?
What if the db is already init-ed?
Can I execute sql statements to init the db?
My foreign keys and indexes are different/messed up in between different databases instances, so I already made a complete schema script and tested it with data export, schema drop, schema re-create and data restore. I am going to do that on all databases to ensure that they are exactly the same.
Yes, you can simply call flyway.init()
You can use flyway.status() to check if the DB has been inited.
This process will become easier with Flyway 1.8, where a new property called initOnMigrate has been introduced. The first time it runs, it will then init an existing non-empty database (PROD) when you run migrate or just migrate on an empty one (DEV).