For our stored procedures, we were using an approach that was working rather well during CD which was making use of the javascript v2 SDK to call container.storedProcedures.upsert. Upsert has now been removed from the API on v3 as it's not supported on non-partitioned collections (which are the only ones you'll be able to create from now on).
I supposed that the v3 SDK would have a way to at least delete and re-create these objects, but for what I can see it only allows creation: https://learn.microsoft.com/en-us/javascript/api/%40azure/cosmos/storedprocedures?view=azure-node-latest
We followed a similar approach for maintaing the index definitions updated and this is the main reason we now need to migrate to the v3 SDK as otherwise updating some kind of indexes fail through v2.
Given that what we want (if possible) is to be able to maintain all of these objects in soure control and automatically deploy them during CD, what would be the recommended way to do this?
(Meanwhile I'm exploring using these powershell commands for it: https://github.com/PlagueHO/CosmosDB but attempting to create a UDF through them caused a very bizzarre outcome in which Azure Portal stopped showing me any UDF on the collection until I removed the one I had created using New-CosmosDbUserDefinedFunction)
There are a few options today and your choices will get better here over the next couple of months.
Cosmos now has support for creating stored procedures, triggers and UDFs using ARM Templates. The second sample on this page has an ARM template that shows this. Cosmos DB ARM Template Samples. This PS tool you are using is not officially supported so you'll need to file an issue there for any questions. We will be releasing PS Cmdlets to create stored procedures, triggers and UDF's but there is no ETA to share at this time.
Related
We have not used Flyway from the beginning of our project. We are at an advanced state of development. An expert review has suggested to use Flyway in our project.
The problem is that we have moved part of our services (microservices) into another testing environment as well.
What is the best way to properly implement Flyway? The requirements are:
In Development environment, no need to alter the schema which is already existing. But all new scripts should be done using Flyway.
In Testing environment, no need to alter the schema which is already existing. But what is not available in testing environment should be created automatically using Flyway when we do migrate project from Dev to test.
When we do migration to a totally new envrionment (UAT, Production etc) the entire schema should be created automatically using Flyway.
From the documentation, what I understood is:
Take a backup of the development schema (both DDL and DML) as SQL script files, give a file name like V1_0_1__initial.sql.
Clean the development database using "flyway clean".
Baseline the Development database "flyway baseline -baselineversion=1.0.0"
Now, execute "flyway migrate" which will apply the SQL script file V1_0_1__initial.sql.
Any new scripts should be written with higher version numbers (like V2_0_1__account_table.sql)
Is this the correct way or is there any better way to do this?
The problem is that I have a test database where we have different set of data (Data in Dev and test are different and I would like to keep the data as it is in both the environments). If so, is it good to separate the DDL and DML in different script files when we take it from the Dev environment and apply them separately in each environment? The DML can be added manually as required; but bit confused if I am doing the right thing.
Thanks in advance.
So, there are actually two questions here. Data management and Flyway management.
In terms of data management, yes, that should be a separate thing. Data grows and grows. Trying to manage data, beyond simple lookup tables, from source control quickly becomes very problematic. Not to mention that you want different data in different environments. This also makes automating deployments much more difficult (branching would be your friend if you insist on going this route, one branch for each data set, then deploy appropriately).
You can implement Flyway on an existing project, yes. The key is establishing the baseline. You don't have to do all the steps you outlined above. Let's say you have an existing database. You have to get the script that defines that database. That single script should include all appropriate DDL (and, if you want, DML). Name it following the Flyway standards. Something like V1.0__Baseline.sql.
With that in place, all you must do is run:
flyway baseline
That will establish your existing code base as the start point. From there, you just have to create scripts following the naming standard: V1.1xxx V2.0xxx V53000.1xxx. And run
flyway migrate
To deploy appropriate changes.
The only caveat to this is that, as the documentation states, you must ensure that all your databases match this V1.0 that you're creating and marking as the baseline. Any deviation will cause errors as you introduce new changes and migrate them into place. As long as you've got matching baseline points, you should be able to proceed with different data in different environments with no issues.
This is my how-to instruction on integration flyway with prod DB: https://delicious-snipe-938.notion.site/How-to-integrate-Flyway-with-existing-MySQL-DB-in-Prod-PostgreSQL-is-similar-1eabafa8a0e844e88205c2f32513bbbe.
I'm investigating how Flyaway works to see if it's suitable for our upgrade and migration needs.
Can somebody tell me if Flyaway automatically handles precedence so that for example, if a table has a trigger, it creates the table before attempting to create the trigger, or is the user responsible to manually creating and ordering the actual SQL Statements required in each schema update?
The latter - Flyway does not keep knowledge of the dependencies between database objects (as it's a very large and gnarly problem, and it's highly db-dependent). Which database do you use? A tool like Redgate Schema Compare will generate the script with all the correct dependency orderings for you.
Does FlywayDB support the use-case where a script can be re-run multiple times using different parameter sets through "placeholder" and be treated either as separate versions or repeatable migration (though with different SQL files)? I have a requirement where we'd want to run the same set of scripts to organize data according to "regions" (US, UK, CA, etc.)
e.g...
Files:
sql/V1__customer_info.sql
sql/V2__customer_address.sql
Commands:
# Migrate US customers
mvn -Dflyway.placeholders.region_id=us flyway:migrate
# Migrate UK customers
mvn -Dflyway.placeholders.region_id=uk flyway:migrate
# Migrate Australian customers
mvn -Dflyway.placeholders.region_id=au flyway:migrate
No is the short answer. I have had a couple of ideas you might like to explore further:
Implement it in a callback specifically afterMigrate.sql and then call as per your example. afterMigrate is called even if there are no pending migrations to apply. This is "extending" the callback feature and you would be constrained by a single sql file so would need to combine info and address into a single file. Java callbacks are more flexible however I have not used them.
Pass a list into the placeholder and have your database split and loop over it. This would be achievable with Oracle and PLSQL but may be tricky with other databases or if you need to support multiple database types.
I'm using SQLite for a few small projects and I've run into an issue today that is easily solved using other SQL databases but apparently it's a major stumbling block here.
Typically, we manage schema updates using a separate file for each update...
setup.001.sql
setup.002.sql
...
setup.011.sql
etc.
Through the use of various if statements, we can check if certain schema updates need to be performed within the SQL scripts themselves such that it's simply a matter of executing each script in order to bring any version of the database to the current version.
So I've found a couple of issues with this in SQLite:
There does not appear to be an if statement
There does not appear to be a clean way to retrieve PRAGMA user_version into a local variable for checking
How then, does one execute updates dependent on this information internally within a SQL Script? I do not want to have to code a separate update script in another language just to be able to run these scripts conditionally. This seems like a pretty basic need for any database provider.
SQLite is an embedded database; it is designed to be used together with a 'real' programming language. You have to put the logic into your own application.
The output of PRAGMA user_version can be read like the output of any other query.
I think the documentation (http://flywaydb.org/getstarted/existingDatabaseSetup.html) is not clear enough and would like the description to be illustrated with an example. I have one for you:
Let's say we have two different versions (1 and 2) of the production database whose schema version is implicit but deterministic by querying the existing tables. How will we then achieve what is described in the documentation?
In my example the two versions both have a script attached:
Version 1: Create table A
Version 2: Create table B
I have created java migration files matching the scripts for versions 1 and 2 but since the flyway metadata is missing I need to query the database whether the scripts have been run and skip them in that case. The problem is that the application crashes since Flyway has not been initialised.
I don't want to initialise Flyway from the commandline since I want this to be done automatically upon deployment (Flyway in embedded mode). From what I've seen this only works with empty databases.
Is there a simple solution to this problem?
For single PROD databases, you can use flyway.initOnMigrate
In your case you would have to wrap Flyway and recreate this manually by inspecting your tables and calling either init with flyway.initialVersion=1 or with flyway.initialVersion=2, followed by a call to migrate.