I am looking to use continuous integration to deploy my SSDT project to a UAT and production environments and use external tables which point to azure data lake. The following statements need to be changed depending on the environment:-
CREATE DATABASE SCOPED CREDENTIAL [ScopeCredential]
WITH IDENTITY = N'KeyDetails' ,
Secret = 'secretsuff';
CREATE EXTERNAL DATA SOURCE [DS1]
WITH (
TYPE = BLOB_STORAGE,
LOCATION = N'https://BlobDataDev.blob.core.windows.net/dir/MyProject/',
CREDENTIAL = [AzureStorageCredential]
);
I understand command variables cant be used in this context. What is the best way to ensure these change automatically between environments and work best with the CI/CD release deployment of the DACPAC.
This needs to be changed prior to the object creates as the create statements will hang if the credentials are wrong.
Thanks
The solution I came up with was to exclude these items from the deployment, using params on the sqlpackage.exe command line. Of course these need to be created initially.
As these items remain constant this method seems to work reasonably well.
Related
We have not used Flyway from the beginning of our project. We are at an advanced state of development. An expert review has suggested to use Flyway in our project.
The problem is that we have moved part of our services (microservices) into another testing environment as well.
What is the best way to properly implement Flyway? The requirements are:
In Development environment, no need to alter the schema which is already existing. But all new scripts should be done using Flyway.
In Testing environment, no need to alter the schema which is already existing. But what is not available in testing environment should be created automatically using Flyway when we do migrate project from Dev to test.
When we do migration to a totally new envrionment (UAT, Production etc) the entire schema should be created automatically using Flyway.
From the documentation, what I understood is:
Take a backup of the development schema (both DDL and DML) as SQL script files, give a file name like V1_0_1__initial.sql.
Clean the development database using "flyway clean".
Baseline the Development database "flyway baseline -baselineversion=1.0.0"
Now, execute "flyway migrate" which will apply the SQL script file V1_0_1__initial.sql.
Any new scripts should be written with higher version numbers (like V2_0_1__account_table.sql)
Is this the correct way or is there any better way to do this?
The problem is that I have a test database where we have different set of data (Data in Dev and test are different and I would like to keep the data as it is in both the environments). If so, is it good to separate the DDL and DML in different script files when we take it from the Dev environment and apply them separately in each environment? The DML can be added manually as required; but bit confused if I am doing the right thing.
Thanks in advance.
So, there are actually two questions here. Data management and Flyway management.
In terms of data management, yes, that should be a separate thing. Data grows and grows. Trying to manage data, beyond simple lookup tables, from source control quickly becomes very problematic. Not to mention that you want different data in different environments. This also makes automating deployments much more difficult (branching would be your friend if you insist on going this route, one branch for each data set, then deploy appropriately).
You can implement Flyway on an existing project, yes. The key is establishing the baseline. You don't have to do all the steps you outlined above. Let's say you have an existing database. You have to get the script that defines that database. That single script should include all appropriate DDL (and, if you want, DML). Name it following the Flyway standards. Something like V1.0__Baseline.sql.
With that in place, all you must do is run:
flyway baseline
That will establish your existing code base as the start point. From there, you just have to create scripts following the naming standard: V1.1xxx V2.0xxx V53000.1xxx. And run
flyway migrate
To deploy appropriate changes.
The only caveat to this is that, as the documentation states, you must ensure that all your databases match this V1.0 that you're creating and marking as the baseline. Any deviation will cause errors as you introduce new changes and migrate them into place. As long as you've got matching baseline points, you should be able to proceed with different data in different environments with no issues.
This is my how-to instruction on integration flyway with prod DB: https://delicious-snipe-938.notion.site/How-to-integrate-Flyway-with-existing-MySQL-DB-in-Prod-PostgreSQL-is-similar-1eabafa8a0e844e88205c2f32513bbbe.
I am developing UWP application using vs studio 2017 version 15.9.6.
I want to use Windows local SQLite database. I want to run an SQL script named mySql.txt when the user first time install the application. I dont want to run it every time when the user run the app as it contain insert statement, which will cause duplicate rows insertion. So I want to run that script only once, preferably in the installation time.
How can I do that? I am very new to UWP and .NET. Please guide me step-by-step if possible.
You can make sure the initialization/seeding is done only once for the app. For that you may utilize ApplicationDate.Current.LocalSettings.
These allow you to write simple data for your application which are bound to your app. Once the user uninstalls the app, these data will be removed as well. This fits your scenario exactly.
Suppose your database initialization code is in the method InitializeDb(). You could use the following to make sure the initialization is done only once:
if (!ApplicationData.Current.LocalSettings.Values.ContainsKey("DbInitialized"))
{
InitializeDb();
ApplicationData.Current.LocalSettings.Values["DbInitialized"] = true;
}
This code first checks if we have initialized the db previously and if not, performs the initialization and stores a flag into app settings to make sure the next time the initialization is skipped.
You can run this code during app initialization, for example in OnLaunched method, or on when the database service is first required.
This is of course the simplest implementation, so you can (and should) add some exception handling, so that if the initialization fails, it can be retried and so on. Also you may want to handle app updates and DB updates - in which case you can use ApplicationData.Current.Version which allows you to track the version of application data and can be used to keep track of DB version as well so you can perform appropriate migrations between versions.
Finally, for even better user convenience, there is also a way to perform the app update steps during updates. See this article for more info.
I have two containers in my meteor app on galaxy servers. I have some background jobs that I want to be executing only on a single container to avoid duplication.
What's the easiest way to achieve this? Is there some procId or the like that I can retrieve during runtime?
If the two servers have their own settings files, you could use a setting to nominate one of the servers as the one that does the background jobs.
There is a package called node-cron that can be used for setting up regular jobs https://www.npmjs.com/package/node-cron
Use a Meteor.startup method to look at the settings file, and if it is the designated server, it can schedule the jobs for itself.
Another technique is to work out the ID of each server, and have database entry containing the id of the nominated server.
On checking further, https://github.com/percolatestudio/meteor-synced-cron supports multiple servers, which should do what you need :)
I was thinking about this more... one solution (which I'm kind of borrowing from meteor-migrations) is to make a simple database collection/entry that holds a startupCodeHasRun flag. Then in a Meteor.startup() block you can check if the flag has been set, if not, set the flag and run the code. This would cause the code to only be run once on only one of your containers that share the same database.
The gotcha is you would have to reset this flag manually before redeploying or else the code would never be run again on redeploy.
Not an ideal solution but could work. This is the same way the above database migrations package works in a multi-container environment. And since it's a one-time database migration, you don't have to worry about redeploy.
I have been applying separate web.config files for each environoment as my ASP.NET application progresses through Development, IT, UA and Production and have been looking for a way to simplify this process.
In the past, these had been updated manually; this was tiresome and prone to human error. More recently, I've been using IBM's uDeploy to push the application with an environment-specific config file deployed with the application depending on the target environment.
I've seen many suggestions, such as separate config files (as per my current setup), use of pre-build events etc. However, I implemented a solution to this issue in our test environment whereby I assigned the database connection strings to environment variables on my application server. The relevant environment variable is then passed into my data access connection method.
In other words, each environment's application server has an environment variable with the same name but with a different value assigned. This solution is quite simple and easily implemented and appears to function correctly.
Does anyone else manage separate environment configurations in this way? Are there any disadvantages to this approach that I have failed to consider?
How can I automate the web-application build process, that includes following steps:
Change connection string.
Recreate database by scripts.
Deploy web-site by ftp.
Copy some files to server in addition to application.
And may be perform some initialize operations.
Should I write any script/programm, use Visual Studio or any another program?
Personally I use a Continuous Integration tool to do this kind of work.
The one I mainly use is Team City by JetBrains.
This kind of software can look at your Source Control repo for new checking, perform builds, publish builds to servers as well as running pre/post build events.
You've to start learning MSBuild. It is VERY simple and straightforward, so just start and you'll see ;)
In adddition to built in features it has Community Pack with many tasty things so you will be able to:
Replace connection string in config file using regex or replace whole config with predefined connection string (FileUpdate or Copy task)
Execute database scripts (MSBuild.Community.Tasks.SqlServer.ExecuteDDL)
Deploy site using Copy task
And many other...
You can run pre and post events in Visual Studio. To do this, simply right click on the project and in the project properties navigate to the 'Build Events' options. Here you can specify the pre and post build events (you can also specify when the event runs - on successful build or otherwise).
Once the project has been successfully built, the post build event can be set up to perform the tasks specified. You can detail the steps either in a separate file or in Visual Studio project's build events itself.
More information
Pre/Post Build event command line arguments
How to: Specify Build Events (C#)
Much along the continuous integration concept Jamie mentions, we use BuildMaster internally for all of our applications since we develop it :)
Now that we have a version offered for free, I'll share some thoughts on each of your bullet points:
Change connection string
This is something that is handled uniquely by the tool. Each environment would get its own "instance" of a configuration file and in a deployment plan you can use the "deploy configuration files" action to put them in any environment. This means there are no transforms to worry about since the config file is stored and versioned within the tool.
Recreate database by scripts
This is another major feature we have. Object code (stored procs, views, etc.) can be run every time with a DROP/CREATE combo, but adding indexes, dropping columns, can only be done once (you can't bring a column's data back without a restore!)
BuildMaster handles these types of change scripts differently - they can only be run at most once against an environment's instance of your database. This makes it super easy to bring any new or existing initialized database schema up-to-date.
Deploy web-site by FTP
Just add an action to your deployment plan, and you click Create Build or Promote Build, it will do that.
Copy some files to server in addition to application
If the process is repeatable you can do this easily, if need be by using a manual action that will remind you to do it.
And may be perform some initialize operations
This sounds like a "change control" to me, a one-time change when you release. We support these as well but not in the free version unfortunately.