The scenario is this. I have a SQL Server database online that I am demoing an application. During development, I have added extra fields, modified field types, changed keys and added some new tables locally.
What's the best way for me to update the online database with the new structure and not lose the data? The database is a SQL Server 2005 one.
Download a trial of Red Gate SQL Compare, compare your two servers and you are done. If you do this often, it is well worth the $400, or get one of their bundles for a better bang for the buck.
And I do not work for Red Gate, just a happy customer!
Write update scripts to modify your live database structure to the new structure, as well as inserting any data which is required.
You may find it necessary to use temporary tables to do this.
It's probably best if you test this process on a test environment, before running the scripts on the live environment.
Depending on what exactly you've done you may be able to get away with alter statements, though from the sounds of it (removing keys and whatnot) you're doing some heavy lifting that may make that a less-than-ideal solution. You should probably look into creating a maintenance plan or, better yet, a SQL Server Integration Services project in Visual Studio. You should be able to migrate the data in the existing database to a new one using those tools.
This probably isn't of huge help retrospectively, but I always script all structural DB changes to my development database and then using a version number to determine the current version of the DB I can run the required scripts on the live DB, hence bringing it back in line at the same time as the new code is uploaded.
This also works for any content changes, for instance if the change in the underlying structure has an effect on the conent stored you can also write scripts to migrate the data accordingly.
Make a copy of the existing database to copy from.
Make another copy and alter it to your new schema. save DDL for reuse.
Write queries that copy data from #1 to #2. Save the queries for reuse.
Check the results.
Repeat until done.
Related
We have not used Flyway from the beginning of our project. We are at an advanced state of development. An expert review has suggested to use Flyway in our project.
The problem is that we have moved part of our services (microservices) into another testing environment as well.
What is the best way to properly implement Flyway? The requirements are:
In Development environment, no need to alter the schema which is already existing. But all new scripts should be done using Flyway.
In Testing environment, no need to alter the schema which is already existing. But what is not available in testing environment should be created automatically using Flyway when we do migrate project from Dev to test.
When we do migration to a totally new envrionment (UAT, Production etc) the entire schema should be created automatically using Flyway.
From the documentation, what I understood is:
Take a backup of the development schema (both DDL and DML) as SQL script files, give a file name like V1_0_1__initial.sql.
Clean the development database using "flyway clean".
Baseline the Development database "flyway baseline -baselineversion=1.0.0"
Now, execute "flyway migrate" which will apply the SQL script file V1_0_1__initial.sql.
Any new scripts should be written with higher version numbers (like V2_0_1__account_table.sql)
Is this the correct way or is there any better way to do this?
The problem is that I have a test database where we have different set of data (Data in Dev and test are different and I would like to keep the data as it is in both the environments). If so, is it good to separate the DDL and DML in different script files when we take it from the Dev environment and apply them separately in each environment? The DML can be added manually as required; but bit confused if I am doing the right thing.
Thanks in advance.
So, there are actually two questions here. Data management and Flyway management.
In terms of data management, yes, that should be a separate thing. Data grows and grows. Trying to manage data, beyond simple lookup tables, from source control quickly becomes very problematic. Not to mention that you want different data in different environments. This also makes automating deployments much more difficult (branching would be your friend if you insist on going this route, one branch for each data set, then deploy appropriately).
You can implement Flyway on an existing project, yes. The key is establishing the baseline. You don't have to do all the steps you outlined above. Let's say you have an existing database. You have to get the script that defines that database. That single script should include all appropriate DDL (and, if you want, DML). Name it following the Flyway standards. Something like V1.0__Baseline.sql.
With that in place, all you must do is run:
flyway baseline
That will establish your existing code base as the start point. From there, you just have to create scripts following the naming standard: V1.1xxx V2.0xxx V53000.1xxx. And run
flyway migrate
To deploy appropriate changes.
The only caveat to this is that, as the documentation states, you must ensure that all your databases match this V1.0 that you're creating and marking as the baseline. Any deviation will cause errors as you introduce new changes and migrate them into place. As long as you've got matching baseline points, you should be able to proceed with different data in different environments with no issues.
This is my how-to instruction on integration flyway with prod DB: https://delicious-snipe-938.notion.site/How-to-integrate-Flyway-with-existing-MySQL-DB-in-Prod-PostgreSQL-is-similar-1eabafa8a0e844e88205c2f32513bbbe.
I looked at the Flyway samples and documentation and tried to understand if it is useful in my environment.
The following conceptual detail is unclear to me: How does Flyway manage the changes between database versions? It obviously does NOT compare database life-instances (see answer here:Can Flyway find out and generate migration files from datamodel?)
In detail my setup looks like this:
I create SQL create and insert scripts when coding (automatically and manually). This means every version of my database is represented by a number of insert/create statements.
In my world I execute these scripts through a database tool (sqlplus from Oracle). Each run would setup the database _from_scratch_ (!).
Can I put these very same scripts 1 to 1 inside the "migration" path of Flyway? What happens if the target database is way older than the last "migration step" I did (or flyway did not yet exist when it was installed)?
Update:
I got some input from another Flyway user:
It seems like each "migration" (version of the database) has to be hand-written SQL/Java code and contains only "updates" from the previous "migration" of database.
If this is true, I wonder how this can be used with traditional coding technics: in my world SQL statements are generated automatically and contain all database init/create statements, not just "updates" to some previous version. If my SQL code generator could do that, then I wouldn't even need a tool like Flyway :-).
Your question about "how to handle a DB that has a longer history than there are migration scripts?" You need to create a V1_ migration/sql script that matches/recreates your latest DB schema. Something that can take a blank DB to what you have today. Create/generate that sql script using your existing DB tools and then put it in flyways migration directory. (And test V1 by using flyway against a clean DB and see if you get what you expect.) http://flywaydb.org/documentation/existing.html
After that point in time, all later versions must be added in as you work. When you decide you need a new table, in your dev environment, write a new V*_.sql that modifies your schema to the way you need it.
This blog goes over this situation for a Spring/SQL application. https://blog.synyx.de/2012/10/database-migration-using-flyway-and-spring-and-existing-data/
I have an ASP.NET project. Naturally, through different releases and development branches, the db schema changes.
What are some ways to cleanly handle the schema changes in a friendly way so that I can easily switch between development branches?
I use SQL Server 2005, but general techniques probably work.
One good way to keep track of schema changes across multiple branches of a development project would be to follow a database refactoring process. Among other benefits, this sort of process incorporates the use of delta and migration scripts to apply schema changes to each environment (or branch in your case). The setup could look something like this:
main
src <-- ASP.NET project source
db <-- Database create scripts
delta <-- Database change scripts (SQL delta files)
branch
src
db <-- usually has the same contents as the copy in main branch
delta <-- only the changes necessary for this branch
Every time you need to change the database schema for a particular branch you create a SQL delta script that is used to apply the change. To make it easier I would suggest naming each script file to include create date and time to keep them in sequence. Example would be:
201102231435_addcolumn.sql
201102231447_addconstraint.sql
201103010845_anotherchange.sql
Add the delta files to source control in the branch where the schema change needs to be made. You should end up with each branch containing exactly what is necessary to change the corresponding database. Some details might need to be tweaked for your situation depending on things like your branching scheme and whether or not your database is preserved during your release process (as opposed to re-created).
Finally, to try and make these concepts simple, I would recommend a tool to help manage the process. My suggestion is to take a look at DBDeploy / DBDeploy.NET. I've been happily using it for years on all my projects.
We put our schema changes in source control in the same place the rest of the code being deployed for that version is.
Having an argument with my team. We are developing an application using SQLite and some want to add it to the repo (GIT) and some don't. Previously with RDBMS system there has been no perceived benefit of using VCS on the DB. However SQLite is a self contained file with no external dependencies so i assume, even though it is binary, that a commit of the project code + the SQLite file will give an accurate snapshot of the state of play at that point.
I also assume that a branch and merge would work as well.
Has anyone actually done this and if so does it work?
You'd get more benefit from GIT's versioning facilities if you stored a dump of the SQLite database (i.e. commands required to create it) rather than the database file itself. That way you could look at the history of the dump file and see tables or data being added etc.
Generally speaking, it's preferable to include full set of dependencies in a VCS repository. This makes your life a whole lot simpler.
If you're after versioning DB schema, check out Wizardby.
I would like to find a way to create and populate a database during asp.net setup.
So, what I'm willing to do is:
Create the database during the setup
Populate the database with some initial data (country codes or something like that)
Create the appropriate connection string in the configuration file
I'm using .NET 3.5 and Visual Studio 2005, and the Database is SQL Server 2005.
Thanks in advance.
If you are creating an installer I'm sure there is a way to do it in there, but I am not all that familiar with that.
Otherwise, what you might do is the following.
Add a application_start handler in the Global.asax, check for valid connection string, if it doesn't exist, continue to step two.
Login to the server using a default connection string
Execute the needed scripts to create the database and objects needed.
Update the web.config with the connection information
The key here is determining what the "default" connection string is. Possibly a second configuration value.
Generally, you'll need to have SQL scripts to do this. I tend to do this anyway, as it makes maintaining and versioning the database much easier in the long run.
The core idea is, upon running the setup program, you'll have a custom action to execute this script. The user executing your setup will need permissions to:
Create a database
Create tables and other database-level objects in the newly-created database
Populate data
Your scripts will take care of all of that, though. You'll have a CREATE DATABASE command, the appropriate CREATE SCHEMA, CREATE TABLE, CREATE VIEW, etc. commands, and then after the schema is built, the appropriate INSERT statements to populate the data.
I normally break this into multiple scripts, but YMMV:
Create schema script
"Common scripts" (one for the equivalent of aspnet_regsql for web projects, one with the creation of the Enterprise Library logging tables and procs)
Create stored procedure script, if necessary (to be executed after the schema's created)
Populate initial data script
For future maintenance, I create upgrade scripts where a single script typically handles the entire upgrade process.
When writing the scripts, be sure to use the appropriate safety checks (IF EXISTS, etc) before creating objects. And I tend to make mine transactional, as well.
Good luck!
Well, actually I found a tutorial on MSDN: Walkthrough: Using a Custom Action to Create a Database at Installation
I'll use that and see how it goes, thanks for your help guys, I'll let you know how it goes.
If you can use Linq to Sql then this is easy.
Just import your entire database into the Linq to Sql designer. This will create objects that describe all objects in your database, including the System.Data.Linq.DataContext derived class that encapsulate the entire database setup.
Now you can call DataContext.CreateDatabase() to create the database.
See here more information.