The normal way of dealing with Doctrine Migrations is via the standard Commands - during development one runs the commands manually to e.g. run diffs and apply the migrations, and deployment typically involves applying them the by the same approach but automatically. Occasionally when working in a team on a local instance there are new migrations, but I've updated my source from version control rather than done a deployment, so I need to apply the new migrations manually, and I need to know that I need to do that! An improvement could be to display a warning on a rendered webpage that migrations are out of sync and action needs to be taken.
Is there a way to access the Migrations API directly in PHP/Symfony code, so that I could detect a mismatch between committed and applied migrations? I haven't found any documentation about that. I've had an initial poke around the code and it seems heavily skewed towards Commands (reasonably enough).
Firstly, updating your source code from version control, is a deployment too, and applying Doctrine Migrations should be part of that. You should create a check list of all the steps you need to do during a deployment, including rollbacks. Depending on the complexity of the application, many things could go wrong.
To answer you question, you can execute, in your code, a diff migration with the Process component and parse the output to determine if there're migrations to be applied.
Related
We have a production system with a large DB (several hundred tables) and would like to begin using Flyway to manage DDL changes that occur through the dev cycle. However, the organization is setup in such a way that there are production DB changes that sometimes occur, mostly just data changes but possibly DDL, that will happen outside of a data migration tool. While this is obviously an organizational challenge, does this fact alone cripple a tool like Flyway? Or is there a workflow where Flyway could rebuild its indices on demand such that any out-of-band DB change like this is pulled in?
We'd love to use Flyway, but would need to integrate it incrementally until all teams using the system are trained/bought in.
When introducing Flyway to a DB with existing data you will need to baseline Flyway to integrate with your existing data. See baseline.
For changes made after this, Flyway will only track and version changes made from its own migration scripts and not changes made externally to it. However, this does not mean you cannot use the two together, but you would need to be more aware of your database structure to avoid conflicts between your flyway migrations and external changes.
Transactional data changes made to production shouldn't impact Flyway as these won't be versioned.
If you're referring to static data (eg lookup data) that you'd like Flyway to manage, then this isn't detected by Flyway (at least not today). If you discover that you have drift you'll need to add the changes as a new migration script using idempotent syntax to ensure that next time this runs against production it doesn't try to make the same changes again.
For out-of-band schema changes, The enterprise edition of Flyway has a drift check, so at least you'd be made aware of them. However, as for the data changes described above, you'll need to manually add these schema changes as an idempotent migration script.
We have not used Flyway from the beginning of our project. We are at an advanced state of development. An expert review has suggested to use Flyway in our project.
The problem is that we have moved part of our services (microservices) into another testing environment as well.
What is the best way to properly implement Flyway? The requirements are:
In Development environment, no need to alter the schema which is already existing. But all new scripts should be done using Flyway.
In Testing environment, no need to alter the schema which is already existing. But what is not available in testing environment should be created automatically using Flyway when we do migrate project from Dev to test.
When we do migration to a totally new envrionment (UAT, Production etc) the entire schema should be created automatically using Flyway.
From the documentation, what I understood is:
Take a backup of the development schema (both DDL and DML) as SQL script files, give a file name like V1_0_1__initial.sql.
Clean the development database using "flyway clean".
Baseline the Development database "flyway baseline -baselineversion=1.0.0"
Now, execute "flyway migrate" which will apply the SQL script file V1_0_1__initial.sql.
Any new scripts should be written with higher version numbers (like V2_0_1__account_table.sql)
Is this the correct way or is there any better way to do this?
The problem is that I have a test database where we have different set of data (Data in Dev and test are different and I would like to keep the data as it is in both the environments). If so, is it good to separate the DDL and DML in different script files when we take it from the Dev environment and apply them separately in each environment? The DML can be added manually as required; but bit confused if I am doing the right thing.
Thanks in advance.
So, there are actually two questions here. Data management and Flyway management.
In terms of data management, yes, that should be a separate thing. Data grows and grows. Trying to manage data, beyond simple lookup tables, from source control quickly becomes very problematic. Not to mention that you want different data in different environments. This also makes automating deployments much more difficult (branching would be your friend if you insist on going this route, one branch for each data set, then deploy appropriately).
You can implement Flyway on an existing project, yes. The key is establishing the baseline. You don't have to do all the steps you outlined above. Let's say you have an existing database. You have to get the script that defines that database. That single script should include all appropriate DDL (and, if you want, DML). Name it following the Flyway standards. Something like V1.0__Baseline.sql.
With that in place, all you must do is run:
flyway baseline
That will establish your existing code base as the start point. From there, you just have to create scripts following the naming standard: V1.1xxx V2.0xxx V53000.1xxx. And run
flyway migrate
To deploy appropriate changes.
The only caveat to this is that, as the documentation states, you must ensure that all your databases match this V1.0 that you're creating and marking as the baseline. Any deviation will cause errors as you introduce new changes and migrate them into place. As long as you've got matching baseline points, you should be able to proceed with different data in different environments with no issues.
This is my how-to instruction on integration flyway with prod DB: https://delicious-snipe-938.notion.site/How-to-integrate-Flyway-with-existing-MySQL-DB-in-Prod-PostgreSQL-is-similar-1eabafa8a0e844e88205c2f32513bbbe.
I am currently using the Flyway Command Line tool to manage our scripts which will be invoked via our release process triggered from our CI Build server.
The issue is I have 274 repeatable migrations covering package specs, package bodies, functions, procedures, views and materialised views.
When I run migrate everything works as expected with migrations executing followed by any changed repeatable migrations but lets say in the next release we want to delete a object which one of the repeatable migrations maintains. For example, we want to remove the repeatable script that defined ProcedureOne (ie R__ProcedureOne.sql).
To do this I would have a new migration script (V3.1.5.1.01__DropProcedureOne.sql) but I would also remove the repeatable migration script so the object isn't created again and maintained.
However, executing flyway info shows the R__ProcedureOne.sql script with the status of MISSING.
While I agree it is missing it is a deliberate action to have it deleted as it is no longer required opposed to being misplaced.
I am aware of the migrate option ignoreMissingMigrations but i think using this introduces risk and could mask genuine missing files.
What is the general guidance on how best to remove a repeatable scripts?
I suggest that you simply keep the file but make it empty (ie zero bytes). Alternatively have a comment in the file that explains that the object it represents has been removed.
As for actually removing it, another option from what you suggested could be to update the repeatable migration to remove itself then update again to zero length. This has the advantage of being able to be replayed into an empty database; since repeatable migrations are applied after versioned the procedure in your example won't exist to be dropped. The disadvantage is running two migrations.
I use Scrum methodology and deploy functionality in builds every sprint.
There is necessity to perform different changes in the stored data (I mean data in database and on filesystem). I'd like to implement it as a PHP scripts invoked from console. But they should be executed only once, during the deployment.
Is there any way to implement it through app/console without listing it in the list of registered Console commands? Or is there any other way to implement runonce scripts?
DoctrineMigrations covers some part of my requirements, but it's hard to implement complex changes in Model. And it does not cover changes in files on the filesystem.
I don't think symfony has a facility for that, and besides, hiding the command is not the same as securing the command.
Instead, I would make the script determine if it has been run already (could be as simple as adding a version number to a file and checking that number before running) and stop if it detects it has already run before.
db-maintain has the notion of repeatable scripts, such as stored procedures/triggers. When there is a change to such a script it needs to be rolled out again. Flyway seemingly always checks for the filename convention. So to have a repeatable script we might need to rename the file. Is there a more efficient way of doing this?
Flyway started to support repeatable migrations in version 4.0. Repeatable migrations are reapplied every time their checksum changes and can be maintained as single files in version control.
It is your responsibility to ensure the same repeatable migration can
be applied multiple times. This usually involves making use of CREATE
OR REPLACE clauses in your DDL statements.
More details here.
This issue has come up in the Issue Tracker and in this other question.
There is currently no out of the box support for this.
Personally I would
package the repeatable actions in a stored procedure or trigger and add it to the DB as part of a regular migration
make sure this procedure/trigger gets invoked once per migration after that (could be as little a one-line statement)
ensure changes to the procedure/trigger necessary after that also happen as part of regular migrations
If that doesn't do it, feel free to star the Issue and comment with details about your use case.
Update: Repeatable scripts are now fully supported as of Flyway 4.0. See https://flywaydb.org/documentation/migration/repeatable