We are using Flyway to keep up-to-date many databases in our test environments with sql scripts and it works fine.
But we have a special need to also update databases with csv files.
I know Flyway offers some Java based migrations to handle more complicated updates.
But the problem is that these Java classes have the wanted version in their names, that would oblige us to recompile the class each time we want to use it.
It would be more simple if we could drop our csv files in migration directories exactly like we do with sql files.
Then some specific Java code would handle these csv files to do the right update.
So how can we extend Flyway with this specific code that would handle our csv files ?
Thanks
There is currently no support this. Sounds like the same issue as https://github.com/flyway/flyway/issues/469
I am still not sure how to resolve this without exposing too much of Flyway's internals.
Related
I'm wondering if Flyway allows the use a master file to re-enforce the execution orders of procedures, functions, views and triggers in repeatable migrations? Currently I have an in-house tool that generates master files for these objects so that they can be installed without errors (resolves the inter-dependency between them).
Has anyone had similar experience and how did you handle this?
Thanks!
When Flyway finds multiple repeatable migrations that need to run (because they are new or have been modified), Flyway executes them in the alphabetical order of their descriptions.
This should easily allow you to achieve something similar to what your in-house tool did with its master file.
Our application has a number of tables containing reference data. We have been using the traditional Flyway approach of creating delta files for each change in data but with frequent changes its a bit hard to manage this way. It would be easier to have a script with a truncate followed by inserts to reload the table from scratch and when data changes the developer would edit this file as needed.
Is there a clean way to accomplish this in Flyway without generating checksum errors? Hopefully without creating a new version of the load script each time a change is needed.
Have you tried adding your reference data as a beforeMigrate callback script?
"Using the default settings, Flyway looks in its default locations (/sql) for the Command-line tool) for SQL files like beforeMigrate.sql, beforeEachMigrate.sql, afterEachMigrate.sql"
Is there a tool for deploying things into exist? if I've got a bundle of, say, schemas and XQuery stored procedures? Is there a way of, say, bundling those into a zip or tar file and uploading them or deploying them into eXist?
Alternatively what is the best way of storing these things in a version controlled way (in a git repo, say) and deploying them to the eXist server? Ideally, it'd be nice to be able to have a simple script in a scripting language so you can simply call "deploy.py" or whatever and it'd take everything from the repository and load it into the XML database.
The EXpath packaging system specifies a format for generating a ZIP file with XQuery procedures (and other content) and deploying it into multiple XQuery databases.
See the specification. You should be able to use the Python zipfile module to generate these if you're inclined to use Python (though personally, I do so from a makefile).
Unfortunately, the process for checking currently installed package versions to upgrade if necessary is not standardized; I have a solution for BaseX, but nothing for eXist immediately at hand. However, eXist's implementation is well-documented, and you should have little trouble working with it.
Having an argument with my team. We are developing an application using SQLite and some want to add it to the repo (GIT) and some don't. Previously with RDBMS system there has been no perceived benefit of using VCS on the DB. However SQLite is a self contained file with no external dependencies so i assume, even though it is binary, that a commit of the project code + the SQLite file will give an accurate snapshot of the state of play at that point.
I also assume that a branch and merge would work as well.
Has anyone actually done this and if so does it work?
You'd get more benefit from GIT's versioning facilities if you stored a dump of the SQLite database (i.e. commands required to create it) rather than the database file itself. That way you could look at the history of the dump file and see tables or data being added etc.
Generally speaking, it's preferable to include full set of dependencies in a VCS repository. This makes your life a whole lot simpler.
If you're after versioning DB schema, check out Wizardby.
I have a client who has set-up a testing environment in some AI language. It basically runs some predefined test cases and stores the results in as log files (comma separated txt files). My job is to identify and suggest a reporting system and I have these options in mind. either
1. Importing the logs into MSSQL and use the reporting(SSRS) it uses
2. or us import the logs to MySQL and use PHP to develop custom reporting.
I am thinking that going with option2 is better. The reason for this is, the logs are inconsistent and contain unexpected wild characters that normally DB's don't accept. So, I can write some scripts in php before loading them to the database.
Can anyone please suggest if this is your problem what will you suggest to do?
It depends how fancy you need to be. If the data is in CSV files, you could even go so simple as to load it into Excel (or their favorite spreadsheet tool), and use spreadsheet macros to analyze it.