We are using Flyway 4 (great tool!) on Oracle.
When invalid DDL is committed, the continuous database build breaks.. and all the team gets an email.. all good so far.
But when, code that breaks one of our stored procedures is committed.. ie procedure gets created, but it fails to compile.. we still get a successful migration reported from Flyway.
During the migration we see something like :
DB: Warning: execution completed with warning (SQL State: 99999 - Error Code: 17110)
..but still the Flyway ant task reports success.
As we have a lot of stored procedures, 9 times out of 10 it is these that are broken by developers, and not the DDL. We really would like Flyway to fail on a warning also. Can anyone advise how best to approach this?
Solved! Found an acceptable solution for us and implemented it as follows, utilising Flyways callback mechanism which is documented on the Flyway website.
There are many callbacks available and are invoked at various points, but the one that appears to suit our needs is afterMigrate. In the callback, we can execute sql (on Oracle) which counts the number of invalid objects in the user schema at hand
So, implementing a java afterMigrate callback as follows does the job:
public void afterMigrate(Connection connection) {
String countInvalidObjs = "select count(*) " +
"from user_objects " +
"where object_type in ('FUNCTION','PROCEDURE','PACKAGE','PACKAGE BODY','TRIGGER') " +
"and status = 'INVALID' ";
int invalidObjCount = -1;
Statement statement;
try {
statement = connection.createStatement();
ResultSet rs = statement.executeQuery(countInvalidObjs);
while (rs.next()) {
invalidObjCount = rs.getInt(1);
}
} catch(Throwable t) {
System.out.println("*error* " + t.getMessage());
} finally {
if(invalidObjCount!=0) {
throw new IllegalArgumentException("fail to complete migration, build finished with databse warnings");
}
}
}
Related
We have integration setup that creates purchase orders on the batch server. For example the batch job may run and pick up 5 invoices coming from an external source and attempt to post them.
If 4 are successful and 1 fails, we catch the error using the code below:
errEnumerator = SysInfologEnumerator::newData(infolog.cut());
while (errEnumerator.moveNext())
{
msgStruct = new SysInfologMessageStruct(errEnumerator.currentMessage());
errException = errEnumerator.currentException();
messageBody += msgStruct.message() + "\n";
}
Which works great in catching the error and then we return it into a log. The issue is the entire message will be shown. "Number of vouchers posted to the journal 1." 4 times and then the error message.
After each successful post we do clear the infolog by doing infolog.clear();.
If you debug this code in X++ it does clear it each time and the error will only show the actual error without the previous successful posts. But the batch job running on the batch server for some reason does not clear the infolog after each successful post. After CILs, restarting services etc. nothing seems to work.
Is there another way to clear the infolog on the batch server? thanks
If your goal is to store only the error lines in messageBody and not the 'success' lines, you don't have to clear the Infolog. You only need to add the following check at the beginning of your while cycle:
if (errEnumerator.currentException() == Exception::Info ||
errEnumerator.currentException() == Exception::Warning)
{
continue;
}
Do not mess with the infolog!
This will hide information, warnings and errors that you will need for example for batch problem solving.
So please do not clear() or cut().
Instead copy what you want:
numLine = infologLine();
try
{
// Do something useful
}
catch (Exception::Error)
{
doTheLog(infolog.copy(numLine + 1, infologLine()));
throw error("That did not work!");
}
First store the current infolog number. On error process the relevant infologs.
If the infolog is long consider transferring the numbers rather than call by value the container:
doTheLog(numLine + 1, infologLine());
Then infolog.copyin the method.
I have a window in a Qt application using PostgreSQL 9.3 database. The window is a form used do display, edit and insert new data. t looks like that:
I have data from 3 sql tables in that view. the tables are related with foreign keys:
contractors (main table) - mapped to "personal data" section
contacts (has foreign key to contractors.ID)
addresses (has foreign key to contractors.ID)
So - in my window's class I have 3 main models (+ 2 proxy models to transpose tables in "personal data" an "address data" sections). I use QSqlTableModel for theese sesctions, and a QSqlRelationalTableModel for contactData section. when opening that window "normally" (to view some contractor), i simply pass contractor's ID to the constructor and store it in proper variable. Also, I call the QSqlTableModel::​setFilter(const QString & filter) method for each of the models, and set the proper filtering. When opening that window in "add new" mode i simply pass a "-1" or "0" value to the ID variable, so no data gets loaded to the model.
All 3 models have QSqlTableModel::OnManualSubmit editStrategy. When saving the data (triggered by clicking a proper button), I start a transaction. And then I submit models one-by-one. personalData model gets submitted first, as I need to obtain it's PK after insert (to set in the FK fields in other models).
When submitting of the model fails, I show a messageBox with the QSqlError content, rollback the transaction and return from the method.
When I have an error on the first model being processed - no problem, as nothing was inserted. But when the first model is saved, but the second or third fails - there is a little problem. So I rollback the transacion as before, and return from the function. But after correcting the data and submitting it again - the first model is not trying to submit - as it doesn't know that there was a rollback, and the data needs to be inserted again. What would be a good way to notice such a model, that it needs to be submited once again?
At the moment I ended up with something like that:
void kontrahenciSubWin::on_btnContractorAdd_clicked() {
//QStringList errorList; // when error occurs in one model - whole transacion gets broken, so no need for a list
QString error;
QSqlDatabase db = QSqlDatabase::database();
//backup the data - in case something fails and we have to rollback the transaction
QSqlRecord personalDataModelrec = personalDataModel->record(0); // always one row. will get erased by SubmitAll, as no filter is set, because I don't have its ID.
QList<QSqlRecord> contactDataModelRecList;
for (int i = 0 ; i< contactDataModel->rowCount(); i++) {
contactDataModelRecList.append( contactDataModel->record(i) );
}
QList<QSqlRecord> addressDataModelRecList;
for (int i = 0 ; i< addressDataModel->rowCount(); i++) {
addressDataModelRecList.append( addressDataModel->record(i) );
}
db.transaction();
if ( personalDataModel->isDirty() && error.isEmpty() ) {
if (!personalDataModel->submitAll()) //submitAll calls select() on the model, which destroys the data as the filter is invalid ("where ID = -1")
//errorList.append( personalDataModel->lastError().databaseText() );
error = personalDataModel->lastError().databaseText();
else {
kontrahentid = personalDataModel->query().lastInsertId().toInt(); //only here can I fetch ID
setFilter(ALL); //and pass it to the models
}
}
if ( contactDataModel->isDirty() && error.isEmpty() )
if (!contactDataModel->submitAll()) //slot on_contactDataModel_beforeInsert() sets FK field
//errorList.append( contactDataModel->lastError().databaseText() );
error = contactDataModel->lastError().databaseText();
if ( addressDataModel->isDirty() && error.isEmpty() )
if (!addressDataModel->submitAll()) //slot on_addressDataModel_beforeInsert() sets FK field
//errorList.append( addressDataModel->lastError().databaseText() );
error = addressDataModel->lastError().databaseText();
//if (!errorList.isEmpty()) {
// QMessageBox::critical(this, tr("Data was not saved!"), tr("The following errors occured:") + " \n" + errorList.join("\n"));
if (!error.isEmpty()) {
QMessageBox::critical(this, tr("Data was not saved!"), tr("The following errors occured:") + " \n" + error);
db.rollback();
personalDataModel->clear();
contactDataModel->clear();
addressDataModel->clear();
initModel(ALL); //re-init models: set table and so on.
//re-add data to the models - backup comes handy
personalDataModel->insertRecord(-1, personalDataModelrec);
for (QList<QSqlRecord>::iterator it = contactDataModelRecList.begin(); it != contactDataModelRecList.end(); it++) {
contactDataModel->insertRecord(-1, *it);
}
for (QList<QSqlRecord>::iterator it = addressDataModelRecList.begin(); it != addressDataModelRecList.end(); it++) {
addressDataModel->insertRecord(-1, *it);
}
return;
}
db.commit();
isInEditMode = false;
handleGUIOnEditModeChange();
}
Does anyone have a better idea? I doubt if it's possible to ommit backing-up the records before trying to insert them. But maybe there is a better way to "re-add" them to the model? I tried to use "setRecord", and "remoweRows" & "insertRecord" combo, but no luck. Resetting the whole model seems easiest (I only need to re-init it, as it loses table, filter, sorting and everything else when cleared)
I suggest you to use a function written in the language PLPGSQL. It has one transaction between BEGIN and END. If it goes wrong at a certain point of the code then will it rollback all data flawlessly.
What you are doing now is not a good design, because you handle the control over a certain functionality (rollback) to an external system with regard to the rollback (it is happening in the database). The external system is not designed to do that, while the database on the contrairy is created and designed for dealing with rollbacks and transactions. It is very good at it. Rebuilding and reinventing this functionality, which is quite complex, outside the database is asking for a lot of trouble. You will never get the same flawless rollback handling as you will have using functions within the database.
Let each system do what it can do best.
I have met your problem before and had the same line of thought to work this problem out using Hibernate in my case. Until I stepped back from my efforts and re-evaluated the situation.
There are three teams working on the rollback mechanism of a database:
1. the men and women who are writing the source code of the database itself,
2. the men and women who are writing the Hibernate code, and
3. me.
The first team is dedicated to the creation of a good rollback mechanism. If they fail, they have a bad product. They succeeded. The second team is dedicated to the creation of a good rollback mechanism. Their product is not failing when it is not working in very complex situations.
The last team, me, is not dedicated to this problem. Who am I to write a better solution then the people of team 2 or team 1 based on the work of team 2 who were not able to get it to the level of team 1?
That is when I decided to use database functions instead.
Using Async-based Webservice and Async framework in WinRT (Win8) to get a large recordsets(1000 to 5000) from a remote Ms SQL Server.
I want to know :
1) Which is the best approach to handle to insert large recordsets into SQLite?
2) Using RollBack transaction will start all over again if there is connection error. The below method will insert whatever and I can update the data later if the records are not complete. Is this a good approach?
3) Any better way to enhance my below solution?
This foreach statement to handle
each reords in returned result which returned from Async-Based WebService:
foreach (WebServiceList _List in IList)
{
InsertNewItems(_List.No, _List.Description, _List.Unit_Price, _List.Base_Unit_of_Measure);
}
private void InsertNewItems(string ItemNo, string ItemName, decimal ItemPrice, string ItemBUoM)
{
var existingItem = (db2.Table().Where(c => c.No == ItemNo)).SingleOrDefault();
if (existingItem != null)
{
existingItem.No = ItemNo;
existingItem.Description = ItemName;
existingItem.Unit_Price = ItemPrice;
existingItem.BaseUnitofMeasure = ItemBUoM;
int success = db2.Update(existingItem);
}
else
{
int success = db2.Insert(new Item()
{
No = ItemNo,
Description = ItemName,
Unit_Price = ItemPrice,
Base_Unit_of_Measure = ItemBUoM
});
}
}
You should use RunInTransaction from sqlite-net. The documentation for it says,
Executes action within a (possibly nested) transaction by wrapping it
in a SAVEPOINT. If an exception occurs the whole transaction is rolled
back, not just the current savepoint. The exception is rethrown.
using (var db = new SQLiteConnection(DbPath))
{
db.RunInTransaction(() =>
{
db.InsertOrReplace(MyObj);
});
}
Wiki article for Transactions at GitHub
The most important performance aspect for bulk inserts is to use a single transaction. If you want to handle aborts, I suggest that you feed the data in sufficiently large parts and restart from that point on next time. An SQL transaction either finishes completely or rolls back completely, so unless the input data changes between two runs, there should be no need to do an insert-or-update.
See, for example, here for a discussion of SQLite bulk insert performance using different methods.
I have a very unusual problem.
I'm trying to create a simple database (6 tables, 4 of which only have 2 columns).
I'm using an in-house database library which I've used in a previous project, and it does work.
However with my current project there are occasional bugs. Basically the database isn't created correctly. It is added to the sdcard but when I access it I get a DatabaseException.
When I access the device from the desktop manager and try to open the database (with SQLite Database Browser v2.0b1) I get "File is not a SQLite 3 database".
UPDATE
I found that this happens when I delete the database manually off the sdcard.
Since there's no way to stop a user doing that, is there anything I can do to handle it?
CODE
public static boolean initialize()
{
boolean memory_card_available = ApplicationInterface.isSDCardIn();
String application_name = ApplicationInterface.getApplicationName();
if (memory_card_available == true)
{
file_path = "file:///SDCard/" + application_name + ".db";
}
else
{
file_path = "file:///store/" + application_name + ".db";
}
try
{
uri = URI.create(file_path);
FileClass.hideFile(file_path);
} catch (MalformedURIException mue)
{
}
return create(uri);
}
private static boolean create(URI db_file)
{
boolean response = false;
try
{
db = DatabaseFactory.create(db_file);
db.close();
response = true;
} catch (Exception e)
{
}
return response;
}
My only suggestion is keep a default database in your assets - if there is a problem with the one on the SD Card, attempt to recreate it by copying the default one.
Not a very good answer I expect.
Since it looks like your problem is that the user is deleting your database, just make sure to catch exceptions when you open it (or access it ... wherever you're getting the exception):
try {
URI uri = URI.create("file:///SDCard/Databases/database1.db");
sqliteDB = DatabaseFactory.open(myURI);
Statement st = sqliteDB.createStatement( "CREATE TABLE 'Employee' ( " +
"'Name' TEXT, " +
"'Age' INTEGER )" );
st.prepare();
st.execute();
} catch ( DatabaseException e ) {
System.out.println( e.getMessage() );
// TODO: decide if you want to create a new database here, or
// alert the user if the SDCard is not available
}
Note that even though it's probably unusual for a user to delete a private file that your app creates, it's perfectly normal for the SDCard to be unavailable because the device is connected to a PC via USB. So, you really should always be testing for this condition (file open error).
See this answer regarding checking for SDCard availability.
Also, read this about SQLite db storage locations, and make sure to review this answer by Michael Donohue about eMMC storage.
Update: SQLite Corruption
See this link describing the many ways SQLite databases can be corrupted. It definitely sounded to me like maybe the .db file was deleted, but not the journal / wal file. If that was it, you could try deleting database1* programmatically before you create database1.db. But, your comments seem to suggest that it was something else. Perhaps you could look into the file locking failure modes, too.
If you are desperate, you might try changing your code to use a different name (e.g. database2, database3) each time you create a new db, to make sure you're not getting artifacts from the previous db.
I am doing a project in wicket
How to solve the problem.
I came across such a message:
WicketMessage: Can't instantiate page using constructor public itucs.blg361.g03.HomePage()
Root cause:
java.lang.UnsupportedOperationException: [SQLITE_BUSY] The database file is locked (database is locked)
at itucs.blg361.g03.CategoryEvents.CategoryEventCollection.getCategoryEvents(CategoryEventCollection.java:41)
public List<CategoryEvent> getCategoryEvents() {
List<CategoryEvent> categoryEvents = new
LinkedList<CategoryEvent>();
try {
String query = "SELECT id, name, group_id"
+ " FROM event_category";
Statement statement = this.db.createStatement();
ResultSet result = statement.executeQuery(query);
while (result.next()) {
int id = result.getInt("id");
String name = result.getString("name");
int group_id = result.getInt("group_id");
categoryEvents.add(new CategoryEvent(id, name, group_id));
}
} catch (SQLException ex) {
throw new UnsupportedOperationException(ex.getMessage());
}
return categoryEvents;
}
at itucs.blg361.g03.HomePage.(HomePage.java:71)
categories = categoryCollection.getCategoryEvents();
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
Sqlite allows only one writer to the whole database at a time and, unless you selected "WAL" journal mode, no reader while writing. Moreover unless you explicitly ask it to wait, it simply returns the SQLITE_BUSY status for any attempt to access the database while conflicting operation is running.
You can tell sqlite to wait for the database to become available for a specified amount of time. The C-level API is sqlite3_busy_timeout; I never used sqlite from Java though, so I don't know where to find it there.
(...) tell sqlite to wait for the database to become available for specified amount of time.
In order to do it from Java, run the following statement just like a simple SQL statement:
pragma busy_timeout=30000; -- Busy timeout set to 30000 milliseconds