Is Flyway suitable for implementation in an application that will support multiple databases?
We don't know what our customers are using - could be either MySQL, Postgres or Oracle. Can we still use Flyway to migrate the database for new versions of the application?
if your question is: does Flyway provide a DDL abstraction layer across the databases it supports, the answer is no.
This was a conscious design decision, to make sure the full power of the underlying database is available and not just the smallest common denominator supported by the migration tool.
For your use case, you could either provide different migration scripts for the different databases. They should be very similar though.
If you do not wish to potentially duplicate the migration scripts and can live with the smallest common denominator approach, have a look at LiquiBase which might be a better fit for your usecase (if you can live with the XML)
You could use jOOQ's parsing connection, which wraps your target JDBC connection and is capable of translating your input DDL to any target dialect (if it's not too fancy and vendor specific). Flyway wouldn't be aware of this translating JDBC proxy, and wouldn't have to be. The online version of the SQL translator can be seen here. For example, if your input SQL is a MySQL specific:
create table t (i int primary key auto_increment);
The output could be:
-- Oracle
create table T (
I number(10) generated by default as identity(start with 1) not null,
primary key (I)
);
-- SQL Server
create table T (
I int identity(1, 1) not null,
primary key (I)
)
-- PostgreSQL
create table T (
I int generated by default as identity not null,
primary key (I)
)
-- PostgreSQL 9.4
create table T (
I serial4 not null,
primary key (I)
)
Disclaimer: I work for the company behind jOOQ.
Related
I'm developing a Rust application for user registration via SSH (like the one working for SDF).
I'm using the SQLite3 database as a backend to store the information about users.
I'm opening the database file (or creating it if it does not exist) but I don't know the approach for checking if the necessary tables with expected structure are present in the database.
I tried to use PRAGMA schema_version for versioning purposes, but this approach is unreliable.
I found that there are posts with answers that are heavily related to my question:
How to list the tables in a SQLite database file that was opened with ATTACH?
How do I retrieve all the tables from database? (Android, SQLite)
How do I check in SQLite whether a table exists?
I'm opening the database file (or creating it if it does not exist)
but I don't know the approach for checking if the necessary tables
I found querying sqlite_master to check for tables, indexes, triggers and views and for columns using PRAGMA table_info(the_table_name) to check for columns.
e.g. the following would allow you to get the core basic information and to then be able to process it with relative ease (just for tables for demonstration):-
SELECT name, sql FROM sqlite_master WHERE type = 'table' AND name LIKE 'my%';
with expected structure
PRAGMA table_info(mytable);
The first results in (for example) :-
Whilst the second results in (for mytable) :-
Note that type is blank/null for all columns as the SQL to create the table doesn't specify column types.
If you are using SQLite 3.16.0 or greater then you could use PRAGMA Functions (e.g. pragma_table_info(table_name)) rather than the two step approach need prior to 3.16.0.
I have to find out in MariaDb how to implement some features used in Oracle . I have :
Load a file: in Oracle I use the external table. Is there a way (fast and efficient one ) to load a file into a table . Has MariaDb a plugin which allows to load well a specific format of files?
In my existing Oracle code I used to developp a java wrap functions which allow those feature (is there a way in MariaDb to do this?), specifically :
1- Searching a files in an OS directory and insert them in a table,
2- send an SNMP trap
3- Send a mail via SMTP
Is there an equivalent to an Oracle job in Mariadb?
Is there an equivalent to Oracle TDE (Transparent data encryption) ?
Is there an equivalent to the VPD (virtual private policy)?
What is the maximum length of a varchar column/variable ? (in Oracle we can use the CLOBs..)
Many Thanks and Best Regards
MariaDB (and MySQL) can do a LOAD DATA on a CSV file. It is probably the most efficient way to convert external data to a table. (There is also ENGINE=CSV, which requires no conversion, but is limited in that it has no indexes, etc.)
MariaDB cannot, for security reasons, issue any arbitrary system calls. No emails, no 'exec', etc.
No Job, TDE, VPD.
Network transmissions can (optionally) use SSL for encryption at that level.
There is a family of virtually identical datatypes for characters:
CHAR(n), VARCHAR(n) -- where n is up to 65535; n is the limit of _characters_, not _bytes_.
TINYTEXT, TEXT, MEDIUMTEXT, LONGTEXT -- of various limits; the last is limited to 4GB.
For non-character storage (eg, images), there is a similar set of datatypes
BINARY(n), VARBINARY(n)
TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB
The various sizes of TEXT and BLOB indicate whether that is a 1-, 2-, 3-, or 4-byte length field in the implementation.
NVARCHAR is a synonym for VARCHAR. Character sets are handled by declaring a column to be, for example, CHARACTER SET utf8 COLLATE utf8_unicode_ci. Such can be defaulted at the database (schema) level, defaulted at the table level, or specified differently for different columns (even in the same table).
Created the following code in SQL however need to use it in sqlite (phonegap specifically).
INSERT INTO actions(Action) VALUES ('Go to the pub');
SET #aid = LAST_INSERT_ID();
INSERT INTO statements(statement, Language) VALUES ('Have a pint', 'English');
SET #sid = LAST_INSERT_ID();
INSERT INTO Relationships(SID,AID) VALUES (#sid,#aid);
The issue we are having however is how to declare the variables in sqlite.
The LAST_INSERT_ID() will become last_insert_rowid(), however what is the sqlite version of SET #aid = ?
SQLite does not have variables.
In an embedded database such as SQLite, there is no separate server machine or even process, so it would not make sense to add a programming language to the DB engine when the same control flow and processing logic could be just as well done in the application itself.
Just use three separate INSERT statements.
(In WebSQL, the result object has the insertId property.)
We have started to use the updated System.Web.Providers provided in the Microsoft.AspNet.Providers.Core package from NuGet. We started to migrate our existing users and found performance slowing and then deadlocks occurring. This was with less than 30,000 users (much less than the 1,000,000+ we need to create). When we were calling the provider, it was from multiple threads on each server and there were multiple servers running this same process. This was to be able to create all the users we required as quickly as possible and to simulate the load we expect to see when it goes live.
The logs SQL Server generated for for a deadlock contained the EF generated sql below:
SELECT
[Limit1].[UserId] AS [UserId]
, [Limit1].[ApplicationId] AS [ApplicationId]
, [Limit1].[UserName] AS [UserName]
, [Limit1].[IsAnonymous] AS [IsAnonymous]
, [Limit1].[LastActivityDate] AS [LastActivityDate]
FROM
(SELECT TOP (1)
[Extent1].[UserId] AS [UserId]
, [Extent1].[ApplicationId] AS [ApplicationId]
, [Extent1].[UserName] AS [UserName]
, [Extent1].[IsAnonymous] AS [IsAnonymous]
, [Extent1].[LastActivityDate] AS [LastActivityDate]
FROM
[dbo].[Users] AS [Extent1]
INNER JOIN [dbo].[Applications] AS [Extent2] ON [Extent1].[ApplicationId] = [Extent2].[ApplicationId]
WHERE
((LOWER([Extent2].[ApplicationName])) = (LOWER(#p__linq__0)))
AND ((LOWER([Extent1].[UserName])) = (LOWER(#p__linq__1)))
) AS [Limit1]
We ran the query manually and the execution plan said that it was performing a table scan even though there was an underlying index. The reason for this is the use of LOWER([Extent1].[UserName]).
We looked at the provider code to see if we were doing something wrong or if there was a way to either intercept or replace the database access code. We didn't see any options to do this but we did find the source of the LOWER issue, .ToLower() is being called on both the column and parameter.
return (from u in ctx.Users
join a in ctx.Applications on u.ApplicationId equals a.ApplicationId into a
where (a.ApplicationName.ToLower() == applicationName.ToLower()) && (u.UserName.ToLower() == userName.ToLower())
select u).FirstOrDefault<User>();
Does anyone know of a way that we change the behaviour of the provider to not use .ToLower() so allowing the index to be used?
You can create an index on lower(username) per Sql Server : Lower function on Indexed Column
ALTER TABLE dbo.users ADD LowerFieldName AS LOWER(username) PERSISTED
CREATE NONCLUSTERED INDEX IX_users_LowerFieldName_ ON dbo.users(LowerFieldName)
I was using the System.Web.Providers.DefaultMembershipProvider membership provider too but found that it was really slow. I changed to the System.Web.Security.SqlMembershipProvider and found it to be much faster (>5 times faster).
This tutorial shows you how to set up the SQL database that you need to use the SqlMembershipProvider http://weblogs.asp.net/sukumarraju/archive/2009/10/02/installing-asp-net-membership-services-database-in-sql-server-expreess.aspx
This database that is auto generated uses stored procedures which may or may not be an issue for your DB guys.
I've got this query
UPDATE linkeddb...table SET field1 = 'Y' WHERE column1 = '1234'
This takes 23 seconds to select and update one row
But if I use openquery (which I don't want to) then it only takes half a second.
The reason I don't want to use openquery is so I can add parameters to my query securely and be safe from SQL injections.
Does anyone know of any reason for it to be running so slowly?
Here's a thought as an alternative. Create a stored procedure on the remote server to perform the update and then call that procedure from your local instance.
/* On remote server */
create procedure UpdateTable
#field1 char(1),
#column1 varchar(50)
as
update table
set field1 = #field1
where column1 = #column1
go
/* On local server */
exec linkeddb...UpdateTable #field1 = 'Y', #column1 = '1234'
If you're looking for the why, here's a possibility from Linchi Shea's Blog:
To create the best query plans when
you are using a table on a linked
server, the query processor must have
data distribution statistics from the
linked server. Users that have limited
permissions on any columns of the
table might not have sufficient
permissions to obtain all the useful
statistics, and might receive aless
efficient query plan and experience
poor performance. If the linked
serveris an instance of SQL Server, to
obtain all available statistics, the
user must own the table or be a member
of the sysadmin fixed server role, the
db_ownerfixed database role, or the
db_ddladmin fixed database role on the
linkedserver.
(Because of Linchi's post, this clarification has been added to the latest BooksOnline SQL documentation).
In other words, if the linked server is set up with a user that has limited permissions, then SQL can't retrieve accurate statistics for the table and might choose a poor method for executing a query, including retrieving all rows.
Here's a related SO question about linked server query performance. Their conclusion was: use OpenQuery for best performance.
Update: some additional excellent posts about linked server performance from Linchi's blog.
Is column1 primary key? Probably not. Try to select records for update using primary key (where PK_field=xxx), otherwise (sometimes?) all records will be read to find PK for records to update.
Is column1 a varchar field? Is that why are you surrounding the value 1234 with single-quotation marks? Or is that simply a typo in your question?