I have the following code in SQL
-- SCHEMA VERSION: 2
-- Pre-update actions
PRAGMA foreign_keys = OFF;
-- end
-- Create HARVEST_PERIOD table
CREATE TABLE "main"."HARVEST_PERIOD" (
"ID" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
"CODE" TEXT(64) NOT NULL,
"PERIOD" TEXT(64) NOT NULL,
"CURRENT_STATE" TEXT(128)
)
;
-- Post-update actions
INSERT OR REPLACE INTO "main"."SETTINGS" ("NAME", "VALUE") values ("SchemaVersion", "2");
PRAGMA foreign_keys = ON;
-- end
The new table is created as expected and the settings table updated as expected, too. What could be the reason for getting this: [Err] 21 - not an error
Is there any better suggested way to create the new schema?
I encounter this error as well. Later I figured it out. It's because another application was connected to the same db. So, my application can't modify the db -- create a table.
I created it successfully just by closing the another db connection.
Related
Answer: Use db.Exec("PRAGMA foreign_keys = ON") to enforce foreign key constraint checks. Thanks #outdead
When I update my SQLite database using GORM, foreign key constraints aren't enforced.
I have these 2 models:
type Cat struct {
ID int
Name string
Breed string
OwnerID int
Owner Owner
}
type Owner struct {
ID int
Name string
Phone string
}
Which correctly creates a foreign key constraint where owner_id references id in owners. This can be verified by running: .schema cats in the SQLite shell:
CREATE TABLE `cats` (`id` integer,`name` text,`breed` text,`owner_id` integer,PRIMARY KEY (`id`),CONSTRAINT `fk_cats_owner` FOREIGN KEY (`owner_id`) REFERENCES `owners`(`id`));
I have tried PRAGMA foreign_keys = ON; which enforces foreign keys when I run commands in the SQLite shell. If I try to update an owner_id to an id that doesn't exist in owners, I get: Error: FOREIGN KEY constraint failed, which is the behaviour that I want, however, GORM is still able to execute these updates without receiving this error.
You need to exec query to turn on PRAGMA foreign_keys before updating
if res := db.Exec("PRAGMA foreign_keys = ON", nil); res.Error != nil {
return res.Error
}
An alternative to the other answer is to append ?_foreign_keys=on to the connection string:
db, err := gorm.Open(sqlite.Open("my.db?_foreign_keys=on"), &gorm.Config{})
See the go-sqlite3 driver and this question.
Verified working with gorm v1.23.1, gorm's sqlite driver v1.3.1, and go-sqlite3 v2.0.3.
For some reason, I cannot get FOREIGN KEY to work.
Any INSERT into primary_specs will go through, despite the 'ad_ids' table being empty.
EDIT: for some reason the INSERT will NOT go through when using DB Browser, but my python script waltzes right pass that constraint and is able to save the data...
Main table:
CREATE TABLE "ad_ids" (
"ad_id" INTEGER,
"ad_url" TEXT,
PRIMARY KEY("ad_id")
)
Secondary table:
CREATE TABLE "primary_specs" (
"ad_id" INTEGER,
"version" TEXT,
"year" INTEGER,
PRIMARY KEY("ad_id"),
FOREIGN KEY("ad_id")
REFERENCES ad_ids("ad_id")
)
PRAGMA foreign_key_list(primary_specs);
returns:
id seq table from to on_update on_delete match
0 0 ad_ids ad_id ad_id NO ACTION NO ACTION NONE
PRAGMA foreign_keys
returns 1
Apparently PRAGMA foreign_keys = ON; is applied on connection and not a database.
I want to use foreign_key_check to check my SQLITE3 database using version 3.21.0.
The following statements applied to an empty table yield error 1 instead of a list of foreign key violations. The tables are clearly empty. Why do I get the error?
CREATE TABLE City(CityName TEXT NOT NULL, CityID INTEGER NOT NULL);
CREATE TABLE Street(StreetName TEXT NOT NULL, CityID INTEGER NOT NULL REFERENCES City(CityID));
PRAGMA foreign_key_check;
CityID is required to be UNIQUE.
In my usecase, the table City contains some cities with different variations of the name (New York == New York City) with the same CityID, hence it is not UNIQUE in my case. I need to change the database structure.
I still wonder why this is error 1. I can create the tables with PRAGMA foreign_keys = ON; and no error.
I tried to set up a database with two tables in sqlite. Once of my table is having a timestamp column . I am trying to implement timestamp mode to capture incremental changes in the DB. Kafka connect is failing with the below error:
ERROR Failed to get current time from DB using Sqlite and query 'SELECT
CURRENT_TIMESTAMP'
(io.confluent.connect.jdbc.dialect.SqliteDatabaseDialect:471)
java.sql.SQLException: Error parsing time stamp
Caused by: java.text.ParseException: Unparseable date: "2019-02-05 02:05:29"
does not match (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q
\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q.\E(\p{Nd}++)
Many thanks for the help
Config:
name=test-query-sqlite-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:sqlite:employee.db
query=SELECT users.id, users.name, transactions.timestamp, transactions.payment_type FROM users JOIN transactions ON (users.id = transactions.user_id)
mode=timestamp
timestamp.column.name=timestamp
topic.prefix=test-joined
DDL:
CREATE TABLE transactions(id integer primary key not null,
payment_type text not null,
timestamp DATETIME DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
user_id int not null,
constraint fk foreign key(user_id) references users(id)
);
CREATE TABLE users (id integer primary key not null,name text not null);
The kafka connect jdbc connector easily detects the changes in the timestamp, if the values of the 'timestamp' column are in the format of the 'UNIX timestamp'.
sqlite> CREATE TABLE transact(timestamp TIMESTAMP DEFAULT (STRFTIME('%s', 'now')) not null,
...> id integer primary key not null,
...> payment_type text not null);
sqlite>
The values can be inserted as:
sqlite> INSERT INTO transact(timestamp,payment_type,id) VALUES (STRFTIME('%s', 'now'),'cash',1);
The timestamp related changes are then detected by the kafka jdbc source connector and the same can be consumed as follows:
kafka-console-consumer --bootstrap-server localhost:9092 --topic jdbc-transact --from-beginning
{"timestamp":1562321516,"id":2,"payment_type":"card"}
{"timestamp":1562321790,"id":1,"payment_type":"online"}
I've reproduced this, and it is already logged as an issue for the JDBC Source connector. You can monitor it here: https://github.com/confluentinc/kafka-connect-jdbc/issues/219
We have a DACPAC (sqlproj) solution which has some tables and a post-deployment script which runs some DML queries.
If the DML query fails (I'm raising an error with severity=20), I would like to rollback all changes - including the DDL changes done by the dacpac and the post-deployment file changes. This is especially useful when I would be upgrading an existing target database.
I'm striving for an atomic DB upgrade when the DACPAC is published - all DDL changes mentioned in the DACPAC solution should be published only when everything in the post deployment script is successful.
Since DACPAC DDL changes are committed before it invokes post-deployment script, I thought generating all the DAC changes as a single script file using DacServices.GenerateDeployScript will help. Doesnt look so straight forward.
Has anyone tried something like this (and failed/passed)?
I'm facing many challenges like...
Create/alter Database shouldnt be in a transaction.
Rollback not happening at all.
[Edit 10Nov]: Pasting the deploy script generated by the dacpac here, so that I can explain my issue better (hopefully)
/*
Deployment script for 9Nov
This code was generated by a tool.
Changes to this file may cause incorrect behavior and will be lost if
the code is regenerated.
*/
GO
SET ANSI_NULLS, ANSI_PADDING, ANSI_WARNINGS, ARITHABORT, CONCAT_NULL_YIELDS_NULL, QUOTED_IDENTIFIER ON;
SET NUMERIC_ROUNDABORT OFF;
GO
:setvar INSTALL_DIR "D:\EDW_9Nov\"
:setvar DatabaseName "9Nov"
:setvar DefaultFilePrefix "9Nov"
:setvar DefaultDataPath "C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\"
:setvar DefaultLogPath "C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\DATA\"
GO
:on error exit
GO
/*
Detect SQLCMD mode and disable script execution if SQLCMD mode is not supported.
To re-enable the script after enabling SQLCMD mode, execute the following:
SET NOEXEC OFF;
*/
:setvar __IsSqlCmdEnabled "True"
GO
IF N'$(__IsSqlCmdEnabled)' NOT LIKE N'True'
BEGIN
PRINT N'SQLCMD mode must be enabled to successfully execute this script.';
SET NOEXEC ON;
END
GO
USE [master];
GO
IF (DB_ID(N'$(DatabaseName)') IS NOT NULL)
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
DROP DATABASE [$(DatabaseName)];
END
GO
PRINT N'Creating $(DatabaseName)...'
GO
CREATE DATABASE [$(DatabaseName)]
ON
PRIMARY(NAME = [$(DatabaseName)], FILENAME = N'$(DefaultDataPath)$(DefaultFilePrefix)_Primary.mdf')
LOG ON (NAME = [$(DatabaseName)_log], FILENAME = N'$(DefaultLogPath)$(DefaultFilePrefix)_Primary.ldf') COLLATE SQL_Latin1_General_CP1_CI_AS
GO
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET ANSI_NULLS ON,
ANSI_PADDING ON,
ANSI_WARNINGS ON,
ARITHABORT ON,
CONCAT_NULL_YIELDS_NULL ON,
NUMERIC_ROUNDABORT OFF,
QUOTED_IDENTIFIER ON,
ANSI_NULL_DEFAULT ON,
CURSOR_DEFAULT LOCAL,
RECOVERY SIMPLE,
CURSOR_CLOSE_ON_COMMIT OFF,
AUTO_CREATE_STATISTICS ON,
AUTO_SHRINK OFF,
AUTO_UPDATE_STATISTICS ON,
RECURSIVE_TRIGGERS OFF
WITH ROLLBACK IMMEDIATE;
ALTER DATABASE [$(DatabaseName)]
SET AUTO_CLOSE OFF
WITH ROLLBACK IMMEDIATE;
END
GO
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET ALLOW_SNAPSHOT_ISOLATION OFF;
END
GO
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET READ_COMMITTED_SNAPSHOT OFF
WITH ROLLBACK IMMEDIATE;
END
GO
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET AUTO_UPDATE_STATISTICS_ASYNC OFF,
PAGE_VERIFY NONE,
DATE_CORRELATION_OPTIMIZATION OFF,
DISABLE_BROKER,
PARAMETERIZATION SIMPLE,
SUPPLEMENTAL_LOGGING OFF
WITH ROLLBACK IMMEDIATE;
END
GO
IF IS_SRVROLEMEMBER(N'sysadmin') = 1
BEGIN
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
EXECUTE sp_executesql N'ALTER DATABASE [$(DatabaseName)]
SET TRUSTWORTHY OFF,
DB_CHAINING OFF
WITH ROLLBACK IMMEDIATE';
END
END
ELSE
BEGIN
PRINT N'The database settings cannot be modified. You must be a SysAdmin to apply these settings.';
END
GO
IF IS_SRVROLEMEMBER(N'sysadmin') = 1
BEGIN
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
EXECUTE sp_executesql N'ALTER DATABASE [$(DatabaseName)]
SET HONOR_BROKER_PRIORITY OFF
WITH ROLLBACK IMMEDIATE';
END
END
ELSE
BEGIN
PRINT N'The database settings cannot be modified. You must be a SysAdmin to apply these settings.';
END
GO
ALTER DATABASE [$(DatabaseName)]
SET TARGET_RECOVERY_TIME = 0 SECONDS
WITH ROLLBACK IMMEDIATE;
GO
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET FILESTREAM(NON_TRANSACTED_ACCESS = OFF),
CONTAINMENT = NONE
WITH ROLLBACK IMMEDIATE;
END
GO
IF EXISTS (SELECT 1
FROM [master].[dbo].[sysdatabases]
WHERE [name] = N'$(DatabaseName)')
BEGIN
ALTER DATABASE [$(DatabaseName)]
SET AUTO_CREATE_STATISTICS ON(INCREMENTAL = OFF),
MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT = OFF,
DELAYED_DURABILITY = DISABLED
WITH ROLLBACK IMMEDIATE;
END
GO
USE [$(DatabaseName)];
GO
IF fulltextserviceproperty(N'IsFulltextInstalled') = 1
EXECUTE sp_fulltext_database 'enable';
GO
PRINT N'Creating [EDW_INTERNAL]...';
GO
CREATE SCHEMA [EDW_INTERNAL]
AUTHORIZATION [dbo];
GO
PRINT N'Creating [EDW_INTERNAL].[DB_VERSIONS]...';
GO
CREATE TABLE [EDW_INTERNAL].[DB_VERSIONS] (
[ID] BIGINT IDENTITY (1, 1) NOT NULL,
[MODULE] VARCHAR (30) NOT NULL,
[FROM_VERSION] VARCHAR (20) NOT NULL,
[TO_VERSION] VARCHAR (20) NOT NULL,
[UPGRADE_DML_APPLIED_YN] VARCHAR (1) NOT NULL,
CONSTRAINT [PK_DB_VERSIONS] PRIMARY KEY CLUSTERED ([MODULE] ASC, [FROM_VERSION] ASC, [TO_VERSION] ASC)
);
GO
PRINT N'Creating [EDW_INTERNAL].[DML_UPGR_SCRIPT_MASTER]...';
GO
CREATE TABLE [EDW_INTERNAL].[DML_UPGR_SCRIPT_MASTER] (
[MODULE] VARCHAR (30) NOT NULL,
[FROM_VERSION] VARCHAR (20) NOT NULL,
[TO_VERSION] VARCHAR (20) NOT NULL,
[APPLY_ORDER] INT NOT NULL,
[UPGR_SCRIPT_FILEPATH] VARCHAR (1024) NOT NULL,
CONSTRAINT [PK_DML_UPGR_SCRIPT_MASTER] PRIMARY KEY CLUSTERED ([APPLY_ORDER] ASC, [TO_VERSION] ASC, [FROM_VERSION] ASC, [MODULE] ASC)
);
GO
PRINT N'Creating unnamed constraint on [EDW_INTERNAL].[DB_VERSIONS]...';
GO
ALTER TABLE [EDW_INTERNAL].[DB_VERSIONS]
ADD DEFAULT 'N' FOR [UPGRADE_DML_APPLIED_YN];
GO
PRINT N'Creating [EDW_INTERNAL].[UPGRADE_DML]...';
GO
CREATE PROCEDURE EDW_INTERNAL.UPGRADE_DML
#Module VARCHAR(30)
AS
BEGIN
DECLARE #Failure bit = 1;
IF #Failure = 1
BEGIN
RAISERROR
(N'One or more database upgrade query statements have failed. Please check the DML Upgrade Log table for details.',
20, -- Severity.
1 -- State
) WITH LOG;
END
END
GO
/*
Post-Deployment Script Template
--------------------------------------------------------------------------------------
This file contains SQL statements that will be appended to the build script.
Use SQLCMD syntax to include a file in the post-deployment script.
Example: :r .\myfile.sql
Use SQLCMD syntax to reference a variable in the post-deployment script.
Example: :setvar TableName MyTable
SELECT * FROM [$(TableName)]
--------------------------------------------------------------------------------------
*/
-- Reference to load the Version Upgrade tables with rows
--:r VersionUpgradeRowsPopulate.sql
-- execute
EXEC [EDW_INTERNAL].[UPGRADE_DML] #MODULE = 'Test_Common'
GO
GO
DECLARE #VarDecimalSupported AS BIT;
SELECT #VarDecimalSupported = 0;
IF ((ServerProperty(N'EngineEdition') = 3)
AND (((##microsoftversion / power(2, 24) = 9)
AND (##microsoftversion & 0xffff >= 3024))
OR ((##microsoftversion / power(2, 24) = 10)
AND (##microsoftversion & 0xffff >= 1600))))
SELECT #VarDecimalSupported = 1;
IF (#VarDecimalSupported > 0)
BEGIN
EXECUTE sp_db_vardecimal_storage_format N'$(DatabaseName)', 'ON';
END
GO
PRINT N'Update complete.';
GO