I tried to set up a database with two tables in sqlite. Once of my table is having a timestamp column . I am trying to implement timestamp mode to capture incremental changes in the DB. Kafka connect is failing with the below error:
ERROR Failed to get current time from DB using Sqlite and query 'SELECT
CURRENT_TIMESTAMP'
(io.confluent.connect.jdbc.dialect.SqliteDatabaseDialect:471)
java.sql.SQLException: Error parsing time stamp
Caused by: java.text.ParseException: Unparseable date: "2019-02-05 02:05:29"
does not match (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q
\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q.\E(\p{Nd}++)
Many thanks for the help
Config:
name=test-query-sqlite-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:sqlite:employee.db
query=SELECT users.id, users.name, transactions.timestamp, transactions.payment_type FROM users JOIN transactions ON (users.id = transactions.user_id)
mode=timestamp
timestamp.column.name=timestamp
topic.prefix=test-joined
DDL:
CREATE TABLE transactions(id integer primary key not null,
payment_type text not null,
timestamp DATETIME DEFAULT(STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
user_id int not null,
constraint fk foreign key(user_id) references users(id)
);
CREATE TABLE users (id integer primary key not null,name text not null);
The kafka connect jdbc connector easily detects the changes in the timestamp, if the values of the 'timestamp' column are in the format of the 'UNIX timestamp'.
sqlite> CREATE TABLE transact(timestamp TIMESTAMP DEFAULT (STRFTIME('%s', 'now')) not null,
...> id integer primary key not null,
...> payment_type text not null);
sqlite>
The values can be inserted as:
sqlite> INSERT INTO transact(timestamp,payment_type,id) VALUES (STRFTIME('%s', 'now'),'cash',1);
The timestamp related changes are then detected by the kafka jdbc source connector and the same can be consumed as follows:
kafka-console-consumer --bootstrap-server localhost:9092 --topic jdbc-transact --from-beginning
{"timestamp":1562321516,"id":2,"payment_type":"card"}
{"timestamp":1562321790,"id":1,"payment_type":"online"}
I've reproduced this, and it is already logged as an issue for the JDBC Source connector. You can monitor it here: https://github.com/confluentinc/kafka-connect-jdbc/issues/219
Related
Before I alter a table I want to make a backup of it, so I use this code :
CREATE TABLE states_BACKUP (
state_id INTEGER NOT NULL,
domain VARCHAR(64),
entity_id VARCHAR(255),
state VARCHAR(255),
attributes TEXT,
event_id INTEGER,
last_changed DATETIME,
last_updated DATETIME,
created DATETIME,
context_id VARCHAR(36),
context_user_id VARCHAR(36), old_state_id INTEGER,
PRIMARY KEY (state_id),
FOREIGN KEY(event_id) REFERENCES events (event_id)
);
INSERT into states_BACKUP
Select *
FROM STATES;
However when the insert part is executed an error message says :
Execution finished with errors.
Result: UNIQUE constraint failed: states_BACKUP.state_id
At line 18:
INSERT into states_BACKUP
Select *
FROM STATES;
when I change the code to
Select distinct *
FROM STATES;
I get the same eror.
Trying to find an answer on the web how to solve this problem, I find this error has to do with duplicate id's. I wonder how this can happen when I just copy a table.
Anyone has a solution for this ?
I have a question for changing the backend connection from SQLite to SQL Server. After passing in the correct connection string for sql_alchemy_conn, I run this command: airflow initdb. I get the following error:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]A table can only have one timestamp column. Because table 'task_reschedule' already has one, the column 'start_date' cannot be added. (2738) (SQLExecDirectW)") [SQL: '\nCREATE TABLE task_reschedule (\n\tid INTEGER NOT NULL IDENTITY(1,1), \n\ttask_id VARCHAR(250) NOT NULL, \n\tdag_id VARCHAR(250) NOT NULL, \n\texecution_date TIMESTAMP NOT NULL, \n\ttry_number INTEGER NOT NULL, \n\tstart_date TIMESTAMP NOT NULL, \n\tend_date TIMESTAMP NOT NULL, \n\tduration INTEGER NOT NULL, \n\treschedule_date TIMESTAMP NOT NULL, \n\tPRIMARY KEY (id), \n\tCONSTRAINT task_reschedule_dag_task_date_fkey FOREIGN KEY(task_id, dag_id, execution_date) REFERENCES task_instance (task_id, dag_id, execution_date)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/f405)
So this works for me:
In the file: 0a2a5b66e19d_add_task_reschedule_table.py add this:
def mysql_datetime():
return mysql.DATETIME(timezone=True)
and replace any lines which has timestamp() such as the below:
sa.Column('execution_date', timestamp(), nullable=False, server_default=None),
with this:
sa.Column('execution_date', mysql_datetime(), nullable=False, server_default=None),
Once I made this change, the above error disappears but I am not sure if there are any other unintended consequences. If so I will update here or just resort to using MySQL database.
I'm trying to create a database from the E/R Diagram I just created using MySQL WorkBench, can someone please help?
Executing SQL script in server
ERROR: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near
CONSTRAINT `IDProveedor`
FOREIGN KEY (`IDproveedor`)
REFERENCES `Repu
at line 11
SQL Code:
-- -----------------------------------------------------
-- Table `Repuestos`.`Articulo`
-- -----------------------------------------------------
CREATE TABLE IF NOT EXISTS `Repuestos`.`Articulo` (
`IDArticulo` INT NOT NULL AUTO_INCREMENT,
`Nombre` VARCHAR(45) NOT NULL,
`IDproveedor` INT NOT NULL,
`Articulocol` VARCHAR(45) NOT NULL,
`Valor_Unitario` INT NOT NULL,
PRIMARY KEY (`IDArticulo`),
INDEX `NitProveedor_idx` (`IDproveedor` ASC) VISIBLE,
CONSTRAINT `IDProveedor`
FOREIGN KEY (`IDproveedor`)
REFERENCES `Repuestos`.`Proveedor` (`IDProveedor`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
SQL script execution finished: statements: 6 succeeded, 1 failed
Fetching back view definitions in final form.
Nothing to fetch.
I have an SQLite database, version 3 and I am using C# to create an application that uses this database.
I want to use a timestamp field in a table for concurrency, but I notice that when I insert a new record, this field is not set, and is null.
For example, in MS SQL Server if I use a timestamp field it is updated by the database and I don't have to set it by myself. Is this possible in SQLite?
Just declare a default value for a field:
CREATE TABLE MyTable(
ID INTEGER PRIMARY KEY,
Name TEXT,
Other STUFF,
Timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
However, if your INSERT command explicitly sets this field to NULL, it will be set to NULL.
You can create TIMESTAMP field in table on the SQLite, see this:
CREATE TABLE my_table (
id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
name VARCHAR(64),
sqltime TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL
);
INSERT INTO my_table(name, sqltime) VALUES('test1', '2010-05-28T15:36:56.200');
INSERT INTO my_table(name, sqltime) VALUES('test2', '2010-08-28T13:40:02.200');
INSERT INTO my_table(name) VALUES('test3');
This is the result:
SELECT * FROM my_table;
Reading datefunc a working example of automatic datetime completion would be:
sqlite> CREATE TABLE 'test' (
...> 'id' INTEGER PRIMARY KEY,
...> 'dt1' DATETIME NOT NULL DEFAULT (datetime(CURRENT_TIMESTAMP, 'localtime')),
...> 'dt2' DATETIME NOT NULL DEFAULT (strftime('%Y-%m-%d %H:%M:%S', 'now', 'localtime')),
...> 'dt3' DATETIME NOT NULL DEFAULT (strftime('%Y-%m-%d %H:%M:%f', 'now', 'localtime'))
...> );
Let's insert some rows in a way that initiates automatic datetime completion:
sqlite> INSERT INTO 'test' ('id') VALUES (null);
sqlite> INSERT INTO 'test' ('id') VALUES (null);
The stored data clearly shows that the first two are the same but not the third function:
sqlite> SELECT * FROM 'test';
1|2017-09-26 09:10:08|2017-09-26 09:10:08|2017-09-26 09:10:08.053
2|2017-09-26 09:10:56|2017-09-26 09:10:56|2017-09-26 09:10:56.894
Pay attention that SQLite functions are surrounded in parenthesis!
How difficult was this to show it in one example?
Have fun!
you can use triggers. works very well
CREATE TABLE MyTable(
ID INTEGER PRIMARY KEY,
Name TEXT,
Other STUFF,
Timestamp DATETIME);
CREATE TRIGGER insert_Timestamp_Trigger
AFTER INSERT ON MyTable
BEGIN
UPDATE MyTable SET Timestamp =STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW') WHERE id = NEW.id;
END;
CREATE TRIGGER update_Timestamp_Trigger
AFTER UPDATE On MyTable
BEGIN
UPDATE MyTable SET Timestamp = STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW') WHERE id = NEW.id;
END;
To complement answers above...
If you are using EF, adorn the property with Data Annotation [Timestamp], then
go to the overrided OnModelCreating, inside your context class, and add this Fluent API code:
modelBuilder.Entity<YourEntity>()
.Property(b => b.Timestamp)
.ValueGeneratedOnAddOrUpdate()
.IsConcurrencyToken()
.ForSqliteHasDefaultValueSql("CURRENT_TIMESTAMP");
It will make a default value to every data that will be insert into this table.
you can use the custom datetime by using...
create table noteTable3
(created_at DATETIME DEFAULT (STRFTIME('%d-%m-%Y %H:%M', 'NOW','localtime')),
title text not null, myNotes text not null);
use 'NOW','localtime' to get the current system date else it will show some past or other time in your Database after insertion time in your db.
Thanks You...
If you use the SQLite DB-Browser you can change the default value in this way:
Choose database structure
select the table
modify table
in your column put under 'default value' the value: =(datetime('now','localtime'))
I recommend to make an update of your database before, because a wrong format in the value can lead to problems in the SQLLite Browser.
I have the following code in SQL
-- SCHEMA VERSION: 2
-- Pre-update actions
PRAGMA foreign_keys = OFF;
-- end
-- Create HARVEST_PERIOD table
CREATE TABLE "main"."HARVEST_PERIOD" (
"ID" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
"CODE" TEXT(64) NOT NULL,
"PERIOD" TEXT(64) NOT NULL,
"CURRENT_STATE" TEXT(128)
)
;
-- Post-update actions
INSERT OR REPLACE INTO "main"."SETTINGS" ("NAME", "VALUE") values ("SchemaVersion", "2");
PRAGMA foreign_keys = ON;
-- end
The new table is created as expected and the settings table updated as expected, too. What could be the reason for getting this: [Err] 21 - not an error
Is there any better suggested way to create the new schema?
I encounter this error as well. Later I figured it out. It's because another application was connected to the same db. So, my application can't modify the db -- create a table.
I created it successfully just by closing the another db connection.