I have this code to insert dataframe into DB (SQLite) :
l1 = df.to_dict(orient='record')
meta = schema.MetaData(bind=db,reflect=True)
t= Table(t1, meta, autoload=True)
Session = sessionmaker(bind=db)
session = Session()
db.execute(t.insert(), l1)
session.commit()
session.close()
It fails because the unique id of the table 'is not unique' (df comes from the db and fields have been modified).
in df, there is one unique id called id, which needs to be incremented.
However, how to specify the unique_id and how to make it increment automatically through sql alchemy?
Solution found (if it helps anybody in same situation):
Just add this code, after creating the list, it would remove the id.
del l1['id']
Related
I am trying to delete some duplicate records from mysql table but it was not working.
I am taking help of https://www.javatpoint.com/mysql-delete-duplicate-records
If I will try with example database which is given example it was working fine.
But in my table it was not working.
DELETE S1 FROM employee_attendance AS S1 INNER JOIN employee_attendance AS S2 WHERE S1.DbKey < S2.DbKey AND S1.DivisionDbKey = S2.DivisionDbKey AND S1.Date = S2.Date AND S1.Month = S2.Month AND S1.FinancialYearDbKey = S2.FinancialYearDbKey AND S1.EmployeeDbKey = S2.EmployeeDbKey AND S1.Attendance = S2.Attendance AND S1.InTime = S2.InTime AND S1.OutTime = S2.OutTime AND S1.EmployeeDbKey = 6798 AND S1.Month = '05' AND S1.FinancialYearDbKey = 5;
I am getting error
#1205 - Lock wait timeout exceeded; try restarting transaction
I have tried with another example https://www.geeksforgeeks.org/sql-query-to-delete-duplicate-rows/
DELETE FROM employee_attendance WHERE DbKey NOT IN (SELECT MAX(DbKey) FROM employee_attendance WHERE EmployeeDbKey = 6798 AND Month = '05' AND FinancialYearDbKey = '5' GROUP BY DivisionDbKey,Date,Month,FinancialYearDbKey,EmployeeDbKey,Attendance,InTime,OutTime)
I am getting same error.
#1205 - Lock wait timeout exceeded; try restarting transaction
Any suggestion will be appriciated. Thank you.
I personally think this is a bad practice. You should instead make a (empty) duplicate of the table employee_attendance then define a UNIQUE KEY on that new table that will prevent duplicate entries.
Consider these steps:
Create a duplicate table:
CREATE TABLE employee_attendance_new LIKE employee_attendance;
Add UNIQUE INDEX - now, this is just a simple example. You can add or reduce columns to the unique index but make sure that you drop the existing unique index first then only you re-create:
ALTER TABLE employee_attendance_new
ADD UNIQUE INDEX unq_idx(EmployeeDbKey, date, InTime, OutTime);
Insert the data into the new table using INSERT IGNORE..:
INSERT IGNORE INTO employee_attendance_new
SELECT * FROM employee_attendance;
Check and compare both table. If you're satisfied with the result, rename those tables:
RENAME TABLE employee_attendance TO employee_attendance_old;
RENAME TABLE employee_attendance_new TO employee_attendance;
Now you have the new table with no duplicates and the old table for reference or in case there are some data you need from it.
Fiddle example
I need to merge some randomly uppercased data that has been collected in an SQLite table key_val, such that key is always lowercase and no vals are lost. There is a unique compound index on key,val.
The initial data looks like this:
key|val
abc|1
abc|5
aBc|1
aBc|5
aBc|3
aBc|2
AbC|1
abC|3
The result after the merge would be
key|val
abc|1
abc|2
abc|3
abc|5
In my programmer brain, I would
for each `key` with upper case letters;
if a lower cased `key` is found with the same value
then delete `key`
else update `key` to lower case
Re implementing the loop has a sub query for each row found with upper case letters, to check if the val already exists as a lower case key
If it does, I can delete the cased key.
From there I can UPDATE key = lower(key) as the "duplicates" have been removed.
The first cut of the programming method of finding the dupes is:
SELECT * FROM key_val as parent
WHERE parent.key != lower(parent.key)
AND 0 < (
SELECT count(s.val) FROM key_val as s
WHERE s.key = lower(parent.key) AND s.val = parent.val
)
ORDER BY parent.key DESC;
I'm assuming there's a better way to do this in SQLite? The ON CONFLICT functionality seems to me like it should be able to handle the dupe deletion on UPDATE but I'm not seeing it.
First delete all the duplicates:
DELETE FROM key_val AS k1
WHERE EXISTS (
SELECT 1
FROM key_val AS k2
WHERE LOWER(k2.key) = LOWER(k1.key) AND k2.val = k1.val AND k2.rowid < k1.rowid
);
by keeping only 1 combination of key and val with the min rowid.
It is not important if you kept the key with all lower chars or not, because the 2nd step is to update the table:
UPDATE key_val
SET key = LOWER(key);
See the demo.
Honestly it might just be easier to create a new table and then insert into it. As it seems you really just want a distinct select here, use:
INSERT INTO kev_val_new ("key", val)
SELECT DISTINCT LOWER("key"), val
FROM key_val;
Once you have populated the new table, you may drop the old one, and then rename the new one to the previous name:
DROP TABLE key_val;
ALTER TABLE key_val_new RENAME TO key_val;
I agree with #Tim that it would be easire to re-create table using simple select distict lower().. statement, but that's not always easy if table has dependant objects (indexes, triggers, views). In this case this can be done as sequence of two steps:
insert lowered keys which are not still there:
insert into t
select distinct lower(tr.key) as key, tr.val
from t as tr
left join t as ts on ts.key = lower(tr.key) and ts.val = tr.val
where ts.key is null;
now when we have all lowered keys - remove other keys:
delete from t where key <> lower(key);
See fiddle: http://sqlfiddle.com/#!5/84db50/11
However this method assumes that key is always populated (otherwise it would be a strange key)
If vals can be null then "ts.val = tr.val" should be replaced with more complex stuff like ifnull(ts.val, -1) = ifnull(tr.val, -1) where -1 is some unused value (can be different). If we can't assume any unused value like -1 then it should be more complex check for null / not null cases.
I am building a shiny application which will allow CRUD operations by a user on a table which exists in an sqlite3 database. I am using the input$table_rows_selected() function in DT to get the index of the rows selected by the user. I am then trying to delete the rows (using an action button deleteRows) from the database which have a matching timestamp (the epoch time stored as the primary key). The following code runs without any error but does not delete the selected rows.
observeEvent(input$deleteRows, {
if(!is.null(input$responsesTable_rows_selected)){
s=input$responsesTable_rows_selected
conn <- poolCheckout(pool)
lapply(length(s), function(i){
timestamp = rvsTL$data[s[i],8]
query <- glue::glue_sql("DELETE FROM TonnageListChartering
WHERE TonnageListChartering.timestamp = {timestamp}
", .con = conn)
dbExecute(conn, sqlInterpolate(ANSI(), query))
})
poolReturn(conn)
# Show a modal when the button is pressed
shinyalert("Success!", "The selected rows have been deleted. Refresh
the table by pressing F5", type = "success")
}
})
pool is a handler at the global level for connecting to the database.
pool <- pool::dbPool(drv = RSQLite::SQLite(),
dbname="data/compfleet.db")
Why does this not work? And if it did, is there any way of refreshing the datatable output without having to reload the application?
As pointed out by #RomanLustrik there was definitely something 'funky' going on with timestamp. I am not well versed with sqlite but running PRAGMA table_info(TonnageListChartering); revealed this:
0|vesselName||0||0
1|empStatus||0||0
2|openPort||0||0
3|openDate||0||0
4|source||0||0
5|comments||0||0
6|updatedBy||0||0
7|timestamp||0||1
8|VesselDetails||0||0
9|Name||0||0
10|VslType||0||0
11|Cubic||0||0
12|DWT||0||0
13|IceClass||0||0
14|IMO||0||0
15|Built||0||0
16|Owner||0||0
I guess none of the variables have a data type defined and I am not sure if that's possible to do it now. Anyway, I changed the query to ensure that the timestamp is in quotes.
query <- glue::glue_sql("DELETE FROM TonnageListChartering
WHERE TonnageListChartering.timestamp = '{timestamp}'
", .con = conn)
This deletes the user selected rows.
However, when I am left with only one row, I am unable to delete it. No idea why. Maybe because of a primary key that I have defined while creating the table?
So I've these two statements:
string insertUserData = "INSERT INTO W711_User_Data(Network_ID, F_Name, M_Name, L_NAME, Badge, Telephone, Org_Code, Org_Name, Req_Head_Network_ID)Values(#networkID1, #firstName1, #middleName1, #lastName1, #badgeNumber1, #telephone1, #orgCode1, #orgName1, #myUserName1)";
string insertReservationData = "INSERT INTO W711_Reservation_Data(ID, Network_ID, EventTitle, StartDate, EndDate, Justification) Values(null, #networkID2, #eventTitle1, #startDate1, #endDate1, #justification1)";
The network id in second string is foreign key relation with network id in first table.
The problem is: When I run the application in VS2010, it gives the error:
Can't insert explicit value for identity columnin table 'W711_Reservation_Data' when IDENTITY_INSERT is set to OFF.
Then I read to do this somewhere:
SET IDENTITY_INSERT W711_Reservation_Data ON
But it also fails and gives the same error again!
Please help
Thanks
p.s. sql server
if your id is an identity (aka auto generated from the database), just do not list the ID field in any place in the INSERT Statement, not as column name and not in the values list:
to get the ID generated by SQL Server you call SCOPE_IDENTITY in this way:
INSERT INTO W711_Reservation_Data(Network_ID, EventTitle, StartDate, EndDate, Justification) Values(#networkID2, #eventTitle1, #startDate1, #endDate1, #justification1)";
RETURN SCOPE_IDENTITY()
Edit: this is the second of your two statements, I have removed the ID and the NULL...
Why are you trying to insert null as the value for the ID column anyway? Assuming that this is the IDENTITY column that is the source of the complaint then it seems more likely you need to just leave it out of the column list and don't pass any explicit value in. i.e.
INSERT INTO W711_Reservation_Data
(Network_ID, EventTitle, StartDate, EndDate, Justification)
Values
(#networkID2, #eventTitle1, #startDate1, #endDate1, #justification1)
There might be two possibility.
1]
if Network_ID in first table is primary key auto generated then
insert data in first table.
then get latest network id from that table and
pass that network id with second query.
2].
If ID column in second table is primary key then
Do not pass null in second query.
either make auto generated or pass uniquer value in query.
I want to create a table in a SQLite database only if doesn't exist already. Is there any way to do this? I don't want to drop the table if it exists, only create it if it doesn't.
From http://www.sqlite.org/lang_createtable.html:
CREATE TABLE IF NOT EXISTS some_table (id INTEGER PRIMARY KEY AUTOINCREMENT, ...);
Am going to try and add value to this very good question and to build on #BrittonKerin's question in one of the comments under #David Wolever's fantastic answer. Wanted to share here because I had the same challenge as #BrittonKerin and I got something working (i.e. just want to run a piece of code only IF the table doesn't exist).
# for completeness lets do the routine thing of connections and cursors
conn = sqlite3.connect(db_file, timeout=1000)
cursor = conn.cursor()
# get the count of tables with the name
tablename = 'KABOOM'
cursor.execute("SELECT count(name) FROM sqlite_master WHERE type='table' AND name=? ", (tablename, ))
print(cursor.fetchone()) # this SHOULD BE in a tuple containing count(name) integer.
# check if the db has existing table named KABOOM
# if the count is 1, then table exists
if cursor.fetchone()[0] ==1 :
print('Table exists. I can do my custom stuff here now.... ')
pass
else:
# then table doesn't exist.
custRET = myCustFunc(foo,bar) # replace this with your custom logic