Delete only one column from the Target table even if there are multiple similar columns (Teradata) - teradata

I came across a situation where i have to delete a column from a table based on a condition from other table
Let me break it down to you!
There is a master table called MORTALITY (containing info regarding deceased individuals)
And another table called INC_MORTALITY (incremental mortality) table which is refreshed on a weekly basis
Note: Both the tables have similar format
So this week’s new records, containing both additional deceased individuals as well as updates of old data for previously delivered records. This is a single file with a column (OP_DIRECTIVE) specifying if it is an “add” or “delete” record.
Processing Weekly Files
To incorporate the weekly update file, we need to execute the following steps in order.
1. Delete rows in the master table which have a OP_DIRECTIVE = 'D' as the operation in the weekly update. For a given delete row, you should delete a single row in the master table which matches the delete record on all fields aside from the “D” operation column. Warning: please ensure you only delete, or mark as deleted, one record, even if more than one historical record fully matches this new delete record.
2. Add rows in the master table which appear in the “Add” file.
Upon completion of these steps, your master table should be the most up to date master of deaths.

(Note: THESE TABLES DOES NOT HAVE PRIMARY KEYS)
SO WHAT I TRIED:
DEL FROM MORTALITY MI
WHERE MI.DATA_SOURCE = INC_MORTALITY.DATA_SOURCE
AND MI.DD_IMP_FLAG = INC_MORTALITY.DD_IMP_FLAG
AND MI.DOB = INC_MORTALITY.DOB
AND MI.DOD = INC_MORTALITY.DOD
AND MI.DEATH_VERIFICATION = INC_MORTALITY.DEATH_VERIFICATION
AND MI.GENDER_PROBABILITY = INC_MORTALITY.GENDER_PROBABILITY
AND MI.GENDER = INC_MORTALITY.GENDER
AND MI.TOKEN_1 = INC_MORTALITY.TOKEN_1
AND MI.TOKEN_2 = INC_MORTALITY.TOKEN_2
AND MI.TOKEN_4 = INC_MORTALITY.TOKEN_4
AND MI.TOKEN_5 = INC_MORTALITY.TOKEN_5
AND MI.TOKEN_7 = INC_MORTALITY.TOKEN_7
AND MI.TOKEN_16 = INC_MORTALITY.TOKEN_16
AND MI.TOKEN_KEY = INC_MORTALITY.TOKEN_KEY
AND INC_MORTALITY.OP_DIRECTIVE = 'D'
The above Delete statement will delete all the rows satisfying the conditions, my requirement is to delete only one record even if more than one historical record fully matches this new delete record,
and if i include ROW NUMBER() stmt like below my DELETE stmt is not working
QUALIFY ROW_NUMBER() OVER (PARTITION BY MI.DATA_SOURCE,MI.DOB,MI.DOD
ORDER BY MI.DOD DESC ) = 1
Any suggestions on how to approach this scenario, Thanks!!

Approach to solution: Copy unmatched rows to a work table, then truncate the original table and replace with contents of the work table. One way to identify unmatched rows would be to tag each of the input rows in a set of duplicates with a unique number, something like this:
INSERT work_table SELECT MI.col1, MI.col2, ...
FROM
(SELECT M.*,
ROW_NUMBER() OVER (PARTITION BY <join cols> ORDER BY <some col(s)>) AS ROWNUM
FROM MORTALITY M) MI
LEFT JOIN
(SELECT I.*,
ROW_NUMBER() OVER (PARTITION BY <join cols> ORDER BY <some col(s)>) AS ROWNUM
FROM INC_MORTALITY I
WHERE OP_DIRECTIVE='D') INC
ON MI.join_col1 = INC.join_col1
AND MI.join_col2 = INC.join_col2
...
AND MI.ROWNUM = INC.ROWNUM
WHERE INC.ROWNUM IS NULL /* "anti-join" keeps only unmatched rows */
;
DELETE FROM MORTALITY;
INSERT MORTALITY SELECT * FROM work_table;
If INC_MORTALILTY never has duplicates, then you can eliminate numbering that relation and change the last join condition to MI.ROWNUM = 1 and use one of the other JOIN columns for the NULL check.

Related

Remove duplicate records from mysql table

I am trying to delete some duplicate records from mysql table but it was not working.
I am taking help of https://www.javatpoint.com/mysql-delete-duplicate-records
If I will try with example database which is given example it was working fine.
But in my table it was not working.
DELETE S1 FROM employee_attendance AS S1 INNER JOIN employee_attendance AS S2 WHERE S1.DbKey < S2.DbKey AND S1.DivisionDbKey = S2.DivisionDbKey AND S1.Date = S2.Date AND S1.Month = S2.Month AND S1.FinancialYearDbKey = S2.FinancialYearDbKey AND S1.EmployeeDbKey = S2.EmployeeDbKey AND S1.Attendance = S2.Attendance AND S1.InTime = S2.InTime AND S1.OutTime = S2.OutTime AND S1.EmployeeDbKey = 6798 AND S1.Month = '05' AND S1.FinancialYearDbKey = 5;
I am getting error
#1205 - Lock wait timeout exceeded; try restarting transaction
I have tried with another example https://www.geeksforgeeks.org/sql-query-to-delete-duplicate-rows/
DELETE FROM employee_attendance WHERE DbKey NOT IN (SELECT MAX(DbKey) FROM employee_attendance WHERE EmployeeDbKey = 6798 AND Month = '05' AND FinancialYearDbKey = '5' GROUP BY DivisionDbKey,Date,Month,FinancialYearDbKey,EmployeeDbKey,Attendance,InTime,OutTime)
I am getting same error.
#1205 - Lock wait timeout exceeded; try restarting transaction
Any suggestion will be appriciated. Thank you.
I personally think this is a bad practice. You should instead make a (empty) duplicate of the table employee_attendance then define a UNIQUE KEY on that new table that will prevent duplicate entries.
Consider these steps:
Create a duplicate table:
CREATE TABLE employee_attendance_new LIKE employee_attendance;
Add UNIQUE INDEX - now, this is just a simple example. You can add or reduce columns to the unique index but make sure that you drop the existing unique index first then only you re-create:
ALTER TABLE employee_attendance_new
ADD UNIQUE INDEX unq_idx(EmployeeDbKey, date, InTime, OutTime);
Insert the data into the new table using INSERT IGNORE..:
INSERT IGNORE INTO employee_attendance_new
SELECT * FROM employee_attendance;
Check and compare both table. If you're satisfied with the result, rename those tables:
RENAME TABLE employee_attendance TO employee_attendance_old;
RENAME TABLE employee_attendance_new TO employee_attendance;
Now you have the new table with no duplicates and the old table for reference or in case there are some data you need from it.
Fiddle example

sqlite shift rowid in multiple records

Hello i have an sqlite db with many records like 10540 record they are ordered by creation time , i want to shift like a record in the middle and like to do it automatically
for example :
select * from table1 where id >= 8521;
UPDATE Table1 SET id = id +1 ;
does not work i get Error: Result: UNIQUE constraint failed:
so i want to shift up all records from 8521 to the last record and get place in the 8520 place for example so i can insert my record in that place of table .
even the
id = select max(id)+1
does not work how can i increment the id from last record to the needed record so i can put a place in the records db
A simple update statement would fail, as it would try to create duplicate values in the primary key.
What you can do is this:
First update the column to the negatives of the values they should have:
update table1
set id = -(id + 1)
where id > 8520;
Now there are no duplicates and you just need to update again to the positive values:
update table1
set id = -id
where id < 0;
This will do the trick, but any kind of updating the primary key is not a recommended practice

Update row with value from next row sqlite

I have the following columns in a SQLite DB.
id,ts,origin,product,bid,ask,nextts
1,2016-10-18 20:20:54.733,SourceA,Dow,1.09812,1.0982,
2,2016-10-18 20:20:55.093,SourceB,Oil,7010.5,7011.5,
3,2016-10-18 20:20:55.149,SourceA,Dow,18159.0,18161.0,
How can I populate the 'next timestamp' column (nextts) with the next timestamp for the same product (ts), from the same source? I've been trying the following, but I can't seem to put a subquery in an UPDATE statement.
UPDATE TEST a SET nextts = (select ts
from TEST b
where b.id> a.id and a.origin = b.origin and a.product = b.product
order by id asc limit 1);
If I call this, I can display it, but I haven't found a way of updating the value yet.
select a.*,
(select ts
from TEST b
where b.id> a.id and a.origin = b.origin and a.product = b.product
order by id asc limit 1) as nextts
from TEST a
order by origin, a.id;
The problem is that you're using table alias for table in UPDATE statement, which is not allowed. You can skip alias from there and use unaliased (but table-name prefixed) reference to its columns (while keeping aliased references for the SELECT), like this:
UPDATE TEST
SET nextts = (
SELECT b.ts
FROM TEST b
WHERE b.id > TEST.id AND
TEST.origin = b.origin AND
TEST.product = b.product
ORDER BY b.id ASC
LIMIT 1
);
Prefixing unaliased column references with the table name is necessary for SQLite to identify that you're referencing to unaliased table. Otherwise the id column whould be understood as the id from the closest[*] possible data source, in which case it's the aliased table (as b alias), while we're interested in the unaliased table, therefore we need to explicitly tell SQLite that.
[*] Closest data source is the one listed in the same query, or parent query, or parent's parent query, etc. SQLite is looking for the first data source (going from inner part to the outside) in the query hierarchy that defines this column.

SQL update based on column in other table

Using SQLite, I am trying to update three columns based on another table (two columns)
The three columns are (Table1):
'AgentCreatedID'
'AgentOwnedID'
'AgentSentID'
The other table (Table2) consists of 'AgentID' and 'Designation'.
If the ID in one of the three columns matches the 'AgentID' in the second table, I want the 'Designation' value to populate. This table is a list of ALL unique IDs and the corresponding designation. Each row of data has a Creator, Owner, and Sender. I need to see what designation that person is from.
In Access, this would look something like this for the first value. I would also need to add the other two values.
UPDATE Table1
LEFT JOIN Table2 ON Table1.AgentCreatedID = Table2.AgentID
SET raw.AgentCreatedID = [ Table2 ]![ Designation];
I am not sure what that ! command is or how it could be used in SQLite.
SQLite does not suport joins in an UPDATE statement.
You have to look up the new value with correlated subqueries:
UPDATE Table1
SET AgentCreatedID = (SELECT Designation
FROM Table2
WHERE AgentID = AgentCreatedID),
AgentOwnedID = (SELECT Designation
FROM Table2
WHERE AgentID = AgentOwnedID),
AgentSentID = (SELECT Designation
FROM Table2
WHERE AgentID = AgentSentID)
The exclamation mark is used to separate the worksheet name from the reference in that worksheet. Here is Microsoft's explanation of cell references.
Now that you know what [ Table2 ]![Designatio] means, you can simplify it to use only the column name.

MySQL Changing Order Depending On Contents of a Column

I have a MySQL table Page with 2 columns: PageID and OrderByMethod.
I also then have a Data table with lots of columns including PageID (the Page the data is on), DataName, and DataDate.
I want OrderByMethod to have one of three entries: Most Recent Data First, Most Recent Data Last, and Alphabetically.
Is there a way for me to tack an "ORDER BY" clause to the end of this query that will vary its ordering method based on the contents of the "OrderByMethod" column? For example, in this query, I would want to have the ORDER BY clause contain whatever ordering rule is stored in Page 1's OrderByMethod column.
GET * FROM `Data` WHERE `Data`.`PageID`=1 ORDER BY xxxxxx;
Maybe a SELECT clause in the ORDER BY clause? I'm not sure how that would work though.
Thanks!
select Data.*
from Data
inner join Page on (Data.PageID=Page.PageID)
where Data.PageID=1
order by
if(Page.OrderByMethod='Most Recent Data First', now()-DataDate,
if(Page.OrderByMethod='Most Recent Data Last', DataDate-now(), DataName)
);
You can probably do this with the IF syntax to generate a column that you can then order by.
SELECT *, IF(Page.OrderBy = 'Alphabetically', Data.DataName, IF(Page.OrderBy = 'Most Recent Data First', NOW() - Data.DataDate, Data.DataDate - NOW())) AS OrderColumn
FROM Data
INNER JOIN Page ON Data.PageID = Page.PageID
WHERE Page.PageID = 1
ORDER BY OrderColumn
The direction of the ordering is determined in the calculation of the data instead of specifying a direction in the ORDER BY
Can you just append the order by clause to the select statement and rebind the table on postback?
If you want to use the content of the column in Page table as an expression in ORDER BY you have to do it using prepared statements. Let say, you store in OrderByMethod something like "field1 DESC, field2 ASC" and you want this string to be used as it is:
SET #order_by =(SELECT OrderByMethod FROM Page WHERE id = [value]);
SET #qr = CONCAT(your original query,' ORDER BY ', #order_by);
PREPARE stmt FROM #qr;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
If you want the result set to be sorted based on the value of OrderByMethod , you can use IF as it was already mentioned by others, or CASE :
...
ORDER BY
CASE OrderByMethod
WHEN 'val1' THEN field_name1
WHEN 'val2' THEN field_name2
....etc
END

Resources