I have below table on Oracle19c(I am an oracle newbie). 4 million rows are inserted into the table daily and for now this table have 40 column and 240 million rows.
I usually search the table with user_id and MyTimestamp columns filter query and it takes 10 minutes to return the answer.
Example:
select * from table where user_id=123581 and MyTimestamp between 1657640396 and 1657777396
Note: Duplicate values are stored in the user_id and MyTimestamp columns.
I want partition monthly on MyTimestamp and index on user_id but which global or local indexes is suitable for indexing and how do I do it?
----------------------------------------------------------------------------------------------------
| id | MyTimestamp | Name | user_id ...
----------------------------------------------------------------------------------------------------
| 0 | 1657640396 | John | 123581 ...
| 1 | 1657638832 | Tom | 168525 ...
| 2 | 1657640265 | Tom | 168525 ...
| 3 | 1657640292 | John | 123581 ...
| 4 | 1657640005 | Jack | 896545 ...
--------------------------------------------------------------------------------------------------
If the majority of your queries contain the partition key, then better create LOCAL indexes:
CREATE INDEX index_name ON table_name (MyTimestamp, user_id) LOCAL;
Local indexes are smaller (i.e. the index partition) and thus faster and you don't have to rebuild the index when you drop an outdated partition.
Related
I have a table with a large amount of data; moving forward, I would like to enforce uniqueness for a given column in this table. However, the table contains a large amount of rows where that column is non-unique. I am not able to delete or alter these rows.
Is it possible to enforce uniqueness over a given date range, or since a specific date, or based on the value of another column (or something else like that) in MariaDB?
You can create a UNIQUE index on multiple columns, where one column is nullable. MariaDB will see each column with NULL values as a different value regarding the UNIQUE index, even if the other column values of the UNIQUE index are the same. Check the MariaDB documentation Getting Started with Indexes - Unique Index:
The fact that a UNIQUE constraint can be NULL is often overlooked. In SQL any NULL is never equal to anything, not even to another NULL. Consequently, a UNIQUE constraint will not prevent one from storing duplicate rows if they contain null values:
CREATE TABLE t1 (a INT NOT NULL, b INT, UNIQUE (a,b));
INSERT INTO t1 values (3,NULL), (3, NULL);
SELECT * FROM t1;
+---+------+
| a | b |
+---+------+
| 1 | 1 |
| 2 | 1 |
| 2 | 2 |
| 3 | NULL |
| 3 | NULL |
+---+------+
You can create such a UNIQUE index on the date column you already have and a new column which indicates if the date value should be unique or not:
CREATE TABLE Foobar(
id INT AUTO_INCREMENT PRIMARY KEY NOT NULL,
createdAt DATE NOT NULL,
dateUniqueMarker BIT NULL DEFAULT 0,
UNIQUE KEY uq_createdAt(createdAt, dateUniqueMarker)
);
INSERT INTO Foobar(createdAt) VALUES ('2021-11-04'),('2021-11-05'),('2021-11-06');
SELECT * FROM Foobar;
+----+------------+------------------------------------+
| id | createdAt | dateUniqueMarker |
+----+------------+------------------------------------+
| 1 | 2021-11-04 | 0x00 |
| 2 | 2021-11-05 | 0x00 |
| 3 | 2021-11-06 | 0x00 |
+----+------------+------------------------------------+
INSERT INTO Foobar(createdAt) VALUES ('2021-11-05');
ERROR 1062 (23000): Duplicate entry '2021-11-05-\x00' for key 'Foobar.uq_createdAt'
UPDATE Foobar SET dateUniqueMarker = NULL WHERE createdAt = '2021-11-05';
INSERT INTO Foobar(createdAt, dateUniqueMarker) VALUES ('2021-11-05', NULL);
SELECT * FROM Foobar;
+----+------------+------------------------------------+
| id | createdAt | dateUniqueMarker |
+----+------------+------------------------------------+
| 1 | 2021-11-04 | 0x00 |
| 2 | 2021-11-05 | NULL |
| 5 | 2021-11-05 | NULL |
| 3 | 2021-11-06 | 0x00 |
+----+------------+------------------------------------+
Without any data example and scenario illustration, it's hard to know. If you can update your question with those information, please do.
"Is it possible to enforce uniqueness over a given date range, or since a specific date, or based on the value of another column (or something else like that) in MariaDB?"
If by "enforce" you mean to create a new column then populate it with unique identifier, then yes it is possible. If what you really mean is to generate a unique value based on other column, that's also possible. Question is, how unique do you want it to be?
Is it like this unique?
column1
column2
column3
unique_val
2021-02-02
ABC
DEF
1
2021-02-02
CBD
FEA
1
2021-02-03
BED
GER
2
2021-02-04
ART
TOY
3
2021-02-04
ZSE
KSL
3
Whereby if it's the same date (on column1), it should have the same unique value regardless of column2 & column3 data.
Or like this?
column1
column2
column3
unique_val
2021-02-02
ABC
DEF
1
2021-02-02
CBD
FEA
2
2021-02-03
BED
GER
3
2021-02-04
ART
TOY
4
2021-02-04
ZSE
KSL
5
Taking all (or certain) columns to consider the unique value.
Both of the scenario above can be achieved in query without the need to alter the table, adding and populate a new column but of course, the latter is also possible.
I'm trying to figure out how to implement a reverse lookup GSI in DyamoDB. I attended an amazing talk about DynamoDB at reInvent this year (https://youtu.be/HaEPXoXVf2k?t=2674). Around 44 minutes into the talk the idea of a Reverse Lookup GSI is presented. I can't figure out how to implement this in Dynamo.
I want to add a single GSI to do a reverse lookup.
My current Scheme looks like:
I would like to be able to query on just the CXSK. I'm planning on overloading the CXSK and would love to be able to do a query with a begins with for that key.
I'm not sure what I'm missing when I go to create the GSI. I'm not sure what should go in the following fields. I'm also curious if it makes sense to have an overloaded Sort Key.
Let's say this is your original table
| pk | sk | prop1 | prop2 | ...
| a | b | xyz | abc
| a | c | lmn | opq
| b | x | rst | lme
| b | b | tuv | opq
in the above table you can do queries like
select * where pk = a It will return row 1 and 2
select * where pk = a and sk = b it will return row 1
Now to do reverse lookup mean you want to aggregate data by some other field name.
Let's say we want to do it by sk. To do this we will create a GSI with sk as partitionKey and pk as SortKey. And this view of table will look like
This will be your GSI1 table
| pk | sk | prop1 | prop2 | ...
| b | a | xyz | abc
| c | a | lmn | opq
| x | b | rst | lme
| b | b | tuv | opq
in the above table you can do queries like
select * where pk = b It will return row 1 and 4
select * where pk = b and sk = a it will return row 1
Considering the above description, in your case you should create GSI with pk as CXSK and sk as USERId
Given a SQLite db.
a table with primary key:
create table t1 (id int not null, CONSTRAINT pk_id PRIMARY KEY (id));
Now query info for it:
PRAGMA TABLE_INFO(t1);
returns:
| cid | name | type | notnull | dflt_value | pk |
| --- | ---- | ---- | ------- | -----------| -- |
| 0 | id | int | 1 | <null> | 1 |
PRAGMA index_list(t1);
returns:
| seq | name | unique | origin | partial |
| --- | ----------------------| ------ | ------ | ------- |
| 0 | sqlite_autoindex_t1_1 | 1 | pk | 0 |
As we can see index_list returns info about the PK but it reports incorrect name ("sqlite_autoindex_t1_1" instead of "pk_t1").
The same problem with UNIQUE constraints. They are created with autogenerated names.
Is it possible to extract real PRIMARY KEY/UNIQUE CONSTRAINT name?
P.S. I can see that JetBrains's DataGrip correctly show PK names in database browser. But sqliteadmin for example shows them with name like sqlite_autoindex_t1_1. For unique constraints even DataGrip doesn't show correct names (actually it doesn't show them at all).
The index and the constraint are different objects.
SQLite has no mechanism to retrieve the constraint name. You'd have to parse the SQL.
I have this database in sqlite (table1):
+-----+-------+-------+
| _id | name | level |
+-----+-------+-------+
| 1 | Mike | 3 |
| 2 | John | 2 |
| 3 | Bob | 2 |
| 4 | David | 1 |
| 5 | Tom | 2 |
+-----+-------+-------+
I want to create a view with all elements of level 2 and then to add a new column indicating the order of the row in the new table. That is, I would want this result:
+-------+------+
| index | name |
+-------+------+
| 1 | John |
| 2 | Bob |
| 3 | Tom |
+-------+------+
I have tried:
CREATE VIEW words AS SELECT _id as index, name FROM table1;
But then I get:
+-------+------+
| index | name |
+-------+------+
| 2 | John |
| 3 | Bob |
| 5 | Tom |
+-------+------+
I suppose it should be something as:
CREATE VIEW words AS SELECT XXXX as index, name FROM table 1;
What should I use instead of XXXX?
When ordered by _id, the number of rows up to and including this one is the same as the number of rows where the _id value is less than or equal to this row's _id:
CREATE VIEW words AS
SELECT (SELECT COUNT(*)
FROM table1 b
WHERE level = 2
AND b._id <= a._id) AS "index",
name
FROM table1 a
WHERE level = 2;
(The computation itself does not actually require ORDER BY _id because the order of the rows does not matter when we're just counting them.)
Please note that words is not guaranteed to be sorted; add ORDER BY "index" if needed.
And this is, of course, not very efficient.
You have two options. First, you could simply add a new column with the following:
ALTER TABLE {tableName} ADD COLUMN COLNew {type};
Second, and more complicatedly, but would actually put the column where you want it, would be to rename the table:
ALTER TABLE {tableName} RENAME TO TempOldTable;
Then create the new table with the missing column:
CREATE TABLE {tableName} (name TEXT, COLNew {type} DEFAULT {defaultValue}, qty INTEGER, rate REAL);
And populate it with the old data:
INSERT INTO {tableName} (name, qty, rate) SELECT name, qty, rate FROM TempOldTable;
Then delete the old table:
DROP TABLE TempOldTable;
I'd much prefer the second option, as it will allow you to completely rename everything if need be.
I need to pull a row in a select statement from a SQL database if a certain value in a table is changed.
For example, I have a column called price in a Price table. If the user changes the value for price (through an asp.net app), I want to select that entire row. This is going to be done in a workflow and an email is sent to the user that the row that was changed AFTER it was changed.
Does this make sense? Can someone point me in the right direction of a procedure or function to use? Thanks.
You could use an SQL trigger to accomplish this.
There is a tutorial (using Price as you described) that shows how to accomplish this here: http://benreichelt.net/blog/2005/12/13/making-a-trigger-fire-on-column-change/
well, in order to update a row, you'll have to update that row "WHERE uniqueID = [someid]". Can't you simply run a select immediately after that? (SELECT * FROM [table] WHERE uniquueID = [someid])
Without knowing what your data looks like (or what database this is, it's a little difficult) but assuming you have a history table with a date and an ID that stays the same like this...
+----+-------+------------+
| ID | PRICE | CHNG_DATE |
+----+-------+------------+
| 1 | 2.5 | 2001-01-01 |
| 1 | 42 | 2001-01-01 |
| 2 | 4 | 2001-01-01 |
| 2 | 4 | 2001-01-01 |
| 3 | 4 | 2001-01-01 |
| 3 | 3 | 2001-01-01 |
| 3 | 2 | 2001-01-01 |
+----+-------+------------+
and your database supports With and Row_number You could write the following
WITH data
AS (SELECT id,
price,
chng_date,
Row_number()
OVER (
partition BY id
ORDER BY chng_date) rn
FROM price)
SELECT data.id,
data.price new,
data_prv.price old,
data.chng_date
FROM data
INNER JOIN data data_prv
ON data.id = data_prv.id
AND data.rn = data_prv.rn + 1
WHERE data.price <> data_prv.price
That would produce this
+----+-----+-----+------------+
| ID | NEW | OLD | CHNG_DATE |
+----+-----+-----+------------+
| 1 | 42 | 2.5 | 2001-01-01 |
| 3 | 3 | 4 | 2001-01-01 |
| 3 | 2 | 3 | 2001-01-01 |
+----+-----+-----+------------+
Demo
If your Database supports LAG() its even eaiser
WITH data
AS (SELECT id,
price new,
chng_date,
Lag(price)
OVER (
partition BY id
ORDER BY chng_date) old
FROM price)
SELECT id,
new,
old,
chng_date
FROM data
WHERE new <> old
Demo