Related
I want to store and retrieve data from a DynamoDB table.
My data (an item = a review a user gave on a feature of an app) have the following attributes :
user string
feature string
appVersion string
timestamp string
rate int
description string
There is multiple features, on multiple versions of the app, and an user can give multiple reviews on these features. So I would like to use (user, appVersion, feature, timestamp) as a primary key.
But it does not seem to be possible to use that much attributes in a primary key in DynamoDB.
The first solution I implemented is to use user as a Partition Key, and a hash of (appVersion, feature, timestamp) as a Sort Key (in a new field named reviewID).
My problem is that, I want to retrieve an item for a given user, feature, appVersion without knowing the timestamp value (let's say I want the item with the latest timestamp, or the list of all items matching the 3 fields)
Without knowing the timestamp, I can't build the Sort Key necessary to retrieve my item. But if I remove the timestamp from the Sort Key, I will not be able to store multiple items having the same (user, appVersion, feature).
What would be the proper way to handle this usecase ?
I am thinking about using a hash of (user, appVersion, feature) as a Partition Key, and the timestamp as a Sort Key, would this be a correct solution ?
Put the timestamp at the end of your SK and then when you Query the data you use begins_with on the SK.
PK SK
UserID appVersion#feature#timestamp
This will allow you to dynamically query the data. For example you want all the users votes for a specific appversion
SELECT * FROM Mytable WHERE PK= 'x' AND SK BEGINS_WITH('{VERSION ID}')
This is done using a Query command.
The answer from Lee Hannigan will work, I like it.
However, keep in mind that accessing a PK is very fast because its hash-based.
I am thinking about using a hash of (user, appVersion, feature) as a
Partition Key, and the timestamp as a Sort Key, would this be a
correct solution?
This might also work, the table would look like this
PK SK
User#{User}AppVersion#{appVersion}#Feature#{feature} TimeStamp#{timestamp}
If you always know the user, appVersion, and the feature, this will be more optimal, because the SK lookup is O(logN)
one way
HASH string "modelName": "user"
RANGE string "id": "b0d5be50-4fae-11ed-981f-dbffcc56c88a"
uuid himself can be used for as timestamp
when searching you could search using reverse index
Another way
HASH string "modelName": "user"
RANGE string "createdAt" "2019-10-12T07:20:50.52Z"
createdAt, use time format rfc3339
when searching you could search using reverse index
Put down on paper what you need and you'll find others way to manage indes HASH/RANGE
In a React Native App I'm attempting to insert data into a local sqlite db
let submissionID = "1-2-3";
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES("+submissionID+",'Test')");
(dbQuery is the name of a function I made to simplify my queries but the statement inside it should be the same)
If I viewed the Submissions table after this insert statement I would expect to see a row with [ID:"1-2-3",Data:"Test"] but instead I see [ID:"-4",Data:"Test"]
I created the table like so
CREATE TABLE IF NOT EXISTS Submissions(ID BLOB PRIMARY KEY NOT NULL, Data BLOB NOT NULL)
I used Blob because I read "The value is a blob of data, stored exactly as it was input." but I've also tried Text. I've also casted submissionID as a string like so
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES("+String(submissionID)+",'Test')");
But none of that worked. I do see here how sqlite takes advantage of arithmetic operators
https://www.w3resource.com/sqlite/arithmetic-operators.php
but I'm not sure how to stop it from doing so.
How would I get sqlite to treat my hyphens as hyphens instead of subtraction signs?
What you're doing is the equivalent of:
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES(1-2-3,'Test')");
passing the numeric expression 1-2-3 to the INSERT statement. The simplest fix is to quote the string literal.
let submissionID = "1-2-3";
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES('"+submissionID+"','Test')");
However, to guard against SQL injection attacks, you really ought to be using prepared statements instead of using string concatenation to build SQL statements.
Enclose the string in single quotes i.e.
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES('"+String(submissionID)+"','Test')");
Thus the value is treated as a literal by SQLite, without enclosing the value it will either be treated as a numeric value or as an identifier (column, table, trigger, view depending upon where it is coded and thus what the parser expects).
The data type (column affinity) has little bearing other than if you specified ID INTEGER PRIMARY KEY, then you could not store anything other than an integer. As ID INTEGER PRIMARY key has a special interpretation that is the column is an alias of the rowid.
I used Blob because I read "The value is a blob of data, stored
exactly as it was input." but I've also tried Text. I've also casted
submissionID as a string like so
That is only if the value to be inserted is a BLOB byte[] or in the case of raw SQL x'FF01FE02', otherwise SQLite will store the value according to how it interprets the type should be stored.
There is a query that scans full table, there are 5million records, it costs about 60s. How to optimize this?
I have tried to use memory mode of sqlite, in theory this should be faster, since whole database is stored in memroy. However, it costs almost the same time.
table schema like this:
CREATE TABLE tbl0(estimateid int, seq int, field1 int NULL, field2 int NULL, field3 int NULL, field4 int NULL);
CREATE INDEX tbl0_idx on tbl0(estimateid);
CREATE TABLE tbl1(seq int, companyid int, field1 int NULL, field2 int NULL, field3 int NULL, field4 int NULL, field5 int NULL);
CREATE INDEX tbl1_idx on tbl1(seq);
CREATE TABLE tbl2(symbolid int, relatedcompanyid int, value char(64), field1 int NULL, field2 int NULL, field3 int NULL, field4 int NULL, field5 int NULL);
CREATE INDEX tbl2_idx on tbl2(relatedcompanyid);
and this is query, query that need join 3 tables:
>explain query plan select tbl0.estimateid, tbl1.seq, tbl1.companyid, tbl2.value from tbl0, tbl1, tbl2 where tbl0.seq = tbl1.seq and tbl1.companyid = tbl2.relatedcompanyid;
0|0|1|SCAN TABLE tbl1
0|1|2|SEARCH TABLE tbl2 USING INDEX tbl2_idx (relatedcompanyid=?)
0|2|0|SEARCH TABLE tbl0 USING AUTOMATIC COVERING INDEX (seq=?)
How to accelerate this query? Seems it's inevitable one table will be fully scanned. Each table contains about 5million records, this query costs very long time(several minutes).
When I put db in memory, use this #sqlite3 :memory:, it doesnot make any difference in speed.
Help is very appreciated.
A full index scan (type: index) is according to the documentation is the 2nd worst possible execution plan after a full table scan, which you chosen.
Full table scan is resource-intensive operation for DB, and there's no magic behind the scenes unless you boost your memory, CPU speed, will index the table, will reduce the number of records, etc. That's why you haven't noticed any drastic speed increase when you moved everything to the memory.
You should try to avoid this and to make the better query, or to optimize the DB and tables structures. Please, reference EXPLAIN QUERY PLAN and Query Planning for getting more details on the execution of your SQL and how it can be optimized.
It's hard to say more and to be more specific, as in your original question you haven't provided the DB structures, the characteristics of your data, your query, etc.
Your database is not in memory; you've done something wrong. I built a program to load 5 million records in another in-memory database system and it took less than 800 milliseconds for a full sequential scan. Even if SQLite is only half as fast as the in-memory database system I used, it should only take a second or two.
another possibility is you're writing to the console after fetching every row, or performing some other logic, that is causing the overall slowness.
In my programming task I've gone down a dark alley and wished I hadn't, but there is no turning back now.
I'm building up a SQL statement where the table name, column name and id value are retrieved from query string parameters i.e. ("SELECT [{0}] FROM [{1}] WHERE [Id] = {2};", c, t, id)
But it isn't as bad as it looks, I'm protected:
Only authenticated users (i.e. signed in users) can execute the Page_Load
I'm checking that both the table and the column exists beforehand
(using GetSchema etc.)
I'm checking that the Id is an integer beforehand
All my tables have Id columns
The database connection is reasonably secure
The field value is expected to be of type NVARCHAR(4000) or NVARCHAR(MAX) so I'm avoiding ExecuteScalar and I'm trying out LINQ ExecuteQuery because I like LINQ. But I'm a bit out of my depth again.
I've got this far:
Dim db As New MyDataContext
Dim result = db.ExecuteQuery(Of ITable)("SELECT [{0}] FROM [{1}] WHERE [Id] = {2};", c, t, id)
Is this the right way to go?
How do I get first row and first column value?
Is there a better alternative?
P.S. It's a SQL Server 2005 database
Any help appreciated.
Thanks.
SQL Server requires the tables ans columns to be statically known. You can't provide them using command parameters. You can't say
select * from #tableName
because the table name can't be a variable.
You need to build the SQL string with C# ensuring proper escaping of identifiers. Escaping works like this:
var escaped = "[" + rawUntrustedUserInput.Replace("]", "]]") + "]";
This is safe.
For example, Assume i have a table with employee list.
The employee id is the primary key, i have other details of employees such as name, age etc.
The employee can have multiple phone numbers. So, it becomes a multivalued attribute.
We don't know how many phones(contact numbers) the employees has.
Is it possible to have multivalued attribute in sqlite3?
Or is there is any method to accommodate this?
Thanks.
You can accomplish this with following ideas:
create 1 column to store all phone numbers (which makes querying based on numbers difficult, not recommended)
you can create x columns for each number (phone1, phone2, ...) (you need to decide what'll be the maximum number of phone numbers for each user)
You can create separate table for phone numbers and link this table with employee table with foreign key. This allows you to store varying list of phone numbers for each employee (requires new table).
Option 3 seems to be most flexible, however in most cases I've seen there was option 2 implemented (usually people have limited number of phone numbers).
You also could create a serializable custom class to store multiple contact numbers. You can then serialize the custom class into a byte[] and then extract an ASCII string from the byte[]. You'll be able to store this string of multiple contact numbers, but you will have to deserialize the data after reading it back in from the database ( reversing the process below.)
public class Contacts : ISerializable
{
}
if (Contacts is ISerializable)
{
string contactNumbers = String.Empty;
using (var stream = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, Contacts);
stream.Close();
var bytes = stream.GetBuffer();
contactNumbers = System.Text.Encoding.ASCII.GetString(bytes));
}
//Perform the update for this column or add to your insert query.
}
This is the wrong approach in relational database terms. Instead, you should create a separate table to hold employee phone numbers. The table will have columns employee_id and phone_number. Each employee may have 0 or more records in this table, depending on how many phone numbers you have on file.
You may also want to include a description column so that you can have information about what each phone number is.
Storing multiple values in a single column is a first-order no-no in relational databases. Please don't do it.
Multivalued fields are considered very bad practice in DB Design as it shows a lack of "normal form". Instead you should:
Create a new EmployeePhoneNumber table with 3 columns. EmployeePhoneNumberId, EmployeeId, PhoneNumber and PhoneType.
Add a constraint to restrict the Type column to be one of (Mobile, Home ..).
Now you can add any number of phone numbers, by type, to the new table. query it by employeeId and type, if needed.
This option is not only more flexible but it will save db storage size as you will only allocate storage for phone numbers that actually exist.