I have column in my database that stores BLOB.
I want to run a query to check if specific byte array value is present in the table.
The value is b'\xf4\x8f\xc6{\xc2mH(\x97\x9c\x83hkE\x8b\x95' (python bytes).
I tried to run this query:
SELECT * from received_message
WHERE "EphemeralID"
LIKE HEX('\xf4\x8f\xc6{\xc2mH(\x97\x9c\x83hkE\x8b\x95');
But I get 0 results though I 100% sure that I store this value in the database.
Is there something wrong with my query?
Your search string is a bit weird-- you appear to have some complex things in there like { and (. Maybe you should search through the blob the way it is stored instead?
From the Sqlite documentation:
BLOB literals are string literals containing hexadecimal data and
preceded by a single "x" or "X" character. Example: X'53514C697465'
So maybe do a like with the ascii representation of the hex value you want? Maybe start with looking for just f48f or F48F if your sqlite stores it upper case.
Related
In a React Native App I'm attempting to insert data into a local sqlite db
let submissionID = "1-2-3";
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES("+submissionID+",'Test')");
(dbQuery is the name of a function I made to simplify my queries but the statement inside it should be the same)
If I viewed the Submissions table after this insert statement I would expect to see a row with [ID:"1-2-3",Data:"Test"] but instead I see [ID:"-4",Data:"Test"]
I created the table like so
CREATE TABLE IF NOT EXISTS Submissions(ID BLOB PRIMARY KEY NOT NULL, Data BLOB NOT NULL)
I used Blob because I read "The value is a blob of data, stored exactly as it was input." but I've also tried Text. I've also casted submissionID as a string like so
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES("+String(submissionID)+",'Test')");
But none of that worked. I do see here how sqlite takes advantage of arithmetic operators
https://www.w3resource.com/sqlite/arithmetic-operators.php
but I'm not sure how to stop it from doing so.
How would I get sqlite to treat my hyphens as hyphens instead of subtraction signs?
What you're doing is the equivalent of:
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES(1-2-3,'Test')");
passing the numeric expression 1-2-3 to the INSERT statement. The simplest fix is to quote the string literal.
let submissionID = "1-2-3";
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES('"+submissionID+"','Test')");
However, to guard against SQL injection attacks, you really ought to be using prepared statements instead of using string concatenation to build SQL statements.
Enclose the string in single quotes i.e.
this.dbQuery("INSERT INTO Submissions (ID, Data) VALUES('"+String(submissionID)+"','Test')");
Thus the value is treated as a literal by SQLite, without enclosing the value it will either be treated as a numeric value or as an identifier (column, table, trigger, view depending upon where it is coded and thus what the parser expects).
The data type (column affinity) has little bearing other than if you specified ID INTEGER PRIMARY KEY, then you could not store anything other than an integer. As ID INTEGER PRIMARY key has a special interpretation that is the column is an alias of the rowid.
I used Blob because I read "The value is a blob of data, stored
exactly as it was input." but I've also tried Text. I've also casted
submissionID as a string like so
That is only if the value to be inserted is a BLOB byte[] or in the case of raw SQL x'FF01FE02', otherwise SQLite will store the value according to how it interprets the type should be stored.
I'm trying to append zeroes to a blob field in a sqlite3 db.
What I tried is this:
UPDATE Logs
SET msg_buffer = msg_buffer || zeroblob(1)
WHERE msg_id = 'F0'
As you can see, the field name is "msg_buffer", and what I want is to append byte zero. It seems that the concat operator || doesn't work.
How could I achieve this?
Reading the doc link posted by Googie (3.2. Affinity Of Expressions), I managed to find the way:
UPDATE Logs
SET msg_buffer = CAST(msg_buffer || zeroblob(1) AS blob)
WHERE msg_id = 'F0'
The CAST operator can take the expression with the concatenate operator and force the blob affinity.
SQLite3 does support datatypes. See https://www.sqlite.org/datatype3.html
They are not strictly linked with declared type of a column, but rather individual per each cell value. The type is determined by how it was created/modified. For example if you insert 5, it will be INTEGER. If you insert 5.5, it will be REAL. If you insert 'test' it will be TEXT, if you insert zeroblob(1), it will be BLOB and if you insert null, it will be NULL.
Now, what you are doing is that you're trying to concatenate current value with a BLOB type. If current value is TEXT (or basically if you use || operator, as you do, you are converting any type into a TEXT), it will be concatenated with byte \x00, which actually determines the end of a string. In other words, you are adding yet another string terminator, to an already existing one, that the TEXT type has.
There will be no change on output of this operation. TEXT always ends with byte zero and it is always excluded from the result, as it's a meta character, not the value itself.
Additional information from http://sqlite.1065341.n5.nabble.com/Append-data-to-a-BLOB-field-td46003.html - appending binary data to BLOB field is not possible. You can modify prealocated blob:
Append is not possible. But if you preallocate space using
zeroblob() or similar, you can write to it using the incremental
blob API:
http://www.sqlite.org/c3ref/blob_open.html
Finally, please see accepted answer, as author of the question found an interesting solution.
I want to create a MVCCKey with a timestamp and pretty value I know. But I realize a roachpb.key is not very straightforward; is there some prefix/suffix involved? Is the database name is also encoded in roachpb.key?
Can anyone please tell me how a MVCCKey is formed? What information does it have? In the documentation, it just says that it looks like /table/primary/key/column.
An engine.MVCCKey combines a regular key with a timestamp. MVCCKeys are encoded into byte strings for use as RockDB keys (RocksDB is configured with a custom comparator so MVCCKeys are sorted correctly even though the timestamp uses a variable-width encoding).
Regular keys are byte strings of type roachpb.Key. For ordinary data records, the keys are constructed from table, column, and index IDs, along with the values of indexed columns. (The database ID is not included here; the database to which a table belongs can be found in the system.descriptors table)
The function keys.PrettyPrint can convert a roachpb.Key to a human-readable form.
How does sqlitebrowser determine that the data within a cell is a blob? For some data it will determine it to be a blob, but other data it will say it is not a blob. My example of this is a set of data where there are 10 doubles written out as a blob, yet it does not call it a blob. So I would really really like to know how it determines a cell to be a blob or not be a blob?
https://github.com/sqlitebrowser/sqlitebrowser/issues/16 might be the better answer. As it seems, there is or was code to determine what to display: for short blobs like the ones shown in the example (with less than 1024 bytes), the second condition holds: if there is a \0 character, the data is just displayed as "BLOB".
The SQLite C API has two different functions for binding a text value or a blob value to an INSERT statement, so only the writing application determines what the type of the value is.
The function sqlite3_column_type() returns the type of a column's value in the current row; this is used by the browser.
The same mechanisms are also directly availabe in SQL:
> SELECT typeof('hello'), typeof(x'1256AB');
text|blob
Is there a way to index blob fields and have the index used for beginning of blob searches?
Currently I have hashes stored as hexadecimal in text fields.
These hashes in hexadecimal form are 32 characters long, and form the bulk of the data in the database.
Problem is, they are often searched by their starting bytes, as in
select * from mytable where hash like '00a1b2%'
I would like to store them as blobs, as this saves about 30% of the database size. However while
select * from mytable where hex(hash) like '00a1b2%'
works, it's also much slower and does not seem to use the index.
Searching for exact blob matches does use the index, so the index is working.
Is there a way to perform a search on a blob start (with binary/memcmp "collation") that would use the index?
I also tried substr(), it's apparently faster than hex() but still not indexed
select * from mytable where substr(hash, 1, 6) = x'00a1b2'
To be able to use an index for LIKE, the table column must have TEXT affinity, and the index must be case insensitive:
CREATE TABLE mytable(... hash TEXT, ...);
CREATE INDEX hash_index ON mytable(hash COLLATE NOCASE);
Functions like hex or substr prevent usage of indexes.
Blobs can be indexed and compared like other types.
This allows you to express a prefix search with two comparisons:
SELECT * FROM mytable WHERE hash >= x'00a1b2' AND hash < x'00a1b3'