I have a table with a datetime column that was encrypted using Always Encrypted feature on SQL Server 2016.
Now I am trying to perform this simple select:
select max(dt_order)
from orders
where customer = 123;
I am running into this error:
Msg 33299, Level 16, State 2, Line 5
Encryption scheme mismatch for columns/variables 'dt_order'. The encryption scheme for the columns/variables is (encryption_type = 'DETERMINISTIC', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'myColHML', column_encryption_key_database_name = 'TESTING') and the expression near line '1' expects it to be (encryption_type = 'PLAINTEXT') (or weaker).
In SSMS I already have set connect options "Column Encryption Setting=Enabled" and also I've set query options "Enable Parametrization for Always Encrypted"
Any idea?
Since your data is encrypted using a key on the client side, SQL Server will not be able to calculate max. This is due to the fact that SQL Server does not have the key. The main value proposition of Always Encrypted is that it protects the data from Admins of SQL Server. Currently, the only operation possible on encrypted columns is equality.
From official documentation
Deterministic encryption always generates the same encrypted value for
any given plain text value. Using deterministic encryption allows
point lookups, equality joins, grouping and indexing on encrypted
columns. However, but may also allow unauthorized users to guess
information about encrypted values by examining patterns in the
encrypted column, especially if there is a small set of possible
encrypted values, such as True/False, or North/South/East/West region.
Deterministic encryption must use a column collation with a binary2
sort order for character columns.
Randomized encryption uses a method that encrypts data in a less
predictable manner. Randomized encryption is more secure, but prevents
searching, grouping, indexing, and joining on encrypted columns.
Related
If the user enters a wrong key for AES decryption, some garbage data is generated. I want to verify the given decryption key and throw an error if the key is incorrect. How can I verify the key entered by the user?
Use an HMAC. The basic premise is that you run the plaintext through an HMAC, add the result to the plaintext and then encrypt. Then do the opposite when decrypting. If the plaintext and HMAC result match, then you know you've got the correct key.
OR, if you want to know prior to decryption, use the key material provided by the user to derive two further keys (using, say PBKDF2). Use one for encryption and another for an HMAC. In this case, encrypt first and then apply the HMAC using the second key. This way you can compute the HMAC and check if it matches before you decrypt.
Simplest approach is to add a magic number to the plaintext file data in a predictable location before encrypting; when decrypting, if the magic number is wrong, you used the wrong key. Downside to this approach is that it cannot validate the integrity/authenticity of the entire message.
To do that, use AES in an authenticated mode (e.g. AES-GCM) which gives stronger guarantees that the rest of the message was not tampered with.
One common way used to verify if a key is correctly entered, without revealing the actual key, is by use of a KCV (Key Check Value). When you create the key you would at the same time calculate the KCV, when the key is then entered manually, you can verify the entry by re-calcuylating the KCV. This is eg. used when entering keys manually into HSM's from physical key letters.
To calculate a KCV for an AES key you encrypt an empty (0x00) block with the key and the first 3 bytes of the resulting encrypted block is then the KCV.
Take a look here
For example, if I create a dictionary in python I can use d.keys() to retrieve the keys.
What is a hash table/dictionary without this kind of access? Storage might be an issue and the keys may be of least importance.
Edit (clarification): I want a data structure that can access values through the key but doesn't know the key, only the hash. For example:
Hash Value
-----------------------------------------------------------------------
2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae|hey!
c9fc5d06292274fd98bcb57882657bf71de1eda4df902c519d915fc585b10190|hello!
If I try and access the data structure with the key "this is a key", it will hash that and get "hello!". If I try to access it with the key "foo", I will get "hey!".
We cannot retrieve the keys from this hash table, but we can access the data. This would be useful in cases where storage is important.
Normally, this would be the table:
Hash Value Key
-------------------------------------------------------------------------------------
2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae|hey! |foo
c9fc5d06292274fd98bcb57882657bf71de1eda4df902c519d915fc585b10190|hello!|this is a key
This is called a Set - in this case the value is the key, and implementations generally use the hashcode and equality operations on the items before adding them to the set.
Some implementations of Set can be sorted, generally those are referred to as SortedSet. Think of Set<T> as an equivalent to Dictionary<T,T> (and SortedSet<T> being approximate of SortedDictionary<T,T> in C# parlance.
Sorted variants are generally implemented using binary trees, whereas unsorted implementations use hashing tables. As the key is the value, most implementations only store the value itself.
Which platform / language are you using? Java?
I have to find out in MariaDb how to implement some features used in Oracle . I have :
Load a file: in Oracle I use the external table. Is there a way (fast and efficient one ) to load a file into a table . Has MariaDb a plugin which allows to load well a specific format of files?
In my existing Oracle code I used to developp a java wrap functions which allow those feature (is there a way in MariaDb to do this?), specifically :
1- Searching a files in an OS directory and insert them in a table,
2- send an SNMP trap
3- Send a mail via SMTP
Is there an equivalent to an Oracle job in Mariadb?
Is there an equivalent to Oracle TDE (Transparent data encryption) ?
Is there an equivalent to the VPD (virtual private policy)?
What is the maximum length of a varchar column/variable ? (in Oracle we can use the CLOBs..)
Many Thanks and Best Regards
MariaDB (and MySQL) can do a LOAD DATA on a CSV file. It is probably the most efficient way to convert external data to a table. (There is also ENGINE=CSV, which requires no conversion, but is limited in that it has no indexes, etc.)
MariaDB cannot, for security reasons, issue any arbitrary system calls. No emails, no 'exec', etc.
No Job, TDE, VPD.
Network transmissions can (optionally) use SSL for encryption at that level.
There is a family of virtually identical datatypes for characters:
CHAR(n), VARCHAR(n) -- where n is up to 65535; n is the limit of _characters_, not _bytes_.
TINYTEXT, TEXT, MEDIUMTEXT, LONGTEXT -- of various limits; the last is limited to 4GB.
For non-character storage (eg, images), there is a similar set of datatypes
BINARY(n), VARBINARY(n)
TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB
The various sizes of TEXT and BLOB indicate whether that is a 1-, 2-, 3-, or 4-byte length field in the implementation.
NVARCHAR is a synonym for VARCHAR. Character sets are handled by declaring a column to be, for example, CHARACTER SET utf8 COLLATE utf8_unicode_ci. Such can be defaulted at the database (schema) level, defaulted at the table level, or specified differently for different columns (even in the same table).
I'm a bit confused by how to properly set up secondary indexes in DynamoDB.
the documentation states secondary indexes are for tables which have a hash and rangekey, but in my case, I have no need of the range key.
The scenario is basically like this. I have a list of mobile clients which will call into my API. those clients are identified by a 6 character unique client ID. Each client also has a unique device ID, which is basically a long GUID -- quite long and inconvenient to use as the primary key.
The question comes when a client registers itself it sends is device ID (the long GUID) in a registration request and the server generates the unique clientID (the six char unique ID) which it returns to the client for future communication. One of the checks that the server side must do is make sure the request is not a duplicate registration, i.e. that the deviceID is not already present in the table under another client ID.
In a SQL table, I would have the clientID be the primary key, and 'd just define the a unique index on the deviceID field, but it seems like I can't do that in DynamoDB, since I only have a hash key on the table, not a hash and range key. I could do a query to find out if there's a dupe deviceID somewhere but that would seem to require a table scan which I'd like to avoid.
What's the proper way to set up something like this in DynamoDB? Do I just use a dummy range key like "foo" on all my rows and use a local secondary index? Seems inefficient somehow.
I personally don't like to use indexes.
What I recommend is to keep two tables.
DEVICES
Hash: device_id
attribute: client_id
CLIENT_DEVICES
Hash: client_id
Range: device_id
This allows you to reason about whether a client has devices, which devices, as well as ask for a device if it attached to a client.
This IMO is more readable than global/local secondary indexes.
I am working on a project that requires key/value caching function but the application will exist in a very limited environment that does not support any of the go-to industry standard memory caching methods such as ASP.NET Cache, memcached, AppFabric.
The only option we have in this restrictive environment is a MS SQL database. We have to create a simple key/value table to meet our key/value caching needs. We will most likely serialize the data as JSON but I am not sure what would be the best data type for the key. It will obviously need to be a unique key and need to be readable by the programmer getting and setting the cache. It also needs to be a fast look up since we will already be loosing performance not having access to an "in memory" cache solution.
I am use to having my primary key column be an int or bigint value. In this case should the primary key (the cache key) be a char or varchar data type as all queries will be:
SELECT value FROM CacheTable WHERE key = 'keyname'
I also saw posts about using an md5 hash but other posts pointed out that hashing cannot be relied on to produce unique keys all the time. I'm basically after some advice on the data type and rather or not the 'key' column should be the primary key or if I should still create an int or bigint primary key (even though it probably will not be used).
The end result we are after is creating a caching class similar to .NET's native caching where we can create a static class that pulls from the database table such as:
CustomDatabaseCache.Set(string key, object value);
CustomDatabaseCache.Get(string key)
I think in your scenario having a clustered primary key on your keyname column would work fine. However, it's worth experimenting with fill factors, because you want a fill factor that is low enough that you don't cause excessive page splits, but one that is high enough to keep the number of page reads low.
A clustered IDENTITY index works better in terms of eliminating page splits on the clustered index - and you could use a unique index on keyname which used an INCLUDE clause to include your values. However - in your case, I don't see the benefit in doing that because you'd have exactly the same page-split problem on your unique index, and the clustered index on keyname would be no more expensive to read because you wouldn't have any extra columns. Plus you would then have index update cost on two indexes for write.
Hope that helps.