What I do
I am reading the CSV file from the Azure storage account and copying the same data to the Azure SQL database in a table. The CSV file contains PII information such as first name, last name, phone and email. The column mapping with the sink database is one to one.
Want solution for:
No issue, all data is properly copied. On top of that I want to encrypt the PII information. After encryption I will see only the encrypted value in the database.
I tried:
I converted the PII value to hashbyte (SHA2_256) but I did not get the actual value back so it is not my solution
NOTE: I am using ADF V2
In advance, thank you for your time. Any of your input will be valuable to me
There are multiple types ways to handle pii in Azure SQL database like encryption or dynamic data masking.
Is there any specific requirement why you want to encrypt the data through ADF?
You can use dataflows within ADF to do the necessy encryption or needed
Related
I want to query encrypted data from my SQLite database.
For each row, I'm using XOR operation on every value, convert it toBase64 and then INSERT it in the database.
Now I need to find a way to SELECT the encrypted values.
i.e:
SELECT *
FROM table
WHERE name_column BETWEEN 'value1' AND 'value2'
Considering the huge information in my database, how can I do that without having to decrypt all the table to get the wanted rows?
It's impossible. You are using BETWEEN 'value1' AND 'value2'. The database can only see the XORed strings and BETWEEN will not work as expected. Even if you find a way to decrypt the strings on-the-fly with SQLITE (remember XOR calling again will decrypt) it's not very efficient and resource consuming when there are thousand of entries.
So in order to continue with your problem you could have a look at this extension list. SQLITE seems to provide some very basic encryption modules, which can XOR the whole database with a key you defined. (not recommended)
This file describes the SQLite Encryption Extension (SEE) for SQLite.
The SEE allows SQLite to read and write encrypted database files. All
database content, including the metadata, is encrypted so that to an
outside observer the database appears to be white noise.
This file contains the complete source code to the SEE variant that
does weak XOR encryption. Do not take this file seriously. It is for
demonstration purposes only. XOR encryption is so weak that it hardly
qualifies as "encryption".
The way you want to do it won't work, unless you read all values of a column to your Qt program, decrypt them and check if VALUE X is BETWEEN A and B.
I am looking for a way to tell if a database file was modified or not.
The amount of data stored is not large, however updates are often and running select statements after any update to create a new checksum of all data would be too much.
Previously most of our data was stored as entries with JSON, so it was much easier to get few rows and create a checksum of it. Now however, we need to use the database properly, so data will be normalized across few tables and multiple rows.
I need this to be handled by the database, so I don't want to create an md5 of the database file and check that.
Is there any way I could achieve that?
Whenever a database is modified, the file change counter in the database header is incremented.
Is there any way to query a SQLite database for basic meta data such as:
Last date/time updated
Hash of database to indicate "state"
I am just looking for a simple, infrastructural way to have a script evaluate different databases and take a reasonable point of view on whether they are the same "state" as other databases in a different environment (PROD and DEV for instance).
In my experience, if no update, new record, or any change is made to the SQLite database file, the last modified time of the file doesn't change. So the last modified time should suffice for the time of any change made to database.
If 2 database files with same state are only accessed for reading, their modified times are always the same.
Similarly you get the file sizes for comparison.
You can use the whole file to calculate hash. If you consider same data in the database as the same "state" regardless of any difference in the past, then maybe you want hash of the all records in database, which is probably not simple.
I'm developing app for storing some private user data. I use sql-server-ce database. Data can contain images. In future, app will be able to synchronize with SkyDrive. So I have a few questions about data encryption:
What is the best way to encrypt data? Encrypt all databse (with Password option), or just data (with AES128, as in database)?
Where to store assymetric key (for AES). Can I store it as constant in code (I guess no: Silverlight - Hardcoding private key), or should I use ProtectedData class? As I understand, ProtectedData items are linked to current device, so I won't be able to synchronize that generated key to another device?
How to store images? As separated encrypted files, or as Blob column in database? For all app it can be about 50 full-res images. I would like to store them at database, but will it impact performance (e.g. I want to show all 50 thumbs that stores at another Blob column, but with LINQ I select all rows, so they will be all loaded to memory?).
I have an asp.net/vb file that receives data and processes it via a stored procedure. The code had the width set to 2 for the year's varchar, so it was chopped, leaving only the first two digits to get inserted into the db.
Is this info possibly retrievable from a system/IIS log file or is it lost forever?
thanks!
That data is lost forever.
Do you have data that isn't corrupted? Are the records in the database sequential or do they have automatically incrementing fields. Do you have timestamps on the records? Do the years correspond to the date when the record was inserted/updated? Depending on your answers to these you may be able to reconstruct the data. In particular using timestamps and/or autoincrement fields may give you the ability to determine a particular ordering between records. If the date field is related to this ordering you may be able to infer the year from the data in other records. It's very unlikely that any log files would be of any use.
Only if the year was part of a querystring or URL...which is unlikely, at best. If your IIS admin happened to turn on logging of POST fields, then you may be able to retrieve it from there. Very few sites that I know of, though, ever log POST data.