I'm trying to encrypt some columns in my Google BigQuery table.The algorithm uses 256-bit Advanced Encryption Standard (AES-256) keys in Galois Counter Mode (GCM), padded with Cloud KMS-internal metadata.
Is it possible from my reporting service (e.g PBI / SSRS) do the decryption? I saw from this documentation we can decrypt using multiple languages, but not sure if it's possible to integrate with my reporting service
https://cloud.google.com/kms/docs/encrypt-decrypt#kms-decrypt-symmetric-gcloud
Related
In this video GCP engineers show how to encrypt data using a customer provided encryption key. This allows data to be encrypted E2E and the customer can keep the private key securely stored in house.
However, does this still allow joining tables based on certain values in the columns? AFAIK, encryption always adds some randomness to the message and as such, each time a value is encrypted, it will result in a different output while in its encrypted state. As such, joins should not work.
Am I correct in this belief? What are possible mitigations to enable joining despite having encryption? Any personally identifiable information is a common target for encryption, however joins, even on an encrypted value such as the customer ID, are essential for many applications.
A pair of Amazon Lambdas will symmetrically encrypt and decrypt a small piece of application data. I want to use Amazon KMS to facilitate this, because it solves the problems of secret storage and key rotation, and then some.
The Amazon KMS Developer Guide indicates:
These operations are designed to encrypt and decrypt data keys. They use an AWS KMS customer master key (CMK) in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. Although you might use them to encrypt small amounts of data, such as a password or RSA key, they are not designed to encrypt application data.
It goes on to recommend using AWS Encryption SDK or the Amazon S3 encryption client for encrypting application data.
While the listed advantages of the AWS Encryption SDK are clear as day, and very attractive, especially to a developer who is not a cryptographer, let's assume for the purpose of this question that circumstances are not favorable to those alternatives.
If my application data is sure never to exceed 4k, why specifically shouldn't I simply use Amazon KMS to encrypt and decrypt this data?
Use case
My team is implementing a new authentication layer to be used across the services and APIs at our company. We're implementing a JWT specification, but whereas we intend to steer clear of the widely documented cryptographic grievances beleaguering JWE / JWS compliant token signing, we're symmetrically encrypting the payload. Thus, we keep the advantage of standard library implementations of non-cryptographic token validation operations (expiry and the rest,) and we leave behind the cryptographic "foot-gun."
I suspect it's about performance: scaling and and latency.
KMS encrypt/decrypt has a limit of 5500 reqs/s per account, which is shared with some other KMS operations.
"Why?" Is also discussed a bit more thoroughly in the FAQ.
Why use envelope encryption? Why not just send data to AWS KMS to encrypt directly?
While AWS KMS does support sending data less than 4 KB to be encrypted, envelope encryption can offer significant performance benefits. When you encrypt data directly with KMS it must be transferred over the network. Envelope encryption reduces the network load for your application or AWS cloud service. Only the request and fulfillment of the data key through KMS must go over the network. Since the data key is always stored in encrypted form, it is easy and safe to distribute that key where you need it to go without worrying about it being exposed. Encrypted data keys are sent to AWS KMS and decrypted under master keys to ultimately allow you to decrypt your data. The data key is available directly in your application without having to send the entire block of data to AWS KMS and suffer network latency.
https://aws.amazon.com/kms/faqs/
I am going through this issue with AWS support right now. There is the throttling limit mentioned in the accepted answer. Also, if you reuse and cache data keys as allowed by the SDK, you can save money at the expense of lowered security (one data key can decrypt multiple objects).
However, if neither of those are relevant to you, direct CMK encryption is appealing. The security is excellent because the data key cannot be leaked, every decryption requires a API call to KMS and can be audited. In the KMS Best Practices whitepaper, it states that encryption of credit card numbers in this way is PCI compliant.
Option 1: If we use an AWS KMS-managed customer master key (CMK), does that provide end-to-end encryption?
Option 2: Or, would we have to use a client-side master key, so that only the client can decrypt their data?
Update: KMS is not asymmetric, though you can use Envelope Encryption to generate a data key from the CMK. The key is generated on a physical HSM making it not accessible externally. You will only have to worry about the access to the CMK which you can achieve using IAM access control.
For a detailed explanation how the mechanism works, check the Envelope Encryption section on the KMS Cryptographic Details white paper.
So if you only worried about eavesdropping can be a good solution. If you are looking for strictly end-to-end encryption you might have to use asymmetric keys on which KMS can help you with too.
Aws kms does not store any data it provide you two keys
1 plain key : with the help of it you encrypt the data and delete it(key)(no need to save anywhere).
2.encrypted data key :- you need to save this key to decrypt the data( to decrypt the data first you got plain key from aws using encrypted data key) and with the help of plain key you decrypt the data.
So encryption is done at client side.
I am developing a web based application that will be employed by numerous third party organisations
in numerous countries around the world.
The browser based client will feed sensitive data into a shared back end database.
All organisations in all countries will Read/Write data into the same database.
I wish to encrypt the data entered within the browser so that its safe while in transit
to the back end database. e.g. client side encryption.
I also wish to encrypt the data while at rest in my database.
This application will be developed using Java, Javascript (REACT.js), and Scala.
The backend database will be MongoDB.
I cannot find a good Key Management example/description e.g. how a key is properly generated,
distributed, stored, replaced, deleted, and recovered during its life time.
I have the following choices/decisions to make:-
Flavour of encryption, e.g TripleDES, RSA, Blowfish, Twofish, AES etc..
Key(s) Symmetric/Asymmetric and its/thier length
How should I securely distribute the keys to my clients.
How to keep my keys safe on my back end servers.
If keys should have a lifecycle of generated, distributed, stored, replaced, deleted.
How do I decrypt data that was encrypted with Key0 when I am now using Key1, or Key2?
How should I store my multiple keys for my multiple clients to enable me to encrypt/decrypt
each clients data.
Use HTTPS with certificate pinning to secure the data in transit.
Use AES for encryption. Do not use TripleDES, RSA, Blowfish or Twofish in new work.
Use an HSM.
Encrypt the data with a long-life key that is not distributed, encrypt that key with short life keys that can be changed as needed.
Considering the scope of the project get a cryptographic domain expert to design the security and vet the code.
I have tried finding the answer to this but have only found anecdotal references. Does Google encrypt the data that is stored on BigQuery? If so, what encryption mechanisms are used?
All new data being added to BigQuery tables are encrypted using strong, industry-standard encryption methods. All old data will be encrypted over time, but currently with no specific timeline. If you'd like more detail on security across the Google Cloud Platform you might want to check out this blog post:
http://googlecloudplatform.blogspot.com/2014/08/googles-cloud-is-secure-but-you-dont.html
BigQuery is a part of Google's Cloud Platform Offering. As part of utilizing BigQuery you need to first load the data into it.
You can load Data 2 ways.
Load jobs support two data sources:
Objects in Google Cloud Storage
Data sent with the job or streaming insert
Data sourced from Google Cloud Storage is always encrypted as per the link below. Streaming data into BigQuery probably depends on how you stream it (I'm haven't found any absolute data on this point)
Loading data into BigQuery
Description of Compute Engine Disks
Yes, data stored in BigQuery is encrypted.
Google Cloud Platform encrypts data stored in BigQuery, without any action required from the user, using one or more encryption mechanisms. Data stored in Google Cloud Platform is encrypted at the storage level using either AES256 or AES128.
There are more details in this whitepaper:
https://cloud.google.com/security/encryption-at-rest/