I need an alternative to initialization vector encryption - encryption

I helped a small non-profit create a cool "anonymous" help line where their clientele can send in SMS questions to a phone number and organization can respond without knowing who the client is. It's proven to be a very helpful tool for the organization. I am in the process of re-writing it from scratch and am looking for advice on how to handle the anonymity aspect of it.
The way that v1 works is by simply encrypting / decrypting the client phone number and using that as the client id in the data model using 'aes-256-ctr'. Since the staff / leadership of the nonprofit do not have access to the encryption key, they can legitimately claim to not be able to have access to any p.i.i. not disclosed voluntary by their clients. When they issue a message response to the client, I simply need to decrypt the client id to get the phone that I need in order to dispatch the message.
v1 encrpytion strategy:
export const encrypt = (text) => {
const cipher = crypto.createCipher(algorithm, password);
let crypted = cipher.update(text, 'utf8', 'hex');
crypted += cipher.final('hex');
return crypted;
};
export const decrypt = (text) => {
const decipher = crypto.createDecipher(algorithm, password);
let dec = decipher.update(text, 'hex', 'utf8');
dec += decipher.final('utf8');
return dec;
};
Lately though, I've been getting console warnings because new node versions will output a warning when using createCipher instead of createCipheriv to utilize initialization vector. On my v2 rewrite I switched it over to createCipheriv, but that has the effect of "randomizing" the encryption output and breaking my ability to rely upon the encrypted phone number as a reliable, consistent client id. Being able to reliably group client messages is an important requirement to provide a conversational view in the UX.
How might I properly encrypt the identifying phone number such that only I can decrypt it at message send time and still have some kind of consistent internal ID by which to group messages? I don't HAVE to rely upon the encrypted phone number as the client id, but I can't think of any reliable alternative (via a lookup table of some kind). If possible I don't want to fall back on createCipher without the I.V. if its not a best practice.
I should note that this application is not dealing with any legally sensitive information or anything high stakes like that. I probably could get away with storing the unencrypted phone number in the database and just not exposing it to the front end and maintain anonymity, I just really don't like that idea if there is a sensible workaround.
Any ideas?

You should be generating a new random (or UUID-style) client id for each new client. The benefit of this is that you can keep using a secure encryption method and still identify and lookup users quickly.
If you want the phone number field to act as a unique lookup, simply take an HMAC of the phone number and use that for lookups (as well as storing encrypted) - it can't ever be decrypted and is guaranteed to be consistent across invocations for the same phone number under a given key.
EDIT: To migrate existing data, you should, for each record, take the client_id (currently the encrypted phone number), decrypt it, HMAC it, and store the result in a new column. Then, generate a new ID and set that to be the client_id. This allows lookup by actual client_id and by phone number.

Related

Encryption and Security with C# and SQL

I have seen the examples but I'm hoping to run this by other programmers. For encryption within my window forms app, I am generating two random numbers and saving them in an SQL Server table like thus:
OPEN SYMMETRIC KEY SymmetricKeyName DECRYPTION BY CERTIFICATE CertificateName;
insert into keyfile(encrypted_key1, encrypted_key2) values
(EncryptByKey(Key_GUID('SymmetricKeyName'), **Key1**),
EncryptByKey(Key_GUID('SymmetricKeyName'), **Key2**))
Then I am using the keys to encrypt a file using AES-256 as follows:
var key = new Rfc2898DeriveBytes(**Key1, Key2**, 1000);
RijndaelManaged AES = new RijndaelManaged();
AES.KeySize = 256;
AES.BlockSize = 128;
AES.Key = key.GetBytes(AES.KeySize / 8);
AES.IV = key.GetBytes(AES.BlockSize / 8);
AES.Padding = PaddingMode.Zeros;
AES.Mode = CipherMode.CBC;
using (var output = File.Create(outputFile))
{
using (var crypto = new CryptoStream(output, AES.CreateEncryptor(), CryptoStreamMode.Write))
{
using (var input = File.OpenRead(inputFile))
{
input.CopyTo(crypto);
}
}
}
etc.
In order to perform decryption both keys that were used to encrypt the file are required. Decryption is possible through software requiring two authenticated users. The keys change every day. The data and the database are sufficiently physically secure. The key table is in a separate database from the certificate.
The question is: Does this secure the data enough to not be readily be decrypted and, if now, why not and what changes might you suggest?
The problem here is that there is a high probability that anyone that is able to obtain the file is also able to obtain the data in the database (which includes the key). Once the data is compromised, it doesn't matter how often you change the key since the attacker would have a copy of the file encrypted with the key that matches it.
Common solutions to this problem are to use an external Hardware Security Module or something like a TPM.
Here is a very useful and related post that enumerates several options.
As suggested by others, you can store the key on a USB, alternatively a network share. However if on a network share you might need to change the Service Logon to an account with access to the network share.
"SYMMETRIC KEY" might be an issue here. They are similar to a xor operation, the same key works to both encrypt and decrypt.
For a 'cooler' method, use ASYMMETRIC keys instead, then the database can keep the 'how to encrypt' half, while your application can have the 'how to decrypt' half. It's a lot of effort, but "not even the DBAs can see the secret data" is a cool feature.

Password decryption with known salt and dictionary

We are migrating a company we acquired a lot of kiosk hardware assets from that are in the field. Since the company was suffering, we are stuck with some issues in migrating the locations fingerprints, usernames and passwords without any implementation docs. Luckily, most of the passwords used are numeric 4-6 PIN or a common used word. I'm stuck with trying to figure out what format the password is hashed in and hopefully can decipher it from there using a dictionary for the majority of the passwords. I have the following format:
"password": "ce62f0002776890507c4050a3b76c064d3d24328aea52a08633b726d352532dc",
"salt": "JUQLSPOYGFURMGSDRYWIWBIWP",
The password above is "password". Hopefully this helps in finding the format.
If it is a hash, looks like a hash, possibly HMAC-SHA256 from the length, you need to run a password cracking program. You should be able to recover well over 90% but most likely not all.
On my laptop I can run a 20 byte password through SHA-512 and compare in under 1us so with just a SHA-512 hash I can try 1,000,000 passwords a second.
You can make a list to check but there are already good lists, see below.
For more information see:
Password list at SecLists.
Infosec password-cracking-tools
Arstechnica How I became a password cracker.
You can implement the old hashing method in your new code. When the password matches (i.e. the one the partner sends) you can then store it in your new format (essentually accepting both). This saves you the need to crack the existing passwords.
For this to work you do need to know how the passwords are hashed and what formatis used, lucikly this seems to be fairly easy (Java sample):
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bos.write("password".getBytes(StandardCharsets.US_ASCII));
bos.write("JUQLSPOYGFURMGSDRYWIWBIWP".getBytes(StandardCharsets.US_ASCII));
MessageDigest md = MessageDigest.getInstance("SHA-256");
byte[] out = md.digest(bos.toByteArray());
System.out.println("hex = " + new HexBinaryAdapter().marshal(out).toLowerCase());
Produces (i.e. concatenate password bytes and salt bytes, non-iteratively calculate SHA256 and convert to hex) the expected hash:
hex = ce62f0002776890507c4050a3b76c064d3d24328aea52a08633b726d352532dc

Theory of Retrieving Hashed Passwords

This is a theory question, as I already know the following:
Hashes are not encryptions
Hashes can be broken https://crackstation.net/
That my code is not serious enough to be used on an enterprise site
But my question is just on the theory of it all.
In theory, if I stored a hashed password into a database, how could I check it when a user comes to log in? Is it that simple or is there a standard way to actually store and retrieve passwords in SQLite3?
If you can offer some SQLite3 code or guidance on where to start, that would be appreciated.
import sqlite3
import hashlib
def encrypt_password(password):
encrypted_pass = hashlib.sha1(password.encode('utf-8')).hexdigest()
return encrypted_pass
cemail = input("E: ")
cpassword = input("P: ")
connection = sqlite3.connect('/Users/Air/Desktop/users.db')
connection.create_function('encrypt', 1, encrypt_password)
cursor_v = connection.cursor()
cursor_v.execute("insert into user (cemail, cpassword) values (?,encrypt(?))", (cemail, cpassword))
connection.commit()
cursor_v.close()
When dealing with passwords, there's an absolute ton of information you need to know as a minimum:
Firstly, hashing passwords must be slow, which is the exact opposite of what a hash function is usually intended for.
Use bcrypt or PBKDF2 for this purpose. MD5/SHA1/256/512 are not sufficient for a password and offer no real protection when faced with a GPU-based bruteforcer (such as hashcat).
Second, verification of passwords must be in constant-time, aka they must complete in the same time regardless of the length of a string.
String comparison functions are at risk of a timing attack, and this means that the hash can be deduced by repeatedly giving different inputs to your verification function until enough information about the hash is revealed.
To get around this, use a cryptography library that does this for you.
If you have no other choice, the following pseudocode should help you see how a constant-time algorithm generally works (though you should still use a library!)
def verify(hash, input):
key = secure_random(32)
hash = hmac_sha_512(key, hash)
input = hmac_sha_512(key, bcrypt(input))
results = []
for i, letter in input
results.add(hash[i] === letter)
return false not in results
Thirdly, passwords should have a cryptographically random salt used throughout the hashing. A secure random source of randomness is generally taken from /dev/urandom or a library dedicated to it.
To answer the question of Well how do I verify a password if I can't see the original value? — simple, just hash whatever a user gives you and see if the hash matches the one in the database. But see above for the huge pitfall there.
You cannot retrieve hashed password but compare new password whether it is the same. IF HASH(new_password) == hashed_password_in_db THEN ..ok.. ELSE ..wrong password....

MAC variant of MSR DUKPT?

Currently I am working on a ChipCard EMV device decryption. Down below is the related data I have after using the transaction (TLV format as Tag Length Value):
<DFDF54> --- It means KSN
0A
950003000005282005B4
<DFDF59>---- per instruction, it is called Encrypted Data Primative
82 ---- length of value in hex, when more than 255 degits, use 82
00D815F35E7846BF4F34E56D7A42E9D24A59CDDF8C3D565CD3D42A341D4AD84B0B7DBFC02DE72A57770D4F795FAB2CE3A1F253F22E0A8BA8E36FA3EA38EE8C95FEBA3767CDE0D3FBB6741A47BE6734046B8CBFB6044C6EE5F98C9DABCD47BC3FD371F777E7E1DCFA16EE5718FKLIOE51A749C7ECC736CB7780AC39DE062DAACC318219E9AAA26E3C2CE28B82C8D22178DA9CCAE6BBA20AC79AB985FF13611FE80E26C34D27E674C63CAC1933E3F9B1BE319A5D12D16561C334F931A5E619243AF398D9636B0A8DC2ED5C6D1C7C795C00D083C08953BC8679C60
I know BDK for this device is 0123456789ABCDEFFEDCBA9876543210. Per decryption instruction, it mentioned that DFDF59 contains the following tags:
FC<len>/* container for encrypted generic data */
F2<len>/*container for Batch Data*/
... /*Batch Data tags*/
F3<len>/*container for Reversal Data, if any*/
... /*Reversal Data tags*/
Per instruction, it mentioned "MAC variant of MSR DUKPT", where MAC stands for message authentication code, and "Parse the data through TLV format. For encrypted data tag, use TDES_Decrypt_CBC to decrypt it".
I tried to use 3DES DUKPT using KSN, BDK, and encrypted data DFDF59. It wouldn't work. Can anyone in decryption field give me some advice? Our vendor is very reluctant to share their knowledge ...
I have no idea how MAC is really playing a role here in decryption.... I thought MAC is just an integrity check.... I am using session key for 3DES DUKPT that was generated from KSN and BDK. this works for other decryptions in this device, but doesn't solve the DFDF59 (chip card EMV decryption).... That is why I start to wonder whether I am using the right session key or not.... Feel free to just throw ideas out there. Thank you!
If you look closely at DUKPT internals it generates a transaction key out of the current future keys and encryption counter. This 'transaction key' for a specific KSN has several variants (which effectively are just xor masks that you put on the transaction key to differentiate it for PIN, MAC req, MAC rsp, data encryption req and rsp usages). These variants mean that you use a different key to generate PIN and different key to encrypt data (so that you cannot ie. decrypt/attack PINblock when able to select data buffer arbitrarily). Using MAC variant means only that for the encryption operation you will be using a certain mask for the DUKPT transaction key.

Effective Password Encryption

I've taken a look at the StackOverflow question, "Password Encryption / Database Layer AES or App Layer AES," and I'd like to effectively and efficiently hash my passwords on registration (web app) and then be able to check they are correct on login. I'm using VB, but comfortable using C#.
I would love to use Jeff Atwood's Encryption class described in ".NET Encryption Simplified" as it's really easy to understand. It has a hashing class—but I have no idea how to "login" and compare hashes after they have been hashed. This is Jeff's demonstration of his hash methods using his Encryption class:
Sub DemoHash()
Dim d As New Encryption.Data( _
"{ts '2004-10-09 08:10:04'}The world is beautiful and needs caring by its children")
Dim hash As New Encryption.Hash(Encryption.Hash.Provider.SHA1)
Dim hash2 As New Encryption.Hash(Encryption.Hash.Provider.SHA256)
Dim hash3 As New Encryption.Hash(Encryption.Hash.Provider.SHA384)
Dim hash4 As New Encryption.Hash(Encryption.Hash.Provider.SHA512)
Dim hash5 As New Encryption.Hash(Encryption.Hash.Provider.MD5)
Dim hash6 As New Encryption.Hash(Encryption.Hash.Provider.CRC32)
hash.Calculate(d)
hash2.Calculate(d)
hash3.Calculate(d)
hash4.Calculate(d)
hash5.Calculate(d)
Console.WriteLine("SHA1: " & hash.Value.Hex)
Console.WriteLine("SHA256: " & hash2.Value.Hex)
Console.WriteLine("SHA384: " & hash3.Value.Hex)
Console.WriteLine("SHA512: " & hash4.Value.Hex)
Console.WriteLine("MD5: " & hash5.Value.Hex)
Console.WriteLine("CRC32: " & hash6.Calculate(d).Hex)
Console.WriteLine()
Dim salt As New Encryption.Data("salty!")
Console.WriteLine("Salted CRC32: " & hash6.Calculate(d, salt).Hex)
Console.WriteLine("Press ENTER to continue...")
Console.ReadLine()
End Sub
So my questions are:
I can encrypt the password (though I have no intention of storing it) and hash a string. If I were to have a user called 'barry' with a password of 'fishlegs', what is the best way to store his password and retrieve it?
In SQL Server; is binary or nvarchar the best option for the storage of the hash?
Based on 'barry' and his password what effectively is the hash storing? Is it an encryption of 'fishlegs' appended to a salt?
Cryptography is hard!
Thanks to anyone who can assist...
Hmm, I think you're just missing some basic concepts related to how hashing works. Let me try to explain briefly. I'm going to start out simple and elaborate on my answer afterwards, so please read through the whole thing, the information at the beginning will not be secure.
What you want to use to store a password is a function known as a "one-way hash". What this means is that, for any input that you feed the function, the same input will always give the same result. However, there is no mathematical process that lets you take that result string and figure out what the original input was.
Let's take MD5 as an example of a hashing function. If I run MD5 on the string "password", I will always get the result "5f4dcc3b5aa765d61d8327deb882cf99". However, if you were to simply give someone that result string ("5f4d..."), it is impossible for them to apply some mathematical process to "reverse" the function and figure out that it came from "password".
What this means is that when a user first sets up their password, you apply a hashing function to it, and store the result. So instead of storing "password", you store "5f4dcc3b5aa765d61d8327deb882cf99". Then, when that user tries to log in, you take whatever they typed into the password box on the login form, and apply the same hashing function. If you get the same result as what's stored in the database, they must have entered the same password as they originally chose, even though you have no idea what that original password actually was.
Now, even though it's impossible to "reverse" a hash function, the fact that the same input always gives the same output means that someone can simply build up a big database of input/output pairs, and use that to effectively reverse hashes. This is called a "rainbow table". There are many of these available on the internet, so it's not safe to use simple hashing, just in case your database ever gets compromised. That is, even though it is mathematically impossible to take "5f4dcc3b5aa765d61d8327deb882cf99" and figure out that it came from running MD5 on "password", it's very easy to determine that in practice. All you have to do is run every word in a dictionary through MD5 and store the results, and you can easily reverse simple passwords.
This is where "salting" comes in. If you generate a random "salt" string for every user and attach that to their password, it effectively ruins rainbow tables. For example, let's say that the same user above registers with their password as "password". We generate a random 8-character salt to attach to their password before hashing it. Let's say that it's "A4BR82QX". Now, instead of hashing "password", we hash "A4BR82QXpassword". This gives the result "87a4ba071c8bcb5efe457e6c4e6c4490", so we store that in the database, along with the salt string. Then when this user tries to log in, instead of directly hashing and comparing the password they entered in the login form, we take what they entered, put "A4BR82QX" in front of it again, and hash that. Just as before, if it matches the stored hash, we know that they entered the right password.
Effectively what you've done here is make it so that pre-generated rainbow tables are useless for trying to crack the passwords in your database. Since the salt is random, and each user has a different one (generally), the attacker will have to re-generate their rainbow tables for every individual user. This is much more difficult.
However, there's one more problem, and that's that generating MD5 hashes is fast. Even though salting like this requires them to re-generate rainbow tables, because of how fast MD5 is, some decently-complete rainbow tables can be created very quickly. So if they just want to crack a high-value account on your site, it's not really a big deal for them to spend some time generating rainbow tables to try and reverse that password. If the high-value account's original password wasn't secure enough by itself, it'll still be found very quickly, even with salting.
So the next step is to find a slow hash function, and use this instead of a fast one like MD5. Having your site take an extra couple of seconds to check a login isn't a big deal at all. But when someone is trying to generate rainbow tables to crack a password, having each entry take several seconds is an absolute killer. I've written enough here, so I'll just finish by linking to this article, which goes into plenty of detail about picking a good, slow hashing function: Enough With The Rainbow Tables: What You Need To Know About Secure Password Schemes.
That was a pretty huge answer, if any of that's unclear, please let me know in a comment and I'll edit to elaborate.
OK so a couple of things about Jeff's class first of all. SHA1 and MD5 are deprecated now. CRC32 is not suitable at all for passwords. Secondly you should be salting every hash, preferably with a different salt value. Generally you choose a cryptographically random block of data for this, but at a push you could use the user name. To salt you prefix, or suffix the salt value somewhere in the process. I tend to hash the password, hash the salt, combine the two, then hash again. But you can swap things around it doesn't really matter that much, as long as you are consistent.
So rather than confuse things further with Jeff's class lets do this the classic way.
First off random salt generation.
public static byte[] GetRandomSalt()
{
int minSaltSize = 16;
int maxSaltSize = 32;
Random random = new Random();
int saltSize = random.Next(minSaltSize, maxSaltSize);
saltBytes = new byte[saltSize];
RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
rng.GetNonZeroBytes(saltBytes);
return saltBytes;
}
Then hashing
public static byte[] ComputeHash(string plainText)
{
byte[] plainTextBytes = Encoding.UTF8.GetBytes(plainText);
HashAlgorithm hash = new SHA256Managed();
return hash.ComputeHash(plainTextWithSaltBytes);
}
So this will compute a SHA256 hash and return it as a byte array.
If you were salting you'd do something like the following
byte[] passwordHash = ComputeHash(password);
byte[] salt = GetRandomSalt();
byte[] saltHash = ComputeHash(salt);
byte[] hashWithSaltBytes = new byte[hashBytes.Length + saltBytes.Length];
for (int i=0; i < hashBytes.Length; i++)
hashWithSaltBytes[i] = hashBytes[i];
for (int i=0; i < saltBytes.Length; i++)
hashWithSaltBytes[hashBytes.Length + i] = saltBytes[i];
And then, if you're bored hash it down again, or leave as is.
To turn a byte array into a string, if you don't fancy storing bytes you can use
string hashValue = Convert.ToBase64String(hashWithSaltBytes);
String comparisons are easier than byte comparisons where you have to iterate over each array, up to you. Just remember if you are using random salts you need to store them beside the password.
Take the user password "Secret!".
Generate a random salt of a few bytes "s4L75a1T".
Concat them to "s4L75a1TSecret!".
Calculate the hash "b9797a5b683804eb195b6ba2a5e368ae74394cd3" (this is SHA-1 in hex)
Store the hash and the salt (both as hex string, Base64 string, or whatever you like) in the database together with the user name and the other user information (in the example the salt is just a plain string and the hash is Base64 encoded).
FirstName LastName Salt Hash
-----------------------------------------------------------------
John Doe s4L75a1T uXl6W2g4BOsZW2uipeNornQ5TNM=
If you want to verify the user password, just take the user name, look up the salt, do the above again and see if the calculated hash and the one in the database match. There is no (known) way to (easily) recover the password.
Don't confuse encrypting passwords with hashing them; with a good cryptographic hash function, you should not be able to reverse the hashing process to retrieve the original string.
Chad's answer above is an excellent point-by-point explanation of the concepts involved.
This subject's been done to death all over the Internet; not just on Stack Overflow; seriously - a simple web search ought to find you a pretty comprehensive guide.
Thanks, Jeff, for spreading yet another bunch of misinformation. I suppose we'll see a whole slew of misguided questions about hashing, encryption, etc. over the next week, not to mention the crap that'll come up about floating-point arithmetic.
I believe the process is like this: Generate a random salt, so 'fishlegs' + 'r4nd0mS4lt' to get 'fishlegsr4nd0mS4lt'. Then you hash that, say with MD5 (though you might want to use SHA256 to be more secure) to get: 593d5518d759b4860188158ef4c71f28. Store that and the randomly generated salt. When the user logs in, append the random salt and then check if his entered password with the salt matches the hash in the database.
What you are essentially wanting to do is:
A) On account creation, you get a username and password from the user, you hash those together with your own salt and store the resultant string in your database like:
Dim sUserName = "barry"
Dim sPassWord = "fishlegs"
Dim mySalt = "A deliciously salty string! fillED WIth all KindS of Junkk(&^&*(£"
Dim d As New Encryption.Data(mySalt + sUserName + sPassWord)
Dim hash As New Encryption.Hash(Encryption.Hash.Provider.SHA256)
hash.Calculate(d)
Dim sTheSaltedHashedUnPassCombination = hash.Value.Hex;
SavenewPasswordHashToDatabase(sTheSaltedHashedUnPassCombination)
You never store sPassWord.
B) When the user logs in you perform exactly the same operation on the provided username and password then compare the resultant hash to the previously stored value in the database so you use:
Dim sThePreviouslyCreatedHashFromTheDatabase = GetHashForUserFromDatabase(usernameProvided)
Dim mySalt = "A deliciously salty string! fillED WIth all KindS of Junkk(&^&*(£"
Dim d As New Encryption.Data(mySalt + usernameProvided+ passwordProvided)
Dim hash As New Encryption.Hash(Encryption.Hash.Provider.SHA256)
hash.Calculate(d)
Dim sTheSaltedHashedUnPassCombination = hash.Value.Hex;
if (sThePreviouslyCreatedHashFromTheDatabase.Equals(sTheSaltedHashedUnPassCombination))
'UN & Password Valid!
else
'UN & PW Invalid!
end
(Pardon any errors, VB aint my language)
To answer your given questions:
1) See above. Never store the password directly, store the hashed value
2) use a char(X), the number of characters returned from the hash is constant so you can give it a known storage size.
3) What you are effectively storing is the password for the user salted with their username and also salted with the constant on your server. This will be a fixed length string and you cannot change this string back into the separate username and password.
The whole point of hashing is that you store the hashes rather than the plaintext passwords and they can't be recovered (at least only with great difficulty). Hash functions are designed to be one-way, so that retrieval is impossible - this is their very essence.
nchar[n] is probably the best option, since common hash algorithms produce results of constant length (SHA1 outputs a 160-bit hash). Common practice is to convert the hash to Base64 so that it can be stored as ASCII text.
A salt is just a method that adds random bits onto a hash in order to complicate the process of cracking. Each extra bit of salt means that a brute-force cracker must take twice as long (and use twice as much memory).
ASP.NET includes a SQL based membership provider that allows you to create, store and verify encryped or hashed passwords, without you having to do anything - as Eric Lippert says:
let me give you all my standard caution about rolling your own cryptographic algorithms and security systems: don't.
You Want Salt With That? Part One: Security vs Obscurity
You Want Salt With That? Part Two: Security vs Obscurity
You Want Salt With That? Part Three: We Need A Hash
You Want Salt With That? Part Four: Challenge-Response

Resources