I have a website that allows users to upload files which are in turn stored in a Microsoft SQL Database.
The database contains hundreds of thousands of images and is becoming unmanagable. As such I've written a powershell script that extracts those images from the database to the filesystem and then generates a SHA1 hash of the file as such:
$SHA1 = Get-Checksum -Algorithm sha1 -File ($Dest + "$i.jpg")
Going forward I no longer want to store images in the database but rather use the file system however I still need to generate SHA1 hashes of each file.
I've been unable to find any documentation on generating SHA1 file hashes in classic asp or asp.net (c# or vb.net).
Note I specifically need to know how to generate FILE HASHES - not hashes of strings.
Has anyone else done this or know if or how it can be done? Any example code you could provide would be greatly appreciated.
Thanks
Brad
using (FileStream fs = new FileStream(#"C:\file\location", FileMode.Open))
using (BufferedStream bs = new BufferedStream(fs))
{
using (SHA1Managed sha1 = new SHA1Managed())
{
byte[] hash = sha1.ComputeHash(bs);
StringBuilder formatted = new StringBuilder(2 * hash.Length);
foreach (byte b in hash)
{
formatted.AppendFormat("{0:X2}", b);
}
}
}
formatted contains the string representation of the SHA-1 hash. Also, by using a FileStream instead of a byte buffer, ComputeHash computes the hash in chunks, so you don't have to load the entire file in one go, which is helpful for large files.
Answer borrowed from mgbowen in How do I do a SHA1 File Checksum in C#?
Related
My question is a bit similar to this one but it is with ASP.NET and my requirements are slightly different: Android append files to a zip file without having to re-write the entire zip file?
I need to insert data to a zip-file downloaded by users (not much 1KB of data at most, this is data for Adword off-line conversion actually). The zip-file is downloaded through an ASP.NET website. Because the zip file is already large enough (10's of MB) to avoid overloading the server, I need to insert these data without re-compressing everything. I can think of two ways to do this.
Way A: Find a zip-technology that lets embed a particular file in the ZIP file, this particular file being embedded uncompressed. Assuming there is no checksum, it'd be then easy to just override the bits of this un-compressed file with my specific data, in the zip file itself. If possible, this would have to be supported by all unzip tools (Windows integrated zip, winrar, 7zip...).
Way B: Append an extra file to the original ZIP file without having to recompress it! This extra file would have to be stored in an embedded folder in the ZIP file.
I looked a bit at SevenZipSharp which has an enumeration SevenZip.CompressionMode with values Create and Append that leads me to think that Way B could be implemented. DotNetZip seems also to work pretty well with Stream according to FAQ.
But if Way A could be possible I'd prefer it much since no extra zip library would be needed on the server side!
Ok, thanks to DotNetZip I am able to do what I want in a very resource efficient way:
using System.IO;
using Ionic.Zip;
class Program {
static void Main(string[] args) {
byte[] buffer;
using (var memoryStream = new MemoryStream()) {
using (var zip = new ZipFile(#"C:\temp\MylargeZipFile.zip")) {
// The file on which to override content in MylargeZipFile.zip
// has the path "Path\FileToUpdate.txt"
zip.UpdateEntry(#"Path\FileToUpdate.txt", #"Hello My New Content");
zip.Save(memoryStream);
}
buffer = memoryStream.ToArray();
}
// Here the buffer will be sent to httpResponse
// httpResponse.Clear();
// httpResponse.AddHeader("Content-Disposition", "attachment; filename=MylargeZipFile.zip");
// httpResponse.ContentType = "application/octe-t-stream";
// httpResponse.BinaryWrite(buffer);
// httpResponse.BufferOutput = true;
// Just to check it worked!
File.WriteAllBytes(#"C:\temp\Result.zip", buffer);
}
}
I want to encrypt large string (200 MB).
The string come from dataUrl (base64) corresponding to file.
I'm doing my encryption in the browser.
My issue is that at the moment, i chunked string into small part into an array.
Then i encrypt this chunks.
At the moment encrypting the string will full the memory.
Here is how i'm doing it.
var encryptChunk = function(chunk, index){
encryptedChunks.push( aesEncryptor.process( chunk ));
sendUpdateMessage( "encryption", index+1, numberOfChunks );
}
chunkedString.forEach(encryptChunk);
encryptedChunks.push( aesEncryptor.finalize() );
I assume that, there should be a better way of doing this. But i can't find a memroy efficient way of doing this.
I am doing something similar to you. To directly answer your question of "is there a more memory efficient way?" .. well I use a web worker for processing progressive ciphering which seems to work.
//pass in what you need here
var worker = new Worker("path/to/worker.js");
worker.postMessage({
key: getKeyAndIvSomehow(),
file: file,
chunkSize: MY_CHUNK_SIZE
});
worker.addEventListener('message', function (e) {
// create the blob from e.data.encrypted
});
You will need to import the cryptoJS script into your worker: importScripts('cryptoJS.all.min.js')
What are you doing with the encrypted chunks? If you're, say, uploading them over the network, you don't need to store them in an array first. Instead, you can upload the encrypted file chunk by chunk, either writing your own chunked upload implementation (it's not terribly hard) or by using an existing library.
Ditto for the input: you can encrypt it as you read it. You can use the JS File API to read the file in chunks, using the .slice() method.
Other than that, your code looks just like the recommended way to progressively encrypt a file.
I'm just learning to use SQLite and I was curious if such is possible:
Encryption of the database file?
Password protect opening of the database?
PS. I know that there is this "SQLite Encryption Extension (SEE).", but according to the documentation, "The SEE is licensed software...." and "The cost of a perpetual source code license for SEE is US $2000."
SQLite has hooks built-in for encryption which are not used in the normal distribution, but here are a few implementations I know of:
SEE - The official implementation.
wxSQLite - A wxWidgets style C++ wrapper that also implements SQLite's encryption.
SQLCipher - Uses openSSL's libcrypto to implement.
SQLiteCrypt - Custom implementation, modified API.
botansqlite3 - botansqlite3 is an encryption codec for SQLite3 that can use any algorithms in Botan for encryption.
sqleet - another encryption implementation, using ChaCha20/Poly1305 primitives. Note that wxSQLite mentioned above can use this as a crypto provider.
The SEE and SQLiteCrypt require the purchase of a license.
Disclosure: I created botansqlite3.
You can password protect SQLite3 DB.
For the first time before doing any operations, set password as follows.
SQLiteConnection conn = new SQLiteConnection("Data Source=MyDatabase.sqlite;Version=3;");
conn.SetPassword("password");
conn.open();
then next time you can access it like
conn = new SQLiteConnection("Data Source=MyDatabase.sqlite;Version=3;Password=password;");
conn.Open();
This wont allow any GUI editor to view Your data.
Later if you wish to change the password, use conn.ChangePassword("new_password");
To reset or remove password, use conn.ChangePassword(String.Empty);
The .net library System.Data.SQLite also provides for encryption.
You can get sqlite3.dll file with encryption support from http://system.data.sqlite.org/.
1 - Go to http://system.data.sqlite.org/index.html/doc/trunk/www/downloads.wiki and download one of the packages. .NET version is irrelevant here.
2 - Extract SQLite.Interop.dll from package and rename it to sqlite3.dll. This DLL supports encryption via plaintext passwords or encryption keys.
The mentioned file is native and does NOT require .NET framework. It might need Visual C++ Runtime depending on the package you have downloaded.
UPDATE
This is the package that I've downloaded for 32-bit development: http://system.data.sqlite.org/blobs/1.0.94.0/sqlite-netFx40-static-binary-Win32-2010-1.0.94.0.zip
Keep in mind, the following is not intended to be a substitute for a proper security solution.
After playing around with this for four days, I've put together a solution using only the open source System.Data.SQLite package from NuGet. I don't know how much protection this provides. I'm only using it for my own course of study. This will create the DB, encrypt it, create a table, and add data.
using System.Data.SQLite;
namespace EncryptDB
{
class Program
{
static void Main(string[] args)
{
string connectionString = #"C:\Programming\sqlite3\db.db";
string passwordString = "password";
byte[] passwordBytes = GetBytes(passwordString);
SQLiteConnection.CreateFile(connectionString);
SQLiteConnection conn = new SQLiteConnection("Data Source=" + connectionString + ";Version=3;");
conn.SetPassword(passwordBytes);
conn.Open();
SQLiteCommand sqlCmd = new SQLiteCommand("CREATE TABLE data(filename TEXT, filepath TEXT, filelength INTEGER, directory TEXT)", conn);
sqlCmd.ExecuteNonQuery();
sqlCmd = new SQLiteCommand("INSERT INTO data VALUES('name', 'path', 200, 'dir')", conn);
sqlCmd.ExecuteNonQuery();
conn.Close();
}
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
bytes = System.Text.Encoding.Default.GetBytes(str);
return bytes;
}
}
}
Optionally, you can remove conn.SetPassword(passwordBytes);, and replace it with conn.ChangePassword("password"); which needs to be placed after conn.Open(); instead of before. Then you won't need the GetBytes method.
To decrypt, it's just a matter of putting the password in your connection string before the call to open.
string filename = #"C:\Programming\sqlite3\db.db";
string passwordString = "password";
SQLiteConnection conn = new SQLiteConnection("Data Source=" + filename + ";Version=3;Password=" + passwordString + ";");
conn.Open();
You can always encrypt data on the client side. Please note that not all of the data have to be encrypted because it has a performance issue.
You can use SQLite's function creation routines (PHP manual):
$db_obj->sqliteCreateFunction('Encrypt', 'MyEncryptFunction', 2);
$db_obj->sqliteCreateFunction('Decrypt', 'MyDecryptFunction', 2);
When inserting data, you can use the encryption function directly and INSERT the encrypted data or you can use the custom function and pass unencrypted data:
$insert_obj = $db_obj->prepare('INSERT INTO table (Clear, Encrypted) ' .
'VALUES (:clear, Encrypt(:data, "' . $passwordhash_str . '"))');
When retrieving data, you can also use SQL search functionality:
$select_obj = $db_obj->prepare('SELECT Clear, ' .
'Decrypt(Encrypted, "' . $passwordhash_str . '") AS PlainText FROM table ' .
'WHERE PlainText LIKE :searchterm');
Well, SEE is expensive. However SQLite has interface built-in for encryption (Pager). This means, that on top of existing code one can easily develop some encryption mechanism, does not have to be AES. Anything really.
Please see my post here: https://stackoverflow.com/a/49161716/9418360
You need to define SQLITE_HAS_CODEC=1 to enable Pager encryption. Sample code below (original SQLite source):
#ifdef SQLITE_HAS_CODEC
/*
** This function is called by the wal module when writing page content
** into the log file.
**
** This function returns a pointer to a buffer containing the encrypted
** page content. If a malloc fails, this function may return NULL.
*/
SQLITE_PRIVATE void *sqlite3PagerCodec(PgHdr *pPg){
void *aData = 0;
CODEC2(pPg->pPager, pPg->pData, pPg->pgno, 6, return 0, aData);
return aData;
}
#endif
There is a commercial version in C language for SQLite encryption using AES256 - it can also work with PHP, but it needs to be compiled with PHP and SQLite extension. It de/encrypts SQLite database file on the fly, file contents are always encrypted. Very useful.
http://www.iqx7.com/products/sqlite-encryption
I had also similar problem. Needed to store sensitive data in simple database (SQLite was the perfect choice except security). Finally I have placed database file on TrueCrypt encrypted valume.
Additional console app mounts temporary drive using TrueCrypt CLI and then starts the database application. Waits until the database application exits and then dismounts the drive again.
Maybe not suitable solution in all scenarios but for me working well ...
I run a rather large site where my members add thousands of images every day. Obviously there is a lot of duplication and i was just wondering if during an upload of an image i can somehow generate a signature or a hash of an image so i can store it. And every time someone uploads the picture i would simply run a check if this signature already exists and fire an error stating that this image already exists. Not sure if this kind of technology already exists for asp.net but i am aware of tineye.com which sort of does it already.
If you think you can help i would appreciate your input.
Kris
A keyword that might be of interest is perceptual hashing.
You use any derived HashAlgorithm to generate a hash from the byte array of the file. Usually MD5 is used, but you could subsitute this for any of those provided in the System.Security.Cryptography namespace. This works for any binary, not just images.
Lots of sites provide MD5 hashes when you download files to verify if you've downloaded the file properly. For instance, an ISO CD/DVD image may be missing bytes when you've received the whole thing. Once you've downloaded the file, you generate the hash for it and make sure it's the same as the site says it should be. If all compares, you've got an exact copy.
I would probably use something similar to this:
public static class Helpers
{
//If you're running .NET 2.0 or lower, remove the 'this' keyword from the
//method signature as 2.0 doesn't support extension methods.
static string GetHashString(this byte[] bytes, HashAlgorithm cryptoProvider)
{
byte[] hash = cryptoProvider.ComputeHash(bytes);
return Convert.ToBase64String(hash);
}
}
Requires:
using System.Security.Cryptography;
Call using:
byte[] bytes = File.ReadAllBytes("FilePath");
string filehash = bytes.GetHashString(new MD5CryptoServiceProvider());
or if you're running in .NET 2.0 or lower:
string filehash = Helpers.GetHashString(File.ReadAllBytes("FilePath"), new MD5CryptoServiceProvider());
If you were to decide to go with a different hashing method instead of MD5 for the miniscule probability of collisions:
string filehash = bytes.GetHashString(new SHA1CryptoServiceProvider());
This way your has method isn't crypto provider specific and if you were to decide you wanted to change which crypto provider you're using, you just inject a different one into the cryptoProvider parameter.
You can use any of the other hashing classes just by changing the service provider you pass in:
string md5Hash = bytes.GetHashString(new MD5CryptoServiceProvider());
string sha1Hash = bytes.GetHashString(new SHA1CryptoServiceProvider());
string sha256Hash = bytes.GetHashString(new SHA256CryptoServiceProvider());
string sha384Hash = bytes.GetHashString(new SHA384CryptoServiceProvider());
string sha512Hash = bytes.GetHashString(new SHA512CryptoServiceProvider());
Typically you'd just use MD5 or similar to create a hash. This isn't guaranteed to be unique though, so I'd recommend you use the hash as a starting point. Identify if the image matches any known hashes you stored, then individually load the ones that it does match and do a full byte comparison on the potential collisions to be sure.
Another, simpler technique though is to simply pick a smallish number of bits and read first part of the image... store that number of starting bits as if they were a hash. This still gives you a small number of potential collisions that you'd need to check, but has much less overhead.
Look in the System.Security.Cryptography namespace. You have your choice of several hashing algorithms/implementations. Here's an example using md5, but since you have a lot of these you might want something bigger like SHA1:
public byte[] HashImage(Stream imageData)
{
return new MD5CryptoServiceProvider().ComputeHash(imageData);
}
I don't know if it already exists or not, but I can't think of a reason you can't do this yourself. Something similar to this will get you a hash of the file.
var fileStream = Request.Files[0].InputStream;//the uploaded file
var hasher = System.Security.Cryptography.HMACMD5();
var theHash = hasher.ComputeHash(fileStream);
System.Security.Cryptography
I want to convert an object of System.Byte[] which is actually a file, retrieved from the database, back to file and store the file in a folder. How can we convert it using ASP.Net with C# ?
It's right way:
byte[] b = YourByteArrayFromDb;
File.WriteAllBytes(MyFilePath, b);
Do something like:
byte[] b = YourByteArrayFromDb;
string s = System.Text.ASCIIEncoding.ASCII.GetString(b);
File.WriteAllText(MyFilePath, s);
Given that this is an ASP.Net application, you may run into permission issues writing to the file system. Place your code in a try/catch block and provide a reasonable error message or log entry if you can't write the file.
If your original file was not encoded in ASCII, you will need to adjust accordingly.