I am creating an app where user can upload their text file and find out about its most used word.
I have tried to follow this doc to get used to the idea of using AZURE STORAGE BLOBS - https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet
But I am super newbie and having a hard time figuring it out how to adapt those blobs methods for my POST method.
This my sudo - what I think I need in my controller and what needs to happen when POST method is triggered.
a.No need for DELETE or PUT, not replacing the data nor deleting in this app
b.Maybe need a GET method, but as soon as POST method is triggered, it should pass the text context to the FE component
POST method
connect with azure storage account
if it is a first time of POST, create a container to store the text file
a. how can I connect with the existing container if the new container has already been made? I found this, but this is for the old CloudBlobContainer. Not the new SDK 12 version.
.GetContainerReference($"{containerName}");
upload the text file to the container
get the chosen file's text content and return
And here is my controller.
public class HomeController : Controller
{
private IConfiguration _configuration;
public HomeController(IConfiguration Configuration)
{
_configuration = Configuration;
}
public IActionResult Index()
{
return View();
}
[HttpPost("UploadText")]
public async Task<IActionResult> Post(List<IFormFile> files)
{
if (files != null)
{
try
{
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
string containerName = "textdata" + Guid.NewGuid().ToString();
BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName);
//Q. How to write a if condition here so if the POST method has already triggered and container already created, just upload the data. Do not create a new container?
string fileName = //Q. how to get the chosen file name and replace with newly assignmed name?
string localFilePath = //Q. how to get the local file path so I can pass on to the FileStream?
BlobClient blobClient = containerClient.GetBlobClient(fileName);
using FileStream uploadFileStream = System.IO.File.OpenRead(localFilePath);
await blobClient.UploadAsync(uploadFileStream, true);
uploadFileStream.Close();
string data = System.IO.File.ReadAllText(localFilePath, Encoding.UTF8);
//Q. If I use fetch('Home').then... from FE component, will it receive this data? in which form will it receive? JSON?
return Content(data);
}
catch
{
//Q. how to use storageExeption for the error messages
}
finally
{
//Q. what is suitable to execute in finally? return the Content(data) here?
if (files != null)
{
//files.Close();
}
}
}
//Q. what to pass on inside of the Ok() in this scenario?
return Ok();
}
}
Q1. How can I check if the POST method has been already triggered, and created the Container? If so how can I get the container name and connect to it?
Q2. Should I give a new assigned name to the chosen file? How can I do so?
Q3. How can I get the chosen file's name so I can pass in order to process Q2?
Q4. How to get the local file path so I can pass on to the FileStream?
Q5. How to return the Content data and pass to the FE? by using fetch('Home').then... like this?
Q6. How can I use storageExeption for the error messages
Q7. What is suitable to execute in finally? return the Content(data) here?
Q8. What to pass on inside of the Ok() in this scenario?
Any help is welcomed! I know I asked a lot of Qs here. Thanks a lot!
Update: add a sample code, you can modify it as per your need.
[HttpPost]
public async Task<IActionResult> SaveFile(List<IFormFile> files)
{
if (files == null || files.Count == 0) return Content("file not selected");
string connectionString = "xxxxxxxx";
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
string containerName = "textdata" + Guid.NewGuid().ToString();;
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
containerClient.CreateIfNotExists();
foreach (var file in files)
{
//use this line of code to get file name
string fileName = Path.GetFileName(file.FileName);
BlobClient blobClient = containerClient.GetBlobClient(fileName);
//directly read file content
using (var stream = file.OpenReadStream())
{
await blobClient.UploadAsync(stream);
}
}
//other code
return View();
}
Original answer:
When using List<IFormFile>, you should use foreach code block to iterate each file in the list.
Q2. Should I give a new assigned name to the chosen file? How can I do
so?
If you want to keep the file original name, in the foreach statement like below:
foreach (var file in myfiles)
{
Path.GetFileName(file.FileName)
//other code
}
And if you want to assign a new file name when uploaded to blob storage, you should define the new name in this line of code: BlobClient blobClient = containerClient.GetBlobClient("the new file name").
Q3. How can I get the chosen file's name so I can pass in order to
process Q2?
refer to Q2.
Q4. How to get the local file path so I can pass on to the FileStream?
You can use code like this: string localFilePath = file.FileName; to get the path, and then combine with the file name. But there is a better way, you can directly use this line of code Stream uploadFileStream = file.OpenReadStream().
Q5. How to return the Content data and pass to the FE? by using
fetch('Home').then... like this?
Not clear what's it meaning. Can you provide more details?
Q6. How can I use storageExeption for the error messages
The storageExeption does not exist in the latest version, you should install the older one.
You can refer to this link for more details.
#Ivan's answer is what the documentation seems the recommend; however, I was having a strange issue where my stream was always prematurely closed before the upload had time to complete. To anyone else who might run into this problem, going the BinaryData route helped me. Here's what that looks like:
await using var ms = new MemoryStream();
await file.CopyToAsync(ms);
var data = new BinaryData(ms.ToArray());
await blobClient.UploadAsync(data);
I have the following code to save an excel file on the server and then read its content:
if (file.Length > 0)
{
string path = _hostingEnvironment.ContentRootPath + "/CSV-import-students/";
FileStream fs = new FileStream(Path.Combine(path, file.FileName), FileMode.Create);
await file.CopyToAsync(fs);
FileInfo fileUploaded = new FileInfo(Path.Combine(path, file.FileName));
using (ExcelPackage package = new ExcelPackage(fileUploaded))
{
StringBuilder sb = new StringBuilder();
ExcelWorksheet worksheet = package.Workbook.Worksheets[0];
int rowCount = worksheet.Dimension.Rows;
int ColCount = worksheet.Dimension.Columns;
bool bHeaderRow = true;
}
The file is saved fine on the server. But, then when I try to access it, I receive "Process cannot access file because it is being used by another process" error. How can I prevent this error? Any ideas?
Almost invariably, when newing up a class that implements IDisposable (such as FileStream), you should do so with a using statement:
using (var fs = new FileStream(Path.Combine(path, file.FileName), FileMode.Create))
{
await file.CopyToAsync(fs);
}
This will automatically dispose of the resource when the using statement goes out of scope, and in the case of FileStream, will flush the write and close the file. That's the source of your issue, so this will solve your problem.
However, you might also need to contend with concurrency. It's possible for two requests to be processed simultaneously that both need to work with the same file. You should plan for concurrency, by catching file access violation exceptions and responding via a retry policy. The Polly exception handling library can help here.
I have a form that uploads multiple files. My model has a List<HttpPostedFileBase> called SchemaFileBases, which is correctly binded. I need to upload these files to s3 and would like to do it in parallel. I'm unable to use asyc and await because this code is run from both ASP.Net and a queue based application that currently doesn't have async/await support (working on it).
If I change the foreach below to Parallel.ForEach(this.SchemaFileBases, schemaFileBase => {... Then I get some funkiness going on. The two files end up being mashed. Each file will contain some of the other files content after it's uploaded. AwsDocument is being used elsewhere in parallel so I don't think it has to do with that. Each AwsDocument has it's own AmazonS3Client.
public override void UploadToS3(IMetadataParser parser)
{
string hash;
string key;
foreach (var schemaFileBase in this.SchemaFileBases)
{
AwsDocument aws = new AwsDocument(AwsBucket.Received);
hash = schemaFileBase.InputStream.Md5Hash().ToByteArray().ToHex();
key = String.Format("{0}/{1}", this.S3Prefix, schemaFileBase.FileName);
Stream inputStream = schemaFileBase.InputStream;
aws.UploadToS3(key, inputStream, hash);
}
}
My coworker suspect's it's something to do with how the InputStream on the HttpPostedFileBase is implemented. Perhaps it is not thread safe, and the streams are both reading from the original request at the same time? I can't imagine MS would do that though.
Multi-threaded version:
public override void UploadToS3(IMetadataParser parser)
{
Parallel.ForEach(this.SchemaFileBases, f =>
{
AwsDocument aws = new AwsDocument(AwsBucket.Received);
string hash = f.InputStream.Md5Hash().ToByteArray().ToHex();
string key = String.Format("{0}/{1}", this.S3Prefix, f.FileName);
Stream inputStream = f.InputStream;
aws.UploadToS3(key, inputStream, hash);
});
}
Above solution is what I tried to multi-thread it. Does not work (files get mixed up all weird).
My server has an API to upload files and convert them to PDF. Right now, the file gets uploaded, saved to disk and then converted. See the (trimmed down) code below:
public class ConversionController : ApiController {
public async Task<HttpResponseMessage> PostData() {
var root = HttpContext.Current.Server.MapPath("~/App_Data");
var provider = new MultipartFormDataStreamProvider(root);
await Request.Content.ReadAsMultipartAsync(provider);
var file = provider.FileData.First();
var originalName = file.Headers.ContentDisposition.FileName;
var fileStream = new FileStream(file.LocalFileName, FileMode.Open, FileAccess.Read);
// convert file stream and return the PDF response ...
}
}
As you can see, I read the file to disk but then immediately get a stream for it so I can feed it to our conversion function (which takes a stream). This seems like a waste to save the file to disk every time. So instead of ReadAsMultipartAsync() which saves to disk, I can use ReadAsStreamAsync() which will give me the stream that I can give directly to the conversion function.
The problem that I'm having with ReadAsMultipartAsync() is that I can't figure out how to get the original file name without having the MultipartFileData instance to work with. I know that the name comes with the request as part of the body, but I can't figure out how to access it. How can I get the name of the uploaded file without writing the uploaded file to disk?
You can use MultipartMemoryStreamProvider, for example:
var provider = new MultipartMemoryStreamProvider();
var task = Request.Content.ReadAsMultipartAsync(provider).ContinueWith(t =>
{
var file = provider.Contents.First();
var fileContents = await file.ReadAsByteArrayAsync();
var filename = file.Headers.ContentDisposition.FileName.Replace("\"", string.Empty);
/// do other stuff
return Request.CreateResponse(HttpStatusCode.OK);
});
In this case the content is read as a byte array, but the same applies to streams.
I am creating a client side application which needs to create a log of the user activity but for various reasons this log must not be human readable.
Currently for my development I am creating a plain text log which looks something like this:
12/03/2009 08:34:21 -> User 'Bob' logged in
12/03/2009 08:34:28 -> Navigated to config page
12/03/2009 08:34:32 -> Option x changed to y
When I deploy my application, the log must not be in plain text, so all text must be encrypted. This doesn't appear to be straightforward to achieve as I need the log file to dynamically update as each entry is added.
The approach I was thinking about was to create a binary file, encrypt each log entry in isolation and then append it to the binary file with some suitable demarcation between each entry.
Does anyone know of any common approaches to this problem, I'm sure there has to be a better solution!
Don't encrypt individual log entries separately and write them to a file as suggested by other posters, because an attacker would easily be able to identify patterns in the log file. See the block cipher modes Wikipedia entry to learn more about this problem.
Instead, make sure that the encryption of a log entry depends on the previous log entries. Although this has some drawbacks (you cannot decrypt individual log entries as you always need to decrypt the entire file), it makes the encryption a lot stronger. For our own logging library, SmartInspect, we use AES encryption and the CBC mode to avoid the pattern problem. Feel free to give SmartInspect a try if a commercial solution would be suitable.
This is not really my thing, I'll admit that readily, but can't you encrypt each entry individually and then append it to the logfile? If you that refrain from encrypting the timestamp, you can easily find entries your are looking for and decrypt those when needed.
My point being mainly that appending individual encrypted entries to a file does not necessarily need to be binary entries appended to a binary file. Encryption with (for example) gpg will yield ascii garble that can be appended to an ascii file. Would that solve you problem?
FWIW, the one time I needed an encrypted logger I used a symmetric key (for performance reasons) to encrypt the actual log entries.
The symmetric 'log file key' was then encrypted under a public key and stored at the beginning of the log file and a separate log reader used the private key to decrypt the 'log file key' and read the entries.
The whole thing was implemented using log4j and an XML log file format (to make it easier for the reader to parse) and each time the log files were rolled over a new 'log file key' was generated.
Assuming you're using some sort of logging framework, e.g., log4j et al, then you should be able to create a custom implementation of Appender (or similar) that encrypts each entry, as #wzzrd suggested.
It is not clear to me wheter your concern is on the security, or the implement.
A simple implement is to hook up with a stream encryptor. A stream encryptor maintains its own state and can encrypt on the fly.
StreamEncryptor<AES_128> encryptor;
encryptor.connectSink(new std::ofstream("app.log"));
encryptor.write(line);
encryptor.write(line2);
...
Very old question and I'm sure the tech world has made much progress, but FWIW Bruce Schneier and John Kelsey wrote a paper on how to do this: https://www.schneier.com/paper-auditlogs.html
The context is not just security but also preventing the corruption or change of existing log file data if the system that hosts the log/audit files is compromised.
Encrypting each log entry individually would decrease the security of your ciphertext a lot, especially because you're working with very predictable plaintext.
Here's what you can do:
Use symmetric encryption (preferably AES)
Pick a random master key
Pick a security window (5 minutes, 10 minutes, etc.)
Then, pick a random temporary key at the beginning of each window (every 5 minutes, every 10 minutes, etc.)
Encrypt each log item separately using the temporary key and append to a temporary log file.
When the window's closed (the predetermined time is up), decrypt each element using the temporary key, decrypt the master log file using the master key, merge the files, and encrypt using the master key.
Then, pick a new temporary key and continue.
Also, change the master key each time you rotate your master log file (every day, every week, etc.)
This should provide enough security.
I'm wondering what kind of application you write. A virus or a Trojan horse? Anyway ...
Encrypt each entry alone, convert it to some string (Base64, for example) and then log that string as the "message".
This allows you to keep parts of the file readable and only encrypt important parts.
Notice that there is another side to this coin: If you create a fully encrypted file and ask the user for it, she can't know what you will learn from the file. Therefore, you should encrypt as little as possible (passwords, IP addresses, costumer data) to make it possible for the legal department to verify what data is leaving.
A much better approach would be to an obfuscator for the log file. That simply replaces certain patterns with "XXX". You can still see what happened and when you need a specific piece of data, you can ask for that.
[EDIT] This story has more implications that you'd think at first glance. This effectively means that a user can't see what's in the file. "User" doesn't necessarily include "cracker". A cracker will concentrate on encrypted files (since they are probably more important). That's the reason for the old saying: As soon as someone gets access to the machine, there is no way to prevent him to do anything on it. Or to say it another way: Just because you don't know how doesn't mean someone else also doesn't. If you think you have nothing to hide, you haven't thought about yourself.
Also, there is the issue of liability. Say, some data leaks on the Internet after you get a copy of the logs. Since the user has no idea what is in the log files, how can you prove in court that you weren't the leak? Bosses could ask for the log files to monitor their pawns, asking to have it encoded so the peasants can't notice and whine about it (or sue, the scum!).
Or look at it from a completely different angle: If there was no log file, no one could abuse it. How about enabling debugging only in case of an emergency? I've configured log4j to keep the last 200 log messages in a buffer. If an ERROR is logged, I dump the 200 messages to the log. Rationale: I really don't care what happens during the day. I only care for bugs. Using JMX, it's simple to set the debug level to ERROR and lower it remotely at runtime when you need more details.
For .Net see Microsoft Application blocks for log and encrypt functionality:
http://msdn.microsoft.com/en-us/library/dd203099.aspx
I would append encrypted log entries to a flat text file using suitable demarcation between each entry for the decryption to work.
I have the exact same need as you. Some guy called 'maybeWeCouldStealAVa' wrote a good implementation in: How to append to AES encrypted file , however this suffered from not being flushable - you would have to close and reopen the file each time you flush a message, to be sure not to lose anything.
So I've written my own class to do this:
import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.security.*;
public class FlushableCipherOutputStream extends OutputStream
{
private static int HEADER_LENGTH = 16;
private SecretKeySpec key;
private RandomAccessFile seekableFile;
private boolean flushGoesStraightToDisk;
private Cipher cipher;
private boolean needToRestoreCipherState;
/** the buffer holding one byte of incoming data */
private byte[] ibuffer = new byte[1];
/** the buffer holding data ready to be written out */
private byte[] obuffer;
/** Each time you call 'flush()', the data will be written to the operating system level, immediately available
* for other processes to read. However this is not the same as writing to disk, which might save you some
* data if there's a sudden loss of power to the computer. To protect against that, set 'flushGoesStraightToDisk=true'.
* Most people set that to 'false'. */
public FlushableCipherOutputStream(String fnm, SecretKeySpec _key, boolean append, boolean _flushGoesStraightToDisk)
throws IOException
{
this(new File(fnm), _key, append,_flushGoesStraightToDisk);
}
public FlushableCipherOutputStream(File file, SecretKeySpec _key, boolean append, boolean _flushGoesStraightToDisk)
throws IOException
{
super();
if (! append)
file.delete();
seekableFile = new RandomAccessFile(file,"rw");
flushGoesStraightToDisk = _flushGoesStraightToDisk;
key = _key;
try {
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
byte[] iv = new byte[16];
byte[] headerBytes = new byte[HEADER_LENGTH];
long fileLen = seekableFile.length();
if (fileLen % 16L != 0L) {
throw new IllegalArgumentException("Invalid file length (not a multiple of block size)");
} else if (fileLen == 0L) {
// new file
// You can write a 16 byte file header here, including some file format number to represent the
// encryption format, in case you need to change the key or algorithm. E.g. "100" = v1.0.0
headerBytes[0] = 100;
seekableFile.write(headerBytes);
// Now appending the first IV
SecureRandom sr = new SecureRandom();
sr.nextBytes(iv);
seekableFile.write(iv);
cipher.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec(iv));
} else if (fileLen <= 16 + HEADER_LENGTH) {
throw new IllegalArgumentException("Invalid file length (need 2 blocks for iv and data)");
} else {
// file length is at least 2 blocks
needToRestoreCipherState = true;
}
} catch (InvalidKeyException e) {
throw new IOException(e.getMessage());
} catch (NoSuchAlgorithmException e) {
throw new IOException(e.getMessage());
} catch (NoSuchPaddingException e) {
throw new IOException(e.getMessage());
} catch (InvalidAlgorithmParameterException e) {
throw new IOException(e.getMessage());
}
}
/**
* Writes one _byte_ to this output stream.
*/
public void write(int b) throws IOException {
if (needToRestoreCipherState)
restoreStateOfCipher();
ibuffer[0] = (byte) b;
obuffer = cipher.update(ibuffer, 0, 1);
if (obuffer != null) {
seekableFile.write(obuffer);
obuffer = null;
}
}
/** Writes a byte array to this output stream. */
public void write(byte data[]) throws IOException {
write(data, 0, data.length);
}
/**
* Writes <code>len</code> bytes from the specified byte array
* starting at offset <code>off</code> to this output stream.
*
* #param data the data.
* #param off the start offset in the data.
* #param len the number of bytes to write.
*/
public void write(byte data[], int off, int len) throws IOException
{
if (needToRestoreCipherState)
restoreStateOfCipher();
obuffer = cipher.update(data, off, len);
if (obuffer != null) {
seekableFile.write(obuffer);
obuffer = null;
}
}
/** The tricky stuff happens here. We finalise the cipher, write it out, but then rewind the
* stream so that we can add more bytes without padding. */
public void flush() throws IOException
{
try {
if (needToRestoreCipherState)
return; // It must have already been flushed.
byte[] obuffer = cipher.doFinal();
if (obuffer != null) {
seekableFile.write(obuffer);
if (flushGoesStraightToDisk)
seekableFile.getFD().sync();
needToRestoreCipherState = true;
}
} catch (IllegalBlockSizeException e) {
throw new IOException("Illegal block");
} catch (BadPaddingException e) {
throw new IOException("Bad padding");
}
}
private void restoreStateOfCipher() throws IOException
{
try {
// I wish there was a more direct way to snapshot a Cipher object, but it seems there's not.
needToRestoreCipherState = false;
byte[] iv = cipher.getIV(); // To help avoid garbage, re-use the old one if present.
if (iv == null)
iv = new byte[16];
seekableFile.seek(seekableFile.length() - 32);
seekableFile.read(iv);
byte[] lastBlockEnc = new byte[16];
seekableFile.read(lastBlockEnc);
cipher.init(Cipher.DECRYPT_MODE, key, new IvParameterSpec(iv));
byte[] lastBlock = cipher.doFinal(lastBlockEnc);
seekableFile.seek(seekableFile.length() - 16);
cipher.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec(iv));
byte[] out = cipher.update(lastBlock);
assert out == null || out.length == 0;
} catch (Exception e) {
throw new IOException("Unable to restore cipher state");
}
}
public void close() throws IOException
{
flush();
seekableFile.close();
}
}
Here's an example of using it:
import org.junit.Test;
import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.io.BufferedWriter;
public class TestFlushableCipher {
private static byte[] keyBytes = new byte[] {
// Change these numbers, lest other StackOverflow readers can decrypt your files.
-53, 93, 59, 108, -34, 17, -72, -33, 126, 93, -62, -50, 106, -44, 17, 55
};
private static SecretKeySpec key = new SecretKeySpec(keyBytes,"AES");
private static int HEADER_LENGTH = 16;
private static BufferedWriter flushableEncryptedBufferedWriter(File file, boolean append) throws Exception
{
FlushableCipherOutputStream fcos = new FlushableCipherOutputStream(file, key, append, false);
return new BufferedWriter(new OutputStreamWriter(fcos, "UTF-8"));
}
private static InputStream readerEncryptedByteStream(File file) throws Exception
{
FileInputStream fin = new FileInputStream(file);
byte[] iv = new byte[16];
byte[] headerBytes = new byte[HEADER_LENGTH];
if (fin.read(headerBytes) < HEADER_LENGTH)
throw new IllegalArgumentException("Invalid file length (failed to read file header)");
if (headerBytes[0] != 100)
throw new IllegalArgumentException("The file header does not conform to our encrypted format.");
if (fin.read(iv) < 16) {
throw new IllegalArgumentException("Invalid file length (needs a full block for iv)");
}
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
cipher.init(Cipher.DECRYPT_MODE, key, new IvParameterSpec(iv));
return new CipherInputStream(fin,cipher);
}
private static BufferedReader readerEncrypted(File file) throws Exception
{
InputStream cis = readerEncryptedByteStream(file);
return new BufferedReader(new InputStreamReader(cis));
}
#Test
public void test() throws Exception {
File zfilename = new File("c:\\WebEdvalData\\log.x");
BufferedWriter cos = flushableEncryptedBufferedWriter(zfilename, false);
cos.append("Sunny ");
cos.append("and green. \n");
cos.close();
int spaces=0;
for (int i = 0; i<10; i++) {
cos = flushableEncryptedBufferedWriter(zfilename, true);
for (int j=0; j < 2; j++) {
cos.append("Karelia and Tapiola" + i);
for (int k=0; k < spaces; k++)
cos.append(" ");
spaces++;
cos.append("and other nice things. \n");
cos.flush();
tail(zfilename);
}
cos.close();
}
BufferedReader cis = readerEncrypted(zfilename);
String msg;
while ((msg=cis.readLine()) != null) {
System.out.println(msg);
}
cis.close();
}
private void tail(File filename) throws Exception
{
BufferedReader infile = readerEncrypted(filename);
String last = null, secondLast = null;
do {
String msg = infile.readLine();
if (msg == null)
break;
if (! msg.startsWith("}")) {
secondLast = last;
last = msg;
}
} while (true);
if (secondLast != null)
System.out.println(secondLast);
System.out.println(last);
System.out.println();
}
}