How to process Excel file in memory? - .net-core

I am trying to create an API that will accept the representation of an Excel file from the client. I wish to return a List<List<string>> as JSON array after processing the first sheet. However, I cannot write the file to disk, and all processing must happen in-memory. What are the ways in which this can be achieved?
I've tried referring to various solutions on the internet but all of them involve writing the file to disk and then using that file for further processing. I'm open to solutions that involve
Accepting base-64 representation of the file from the POST request body
Accepting file as part of multipart/form-data request
Any other standard request formats that accept files
The only condition is that the API should return a JSON array representation of the spreadsheet.

Here I am sending a file as part of multipart/form-data request to the API which written in .NET core.
which support .xlsx , .xls and .csv format
use ExcelDataReader and ExcelDataReader.DataSet NuGet packages for reading excel and convert in the dataset.
Here one problem i faced and solution in .NET core.
By default, ExcelDataReader throws a NotSupportedException "No data is available for encoding 1252." on .NET Core.
To fix, add a dependency to the package System.Text.Encoding.CodePages and then add code to register the code page in starting of API
System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
This is required to parse strings in binary BIFF2-5 Excel documents encoded with DOS-era code pages. These encodings are registered by default in the full .NET Framework, but not on .NET Core.
public ActionResult ExcelOrCsvToArray()
{
if (Request.Form.Files.Count > 0)
{
IFormFile file = Request.Form.Files[0];
string fileName = file.FileName;
string fileContentType = file.ContentType;
System.Text.Encoding.RegisterProvider(System.Text.CodePagesEncodingProvider.Instance);
Stream stream = file.OpenReadStream();
try
{
if (fileName.EndsWith(".csv"))
{
using (var reader = ExcelReaderFactory.CreateCsvReader(stream))
{
var result = SetAsDataSet(reader);
DataTable table = result.Tables[0];
return new OkObjectResult(table);
}
}
else
{
using (var reader = ExcelReaderFactory.CreateReader(stream))
{
var result = SetAsDataSet(reader);
DataTable table = result.Tables[0];
return new OkObjectResult(table);
}
}
}
catch (Exception e)
{
return new BadRequestObjectResult(e);
}
}
else
{
return new BadRequestResult();
}
}
private DataSet SetAsDataSet(IExcelDataReader reader)
{
var result = reader.AsDataSet(new ExcelDataSetConfiguration()
{
ConfigureDataTable = (_) => new ExcelDataTableConfiguration()
{
UseHeaderRow = true,
}
});
return result;
}

Related

How to use "Azure storage blobs" for POST method in controller

I am creating an app where user can upload their text file and find out about its most used word.
I have tried to follow this doc to get used to the idea of using AZURE STORAGE BLOBS - https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet
But I am super newbie and having a hard time figuring it out how to adapt those blobs methods for my POST method.
This my sudo - what I think I need in my controller and what needs to happen when POST method is triggered.
a.No need for DELETE or PUT, not replacing the data nor deleting in this app
b.Maybe need a GET method, but as soon as POST method is triggered, it should pass the text context to the FE component
POST method
connect with azure storage account
if it is a first time of POST, create a container to store the text file
a. how can I connect with the existing container if the new container has already been made? I found this, but this is for the old CloudBlobContainer. Not the new SDK 12 version.
.GetContainerReference($"{containerName}");
upload the text file to the container
get the chosen file's text content and return
And here is my controller.
public class HomeController : Controller
{
private IConfiguration _configuration;
public HomeController(IConfiguration Configuration)
{
_configuration = Configuration;
}
public IActionResult Index()
{
return View();
}
[HttpPost("UploadText")]
public async Task<IActionResult> Post(List<IFormFile> files)
{
if (files != null)
{
try
{
string connectionString = Environment.GetEnvironmentVariable("AZURE_STORAGE_CONNECTION_STRING");
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
string containerName = "textdata" + Guid.NewGuid().ToString();
BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName);
//Q. How to write a if condition here so if the POST method has already triggered and container already created, just upload the data. Do not create a new container?
string fileName = //Q. how to get the chosen file name and replace with newly assignmed name?
string localFilePath = //Q. how to get the local file path so I can pass on to the FileStream?
BlobClient blobClient = containerClient.GetBlobClient(fileName);
using FileStream uploadFileStream = System.IO.File.OpenRead(localFilePath);
await blobClient.UploadAsync(uploadFileStream, true);
uploadFileStream.Close();
string data = System.IO.File.ReadAllText(localFilePath, Encoding.UTF8);
//Q. If I use fetch('Home').then... from FE component, will it receive this data? in which form will it receive? JSON?
return Content(data);
}
catch
{
//Q. how to use storageExeption for the error messages
}
finally
{
//Q. what is suitable to execute in finally? return the Content(data) here?
if (files != null)
{
//files.Close();
}
}
}
//Q. what to pass on inside of the Ok() in this scenario?
return Ok();
}
}
Q1. How can I check if the POST method has been already triggered, and created the Container? If so how can I get the container name and connect to it?
Q2. Should I give a new assigned name to the chosen file? How can I do so?
Q3. How can I get the chosen file's name so I can pass in order to process Q2?
Q4. How to get the local file path so I can pass on to the FileStream?
Q5. How to return the Content data and pass to the FE? by using fetch('Home').then... like this?
Q6. How can I use storageExeption for the error messages
Q7. What is suitable to execute in finally? return the Content(data) here?
Q8. What to pass on inside of the Ok() in this scenario?
Any help is welcomed! I know I asked a lot of Qs here. Thanks a lot!
Update: add a sample code, you can modify it as per your need.
[HttpPost]
public async Task<IActionResult> SaveFile(List<IFormFile> files)
{
if (files == null || files.Count == 0) return Content("file not selected");
string connectionString = "xxxxxxxx";
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
string containerName = "textdata" + Guid.NewGuid().ToString();;
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);
containerClient.CreateIfNotExists();
foreach (var file in files)
{
//use this line of code to get file name
string fileName = Path.GetFileName(file.FileName);
BlobClient blobClient = containerClient.GetBlobClient(fileName);
//directly read file content
using (var stream = file.OpenReadStream())
{
await blobClient.UploadAsync(stream);
}
}
//other code
return View();
}
Original answer:
When using List<IFormFile>, you should use foreach code block to iterate each file in the list.
Q2. Should I give a new assigned name to the chosen file? How can I do
so?
If you want to keep the file original name, in the foreach statement like below:
foreach (var file in myfiles)
{
Path.GetFileName(file.FileName)
//other code
}
And if you want to assign a new file name when uploaded to blob storage, you should define the new name in this line of code: BlobClient blobClient = containerClient.GetBlobClient("the new file name").
Q3. How can I get the chosen file's name so I can pass in order to
process Q2?
refer to Q2.
Q4. How to get the local file path so I can pass on to the FileStream?
You can use code like this: string localFilePath = file.FileName; to get the path, and then combine with the file name. But there is a better way, you can directly use this line of code Stream uploadFileStream = file.OpenReadStream().
Q5. How to return the Content data and pass to the FE? by using
fetch('Home').then... like this?
Not clear what's it meaning. Can you provide more details?
Q6. How can I use storageExeption for the error messages
The storageExeption does not exist in the latest version, you should install the older one.
You can refer to this link for more details.
#Ivan's answer is what the documentation seems the recommend; however, I was having a strange issue where my stream was always prematurely closed before the upload had time to complete. To anyone else who might run into this problem, going the BinaryData route helped me. Here's what that looks like:
await using var ms = new MemoryStream();
await file.CopyToAsync(ms);
var data = new BinaryData(ms.ToArray());
await blobClient.UploadAsync(data);

Null response creating file using Google Drive .NET API

I am trying to upload a file onto my Drive using Google Drive .NET API v3. My code is below
static string[] Scopes = { DriveService.Scope.Drive,
DriveService.Scope.DriveAppdata,
DriveService.Scope.DriveFile,
DriveService.Scope.DriveMetadataReadonly,
DriveService.Scope.DriveReadonly,
DriveService.Scope.DriveScripts };
static string ApplicationName = "Drive API .NET Quickstart";
public ActionResult Index()
{
UserCredential credential;
using (var stream =
new FileStream("C:/Users/admin1/Documents/visual studio 2017/Projects/TryGoogleDrive/TryGoogleDrive/client_secret.json", FileMode.Open, FileAccess.Read))
{
string credPath = Environment.GetFolderPath(
System.Environment.SpecialFolder.Personal);
credPath = Path.Combine(credPath, ".credentials/drive-dotnet-quickstart.json");
credential = GoogleWebAuthorizationBroker.AuthorizeAsync(
GoogleClientSecrets.Load(stream).Secrets,
Scopes,
"user",
CancellationToken.None,
new FileDataStore(credPath, true)).Result;
Debug.WriteLine("Credential file saved to: " + credPath);
}
// Create Drive API service.
var service = new DriveService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = ApplicationName,
});
// Define parameters of request.
FilesResource.ListRequest listRequest = service.Files.List();
listRequest.PageSize = 10;
listRequest.Fields = "nextPageToken, files(id, name)";
// List files.
IList<Google.Apis.Drive.v3.Data.File> files = listRequest.Execute()
.Files;
Debug.WriteLine("Files:");
if (files != null && files.Count > 0)
{
foreach (var file in files)
{
Debug.WriteLine("{0} ({1})", file.Name, file.Id);
}
}
else
{
Debug.WriteLine("No files found.");
}
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Name = "report.csv",
MimeType = "text/csv",
};
FilesResource.CreateMediaUpload request;
using (var stream = new FileStream("C:/debugging/report.csv",
FileMode.Open))
{
request = service.Files.Create(
fileMetadata, stream, "text/csv");
request.Fields = "id";
request.Upload();
}
var response = request.ResponseBody;
Console.WriteLine("File ID: " + response.Id);
return View();
}
The problem I'm facing is that response is always null. I looked into it a bit further and found that the request returned a 403 resultCode. I also took a look at some other questions on SO this and this but neither were of any help.
Edit: I forgot to mention that the first part of the code is working correctly - it lists all the files in my Drive. Only the second part is not working (the upload file part)
string[] Scopes = { DriveService.Scope.Drive };
Change the Drive scope then delete the file token.json
in vs2017 you can see token.json file in token.json folder when client_secret.json file present.
Try to visit this post from ASP.NET forum.
The same idea as what you want to do in your app, since you are dealing with uploading a file in Google Drive using .net.
You may try to call rest api directly to achieve your requirement :
The quickstart from .net will help you to make requests from/to the Drive API.
Upload Files:
The Drive API allows you to upload file data when create or
updating a File resource.
You can send upload requests in any of the following ways:
Simple upload: uploadType=media. For quick transfer of a small file (5 MB or less). To perform a simple upload, refer to Performing
a Simple Upload.
Multipart upload: uploadType=multipart. For quick transfer of a small file (5 MB or less) and metadata describing the file, all in a
single request. To perform a multipart upload, refer to Performing a
Multipart Upload.
Resumable upload: uploadType=resumable. For more reliable transfer, especially important with large files. Resumable uploads are
a good choice for most applications, since they also work for small
files at the cost of one additional HTTP request per upload. To
perform a resumable upload, refer to Performing a Resumable
Upload.
You may try this code from the documentation on uploading sample file.
var fileMetadata = new File()
{
Name = "photo.jpg"
};
FilesResource.CreateMediaUpload request;
using (var stream = new System.IO.FileStream("files/photo.jpg",
System.IO.FileMode.Open))
{
request = driveService.Files.Create(
fileMetadata, stream, "image/jpeg");
request.Fields = "id";
request.Upload();
}
var file = request.ResponseBody;
Console.WriteLine("File ID: " + file.Id);
You may check the errors you may encounter in this documentation.
Have a look at what request.Upload() returns. For me when I was having this issue it returned:
Insufficient Permission Errors [Message[Insufficient Permission] Location[ - ]
I changed my scope from DriveService.Scope.DriveReadonly to DriveService.Scope.Drive and I was in business.
Change static string[] Scopes = { DriveService.Scope.DriveReadonly }; to static string[] Scopes = { DriveService.Scope.Drive };.
After changes, take a look into token.json file and check does it change its scope from DriveReadonly to Drive.
If you are seeing DriveReadonly then delete the token.json file and run the application again.

Uploading multiple HttpPostedFileBase using Parallel.ForEach breaking files

I have a form that uploads multiple files. My model has a List<HttpPostedFileBase> called SchemaFileBases, which is correctly binded. I need to upload these files to s3 and would like to do it in parallel. I'm unable to use asyc and await because this code is run from both ASP.Net and a queue based application that currently doesn't have async/await support (working on it).
If I change the foreach below to Parallel.ForEach(this.SchemaFileBases, schemaFileBase => {... Then I get some funkiness going on. The two files end up being mashed. Each file will contain some of the other files content after it's uploaded. AwsDocument is being used elsewhere in parallel so I don't think it has to do with that. Each AwsDocument has it's own AmazonS3Client.
public override void UploadToS3(IMetadataParser parser)
{
string hash;
string key;
foreach (var schemaFileBase in this.SchemaFileBases)
{
AwsDocument aws = new AwsDocument(AwsBucket.Received);
hash = schemaFileBase.InputStream.Md5Hash().ToByteArray().ToHex();
key = String.Format("{0}/{1}", this.S3Prefix, schemaFileBase.FileName);
Stream inputStream = schemaFileBase.InputStream;
aws.UploadToS3(key, inputStream, hash);
}
}
My coworker suspect's it's something to do with how the InputStream on the HttpPostedFileBase is implemented. Perhaps it is not thread safe, and the streams are both reading from the original request at the same time? I can't imagine MS would do that though.
Multi-threaded version:
public override void UploadToS3(IMetadataParser parser)
{
Parallel.ForEach(this.SchemaFileBases, f =>
{
AwsDocument aws = new AwsDocument(AwsBucket.Received);
string hash = f.InputStream.Md5Hash().ToByteArray().ToHex();
string key = String.Format("{0}/{1}", this.S3Prefix, f.FileName);
Stream inputStream = f.InputStream;
aws.UploadToS3(key, inputStream, hash);
});
}
Above solution is what I tried to multi-thread it. Does not work (files get mixed up all weird).

Trying to get DHL shipping rates using ASP.Net but DHL's docs/samples are J2EE-geared. Anyone doing this in .Net that can provide some direction?

I'd like to get DHL shipping rates either per transaction or in batch all at once (to store in a table for later use) from an ASP.Net e-commerce application that ships product internationally, but after downloading their J2EE-based toolkit (https://xmlpi-ea.dhl.com) and reviewing the documentation & samples, I'm not quite sure how to do it in .Net. If anyone has experience with getting DHL shipping rates, I'd appreciate a point in the right direction using .Net. as I don't know Java.
Edit
Just found out the servlet is not discoverable, which means I cannot WSDL it to get a proxy class and will have to rely on tons of their XML samples to build my own client. Anyone done this in .NET already?
Looks like they have web services that you can use.
http://www.dhl-usa.com/en/express/resource_center/integrated_shipping_solutions.html
Sorry to be late. I just finished developing an integration and I give you here the way.
First you have to use xsd2code++ because the XSD.EXE from Microsoft doesn't work. Don't ask me why but it doesn't find the import included in the XSD file or maybe I didn't dig enough why and once I tried xsd2code++ it was a breeze to just right click the XSD in Visual Studio and use the option there.
Once you have your XSD converted to classes you consume it with the 3 methods bellow. See the 2 following lines of code that use the methods. Don't forget to add the necessary usings for XDocument.
Once you register on DHL web site you can download the DHL Toolkit PI which contains the folder XSD where all the XSD files are located.
NOTE : An alternative to Xsd2Code++ is Xsd2code on CodePlex : XSD2CODE hurry up because CodePlex is closing
string Request = XDocument.Parse(SerializeToXML(Quote)).ToString();
string Response = XDocument.Parse(SendRequest(Request)).ToString();
if (Response.IndexOf("DCTResponse") != -1)
DCTResponse = DeserializeFromXML<DHL.Response.DCTResponse>(Response);
else
DCTErrorResponse = DeserializeFromXML<DHL.Response.ErrorResponse>(Response);
public static string SendRequest(string XML)
{
string Response = "";
try
{
HttpWebRequest myReq = null;
myReq = WebRequest.Create(Properties.Settings.Default.DHLURL) as HttpWebRequest;
myReq.ContentType = "application/x-www-form-urlencoded";
myReq.Method = "POST";
using (System.IO.Stream stream = myReq.GetRequestStream())
{
byte[] arrBytes = ASCIIEncoding.ASCII.GetBytes(XML);
stream.Write(arrBytes, 0, arrBytes.Length);
stream.Close();
}
WebResponse myRes = myReq.GetResponse();
System.IO.Stream respStream = myRes.GetResponseStream();
System.IO.StreamReader reader = new System.IO.StreamReader(respStream, System.Text.Encoding.ASCII);
Response = reader.ReadToEnd();
myRes.Close();
myRes = null;
}
catch (Exception ex)
{
Response = ex.ToString();
}
return Response;
}
public static string SerializeToXML<T>(T toSerialize)
{
string Result = "";
XmlSerializerNamespaces ns = new XmlSerializerNamespaces(); ns.Add("", "");
using (TextWriter tw = new StringWriter())
{
using (XmlWriter writer = XmlWriter.Create(tw, new XmlWriterSettings { OmitXmlDeclaration = true }))
{
new XmlSerializer(typeof(T)).Serialize(writer, toSerialize, ns);
Result = tw.ToString();
}
}
return Result;
}
public static T DeserializeFromXML<T>(string xml)
{
var serializer = new XmlSerializer(typeof(T));
return (T)serializer.Deserialize(new StringReader(xml));
}
Hope this helps...

how do I handle and deflate a GZipped form post in asp.net MVC?

I have an iPad app that submits orders to an ASP.NET MVC web site via form post. It is posting JSON which can be fairly large for a mobile device to send (200~300K) under certain conditions. I can GZip the form post but then my asp.net mvc chokes on the gzipped content.
How can I handle a GZipped form post in asp.net mvc?
UPDATE:
Darin's answer puts me on the right track but I still have no idea how to do what he suggests, so here is where I am at:
Have this code to decompress a string:
http://dotnet-snippets.com/dns/compress-and-decompress-strings-SID612.aspx
And I get the string like so:
StreamReader reader = new StreamReader(Request.InputStream);
string encodedString = reader.ReadToEnd();
but this gives me the error:
The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or a non-white space character among the padding characters.
EDIT - COMPLETED CODE
I am using asp.net MVC and this is working great for me. I also had to deal with some other encoding that happens when my gzipping occurs:
[Authorize]
[HttpPost]
[ValidateInput(false)]
public ActionResult SubmitOrder()
{
GZipStream zipStream = new GZipStream(Request.InputStream, CompressionMode.Decompress);
byte[] streamBytes = ReadAllBytes(zipStream);
var result = Convert.ToBase64String(streamBytes);
string sample = System.Text.Encoding.UTF8.GetString(Convert.FromBase64String(result));
string escaped = Uri.UnescapeDataString(sample);
// escaped now has my form values as a string like so: var1=value1&var2=value2&ect...
//more boring code
}
public static byte[] ReadAllBytes(Stream input)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
You can do this without a custom model binder. Write an Action that accepts HttpPostedFileBase, i.e, treat this as a file upload.
[HttpPost]
public ActionResult UploadCompressedJSON(HttpPostedFileBase file)
{
if (file != null && file.ContentLength > 0)
{
GZipStream zipStream = new GZipStream(file.InputStream, CompressionMode.Decompress);
byte[] streamBytes = ReadAllBytes(zipStream);
var result = Convert.ToBase64String(streamBytes);
}
return RedirectToAction("Index");
}
You are going to need to change your client side code to send a file upload request but that should be fairly easy. For example you can look at this code.
How can I handle a GZipped form post in asp.net mvc?
You could write a custom model binder that will directly read the Request.InputStream, unzip it and then parse the contents and instantiate some view model you want to bind to.
Use the System.IO.Compression.GZipStream class.
Codeproject example

Resources