I'm using asp.net core and web api for uploading images.
On server:
[Produces("application/json")]
[Route("api/Upload")]
public class UploadApiController : Controller
{
private readonly IHostingEnvironment _environment;
public UploadApiController(IHostingEnvironment environment)
{
_environment = environment;
}
[HttpPost]
public async Task Post(ICollection<IFormFile> files)
{
//...
}
}
On client:
// Files is an array that contains all temporary images for uploading.
let Files = [];
let image_preview = function (file) {
file['Alternate'] = 'alternate_text';
Files.push(file);
// other implements...
};
$('button#upload').click(function () {
let formData = new FormData();
for (let i = 0; i < Files.length; i++) {
formData.append('files', Files[i])
}
let xhr = new XMLHttpRequest();
xhr.open('POST', '/api/upload', true);
xhr.onload = function () {
console.log('uploading...')
};
xhr.send(formData);
});
Snapshot:
My question: how to add new property "Alternate" to ICollection<IFormFile> files to detect property Alternate that is sent from client (formData)?
It's not the answer for question How to add property “Alternate” to an image file on server? but it seems like solving the problem (sending image file with alternate text).
On server:
using Newtonsoft.Json;
[HttpPost]
public async Task Post(ICollection<IFormFile> files, IList<string> alts)
{
IDictionary<string, string> _alts = new Dictionary<string, string>();
foreach (var alt in alts)
{
IDictionary<string, string> temp = JsonConvert.DeserializeObject<Dictionary<string, string>>(alt);
foreach (var item in temp)
{
_alts.Add(item.Key, item.Value);
}
}
}
On client:
for (let i = 0; i < Files.length; i++) {
formData.append('files', Files[i]);
let name = Files[i]['name'],
alt = {};
alt[name] = 'alt_text';
formData.append('alts', JSON.stringify(alt));
}
We would never get duplicate key in the dictionary because Files[i]['name'] is always primary and cannot be changed (hacking by someone) if we've checked duplicate uploading file before.
Then, we can merge the file name (in files) with Key in _alts to get Alternate text.
Snapshot:
UPDATE: The code in the snapshot was wrong.
Related
I have a service bus triggered Azure Function which listens for messages containing just blob URL strings of JSON data which each one of them is at least 10MB.
Message queue is near real-time(If I use the correct term) so producers keep putting messaging to the queue with a frequency so there is always data in the queue to be processed.
I have designed a solution but it gets OutOfMemoryException most of the time. The steps involved in the current solution sequentially are:
Consume a message
Download the file from the URL within the consumed message to a temporary folder
Read the whole file as a string
Deserialize it to an object
Partition into the chunks to supply Mongo bulk upsert limit
Bulk upsert to Mongo
I have tried to solve OutOfMemoryException and I thought that it's because my function/consumer don't have the same pace with the producer, so I think that at the time t1 when it gets the first message and process it and then while it's upserting to the mongo the function keeps getting the messages and they accumulate in the memory and waiting to be upserted.
Is my reasoning right?
Thus I think that If I could implement a streaming solution starting from #3, reading from file by chunking and putting it to a stream then I would prevent the memory keep growing and reduce time also. I have mostly Java background and I somehow know that with custom iterator/spliterator/iterable it is possible to do streaming and asynchronous processing.
How can I do asynchronous data streaming with .Net Core in an Azure Function?
Are there other approaches to solve this problem?
namespace x.y.Z
{
public class MyFunction
{
//...
[FunctionName("my-func")]
public async Task Run([ServiceBusTrigger("my-topic", "my-subscription", Connection = "AzureServiceBus")] string message, ILogger log, ExecutionContext context)
{
var data = new PredictionMessage();
try
{
data = myPredictionService.genericDeserialize(message);
await myPredictionService.ValidateAsync(data);
await myPredictionService.AddAsync(data);
}
catch (Exception ex)
{
//...
}
}
}
}
public class PredictionMessage
{
public string BlobURL { get; set; }
}
namespace x.y.z.Prediction
{
public abstract class BasePredictionService<T> : IBasePredictionService<T> where T : PredictionMessage, new()
{
protected readonly ILogger log;
private static JsonSerializer serializer;
public BasePredictionService(ILogger<BasePredictionService<T>> log)
{
this.log = log;
serializer = new JsonSerializer();
}
public async Task ValidateAsync(T message)
{
//...
}
public T genericDeserialize(string message)
{
return JsonConvert.DeserializeObject<T>(message);
}
public virtual Task AddAsync(T message)
{
throw new System.NotImplementedException();
}
public async Task<string> SerializePredictionResult(T message)
{
var result = string.Empty;
using (WebClient client = new WebClient())
{
var tempPath = Path.Combine(Path.GetTempPath(), DateTime.Now.Ticks + ".json");
Uri srcPath = new Uri(message.BlobURL);
await client.DownloadFileTaskAsync(srcPath, tempPath);
using (FileStream fs = File.Open(tempPath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs))
{
result = sr.ReadToEnd();
}
}
Task.Run(() =>
{
File.Delete(tempPath);
});
return result;
}
}
protected TType StreamDataDeserialize<TType>(string streamResult)
{
var body = default(TType);
using (MemoryStream stream = new MemoryStream(Encoding.Default.GetBytes(streamResult)))
{
using (StreamReader streamReader = new StreamReader(stream))
{
body = (TType)serializer.Deserialize(streamReader, typeof(TType));
}
}
return body;
}
protected List<List<TType>> Split<TType>(List<TType> list, int chunkSize = 1000)
{
List<List<TType>> retVal = new List<List<TType>>();
while (list.Count > 0)
{
int count = list.Count > chunkSize ? chunkSize : list.Count;
retVal.Add(list.GetRange(0, count));
list.RemoveRange(0, count);
}
return retVal;
}
}
}
namespace x.y.z.Prediction
{
public class MyPredictionService : BasePredictionService<PredictionMessage>, IMyPredictionService
{
private readonly IMongoDBRepository<MyPrediction> repository;
public MyPredictionService(IMongoDBRepoFactory mongoDBRepoFactory, ILogger<MyPredictionService> log) : base(log)
{
repository = mongoDBRepoFactory.GetRepo<MyPrediction>();
}
public override async Task AddAsync(PredictionMessage message)
{
string streamResult = await base.SerializePredictionResult(message);
var body = base.StreamDataDeserialize<List<MyPrediction>>(streamResult);
if (body != null && body.Count > 0)
{
var chunkList = base.Split(body);
await BulkUpsertProcess(chunkList);
}
}
private async Task BulkUpsertProcess(List<List<MyPrediction>> chunkList)
{
foreach (var perChunk in chunkList)
{
var filterContainers = new List<IDictionary<string, object>>();
var updateContainer = new List<IDictionary<string, object>>();
foreach (var item in perChunk)
{
var filter = new Dictionary<string, object>();
var update = new Dictionary<string, object>();
filter.Add(/*...*/);
filterContainers.Add(filter);
update.Add(/*...*/);
updateContainer.Add(update);
}
await Task.Run(async () =>
{
await repository.BulkUpsertAsync(filterContainers, updateContainer);
});
}
}
}
}
I'm using Asp.Net Core as a Rest Api Service.
I need access to request and response in ActionFilter. Actually, I found the request in OnActionExcecuted but I can't read the response result.
I'm trying to return value as follow:
[HttpGet]
[ProducesResponseType(typeof(ResponseType), (int)HttpStatusCode.OK)]
[Route("[action]")]
public async Task<IActionResult> Get(CancellationToken cancellationToken)
{
var model = await _responseServices.Get(cancellationToken);
return Ok(model);
}
And in ActionFilter OnExcecuted method as follow:
_request = context.HttpContext.Request.ReadAsString().Result;
_response = context.HttpContext.Response.ReadAsString().Result; //?
I'm trying to get the response in ReadAsString as an Extension method as follow:
public static async Task<string> ReadAsString(this HttpResponse response)
{
var initialBody = response.Body;
var buffer = new byte[Convert.ToInt32(response.ContentLength)];
await response.Body.ReadAsync(buffer, 0, buffer.Length);
var body = Encoding.UTF8.GetString(buffer);
response.Body = initialBody;
return body;
}
But, there is no result!
How I can get the response in OnActionExcecuted?
Thanks, everyone for taking the time to try and help explain
If you're logging for json result/ view result , you don't need to read the whole response stream. Simply serialize the context.Result:
public class MyFilterAttribute : ActionFilterAttribute
{
private ILogger<MyFilterAttribute> logger;
public MyFilterAttribute(ILogger<MyFilterAttribute> logger){
this.logger = logger;
}
public override void OnActionExecuted(ActionExecutedContext context)
{
var result = context.Result;
if (result is JsonResult json)
{
var x = json.Value;
var status = json.StatusCode;
this.logger.LogInformation(JsonConvert.SerializeObject(x));
}
if(result is ViewResult view){
// I think it's better to log ViewData instead of the finally rendered template string
var status = view.StatusCode;
var x = view.ViewData;
var name = view.ViewName;
this.logger.LogInformation(JsonConvert.SerializeObject(x));
}
else{
this.logger.LogInformation("...");
}
}
I know there is already an answer but I want to also add that the problem is the MVC pipeline has not populated the Response.Body when running an ActionFilter so you cannot access it. The Response.Body is populated by the MVC middleware.
If you want to read Response.Body then you need to create your own custom middleware to intercept the call when the Response object has been populated. There are numerous websites that can show you how to do this. One example is here.
As discussed in the other answer, if you want to do it in an ActionFilter you can use the context.Result to access the information.
For logging whole request and response in the ASP.NET Core filter pipeline you can use Result filter attribute
public class LogRequestResponseAttribute : TypeFilterAttribute
{
public LogRequestResponseAttribute() : base(typeof(LogRequestResponseImplementation)) { }
private class LogRequestResponseImplementation : IAsyncResultFilter
{
public async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next)
{
var requestHeadersText = CommonLoggingTools.SerializeHeaders(context.HttpContext.Request.Headers);
Log.Information("requestHeaders: " + requestHeadersText);
var requestBodyText = await CommonLoggingTools.FormatRequestBody(context.HttpContext.Request);
Log.Information("requestBody: " + requestBodyText);
await next();
var responseHeadersText = CommonLoggingTools.SerializeHeaders(context.HttpContext.Response.Headers);
Log.Information("responseHeaders: " + responseHeadersText);
var responseBodyText = await CommonLoggingTools.FormatResponseBody(context.HttpContext.Response);
Log.Information("responseBody: " + responseBodyText);
}
}
}
In Startup.cs add
app.UseMiddleware<ResponseRewindMiddleware>();
services.AddScoped<LogRequestResponseAttribute>();
Somewhere add static class
public static class CommonLoggingTools
{
public static async Task<string> FormatRequestBody(HttpRequest request)
{
//This line allows us to set the reader for the request back at the beginning of its stream.
request.EnableRewind();
//We now need to read the request stream. First, we create a new byte[] with the same length as the request stream...
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
//...Then we copy the entire request stream into the new buffer.
await request.Body.ReadAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
//We convert the byte[] into a string using UTF8 encoding...
var bodyAsText = Encoding.UTF8.GetString(buffer);
//..and finally, assign the read body back to the request body, which is allowed because of EnableRewind()
request.Body.Position = 0;
return $"{request.Scheme} {request.Host}{request.Path} {request.QueryString} {bodyAsText}";
}
public static async Task<string> FormatResponseBody(HttpResponse response)
{
//We need to read the response stream from the beginning...
response.Body.Seek(0, SeekOrigin.Begin);
//...and copy it into a string
string text = await new StreamReader(response.Body).ReadToEndAsync();
//We need to reset the reader for the response so that the client can read it.
response.Body.Seek(0, SeekOrigin.Begin);
response.Body.Position = 0;
//Return the string for the response, including the status code (e.g. 200, 404, 401, etc.)
return $"{response.StatusCode}: {text}";
}
public static string SerializeHeaders(IHeaderDictionary headers)
{
var dict = new Dictionary<string, string>();
foreach (var item in headers.ToList())
{
//if (item.Value != null)
//{
var header = string.Empty;
foreach (var value in item.Value)
{
header += value + " ";
}
// Trim the trailing space and add item to the dictionary
header = header.TrimEnd(" ".ToCharArray());
dict.Add(item.Key, header);
//}
}
return JsonConvert.SerializeObject(dict, Formatting.Indented);
}
}
public class ResponseRewindMiddleware {
private readonly RequestDelegate next;
public ResponseRewindMiddleware(RequestDelegate next) {
this.next = next;
}
public async Task Invoke(HttpContext context) {
Stream originalBody = context.Response.Body;
try {
using (var memStream = new MemoryStream()) {
context.Response.Body = memStream;
await next(context);
//memStream.Position = 0;
//string responseBody = new StreamReader(memStream).ReadToEnd();
memStream.Position = 0;
await memStream.CopyToAsync(originalBody);
}
} finally {
context.Response.Body = originalBody;
}
}
You can also do...
string response = "Hello";
if (result is ObjectResult objectResult)
{
var status = objectResult.StatusCode;
var value = objectResult.Value;
var stringResult = objectResult.ToString();
responce = (JsonConvert.SerializeObject(value));
}
I used this in a .net core app.
Hope it helps.
I am passing Json Data from Angular JS Controller. The Json Data contains two strings called name attribute and comment attribute and a list of files. The controller code for angular is given below:
app.controller("demoController", function ($scope, $http) {
//1. Used to list all selected files
$scope.files = [];
//2. a simple model that want to pass to Web API along with selected files
$scope.jsonData = {
name: "Sibnz",
comments: "This is a comment"
};
//3. listen for the file selected event which is raised from directive
$scope.$on("seletedFile", function (event, args) {
$scope.$apply(function () {
//add the file object to the scope's files collection
$scope.files.push(args.file);
});
});
//4. Post data and selected files.
$scope.save = function () {
$http({
method: 'POST',
url: "http://localhost:51739/PostFileWithData",
headers: { 'Content-Type': undefined },
transformRequest: function (data) {
var formData = new FormData();
formData.append("model", angular.toJson(data.model));
for (var i = 0; i < data.files.length; i++) {
formData.append("file" + i, data.files[i]);
}
return formData;
},
data: { model: $scope.jsonData, files: $scope.files }
}).
success(function (data, status, headers, config) {
alert("success!");
}).
error(function (data, status, headers, config) {
alert("failed!");
});
};
});
In the Web API, controller I am receiving the JSON data by using the following code:
[HttpPost]
[Route("PostFileWithData")]
public async Task<HttpResponseMessage> Post()
{
if (!Request.Content.IsMimeMultipartContent())
{
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}
var root = HttpContext.Current.Server.MapPath("~/App_Data/Uploadfiles");
Directory.CreateDirectory(root);
var provider = new MultipartFormDataStreamProvider(root);
var result = await Request.Content.ReadAsMultipartAsync(provider);
var model = result.FormData["jsonData"];
var g = result.FileData;
if (model == null)
{
throw new HttpResponseException(HttpStatusCode.BadRequest);
}
//TODO: Do something with the JSON data.
//get the posted files
foreach (var file in result.FileData)
{
//TODO: Do something with uploaded file.
var f = file;
}
return Request.CreateResponse(HttpStatusCode.OK, "success!");
}
When I debug the code, I find that the JSON data is populating the var model and var g variables. I want to extract the name and comment attributes from the Json Data and store them in the Database. And also want to copy the file into /App_Data/Uploadfiles directory and store the file location in the database.
You need to create a model in your Web API and deserialize JSON data to this model, you can use Newtonsoft.Json NuGet package for that
Install-Package Newtonsoft.Json
class DataModel
{
public string name { get; set; }
public string comments { get; set; }
}
In Web API controller
using Newtonsoft.Json;
HttpRequest request = HttpContext.Current.Request;
var model = JsonConvert.DeserializeObject<DataModel>(request.Form["jsonData"]);
// work with JSON data
model.name
model.comments
To work with files
// Get the posted files
if (request.Files.Count > 0)
{
for (int i = 0; i < request.Files.Count; i++)
{
Stream fileStream = request.Files[i].InputStream;
Byte[] fileBytes = new Byte[stampStream.Length];
// Do something with uploaded file
var root = HttpContext.Current.Server.MapPath("~/App_Data/Uploadfiles/");
string fileName = "image.jpg";
File.WriteAllBytes(root + fileName, stampBytes);
// Save only file name to your database
}
}
I have a control on view page. When user selects the file and clicks on submit button this makes ajax call to upload the file on server. Unfortunately my server method accepts file path (like C:/Videos/1.mp4) to upload. This works great with string demoPath in the code below but I'm not sure how to get similar path when user selects in control. Due to sercurity reasons modern browsers not allows exposing paths. How to achieve this?
[HttpPost]
public async Task<JsonResult> Upload(string lectureId, string filepath)
{
for (int i = 0; i < Request.Files.Count; i++)
{
//// This works great
//string demoPath = "C:/Users/abchi/Desktop/BigBuckBunny.mp4";
var file = Request.Files[i];
var fileName = Path.GetFileName(file.FileName);
//var path = Path.Combine(Server.MapPath("~/User/"), fileName);
//file.SaveAs(path);
//await RunUploader(demoPath);
await RunUploader(get_path_from_posted_file_or_request);
}
return Json(new { error = false, message = "Video uploaded." });
}
public async Task RunUploader(string filePath)
{
// :::::::
using (var fileStream = new FileStream(filePath, FileMode.Open))
{
// ::::
}
// ::::::
}
I'm not sure this is expected because I did not quite understand.
Download the file path of the user's computer can not be - https://stackoverflow.com/a/15201258/4599089
but if you want to have access to the FileStream on your server:
File has InputStream and you can use this:
[HttpPost]
public async Task<JsonResult> Upload(string lectureId, string filepath)
{
for (int i = 0; i < Request.Files.Count; i++)
{
var file = Request.Files[i];
var fileName = Path.GetFileName(file.FileName);
var path = Path.Combine(Server.MapPath("~/User/"), fileName);
var fileStream = new FileStream(path, FileMode.Create, FileAccess.ReadWrite);
file.InputStream.CopyTo(fileStream);
fileStream.Close();
await RunUploader(path); //path or stream
}
return Json(new { error = false, message = "Video uploaded." });
}
public async Task RunUploader(string filePath)
{
// :::::::
using (var fileStream = new FileStream(filePath, FileMode.Open))
{
// ::::
}
// ::::::
}
I asked my fellow dev to make necessary changes in public async Task RunUploader(string filePath) parameters. Said code was part of YouTube .NET samples for console apps. Now we are developing for web, in this case we can't pass full path. So they made following changes:
[HttpPost]
public async Task<JsonResult> Upload(string lectureId)
{
for (int i = 0; i < Request.Files.Count; i++)
{
var file = Request.Files[i];
Stream fileStream = file.InputStream;
await Run(fileStream);
}
return Json(new { error = false, message = "Video uploaded." });
}
public async Task Run(Stream fileStream)
{
// ::::::::::
using (fileStream)
{
// ::::::
}
// ::::::::::
}
Now with this change everything started working.
I created a test source which should send a message to the client every x time. This is the ApiController:
public class TestSourceController : ApiController
{
private static readonly ConcurrentQueue<StreamWriter> ConnectedClients = new ConcurrentQueue<StreamWriter>();
[AllowAnonymous]
[Route("api/sources/test")]
public HttpResponseMessage Get()
{
var response = Request.CreateResponse();
response.Content = new PushStreamContent((Action<Stream, HttpContent, TransportContext>) OnStreamAvailable,
"text/event-stream");
return response;
}
private static void OnStreamAvailable(Stream stream, HttpContent headers, TransportContext context)
{
var clientStream = new StreamWriter(stream);
ConnectedClients.Enqueue(clientStream);
}
private static void DoThings()
{
const string outboundMessage = "Test";
foreach (var clientStream in ConnectedClients)
{
clientStream.WriteLine("data:" + JsonConvert.SerializeObject(outboundMessage));
clientStream.Flush();
}
}
}
The clientStream.Flush(); is called like expected and without exceptions.
I handle it in AngularJS like this:
$scope.handleServerCallback = function (data) {
console.log(data);
$scope.$apply(function() {
$scope.serverData = data;
});
};
$scope.listen = function () {
$scope.eventSource = new window.EventSource("http://localhost:18270/api/sources/test");
$scope.eventSource.onmessage = $scope.handleServerCallback;
$scope.eventSource.onopen = function() { console.log("Opened source"); };
$scope.eventSource.onerror = function (e) { console.error(e); };
};
$scope.listen();
My guess is it's a problem with the server since I can see the "EventStream" from the test call is empty in the chrome debugger.
Does anyone know how to make sure the messages arrive at the client?
The solution was quite easy, according to the spec every line has to end with "\n" and the very last line with "\n\n".
So:
clientStream.WriteLine("data:" + JsonConvert.SerializeObject(outboundMessage) + "\n\n");
Solves it.