Track writen data instead of read data in a file stream - .net-core

I'm trying to follow this guide for making a progressbar with blazor server-side
https://www.meziantou.net/file-upload-with-progress-bar-in-blazor.htm
I then want to write the files to the file system, the problem is that reading it is much faster than actually writing it to the disk meaning it seems stuck at 100% for quite some time at large files.
So can i make it track how much it have written to the disk instead. or maybe them synchronized.
await using FileStream fs = new(path, FileMode.Create);
using var stream = file.OpenReadStream(maxFileSize);
while (await stream.ReadAsync(buffer) is int read && read > 0)
{
//Writing to disk
await fs.WriteAsync(buffer.AsMemory(0, read));
//Updating how much data has been read
uploadedFiles[startIndex].UploadedBytes += read;
}
EDIT:
After trying the answer it made more sense, and because i'm using an azure file share. There is some delay for that.
Checking the filesize directly on the file would give me the right progress but way to slow.
Think i will store the file locally, before transfering to the fileShare. Or looking into blobStorage transfer.

Reading and Writing are already happening interleaved.
The only thing that could possibly help here is to Flush:
while (await stream.ReadAsync(buffer) is int read && read > 0)
{
//Writing to disk
await fs.WriteAsync(buffer.AsMemory(0, read));
// Force the actual writing
await fs.FlushAsync();
//Updating how much data has been read
uploadedFiles[startIndex].UploadedBytes += read;
}
Without this the FileSystem is free to keep the data in a buffer until you Close the file.
Do note that repeatedly Flushing will probably increase the total time for the upload.

Related

firebase real time database takes a long time to load data

Firebase realtime database is taking a long time to load data. Here's a screenshot of the data that I have in the database. What can I do to optimize the loading? Also are there other places that I can store the data other than firebase? The data is 3.8MB in size, and has the following structure
{"10-happier": {body: "test"}, "zero-to-one": {body: "test2"}}
Here's my code
var defer = Q.defer();
app.database().ref('content').once('value').then(snapshot => {
if (snapshot && snapshot.val()) {
defer.resolve(snapshot.val());
} else {
defer.resolve({});
}
}).catch( error => {
defer.reject(error);
});
return defer.promise;
There's nothing you can do to optimize a query like this. When you fetch an entire node:
app.database().ref('content').once('value')
The SDK will download everything, and it will take as long as it takes to get the whole thing. The total performance is going to be determined by the speed of the client's connection to the server. If you want the query to be faster, your only viable option is to get a faster connection to Realtime Database.
Alternatively, you can bypass the use of a database altogether and use a different method of storage that involves some form of compression or CDN to deliver the content more efficiently for the end user. Since recommendations for software and services are off-topic for Stack Overflow, you will have to do some research to figure out what your options are and what will work best for your specific situation.

Safe to Save Binary Data in Cloud Firestore Database?

I've always used the Cloud Firestore Database (and the old real-time one) to store text, and then use the Storage for images.
While using SurveyJS and AngularFirestore, I discovered I can push binary files into and out of the Firestore Database with the attached code. My question is: Is this OK?? I mean it works great, but I don't want to incur a cost or network slowdown...Thanks
var resultAsString = JSON.stringify(this.survey.data);
this.qs.saveSupplierQuestionnaire(this.companyid, this.id,this.survey.data)
...
saveSupplierQuestionnaire(userid:string, questionnaireid:string, questionnaireData:any) {
var resultAsString = JSON.stringify(questionnaireData);
var numCompleted = 0; /////test grading
const dbRef = this.afs.collection<questionnaire>('companies/' + userid + '/questionnaires/').doc(questionnaireid).update({results:resultAsString})
If it meets the needs of your application, then it's OK.
You should be aware than any time a document is read, the entire document is transferred to the client. So, even if you don't use the field with the binary data, you are going to make the user wait for the entire contents to be downloaded. This is true for all fields of a document, regardless of their types. There is really nothing special about binary fields, other than how the data is typed.

QNetworkAccessManager: post http multipart from serial QIODevice

I'm trying to use QNetworkAccessManager to upload http multiparts to a dedicated server.
The multipart consists of a JSON part describing the data being uploaded.
The data is read from a serial QIODevice, which encrypts the data.
This is the code that creates the multipart request:
QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
QHttpPart metaPart;
metaPart.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
metaPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"metadata\""));
metaPart.setBody(meta.toJson());
multiPart->append(metaPart);
QHttpPart filePart;
filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant(fileFormat));
filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"file\""));
filePart.setBodyDevice(p_encDevice);
p_encDevice->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart
multiPart->append(filePart);
QNetworkAccessManager netMgr;
QScopedPointer<QNetworkReply> reply( netMgr.post(request, multiPart) );
multiPart->setParent(reply.data()); // delete the multiPart with the reply
If the p_encDevice is an instance of QFile, that file gets uploaded just fine.
If the specialised encrypting QIODevice is used (serial device) then all of the data is read from my custom device. however QNetworkAccessManager::post() doesn't complete (hangs).
I read in the documentation of QHttpPart that:
if device is sequential (e.g. sockets, but not files),
QNetworkAccessManager::post() should be called after device has
emitted finished().
Unfortunately I don't know how do that.
Please advise.
EDIT:
QIODevice doesn't have finished() slot at all. What's more, reading from my custom IODevice doesn't happen at all if QNetworkAccessManager::post() is not called and therefore the device wouldn't be able to emit such an event. (Catch 22?)
EDIT 2:
It seems that QNAM does not work with sequential devices at all. See discussion on qt-project.
EDIT 3:
I managed to "fool" QNAM to make it think that it is reading from non-sequential devices, but seek and reset functions prevent seeking. This will work until QNAM will actually try to seek.
bool AesDevice::isSequential() const
{
return false;
}
bool AesDevice::reset()
{
if (this->pos() != 0) {
return false;
}
return QIODevice::reset();
}
bool AesDevice::seek(qint64 pos)
{
if (this->pos() != pos) {
return false;
}
return QIODevice::seek(pos);
}
You'll need to refactor your code quite a lot so that the variables you pass to post are available outside that function you've posted, then you'll need a new slot defined with the code for doing the post inside the implementation. Lastly you need to do connect(p_encDevice, SIGNAL(finished()), this, SLOT(yourSlot()) to glue it all together.
You're mostly there, you just need to refactor it out and add a new slot you can tie to the QIODevice::finished() signal.
I've had more success creating the http post data manually than with using QHttpPart and QHttpMultiPart. I know it's probably not what you want to hear, and it's a little messy, but it definitely works. In this example I am reading from a QFile, but you can call readAll() on any QIODevice. It also is worth noting, QIODevice::size() will help you check if all the data has been read.
QByteArray postData;
QFile *file=new QFile("/tmp/image.jpg");
if(!(file->open(QIODevice::ReadOnly))){
qDebug() << "Could not open file for reading: "<< file->fileName();
return;
}
//create a header that the server can recognize
postData.insert(0,"--AaB03x\r\nContent-Disposition: form-data; name=\"attachment\"; filename=\"image.jpg\"\r\nContent-Type: image/jpeg\r\n\r\n");
postData.append(file->readAll());
postData.append("\r\n--AaB03x--\r\n");
//here you can add additional parameters that your server may need to parse the data at the end of the url
QString check(QString(POST_URL)+"?fn="+fn+"&md="+md);
QNetworkRequest req(QUrl(check.toLocal8Bit()));
req.setHeader(QNetworkRequest::ContentTypeHeader,"multipart/form-data; boundary=AaB03x");
QVariant l=postData.length();
req.setHeader(QNetworkRequest::ContentLengthHeader,l.toString());
file->close();
//free up memory
delete(file);
//post the data
reply=manager->post(req,postData);
//connect the reply object so we can track the progress of the upload
connect(reply,SIGNAL(uploadProgress(qint64,qint64)),this,SLOT(updateProgress(qint64,qint64)));
Then the server can access the data like this:
<?php
$filename=$_REQUEST['fn'];
$makedir=$_REQUEST['md'];
if($_FILES["attachment"]["type"]=="image/jpeg"){
if(!move_uploaded_file($_FILES["attachment"]["tmp_name"], "/directory/" . $filename)){
echo "File Error";
error_log("Uploaded File Error");
exit();
};
}else{
print("no file");
error_log("No File");
exit();
}
echo "Success.";
?>
I hope some of this code can help you.
I think the catch is that QNetworkAccessManager does not support chunked transfer encoding when uploading (POST, PUT) data. This means that QNAM must know in advance the length of the data it's going to upload, in order to send the Content-Length header. This implies:
either the data does not come from sequential devices, but from random-access devices, which would correctly report their total size through size();
or the data comes from a sequential device, but the device has already buffered all of it (this is the meaning of the note about finished()), and will report it (through bytesAvailable(), I suppose);
or the data comes from a sequential device which has not buffered all the data, which in turn means
either QNAM reads and buffers itself all the data coming from the device (by reading until EOF)
or the user manually set the Content-Length header for the request.
(About the last two points, see the docs for the QNetworkRequest::DoNotBufferUploadDataAttribute.)
So, QHttpMultiPart somehow shares these limitations, and it's likely that it's choking on case 3. Supposing that you cannot possibly buffer in memory all the data from your "encoder" QIODevice, is there any chance you might know the size of the encoded data in advance and set the content-length on the QHttpPart?
(As a last note, you shouldn't be using QScopedPointer. That will delete the QNR when the smart pointer falls out of scope, but you don't want to do that. You want to delete the QNR when it emits finished()).
From a separate discussion in qt-project and by inspecting the source code it seems that QNAM doesn't work with sequential at all. Both the documentation and code are wrong.

Qt Streaming Large File via HTTP and Flushing to eMMC Flash

I'm streaming a large file ( 1Gb ) via HTTP to my server in Qt on a very memory constrained embedded Linux device. When I first receive the header I determine where to write the data on the filesystem, create a QFile pointer to that location, and open the file for appending. There is an 'accumulate' function in the server that is called each time new data arrives to the socket. From that accumulate function I want to stream the data right to the file via write(). You can see my accumulate function below.
My problem is memory usage when doing this -- I run out of memory. Shouldn't I be able to flush() and fsync() each iteration of the accumulation and not have to worry about RAM usage? What am I doing wrong and how can I fix this? Thanks -
I open my file once before the accumulate function:
// Open the file
filePointerToWriteTo->open(QIODevice::WriteOnly | QIODevice::Append | QIODevice::Unbuffered)
Here is a portion of the accumulate function:
// Extract the QFile pointer from the QVariant
QFile *filePointerToWriteTo = (QFile *)(containerForPointer->pointer).value<void *>();
qDebug() << "APPENDING bytes: " << data.length();
// Write to the file and sync
filePointerToWriteTo->write(data);
filePointerToWriteTo->waitForBytesWritten(-1);
filePointerToWriteTo->flush(); // Flush
fsync(filePointerToWriteTo->handle()); // Make sure bytes are written to disk
EDIT:
I instrumented my code and the 'waitForBytesWritten(-1)' call ALWAYS return 'false'. The docs say this should wait until data is written to the device.
Also, If I uncomment only the 'write(data)' line, then my free memory never decreases. What could be going on? How does 'write' consume so much memory?
EDIT:
Now I am doing the following. I do not run out of memory, but my free memory drops to 2Mb and hovers there until the entire file is transferred. At which point, the memory is released. If I kill the transfer in the middle, the kernel seems to hold on to the memory because it stays around 2Mb free until I restart the process and try to write to the same file. I still think I should be able to use and flush the memory each iteration:
// Extract the QFile pointer from the QVariant
QFile *filePointerToWriteTo = (QFile *)(containerForPointer->pointer).value<void *>();
int numberOfBytesWritten = filePointerToWriteTo->write(data);
qDebug() << "APPENDING bytes: " << data.length() << " ACTUALLY WROTE: " << numberOfBytesWritten;
// Flush and sync
bool didWaitForWrite = filePointerToWriteTo->waitForBytesWritten(-1); // <----------------------- This ALWAYS returns false!
filePointerToWriteTo->flush(); // Flush
fsync(filePointerToWriteTo->handle()); // Make sure bytes are written to disk
fdatasync(filePointerToWriteTo->handle()); // Specific Sync
sync(); // Total sync
EDIT:
This kind of sounds like me misunderstanding Linux caching. After reading this post --> http://blog.scoutapp.com/articles/2010/10/06/determining-free-memory-on-linux, it's possible that I am misunderstanding the output of 'free -mt'. I have been watching the 'free' field in that output and see it drop to hover around 2MB on the massive file transfer. I would just like to see it return to high levels of free data when the file transfer is done.
I think Linux is just caching everything it can and frees what it can spare around the 2MB free memory limit. I do not run out of memory when receiving or sending out ~2Gb of files on a 512 MB RAM system. In my Qt program, after receiving all of the data, appending to file, and closing the file. I do the following in a QProcess to see my 'free' memory return in the 'free -mt' command in a separate terminal:
// Now we've returned a large file - so free up cache in linux
QProcess freeCachedMemory;
freeCachedMemory.start("sh");
freeCachedMemory.write("sync; echo 3 > /proc/sys/vm/drop_caches"); // Sync to disk and clear Linux cache
freeCachedMemory.waitForFinished();
freeCachedMemory.close();

Please suggest a way to store a temp file in Windows Azure

Here I have a simple feature on ASP.NET MVC3 which host on Azure.
1st step: user upload a picture
2nd step: user crop the uploaded picture
3rd: system save the cropped picture, delete the temp file which is the uploaded original picture
Here is the problem I am facing now: where to store the temp file?
I tried on windows system somewhere, or on LocalResources: the problem is these resources are per Instance, so here is no guarantee the code on an instance shows the picture to crop will be the same code on the same instance that saved the temp file.
Do you have any idea on this temp file issue?
normally the file exist just for a while before delete it
the temp file needs to be Instance independent
Better the file can have some expire setting (for example, 1H) to delete itself, in case code crashed somewhere.
OK. So what you're after is basically somthing that is shared storage but expires. Amazon have just announced a rather nice setting called object expiration (https://forums.aws.amazon.com/ann.jspa?annID=1303). Nothing like this for Windows Azure storage yet unfortunately, but, doesnt mean we can't come up with some other approach; indeed even come up with a better (more cost effective) approach.
You say that it needs to be instance independant which means using a local temp drive is out of the picture. As others have said my initial leaning would be towards Blob storage but you will have cleanup effort there. If you are working with large images (>1MB) or low throughput (<100rps) then I think Blob storage is the only option. If you are working with smaller images AND high throughput then the transaction costs for blob storage will start to really add up (I have a white paper coming out soon which shows some modelling of this but some quick thoughts are below).
For a scenario with small images and high throughput a better option might be to use the Windows Azure Cache as your temporary storaage area. At first glance it will be eye wateringly expensive; on a per GB basis (110GB/month for Cache, 12c/GB for Storage). But, with storage your transactions are paid for whereas with Cache they are 'free'. (Quotas are here: http://msdn.microsoft.com/en-us/library/hh697522.aspx#C_BKMK_FAQ8) This can really add up; e.g. using 100kb temp files held for 20 minutes with a system throughput of 1500rps using Cache is about $1000 per month vs $15000 per month for storage transactions.
The Azure Cache approach is well worth considering, but, to be sure it is the 'best' approach I'd really want to know;
Size of images
Throughput per hour
A bit more detail on the actual client interaction with the server during the crop process? Is it an interactive process where the user will pull the iamge into their browser and crop visually? Or is it just a simple crop?
Here is what I see as a possible approach:
user upload the picture
your code saves it to a blob and have some data backend to know the relation between user session and uploaded image (mark it as temp image)
display the image in the cropping user interface interface
when user is done cropping on the client:
4.1. retrieve the original from the blob
4.2. crop it according the data sent from the user
4.3. delete the original from the blob and the record in the data backend used in step 2
4.4. save the final to another blob (final blob).
And have one background process checking for "expired" temp images in the data backend (used in step 2) to delete the images and the records in the data backend.
Please note that even in WebRole, you still have the RoleEntryPoint descendant, and you still can override the Run method. Impleneting the infinite loop in the Run() (that method shall never exit!) method, you can check if there is anything for deleting every N seconds (depending on your Thread.Sleep() in the Run().
You can use the Azure blob storage. Have look at this tutorial.
Under sample will be help you.
https://code.msdn.microsoft.com/How-to-store-temp-files-in-d33bbb10
you have two way of temp file in Azure.
1, you can use Path.GetTempPath and Path.GetTempFilename() functions for the temp file name
2, you can use Azure blob to simulate it.
private long TotalLimitSizeOfTempFiles = 100 * 1024 * 1024;
private async Task SaveTempFile(string fileName, long contentLenght, Stream inputStream)
{
try
{
//firstly, we need check the container if exists or not. And if not, we need to create one.
await container.CreateIfNotExistsAsync();
//init a blobReference
CloudBlockBlob tempFileBlob = container.GetBlockBlobReference(fileName);
//if the blobReference is exists, delete the old blob
tempFileBlob.DeleteIfExists();
//check the count of blob if over limit or not, if yes, clear them.
await CleanStorageIfReachLimit(contentLenght);
//and upload the new file in this
tempFileBlob.UploadFromStream(inputStream);
}
catch (Exception ex)
{
if (ex.InnerException != null)
{
throw ex.InnerException;
}
else
{
throw ex;
}
}
}
//check the count of blob if over limit or not, if yes, clear them.
private async Task CleanStorageIfReachLimit(long newFileLength)
{
List<CloudBlob> blobs = container.ListBlobs()
.OfType<CloudBlob>()
.OrderBy(m => m.Properties.LastModified)
.ToList();
//get total size of all blobs.
long totalSize = blobs.Sum(m => m.Properties.Length);
//calculate out the real limit size of before upload
long realLimetSize = TotalLimitSizeOfTempFiles - newFileLength;
//delete all,when the free size is enough, break this loop,and stop delete blob anymore
foreach (CloudBlob item in blobs)
{
if (totalSize <= realLimetSize)
{
break;
}
await item.DeleteIfExistsAsync();
totalSize -= item.Properties.Length;
}
}

Resources