I am very new to grpc and I am refactoring some http handlers to grpc. In there I have found a handler which is relevant to upload a file. In the request, it is sending a file as http.FormFile using http multipart form data.
What I found is, there is a way using request chunk data stream to upload file. But what i need is avoid streaming and do it in stateless manner.
I have searched a way to solve this, but I couldn't find a way to do that. Highly appreciate if someone give me a propper solution to do this
Tl;dr
gRPC was not designed to handle large file uploads in the same way that you would using http multipart form data uploads. gRPC has a (slightly) arbitrary 4MB message limit (also see this). In my experience, the proper solution is to not use gRPC for large file uploads. That being said, there may be a few options you can try.
Changing gRPC Call Options
You can manually override the default 4MB message limit using connection options when you dial the gRPC server. For example, see this:
client, err := grpc.Dial("...",
grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(4096)))
Using gRPC Streams
I have needed to use gRPC for file uploads without changing the default message limit options by implementing my own chunking package to handle it for me over a unary gRPC stream. You mentioned you wanted to avoid using a stream, however I'm providing this as a resource for those who want to avoid changing their gRPC message limit. Using the library, you'll need to wrap your gRPC client. Once a wrapper is created, you can upload anything that is compatible with the io.Reader interface.
err := chunk.UploadFrom(reader, WrapUploadFileClient(client))
Related
I am logging raw http requests to kafka.
GET /api/v1/user
GET /api/v1/friends
POST /api/v2/problem/solve HTTP/1.1
Host: domain.com
Content-Length:111
Cookie:...
{
"input":"{...}"
}
Obstruction, such as sessions and JWT tokens, will modify the source of the server.
I would like to make new requests based on these data, test them, and see the results, can you recommend a suitable solution?
I looked for Jmeter, K6, and others, but it was not a satisfactory solution.
You're looking for a load testing tool which will take the stuff you put to Kafka and generate end-to-end load test out of it? Good luck.
JMeter might be used in at least 2 ways:
If you're capable of exporting messages from Kafka into files you could consider using HTTP Raw Request sampler which has more or less the same syntax as your records
If not, you could use JSR223 PreProcessor for reading the records from Kafka and constructing the HTTP Request out of them, the JMeter API class you will need is HTTPSamplerProxy
We realize that if we want to produce a multipart query that contains a video file of 15GB, it is impossible to allocate in memory the size needed for such a large amount of data, most devices have only 2 or 3GB of RAM.
It is therefore absolutely necessary to switch to the uploadTask method which will push to the server the contents of a block file of the maximum size allowed by the IP packets sent to the server.
This is a POST method. However, it does not contain parameters such as the folder id or the file name. So you need a way to transmit these parameters. The best way is to code them in the URL.
I proposed an encoding format in the form of a path behind the endpoint of the API, but we can also very well encode these two parameters in a classic way in the URL, eg:
/api/upload?id=123&filename=video.mp4
From what I read on Stackoverflow, it's trivial with Symfony to retrieve id and filename. Then all the data received in the body of the POST request can be written in a raw way directly into a file, without also passing through a buffer in server-side memory.
The user data must imperatively be streamed, whether mobile side or server side, and whether upload or download. Loading user content in memory is also very dangerous in terms of security.
In symfony, how can I do that?
This goes way beyond Symfony and depends on the web server you are using.
By default with apache/nginx and php you will receive an already buffered request, so you cannot stream it to a file.
However, there are solutions, for example with Apache you can stream requests, see http://hc.apache.org/httpclient-3.x/performance.html#Request_Response_entity_streaming
Probably nginx also has options for it, but I don't know about those.
Another option might be websockets, see http://en.wikipedia.org/wiki/WebSocket
I am looking for a way on how to implement file download functionality using gRPC but I can't find in the documentation how this is done.
What is an optimal way to do this? I wanted to have a gRPC server that holds files and a gRPC client to request something from gRPC.
I have looked at the examples in Java but could not figure out a way on how to do this. I just started reading about gRPC today.
You must define a specific message as data container and you must transfer it to the client in chunks using a stream.
Example of proto Message and RPC service:
message DataChunk {
bytes data = 1;
}
rpc DownloadProductImage(DownloadProductImageRequest) returns(stream DataChunk){}
In java you can use BufferedInputStream to stream the resource to the client.
protobuf / grpc
server side code
client side code
There are a few possible approaches.
If your file size is small enough, you can define a proto file with a single String field, and dump the entire text content into that field.
If your file size is too large to do this, you might want to chunk the file and use the streaming API to send the file chunk by chunk.
Below is a good guide for getting started with gRPC.
https://grpc.io/docs/tutorials/basic/java.html
I have taken the working example from db80 and have created a PR with a full example: https://github.com/grpc/grpc-java/pull/5126
Hopefully they will merge this then everyone can have a working example of this common use-case for GRPC.
There is probably an answer within reach, but most of the search results are "handling large file uploads" where the user does not know what they're doing or "handing many uploads" where the answer consistently is just an explanation of how to work with multipart requests and/or Flash uploader widgets.
I haven't had time to sift through Go's HTTP implementation, yet, but when does the application have the first chance to see the incoming body? Not until it has been completely received?
If I were to [poorly] decide to use HTTP to transfer a large amount of data and posted a single request with several 10-gigabyte parts, would I have to wait for the whole thing to be received before processing it or does the io.Reader with the body iteratively process it?
This is only tangentially related, but I also haven't been able to get a clear answer about whether I can choose to forcibly close the connection in the middle; whether or not, even if I close it, it will just keep receiving it on the port.
Thanks so much.
An application's handler is called after the headers are parsed and before the request body is read. The handler can read the request body as soon as the handler is called. The server does not buffer the entire request body.
An application can read file uploads without buffering the entire request by getting a multipart reader and iterating through the parts.
An application can replace the request body with a MaxBytesReader to force close the connection after a specified limit is breached.
The above comments are about the net/http server included in the standard library. The comments may not apply to other servers.
While I haven't done this with GB size files, my strategy with file processing (mostly stuff I read from and write to S3) is to use https://golang.org/pkg/os/exec/ with a cmd line utility that handles chunking a way you like. Then read and process by tailing the file as explained here: Reading log files as they're updated in Go
In my situations, network utilities can download the data far faster than my code can process it, so it makes sense to send it to disk and pick it up as fast as I can, that way I'm not holding some connection open while I process.
I have an ASP.NET web application and I want my users to be able to upload large files. However, some files are very large and uses too much memory.
In principle it should be possible to receive the request stream and write it directly to a FileWriter stream, removing any need to load the entire file into memory first.
I've tried accessing Request.InputStream and writing it directly to a file. It works, but a test using larger files reveal that Request.InputStream is only available after the entire request is already loaded into memory.
Can someone tell me an approach I can use to receive a normal Request.InputStream in ASP.NET and directly write it to a file without first loading it into memory?
Note, the file is sent through a normal request in a browser by posting a form with a file field.
(I actually use BlueImp JQuery File Upload but I don't think it's relevant to this question)
The process is called byte serving.
Byte Serving:
Byte serving is the process of sending only a portion of an HTTP/1.1 message from a server to a client. Byte serving begins when an HTTP server advertises its willingness to serve partial requests using the Accept-Ranges response header. A client then requests a specific part of a file from the server using the Range request header.
Is seems that IIS and ASP.NET are capable of handling Accept-Range headers. There is a Range Controller on Microsoft git repositories.
Here is an article that may be useful in configuring IIS to handle these requests.