I want to serve files(images) via gRPC Gateway from gRPC server. Since protocol buffers messages have sctructure, I don't see how I could ensure the gateway to send content of the bytes field of the response message instead of the entire json-encoded message. Is there a native solution for this or does one simply have to write a dedicated http muxer to handle these requests?
Rather than sending arbitrarily-sized files (probably as bytes), it would probably be better to include a URL to the file in the message and then host/serve the file over HTTP (e.g. from S3, Google Cloud Storage etc. from which you could generate signed URLs to limit access).
I think the max message size is 2GB (source?) and the recommendation (Large Data Sets) is to consider alternative techniques once messages sizes exceed few MBs.
Related
I would like to create a mechanism to allow clients to subscribe to the contents of a blob-storage bucket.
Traditionally (HTTP 1.1) this would involve polling for new items, then issuing a GET requests for each item.
Mechanisms such as gRPC allow responses to be streamed, however that particular mechanism relies on messages being loaded into memory, limiting incoming messages to a few MB.
Does a "push" mechanism exist that would allow a server to transmit larger payloads to clients without requiring the client to request that specific payload?
Note: My question remains un-answered.
After hours of reading I'm confused about download/upload speeds.
Let's suppose I sent an http request to fetch an image, all websites mention this as "Download" but why?
My computer is sending the request and getting a response back, so it's both uploading (sending packet) and downloading (getting packet). Thus it's not pure Download.
Plus, I read that internet provides prioterize download over upload but how? given a packet from A to B how can they decide if it's an upload or download?
From A's perspective it's Upload, but according to B it's Download...
Where did you read about prioritizing download speed over upload? ISPs can only prioritize speed traffic by IP (to move traffic to another chanel out of shaper or to make this traffic use another shaper's rules) or by shaper set rules for UDP, Vo-Ip or traffic per VLAN/port. Web sites can optimize data for displaying content cached and compressed, for example. Or to use the uploading feature on other upstream servers. But you, as a user, upload to the server RAW data (an uncompressed set of bytes). For managing speeds from an ISP, the ISP must monitor each packet from each user. And then it can limit speeds either by packet size (but it will work 50/50 'cos files are uploaded by a few requests) or by meta data from a packet. It is a massive load for all infrastructure. And accordingly, it is a very costly process for businesses.
Does sending large files over a websocket "block" websocket for other messages while the large files are being sent?
Does sending the files via independent Http requests while the other messages continue to be sent over WS have any distinct advantage "in keeping the WS unblocked"?
Assume 1 network card.
In case of WebSocket over HTTP/1.1, yes, the upload of a large file (in the form of a large WebSocket message) blocks the WebSocket connection.
In case of WebSocket over HTTP/2 (if supported by both the client and server), one HTTP/2 stream will upload the large file, and another HTTP/2 stream is be used to carry WebSocket messages. In this case, the problem becomes the HTTP/2 flow control window, which may be exhausted by the large upload stream, leaving the WebSocket message stream stalled (so that messages are queued and delayed). Unfortunately, the details of this queueing/delay depend on the client and on the server implementations, so you have to try.
Typically implementations do a good job at interleaving streams, so rarely the possible stalls are a problem.
For WebSocket over HTTP/1.1, if you open multiple WebSocket connections, you may be able to send files and messages in parallel, using N WebSocket connections for the files, and 1 WebSocket connection for the messages, for example.
Some non-browser clients allow you to open multiple HTTP/2 connections to the same domain, so again you will have the chance to send files and messages in parallel. However, to my knowledge, browsers do not allow more than 1 HTTP/2 connection to the same domain, so the parallelism is there, but constrained by the HTTP/2 flow control window.
Not sure what you mean by "keeping the WS unblocked", but HTTP/1.1 works in the same way as WebSocket for what pertains its usage of connections.
If you are in a browser environment, browsers allow 6-8 HTTP connections to the same domain, and typically unlimited (or at least many more) WebSocket connections.
So if you want to send, say, 10 large files, 6-8 of them will be uploaded via HTTP, but the remaining will be queued waiting for one of the HTTP connections to finish the previous upload.
Meanwhile, you can use the WebSocket connection to send messages.
In case of HTTP/2, browsers only open 1 connection, so you may use HTTP/2 streams for the uploads and a WebSocket over HTTP/2 stream for the messages, but they will all share the same HTTP/2 flow control window, potentially stalling each other.
All in all, WebSocket has not been designed for large uploads.
I would not be surprised if you hit WebSocket message size limits, as servers cannot allow clients to upload messages of arbitrary size (as it will blow up the server memory). The same is true for clients; browsers have typically small limits for the size of WebSocket messages that they receive, independently of whether HTTP/1.1 or HTTP/2 is used.
If you really need to upload large files, I think a solution where you upload via HTTP (which allow larger sizes, for example when using multipart/form-data), and keep small messaging via WebSocket is optimal.
The use of HTTP/2 may hit the HTTP/2 flow control window limit, but you have a limit in 6-8 connections in HTTP/1.1 too, so again you have to try and see if you hit any limit, and if you do, which limit it is in what case.
Using HTTP for uploads makes less likely that you hit WebSocket message size limits that are not known in advance and possibly different from client to client (browser to browser), and you don't want to implement your own splitting and merging of large uploads via WebSocket to respect those limits.
As far as I can tell, Firestore uses protocol buffers when making a connection from an android/ios app. Out of curiosity I want to see what network traffic is going up and down, but I can't seem to make charles proxy show any real decoded info. I can see the open connection, but I'd like to see what's going over the wire.
Firestores sdks are open source it seems. So it should be possible to use it to help decode the output. https://github.com/firebase/firebase-js-sdk/tree/master/packages/firestore/src/protos
A few Google services (like AdMob: https://developers.google.com/admob/android/charles) have documentation on how to read network traffic with Charles Proxy but I think your question is, if it’s possible with Cloud Firestore since Charles has support for protobufs.
The answer is : it is not possible right now. The firestore requests can be seen, but can't actually read any of the data being sent since it's using protocol buffers. There is no documentation on how to use Charles with Firestore requests, there is an open issue(feature request) on this with the product team which has no ETA. In the meanwhile, you can try with the Protocol Buffers Viewer.
Alternatives for viewing Firestore network traffic could be :
From Firestore documentation,
For all app types, Performance Monitoring automatically collects a
trace for each network request issued by your app, called an HTTP/S
network request trace. These traces collect metrics for the time
between when your app issues a request to a service endpoint and when
the response from that endpoint is complete. For any endpoint to which
your app makes a request, Performance Monitoring captures several
metrics:
Response time — Time between when the request is made and when the response is fully received
Response payload size — Byte size of the network payload downloaded by the app
Request payload size — Byte size of the network payload uploaded by the app
Success rate — Percentage of successful responses compared to total responses (to measure network or server failures)
You can view data from these traces in the Network requests subtab of
the traces table, which is at the bottom of the Performance dashboard
(learn more about using the console later on this page).This
out-of-the-box monitoring includes most network requests for your app.
However, some requests might not be reported or you might use a
different library to make network requests. In these cases, you can
use the Performance Monitoring API to manually instrument custom
network request traces. Firebase displays URL patterns and their
aggregated data in the Network tab in the Performance dashboard of the
Firebase console.
From stackoverflow thread,
The wire protocol for Cloud Firestore is based on gRPC, which is
indeed a lot harder to troubleshoot than the websockets that the
Realtime Database uses. One way is to enable debug logging with:
firebase.firestore.setLogLevel('debug');
Once you do that, the debug output will start getting logged.
Firestore use gRPC as their API, and charles not support gRPC now.
In this case you can use Mediator, Mediator is a Cross-platform GUI gRPC debugging proxy like Charles but design for gRPC.
You can dump all gRPC requests without any configuration.
For decode the gRPC/TLS traffic, you need download and install the Mediator Root Certificate to your device follow the document.
For decode the request/response message, you need download proto files which in your description, then configure the proto root in Mediator follow the document.
Do downloads use HTTP? How can they resume downloads after they have been suspended for several minutes? Can they request a certain part of the file?
Downloads are done over either HTTP or FTP.
For a single, small file, FTP is slightly faster (though you'll barely notice a differece). For downloading large files, HTTP is faster due to automatic compression. For multiple files, HTTP is always faster due to reusing existing connections and pipelining.
Parts of a file can indeed be requested independent of the whole file, and this is actually how downloads work. This is a process known as 'Chunked Encoding'. A browser requests individual parts of a file, downloads them independently, and assembles them in the correct order once all parts have been downloaded:
In chunked transfer encoding, the data stream is divided into a series of non-overlapping "chunks". The chunks are sent out and received independently of one another. No knowledge of the data stream outside the currently-being-processed chunk is necessary for both the sender and the receiver at any given time.
And according to FTP vs HTTP:
During a "chunked encoding" transfer, the sending party sends a stream of [size-of-data][data] blocks over the wire until there is no more data to send and then it sends a zero-size chunk to signal the end of it.
This is combined with a process called 'Byte Serving' to allow for resuming of downloads:
Byte serving begins when an HTTP server advertises its willingness to serve partial requests using the Accept-Ranges response header. A client then requests a specific part of a file from the server using the Range request header. If the range is valid, the server sends it to the client with a 206 Partial Content status code and a Content-Range header listing the range sent.
Do downloads use HTTP?
Yes. Especially since major browsers had deprecated FTP.
How can they resume downloads after they have been suspended for several minutes?
Not all downloads can resume after this long. If the (TCP or SSL/TLS) connection had been closed, another one has to be initiated to resume the download. (If it's HTTP/3 over QUIC, then it's another story.)
Can they request a certain part of the file?
Yes. This can be done with Range Requests. But it require server-side support (especially when the requested resource is provided by a dynamic script).
That other answer mentioning chunked transfer had mistaken it for the underlaying mechanism of TCP. Chunked transfer is not designed for the purpose of resuming partial downloads. It's designed for delimiting message boundary when the Content-Length header is not present, and when the communicating parties wish to reuse the connection. It is also used when the protocol version is HTTP/1.1 and there's a trailer fields section (which is similar to header fields section, but comes after the message body). HTTP/2 and HTTP/3 have their own way to convey trailers.
Even if multiple non-overlapping "chunks" of the resource is requested, it's encapsulated in a multipart/* message.