I am trying to stream from firebase REST API:
I have around 500MB of data in firebase to a path, when i initiate a stream connection to it, it first wait for 2min and then end up closing connection with response pasted below.
Request: curl -Lv -X GET -H "Accept: text/event-stream" https://{URL} -o backup.txt
After trying 5 times, i have received all these response:
Response: Data requested exceeds the maximum size that can be accessed with a single request.
Response: {
"error" : "Internal server error."
}
My question is not about how to download all data, it is much about how to add stream Listener to a child in firebase database which has huge data in it.
When i add a listener to a path as mentioned in request, it close the connection because of all previous data present, How to achieve this?
The error Data requested exceeds the maximum size that can be accessed with a single request would suggest that you cannot perform the request you are trying because it is too large.
You will need to split it up into multiple requests and then join them afterwards.
Related
I'm trying to get fluenbit (td-agent-bit-1.6.3-win64) to send log lines from a file to a HTTP endpoint.
My problem is, that the endpoint can only accept 1 JSON object by POST at a time, and it looks like fluentbit always sends several JSON objects in a batch to the endpoint.
My configuration is:
[SERVICE]
flush 2
daemon off
log_level debug
[INPUT]
name tail
path c:\something\*.log
refresh_interval 2
[OUTPUT]
name http
match *
host myhost
port 1234
uri /something/logs
header content-type application/json
format json_stream
According to the docs, there are several formats for this output (https://docs.fluentbit.io/manual/pipeline/outputs/http)
I've tried using the json output format, but that sends multiple JSON objects wrapped by an array.
The json_stream format appears to send multiple JSON objects as well, separated by commas
And the json_lines also sends multiple objects.
I'd really appreciate any help with getting fluentbit to POST my logs one at a time
Thanks!
When http client downloads something via GET method and receives status code 200, is all response content already buffered in memory ? Response content maybe quite large, like 1G.
The status line containing the status code is the first thing that is received, the rest of the data can still be in-flight or not even sent yet.
I'm writing a NGINX module, and I would like to know if there is an option to send the client a response in parts? Meaning, in case of large files my server returns the response to the GET request in parts, I would like to send each part to the user without saving it to a temp file. I would like each part of the response to be sent and the client will get a few responses to the same request.
I am having difficulties understanding the HTTP request and response flow. I am working with a system where I can "hijack" incoming HTTP request and give my own response. The problem I am having is that some type of GET request seem to assume that all data is sent back in first request.
For instance, JPEG image requests, no matter the size (my tests include 0-20 MB JPEG files) seems to assume that the entire data is sent in the first response. Even if I don't send any data and explicitly set range header to 0 I never get a response back from the client asking for the data.
Other data request types, such as mp4 video, the client seems perfectly fine with getting a response with only header information with no data and then sends a new request explicitly asking for byte range 0-.
Is there some kind of agreement between the the client and server that some types should be sent back in one request while others can be split up in a number of requests?
I want to create a script that checks an URL and perform an action (download + unzip) when the "Last-Modified" header of the remote file changed. I thought about fetching the header with curl but then I have to store it somewhere for each file and perform a date comparison.
Does anyone have a different idea using (mostly) standard unix tools?
thanks
A possible solution would be periodically running this algorithm on the client box.
Create a HTTP request indicating the If-Modified-Since header equal to the date of your local file. If the file does not exist yet do not include this header;
The server will either send you the file if it was changed since the If-Modified-Since header in the payload or send 304 Not Modified HTTP status.
If you receive a 200 OK HTTP status simply get the payload from the HTTP body and unzip the file.
If in the other hand you received a 304 Not Modified you know that your file is up-to-date.
Use the Last-Modified header to touch your local file. This way you will be in sync with the server datetime.
Another way would be for the server to push notifications (a broadcast package for example) when the file is changed. When the notification is received the client would then execute the above algorithm. This would imply code to live in the HTTP server that listens for file system changes and then broadcast them to interested parties.
Perhaps this info for the curl command is of some importance:
TIME CONDITIONS
HTTP allows a client to specify a time
condition for the document it
requests. It is If-Modified-Since or
If-Unmodified-Since. Curl allow you to
specify them with the -z/--time-cond
flag.
For example, you can easily make a
download that only gets performed if
the remote file is newer than a local
copy. It would be made like:
curl -z local.html
http://remote.server.com/remote.html
Or you can download a file only if the
local file is newer than the remote
one. Do this by prepending the date
string with a '-', as in:
curl -z -local.html
http://remote.server.com/remote.html
You can specify a "free text" date as
condition. Tell curl to only download
the file if it was updated since
yesterday:
curl -z yesterday
http://remote.server.com/remote.html
Curl will then accept a wide range of
date formats. You always make the date
check the other way around by
prepending it with a dash '-'.prepending it with a dash '-'.
To sum up, you will need:
curl command
touch command
some bash scripting
is Java applicable in your case? I did a similar thing in one of my homework using the Apache HTTPcore library, you need to add the header "If-Modified-Since" to your HTTP request before you send it to the server, if the status code of the response that you receive from the server is not 304 then you know that the file has changed since the time value that you're checking against.