I'm trying to get fluenbit (td-agent-bit-1.6.3-win64) to send log lines from a file to a HTTP endpoint.
My problem is, that the endpoint can only accept 1 JSON object by POST at a time, and it looks like fluentbit always sends several JSON objects in a batch to the endpoint.
My configuration is:
[SERVICE]
flush 2
daemon off
log_level debug
[INPUT]
name tail
path c:\something\*.log
refresh_interval 2
[OUTPUT]
name http
match *
host myhost
port 1234
uri /something/logs
header content-type application/json
format json_stream
According to the docs, there are several formats for this output (https://docs.fluentbit.io/manual/pipeline/outputs/http)
I've tried using the json output format, but that sends multiple JSON objects wrapped by an array.
The json_stream format appears to send multiple JSON objects as well, separated by commas
And the json_lines also sends multiple objects.
I'd really appreciate any help with getting fluentbit to POST my logs one at a time
Thanks!
Related
I am trying to stream from firebase REST API:
I have around 500MB of data in firebase to a path, when i initiate a stream connection to it, it first wait for 2min and then end up closing connection with response pasted below.
Request: curl -Lv -X GET -H "Accept: text/event-stream" https://{URL} -o backup.txt
After trying 5 times, i have received all these response:
Response: Data requested exceeds the maximum size that can be accessed with a single request.
Response: {
"error" : "Internal server error."
}
My question is not about how to download all data, it is much about how to add stream Listener to a child in firebase database which has huge data in it.
When i add a listener to a path as mentioned in request, it close the connection because of all previous data present, How to achieve this?
The error Data requested exceeds the maximum size that can be accessed with a single request would suggest that you cannot perform the request you are trying because it is too large.
You will need to split it up into multiple requests and then join them afterwards.
I am working on a project which sends GPS longtitude and latitude to a server using Http POST method. I use GPRS of sim908 module and AT+Commands to communicate with this module.
Here are the commands related to Http Post:
AT+HTTPPARA="URL","http://'server'/'path':tcpPort'"
AT+HTTPACTION=1
AT+HTTPDATA= 'size','time'
The first command is used to set http parameters:
'server' = FQDN or IP address
'path' = path of file or directoy
'tcpPort' = default is 80
The second commands tells the module whether to use GET or POST method which is POST here.
The third one is used to recieve server response:
'size' = number of characters to read
'time' = set enough time to input all data with length of 'size'
I know how to send data using GET method. I must put a string like: "?var1=value1&var2=value2" at the end of the url. Here is an example: "http://www.example.com/test/getdata.php?TI=12.1&TO=22.2&TR=33.3"
But how is the POST method? Could anyone help me please?
Thanks alot.
[SOLVED]:
SIM908 does actually support post even though it is poorly documented:
AT+HTTPINIT
AT+HTTPPARA="CID",1
AT+HTTPPARA="URL","http://108.167.133.20/.../index.php"
NB*** AT+HTTPDATA=, ****NB
Wait for DOWNLOAD response then send data (bytes needs to be exact)
AT+HTTPACTION=1
AT+HTTPREAD=1,100000
I'm writing a NGINX module, and I would like to know if there is an option to send the client a response in parts? Meaning, in case of large files my server returns the response to the GET request in parts, I would like to send each part to the user without saving it to a temp file. I would like each part of the response to be sent and the client will get a few responses to the same request.
I want to do the following using NGINX Module :
Nginx receives a request, checks if it has the key to decode the request in the cache(custom)
if YES, then decode request, obtain an ID from it and check if there is a value against this ID in a key-value store (asynchronously) and return it in the response
if NO, then get the new key from the key-value store (asynchronously) and then store this key in the cache and use it to decode the request. Obtain the ID and check if there is a value against this ID in the key-value store(asynchronously) and send it in the response.
I was able to figure out how to do step 1, i wrote an upstream module by referring openresty's nginx module from github. For achieving step 2 functionality, i tried creating a new upstream request in the process_header() function of the first upstream call (i.e getting the key from the store), but this didn't work. How to achieve this ?
Thanks in advance.
I see 2 approaches:
You may do it all in Lua using lua-nginx-module and lua-resty-redis library. Here you may find some info Configure-nginx-to-get-url-from-redis-with-key-and-proxy-the-url-to-other-server
Write nginx C module, use redis2-nginx-module as upstream module, send subrequest. Take a look at my answer to Subrequests are not sent or the request hangs It shows how to send subrequests.
I want to create a script that checks an URL and perform an action (download + unzip) when the "Last-Modified" header of the remote file changed. I thought about fetching the header with curl but then I have to store it somewhere for each file and perform a date comparison.
Does anyone have a different idea using (mostly) standard unix tools?
thanks
A possible solution would be periodically running this algorithm on the client box.
Create a HTTP request indicating the If-Modified-Since header equal to the date of your local file. If the file does not exist yet do not include this header;
The server will either send you the file if it was changed since the If-Modified-Since header in the payload or send 304 Not Modified HTTP status.
If you receive a 200 OK HTTP status simply get the payload from the HTTP body and unzip the file.
If in the other hand you received a 304 Not Modified you know that your file is up-to-date.
Use the Last-Modified header to touch your local file. This way you will be in sync with the server datetime.
Another way would be for the server to push notifications (a broadcast package for example) when the file is changed. When the notification is received the client would then execute the above algorithm. This would imply code to live in the HTTP server that listens for file system changes and then broadcast them to interested parties.
Perhaps this info for the curl command is of some importance:
TIME CONDITIONS
HTTP allows a client to specify a time
condition for the document it
requests. It is If-Modified-Since or
If-Unmodified-Since. Curl allow you to
specify them with the -z/--time-cond
flag.
For example, you can easily make a
download that only gets performed if
the remote file is newer than a local
copy. It would be made like:
curl -z local.html
http://remote.server.com/remote.html
Or you can download a file only if the
local file is newer than the remote
one. Do this by prepending the date
string with a '-', as in:
curl -z -local.html
http://remote.server.com/remote.html
You can specify a "free text" date as
condition. Tell curl to only download
the file if it was updated since
yesterday:
curl -z yesterday
http://remote.server.com/remote.html
Curl will then accept a wide range of
date formats. You always make the date
check the other way around by
prepending it with a dash '-'.prepending it with a dash '-'.
To sum up, you will need:
curl command
touch command
some bash scripting
is Java applicable in your case? I did a similar thing in one of my homework using the Apache HTTPcore library, you need to add the header "If-Modified-Since" to your HTTP request before you send it to the server, if the status code of the response that you receive from the server is not 304 then you know that the file has changed since the time value that you're checking against.