Using one of these parser plugin configurations:
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
# Command | Decoder | Field | Optional Action
# =============|==================|=================
Decode_Field_As escaped_utf8 log do_next
Decode_Field_As json log
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
Is it possible to determine the depth to which the the plugin parses fields.
To make that more clear,
Let's say that you're trying to log an HTTP response from a Docker container containing a large body with multiple items.
Fluentbit should parse the Docker log, which it does. Great.
But it shouldn't be parsing the entire body as well.
I don't want 1000 fields in Elasticsearch, because it's trying to parse all items in my response body too.
{
"length":"value",
"timestamp":"timestamp",
"items": [
{...},
{...},
{...},
...
]
}
Is there any way to configure the depth to which Fluentbit will parse JSON?
Thanks!
Related
Attempting to parse some Tomcat logs that contain log Exception messages using Fluent Bit but I am struggling to parse the multiline exception messages and logs into a single log entry. I can successfully parse the logs the way I desire, when the log is static and is not being written to and enabling read_from_head true; I can confirm this because this is what a sample output looks like (Exception is successfully written to one log entry).
{
"filename": "/opt/tomcat/catalina.out",
"fluentTimestamp": 1660079524549258200,
"hostname": "ip-172-22-11-142",
"msg": "com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container\n\tjava.io.EOFException: No content to map to Object due to end of input\n\ blah blah blahg you get the idea",
"severity": "SEVERE",
"thread": "http-nio-8080-exec-4",
"timestamp": "09-Aug-2022 01:42:35.475"
}
But when I tail the file live, Fluent bit appears to ignore my parsing regex as I only get a "log" statement and none of the other fields. I'm assuming because the logs are partially being streamed and written and my regex is invalid. Here's a sample of my log file:
09-Aug-2022 06:36:45.901 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 18109 ms
09-Aug-2022 06:43:04.787 SEVERE [http-nio-8080-exec-2] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.io.EOFException: No content to map to Object due to end of input
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2766)
at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2682)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1308)
at org.codehaus.jackson.jaxrs.JacksonJsonProvider.readFrom(JacksonJsonProvider.java:419)
at com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.readFrom(JacksonProviderProxy.java:139)
at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:490)
at com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
at com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:86)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at java.base/java.lang.Thread.run(Thread.java:829)
09-Aug-2022 06:43:07.060 SEVERE [http-nio-8080-exec-3] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
My config looks like this:
[SERVICE]
flush 5
daemon Off
log_level debug
parsers_file parsers.conf
[INPUT]
name tail
tag tomcat
path /opt/tomcat/catalina.out
multiline on
parser_firstline java_multiline
mem_buf_limit 5MB
skip_long_lines on
#read_from_head true
refresh_interval 10
[FILTER]
Name record_modifier
Match *
Record hostname ${HOSTNAME}
[OUTPUT]
name http
match *
host somelocation
My parser config:
[PARSER]
Name java_multiline
Format regex
Regex (?<timestamp>([012][0-9])-(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)-\d\d\d\d [012]\d:[0-6]\d:[0-6]\d\.\d\d\d)\s(?<severity>[^\s]+)\s\[(?<thread>[^\]]+)\]\s(?<msg>[^*]+)
I also tried to do a multiline parser but still to no avail:
[MULTILINE_PARSER]
name multiline-regex-test
type regex
flush_timeout 1000
#
# Regex rules for multiline parsing
# ---------------------------------
#
# configuration hints:
#
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
#
# rules | state name | regex pattern | next state
# ------|---------------|--------------------------------------------
rule "start_state" "/([012][0-9])-(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)-\d\d\d\d [012]\d:[0-6]\d:[0-6]\d\.\d\d\d/" "cont"
rule "cont" "/[^\t].+/" "cont"
Again, it works fine when parsing it statically but doesn't seem to work when I'm tailing the file live. Is there something I'm missing here?
What version are you using? If using 1.8+ then use the built-in java parser. See https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support
[INPUT]
name tail
...
multiline.parser java
I would like to mark the last time entry for a user as the current time entry. In other words, I would like to clear the end field of a given time entry, to indicate that it it not finished yet, it's still running.
Is it possible to do that?
I tried using the Update time entry on workspace endpoint, like so:
curl -H 'Content-Type':'application/json' \
-H 'X-Api-Key':'API_KEY' \
-X PUT https://api.clockify.me/api/workspaces/WORKSPACE_ID/timeEntries/TIME_ENTRY_ID \
-d '{"end":null}'
But the response code is 400 with the following response:
{"message":"text","code":3002}
What does this error code mean?
So the documentation for the UpdateTimeEntryRequest (https://clockify.github.io/clockify_api_docs/#/definitions/UpdateTimeEntryRequest) is a bit wrong or incomplete. I figured it out through some testing, here's what you need to do:
First, you need to make sure you have the current data of the time entry, specifically the start date and billable status (as these are the only two "required" fields in the API call). You are getting the 400 bad request because you are missing these two fields - the error message is obviously no help here, but a 400 response code means "the server was unable to process the request sent by the client due to invalid syntax.", which makes sense, since your input is invalid.
Assuming you have the time entry information already (which is the only way I could see how you would know the ID), you could recreate the whole UpdateTimeEntryRequest data but then null out the end field or leave it out completely, like this:
{
"start": START_FROM_THE_TIME_ENTRY,
"billable": BILLABLE_FROM_THE_TIME_ENTRY,
"description": DESC_FROM_THE_TIME_ENTRY,
"projectId": PROJECT_ID,
"taskId": TASK_ID,
"end": null,
"tagIds": TAGS_ARRAY
}
From my testing, this restarts the timer. If you leave out any field, those fields are reset, for example:
{
"start": START_FROM_THE_TIME_ENTRY,
"billable": BILLABLE_FROM_THE_TIME_ENTRY
}
would restart the timer, but it would clear the description, project/task, and tags, which is probably not preferable.
I am trying to make a post request via R using the httr package to composer rest server. I have written a code and then created the composer rest server from it. These are my details
Request URL : http://localhost:3000/api/nl.amis.registry.fruits
Body: {
"$class": "nl.amis.registry.fruits",
"Id": "9",
"name": "orange",
"description": "string",
"count": ""
}
First, I have tried with the composer rest server. For my purpose, I needed the count to be blank and the value will be appended by another API call. I was able to make the transaction successfully with the count: "". This I was able to check in the test section of the composer playground. The remaining code works fine which appends the count variable later on.
Now I am writing an R code to make a similar transaction through POST request. Here I am facing an error that "count cannot be blank" and returns with error 422 Unprocessable entity. The content type I was used was application/json. While using the "count":{} , the post request process fine and i am getting "count":[object Object] in the response. But the later on code which does the appending will do something like count:"[object Object],1" wherein I am expecting "count":"1". Everything works fine while using the test in composer playground but while trying to access externally via rest API is creating the problem. Please help.
you can use an Optional keyword to declare a count in an asset of the model file. using Optional keyword you can post an empty value of count.
for example:
asset fruits identified by Id {
o String Id
o String name
o String description
o String count optional
}
I'm trying to use Riak's mapreduce via http. his is what i'm sending:
{
"inputs":{
"bucket":"test",
"key_filters":[["matches", ".*"]]
},
"query":[
{
"map":{
"language":"erlang",
"source":"value(RiakObject, _KeyData, _Arg) -> Key = riak_object:key(RiakObject), Count = riak_kv_crdt:value(RiakObject, <<\"riak_kv_pncounter\">>), [ {Key, Count} ]."
}
}
]}
Riak fails with "[worker_startup_failed]", which isn't very informative. Could anyone please help me get this to actually execute the function?
WARNING
Allowing arbitrary Erlang functions via map-reduce is a security risk. Any valid Erlang can be executed, including sending your entire data set offsite or formatting the hard drive.
You have been warned.
However, if you implicitly trust any client that may connect to your cluster, you can allow Erlang source to be passed in a map-reduce request by setting {allow_strfun, true} in the riak_kv section of app.config, (or in the advanced.config if you are using riak.conf).
Once you have allowed passing an Erlang function in a map-reduce phase, you need to pass in a function of the form fun(RiakObject,KeyData,Arg) -> [result] end. Note that this must be an anonymous fun, so fun is a keyword, not a name, and it must end with end.
Your function should handle the case where {error,notfound} is passed as the first argument instead of an object. Simply adding a catch-all clause to the function could accomplish that.
Perhaps something like:
{
"inputs":{
"bucket":"test",
"key_filters":[["matches", ".*"]]
},
"query":[
{
"map":{
"language":"erlang",
"source":"fun(RiakObject, _KeyData, _Arg) ->
Key = riak_object:key(RiakObject),
Count = riak_kv_crdt:value(
RiakObject,
<<\"riak_kv_pncounter\">>),
[ {Key, Count} ];
(_,_,_) -> [{error,0}]
end."
}
}
]}
Allowing the source to be passed in the request is very useful while developing and debugging. For production, you really should put the functions in a dedicated pre-compiled module that you copy to the code path of each node so that the phase spec can specify the module and function by name instead of providing arbitrary code.
{"map":{
"language":"erlang",
"module":"yourprecompiledmodule",
"function":"functionname"}}
You need to enable allow_strfun on all nodes in your cluster. To do so in Riak 2, you will need to use the advanced.config file to add this to the riak_kv configuration:
[
{riak_kv, [
{allow_strfun, true}
]}
].
The other option is to create your own Erlang module by using the compiler shipped with Riak and placing the *.beam file in a well-known location for Riak to find. The basho-patches directory is one such place.
Please see the documentation as well:
advanced.config
Installing custom Erlang code
HTTP MapReduce
Using MapReduce
Advanced MapReduce
MapReduce / curl example
I want to use Jmeter to update a document on solr using http post.
I want it to take a different file to update in every iteration, create a proper http post request and monitor the responses from the server.
Can someone guide me of how this can be done:
Taking a different file every time.
Creating a http post from it.
Your use case can be split into 2 parts:
Get list of files to send
Send them to server
In regards to point 1, I would suggest to obtain file list via scripting.
Assuming following Test Plan Structure:
Add a Thread Group (all defaults)
Add a JSR223 Sampler as a child of Thread Group
Select "beanshell" as language
In "Script" area add following code:
File folder = new File("PATH TO FOLDER WITH YOUR FILES");
File [] files2send = folder.listFiles();
int counter = 1;
for (File file : files2send)
{
vars.put("FILE_" + counter, file.getPath());
counter++;
}
This will save files, you will be sending as JMeter Variables like:
FILE_1=d:\2solr\sometxtfile.txt
FILE_2=d:\2solr\somewordfile.docx
FILE_3=d:\2solr\someexcelfile.xlsx
After that you can use For Each Controller to iterate through files and add them to your request
Add For Each Controller as a child of Thread Group (same level as JSR223 Sampler)
Make sure that For Each Controller has following configuration:
Input variable prefix: FILE
Output variable name: CURRENTFILE
Add _ before number is checked
Add HTTP Request as a child of For Each Controller
Access file you want to send as ${CURRENTFILE} in "Send Files With The Request" stanza of the HTTP Request
It's just one of the approaches, if you are not very comfortable with JSR233 or Beanshell you may wish to use CSV Data Set Config instead.