Using One of the cookbook examples, I am trying to make a server like so in Rebol2:
listen: open tcp://:9999
waitports: [listen]
forever [
data: wait waitports
either same? data listen [
active-port: first listen
append waitports active-port
][
incoming-from-remote: first data
print incoming-from-remote
]
]
With a client I can connect and send messages by inserting them, but when I close the port from the client side, I get the following error on the server:
** Script Error: Out of range or past end
** Where: forever
** Near: incoming-from-remote: first data
print incoming-from-remote
How can I handle this better?
put this in your either block
either any [
not incoming-from-remote: copy data
"" = incoming-from-remote
] [
attempt [
close data
remove find waitports data
]
] [
print ..
]
Related
I am trying to send fluentbit metrics to an external source for processing. My understanding from the documentation is that the fluentbit_metrics input is intended to be used with output plugins that are for specific telemetry solutions like Prometheus, OpenTelemetry, etc. However, for my purposes, I cannot actually use any of those solutions and instead have to use a different bespoke metrics solution. For this to work, I would like to just send lines of text to a port that my metrics solution is listening on.
I am trying to use the fluentbit forward output to send data to this endpoint, but I am getting an error in response from my metrics solution because it is receiving a big JSON object which it can't parse. However, when I output the same fluentbit_metrics input to a file or to stdout, the contents of the file is more like what I would expect, where each metric is just a line of text. If these text lines were what was being sent to my metrics endpoint, I wouldn't have any issue ingesting them.
I know that I could take on the work to change my metrics solution to parse and process this JSON map, but before I do that, I wanted to check if this is the only way forward for me. So, my question is, is there a way to get fluentbit to send fluentbit_metrics to a forward output where it does not convert the metrics into a big JSON object? Is the schema for that JSON object specific to prometheus? Is there a reason why the outputs differ so substantially?
Here is a copy of an example config I am using with fluentbit:
[SERVICE]
# This is a commented line
Daemon off
log_level info
log_file C:\MyFolder\fluentlog.txt
flush 1
parsers_file .\parsers.conf
[INPUT]
name fluentbit_metrics
tag internal_metrics
scrape_interval 2
[OUTPUT]
Name forward
Match internal_metrics
Host 127.0.0.1
Port 28232
tag internal_metrics
Time_as_Integer true
[OUTPUT]
name stdout
match *
And here is the output from the forward output plugin:
{
"meta": {
"cmetrics": {},
"external": {},
"processing": {
"static_labels": []
}
},
"metrics": [
{
"meta": {
"ver": 2,
"type": 0,
"opts": {
"ns": "fluentbit",
"ss": "",
"name": "uptime",
"desc": "Number of seconds that Fluent Bit has been running."
},
"labels": [
"hostname"
],
"aggregation_type": 2
},
"values": [
{
"ts": 1670884364820306500,
"value": 22,
"labels": [
"myHostName"
],
"hash": 16603984480778988994
}
]
}, etc.
and here is the output of the same metrics from stdout:
2022-12-12T22:02:13.444100300Z fluentbit_uptime{hostname="myHostName"} = 2
2022-12-12T22:02:11.721859000Z fluentbit_input_bytes_total{name="tail.0"} = 1138
2022-12-12T22:02:11.721859000Z fluentbit_input_records_total{name="tail.0"} = 12
2022-12-12T22:02:11.444943400Z fluentbit_input_files_opened_total{name="tail.0"} = 1
I am sending logs to an azure eventhub with Serilog (using WriteTo.AzureEventHub(eventHubClient)), after that I am running a filebeat process with the azure module enabled, so I send these logs to elasticsearch to be able to explore them with Kibana.
The problem I have is that all the information goes to the field "message", I would need to separate the information of my logs in different fields to be able to do good queries.
The way I found was create an ingest pipeline in Kibana and through a grok processor I separate the fields inside the "meessage" and generate multiple fields. In the filebeat.yml I set the pipeline name, but nothing happen, it seems the pipeline is not working.
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
pipeline: "filebeat-otc"
Does anybody knows what I am missing? THANKS in advance.
EDITION. I will add an example of my pipeline and my data. In the simulation is working properly:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIME:timestamp}\\s%{LOGLEVEL}\\s{[a-zA-Z]*:%{UUID:CorrelationID},[a-zA-Z]*:%{TEXT:OperationTittle},[a-zA-Z]*:%{TEXT:OriginSystemName},[a-zA-Z]*:%{TEXT:TargetSystemName},[a-zA-Z]*:%{TEXT:OperationProcess},[a-zA-Z]*:%{TEXT:LogMessage},[a-zA-Z]*:%{TEXT:ErrorMessage}}"
],
"pattern_definitions": {
"LOGLEVEL" : "\\[[^\\]]*\\]",
"TEXT" : "[a-zA-Z0-9- ]*"
}
}
}
]
},
"docs": [
{
"_source": {
"message": "15:13:59 [INF] {CorrelationId:83355884-a351-4c8b-af8d-b77c48462f36,OperationTittle:Operation1,OriginSystemName:Fexa,TargetSystemName:Usina,OperationProcess:Testing Log Data,LogMessage:Esto es una buena prueba,ErrorMessage:null}"
}
},
{
"_source": {
"message": "20:13:48 [INF] {CorrelationId:8451ee54-efca-40be-91c8-8c8e18e33f58,OperationTittle:null,OriginSystemName:Fexa,TargetSystemName:Donna,OperationProcess:Testing Log Data,LogMessage:null,ErrorMessage:null}"
}
}
]
}
It seems when you use a module it will create and use an ingest pipeline in elasticsearch, and the pipeline option in the output is ignored.
So my solution was modify the index.final_pipeline. For this, in Kibana I went to Stack Management / Index Management there I found my index, there I went to Edit Settings and set "index.final_pipeline": "the-name-of-my-pipeline".
I hope this helps to anybody.
This was thanks to leandrojmp
I have a lambda function that converts my logs to this format:
{
"events": [
{
"field1": "value",
"field2": "value",
"field3": "value"
}, (...)
]
}
When I query it on S3, I get in this format:
[
{
"events": [
{ (...) }
]
}
]
And I'm trying to run a custom classifier for it because the data I want is inside the objects kept by 'events' and not events itself.
So I started with the simplest path I could think that worked in my tests (https://jsonpath.curiousconcept.com/)
$.events[*]
And, sure, worked in the tests but when I run a crawler against the file, the table created includes only an events field with a struct inside it.
So I tried a bunch of other paths:
$[*].events
$[*].['events']
$[*].['events'].[*]
$.[*].events[*]
$.events[*].[*]
Some of these does not even make sense and absolutely every one of those got me an schema with an events field marked as array.
Can anyone point me to a better direction to handle this issue?
I have a druid service which runs at my local machine at port 8082 as follows:
Method POST: http://localhost:8082/druid/v2/?pretty
Body:
{
"queryType" : "topN",
"dataSource" : "some_source",
"intervals" : ["2015-09-12/2015-09-13"],
"granularity" : "all",
"dimension" : "page",
"metric" : "edits",
"threshold" : 25,
"filter": {
"type": "and",
"fields": [
{
"type": "selector",
"dimension": "pix_id",
"value": "1234"
}
}
}
Hitting this query gives me a list of records based on the value of the dimension 'pix_id'.
Now, I want to setup an nginx such that the external application should not have any clue about my druid service. I just want the external application to hit the URL:
http://localhost:80/pix_id/98765
This url should dynamically generate a JSON with the above mentioned pix_id and send a request to druid and return the response to the user.
Is it possible to do this in nginx?
Yes you can do this, but rather I would suggest to have a php or python script in between to give the results.
So the setup would be -
Have php page receive the request.
make a curl call from php to the druid, locally.
get the result and pass on the response.
There are multiple benefits of doing this eg. -
You completely mask druid, and not necessarily limited to druid.
You can do more calculations in php before sending the request to druid.
caching at php end.
If you build up a block structure, convert it to a string with MOLD, and write it to a file like this:
>> write %datafile.dat mold [
[{Release} 12-Dec-2012]
[{Conference} [12-Jul-2013 .. 14-Jul-2013]]
]
You can LOAD it back in later. But what about headers? If a file contains code, it is supposed to start with a header like:
rebol [
title: "Local Area Defringer"
date: 1-Jun-1957
file: %defringe.r
purpose: {
Stabilize the wide area ignition transcriber
using a double ganged defringing algorithm.
}
]
If you are just writing out data and reading it back in, are you expected to have a rebol [] header, and extend it with any properties you want to add? Should you come up with your own myformat [] header concept with your own properties?
Also, given that LOAD does binding, does it make sense to use it for data or is there a different operation?
Rebol data doesn't have to have a header, but is best practice to include one (even if it's just data).
Some notes:
SAVE is your best bet for serializing to file! or port! and has a mechanism for including a header.
MOLD and SAVE both have an /ALL refinement that corresponds to LOAD (without /ALL, some data from MOLD and SAVE cannot be reliably recovered, including Object, Logic and None values).
LOAD discards the header, though you can load it using the /HEADER refinement.
Putting this together:
save/all/header %datafile.dat reduce [next "some" 'data][
title: "Some Data"
]
header: take data: load/header %datafile.dat
To use a header other than Rebol [], you'd need to devise a separate loader/saver.
For the case of reading, construct works very well alongside load to prevent evaluation (of code as opposed to data):
prefs: construct/with load %options.reb default-prefs
It is:
Similar to context
obj: [
name: "Fred"
age: 27
city: "Ukiah"
]
obj-context: context obj
obj-construct: construct obj
In this case, the same:
>> obj-context = obj-construct
== true
Different
when it comes to evaluating code:
obj-eval: [
name: uppercase "Fred"
age: 20 + 7
time: now/time
]
obj-eval-context: context obj-eval
obj-eval-construct: construct obj-eval
This time parsing differently:
>> obj-eval-context = obj-eval-construct
false
>> ?? obj-eval-construct
obj-eval-construct: make object! [
name: 'uppercase
age: 20
time: now/time
]
Aside:
This is the point I realize the following code wasn't behaving as I expected:
obj-eval: [
title: uppercase "Fred"
age: 20 + 7
city: "Ukiah"
time: now/time
]
gives in red (and by extension, rebol2):
>> obj-eval-construct: construct obj-eval
== make object! [
title: 'uppercase
age: 20
city: "Ukiah"
time: now/time
]
lit-word! and lit-path! is different.
TODO: question
It has also
Useful refinement /with
Which can be used for defaults, similar to make