We are using fluent-bit plugin to tail from a file and send to an HTTP endpoint.
The sample log looks like the following.
tenant 1 testing 100
The configuration for input looks like the following.
[INPUT]
Name tail
Path /var/log/input/**/*.log
Tag tenant
Path_Key filename
We then use a lua filter to add a key based on the filepath. This works as expected.
[FILTER]
Name lua
Match *
script /etc/td-agent-bit/test.lua
call extract_id
At this point, we try to filter the message and rewrite the tag based on the tenantid.
[FILTER]
Name rewrite_tag
Match *
Rule $tenantid ^([a-z]+)-([0-9]+)$ from.$tenantid false
Emitter_Name re_emitted
With a stdout, like below,
[OUTPUT]
Name stdout
Match *
we verified the message to be like the following.
tenant: [1630073320.394812583, {"log"=>"tenant 1 testing 100", "tenantid"=>"tenant1", "filename"=>"/var/log/input/tenant1/file1.log"}]
It looks like the rewrite_tag plugin is not able to work and change the tag as expected. Is there a problem with the regex pattern ? Any help on this will be hugely appreciated.
I believe your regex needs a very slight tweak to remove the dash in the match pattern. At the moment, it's looking for abc-123, but your tenant id format is abc123.
By removing the dash from your regex, the example tenantid field should match:
^([a-z]+)-([0-9]+)$ will match "tenant-1"
^([a-z]+)([0-9]+)$ will match "tenant1"
Related
I'm trying to create a fluent-bit config which uses a record's timestamp to a custom key using a filter. Something like:
[INPUT]
Name tail
Path /some/path
...
[FILTER]
Name record_modifier
Match *
Record fluenbit_orig_ts SOME_MAGIC_WAY_TO_GET_UNIXTIME
[OUTPUT]
Name stdout
Match *
The rationale for this that I'm using several parsers, each has its own time format (Time_Format, as it's used in the regular expression parser. Even if I used Time_Keep, it won't help because the time is specified differently by different services, with a different Time_Format). I'd like the records getting to an [OUTPUT} to have the same key to describe the timestamp. In this example that key would be fluenbit_orig_ts
Is this possible?
I got an answer from the fluent-bit slack channel.
Seems like this is possible using lua filters. Specifically, this example seems to be relevant: https://github.com/fluent/fluent-bit/blob/master/scripts/append_tag.lua
I've same issue.
According to #BugoK, I've solved.
This is lua script.
function append_tag(tag, timestamp, record)
new_record = record
new_record["log_time"] = os.date("%Y-%m-%d %H:%M:%S")
return 1, timestamp, new_record
end
And this is td-agent-bit config.
[FILTER]
Name lua
Match nginx.access
script override_time.lua
call append_tag
And then restart td-agent-bit. It works~!
I'm attempting to construct a series of Paw calls using the variables feature. I have one situation I'm unable to solve.
At authentication into the server I'm using, I get a JSON response, with one value that looks like this:
endpoint = "https://sub.something.com/thingone/thingtwo.php?token=sometoken&id=blahblah"
The endpoint portion "https://sub.something.com/" is then used as the base for subsequent calls, where a call might be "GET https://sub.something.com/data?id=123".
I don't want to hardcode the endpoint in Paw, as the endpoint will vary based on factors I can't predict at my end.
Is there a way to do basic string processing like this either in Paw, or by calling out to a shell script and using the return value of said script as a Paw variable?
That's doable using that RegExp Match dynamic value extension. Click on that previous link and hit Install Extension.
Type "Regexp" in the field you expect this value to be used. Pick Regexp Match from the completion results:
Then enter a regexp that matches your need, https?://[^/]+/? should be good:
I've put your example string in the screenshot above to show that it works, but you can instead put a "pointer" (Response Dynamic Value) to the response you want:
In the choices, pick Response Parsed Body if you want to parse a JSON or XML from the reponse. If the string is simply in plain text in the response body, pick Response Raw Body.
Once these steps are completed, you've got a working "Pointer" + "Parser" to the response that extract the part of the string you need. You can do the same operation with another regex for the token…
Tip: these dynamic value tokens can be selected like text and copy/pasted (Cmd+C/Cmd+V) :-)
Hello i need a way to find out the host part of an url , i've tried
Request.Url.Host.Split('.')
but it doesn't work with url like this:
sub.sub.domain.com
or
www.domain.co.uk
since you can have a variable number of dots before and after the domain
i need to get only "domain"
Check out the second answer at Get just the domain name from a URL?
I checked the pastebin link; it's active. I didn't test the code myself, but if it outputs as he describes, you can .split() from there.
If you need to be totally flexibel, you need to make a list of all possible top-level-domains, and try to remove those, with dot, from the end of your string, resulting in
www.domain
or
sub.sub.domain
Then take the last characters after the last dot.
i am trying to rewrite any URL that match this pattern:
~/Ahmed
~/Name
to this:
~/User/Ahmed/Ahmed.aspx
~/User/Name/Name.aspx
and i can write them individually but what i am trying to do is detect any URL that look like "~/User/Ahmed/Ahmed" and auto rewrite them to this "Ahmed"
thanks
Hopefully you're using the UrlRewritingNet library, not UrlRewriter? The former is suggested over the latter.
However, in either you can use a regex:
"~/User/([^/\\]+)/\1.aspx" -> "~/$1" //For ".aspx" in the URL
"~/([A-Za-z]+)" to "~/User/$1/$1.aspx" //For /Name in the URL.
Note the ([^/\]+) means any set of characters without slashes,
and "\1" is a backreference to the previous capture, that ensures the name is an exact duplicate. Note that you should enable "ignore case" if you want to support "/User/ahmed/Ahmed.aspx" and not just "/User/Ahmed/Ahmed.aspx".
The reason I'm asking is because IIS protects certain ASP.NET folders, like Bin, App_Data, App_Code, etc. Even if the URL does not map to an actual file system folder IIS rejects a URL with a path segment equal to one of the mentioned names.
This means I cannot have a route like this:
{controller}/{action}/{id}
... where id can be any string e.g.
Catalog/Product/Bin
So, instead of disabling this security measure I'm willing to change the route, using a suffix before the id, like these:
{controller}/{action}_{id} // e.g. Catalog/Product_Bin
{controller}/{action}/_{id} // e.g. Catalog/Product/_Bin
But these routes won't work if the id contains the new delimeter, _ in this case, e.g.
// These URL won't work (I get 404 response)
Catalog/Product_Bin_
Catalog/Product/_Bin_
Catalog/Product/__Bin
Why? I don't know, looks like a bug to me. How can I make these routes work, where id can be any string?
Ok, I have a definitive answer. Yes, this is a bug. However, at this point I regret to say we have no plans to fix it for a couple of reasons:
It's a breaking change and could be a very hard to notice one at that.
There's an easy workaround.
What you can do is change the URL to not have the underscore:
{controller}/{action}/_{id}
Then add a route constraint that requires that the ID parameter starts with an underscore character.
Then within your action method you trim off the underscore prefix from the id parameter. You could even write an action filter to do this for you if you liked. Sorry for the inconvenience.
You can use characters that are not allowed for a directory or file name like: *,?,:,",<,>,|.
With ASP.NET MVC if you look at the source they have a hard-coded value for the path separator (/) and to my knowledge cannot be changed.