Ok im fishing out, amongst other things, the first segment of a unique ID from a log line with a grok filter, like this (Its only the first segment that I care about, throw away the rest). This segment is hex ,and I want it in binary.
The line:
Transaction: 000178ec-XXXX-XXXX-XXXX-XXXXXXXXXXXX
The filter is like :
Transaction: %{BASE16NUM:transaction_id}-%{GREEDYDATA:otherpartsidontcareabout}
But it just gives me this result:
{
"transaction_id": [
[
"000178ec"
],
"otherpartsidontcareabout":
[
"XXXX-XXXX-XXXX-XXXXXXXXXXXX"
]]
}
Where I had expected it to be transformed into decimal:
{
"transaction_id": [
[
"96492"
],
"otherpartsidontcareabout":
[
"XXXX-XXXX-XXXX-XXXXXXXXXXXX"
]]
}
Am I doing this wrong? Im really stuck.
BASE16NUM just indicates the pattern of the text. To convert it to an integer you are going to have to do something like this:
filter {
ruby {
code => "event['transaction_id'] = event['transaction_id'].hex"
}
}
or for logstash 5
filter {
ruby {
code => "event.set('transaction_id', event.get('transaction_id').hex)"
}
}
Related
I am using wiremock for stubbing and it uses Jayway JsonPath.
I want to create a stub only when the json element doesn't contain exactly 10 digits.
stub is
"request": {
"bodyPatterns": [
{
"matchingJsonPath": "$.employees[?(#.employeeContact =~/[^0-9]{10}/)]"
}
]
}
I have tried multiple combinations like:
1. $.employees[?(#.employeeContact =~/[^0-9]{10}/)]
2. $.employees[?(#.employeeContact =~/^[0-9]{10}/)]
3. $.employees[?(#.employeeContact !=~/[0-9]{10}/)]
4. $.employees[?(#.employeeContact <>~/[^0-9]{10}/)]
But none of these have worked.
Example json which should NOT work:
{
"employee": {
"employeeContact": "1234567890"
}
}
while these employee should work (anything other than 10 digits):
1. "employeeContact": "1a34567890" // character in between
2. "employeeContact": "12345678901" // more than 10
3. "employeeContact": "123456789" // less than 10
4. "employeeContact": "123456 89" //space
You could use the logical or operator to match for lengths less than 10 and greater than 10.
"bodyPatterns": [
"or": [
{ "matchingJsonPath": "$.employees[?(#.employeeContact =~/[^0-9]{1,9}/)]" },
{ "matchingJsonPath": "$.employees[?(#.employeeContact =~/[^0-9]{11,}/)]" }
]
]
This is what worked for me:
"bodyPatterns": [{
"matchesJsonPath": "$.employees[?(#.employeeContact =~/[^0-9]{1,9}/ || $.employees[?(#.employeeContact =~/[^0-9]{11,}/)]"
}]
Watch that it is matchesJsonPath instead of matchingJsonPath.
Even with that "or" didnt work with my wiremock 2.28.1 so may well be a wiremock bug.
I am writing a pact test for a request that filters out the response data by a certain field, hense I would like to create a matcher that would match an array of objects with an exact match on that field.
I tried doing the following two aproaches:
body: eachLike({
color: 'red',
name: like('any'),
}),
body: eachLike({
color: extractPayload('red'),
name: like('any'),
}),
Bot both of them produce the same result:
"matchingRules": {
"$.body": {
"min": 1
},
"$.body[*].*": {
"match": "type"
},
"$.body[*].name": {
"match": "type"
}
}
It seems to me that having "$.body[*].*": {"match": "type"} in there negates the exact matching for the color field. Am I wrong in that assumption, or is there a correct approach that would resolve this issue?
Yes, the issue is that the type matching is cascading and is not being reset.
The equal matcher (V3 only) will reset the matching rule for this context.
It's available in the latest beta: https://github.com/pact-foundation/pact-js/tree/feat/v3.0.0#using-the-v3-matching-rules
To work it in v2, I would use the regex that matches a single string value here.
I have a lambda function that converts my logs to this format:
{
"events": [
{
"field1": "value",
"field2": "value",
"field3": "value"
}, (...)
]
}
When I query it on S3, I get in this format:
[
{
"events": [
{ (...) }
]
}
]
And I'm trying to run a custom classifier for it because the data I want is inside the objects kept by 'events' and not events itself.
So I started with the simplest path I could think that worked in my tests (https://jsonpath.curiousconcept.com/)
$.events[*]
And, sure, worked in the tests but when I run a crawler against the file, the table created includes only an events field with a struct inside it.
So I tried a bunch of other paths:
$[*].events
$[*].['events']
$[*].['events'].[*]
$.[*].events[*]
$.events[*].[*]
Some of these does not even make sense and absolutely every one of those got me an schema with an events field marked as array.
Can anyone point me to a better direction to handle this issue?
Its seems like Bing Spell Check API does not work as I excepted.
A lot of mistakes are ignored...
For example:
"lets go to the see and then to gfgdf." response: "flaggedTokens": []
"lets blhblh to the sea" response: "flaggedTokens": []
Where: "lets go to the see" response:
{
"flaggedTokens": [
{
...
"suggestions": [
{
"suggestion": "let's",
}
]
},
{
...
"suggestions": [
{
"suggestion": "sea",
}
]
}
],
"_type": "SpellCheck"
}
Can I do something to get more reliable results?
Thanks
It's worst on my side...
I always got the same result:
{
"_type": "SpellCheck",
"flaggedTokens": []
}
If somebody happens to kown how to prevent that I would like to know.
I guess that's maybe about migrating to Azure. Maybe thoses endpoints are not functionning correctly now.
I have been researching Firebase as an alternative to the recently deprecated Dropbox Datastore API. I read the articles about structuring data, but I’m still a little unclear.
I have a bunch of users:
users
- name
- email
...and each user has three database “tables”, aircraft, entries, and customFields.
aircraft
- name
- category
- make
entries
- flightDate
- departure
- destination
customFields
- name
- type
So would my Firebase data structure look something like this?
{
“users”: {
“bob”: {
“name”: …
“email”: …
},
“sally”: {
“name”: …
“email”: …
}
},
“aircraft”:{
???
},
“entries”:{
???
},
“customFields”:{
???
}
}
Thanks in advance.
Are you familiar with OOP? Each "table" is an object. Personally I would do something as follows. Since I don't understand what you're trying to achieve with the database and their objects, this may not be correct:
{
"user": {
"name": "bob",
"aircraft": {
"name": "name"
},
"entries": {
"flightdate": "27/05/2015"
}
}
}
Think in objects, not tables. Think parent and child.
But in your example, if each object (user, aircraft, entries etc.) was plurals, you can treat them as a "table", it would just be an array of objects:
{
"aircrafts":[
{
"id":1,
"name": "name"
},
{
"id":2,
"name": "name"
}
]
}
Edit: My first example was if each user had an aircraft, in retrospect it was silly, but my point still stands.