I am trying to define an array and loop through it looking up traces for where the message contains element in my array. Is it possible to do this? For example:
let myIds = datatable (name: string)
[
"111",
"222",
"333",
];
forach (id in myIds)
{
traces
| where message contains id
}
I know this isn't the right syntax above but hopefully it explains what I am trying to achieve. In a nutshell, loop through an array and perform a lookup in my logs (specifically traces).
The first option is to use has_any. This is a simpler solution that might work for your use case but only if your ID appears as a discrete term within the message.
So if the message is in the form "blah blah ID: 111" it will get picked up, but if it's part of another word then it won't (because has works a little differently from contains).
let myIds = datatable (name: string) [ "111","222","333"];
let traces=datatable(message:string) ["aaaaaaaaaaaaaaaa", "blah blah 111", "blah111 blah", "111blah"];
traces
| where message has_any (myIds)
If you need the functionality of contains (if you need to find every single instance of 111) then you can use mv-apply. This loops through your myIds subtable and does the comparison against each entry individually and then unions all the results. Be aware this means you can get duplicates if multiple IDs are matched in the same message.
let myIds = datatable (name: string) [ "111","222","333"] | summarize make_set(name);
let traces=datatable(message:string) ["aaaaaaaaaaaaaaaa", "blah blah 111", "blah111 blah", "111blah"];
traces
| mv-apply id=toscalar(myIds) to typeof(string) on (where message contains id)
you can look into using mv-expand or mv-apply
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/mvexpandoperator
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/mv-applyoperator
Related
Lets say I have this schema:
source_id -> String, HashKey
created_at -> String, RangeKey
required_capabilities -> StringSet
required_capabilities is a Set of Strings that we need to provide in the query in order to be able to retrieve a particular element.
For example:
If I have this three elements:
{
"source_id": "1",
"created_at": "2021-01-18T10:53:25Z",
"required_capabilities": ["Cap1", "Cap2", "Cap3"]
},
{
"source_id": "1",
"created_at": "2021-01-18T10:59:31Z",
"required_capabilities": ["Cap1", "Cap3"]
},
{
"source_id": "1",
"created_at": "2021-01-18T11:05:15Z"
}
I want to create a query, filtering for example source_id = "1" and providing a FilterExpression with the required_capabilities = ["Cap1", "Cap3", "Cap4"].
And I would expect as a result:
{
"source_id": "1",
"created_at": "2021-01-18T10:59:31Z",
"required_capabilities": ["Cap1", "Cap3"] // Since I've provided "Cap1", "Cap3" and "Cap4"
},
{
"source_id": "1",
"created_at": "2021-01-18T11:05:15Z" // Since it doesn't require any capability.
}
I've tried the IN operator as follows, since the stored StringSet should be IN (or Contained by) the given SS, but it didn't work.
aws dynamodb query --table-name TableName --key-condition-expression "source_id = :id" --filter-expression "required_capabilities IN (:rq)" --expression-attribute-values '{":id": {"S": "1"}, ":rq": { "SS": ["Cap1", "Cap3", "Cap4"] }}'
It works only when I provide the exact same StringSet, but If I provide a set that contains the saved one and also have more values, it doesn't return anything.
it seems your issue is around the use of the IN keyword, which does not work with sets. From the docs on conditionals
IN : Checks for matching elements in a list.
AttributeValueList can contain one or more AttributeValue elements of type String, Number, or Binary. These attributes are compared against an existing attribute of an item. If any elements of the input are equal to the item attribute, the expression evaluates to true.
I believe you want the CONTAINS keyword:
CONTAINS : Checks for a subsequence, or value in a set.
AttributeValueList can contain only one AttributeValue element of type String, Number, or Binary (not a set type). If the target attribute of the comparison is of type String, then the operator checks for a substring match. If the target attribute of the comparison is of type Binary, then the operator looks for a subsequence of the target that matches the input. If the target attribute of the comparison is a set ("SS", "NS", or "BS"), then the operator evaluates to true if it finds an exact match with any member of the set. CONTAINS is supported for lists: When evaluating "a CONTAINS b", "a" can be a list; however, "b" cannot be a set, a map, or a list.
Actually, I found out that dynamodb doesn't support the use case I needed, so I found a workaround.
Basically instead of modelling the required_capabilities as a StringSet, I've created a field called required_capability, containing a single required capability (which is ok so far for me) and using the IN operator to check.
If in the future I need to check for more than one capability, I just need to add new fields required_capability_2 and required_capability_3.
It's clearly not ideal, but I guess it's good enough, considering I won't have a lot of required capabilities in a single record, it's usually one, maybe two.
I'm new to F# so forgive me in advance if this is a stupid question or if the syntax may be a bit off. Hopefully it's possible to understand the gist of the question anyways.
What I'd like to achieve is the possibility to compose e.g. Result's (or an Either or something similar) having different error types (discriminated unions) without creating an explicit discriminated union that includes the union of the two other discriminated unions.
Let me present an example.
Let's say I have a type Person defined like this:
type Person =
{ Name: string
Email: string }
Imagine that you have a function that validates the name:
type NameValidationError =
| NameTooLong
| NameTooShort
let validateName person : Result<Person, NameValidationError>
and another that validates an email address:
type EmailValidationError =
| EmailTooLong
| EmailTooShort
let validateEmail person : Result<Person, EmailValidationError>
Now I want to compose validateName and validateEmail, but the problem is that the error type in the Result has different types. What I'd like to achieve is a function (or operator) that allows me to do something like this:
let validatedPerson = person |> validateName |>>> validateEmail
(|>>> is the "magic operator")
By using |>>> the error type of validatedPerson would be a union of NameValidationError and EmailValidationError:
Result<Person, NameValidationError | EmailValidationError>
Just to make it clear, it should be possible to an use arbitrary number of functions in the composition chain, i.e.:
let validatedPerson : Result<Person, NameValidationError | EmailValidationError | XValidationError | YValidationError> =
person |> validateName |>>> validateEmail |>>> validateX |>>> validateY
In languages like ReasonML you can use something called polymorphic variants but this is not available in F# as afaict.
Would it be possible to somehow mimic polymorphic variants using generics with union types (or any other technique)?! Or is this impossible?
There's some interesting proposals for erased type unions, allowing for Typescript-style anonymous union constraints.
type Goose = Goose of int
type Cardinal = Cardinal of int
type Mallard = Mallard of int
// a type abbreviation for an erased anonymous union
type Bird = (Goose | Cardinal | Mallard)
The magic operator which would give you a NameValidationError | EmailValidationError would have its type exist only at compile-time. It would be erased to object at runtime.
But it's still on the anvil, so maybe we can still have some readable code by doing the erasing ourselves?
The composition operator could 'erase' (box, really) the result error type:
let (|>>) input validate =
match input with
| Ok(v) -> validate v |> Result.mapError(box)
| Error(e) -> Error(box e)
and we can have a partial active pattern to make type-matching DU cases palatable.
let (|ValidationError|_|) kind = function
| Error(err) when Object.Equals(kind, err) -> Some ()
| _ -> None
Example (with super biased validations):
let person = { Name = "Bob"; Email = "bob#email.com "}
let validateName person = Result.Ok(person)
let validateEmail person = Result.Ok(person)
let validateVibe person = Result.Error(NameTooShort)
let result = person |> validateName |>> validateVibe |>> validateEmail
match result with
| ValidationError NameTooShort -> printfn "Why is your name too short"
| ValidationError EmailTooLong -> printfn "That was a long address"
| _ -> ()
This will shunt on validateVibe
This is probably more verbose than you would like but it does allow you to put things into a DU without explicitly defining it.
F# has Choice types which are defined like this:
type Choice<'T1,'T2> =
| Choice1Of2 of 'T1
| Choice2Of2 of 'T2
type Choice<'T1,'T2,'T3> =
| Choice1Of3 of 'T1
| Choice2Of3 of 'T2
| Choice3Of3 of 'T3
// Going up to ChoiceXOf7
With your existing functions you would use them like this:
// This function returns Result<Person,Choice<NameValidationError,EmailValidationError>>
let validatePerson person =
validateName person
|> Result.mapError Choice1Of2
|> Result.bind (validateEmail >> Result.mapError Choice2Of2)
This is how you would consume the result:
let displayValidationError person =
match person with
| Ok p -> None
| Error (Choice1Of2 NameTooLong) -> Some "Name too long"
| Error (Choice2Of2 EmailTooLong) -> Some "Email too long"
// etc.
If you want to add a third validation into validatePerson you'll need to switch to Choice<_,_,_> DU cases, e.g. Choice1Of3 and so on.
I am just getting started with eJabberd and am writing a custom module with HTTP access.
I have the request going through, but am now trying to retrieve a custom header and that's where I'm having problems.
I've used the Request record to get the request_headers list and can see that it contains all of the headers I need (although the one I'm after is a binary string on both the key and value for some reason...) as follows:
[
{ 'Content-Length', <<"100">> },
{ <<"X-Custom-Header">>, <<"CustomValue">> },
{ 'Host', <<"127.0.0.1:5280">> },
{ 'Content-Type', <<"application/json">> },
{ 'User-Agent', <<"Fiddler">> }
]
This is also my first foray into functional programming, so from procedural perspective, I would loop through the list and check if the key is the one that I'm looking for and return the value.
To this end, I've created a function as:
find_header(HeaderKey, Headers) ->
lists:foreach(
fun(H) ->
if
H = {HeaderKey, Value} -> H;
true -> false
end
end,
Headers).
With this I get the error:
illegal guard expression
I'm not even sure I'm going about this the right way so am looking for some advice as to how to handle this sort of scenario in Erlang (and possibly in functional languages in general).
Thanks in advance for any help and advice!
PhilHalf
The List that you have mentioned is called a "Property list", which is an ordinary list containing entries in the form of either tuples, whose first elements are keys used for lookup and insertion or atoms, which work as shorthand for tuples {Atom, true}.
To get a value of key, you may do the following:
proplists:get_value(Key,List).
for Example to get the Content Length:
7> List=[{'Content-Length',<<"100">>},
{<<"X-Custom-Header">>,<<"CustomValue">>},
{'Host',<<"127.0.0.1:5280">>},
{'Content-Type',<<"application/json">>},
{'User-Agent',<<"Fiddler">>}].
7> proplists:get_value('Content-Type',List).
<<"application/json">>
You can use the function lists:keyfind/3:
> {_, Value} = lists:keyfind('Content-Length', 1, Headers).
{'Content-Length',<<"100">>}
> Value.
<<"100">>
The 1 in the second argument tells the function what tuple element to compare. If, for example, you wanted to know what key corresponds to a value you already know, you'd use 2 instead:
> {Key, _} = lists:keyfind(<<"100">>, 2, Headers).
{'Content-Length',<<"100">>}
> Key.
'Content-Length'
As for how to implement this in Erlang, you'd write a recursive function.
Imagine that you're looking at the first element of the list, trying to figure out if this is the entry you're looking for. There are three possibilities:
The list is empty, so there is nothing to compare.
The first entry matches. Return it and ignore the rest of the list.
The first entry doesn't match. Therefore, the result of looking for this key in this list is the same as the result of looking for it in the remaining elements: we recurse.
find_header(_HeaderKey, []) ->
not_found;
find_header(HeaderKey, [{HeaderKey, Value} | _Rest]) ->
{ok, Value};
find_header(HeaderKey, [{_Key, _Value} | Rest]) ->
find_header(HeaderKey, Rest).
Hope this helps.
I need to serialize arbitrary records into maps/dictionary.
I imagine my end type look like this:
type TabularData= array<Map<string, obj>>
But I have problems in build a generic function that accept any record and turn them into Map.
In practice, the best advice is probably to use some existing serialization library like FsPickler. However, if you really want to write your own serialization for records, then GetRecordFields (as mentioned in the comments) is the way to go.
The following takes a record and creates a map from string field names to obj values of the fields. Note that it does not handle nested records and it is not particularly fast:
open Microsoft.FSharp.Reflection
let asMap (recd:'T) =
[ for p in FSharpType.GetRecordFields(typeof<'T>) ->
p.Name, p.GetValue(recd) ]
|> Map.ofSeq
Here is a little example that calls this with a simple record:
type Person =
{ Name : string
Age : int }
asMap { Name = "Tomas"; Age = -1 }
Using the same idea mentioned by Tomas, you can create a IDictionary<K,V> from a record like this:
let asDictionary (entity: 'T) =
seq {
for prop in FSharpType.GetRecordFields(typeof<'T>) ->
prop.Name, prop.GetValue(entity)
} |> dict
I'm looking to get all values associated with a key in a dictionary. Sometimes the key holds a single dictionary, sometimes a list of dictionaries.
a = {
'shelf':{
'book':{'title':'the catcher in the rye', 'author':'j d salinger'}
}
}
b = {
'shelf':[
{'book':{'title':'kafka on the shore', 'author':'haruki murakami'}},
{'book':{'title':'atomised', 'author':'michel houellebecq'}}
]
}
Here's my method to read the titles of every book on the shelf.
def print_books(d):
if(len(d['shelf']) == 1):
print d['shelf']['book']['title']
else:
for book in d['shelf']:
print book['book']['title']
It works, but doesn't look neat or pythonic. The for loop fails on the single value case, hence the if/else.
Can you improve on this?
Given your code will break if you have a list with a single item (and this is how I think it should be), if you really can't change your data structure this is a bit more robust and logic:
def print_books(d):
if isinstance(d['shelf'], dict):
print d['shelf']['book']['title']
else:
for book in d['shelf']:
print book['book']['title']
Why not always make 'shelf' map to a list of elements, but in the single element case it's a ... single element list? Then you'd always be able to treat each bookshelf the same.
def print_books(d):
container = d['shelf']
books = container if isinstance(container, list) else [container['book']]
books = [ e['book'] for e in books ]
for book in books:
print book['title']
I would first get the input consistent, then loop through all the books even if only one.
def print_books(d):
books = d['shelf'] if type(d['shelf']) is list else [ d['shelf'] ]
for book in books:
print book['book']['title']
I think this looks a little neater and pythonic, although some might argue not as efficient as your original code to create an array with one element and loop through it.