I have a JSON property bag being received as an update to configuration. I would like to retrieve the existing latest property bag from the table, manipulate it to remove the keys that are being updated using the bag_remove_keys and bag_merge operators.
This is being used in an update policy, so I have the new property bag in an extend from the input data, I need to perform a new extend to retrieve the current latest property set existing in the table.
Something similar to the below:
rawhsimessages
//Get #RecType
| extend ParsedMessage = parse_json(Message)
| extend Objects = ParsedMessage["ExportedConfig"]["Objects"]
| parse Objects with "[" Objects "]"
| extend Properties = parse_json(Objects)
| extend RecType = Properties["#RecType"]
| where RecType == "CHANGE"
| extend latestconfig = XXXX(GeoSCADAConfigurationTest | where Id == Properties["Id"] | summarize arg_max(ConfigTime, Properties)
| project-away Message, ParsedMessage, Objects
Can I replace the XXXX with anything that will allow me to do this?
If not, is there a better approach I can take?
Related
I'm coding in python (PySide2), but the overall concept concerns the Qt Framework.
Here are two tables of mine:
Table "recordings":
| Column | Type |
| -------- | -------------------|
| id | int (primary key) |
| param1 | int |
| param2 | int |
| ... | int |
| paramN | int |
Table "analyzed_recs":
| Column | Type |
| -------- | -------------------|
| id | int (primary key) |
| rec_id | int (foreign key) | <-- Points to recordings.id
| paramN | int |
I need in my program to display param1 and param2 from the former. In Qt I used a QSqlRelationalTable to fulfill this objective:
def _init_db_models(self):
self.analyzed_recs_sql_model = QSqlTableModel(self, self.db)
self.analyzed_recs_sql_model.setTable("analyzed_recs")
rec_id = self.analyzed_recs_sql_model.fieldIndex('rec_id')
self.analyzed_recs_sql_model.setRelation(rec_id, QSqlRelation("recordings", "id", "param1"))
self.analyzed_recs_sql_model.setRelation(rec_id, QSqlRelation("recordings", "id", "param2"))
self.analyzed_recs_sql_model.select()
self.analyzed_data_table.setModel(self.analyzed_recs_sql_model)
This code works fine in displaying the desired fields.
However, when it comes to update a record in analyzed_recs:
record = self.analyzed_recs_sql_model.record()
record.remove(record.indexOf("id"))
record.setValue("rec_id", self.current_rec_id)
record.setValue("param1", self.param1)
record.setValue("param2", param2)
self.analyzed_recs_sql_model.insertRecord(-1, record)
self.analyzed_recs_sql_model.submitAll()
The column rec_id is not set (NULL) into the table (the other params are correctly inserted into the table).
On the contrary, if I avoid using QSqlRelationalTableModel and take QSqlTableModel instead, the insertion is performed correctly (as I expected), but I lose the INNER JOIN display feature.
I was thinking as a work around to create two distinct models, a QSqlRelationalTableModel only for displaying and a QSqlTableModel only for editing the data. However I don't like the extra workload of syncing the two.
I'm sure there is a Qt feature to achieve this, but unfortunately I'm missing it.
Any suggestion?
I've had the same problem using PYQT.
The record object returned by calling record() method has no fields named 'rec_id' because the QSqlRelationalTableModel changes it with the referenced field name 'param1'. We can verify the field names using:
fieldcount = record.count()
for i in range(fieldcount):
logging.info("field %s %s", i, record.fieldName(i))
so we need to add the field before assigning it:
record = self.analyzed_recs_sql_model.record()
record.remove(record.indexOf("id"))
record.append(QSqlField("rec_id"))
record.setValue("rec_id", self.current_rec_id)
I'm trying to display custom names for the Row data fetched using IN condition in Kusto.
Below is the table structure-
Below is the query I've used-
customEvents
| project Action=customDimensions["ActionInvoked"]
| where Action in (
"viewSelected",
"myProjectsSelected",
"watchlistSelected"
)
| summarize count() by tostring(Action)
| sort by count_
| render columnchart
The output to the query is as below-
As noticed in the output highlighted column names ("viewSelected","myProjectsSelected" & "watchlistSelected") are being rendered as is.
These are not User friendly and I'd like to change it.
NOTE: I'm just a day old to Kusto, so my query might be bad. Please feel free to change it to a better one if needed.
You'll need to replace your original values in the Action column by the names you want to be displayed. The easiest way to do it is to use the case function like this (look at the | extend Action = case(...) part):
customEvents
| project Action = tostring(customDimensions["ActionInvoked"])
| where Action in (
"viewSelected",
"myProjectsSelected",
"watchlistSelected"
)
| extend Action = case(
Action == "viewSelected", "Default Views",
Action == "myProjectsSelected", "Custom Views",
"Watchlist")
| summarize count() by Action
| sort by count_
| render columnchart
By the way, note that I moved the tostring higher up in the query, for convenience.
And here's how the new query works on your data:
datatable(Action:string) [
"viewSelected",
"myProjectsSelected",
"watchlistSelected",
"watchlistSelected",
"viewSelected"
]
| extend Action = case(
Action == "viewSelected", "Default Views",
Action == "myProjectsSelected", "Custom Views",
"Watchlist")
| summarize count() by Action
Output:
Action
count_
Default Views
2
Custom Views
1
Watchlist
2
pack_all() function considers all the input columns while making a dynamic object. Is it possible to somehow force it to consider only non-empty & non-null columns? If not, is there any workaround to apply filter on top of the resulting dynamic value?
There is no flavor of pack_all that will do it, but as an alternative, you can combine mv-apply and mv-expand operators to achieve this. Here is an example (adapted from the docs):
datatable(SourceNumber:string,TargetNumber:string,CharsCount:long)
[
'555-555-1234','555-555-1212',46,
'555-555-1234','555-555-1213',50,
'555-555-1212','',int(null)
]
| extend values =pack_all()
| mv-apply removeProperties = values on
(
mv-expand kind = array values
| where isempty(values[1])
| summarize propsToRemove = make_set(values[0])
)
| extend values = bag_remove_keys(values, propsToRemove)
| project-away propsToRemove
It should be added as a new answer, that pack_all() did in the meantime get a new option to exclude null/empty values
pack_all([ignore_null_empty])
ignore_null_empty: An optional bool indicating whether to
ignore null/empty columns and exclude them from the resulting property
bag. Default: false.
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/packallfunction
I'm pretty new to KQL and I'm having a difficult time with it (I don't have a background in stats, and I'm not very good at SQL either). I have telemetry data coming in from Microsoft AppCenter that I want to parse out into some charts but I'm trying to first figure out how to split a concatenated string that is essentially a dictionary that has two possible values: true and false. I want to count the number of each, so every key would have 2 values (true/false) which would also each have a numerical count value.
The input string I'm trying to get this data from is of the format Remove Splash/Main Menu Branding=True;Disable Aim Assist=False - unique items are split by ; and each pair is split by =. I am trying to figure out which options my users are using this way. The example string here would be split into:
Remove Splash/Main Menu Branding = True (count 1)
Disable Aim Assist = False (count 1).
If a new item came in that was Remove Splash/Main Menu Branding=True;Disable Aim Assist=True the summarized data would be
Remove Splash/Main Menu Branding = True (count 2)
Disable Aim Assist = False (count 1).
Disable Aim Assist = True (count 1).
So far I've got a query that selects a single item, but I don't know how to count this across multiple rows:
customEvents
| where timestamp > ago(7d)
| where name == "Installed a mod"
| extend Properties = todynamic(tostring(customDimensions.Properties))
| where isnotnull(Properties.["Alternate Options Selected"])
| extend OptionsStr = Properties.["Alternate Options Selected"] //The example string in above
| extend ModName = Properties.["Mod name"]
| where ModName startswith "SP Controller Support" //want to filter only to one mod's options
| extend optionsSplit = split(OptionsStr, ";")
| summarize any(optionsSplit)
I'm not sure how to make counts of it in a dictionary though. If anyone has any suggestions or tips or examples on something like this, I would really appreciate it, thanks.
Here you go:
let MyTable = datatable(Flags:string) [
"Remove Splash/Main Menu Branding=True;Disable Aim Assist=False",
"Remove Splash/Main Menu Branding=True;Disable Aim Assist=True"
];
MyTable
| extend Flags = split(Flags, ";")
| mv-expand Flag = Flags to typeof(string)
| summarize Count = count() by Flag
The output of this is:
| Flag | Count |
|---------------------------------------|-------|
| Remove Splash/Main Menu Branding=True | 2 |
| Disable Aim Assist=False | 1 |
| Disable Aim Assist=True | 1 |
Explanation:
First you split every input string (that contains multiple flags) into substrings, so that each will only have a single flag - you achieve this by using split.
Now your new Flags column has a list of strings (each one containing a single flag), and you want to create a record with every string, so you use the mv-expand operator
Lastly, you want to count how many times every key=value pair appears, and you do it with summarize count() by Flag
In case you want to see one record (in the output) per Key, then you can use the following query instead:
let MyTable = datatable(Flags:string) [
"Remove Splash/Main Menu Branding=True;Disable Aim Assist=False",
"Remove Splash/Main Menu Branding=True;Disable Aim Assist=True"
];
MyTable
| extend Flags = split(Flags, ";")
| mv-expand Flag = Flags to typeof(string)
| parse Flag with Key "=" Value
| project Key, Value
| evaluate pivot(Value, count(Value))
Its output is:
| Key | False | True |
|----------------------------------|-------|------|
| Remove Splash/Main Menu Branding | 0 | 2 |
| Disable Aim Assist | 1 | 1 |
You wrote that you're new to KQL, so you might find the following free Pluralsight courses interesting:
How to start with Microsoft Azure Data Explorer
Basic KQL
Azure Data Explorer – Advanced KQL
P.S. In the future please provide sample input in datatable format (if you're using Kusto Explorer, just select the relevant query results, right-click on the selection, and click Copy as datatable() literal), and also the expected output in a table format, so that it will be easier to understand what you want to achieve.
I am using Janusgraph 0.2.0 and have the following vertex defined (in Python):
class Airport(TypedVertex):
type = goblin.VertexProperty(goblin.String, card=Cardinality.single)
airport_code = goblin.VertexProperty(goblin.String,
card=Cardinality.single)
airport_city = goblin.VertexProperty(goblin.String,
card=Cardinality.single)
airport_name = goblin.VertexProperty(goblin.String,
card=Cardinality.single)
airport_region = goblin.VertexProperty(goblin.String,
card=Cardinality.single)
airport_runways = goblin.VertexProperty(goblin.Integer,
card=Cardinality.single)
airport_longest_runway = goblin.VertexProperty(goblin.Integer,
card=Cardinality.single)
airport_elev = goblin.VertexProperty(goblin.Integer,
card=Cardinality.single)
airport_country = goblin.VertexProperty(goblin.String,
card=Cardinality.single)
airport_lat = goblin.VertexProperty(goblin.Float,
card=Cardinality.single)
airport_long = goblin.VertexProperty(goblin.Float,
card=Cardinality.single)
I then defined an index for this node on the airport code field using the following commands (some commands were excluded to keep it short).
mgmt.makePropertyKey('type').dataType(String.class).cardinality(Cardinality.SINGLE).make()
mgmt.makePropertyKey('airport_city').dataType(String.class).cardinality(Cardinality.SINGLE).make()
mgmt.makePropertyKey('airport_code').dataType(String.class).cardinality(Cardinality.SINGLE).make()
mgmt.makePropertyKey('airport_country').dataType(String.class).cardinality(Cardinality.SINGLE).make()
airport_code = mgmt.getPropertyKey('airport_code')
airport_city = mgmt.getPropertyKey('airport_city')
airport_country = mgmt.getPropertyKey('airport_country')
mgmt.buildIndex('by_airport_code_unique', Vertex.class).addKey(airport_code).unique().buildCompositeIndex()
mgmt.buildIndex('by_airport_city', Vertex.class).addKey(airport_city).buildCompositeIndex()
mgmt.buildIndex('by_airport_country', Vertex.class).addKey(airport_country).buildCompositeIndex()
mgmt.awaitGraphIndexStatus(graph, 'by_airport_code_unique').call()
mgmt.awaitGraphIndexStatus(graph, 'by_airport_city').call()
mgmt.awaitGraphIndexStatus(graph, 'by_airport_country').call()
After the creating, I use a script to describe the :schema and I see that all the indexes are Registered:
| Graph Index . | Type . | Element | Unique | Backing | PropertyKey | Status |
|-----------------------:|:-----|:--------|:-------|:--------|:-----------|:--------|
| by_airport_code_unique | Composite | JanusGraphVertex | true | internalindex | airport_code | REGISTERED |
| by_airport_city | Composite | JanusGraphVertex | false | internalindex | airport_city | REGISTERED |
| by_airport_country | Composite | JanusGraphVertex | false | internalindex | airport_country | REGISTERED |
When I try to insert the second vertex with the same airport_code, as expected, I get an exception on constraint violation. However, if I go into the gremlin console and run a traversal to retrieve the vertices by their airport_code:
g.V().has('airport_code').values()
I get a warning: WARN org.janusgraph.graphdb.transaction.StandardJanusGraphTx - Query requires iterating over all vertices [()]. For better performance, use indexes
I had a similar problem a few weeks ago, and the issue was that I was trying to define indexes based on labels and I was told that at the time, janusgraph does not support indexes on labels. However, I don't think this is the case here.
Any suggestions or ideas on why my index is not working or not being used?
Thanks in advance for any help.
--MD
You are seeing the warning because your query does not utilize the index. A composite index is used for equality matches.
Composite indexes are very fast and efficient but limited to equality lookups for a particular, previously-defined combination of property keys. Mixed indexes can be used for lookups on any combination of indexed keys and support multiple condition predicates in addition to equality depending on the backing index store.
In order to leverage a composite index, you need to provide the property and a value to match. For example:
g.V().has('airport_code', 'JFK').toList()
I'm not sure why the index wasn't ENABLED after creation, perhaps something in the steps you left out. If you create the index within the same management transaction as the property keys, it should be ENABLED rather than REGISTERED. Check out the index lifecycle wiki.