KQL beginner here - I have some CEF logs hitting one of my servers and I need to get into the data to get some meaningful reports from it.
Take this log - not json, just a string
CEF:0|vendor1|vendorproduct|1.0|Event1|Event2|1|source_ip=0.0.0.0 rt=2020-04-28T04:17:05.475Z data1=example1 group=example2 endpoint=55555555 user=444444
I want to access each field and store as a var for further query use. What is the best way to achieve this so I can have results such as the below? Regex? String functions?
| extend vendorname = // = vendor1
| extend source_ip = // = 0.0.0.0
| extend endpoint = // = 55555555
// etc
OK, I figured this one out - see below for KQL to achieve what I was looking for:
Syslog
| where SyslogMessage has "vendor-name"
| extend logs = split(SyslogMessage, "|")
| extend vendor = logs[1]
| extend app = logs[2]
| extend version = logs[3]
| extend event = logs[4]
| extend msg = logs[5]
| parse SyslogMessage with * "source_ip=" source_ip "rt=" rt " id=" id " data1=" data1 " group=" group " endpoint=" endpoint "user=" user
| project vendor, app, version, event, msg, rt, data1, source_ip, id, group, endpoint, user
Related
I am trying to access function parameters within the 'case' statement in that function and displaying data/"filtered" based on the permission flag..Is it possible?
Usecase: TypeCast the value based on the columnType and check if the user has the permission to view the column based on which you display either the value or say something like "filtered"
Here is what I tried
function rls_columnCheck
.create-or-alter function rls_columnCheck(tableName:string, columnName: string, value:string, columnType:string, IsInGroupPII:bool, IsInGroupFinance:bool) {
let PIIColumns = rls_getTablePermissions(tableName, "PII");
let FinanceColumns = rls_getTablePermissions(tableName, "Finance");
let val= case(columnType=="bool", tobool(value),
columnType=="datetime", todatetime(value),
columnType=="int", toint(value),
value);
iif(columnName in (PIIColumns),
iif(columnName in (FinanceColumns),
iif(IsInGroupPII == true and IsInGroupFinance == true,
val,
"filtered"), // PII True, Fin True
iif(IsInGroupPII == true,
val,
"filtered") // PII True, Fin False
),
iif(columnName in (FinanceColumns),
iif(IsInGroupFinance == true,
val,
"filtered"), // PII False, Fin True
val // PII False, Fin False
)
);
}
Error:
Call to iff(): #then data type (int) must match the #else data type (string)
val in your function must have a single and well-defined data type, that is known at "compile" time of the query.
you can't have different cases, where in each it has a different type (bool, datetime, int, string - in your case statement) - hence the error.
if it makes sense in your use case, you can try to always have val typed as string.
This is not a good approach to use RLS because this will actually cause the engine to run a function for every column of every record. It has many downsides:
Performance of displaying the table’s contents (even if you have full permissions)
Queries on the table won’t benefit from the indexes Kusto stores (suppose you query PermissionTesting2 | where Col1 has “blablabla” - instead of checking the index for “blablabla”, the engine will have to scan all the data, because it has to apply a function for every single cell)
A better approach is to do something like this:
let UserCanSeePII = current_principal_is_member_of('aadgroup=group1#domain.com');
let UserCanSeeFinance = current_principal_is_member_of('aadgroup=group2#domain.com');
let ResultWithPII = YourTable | where UserCanSeePII and (not UserCanSeeFinance) | where ... | extend ...;
let ResultWithFinance = YourTable | where UserCanSeeFinance and (not UserCanSeePII) | where ... | extend ...;
let ResultWithPIIandFinance = YourTable | where UserCanSeeFinance and UserCanSeePII | where ... | extend ...;
let ResultWithoutPIIandFinance = YourTable | where (not UserCanSeePII) and (not UserCanSeeFinance) | where ... | extend ...;
union ResultWithPII, ResultWithFinance, ResultWithPIIandFinance, ResultWithoutPIIandFinance
Learning more about how to write a query in Kusto. I have a column in 2 tables that have different Roles, but the column header is Role, that I'd like to combine the data into one column called Roles.
I tried, adding this, | extend Roles = strcat (RoleName, Role), but that just combined the data.
Here is my query attempt, I'm joining 3 tables, 2 of which have the roles. The third is where I'm validating the user aliases.
(cluster('****').database('****').****_****** | where Discriminator == 'Service'| where DivisionOid == '******')
| join kind = leftouter cluster('****').database('****').Release_Users on SubscriptionId
| join kind = leftouter (cluster('****').database('****').Release_AzureAccess
| where RoleId contains "****" and PrincipalType !contains "ServicePrincipal") on SubscriptionId
| join kind = leftouter cluster('****').database('****').Headtrax_PeopleHierarchyV1 on $left.PrincipalName == $right.EmailAddress and $left.LiveEmailId == $right.EmailAddress
| extend Roles = strcat (RoleName, Role)<<--this was my failed attempt at combining the Role columns. That just concatenated.
I want to validate each user is active from 2 different tables against a person table. I'm a novice and am struggling with how to get this right. I'm thinking I want to combine the 2 tables into one list rather than trying to combine one column out of the 2 tables. Anyone have any advice?
This seems like a job for the union operator.
Union takes two or more tables and returns the rows of all of them.
From: https://learn.microsoft.com/en-us/azure/kusto/query/unionoperator?pivots=azuredataexplorer
Here's an example query from the docs above -
let View_1 = view () { print x=1 };
let View_2 = view () { print x=toint(2) };
let View_3 = view () { print x_long=3 };
union withsource=TableName View_1, View_2, View_3
produces:
The answer from Irwin will certainly work. If you want to take his solution and flatten it into one row, you could do that with the summarize function.
let View_1 = view () { print x=1 };
let View_2 = view () { print x=toint(2) };
let View_3 = view () { print x_long=3 };
union withsource=TableName View_1, View_2, View_3
| summarize sum(x_long1), sum(x_int), sum(x_long)
My use case is that I have an <iron-form> with a single <paper-textarea> field that accepts a string list of email addresses which I parse into an array, then I want to:
Store the individual email addresses in my Firebase (for indexing and lookup purposes),
at multiple locations (per data fan out technique),
with a single write operation (because I don't want to make 100 API calls if the list is that long) and
without overwriting any existing data.
Specifically, I want to start with State A, as follows:
State A
my-app
|
- emails
| |
| - email1#example,com
| |- old: "data"
| - email2#example,com
| |- old: "data"
- users
|
- c0djhbQi6vc4YMB-fDgJ
And achieve State B as follows:
State B
my-app
|
- emails
| |
| - email1#example,com
| |- old: "data"
| - email2#example,com
| |- old: "data"
| - email3#example,com
| |- new: "data"
| - email4#example,com
| |- new: "data"
- users
|
- c0djhbQi6vc4YMB-fDgJ
|
- emails
|
- email3#example,com
|- new: "data"
- email4#example,com
|- new: "data"
Notice: The {old: "data"} is not overwritten.
Background
I seek to extend this SO question and answer.
There, we inserted a single node in a new location with three options:
using firebase-query
JS SDK
using firebase-document
Now, I need to do the same type of insertion (without deletion or replacing old data) for multiple nodes (with a user defined, not autogenerated, key; i.e., keys are specific email addresses). I also need to use the data fan out technique to update multiple paths with a single write operation.
Similar to what's shown here.
https://firebase.google.com/docs/database/web/read-and-write#update_specific_fields
function writeNewPost(uid, username, picture, title, body) {
// A post entry.
var postData = {
author: username,
uid: uid,
body: body,
title: title,
starCount: 0,
authorPic: picture
};
// Get a key for a new Post.
var newPostKey = firebase.database().ref().child('posts').push().key;
// * * * * * * * * * * * * * * * *
// THE ABOVE LINE NEEDS TO CHANGE TO SUPPORT USER-GENERATED KEYS SUCH AS EMAIL ADDRESSES
// * * * * * * * * * * * * * * * *
// Write the new post's data simultaneously in the posts list and the user's post list.
var updates = {};
updates['/posts/' + newPostKey] = postData;
updates['/user-posts/' + uid + '/' + newPostKey] = postData;
return firebase.database().ref().update(updates);
}
Also note, one of the comments mentions:
There's no reason newPostKey above couldn't be an email address...
The challenge is that I need to write multiple keys to multiple locations simultaneously in a single call.
The Firebase Realtime Database supports arbitrarily complex atomic deep updates (blog post). It works like so:
You can update any arbitrarily deep path with a single .update() call
The full path on the key side of your update map will be replaced, so you must address subkeys directly if you don't want to blow away the parent
Paths are relative to your current ref
So let's take your example:
var update = {};
update['emails/email3#example,com'] = {new: 'data'};
update['emails/email4#example,com'] = {new: 'data'};
update['users/c0djhbQi6vc4YMB-fDgJ/emails/email3#example,com'] = {new: 'data'};
update['users/c0djhbQi6vc4YMB-fDgJ/emails/email4#example,com'] = {new: 'data'};
firebase.database().ref().update(update);
This will update all of the locations simultaneously. To make it dynamic, simply use string interpolation when constructing the keys.
Just beginning with firebase... :-(
How do I set a property of an item?
This is my data structure:
myproject
|
- players
|
- -JPUAYuKUNeevXxaCMxM
|
- name: "John"
|
- skill
|
- mu: 25
|
- sigma: 8.333
- -JPUAYuRyJBH8sF93pNt
...
I can add a player with:
var ref = new Firebase(FIREBASE_URL + '/' + 'players');
ref.child(id).set(player);
The question is: how do I update only one property of an item (for example, 'skill') ?
I did try with:
var skill = {};
skill.mu = 27.0;
skill.sigma = 7.0;
ref.child(id).skill.update(skill);
I think I know what's going on here.
You expected ref.child(id) to have a property skill. However, you actually want the "skill" child; ref.child(id).child("skill").
How can I get the date/time at which the item was last published.
I tried to create object for PublishInfoData inorder to use PublishedAt.
PublishInfoData pobj = csClient.Read(pageTCMID, readoptions) as PublishInfoData;
But this gives error like cannot convert IdentifiableObjectData to PublishInfoData.
Please suggest.
This will give you all publish info:
csClient.GetListPublishInfo(pageTCMID);
And then you have to select the latest:
var publishInfo = csClient.GetListPublishInfo(pageTCMID);
var lastPublishedAt = publishInfo.OrderByDescending(pi => pi.PublishedAt).First().PublishedAt;
I add this Powershell code for the record to get the last published date of a page
Set-TridionCoreServiceSettings sdswpap004 2013-SP1
$client = Get-TridionCoreServiceClient
Write-Output "Getting the title"
$pageTitle = Get-TridionItem -Id "tcm:30-7386-64" | Select-Object Title
Write-Output $pageTitle
$publishInfo = $client.GetListPublishInfo("tcm:30-7386-64")
if ($publishInfo) {
Write-Output "Getting the last pablished date: "
Write-Output $publishInfo.PublishedAt | Out-String
}