Separate apdex threshold in Application Insights Analytics - azure-application-insights

Using default Apdex workbook of Application Insights, I'm trying to separate apdex threshold by query name, because different queries have different satisfaction request duration.
I've added two parameters, each for particular query name.
I've extended analytics query by apdexThresholdByName column, which differs for each query name.
Then I've changed this line: extend UserExperience = case(AverageDuration <= apdexThresholdByName, 'Satisfied', AverageDuration <= 4 * apdexThreshhold, 'Tolerating', 'Frustrated') put there my apdexThresholdByName column name instead of static single parameter, which was before.
And this doesn't work.
Syntax Error: "Failed to resolve entity 'apdexThresholdByName'"
let apdexData = {Type}
| where timestamp {TimeRange}
| where name in ({Operations}) or '*' in ({Operations})
{OperationsFilter}
| extend success = columnifexists('success', true)
| extend Failure = iff('{Calculations}' == 'ConsiderFailures' and success == false, 1, 0)
| extend InterestingDimension = iff(isempty({SegmentBy})== true, 'Unknown', {SegmentBy})
| extend apdexThresholdByName = iff('POST Insurance/PostModel [actionName/elementid/widgetname/workspace]' == ["name"], {Insurance_PostModel_actionName_elementid_widgetname_workspace_threshold}, iff('POST Insurance/PostModel [actionName/elementid/widgetname/workspace]' == ["name"], {Insurance_PostModel_widgetname_workspace_threshold}, 0))
| where InterestingDimension in ({SegmentFilters}) or '*' in ({SegmentFilters})
| summarize AverageDuration = avg(duration), Failures = sum(Failure) by user_Id, InterestingDimension
| extend UserExperience = case(AverageDuration <= apdexThresholdByName, 'Satisfied', AverageDuration <= 4 * apdexThreshhold, 'Tolerating', 'Frustrated')
| extend UserExperience = case(Failures > 0, "Frustrated", UserExperience)
| summarize Satisfied = countif(UserExperience == 'Satisfied'), Tolerating = countif(UserExperience == 'Tolerating'), Frustrated = countif(UserExperience == 'Frustrated'), Total = count() by InterestingDimension
| project InterestingDimension, ["Satisfied Users"] = Satisfied, ["Tolerating Users"] = Tolerating, ["Frustrated Users"] = Frustrated, ["Apdex Score"] = round((Satisfied + (Tolerating / 2.0)) / Total, 2), Total
| extend Relevance = iff(["Apdex Score"] == 0, pow(Total, 1.6), Total / ["Apdex Score"])
| project-rename Users = Total
| order by {ShowSegmentsBy}
| project-away Users, Relevance;
apdexData
| extend ["Apdex Interpretation"] = case(["Apdex Score"] <= 0.5, '⛔ Unacceptable', ["Apdex Score"] <= 0.7, '⚠️ Poor', ["Apdex Score"] <= 0.85, '⚠️ Fair', ["Apdex Score"] <= 0.94, '✔️ Good', '✔️ Excellent')
| project Values = InterestingDimension, ["Apdex Score"], ["Apdex Interpretation"], ["Satisfied Users"], ["Tolerating Users"], ["Frustrated Users"]

The field apdexThresholdByName does not pass the summarize clause that follows it.
This summarize clause contains neither "name" or "apdexThresholdByName" in the by section:
| summarize AverageDuration = avg(duration), Failures = sum(Failure) by user_Id, InterestingDimension
so that segmentation does not propagate onward

Related

Is it possible to iterate over the row values of a column in KQL to feed each value through a function

I am applying the series_decompose_anomalies algorithm to time data coming from multiple meters. Currently, I am using the ADX dashboard feature to feed my meter identifier as a parameter into the algorithm and return my anomalies and scores as a table.
let dt = 3hr;
Table
| where meter_ID == dashboardParameter
| make-series num=avg(value) on timestamp from _startTime to _endTime step dt
| extend (anomalies,score,baseline) = series_decompose_anomalies( num, 3,-1, 'linefit')
| mv-expand timestamp, num, baseline, anomalies, score
| where anomalies ==1
| project dashboardParameter, todatetime(timestamp), toreal(num), toint(anomalies), toreal(score)
I would like to bulk process all my meters in one go and return a table with all anomalies found across them. Is it possible to feed an array as an iterable in KQL or something similar to allow my parameter to change multiple times in a single run?
Simply add by meter_ID to make-series
(and remove | where meter_ID == dashboardParameter)
| make-series num=avg(value) on timestamp from _startTime to _endTime step dt by meter_ID
P.S.
Anomaly can be positive (num > baseline => flag = 1) or negative (num < baseline => flag = -1)
Demo
let _step = 1h;
let _endTime = toscalar(TransformedServerMetrics | summarize max(Timestamp));
let _startTime = _endTime - 12h;
TransformedServerMetrics
| make-series num = avg(Value) on Timestamp from _startTime to _endTime step _step by SQLMetrics
| extend (flag, score, baseline) = series_decompose_anomalies(num , 3,-1, 'linefit')
| mv-expand Timestamp to typeof(datetime), num to typeof(real), flag to typeof(int), score to typeof(real), baseline to typeof(real)
| where flag != 0
SQLMetrics
num
Timestamp
flag
score
baseline
write_bytes
169559910.91717172
2022-06-14T15:00:30.2395884Z
-1
-3.4824039875238131
170205132.25708669
cpu_time_ms
17.369556143036036
2022-06-14T17:00:30.2395884Z
1
7.8874529842826
11.04372634506527
percent_complete
0.04595588235294118
2022-06-14T22:00:30.2395884Z
1
25.019464868749985
0.004552738927738928
blocking_session_id
-5
2022-06-14T22:00:30.2395884Z
-1
-25.019464868749971
-0.49533799533799527
pending_disk_io_count
0.0019675925925925924
2022-06-14T23:00:30.2395884Z
1
6.4686836384225685
0.00043773741690408352
Fiddle

Make the value return timespan(0), if rows not available or condition is not met in Kusto?

I have the following code:
Telemetry
| where DataMetadata["category"] == "Warning"
| summarize
Duration = sum(case(Name == "Event", totimespan(Value), totimespan(0))),
Text = min(case(Name == "Information", tostring(Value), "N/A")),
DeviceID = min(case(Name == "Ident", tostring(Value), "N/A"))
by Timestamp
| summarize TotalDuration = sum(Duration) by Text,DeviceID
| top 2 by TotalDuration
| summarize Duration = max(case(isnotnull(TotalDuration) or isnotempty(TotalDuration), strcat("Duration: ",format_timespan(TotalDuration, 'dd:hh:mm:ss'), "[sec] ",DeviceID," - ",Text), tostring(timespan(0))))
Checking the last hour of data, the condition DataMetadata["category"] == "Warning" is not met and in this case I want to display as a result 00:00:00:00 as shown in the summarize at the end of the code.
However, what I get as a result is the following:
What is the issue here and how can I solve it ?
I assume that you do want the top 2 records by TotalDuration, in case there are any.
let Telemetry = datatable(DataMetadata:dynamic, Name:string, Timestamp:datetime, Value:string)[];
Telemetry
| where DataMetadata["category"] == "Warning"
| summarize
Duration = sum(case(Name == "Event", totimespan(Value), totimespan(0))),
Text = min(case(Name == "Information", tostring(Value), "N/A")),
DeviceID = min(case(Name == "Ident", tostring(Value), "N/A"))
by Timestamp
| summarize TotalDuration = sum(Duration) by Text,DeviceID
| union (print TotalDuration = 0s, Text = "NA", DeviceID = "NA")
| top 2 by TotalDuration
| project Duration = strcat("Duration: ",format_timespan(TotalDuration, 'dd:hh:mm:ss'), "[sec] ",DeviceID," - ",Text)
Duration
Duration: 00:00:00:00[sec] NA - NA
Fiddle

Kusto for sliding window

I am new to Kusto Query language. Requirement is to alert when the continuous 15 minute value of machine status is 1.
I have two columns with column1:(timestamp in every second) and column2:machine status(values 1 and 0).How can I use a sliding window to find if the machine is 1 for continuous 15 minutes.
Currently I have used the bin function, but it does not seem to be the proper one.
summarize avg_value = avg(status) by customer, machine,bin(timestamp,15m)
What could be the better solution for this.
Thanks in advance
Here is another option using time series functions:
let dt = 1s;
let n_bins = tolong(15m/dt);
let coeffs = repeat(1, n_bins);
let T = view(M:string) {
range Timestamp from datetime(2022-01-11) to datetime(2022-01-11 01:00) step dt
| extend machine = M
| extend status = iif(rand()<0.002, 0, 1)
};
union T("A"), T("B")
| make-series status=any(status) on Timestamp step dt by machine
| extend rolling_status = series_fir(status, coeffs, false)
| extend alerts = series_equals(rolling_status, n_bins)
| project machine, Timestamp, alerts
| mv-expand Timestamp to typeof(datetime), alerts to typeof(bool)
| where alerts == 1
You can also do it using the scan operator.
thanks
Here is one way to do it, the example uses generated data, hopefully it fits in your scenario:
let view = range x from datetime(2022-01-10 13:00:10) to datetime(2022-01-10 13:10:10) step 1s
| extend status = iif(rand()<0.01, 0, 1)
| extend current_sum = row_cumsum(status)
| extend prior_sum = prev(current_sum, 15)
| extend should_alert = (current_sum-prior_sum != 15 and isnotempty(prior_sum))
If you have multiple machines you need to sort it first by machines and restart the row_cumsum operation:
let T = view(M:string) {
range Timestamp from datetime(2022-01-10 13:00:10) to datetime(2022-01-10 13:10:10) step 1s
| extend machine = M
| extend status = iif(rand()<0.01, 0, 1)
};
union T("A"), T("B")
| sort by machine asc, Timestamp asc
| extend current_sum = row_cumsum(status, machine != prev(machine))
| extend prior_sum = iif(machine == prev(machine, 15), prev(current_sum, 15), int(null))
| extend should_alert = (current_sum-prior_sum != 15 and isnotempty(prior_sum))

kusto query in log analytics workbook

helo iam new to kusto query can i get the query for yesterday's date in log analytics workbook ?
let PGBARI1_Extraction_st = ADFPipelineRun
| where parse_json(Parameters).Process_Unique_RunID startswith "PGBARI1"
| where End >= Start
| where Start >= todatetime (strcat (tostring(format_datetime(now(),'yyyy-MM-dd'))))
| summarize time1 = min(Start) by TenantId , process_name = 'PGBARI1';
let PGBARI1_Compaction_end = ADFPipelineRun
| where Parameters contains "PGBARI1"
| where _ResourceId endswith "df-pr-dna-eus2"
| where OperationName contains "orchestrator"
| where End >= Start
| where Start >= todatetime (strcat (tostring(format_datetime(now(),'yyyy-MM-dd'))))
| summarize time2 = max(End) by TenantId , process_name = 'PGBARI1';
let PGBARI1 = PGBARI1_Extraction_st | join PGBARI1_Compaction_end on $left.process_name==$right.process_name
| summarize Runtime_in_minutes = sum( toint(datetime_diff('minute',time2,time1))), process_name = 'PGBARI1';
let PGBARI1_1 = ADFPipelineRun
| where Parameters contains "PGBARI1"
| where End >= Start
| where Start >= todatetime (strcat (tostring(format_datetime(now(),'yyyy-MM-dd'))))
| where Status != 'Succeeded'
| summarize Failed = count(Status);
let PGBARI_Status = PGBARI1_1 | project Process_name = 'PGBARI1', Status = iff(Failed>0,'Failed','Succeeded');
PGBARI1 | union PGBARI_Status

Alert on error rate exceeding threshold using Azure Insights and/or Analytics

I'm sending customEvents to Azure Application Insights that look like this:
timestamp | name | customDimensions
----------------------------------------------------------------------------
2017-06-22T14:10:07.391Z | StatusChange | {"Status":"3000","Id":"49315"}
2017-06-22T14:10:14.699Z | StatusChange | {"Status":"3000","Id":"49315"}
2017-06-22T14:10:15.716Z | StatusChange | {"Status":"2000","Id":"49315"}
2017-06-22T14:10:21.164Z | StatusChange | {"Status":"1000","Id":"41986"}
2017-06-22T14:10:24.994Z | StatusChange | {"Status":"3000","Id":"41986"}
2017-06-22T14:10:25.604Z | StatusChange | {"Status":"2000","Id":"41986"}
2017-06-22T14:10:29.964Z | StatusChange | {"Status":"3000","Id":"54234"}
2017-06-22T14:10:35.192Z | StatusChange | {"Status":"2000","Id":"54234"}
2017-06-22T14:10:35.809Z | StatusChange | {"Status":"3000","Id":"54234"}
2017-06-22T14:10:39.22Z | StatusChange | {"Status":"1000","Id":"74458"}
Assuming that status 3000 is an error status, I'd like to get an alert when a certain percentage of Ids end up in the error status during the past hour.
As far as I know, Insights cannot do this by default, so I would like to try the approach described here to write an Analytics query that could trigger the alert. This is the best I've been able to come up with:
customEvents
| where timestamp > ago(1h)
| extend isError = iff(toint(customDimensions.Status) == 3000, 1, 0)
| summarize failures = sum(isError), successes = sum(1 - isError) by timestamp bin = 1h
| extend ratio = todouble(failures) / todouble(failures+successes)
| extend failure_Percent = ratio * 100
| project iff(failure_Percent < 50, "PASSED", "FAILED")
However, for my alert to work properly, the query should:
Return "PASSED" even if there are no events within the hour (another alert will take care of the absence of events)
Only take into account the final status of each Id within the hour.
As the request is written, if there are no events, the query returns neither "PASSED" nor "FAILED".
It also takes into account any records with Status == 3000, which means that the example above would return "FAILED" (5 out of 10 records have Status 3000), while in reality only 1 out of 4 Ids ended up in error state.
Can someone help me figure out the correct query?
(And optional secondary questions: Has anyone setup a similar alert using Insights? Is this a correct approach?)
As mentioned, since you're only querying on a singe hour your don't need to bin the timestamp, or use it as part of your aggregation at all.
To answer your questions:
The way to overcome no data at all would be to inject a synthetic row into your table which will translate to a success result if no other result is found
If you want your pass/fail criteria to be based on the final status for each ID, then you need to use argmax in your summarize - it will return the status corresponding to maximal timestamp.
So to wrap it all up:
customEvents
| where timestamp > ago(1h)
| extend isError = iff(toint(customDimensions.Status) == 3000, 1, 0)
| summarize argmax(timestamp, isError) by tostring(customDimensions.Id)
| summarize failures = sum(max_timestamp_isError), successes = sum(1 - max_timestamp_isError)
| extend ratio = todouble(failures) / todouble(failures+successes)
| extend failure_Percent = ratio * 100
| project Result = iff(failure_Percent < 50, "PASSED", "FAILED"), IsSynthetic = 0
| union (datatable(Result:string, IsSynthetic:long) ["PASSED", 1])
| top 1 by IsSynthetic asc
| project Result
Regarding the bonus question - you can setup alerting based on Analytics queries using Flow. See here for a related question/answer
I'm presuming that the query returns no rows if you have no data in the hour, because the timestamp bin = 1h (aka bin(timestamp,1h)) doesn't return any bins?
but if you're only querying the last hour, i don't think you need the bin on timestamp at all?
without having your data it's hard to repro exactly but... you could try something like (beware syntax errors):
customEvents
| where timestamp > ago(1h)
| extend isError = iff(toint(customDimensions.Status) == 3000, 1, 0)
| summarize totalCount = count(), failures = countif(isError == 1), successes = countif(isError ==0)
| extend ratio = iff(totalCount == 0, 0, todouble(failures) / todouble(failures+successes))
| extend failure_Percent = ratio * 100
| project iff(failure_Percent < 50, "PASSED", "FAILED")
hypothetically, getting rid of the hour binning should just give you back a single row here of
totalCount = 0, failures = 0, successes = 0, so the math for failure percent should give you back 0 failure ratio, which should get you "PASSED".
without being to try it i'm not sure if that works or still returns you no row if there's no data?
for your second question, you could use something like
let maxTimestamp = toscalar(customEvents where timestamp > ago(1h)
| summarize max(timestamp));
customEvents | where timestamp == maxTimestamp ...
// ... more query here
to get just the row(s) that have that have a timestamp of the last event in the hour?

Resources