ADX Kusto find most recent rows for multiple id tuples - azure-data-explorer

Given I have Devices with an ID and each Device has different telemetry aspects which are stored to an ADX table. Each of these aspects is identified by a TelemetryId. The DeviceId is unique, but the TelemetryId is not. This Device-Telemery tuples are stored as individual rows in ADX. So the table looks like:
|DeviceId|TelemetryId|Value|TimeStamp|
|--------|-----------|-----|----------------------------|
|1 | 1 | 10 | 2022-06-15 08:50:42.016496 |
|1 | 2 | 20 | 2022-06-15 08:50:41.072425 |
|1 | 3 | 30 | 2022-06-15 08:50:40.199198 |
|2 | 1 | 11 | 2022-06-15 08:50:39.366868 |
|2 | 2 | 22 | 2022-06-15 08:50:38.536320 |
|2 | 3 | 33 | 2022-06-15 08:50:37.602781 |
|2 | 4 | 44 | 2022-06-15 08:50:36.702367 |
|2 | 5 | 55 | 2022-06-15 08:50:35.677494 |
|1 | 1 | 12 | 2022-06-15 08:50:34.750020 |
|1 | 2 | 22 | 2022-06-15 08:50:33.752211 |
Now I need the information about the most recent data containing all Telementry data for a specific DeviceId. From the query consumer perspective, I know which DeviceId exist in the system and which TelemetryId are assigned to each DeviceId. So I can use this information to build the query.
So for example, if I would query for DeviceId == 1 I would expect this result:
|DeviceId|TelemetryId|Value|TimeStamp|
|--------|-----------|-----|----------------------------|
|1 | 1 | 10 | 2022-06-15 08:50:42.016496 |
|1 | 2 | 20 | 2022-06-15 08:50:41.072425 |
|1 | 3 | 30 | 2022-06-15 08:50:40.199198 |
But I’m not sure if this can be achieved with a single Kusto Query? (I know how to do it with multiple ADX queries (one query for each Device-Telemetry tuple) but I am wondering if it is possible to get it done with one Query).

arg_max()
For performance reasons, you should consider narrowing the dataset by TimeStamp before applying the aggregation, E.g. querying only the last week/day/hour, depends on your data ingestion behaviour.
datatable (DeviceId:int, TelemetryId:int, Value:int, TimeStamp:datetime)
[
1 ,1 ,10 ,"2022-06-15 08:50:42.016496"
,1 ,2 ,20 ,"2022-06-15 08:50:41.072425"
,1 ,3 ,30 ,"2022-06-15 08:50:40.199198"
,2 ,1 ,11 ,"2022-06-15 08:50:39.366868"
,2 ,2 ,22 ,"2022-06-15 08:50:38.536320"
,2 ,3 ,33 ,"2022-06-15 08:50:37.602781"
,2 ,4 ,44 ,"2022-06-15 08:50:36.702367"
,2 ,5 ,55 ,"2022-06-15 08:50:35.677494"
,1 ,1 ,12 ,"2022-06-15 08:50:34.750020"
,1 ,2 ,22 ,"2022-06-15 08:50:33.752211"
]
| summarize arg_max(TimeStamp, *) by DeviceId, TelemetryId
DeviceId
TelemetryId
TimeStamp
Value
2
5
2022-06-15T08:50:35.677494Z
55
2
4
2022-06-15T08:50:36.702367Z
44
2
3
2022-06-15T08:50:37.602781Z
33
2
2
2022-06-15T08:50:38.53632Z
22
2
1
2022-06-15T08:50:39.366868Z
11
1
3
2022-06-15T08:50:40.199198Z
30
1
2
2022-06-15T08:50:41.072425Z
20
1
1
2022-06-15T08:50:42.016496Z
10
Fiddle

Related

SQL LEFT JOIN not pulling intended data

I have the below tables:
CATEGORIES:
id | name | category_group | cate_type_id
1 | Entertainment | Entertainment | 1
2 | Electricity | Utilities | 8
3 | Water | Utilities | 8
4 | Rent | Living Exp | 6
5 | credit card | Finance | 5
BUDGET-ITEMS:
id | budget_id | cat_id | category_group| budget_yr | budget_01 | budget_02 | ... | budget_12
1 | 1 | 1 | Entertainment | 2022 | 500 | |
2 | 1 | 2 | Utilities | 2022 | 1500 | |
3 | 1 | 3 | Utilities | 2022 | | 250 |
I want to pull all items from Category table with mapping budget columns. Below is my JOIN.
SELECT c.id as base_id,c.name,c.category_type_id, c.category_group as base_group, b.*
FROM category c
LEFT JOIN budget_items b ON c.id = b.category_id
WHERE c.category_type_id NOT IN (5)
ORDER BY c.category_type_id, c.category_group ASC
I expect the below output:
id | budget_id | cat_id | catgroup | budget_yr | budget_01 | budget_02 | ... | budget_12
1 | 1 | 1 | Entertainment | 2022 | 500 | |
2 | 1 | 2 | Utilities | 2022 | 1500 | |
3 | 1 | 3 | Utilities | 2022 | | 250 |
4 | 1 | 4 | Living Exp | 2022 | | |
However, I get like below (truncated base* columns here for space):
id | budget_id | cat_id | catgroup | budget_yr | budget_01 | budget_02 | ... | budget_12
1 | 1 | 1 | Entertainment | 2022 | 500 | |
2 | 1 | 2 | Utilities | 2022 | | 1500 |
3 | 1 | 3 | Utilities | 2022 | | |
4 | 1 | 4 | Living Exp | 2022 | | |
My query looks OK, not sure where it is going wrong. Does anyone see the issue?
Thanks in advance for your kind help.
Edit:
I have truncated some columns for space here. the problem is the values are aligned to different budget columns. I get the columns correctly from the left table.
Edit:
Thanks to everyone who pitched in to help, I finally figured the issue was with my data. The query was actually working fine. This community is amazing.
Not sure what you're doing, but it seems to work as expected for me:
Schema (SQLite v3.30)
CREATE TABLE items (
`id` INTEGER,
`budget_id` INTEGER,
`cat_id` INTEGER,
`catgroup` VARCHAR(13),
`budget_yr` INTEGER,
`budget_01` INTEGER,
`budget_02` INTEGER
);
INSERT INTO items
(`id`, `budget_id`, `cat_id`, `catgroup`, `budget_yr`, `budget_01`, `budget_02`)
VALUES
('1', '1', '1', 'Entertainment', '2022', '500', null),
('2', '1', '2', 'Utilities', '2022', '1500', null),
('3', '1', '3', 'Utilities', '2022', null, '250');
CREATE TABLE cats (
`id` INTEGER,
`name` VARCHAR(13),
`category_group` VARCHAR(13),
`cate_type_id` INTEGER
);
INSERT INTO cats
(`id`, `name`, `category_group`, `cate_type_id`)
VALUES
('1', 'Entertainment', 'Entertainment', '1'),
('2', 'Electricity', 'Utilities', '8'),
('3', 'Water', 'Utilities', '8'),
('4', 'Rent', 'Living Exp', '6'),
('5', 'credit card', 'Finance', '5');
Query
SELECT c.id
, budget_id
, cat_id
, catgroup
, budget_yr
, budget_01
, budget_02
FROM cats c
LEFT JOIN items i ON c.id = i.cat_id
WHERE c.cate_type_id <> 5;
id
budget_id
cat_id
catgroup
budget_yr
budget_01
budget_02
1
1
1
Entertainment
2022
500
2
1
2
Utilities
2022
1500
3
1
3
Utilities
2022
250
4
View on DB Fiddle

Split column into different rows on SQLite recursively using delimiter ","

I have a SQLite table just like this:
the table name is 'surat'
But i want to make id_ayat to be split into different rows using SQLite query, and expected result just like this:
_id|id_surat|id_ayat
---+--------+-------
3 | 2 | 112
3 | 2 | 213
3 | 3 | 19
3 | 3 | 83
3 | 3 | 85
3 | 3 | 102
is that possible? what query that i can use in SQLite format?
With a recursive CTE:
with recursive cte as (
select _id, id_surat, id_ayat,
id_ayat + 0 col
from tablename
union all
select _id, id_surat, trim(substr(id_ayat, length(col) + 2)),
trim(substr(id_ayat, length(col) + 2)) + 0
from cte
where instr(id_ayat, ',')
)
select _id, id_surat, col id_ayat
from cte
order by _id, id_surat
See the demo.
Results:
| _id | id_surat | id_ayat |
| --- | -------- | ------- |
| 3 | 2 | 112 |
| 3 | 2 | 213 |
| 3 | 3 | 19 |
| 3 | 3 | 83 |
| 3 | 3 | 85 |
| 3 | 3 | 102 |

Select values from one table, count common values from other table, show 0 if no common values

I have two tables that are something as follows:
WORKDAYS
DATE | WORKDAY_LENGHT |
-----------+----------------+
12-05-2018 | 8 |
13-05-2018 | 6.5 |
14-05-2018 | 7.5 |
15-05-2018 | 8 |
ACCIDENTS
TOD | SEVERITY |
-----------------+-----------+
12-05-2018 12:00 | minor |
12-05-2018 15:00 | minor |
13-05-2018 08:00 | severe |
13-05-2018 12:00 | severe |
14-05-2018 10:30 | severe |
And I need a result that is as follows:
WORKDAYS
DATE | WORKDAY_LENGHT | ACCIDENTS_COUNT|
-----------+----------------+----------------+
12-05-2018 | 8 | 2 |
13-05-2018 | 6.5 | 2 |
14-05-2018 | 7.5 | 1 |
15-05-2018 | 8 | 0 |
What I so far have tried is this:
SELECT DISTINCT
w.date,
(
SELECT
COUNT(*)
FROM
accidents a
WHERE
date(w.date) = date(a.tod)
)
AS accidents_count
FROM
workdays w
Which gives me an answer that is somewhat in the right direction. Something like this:
WORKDAYS
DATE | WORKDAY_LENGHT | ACCIDENTS_COUNT|
-----------+----------------+----------------+
12-05-2018 | 8 | 1 |
12-05-2018 | 8 | 1 |
13-05-2018 | 6.5 | 1 |
13-05-2018 | 6.5 | 1 |
14-05-2018 | 7.5 | 1 |
15-05-2018 | 8 | 0 |
This is sqlite, so the date values are stored as strings. The date function therefore should make them just dates, right? Or is that the one causing problems?
I was missing a group by and feel ashamed for opening a question before figuring this out.
adding GROUP BY date(w.date) is the solution here.

Master-Detail show data SQL

I'm working with SQL Server and I have this 3 tables
STUDENTS
| id | student |
-------------
| 1 | Ronald |
| 2 | Jenny |
SCORES
| id | score | period | student |
| 1 | 8 | 1 | 1 |
| 2 | 9 | 2 | 1 |
PERIODS
| id | period |
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
And I want a query that returns this result:
| student | score1 | score2 | score3 | score4 |
| Ronald | 8 | 9 | null | null |
| Jenny | null | null | null | null |
As you can see, the number of scores depends of the periods because sometimes it can be 4 o 3 periods.
I don't know if I have the wrong idea or should I make this in the application, but I want some help.
You need to PIVOT your data e.g.
select Y.Student, [1], [2], [3], [4]
from (
select T.Student, P.[Period], S.Score
from Students T
cross join [Periods] P
left join Scores S on S.[Period] = P.id and S.Student = T.id
) X
pivot
(
sum(Score)
for [Period] in ([1],[2],[3],[4])
) Y
Reference: https://learn.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot?view=sql-server-20

how to reference a result in a subquery

I have the following table in an sqlite database
+----+-------------+-------+
| ID | Week Number | Count |
+----+-------------+-------+
| 1 | 1 | 31 |
| 2 | 2 | 16 |
| 3 | 3 | 73 |
| 4 | 4 | 59 |
| 5 | 5 | 44 |
| 6 | 6 | 73 |
+----+-------------+-------+
I want to get the following table out. Where I get this weeks sales as one column and then the next column will be last weeks sales.
+-------------+-----------+-----------+
| Week Number | This_Week | Last_Week |
+-------------+-----------+-----------+
| 1 | 31 | null |
| 2 | 16 | 31 |
| 3 | 73 | 16 |
| 4 | 59 | 73 |
| 5 | 44 | 59 |
| 6 | 73 | 44 |
+-------------+-----------+-----------+
This is the select statement i was going to use:
select
id, week_number, count,
(select count from tempTable
where week_number = (week_number-1))
from
tempTable;
You are comparing values in two different rows. When you are just writing week_number, the database does not know which one you mean.
To refer to a column in a specific table, you have to prefix it with the table name: tempTable.week_number.
And if both tables have the same name, you have to rename at least one of them:
SELECT id,
week_number,
count AS This_Week,
(SELECT count
FROM tempTable AS T2
WHERE T2.week_number = tempTable.week_number - 1
) AS Last_Week
FROM tempTable;
In case of you want to take a query upon a same table twice, you have to put aliases on the original one and its replicated one to differentiate them
select a.week_number,a.count this_week,
(select b.count from tempTable b
where b.week_number=(a.week_number-1)) last_week
from tempTable a;

Resources