How to split a single string into multiple columns in Kusto? - azure-data-explorer

I am getting data as a single string. For example, my one record looks like - 2020/01/01 "Anna Thomas" 21.(Note: fields are space separated). I want them in a Kusto table with schema Date, Name, and Age. I read about first inserting them to a source table and then ingesting into Target Table using the update policy. But how will I parse this type of records (thought of using splitting the record based on single space but then the name also gets splitted)?
Date
Name
age
2020/01/01
Anna Thomas
21

You should use the parse operator:
let MyTable = datatable(str:string) [
'2020-01-01 "Anna Thomas" 21',
'2020-01-05 "Slavik Kusto" 32'
];
MyTable
| parse str with Timestamp:datetime ' "' Name:string '" ' Age:long
| project-away str
Output:
Timestamp
Name
Age
2020-01-01 00:00:00.0000000
Anna Thomas
21
2020-01-05 00:00:00.0000000
Slavik Kusto
32
Note: I changed the / to - in your timestamps, in order for the string to comply with the ISO 8601 datetime format that is required by Kusto. If you have no control over the input data, then you can do the replace as part of the Kusto query, like this:
let MyTable = datatable(str:string) [
'2020/01/01 "Anna Thomas" 21',
'2020/01/05 "Slavik Kusto" 32'
];
MyTable
| parse str with Timestamp:string ' "' Name:string '" ' Age:long
| extend Timestamp = todatetime(replace("/", "-", Timestamp))
| project-away str
Output:
Timestamp
Name
Age
2020-01-01 00:00:00.0000000
Anna Thomas
21
2020-01-05 00:00:00.0000000
Slavik Kusto
32

Related

misuse of aggregate: group_concat()

I have a problem with sqlite3 in this string
select group_concat(persons.name, ', ') as Актеры
from films_persons join persons on persons.id = films_persons.person_id
where films_persons.film_id = 1 and role = Актеры
films_persons looks like this:
film_id | person_id | role
1 | 1 | "Actors" (or "Актеры" in my case)
persons:
id | name | birth_date
1 | Leonardo | 11.11.1974
I need to group all persons.name in 1 string, and this call must return "Leonardo".
Full name of error is "sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) misuse of aggregate: group_concat()".
In mysql, you need to do:
group_concat(persons.name separator ', ')
instead of:
group_concat(persons.name, ', ')
https://dbfiddle.uk/?rdbms=mariadb_10.5&fiddle=20172b9b99dfcf7dcdf7ccf00efd49f4
But it appears from your error that you are using sqlite, not mysql, in which case your syntax is correct: https://dbfiddle.uk/?rdbms=sqlite_3.27&fiddle=691caef008dbade5c1dde83e4e495c7f

How can I fix wrong date format in SQLite

I'm working on app where I use SQLite to store data.
I created column Date. Since I'm beginner I made a mistake by inputing date as %m/%d/%Y (for example: 2/20/2020)
Now I've got a problem while taking out rows between selected dates.
I tried using this code:
SELECT * FROM Table WHERE Date BETWEEN strftime('%m/%d/%Y','2/5/2019') AND strftime('%m/%d/%Y','2/20/2020')
But that's not working.
Example table:
ID | Date
01 | 9/2/2019
02 | 2/20/2020
Thank you in advance for your help.
Update your dates to the only valid for SQLite date format which is YYYY-MM-DD:
update tablename
set date = substr(date, -4) || '-' ||
substr('00' || (date + 0), -2, 2) || '-' ||
substr('00' || (substr(date, instr(date, '/') + 1) + 0), -2, 2);
See the demo.
Results:
| ID | Date |
| --- | ---------- |
| 1 | 2019-09-02 |
| 2 | 2020-02-20 |
Now you can set the conditions like:
Date BETWEEN '2019-02-05' AND '2020-02-20'
If you do this change then you can use the function strftime() in select statements to return the dates in any format that you want:
SELECT strftime('%m/%d/%Y', date) date FROM Table
If you don't change the format of date column then every time you need to compare dates you will have to transform the value with the expression used in the UPDATE statement, and this is the worst choice that you could make.

hive time_stamp convert into UTC with time_offset in UTC

I have 2 columns: time_stamp and time_offset. Both are STRING data type.
How can we convert one column values into UTC with the help of second column which is in UTC? Is their any hive or from unix solution to convert time_stamp column into UTC?
hive> select time_stamp from table1 limit 2;
OK
20170717-22:31:57.348
20170719-21:10:15.393
[yyyymmdd-hh:mm:ss.msc] this column is in local time
hive> select time_offset from table1 limit 2;
OK
-05:00
+05:00
[‘+hh:mm’ or ‘-hh:mm’ ] this column is in UTC
You can use the Hive Date Functions unix_timestamp and from_unixtime to perform the conversion.
Code
WITH table1 AS (
SELECT '20170717-22:31:57.348' AS time_stamp, '-05:00' AS time_offset UNION ALL
SELECT '20170719-21:10:15.393' AS time_stamp, '+05:00' AS time_offset
)
SELECT
time_stamp,
time_offset,
unix_timestamp(concat(time_stamp, ' ', time_offset), 'yyyyMMdd-HH:mm:ss.SSS X') AS unix_timestamp_with_offset,
from_unixtime(unix_timestamp(concat(time_stamp, ' ', time_offset), 'yyyyMMdd-HH:mm:ss.SSS X'), 'yyyyMMdd-HH:mm:ss.SSS') AS string_timestamp_with_offset
FROM table1
;
Result Set
+------------------------+--------------+-----------------------------+-------------------------------+--+
| time_stamp | time_offset | unix_timestamp_with_offset | string_timestamp_with_offset |
+------------------------+--------------+-----------------------------+-------------------------------+--+
| 20170717-22:31:57.348 | -05:00 | 1500348717 | 20170717-20:31:57.000 |
| 20170719-21:10:15.393 | +05:00 | 1500480615 | 20170719-09:10:15.000 |
+------------------------+--------------+-----------------------------+-------------------------------+--+
Explanation
unix_timestamp can accept an optional format string in the same syntax as Java SimpleDateFormat. I am guessing that your offsets are using the ISO 8601 syntax, so let's use the X format specifier. Then, we can use the concat String Operator to combine time_stamp and time_offset before passing to unix_timestamp.
The unix_timestamp function results in a numeric timestamp specified as seconds since epoch. To convert that back to a string representation, we can pass the result obtained from unix_timestamp through from_unixtime, this time specifying our original format specifier.
(Please do test thoroughly to make sure the results are making sense in your environment. Time zone math can be tricky.)

how to concatenate multiple column values into a single column?

i have the following table friend
id | first_name | last_name | gender | age | mobile
1 | bobby | roe | male | 21 | 541-5780
how to concatenate multiple column (first_name & last_name) values into a single column to get the following result?
full_name
bobby roe
i have writen the following query but it does not work
declare #full_name varchar(max)
select #full_name = COALESCE(#full_name + ', ', '') + first_name, last_name
from friend
select #full_name
More than one way to achieve this:
SELECT CONCAT(first_name, ' ' ,last_name) AS full_name;
For earlier versions (Where CONCAT is not a built in function):
SELECT first_name + ISNULL(' ' + last_name, '') as Full_Name from [YourTable]
This as well should give you the same result
SELECT COALESCE(first_name, '') + COALESCE(last_name, '') as FullName FROM [YourTable]

Dynamic query based on second table

I have a table of price quotes for multiple symbols
Table QUOTES
ID INT
SYMBOL NVARCHAR(6)
DT DATETIME
PRICE DECIMAL(18,5)
Table TempSymbol
SYMBOL NVARCHAR(6)
I want to extract only those symbols from QUOTES whose symbols are also in a temp table that could vary based on user request
Create TABLE TempSymbol
(
SYMBOL NVARCHAR(6) NOT NULL
);
INSERT INTO TempSymbol(SYMBOL) VALUES ('MSFT');
INSERT INTO TempSymbol(SYMBOL) VALUES ('INTC');
INSERT INTO TempSymbol(SYMBOL) VALUES ('AAPL');
I want a query that will return from QUOTES the following data...
datetime symbol1 | price1 | symbol2 | price2 | symbol3 | price3
2012-11-12 12:10:00 MSFT | 12.10 | INTC | 5.68 | AAPL | 16.89
2012-11-12 12:15:00 MSFT | 12.22 | INTC | 5.97 | AAPL | 16.22
....
...
..
SELECT DT, SYMBOL, PRICE FROM QUOTE AS Q INNER JOIN TempSymbol AS TS ON Q.SYMBOL = TS.SYMBOL
This returns records that I need to pivot but that's not available in SQLite is there an another way I should be attempting this? Any help is appreciated.
try out this
SELECT DT, SYMBOL, PRICE FROM QUOTE where SYMBOL in (Select SYMBOL from TempSymbol)
SQL is doing the part of your problem that it's designed to do: retrieve the data. You can add ORDER BY DT to make the records for the same date-time adjacent.
If you think about it a minute you'll see that a SELECT can't possibly return what you want. It returns table rows, and SQL table rows have constant length. So doing what you call a "pivot" is not a SELECT operation. You may be thinking of pivots in spreadsheets. Databases aren't spreadsheets.
After that, producing the report you want is best done with a little program in any of the languages with an SQLite interface (in Android for example that's Java; otherwise C or TCL). Make the query. Get the rows back as hashes, arrays, or ODM records. The rest is a couple of loops over this data. The algorithm is:
last_dt = null
for row in all rows
if row.dt != last_dt
start new output line
print dt
last_dt = dt
end
print ' | ', row.symbol, ' | ', row.price
end
Another note: With advanced DB features like stored procedures and XML objects you could implement this in SQL. XML objects can have variable numbers of fields. Here the limit is SQLite, which doesn't provide these features.

Resources