Wondering if its possible to ingest multiple CSV with different number/ type of fields to a single ADX table? (Refer to the csv sample below)
Any way for me to use the header of the csv as the fields?
CSV type sample:
Type A
+--------+-----+--------+
| Name | Age | Uni |
+--------+-----+--------+
| Hazriq | 27 | UNITEN |
+--------+-----+--------+
Type B
+------+------+-----+
| Name | Uni | Age |
+------+------+-----+
| John | UNIx | 31 |
+------+------+-----+
Type C
+------+------+--------------+-----+
| Name | Uni | Hometown | Age |
+------+------+--------------+-----+
| Jane | UNIt | Kuala Lumpur | 31 |
+------+------+--------------+-----+
Yes, you can create multiple CSV mappings and provide the applicable mapping for a given file. You cannot use the CSV headers. Here is an example:
.create table demo(name:string, age:int, uni:string, hometown:string)
//define the mappings
.create table demo ingestion csv mapping "typeA" '[{"Name":"name", "Ordinal":0}, {"Name":"age", "Ordinal":1}, {"Name":"uni", "Ordinal":2}]'
.create table demo ingestion csv mapping "typeB" '[{"Name":"name", "Ordinal":0}, {"Name":"uni", "Ordinal":1}, {"Name":"age", "Ordinal":2}]'
.create table demo ingestion csv mapping "typeC" '[{"Name":"name", "Ordinal":0}, {"Name":"uni", "Ordinal":1}, {"Name":"age", "Ordinal":3}, {"Name":"hometown", "Ordinal":2} ]'
//Ingest some test date
.ingest inline into table demo with (csvMappingReference="typeA", pushbypull=true) <| Hazriq,27,UNITEN
.ingest inline into table demo with (csvMappingReference="typeB", pushbypull=true) <| John,UNIx,31
.ingest inline into table demo with (csvMappingReference="typeC", pushbypull=true) <| Jane,UNIt,Kuala Lumpur,31
//test
demo
Related
I have a column 'Apples' in azure table that has this string: "Colour:red,Size:small".
Current situation:
|-----------------------|
| Apples |
|-----------------------|
| Colour:red,Size:small |
|-----------------------|
Desired Situation:
|----------------|
| Colour | Size |
|----------------|
| Red | small |
|----------------|
Please help
I'll answer the title as I noticed many people searched for a solution.
The key here is mv-expand operator (expands multi-value dynamic arrays or property bags into multiple records):
datatable (str:string)["aaa,bbb,ccc", "ddd,eee,fff"]
| project splitted=split(str, ',')
| mv-expand col1=splitted[0], col2=splitted[1], col3=splitted[2]
| project-away splitted
project-away operator allows us to select what columns from the input exclude from the output.
Result:
+--------------------+
| col1 | col2 | col3 |
+--------------------+
| aaa | bbb | ccc |
| ddd | eee | fff |
+--------------------+
This query gave me the desired results:
| parse Apples with "Colour:" AppColour ", Size:" AppSize
Remember to include all the different delimiters preceding each word you want to extract, e.g ", Size". Mind the space between.
This helped me then i used my intuition to customize the query according to my needs:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/parseoperator
I'm quite new to SQLite and I'm trying to use an EXCEPT statement in order to compare two tables with very similar data. The data comes from a CSV file I download daily, and within the file new rows are added and deleted, and old rows can have one or more columns change each day. I'm trying to find a way to select rows that have had a column's data change, when I am unable to predict which column's data will change.
Say for example I have:
TABLE contracts:
|ID|Description|Name|Contract Type|
|1 |Plumbing |Bob |Paper |
|2 |Cooking |Ryan|Paper |
|3 |Driving |Eric|Paper |
|4 |Dancing |Emma|Paper |
and:
TABLE updated_contracts:
|ID|Description|Name|Contract Type|
|1 |Hiking |Bob |Paper |
|2 |Cooking |Ryan|Paper |
|3 |Driving |Eric|Paper |
|4 |Dancing |Emma|Digital |
I'd like it to return:
|1 |Hiking |Bob |Paper |
|4 |Dancing |Emma|Digital |
because contract 1 has changed the description and contract 4 has changed the contract type.
Is it possible to do this in SQLite?
One way to do it is with a LEFT join of updated_contracts to contracts where the matching rows are filtered out:
select uc.*
from updated_contracts uc left join contracts c
using(id, Description, Name, `Contract Type`)
where c.id is null
EXCEPT can also be used like this:
select * from updated_contracts
except
select * from contracts
This will work only if the tables have the same number of columns and its advantage is that it compares null values in columns and returns true if they are both null.
See the demo.
Results:
| ID | Description | Name | Contract Type |
| --- | ----------- | ---- | ------------- |
| 1 | Hiking | Bob | Paper |
| 4 | Dancing | Emma | Digital |
My Psql database has a table which has jsonb as type for some columns, when i tried to upload these tables in amazon quicksight for some analysis purpose, am getting an error says unsupported datatype and the columns are getting skipped in amazom Quicksight.
Please help me to convert these into some supported type in amazon Quicksight.
Column | Type | Collation | Nullable | Default
---------------+-----------------------------+-----------+----------+-----------------------------------------------
id | bigint | | not null | nextval('solera_progresses_id_seq'::regclass)
milestones | jsonb | | |
reference_id | character varying | | |
response_code | integer | | |
activity | jsonb | | |
response | jsonb | | |
user_id | bigint | | |
You can use custom SQL to convert the data to the supported type before loading to Quicksight.
For instance, if your jsonb column contains objects like {"name": "John"}, you can create a column name in Quicksight using the query:
SELECT column_name->'name' AS name
FROM table_name
I have this sqlite3 table (simplified):
+--------+----------+-------+
| ROUTE | WPNumber | WPID |
+--------+----------+-------+
| A123 | 1 | WP001 |
| A123 | 2 | WP002 |
| A123 | 3 | WP003 |
| [...] | [...] | [...] |
| A123 | 20 | WP020 |
+--------+----------+-------+
Lets say I want to travel this route in the reverse direction (020 to 001).
How do I get all the WPID's in between? I know it is possible to build a query using BETWEEN and DESC, but then I'd have to build two seperate queries and have Python check when to use which query. Is it possible to have sqlite3 do the work, independent of the direction (reverse or not).
You can reverse the sorting order by reversing the number used in the ORDER BY clause.
Set the parameter ? to either 1 or -1:
SELECT WPID
FROM ThisTable
WHERE ROUTE = 'A123'
ORDER BY WPNumber * ?
If you would just use a similar query with DESC, the database would have a better opportunity to optimize the sorting with an index.
How do I format table output in sqlite? For example, if the default behavior for showing entries is hard to read when one of the columns has variable length text. If I do select a,b,c from table I get
a1|b1|c1
a2|b2|c2
while I want some nicer and easily readible formatting like..
++++++++++++++++
| a1 | b1 | c1 |
| a2 | b2 | c2 |
++++++++++++++++
you can use the dot.commands ".mode columns / .width"
sqlite3 test.db <<SQL
.mode columns
.width 3 10 10 4
select 'ABCDEFGHIJKLM','ABCDEFGHIJKLM','ABC','ABCDEFGHIJKLM';
SQL
will receive
ABC ABCDEFGHIJ ABC ABCD
but with no other formatings like borders etc, so very rudimentary
ant
For sqlite3 you just need a command
.mode table
and your tables should look like this:
+----------+-----------+
| per_id | per2_id |
+----------+-----------+
| 13449958 | 5135490 |
| 21085892 | 5135490 |
| 3291872 | 5135490 |
+----------+-----------+
In the case you want this to be permanent change, you need to write the same command .mode table in the file called .sqliterc (if it doesn't exist just make one) in your home directory.
And that should do it.