I would like to add a column to an existing Materialized View in Kusto.
In this example I am using the table test2
.create-merge table test2 (myid:int, mydate:datetime, myval:int) with (folder = "test/z001uw6n")
.create materialized-view with ( lookback=time(5.00:00:00),docString='test') test_view2 on table test2
{ test2
| summarize take_any(1) by myid, mydate
}
Now I am trying to add a column myval:
.alter materialized-view with ( lookback=time(5.00:00:00),docString='test') test_view2 on table test2
{ test2
| summarize take_any(1) by myid, mydate, myval
}
However, I get an exception: Cannot create materialized view 'test_view2': .alter materialized-view does not support changing group-by expressions. Current: 'myid, mydate', New: 'myid, mydate, myval'.
According to the documentation
, I expected that this is possible. Do you know how to add a column ?
See limitations in the same link you referred to:
* Changes to the materialized view group by expressions aren't supported.
The documentation clearly states:
Changes to the materialized view group by expressions aren't supported.
Related
I know that in ADX I can't update an existing row because it's an append-only system.
I want to add a specific row with a condition:
if there is no other row inside the table with the same values on certain columns.
I came up with this logic but I think this can be done much simpler:
.set-or-append target_table <|
let exists_row_count = old_table | where field1 == value1 and field2 == value2 | count()
let appended_row = case(exists_row_count == 0, <the whole record>, <empty record or null>)
appended_row
*need to mention that I get the value1, value2, and when I'm using a logic app here and I can iterate on each and every new record that I want to insert into the table, and of course, that the record is in tabular form.
You can create a Materialized View on top of your original table, and dedup by the columns you choose.
I'm writing a function which takes in raw data table (contains multijson telemetry data) and reformat it to a multiple cols. I use .set MyTable <| myfunction|limit 0 to create my target table based off of the function and use update policy to alert my target table.
Here is the code :
.set-or-append MyTargetTable <|
myfunction
| limit 0
.alter table MyTargetTable policy update
#'[{ "IsEnabled": true, "Source": "raw", "Query": "myfunction()", "IsTransactional": false, "PropagateIngestionProperties": false}]'
But I'm getting ingestion failures: Here is the ingestion failure message :
Failed to invoke update policy. Target Table = 'MyTargetTable', Query = '
let raw = __table("raw", 'All', 'AllButRowStore')
| where extent_id() in (guid(659e3b3c-6859-426d-9c37-003623834455));
myfunction()': Query schema does not match table schema
I double check the query schema and target table; they are the same . I'm not sure what this error means.
Also, I ran count on both the raw and mytarget tables; there are relatively large discrepancies (400 rows for My target and 2000 rows in raw table).
Any advise will be appreciated.
Generally speaking - to find the root of the mismatch between schemas, you can run something along the following lines, and filter for differences:
myfunction
| getschema
| join kind=leftouter (
table('MyTargetTable')
| getschema
) on ColumnOrdinal, ColumnType
In addition - you should make sure the output schema of the function you use in your update policy is 'stable', i.e. isn't affected by the input data
The output schema of some query plugins such as pivot() and bag_unpack() depends on the input data, and therefore it isn't recommended to use those in update policies.
I have a need to be able to query Azure Data Explorer (ADX) tables dynamically, that is, using application-specific metadata that is also stored in ADX.
If this is even possible, the way to do it seems to be via the table() function. In other words, it feels like I should be able to simply write:
let table_name = <non-trivial ADX query that returns the name of a table as a string>;
table(table_name) | limit 10
But this query fails since I am trying to pass a variable to the table() function, and "a parameter, which is not scalar constant string can't be passed as parameter to table() function". The workaround provided doesn't really help, since all the possible table names are not known ahead of time.
Is there any way to do this all within ADX (i.e. without multiple queries from the client) or do I need to go back to the drawing board?
if you know the desired output schema, you could potentially achieve that using union (note that in this case, the result schema will be the union of all tables, and you'll need to explicitly project the columns you're interested in)
let TableA = view() { print col1 = "hello world"};
let TableB = view() { print col1 = "goodbye universe" };
let LabelTable = datatable(table_name:string, label:string, updated:datetime)
[
"TableA", "MyLabel", datetime(2019-10-08),
"TableB", "MyLabel", datetime(2019-10-02)
];
let GetLabeledTable = (l:string)
{
toscalar(
LabelTable
| where label == l
| order by updated desc
| limit 1
)
};
let table_name = GetLabeledTable('MyLabel');
union withsource = T *
| where T == table_name
| project col1
I am trying to copy all data from sqlite table to another one (reason being changing primary key and recommended way is to create new table, copy data, rename, etc).
Something along:
insert into new_table select * from old_table;
drop old_table;
alter table new_table rename to old_table;
Now problem is, that sometimes when I create new table, it's order of columns is not the same and data is completely messed up. Not sure why, probably cause some people have brand new table and some have old table which lived trough lot of upgrade scripts. Anyway my point is, I need to make sure every column is inserted into correct column. I can for sure just name all the columns manually, but I'd like this to work even if I add some columns in future and also for other tables. I can probably somehow query name of columns and then construct query in language that I am using above (kotlin), but I'd like to do this on db level if possible.
It's on android, but not very relevant to the question :)
Thanks!
Sadly I had add 1 extra query to get columns.
Other queries had all columns specifically mentioned instead of using star to ensure correct order. My kotlin way:
fun columns(db: SQLiteDatabase, tableName: String): List<String> {
val sql = "pragma table_info($tableName)"
return db.rawQuery(sql, arrayOf()).use { cursor ->
(1..cursor.count).map {
cursor.moveToNext()
cursor.getString(cursor.getColumnIndexOrThrow("name"))
}
}
}
fun copyFromTableToTable(db: SQLiteDatabase, originalTableName: String, backupTableName: String) {
val columns = columns(db, backupTableName).joinToString(separator = ",")
db.execSQL("""INSERT INTO $backupTableName
($columns)
SELECT $columns FROM $originalTableName
""".trimMargin())
//db.execSQL("DROP TABLE $originalTableName;")
//db.execSQL("ALTER TABLE $backupTableName RENAME TO $originalTableName;")
}
I'm sure I can check if a row exists by selecting it but I'm wondering if there's a slicker way that I'm just not aware of -- seems like a common enough task that there might be. This SQLite table looks something like this:
rowID QID ANID value
------ ------ ----- ------
0 axo 1 45
1 axo 2 12
If the combination of QID and ANID already exists, that value should be updated, if the combination of QID and ANID doesn't already exist then it should be inserted. While its simple enough to write:
SELECT * where QID = 'axo' and ANID = 3;
And check if the row exists then branch and either insert/update I can't help but look for a better way. Thanks in advance!
Beware: REPLACE doesn't really equal 'UPDATE OR INSERT'...REPLACE replaces the entire row. Therefore if you don't specify values for EVERY column, you'll replace the un-specified columns with NULL or the default values.
In a simple example such as above, it's likely fine, but if you get into the habit of using REPLACE as 'UPDATE OR INSERT' you'll nuke data when you forget to specify a value for every field...just a warning from experience.
The insert documentation details a REPLACE option
INSERT OR REPLACE INTO tabname (QID,ANID,value) VALUES ('axo',3,45)
The "INSERT OR REPLACE" can abbreviated to REPLACE.
Example in Perl
Revised the following example to utilize a composite primary key
use strict;
use DBI;
### Connect to the database via DBI
my $dbfile = "simple.db";
unlink $dbfile;
my $dbh = DBI->connect("dbi:SQLite:dbname=$dbfile");
### Create a table
$dbh->do("CREATE TABLE tblData (qid TEXT, anid INTEGER, value INTEGER, PRIMARY KEY(qid, anid))");
### Add some data
my $insert = $dbh->prepare("INSERT INTO tblData (qid,anid,value) VALUES (?,?,?)");
$insert->execute('axo', 1, 45);
$insert->execute('axo', 2, 12);
$insert->finish;
### Update data
my $insert_update = $dbh->prepare("REPLACE INTO tblData (qid,anid,value) VALUES (?,?,?)");
$insert_update->execute('axo', 2, 500);
$insert_update->execute('axo', 10, 500);
$insert_update->finish;
### Print out the data
my $select = $dbh->prepare("SELECT * FROM tblData ORDER BY 1,2");
$select->execute;
while (my #row = $select->fetchrow_array()) {
printf "Row: %s\n", join(" - ", #row);
}
$select->finish;
$dbh->disconnect;
exit 0;
Produces the following output, demonstrating the update of one row and the insert of another
Row: axo - 1 - 45
Row: axo - 2 - 500
Row: axo - 10 - 500
you can use the REPLACE command. Docs
In this particular situation it turns out that there's no easy solution - at least not that I've been able to find, despite the efforts of Mark to suggest one.
The solution I've ended up using might provide useful to someone else so I'm posting it here.
I've defined a multi-column primary key as QID + ANID and attempt to insert into the table. If that particular combination exists, it will produce an error code of 23000 which I can then use an an indicator that it should be an UPDATE instead. Here's the basic gist (in PHP):
try {
$DB->exec("INSERT INTO tblData (QID,ANID,value) VALUES ('xxx','x',1)");
}
catch(PDOException $e) {
if($e->getCode() == 23000) {
$DB->exec("UPDATE tblData SET value=value+1 WHERE ANID='x' AND QID='xxx'");
}
}
The actual code I've used is a bit more complex using prepare/placeholders and handling the error if the code isn't 23000, etc.