I am trying to return more than 1 million records from Kusto database in Kusto explorer but I am getting this error below
Query result set has exceeded the internal record count limit 500000 (E_QUERY_RESULT_SET_TOO_LARGE; see http://aka.ms/kustoquerylimits)
I think the limit is 5000000. Any ideas how can I achieve this? thanks
set notruncation;
It's strongly recommended that, in this case, some form of limitation
is still put in place.
set truncationmaxsize=YOUR_LIMIT;
set truncationmaxrecords=YOUR_LIMIT;
Reference : https://learn.microsoft.com/en-us/azure/data-explorer/kusto/concepts/querylimits#limit-on-result-set-size-result-truncation
For large data exports - please use server-export options described here:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/data-export/
Service-side export (push): The methods above are somewhat limited since the query results have to be streamed through a single network connection between the producer doing the query and the consumer writing its results. For scalable data export, Kusto provides a "push" export model in which the service running the query also writes its results in an optimized manner. This model is exposed through a set of .export control commands, supporting exporting query results to an external table, a SQL table, or an external Blob storage.
Additional option is to use Kusto Explorer "run query into csv" button, this will add the set notruncation; for you and will save the results directly to disk, so that you can easily open the results in other tools such as Excel.
Related
I get below error on running my kusto query(which i am still learning) in Azure Data Explorer. and the reason is result is more than 64 mb.. Now when I read the MS doc they mentioned we have something called Export which we can use to export the query result in a storage blob.. However I couldnt find any example as how we can do it? Did someone tried it? Can they provide a sample as how we can export to a storage account when my resulted set is more than 64 mb?
Here is what I have for now, I have data in an external table which I would like to query. so for example if my external table name is TestKubeLogs and I am quyering
external_table("TestKubeLogs")
| where category == 'xyz'
and then I get error as
The Kusto DataEngine has failed to execute a query: 'Query result set has exceeded the internal data size limit 67108864 (E_QUERY_RESULT_SET_TOO_LARGE).'
so now I am trying to export this data. How should I do it. I started writing this but how do I specify the category and table name.
.export
async compressed
to json (
h#"https://azdevstoreforlogs.blob.core.windows.net/exportinglogs;mykey==",
)
You are not in the right direction.
Regardless the source of your data, whether you're querying an external table or managed table, it does not make sense to pull large data volume to your client.
What would you do with millions of records presented on your screen?
This is something that is true to all data systems.
Please read the relevant documentation:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/concepts/resulttruncation
Can we insert data into kusto table using flow?
I tried to insert data into kusto table using .ingest inline command but it throws a bad request error shown below:
Bad request: Control commands (starting with a dot '.') cannot be served from the query endpoint unless they are .show control commands.\r\nPlease provide the following information when contacting the Kusto.
So can we insert data into kusto table using flow?
it is possible, you just need to choose the Run control command... action instead of the Run query ... option (as .ingest, like any other command that starts with a dot (.) is a control command, and not a query)
that said, using direct ingestion is not necessarily recommended for large scale - you can read more about why here: https://learn.microsoft.com/en-us/azure/kusto/management/data-ingestion/
Inline ingestion (push): A control command (.ingest inline) is sent to the engine, with the data to be ingested being a part of the command text itself. This method is primarily intended for ad-hoc testing purposes, and should not be used for production purposes.
#Yoni L
Thank you it worked but we need to choose Chart Type as Html Table
I have been using DBExpress connections to various databases (mostly MSSQL, Sybase SQL) with:
SQLConnection -> SQLDataSet -> DataSetProvider -> ClientDataSet.
I need to connect to the databases in a fashion that does NOT write changes back to the tables.
So, the DataSetProvider has ResolveToDataSet:=false, and the ClientDataSet has LogChanges:=false (for performance).
In use I connect the SQLConnection, open the ClientDataSet, and then close the SQLConnection.
I can then manipulate the ClientDataSet without fear of changing the underlying table.
I'm new to FireDAC (XE8), and I'm looking to establish the same sort of scenario - load data into memory from a SQL query, and safely manipulate this data in memory without accidentally updating the source table(s).
I'm currently using:
FDConnection -> FDQuery and a FDMemTable
The FDQuery has CachedUpdates := true and I perform:
FDQ.Open;
FDQ.FetchAll;
FDMemT.CloneCursor(FDQ,true,false);
FDQ.Close;
I think this is pretty much equivalent - I end up with the data in an FDMemTable such that editing the data will not be able to "write back" to tables.
One other issue - in the dbExpress scenario, I often add InternalCalc Fields to the ClientDataSet. It isn't clear to me that I can do that (and have persistent field names) if I'm performing a CloneCursor operation.
Is there a simpler way of ensuring the data never updates the database? Setting the FDQuery to read-only doesn't work - I often have to modify records (but do not wish to persist these changes).
TIA.
EdB
There is a much easier way. Use the FDMemTable's CopyDataSet method. This will copy both the data and the metadata. Changes to the FDMemTable will not be written to the underlying dataset, and internal calc fields (and calculated field) will be copied as well, though you'll have to wire up the OnCalcFields event handler.
FDMemTable1.CopyDataSet( FDQuery1, [coStructure, coRestart, coAppend]);
I am using MongoTemplate to execute my Mongo queries.
I wanted to know if count works with limit set?
Also why find query searches full collection (according to query) although limit is set?
For e.g. the query i wrote might result in having 10000 records, but i want only 100 records and for that i have set limit to 100 and then fired find query. But still query goes on to search full 10000 records.
dataQuery.limit(100);
List<logs> logResultsTemp = mongoTemplate1.find(dataQuery, logs.class);
Is their any limitations in using limit command?
Limit works fine (at least on spring data version 1.2.1 that I use). Perhaps it was a problem on your your version?
About count, there is a specific method to get your collection count, so you don't need to care about the amount of data that your system will fetch:
mongoTemplate.count(new Query(), MyCollection.class)
Btw, if you try this directly on your mongodb console: db.myCollection.limit(1).count() you will get the actual total of documents in your collection, not only one. An so it is for the mongoTemplate.count method, so:
mongoTemplate.count(new Query().limit(1), MyCollection.class)
will work the same way.
I need to be able to run a query such as
SELECT * FROM atable WHERE MyFunc(afield) = "some text"
I've written MyFunc in a VB module but the query results in "Undefined function 'MyFunc' in expression." when executed from .NET
From what I've read so far, functions in Access VB modules aren't available in .NET due to security concerns. There isn't much information on the subject but this avenue seems like a daed end.
The other possibility is through the CREATE PROCEDURE statement which also has precious little documentation: http://msdn.microsoft.com/en-us/library/bb177892%28v=office.12%29.aspx
The following code does work and creates a query in Access:
CREATE PROCEDURE test AS SELECT * FROM atable
However I need more than just a simple select statement - I need several lines of VB code.
While experimenting with the CREATE PROCEDURE statement, I executed the following code:
CREATE PROCEDURE test AS
Which produced the error "Invalid SQL statement; expected 'DELETE', 'INSERT', 'PROCEDURE', 'SELECT', or 'UPDATE'."
This seems to indicate that there's a SQL 'PROCEDURE' statement, so then I tried
CREATE PROCEDURE TEST AS PROCEDURE
Which resulted in "Syntax error in PROCEDURE clause."
I can't find any information on the SQL 'PROCEDURE' statement - maybe I'm just reading the error message incorrectly and there's no such beast. I've spent some time experimenting with the statement but I can't get any further.
In response to the suggestions to add a field to store the value, I'll expand on my requirements:
I have two scenarios where I need this functionality.
In the first scenario, I needed to enable the user to search on the soundex of a field and since there's no soundex SQL function in Access I added a field to store the soundex value for every field in every table where the user wants to be able to search for a record that "soundes like" an entered value. I update the soundex value whenever the parent field value changes. It's a fair bit of overhead but I considered it necessary in this instance.
For the second scenario, I want to normalize the spacing of a space-concatenation of field values and optionally strip out user-defined characters. I can come very close to acheiving the desired value with a combination of TRIM and REPLACE functions. The value would only differ if three or more spaces appeared between words in the value of one of the fields (an unlikely scenario). It's hard to justify the overhead of an extra field on every field in every table where this functionality is needed. Unless I get specific feedback from users about the issue of extra spaces, I'll stick with the TRIM & REPLACE value.
My application is database agnostic (or just not very religious... I support 7). I wrote a UDF for each of the other 6 databases that does the space normalization and character stripping much more efficiently than the built-in database functions. It really annoys me that I can write the UDF in Access as a VB macro and use that macro within Access but I can't use it from .NET.
I do need to be able to index on the value, so pulling the entire column(s) into .NET and then performing my calculation won't work.
I think you are running into the ceiling of what Access can do (and trying to go beyond). Access really doesn't have the power to do really complex TSQL statements like you are attempting. However, there are a couple ways to accomplish what you are looking for.
First, if the results of MyFunc don't change often, you could create a function in a module that loops through each record in atable and runs your MyFunc against it. You could either store that data in the table itself (in a new column) or you could build an in-memory dataset that you use for whatever purposes you want.
The second way of doing this is to do the manipulation in .NET since it seems you have the ability to do so. Do the SELECT statement and pull out the data you want from Access (without trying to run MyFunc against it). Then run whatever logic you want against the data and either use it from there or put it back into the Access database.
Why don't you want to create an additional field in your atable, which is atable.afieldX = MyFunc(atable.afield)? All what you need - to run UPDATE command once.
You should try to write a SQL Server function MyFunc. This way you will be able to run the same query in SQLserver and in Access.
A few usefull links for you so you can get started:
MSDN article about user defined functions: http://msdn.microsoft.com/en-us/magazine/cc164062.aspx
SQLServer user defined functions: http://www.sqlteam.com/article/intro-to-user-defined-functions-updated
SQLServer string functions: http://msdn.microsoft.com/en-us/library/ms181984.aspx
What version of JET (now called Ace) are you using?
I mean, it should come as no surprise that if you going to use some Access VBA code, then you need the VBA library and a copy of MS Access loaded and running.
However, in Access 2010, we now have table triggers and store procedures. These store procedures do NOT require VBA and in fact run at the engine level. I have a table trigger and soundex routine here that shows how this works:
http://www.kallal.ca/searchw/WebSoundex.htm
The above means if Access, or VB.net, or even FoxPro via odbc modifies a row, the table trigger code will fire and run and save the soundex value in a column for you. And this feature also works if you use the new web publishing feature in access 2010. So, while the above article is written from the point of view of using Access Web services (available in office 365 and SharePoint), the above soundex table trigger will also work in a stand a alone Access and JET (ACE) only application.