I am currently trying to upload some data to a Kusto cluster using the C# API. What I am basically trying to perform looks currently like this:
using (IKustoIngestClient client = KustoIngestFactory.CreateStreamingIngestClient(kustoConnection))
{
var ingestionProperties = new KustoIngestionProperties()
{
DatabaseName = _kustoSettings.DatabaseName,
TableName = tableName,
Format = DataSourceFormat.json
};
// ms is a Memory Stream
client.IngestFromStreamAsync(ms, ingestionProperties);
}
The problem with the current code or tactics that I see in multiple samples is that they use Thread.Sleep and then check for the result. This is something that in my opinion is not really useful since the application has to know if the operation succeeded or not.
So is there a way for the above code to validate and properly throw an exception in the case of any failure. Cause now despite failing there are no exceptions being thrown.
Here is some recommended reading:
Kusto ingest client library - Best practices
Tracking Ingestion Status (KustoQueuedIngestClient)
The bottom line:
Async ingestion has some major advantages over Sync ingestion.
Tracking each ingestion operation for a heavy load might have significant implications on the ingestion latency and the cluster responsiveness.
Related
I have a CosmosDB collection with a number of different partitions. I want to delete all of the data in one of the partitions so I tried to run the command:
db.myCollection.deleteAll({PartitionKey: 'pop-9q'})
Where PartitionKey is the field that I partition/shard based on. But when I execute this it returns the not very helpful message:
ERROR: An Error has occurred
Why would I be getting this message and how can I either get more details on the cause or find a resolution?
Currently, at this time, you are unable to perform a bulk delete. Please Up Vote and Comment on this functionality: Add the ability to delete ALL data in a partition
Additionally, which API are you consuming? For Gremlin API you could execute something like the following: g.V().drop()
The Microsoft.Azure.Cosmos SDK has added this ability - currently only available as a preview feature (which requires you to opt-in via the portal)
See here for more details:
https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-delete-by-partition-key?tabs=dotnet-example
Sample code included there:
// Get reference to the container
var container = cosmosClient.GetContainer("DatabaseName", "ContainerName");
// Delete by logical partition key
ResponseMessage deleteResponse = await container.DeleteAllItemsByPartitionKeyStreamAsync(new PartitionKey("Contoso"));
if (deleteResponse.IsSuccessStatusCode) {
Console.WriteLine($"Delete all documents with partition key operation has successfully started");
}
As #Mike said, a "delete all data" feature is not supported yet in Cosmos db SQL API and Mongo API. I notice that you have already added comments in above link. I just provide you with a workaround here that using bulk delete stored procedure for Cosmos db SQL API.
(sample code: https://gist.github.com/deepumi/2a23c5380202bddf0b85e83baf5833be)
For Mongo API, unfortunately, even stored procedure is not supported. You could create an Azure HTTP Trigger Function to execute bulk delete code in the function whenever you want or merge it into your program code.
I have a mobile service sync table that is giving me absolutely HORRENDOUS performance.
The table is declared as:
IMobileServiceSyncTable<Myclass> myclassTable;
this.client = new MobileServiceClient("my url here");
var store = new MobileServiceSQLiteStore(“localdb.db”);
store.DefineTable<Myclass>();
this.client.SyncContext.InitializeAsync(store);
this.myclassTable = client.GetSyncTable<Myclass>();
Than later in a button handler I’m calling into:
this.myclassTable.ToCollectionAsync();
The problem is, the performance is horrific. It takes at best minutes and most times just sits there indefinitely.
Is there anything in the above that I’ve done that would explain why performance is so absolutely terrible?
this.myclassTable.ToCollectionAsync();
For IMobileServiceSyncTable table, the above method would execute the SELECT * FROM [Myclass] sql statement against your local sqlite db.
The problem is, the performance is horrific. It takes at best minutes and most times just sits there indefinitely.
AFAIK, when working with offline sync, we may invoke the pull operation for retrieving a subset of the server data, then insert the retrieved data into the local store table. For await this.myclassTable.PullAsync(), it would send request and retrieve the server data with the MaxPageSize in 50, and the client SDK would send another request to confirm whether there has more data and pull them automatically.
In summary, I would recommend you checking with your code to locate the specific code which causes this poor performance. Also, you could leverage adding diagnostic logging, capturing the network traces via Fiddler to troubleshoot with this issue.
I would like to know the Process Integration steps.
Through Outbound ports
If any of the event occurs at AX Dynamics, we just want to know that events in the form of XML(Process Integration).
Example: Sales Order Creation, Customer Creation, Purchase Order Creation..
Outbound ports are only useful for asynchronous communication.
See AX 2012 Export Data with Outbound ports for an example (using the file system).
The steps to initiate sending data is in the AIF_SendCustomer.
As this is no lightweight operation, you may consider logging the records which needs integration in a custom integration table, then doing the processing in batch.
This is done in the insert and/or update and maybe delete method.
Deletes requires you store the RecId field value in the external system to be used for delete requests. The following does not cover this.
For logged table make the following method:
void syncRecord()
{
XXXRecordLog log;
log.RefTableId = this.TableId;
log.RefRecId = this.RecId;
log.insert();
}
Then call this.syncRecord() in the insert and update methods.
In the query to the outbound service be sure to exists join your table and the log table. This way only changed records are exported.
Make a batch job to do the transfer using the AIF_SendCustomer as a template.
After a synchronous (AifSendMode::Sync) transfer of the records, delete the log records (or mark them transferred).
Finally call AIFoutboundProcessingService to flush the file:
new AIFoutboundProcessingService().run();
Try to keeps things simple. It might be simpler to do a comma file export of the changed records!
I'am working on a Symfony app that provides a rest web service (simple HTTP Request with JSON).
That service check some rules and inserts few lines in two MySQL table (write only).
For optimize reason, even if Doctrine bundle is available, i use native MySQL Query (with bind params) to insert this lines.
My need is : If for any reason, the database is not available, write "runnables" queries into a log file.
The final purpose is that when database is back, i want to be able to execute directly the file's content on the database.
Note that there is no unique constraint (pk is a generated uuid) and no lock or transaction to handle (simple insert statements).
I write a custom SQLLogger, but when $connection->insert(...) is called, the connect fail before logger is called.
So, my question is : There is a way to get the final query (with binded parameters) without database connection ?
Or should i rewrite the mecanism that bind params into query and log it myself when database is not available ?
Best regards,
Julien
As the final query with parameters is build by the database, there is just no way to build the query with PHP and to be garanteed that the query will be the same as the database.
The only way si to build query without binded parameters, but this is clearly not a good practice.
So, i finally decided to store all the JSON (API request body) in a file if the database is not available.
So when the database is back, instead of replay SQL queries, i can replay the original HTTP query.
Hope this late self-anwser will help someone.
Best regards.
I have an application that will reside within a business2business network that will communicate with our AS400 in our internal network environment. The firewall has been configured to allow the data request through to our AS400, but we are seeing a huge lag time in connection speed and response time. For example what takes less than a half second in our local development environments is taking upwards of 120 seconds in our B2B environment.
This is the function that we are utilizing to get our data. We are using the enterprise library application blocks, so the ASI object is the Database...
/// <summary>
/// Generic function to retrieve data table from AS400
/// </summary>
/// <param name="sql">SQL String</param>
/// <returns></returns>
private DataTable GetASIDataTable(string sql)
{
DataTable tbl = null;
HttpContext.Current.Trace.Warn("GetASIDataTable(" + sql + ") BEGIN");
using (var cmd = ASI.GetSqlStringCommand(sql))
{
using (var ds = ASI.ExecuteDataSet(cmd))
{
if (ds.Tables.Count > 0) tbl = ds.Tables[0];
}
}
HttpContext.Current.Trace.Warn("GetASIDataTable() END");
return tbl;
}
I am trying to brainstorm some ideas to consider as to why this is occurring.
Have never used ASP.NET or AS400 in anger, but I have seen this kind of behaviour before and it usually indicated some kind of network problem, typically a reverse DNS lookup that is timing out.
Assuming you have ping enabled through your firewall, check that you can ping in both directions.
Also run traceroute from each machine to try and diagnose where a delay might be.
Hope that helps.
Sorry but I can't tell you what is going on but I just have a couple comments...
First I would output the sql, see if it has a lot of joins and/or is hitting a table (file) with a large amount of records. If you really want to dig in fire up your profiler of choice (I use Ants Profiler) and try to find a profiler for the 400 - see what the server resources are as well as actual query after it goes thru the odbc driver.
I have worked with asp.net and as400 a few times and the way I have been most successful is actually using sql server with a linked server to AS400. I created a view to make it simpler to work with - hiding the oddities of as400 naming. It worked well in my scenario because the application needed to pull information from sql server anyway.
I thought I would mention it in case it helps... best of luck
Check the size of your iSeries system as well. Depending on the size of the query and if the system is undersized for the applications running on it, this may take time. While it shouldn't be thrown out as a posibility, I have seen a similar behavior in the past. But of course more likely is a network issue.
The other idea if you can solve the speed issue or is a sizing problem is to store it in an MS SQL Server then write the records from SQL Server to the iSeries from there.