If we follow the scenario described in the docs titled 'Zero retention on source table' , i.e. we have set transactional update policy and treating source tables are only temporary landing points and thus setting softdelete as 0s:-
.alter-merge table <TableName> policy retention softdelete=0s
Now since the update policy in question is transactional in nature , lets say update policy execution (execution of stored function executed by the update policy) fails , will there be retry ? and how long will Kusto keep retrying? Until the time it attempts retry , where does the data reside? Because source tables are 0 retention , so it won't even exist in source table I believe.
Yes, the retry logic is described in the "transactional policy" section. When the ingestion fails due to transactional update policy, the Data Management cluster will simply send the ingestion command to the source table again based on the logic described in the docs, until the full ingestion command succeeded the data will not be in the source or target tables:
Related
The documentation says:
The update policy function schema and the target table schema must match in their column names, types, and order.
What is the behavior when the target table schema needs to be updated? Will it fail or will that cause the update policy to fail?
If none fails, will the source table data stop being processed into the target table?
if there is a mismatch between the schema of the target table and the output schema of the query in the update policy - the update policy will fail.
And, if the update policy is configured with IsTransactional set to true - the entire ingestion will fail.
In managed ingestion flows, e.g. ingestion from event hub, or any other kind of queued ingestion - the service will automatically retry the ingestion operation in this failure scenario.
This means that there's a 'grace period' during which you can perform the desired schema changes, after which these retries will succeed (assuming you've resolved all schema mismatches)
This, too, is mentioned in the documentation (partial snippet included below)
The PDmlink records are hard deleted in WIndchill from the backend tables.
Users have access to delete the objects created and hence i need to find a way to identify the records deleted.
Is there any table which gives this information in PDMlink database?
Regards
Maha
There is a table called AUDITRECORD in DB which has the details about all the events happened it includes Delete event. But the information will be available only if you have configured Audit Event Recording.
If you have configured then you can get the details by using below query.
select * from auditrecord where eventlabel='Delete';
Refer Windchill helpcenter for steps to configure audit event recording.
In case you have not done this configuration and still wants to fetch the data about deleted objects then
Apache Web server logs and looking at backup of database or current
database doing a negation query to match part object identifer in
apache logs against those still remaining in database. Lot more work,
but not impossible.
I am trying to debug a performance issue in an ASP.NET application using .NET 4.5, EF5 (with a 2nd level cache and lazy loaded navigation properties) and SQL Server 2014. We are experiencing a number of wait locks in the SQL server. When I look at the locking transactions, they contain a very quick UPDATE, and then a very large SELECT. The UPDATE is ostensibly a necessary one, but I am confused as to why the SELECT is being run in the same transaction (and why anything is being selected at all). The fundamental issue is that the table referenced in the UPDATE statement is locked for the duration of the SELECT statement.
We use repository pattern for getting data from the db, and DbContext.SaveChanges() for committing changes. I cannot figure out how it is possible that EF produces a transaction where there is both a write and a read, and I am not getting relevant results when I try to search Google.
We have a number of interfaces into the system, and a couple of console applications working on the database as well, but they all go through the same setup/versions of .NET and EF.
I figure that it must be through SaveChanges, since this is (AFAIK) the only time that things are written to the database.
Does anyone here have a hint as to how these locking transactions might be produced?
The fundamental issue is that the table referenced in the UPDATE
statement is locked for the duration of the SELECT statement.
The answer is in your question:
the SELECT is being run in the same transaction
X lock is always held until the end of the transaction, i.e. until it commits or rolls back. So if after your quick update there is a long select, all that update locked in your table remains locked until your select ends.
You can separate your update and select if your business rules permit, you can add an appropriate index on the updated table to lock only some rows and not the whole table, or you can optimize your select to execute faster.
we have a scenario where we need to populate the collection every one hour with the latest data whenever we receive the data file in blob from external sources and at the same time , we do not want to impact the live users while updating the collection.
So, we have done below
Created 2 databases and collection 1 in both databases
Created a another collection in different database( configuration database ) with property as Active and Passive and this will have the Database1 and Database2 as values for the above properties
Now , our web job will run every time it sees the file in blob and check this configuration database and identify which one is active or passive and process the xml file and update the collection in passive database as that is not used by the live feed and once it is done , will update the active database to current and passive to live
now , our service will always check which one is active and passive and fetch the data accordingly and show to user
As we have to delete the data and insert the newly data in web job , wanted to know is this is best design we have come up with ? Does deleting and inserting the data will cost ? Is there better way to do bulks delete and insert as we are doing sequentially now
wanted to know is this is best design we have come up with ?
As David Makogon said, as for your solution, you need to manage and pay for multiple databases. If possible, you could create new documents in same collection and control which document is active in your program logic.
Does deleting and inserting the data will cost ?
the operation/request will consume the request units, which will be charged. To know Request Units and DocumentDB Pricing details, please refer to:
What is a Request Unit
DocumentDB pricing details
Is there better way to do bulks delete and insert as we are doing sequentially now
Stored Procedure that provides a way to group operations like inserts and submit them in bulk. You could create the stored procedures and then execute the stored procedure in your Webjobs function.
I would like to know the Process Integration steps.
Through Outbound ports
If any of the event occurs at AX Dynamics, we just want to know that events in the form of XML(Process Integration).
Example: Sales Order Creation, Customer Creation, Purchase Order Creation..
Outbound ports are only useful for asynchronous communication.
See AX 2012 Export Data with Outbound ports for an example (using the file system).
The steps to initiate sending data is in the AIF_SendCustomer.
As this is no lightweight operation, you may consider logging the records which needs integration in a custom integration table, then doing the processing in batch.
This is done in the insert and/or update and maybe delete method.
Deletes requires you store the RecId field value in the external system to be used for delete requests. The following does not cover this.
For logged table make the following method:
void syncRecord()
{
XXXRecordLog log;
log.RefTableId = this.TableId;
log.RefRecId = this.RecId;
log.insert();
}
Then call this.syncRecord() in the insert and update methods.
In the query to the outbound service be sure to exists join your table and the log table. This way only changed records are exported.
Make a batch job to do the transfer using the AIF_SendCustomer as a template.
After a synchronous (AifSendMode::Sync) transfer of the records, delete the log records (or mark them transferred).
Finally call AIFoutboundProcessingService to flush the file:
new AIFoutboundProcessingService().run();
Try to keeps things simple. It might be simpler to do a comma file export of the changed records!