I have a bit of code that I'm using to pull items from a database one at a time. On each get, it updates the status of the row to indicate that it has been retrieved so as to not retrieve the same row multiple times.
The below code is run from a backend service serving a Web API. Sometimes when multiple requests come in, it will return the same row (tasks with the same ID).
I was under the impression that having a transaction around it would mean that an update from one of the runs would preclude the row from being returned by a second query.
Any help would be appreciated.
public async Task<TaskDetail> GetTask()
{
using (var db = new SqlConnection(""))
{
using (var tran = db.BeginTransaction())
{
try
{
var sql = $#"
SELECT TOP 1 * FROM
(SELECT TOP 150 t.*
FROM Task t
INNER JOIN TaskStatus ts ON t.Id = ts.TaskId AND ts.Status = #taskStatus) t
ORDER BY NEWID();";
var chosen = await db
.QuerySingleOrDefaultAsync<TaskDetail>(
sql,
param: new
{
taskStatus = TaskStatusEnum.Ready
},
transaction: tran
);
if (chosen == null)
{
throw new InvalidOperationException();
}
var expiry = await db.ExecuteAsync("UPDATE TaskStatus SET Status = #status WHERE TaskId = #taskId", new {status = TaskStatusEnum.Done, taskId = chosen.TaskId}, tran);
tran.Commit();
return chosen;
}
catch
{
tran.Rollback();
throw;
}
}
}
}
A few wrong configurations can be the reason;
1- Check the SQL Server isolation level, be careful about dirty reads.
2- Be sure that you handle the error in web API properly, because especially transaction error is not displayed properly.
3- And, please remove t-sql from code :)
Select query doesn't block rows data from other transactions, two select in different transaction will be performed at the same time. You can try to set a session id on the row and then select it.
EDIT: hope this help, when update is executed it should block the row for others transactions
public async Task<TaskDetail> GetTask()
{
using (var db = new SqlConnection(""))
{
using (var tran = db.BeginTransaction())
{
try
{
var mySessionId = Guid.NewGuid();
var sql = $#"
UPDATE TaskStatus SET Status = #status, SessionId = #mySessionId WHERE TaskId in
(SELECT TOP 1
t.Id
FROM Task t
INNER JOIN TaskStatus ts ON t.Id = ts.TaskId AND ts.Status = #taskStatus ORDER BY NEWID());";
await db
.QuerySingleOrDefaultAsync<TaskDetail>(
sql,
param: new
{
taskStatus = TaskStatusEnum.Ready
status = TaskStatusEnum.InProgress,
mySessionId = mySessionId
},
transaction: tran
);
var sql = $#"
SELECT TOP 150 t.*
FROM Task t
INNER JOIN TaskStatus ts ON t.Id = ts.TaskId AND ts.Status = #taskStatus
WHERE ts.SessionId = #mySessionId;";
var chosen = await db
.QuerySingleOrDefaultAsync<TaskDetail>(
sql,
param: new
{
taskStatus = TaskStatusEnum.InProgress,
mySessionId = mySessionId
},
transaction: tran
);
if (chosen == null)
{
throw new InvalidOperationException();
}
var expiry = await db.ExecuteAsync("UPDATE TaskStatus SET Status = #status WHERE TaskId = #taskId", new {status = TaskStatusEnum.Done, taskId = chosen.TaskId}, tran);
tran.Commit();
return chosen;
}
catch
{
tran.Rollback();
throw;
}
}
}
}
Related
I have a dynamo db table CustomerOrders with following fields
Primary partition key: CustomerId (Number)
Primary sort key: DepartmentId (Number)
Order (Serialized Json String)
I would like to do a query on multiple customers in one request without using Sort Key (DepartmentId). So I created a Global Secondary Index on CustomerId and would like to use that to query just using the CustomerId. I see documentation only related to BatchGetItemAsync for running batch queries. I don't see a way to set the IndexName on a BatchGetItemRequest. How can that be done?
Below is my code segment so far:
public async Task<List<CustomerOrder>> GetOrdersAsync(List<int> customerIds)
{
var orders = new List<CustomerOrder>();
var tableKeys = new List<Dictionary<string, AttributeValue>>();
foreach (var x in customerIds)
{
tableKeys.Add(new Dictionary<string, AttributeValue> { { "CustomerId", new AttributeValue { N = x.ToString() } } });
}
var dynamoTable = $"CustomerOrders";
var keysAndAttributes = new KeysAndAttributes
{
AttributesToGet = new List<string> { "CustomerId", "DepartmentId", "Order" },
Keys = tableKeys
};
var request = new BatchGetItemRequest
{
ReturnConsumedCapacity = ReturnConsumedCapacity.INDEXES, // Not sure what this does
RequestItems = new Dictionary<string, KeysAndAttributes> { { dynamoTable, keysAndAttributes } }
};
BatchGetItemResponse result;
do
{
result = await dynamoDbClient.BatchGetItemAsync(request); // Exception gets thrown from here
var responses = result.Responses;
foreach (var tableName in responses.Keys)
{
var tableItems = responses[tableName];
foreach (var item in tableItems)
{
orders.Add(new CustomerOrder
{
CustomerId = int.Parse(item["CustomerId"].N),
DepartmentId = int.Parse(item["DepartmentId"].N),
Order = JsonConvert.DeserializeObject<Order>(item["Order"].S)
});
}
}
// Set RequestItems to the result's UnprocessedKeys and reissue request
request.RequestItems = result.UnprocessedKeys;
} while (result.UnprocessedKeys.Count > 0);
return orders;
}
I am getting The provided key element does not match the schema error with the above code. Please help!
You can't "set the IndexName on a BatchGetItemRequest"
In fact, you can't GetItem() on a GSI/LSI either. GetItem() only works on the table.
And GetItem() always requires the full primary key.
With a partial key, you'd need to perform multiple Query(), one for each hash key.
The GSI isn't doing anything for you. Department as a sort key really isn't doing anything for you either since I assume customerId is unique.
A better structure might have been to have the table defined with only hash key for the primary key;
I have the following linq expression, I am trying to create a new record in to address table and Update customer table with the newly created AdressID, How can i create a new address get the new AddressID and update customer table
if (request == null)
throw new ArgumentNullException(nameof(request));
if (request.AddressToCreate == null)
throw new ArgumentNullException(nameof(request.AddressToCreate));
var address = Mapper.Map<Address>(request.AddressToCreate);
address.CreatedBy = request.AddressToCreate.CreatedBy;
address.CreatedDate = SystemClock.UtcNow;
address.UpdatedBy = request.AddressToCreate.UpdatedBy;
address.UpdatedDate = SystemClock.UtcNow;
await Context.AddAsync(address);
var rps = Context.Customers .Where(rc => rc.ID == request.ID).SingleOrDefault();
rps.AddressID = request.AddressToCreate.ID;
await Context.SaveChangesAsync();
But request.AddressToCreate.ID; returns 0, How can i modify my code to get the New AddressID
I would recommend this:
Firstly you may not have an Id property to your AddressToCreate.
Secondly AddAsync() may not call Context.SaveChangesAsync() so the entity is not commited to database and the id is 0.
Try it like this;
await Context.AddAsync(address);
await Context.SaveChangesAsync(); // this is optional only if you have modified AddAsync() and it doesnt call SaveChangesAsync inside
var rps = Context.Customers.FirstOrDefault(rc => rc.ID == request.ID);
rps.AddressID = address.ID;
await Context.UpdateAsync(rps);
await Context.SaveChangesAsync();
I have standard lines of code, to fetch data with pagination. It used to work until a month ago, and then stopped. On running ExecuteNextAsync() it stops execution, and displays this following information in Output window:
DocDBTrace Information: 0 : DocumentClient with id 2 disposed.
DocDBTrace Information: 0 : DocumentClient with id 1 disposed.
Not Working Code:
var query =
client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(databaseId, "TestCollection"), "select c.id from TestCollection c",
new FeedOptions
{
//MaxDegreeOfParallelism=-1,
MaxItemCount = maxItemCount,
PopulateQueryMetrics=true
//RequestContinuation = requestContinuation,
//EnableScanInQuery = true,
//EnableCrossPartitionQuery = true
});
var queryAll = query.AsDocumentQuery();
var results = new List<TDocument>();
while (queryAll.HasMoreResults)
{
try
{
var result = await queryAll.ExecuteNextAsync();
var queryMetrics = result.QueryMetrics;
if (result.ResponseContinuation != null) requestContinuation = result.ResponseContinuation;
//Do something here
}
catch (Exception ex)
{
}
}
For the same client object, Create/Update or fetching all items together is working. So JsonSerializer or DocumentClient object cannot be a problem.
Working code:
var query2 = client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(databaseId, collectionName), "select * from TestCollection c",
new FeedOptions { MaxItemCount = -1 });
//.Where(l => l.Id == qId);
var testData2= query2.ToList();
This has stopped our services and their development. Any help is appreciated.
Your query is wrong.
UriFactory.CreateDocumentCollectionUri(databaseId, "TestCollection") will already let the SDK know what to query.
If you simply change
select c.id from TestCollection c
to
select c.id from c
It will work. Currently it is failing to resolve the c alias because you also have TestCollection there.
The only reason your other queries that use * are working is because you aren't using the c. there.
I am preparing store procedure on cosmosdb by Javascript, however, it gets less documents than the real number of documents in collection.
The sproc is called by C#, C# pass a parameter "transmitterMMSI" which is also the partition key of this collection.
First, the following query is executed in sproc:
var query = 'SELECT COUNT(1) AS Num FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
The result is output in response, and the value is 5761, which is the same as the real number of documents in collection.
However, when I change the query to the following:
var query = 'SELECT * FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
The documents.length is output as 5574, which is smaller than the real number.
I have already changed the pageSize: -1, which should mean unlimited.
I did some search with google and stack overflow, it seems that continuation can be help. However, I tried some examples, and they don't work.
Anyone familiar with this can help?
The following list the scripts.
The sproc js script is here, which is also the file "DownSampling.js" used in the C# code:
function DownSampling(transmitterMMSI, interval) {
var context = getContext();
var collection = context.getCollection();
var response = context.getResponse();
var receiverTime;
var tempTime;
var groupKey;
var aggGroup = new Object();
var query = 'SELECT * FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
var accept = collection.queryDocuments(collection.getSelfLink(), query, { pageSize: -1},
function (err, documents, responseOptions) {
if (err) throw new Error("Error" + err.message);
// Find the smallest deviation comparting to IntervalTime in each group
for (i = 0; i < documents.length; i++) {
receiverTime = Date.parse(documents[i].ReceiverTime);
tempTime = receiverTime / 1000 + interval / 2;
documents[i].IntervalTime = (tempTime - tempTime % interval) * 1000;
documents[i].Deviation = Math.abs(receiverTime - documents[i].IntervalTime);
// Generate a group key for each group, combinated of TransmitterMMSI and IntervalTime
groupKey = documents[i].IntervalTime.toString();
if (typeof aggGroup[groupKey] === 'undefined' || aggGroup[groupKey] > documents[i].Deviation) {
aggGroup[groupKey] = documents[i].Deviation;
}
}
// Tag the downsampling
for (i = 0; i < documents.length; i++) {
groupKey = documents[i].IntervalTime;
if (aggGroup[groupKey] == documents[i].Deviation) {
documents[i].DownSamplingTag = 1;
} else {
documents[i].DownSamplingTag = 0;
}
// Remove the items that are not used
delete documents[i].IntervalTime;
delete documents[i].Deviation;
// Replace the document
var acceptDoc = collection.replaceDocument(documents[i]._self, documents[i], {},
function (errDoc, docReplaced) {
if (errDoc) {
throw new Error("Update documents error:" + errDoc.message);
}
});
if (!acceptDoc) {
throw "Update documents not accepted, abort ";
}
}
response.setBody(documents.length);
});
if (!accept) {
throw new Error("The stored procedure timed out.");
}
}
And the C# code is here:
private async Task DownSampling()
{
Database database = this.client.CreateDatabaseQuery().Where(db => db.Id == DatabaseId).ToArray().FirstOrDefault();
DocumentCollection collection = this.client.CreateDocumentCollectionQuery(database.SelfLink).Where(c => c.Id == AISTestCollectionId).ToArray().FirstOrDefault();
string scriptFileName = #"..\..\StoredProcedures\DownSampling.js";
string scriptId = Path.GetFileNameWithoutExtension(scriptFileName);
var sproc = new StoredProcedure
{
Id = scriptId,
Body = File.ReadAllText(scriptFileName)
};
await TryDeleteStoredProcedure(collection.SelfLink, sproc.Id);
sproc = await this.client.CreateStoredProcedureAsync(collection.SelfLink, sproc);
IQueryable<dynamic> query = this.client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(DatabaseId, AISTestCollectionId),
new SqlQuerySpec()
{
//QueryText = "SELECT a.TransmitterMMSI FROM " + AISTestCollectionId + " a",
QueryText = "SELECT a.TransmitterMMSI FROM " + AISTestCollectionId + " a WHERE a.TransmitterMMSI=\"219633000\"",
}, new FeedOptions { MaxItemCount = -1, EnableCrossPartitionQuery = true, MaxDegreeOfParallelism = -1, MaxBufferedItemCount = -1 });
List<dynamic> transmitterMMSIList = query.ToList(); //TODO: Remove duplicates
Console.WriteLine("TransmitterMMSI count: {0}", transmitterMMSIList.Count());
HashSet<string> exist = new HashSet<string>();
foreach (var item in transmitterMMSIList)
{
//int transmitterMMSI = Int32.Parse(item.TransmitterMMSI.ToString());
string transmitterMMSI = item.TransmitterMMSI.ToString();
if (exist.Contains(transmitterMMSI))
{
continue;
}
exist.Add(transmitterMMSI);
Console.WriteLine("TransmitterMMSI: {0} is being processed.", transmitterMMSI);
var response = await this.client.ExecuteStoredProcedureAsync<string>(sproc.SelfLink,
new RequestOptions { PartitionKey = new PartitionKey(transmitterMMSI) }, transmitterMMSI, 30);
string s = response.Response;
Console.WriteLine("TransmitterMMSI: {0} is processed completely.", transmitterMMSI);
}
}
private async Task TryDeleteStoredProcedure(string collectionSelfLink, string sprocId)
{
StoredProcedure sproc = this.client.CreateStoredProcedureQuery(collectionSelfLink).Where(s => s.Id == sprocId).AsEnumerable().FirstOrDefault();
if (sproc != null)
{
await client.DeleteStoredProcedureAsync(sproc.SelfLink);
}
}
I tried to comment the 2 loops in the JS codes, only the documents.length output, while the response number is still less. However, I changed the query to SELECT a.id, the documents.length is correct. Looks like it is the continuation issue.
The sproc is probably timing out. To use a continuation token in these circumstances, you will need to return it to your C# calling code then make another call to the sproc passing in your token. If you show us your sproc code we can help more.
You can use a continuation token to make repeated calls to queryDocuments() from within the sproc without additional roundtrips to the client. Keep in mind that if you do this too many times your sproc will eventually timeout, though. In your case, it sounds like you're already very close to getting all of the documents you're seeking so maybe you will be OK.
Here is an example of using a continuation token within a sproc to query multiple pages of data:
function getManyThings() {
var collection = getContext().getCollection();
var query = {
query: 'SELECT r.id, r.FieldOne, r.FieldTwo FROM root r WHERE r.FieldThree="sought"'
};
var continuationToken;
getThings(continuationToken);
function getThings(continuationToken) {
var requestOptions = {
continuation: continuationToken,
pageSize: 1000 // Adjust this to suit your needs
};
var isAccepted = collection.queryDocuments(collection.getSelfLink(), query, requestOptions, function (err, feed, responseOptions) {
if (err) {
throw err;
}
for (var i = 0, len = feed.length; i < len; i++) {
var thing = feed[i];
// Do your logic on thing...
}
if (responseOptions.continuation) {
getThings(responseOptions.continuation);
}
else {
var response = getContext().getResponse();
response.setBody("RESULTS OF YOUR LOGIC");
}
});
if (!isAccepted) {
var response = getContext().getResponse();
response.setBody("Server rejected query - please narrow search criteria");
}
}
}
I'd like to know how to run this query in Linq way.
UPDATE orders SET shipDate = '6/15/2012' WHERE orderId IN ('123123','4986948','23947439')
My Codes,
[HttpGet]
public void test()
{
EFOrdersRepository ordersRepository = new EFOrdersRepository();
var query = ordersRepository.Orders;
// How to run this query in LINQ
// Query : UPDATE orders SET shipDate = '6/15/2012' WHERE orderId IN ('123123','4986948','23947439')
}
EFOrdersRepository.cs
public class EFOrdersRepository
{
private EFMysqlContext context = new EFMysqlContext();
public IQueryable<Order> Orders
{
get { return context.orders; }
}
}
EFMysqlContext.cs
class EFMysqlContext : DbContext
{
public DbSet<Order> orders { get; set; }
}
Actually it's pretty easy check the following code
EFOrdersRepository db = new EFOrdersRepository();
int[] ids= new string[] { "123123", "4986948", "23947439"};
//this linq give's the orders with the numbers
List<Order> orders = db.Order().ToList()
.Where( x => ids.Contains(x.orderId.Contains));
foreach(var order in orders)
{
order.ShipDate = '06/15/2012';
db.Entry(usuario).State = EntityState.Modified;
}
db.SaveChanges();
Something like this should work (warning Pseudo code ahead!!)
EDIT I like using the Jorge's method of retrieving the orders better (using contains), but leaving this here as another alternative. The statements below the code sample still hold true however.
[HttpGet]
public void test()
{
EFOrdersRepository ordersRepository = new EFOrdersRepository();
var query = ordersRepository.Orders.Where(x=>x.orderId == '123123' ||
x.orderId == '4986948' || x.orderId = '23947439').ToList();
foreach(var order in query){
var localOrder = order;
order.ShipDate = '06/15/2012';
}
ordersRepository.SaveChanges();
}
Basically, LINQ does not do 'bulk updates' well. You either have to fetch and loop through your orders or write a stored procedure that can take an array of ids and bulk update them that way. If you are only doing a few at a time, the above will work ok. If you have tons of orders that need to be updated, the ORM probably will not be the best choice. I look forward to see if anyone else has a better approach.
Disclaimer: the var localOrder = order line is to ensure that there are no modified closure issues. Also, ReSharper and other tools may have a less verbose way of writing the above.
Note: You need to call SaveChanges from your DBContext at the end
Short answer:
var f = new[] { 123123, 4986948, 23947439 };
var matchingOrders = orders.Where(x => f.Contains(x.ID)).ToList();
matchingOrders.ForEach(x => x.ShipDate = newDate);
Complete test:
// new date value
var newDate = new DateTime(2012, 6, 15);
// id's
var f = new[] { 123123, 4986948, 23947439 };
// simpulating the orders from the db
var orders = Builder<Order2>.CreateListOfSize(10).Build().ToList();
orders.Add(new Order2 { ID = 123123 });
orders.Add(new Order2 { ID = 4986948 });
orders.Add(new Order2 { ID = 23947439 });
// selecting only the matching orders
var matchingOrders = orders.Where(x => f.Contains(x.ID)).ToList();
matchingOrders.ForEach(x => Console.WriteLine("ID: " + x.ID + " Date: " + x.ShipDate.ToShortDateString()));
// setting the new value to all the results
matchingOrders.ForEach(x => x.ShipDate = newDate);
matchingOrders.ForEach(x => Console.WriteLine("ID: " + x.ID + " Date: " + x.ShipDate.ToShortDateString()));
Output:
ID: 123123 Date: 1/1/0001
ID: 4986948 Date: 1/1/0001
ID: 23947439 Date: 1/1/0001
ID: 123123 Date: 6/15/2012
ID: 4986948 Date: 6/15/2012
ID: 23947439 Date: 6/15/2012
In ORMs, You have to fetch the record first make the change to the record then save it back. To do that, I will add an UpdateOrder method to my Repositary like this
public bool UpdateOrder(Order order)
{
int result=false;
int n=0;
context.Orders.Attach(order);
context.Entry(order).State=EntityState.Modified;
try
{
n=context.SaveChanges();
result=true;
}
catch (DbUpdateConcurrencyException ex)
{
ex.Entries.Single().Reload();
n= context.SaveChanges();
result= true;
}
catch (Exception ex2)
{
//log error or propogate to parent
}
return result;
}
And i will call it from my Action method like this
int orderId=123232;
var orders=ordersRepository.Orders.Where(x=> x.orderId.Contains(orderId)).ToList();
if(orders!=null)
{
foreach(var order in orders)
{
order.ShipDate=DateTime.Parse('12/12/2012);
var result= ordersRepository.UpdateOrder();
}
}
In this Approach, if you have to update many number of records, you are executing thatn many number of update statement to the database. In this purticular case, i would like to execute the Raw SQL statement with only one query using the Database.SqlQuery method
string yourQry="UPDATE orders SET shipDate = '6/15/2012'
WHERE orderId IN ('123123','4986948','23947439')";
var reslt=context.Database.SqlQuery<int>(yourQry);