I have standard lines of code, to fetch data with pagination. It used to work until a month ago, and then stopped. On running ExecuteNextAsync() it stops execution, and displays this following information in Output window:
DocDBTrace Information: 0 : DocumentClient with id 2 disposed.
DocDBTrace Information: 0 : DocumentClient with id 1 disposed.
Not Working Code:
var query =
client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(databaseId, "TestCollection"), "select c.id from TestCollection c",
new FeedOptions
{
//MaxDegreeOfParallelism=-1,
MaxItemCount = maxItemCount,
PopulateQueryMetrics=true
//RequestContinuation = requestContinuation,
//EnableScanInQuery = true,
//EnableCrossPartitionQuery = true
});
var queryAll = query.AsDocumentQuery();
var results = new List<TDocument>();
while (queryAll.HasMoreResults)
{
try
{
var result = await queryAll.ExecuteNextAsync();
var queryMetrics = result.QueryMetrics;
if (result.ResponseContinuation != null) requestContinuation = result.ResponseContinuation;
//Do something here
}
catch (Exception ex)
{
}
}
For the same client object, Create/Update or fetching all items together is working. So JsonSerializer or DocumentClient object cannot be a problem.
Working code:
var query2 = client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(databaseId, collectionName), "select * from TestCollection c",
new FeedOptions { MaxItemCount = -1 });
//.Where(l => l.Id == qId);
var testData2= query2.ToList();
This has stopped our services and their development. Any help is appreciated.
Your query is wrong.
UriFactory.CreateDocumentCollectionUri(databaseId, "TestCollection") will already let the SDK know what to query.
If you simply change
select c.id from TestCollection c
to
select c.id from c
It will work. Currently it is failing to resolve the c alias because you also have TestCollection there.
The only reason your other queries that use * are working is because you aren't using the c. there.
Related
When I query the status data from Azure Cosomos DB, I encountered a problem and can't find solution.
Code:
public static async Task<List<StatusData>> GenStatusData()
{
var sqlQueryText = $"Select * from (SELECT c.task,c.status, count(c.task) as countnum FROM c group by c.status, c.task ) as e order by e.task";
QueryDefinition queryDefinition = new QueryDefinition(sqlQueryText);
List<StatusData> models = new List<StatusData>();
//Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.OrderBy.OrderByQueryResult.get_Rid()
var queryResultSetIterator = container.GetItemQueryIterator<StatusData>(queryDefinition);
while (queryResultSetIterator.HasMoreResults)
{
FeedResponse<StatusData> currentResultSet = await queryResultSetIterator.ReadNextAsync();
//Console.WriteLine(currentResultSet.Count);
foreach (StatusData model in currentResultSet)
{
models.Add(model);
//Console.WriteLine("\tRead {0}\n", model);
}
}
return await Task.FromResult(models);
}
Issue Encountered:
An unhandled exception occurred while processing the request.
InvalidOperationException: Underlying object does not have an '_rid' field.
Microsoft.Azure.Cosmos.Query.Core.Pipeline.CrossPartition.OrderBy.OrderByQueryResult.get_Rid()
I have a bit of code that I'm using to pull items from a database one at a time. On each get, it updates the status of the row to indicate that it has been retrieved so as to not retrieve the same row multiple times.
The below code is run from a backend service serving a Web API. Sometimes when multiple requests come in, it will return the same row (tasks with the same ID).
I was under the impression that having a transaction around it would mean that an update from one of the runs would preclude the row from being returned by a second query.
Any help would be appreciated.
public async Task<TaskDetail> GetTask()
{
using (var db = new SqlConnection(""))
{
using (var tran = db.BeginTransaction())
{
try
{
var sql = $#"
SELECT TOP 1 * FROM
(SELECT TOP 150 t.*
FROM Task t
INNER JOIN TaskStatus ts ON t.Id = ts.TaskId AND ts.Status = #taskStatus) t
ORDER BY NEWID();";
var chosen = await db
.QuerySingleOrDefaultAsync<TaskDetail>(
sql,
param: new
{
taskStatus = TaskStatusEnum.Ready
},
transaction: tran
);
if (chosen == null)
{
throw new InvalidOperationException();
}
var expiry = await db.ExecuteAsync("UPDATE TaskStatus SET Status = #status WHERE TaskId = #taskId", new {status = TaskStatusEnum.Done, taskId = chosen.TaskId}, tran);
tran.Commit();
return chosen;
}
catch
{
tran.Rollback();
throw;
}
}
}
}
A few wrong configurations can be the reason;
1- Check the SQL Server isolation level, be careful about dirty reads.
2- Be sure that you handle the error in web API properly, because especially transaction error is not displayed properly.
3- And, please remove t-sql from code :)
Select query doesn't block rows data from other transactions, two select in different transaction will be performed at the same time. You can try to set a session id on the row and then select it.
EDIT: hope this help, when update is executed it should block the row for others transactions
public async Task<TaskDetail> GetTask()
{
using (var db = new SqlConnection(""))
{
using (var tran = db.BeginTransaction())
{
try
{
var mySessionId = Guid.NewGuid();
var sql = $#"
UPDATE TaskStatus SET Status = #status, SessionId = #mySessionId WHERE TaskId in
(SELECT TOP 1
t.Id
FROM Task t
INNER JOIN TaskStatus ts ON t.Id = ts.TaskId AND ts.Status = #taskStatus ORDER BY NEWID());";
await db
.QuerySingleOrDefaultAsync<TaskDetail>(
sql,
param: new
{
taskStatus = TaskStatusEnum.Ready
status = TaskStatusEnum.InProgress,
mySessionId = mySessionId
},
transaction: tran
);
var sql = $#"
SELECT TOP 150 t.*
FROM Task t
INNER JOIN TaskStatus ts ON t.Id = ts.TaskId AND ts.Status = #taskStatus
WHERE ts.SessionId = #mySessionId;";
var chosen = await db
.QuerySingleOrDefaultAsync<TaskDetail>(
sql,
param: new
{
taskStatus = TaskStatusEnum.InProgress,
mySessionId = mySessionId
},
transaction: tran
);
if (chosen == null)
{
throw new InvalidOperationException();
}
var expiry = await db.ExecuteAsync("UPDATE TaskStatus SET Status = #status WHERE TaskId = #taskId", new {status = TaskStatusEnum.Done, taskId = chosen.TaskId}, tran);
tran.Commit();
return chosen;
}
catch
{
tran.Rollback();
throw;
}
}
}
}
I am preparing store procedure on cosmosdb by Javascript, however, it gets less documents than the real number of documents in collection.
The sproc is called by C#, C# pass a parameter "transmitterMMSI" which is also the partition key of this collection.
First, the following query is executed in sproc:
var query = 'SELECT COUNT(1) AS Num FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
The result is output in response, and the value is 5761, which is the same as the real number of documents in collection.
However, when I change the query to the following:
var query = 'SELECT * FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
The documents.length is output as 5574, which is smaller than the real number.
I have already changed the pageSize: -1, which should mean unlimited.
I did some search with google and stack overflow, it seems that continuation can be help. However, I tried some examples, and they don't work.
Anyone familiar with this can help?
The following list the scripts.
The sproc js script is here, which is also the file "DownSampling.js" used in the C# code:
function DownSampling(transmitterMMSI, interval) {
var context = getContext();
var collection = context.getCollection();
var response = context.getResponse();
var receiverTime;
var tempTime;
var groupKey;
var aggGroup = new Object();
var query = 'SELECT * FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
var accept = collection.queryDocuments(collection.getSelfLink(), query, { pageSize: -1},
function (err, documents, responseOptions) {
if (err) throw new Error("Error" + err.message);
// Find the smallest deviation comparting to IntervalTime in each group
for (i = 0; i < documents.length; i++) {
receiverTime = Date.parse(documents[i].ReceiverTime);
tempTime = receiverTime / 1000 + interval / 2;
documents[i].IntervalTime = (tempTime - tempTime % interval) * 1000;
documents[i].Deviation = Math.abs(receiverTime - documents[i].IntervalTime);
// Generate a group key for each group, combinated of TransmitterMMSI and IntervalTime
groupKey = documents[i].IntervalTime.toString();
if (typeof aggGroup[groupKey] === 'undefined' || aggGroup[groupKey] > documents[i].Deviation) {
aggGroup[groupKey] = documents[i].Deviation;
}
}
// Tag the downsampling
for (i = 0; i < documents.length; i++) {
groupKey = documents[i].IntervalTime;
if (aggGroup[groupKey] == documents[i].Deviation) {
documents[i].DownSamplingTag = 1;
} else {
documents[i].DownSamplingTag = 0;
}
// Remove the items that are not used
delete documents[i].IntervalTime;
delete documents[i].Deviation;
// Replace the document
var acceptDoc = collection.replaceDocument(documents[i]._self, documents[i], {},
function (errDoc, docReplaced) {
if (errDoc) {
throw new Error("Update documents error:" + errDoc.message);
}
});
if (!acceptDoc) {
throw "Update documents not accepted, abort ";
}
}
response.setBody(documents.length);
});
if (!accept) {
throw new Error("The stored procedure timed out.");
}
}
And the C# code is here:
private async Task DownSampling()
{
Database database = this.client.CreateDatabaseQuery().Where(db => db.Id == DatabaseId).ToArray().FirstOrDefault();
DocumentCollection collection = this.client.CreateDocumentCollectionQuery(database.SelfLink).Where(c => c.Id == AISTestCollectionId).ToArray().FirstOrDefault();
string scriptFileName = #"..\..\StoredProcedures\DownSampling.js";
string scriptId = Path.GetFileNameWithoutExtension(scriptFileName);
var sproc = new StoredProcedure
{
Id = scriptId,
Body = File.ReadAllText(scriptFileName)
};
await TryDeleteStoredProcedure(collection.SelfLink, sproc.Id);
sproc = await this.client.CreateStoredProcedureAsync(collection.SelfLink, sproc);
IQueryable<dynamic> query = this.client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(DatabaseId, AISTestCollectionId),
new SqlQuerySpec()
{
//QueryText = "SELECT a.TransmitterMMSI FROM " + AISTestCollectionId + " a",
QueryText = "SELECT a.TransmitterMMSI FROM " + AISTestCollectionId + " a WHERE a.TransmitterMMSI=\"219633000\"",
}, new FeedOptions { MaxItemCount = -1, EnableCrossPartitionQuery = true, MaxDegreeOfParallelism = -1, MaxBufferedItemCount = -1 });
List<dynamic> transmitterMMSIList = query.ToList(); //TODO: Remove duplicates
Console.WriteLine("TransmitterMMSI count: {0}", transmitterMMSIList.Count());
HashSet<string> exist = new HashSet<string>();
foreach (var item in transmitterMMSIList)
{
//int transmitterMMSI = Int32.Parse(item.TransmitterMMSI.ToString());
string transmitterMMSI = item.TransmitterMMSI.ToString();
if (exist.Contains(transmitterMMSI))
{
continue;
}
exist.Add(transmitterMMSI);
Console.WriteLine("TransmitterMMSI: {0} is being processed.", transmitterMMSI);
var response = await this.client.ExecuteStoredProcedureAsync<string>(sproc.SelfLink,
new RequestOptions { PartitionKey = new PartitionKey(transmitterMMSI) }, transmitterMMSI, 30);
string s = response.Response;
Console.WriteLine("TransmitterMMSI: {0} is processed completely.", transmitterMMSI);
}
}
private async Task TryDeleteStoredProcedure(string collectionSelfLink, string sprocId)
{
StoredProcedure sproc = this.client.CreateStoredProcedureQuery(collectionSelfLink).Where(s => s.Id == sprocId).AsEnumerable().FirstOrDefault();
if (sproc != null)
{
await client.DeleteStoredProcedureAsync(sproc.SelfLink);
}
}
I tried to comment the 2 loops in the JS codes, only the documents.length output, while the response number is still less. However, I changed the query to SELECT a.id, the documents.length is correct. Looks like it is the continuation issue.
The sproc is probably timing out. To use a continuation token in these circumstances, you will need to return it to your C# calling code then make another call to the sproc passing in your token. If you show us your sproc code we can help more.
You can use a continuation token to make repeated calls to queryDocuments() from within the sproc without additional roundtrips to the client. Keep in mind that if you do this too many times your sproc will eventually timeout, though. In your case, it sounds like you're already very close to getting all of the documents you're seeking so maybe you will be OK.
Here is an example of using a continuation token within a sproc to query multiple pages of data:
function getManyThings() {
var collection = getContext().getCollection();
var query = {
query: 'SELECT r.id, r.FieldOne, r.FieldTwo FROM root r WHERE r.FieldThree="sought"'
};
var continuationToken;
getThings(continuationToken);
function getThings(continuationToken) {
var requestOptions = {
continuation: continuationToken,
pageSize: 1000 // Adjust this to suit your needs
};
var isAccepted = collection.queryDocuments(collection.getSelfLink(), query, requestOptions, function (err, feed, responseOptions) {
if (err) {
throw err;
}
for (var i = 0, len = feed.length; i < len; i++) {
var thing = feed[i];
// Do your logic on thing...
}
if (responseOptions.continuation) {
getThings(responseOptions.continuation);
}
else {
var response = getContext().getResponse();
response.setBody("RESULTS OF YOUR LOGIC");
}
});
if (!isAccepted) {
var response = getContext().getResponse();
response.setBody("Server rejected query - please narrow search criteria");
}
}
}
In DocumentDB I've create a pre trigger on Create operation. The trigger code is the following
function createBlock() {
var collection = getContext().getCollection();
var request = getContext().getRequest();
var docToCreate = request.getBody();
if (docToCreate.DocumentType)
{
var query = "SELECT TOP 1 a.BlockSequence FROM a ORDER BY a.BlockSequence DESC";
var isAccepted = collection.queryDocuments(collection.getSelfLink(), query, function (err, feed, options) {
if (err)
throw err;
if (!feed)
throw new Error("Failed to find the document.");
if (feed.length)
{
docToCreate.BlockCode += (feed[0].BlockSequence + 1);
docToCreate.BlockSequence = feed[0].BlockSequence + 1;
}
else
{
docToCreate.BlockCode += "1";
docToCreate.BlockSequence = 1;
}
var isAccepted = collection.createDocument(collection.getSelfLink(), docToCreate);
if (!isAccepted)
throw new Error("The call createDocument returned false.");
});
}
else
throw new Error("DocumentType property is required.");
if (!isAccepted)
throw new Error("The call queryDocuments returned false.");
}
The trigger is executed up to the line immediately above the var isAccepted = collection.createDocument(collection.getSelfLink(), docToCreate);.
When the var isAccepted = collection.createDocument(collection.getSelfLink(), docToCreate); is executed, this error is thrown Message: {"Errors":["Resource with specified id or name already exists"]}
I've checked and no documents with the same id of the new document is stored into this collection.
You shouldn't try to do the write in your trigger. You should simply modify the body or throw an error. In modifying the body, you change the document that is created. In throwing an error you abort the operation.
So instead of:
var isAccepted = collection.createDocument(collection.getSelfLink(), docToCreate);
Do:
return request.setBody(docToCreate);
I'm trying to retrieve all phone calls related to opportunity, which statecode isn't equal 1. Tried QueryByAttribute, QueryExpression and RetrieveMultipleRequest, but still has no solution.
Here some code i wrote.
IContextService contextService = (IContextService)executionContext.GetService(typeof(IContextService));
IWorkflowContext context = contextService.Context;
ICrmService crmService = context.CreateCrmService(true);
if (crmService != null)
{
QueryByAttribute query = new Microsoft.Crm.Sdk.Query.QueryByAttribute();
query.ColumnSet = new Microsoft.Crm.Sdk.Query.AllColumns();
query.EntityName = EntityName.phonecall.ToString();
query.Attributes = new string[] { "regardingobjectid" };
query.Values = new string[] { context.PrimaryEntityId.ToString() };
RetrieveMultipleRequest retrieve = new RetrieveMultipleRequest();
retrieve.Query = query;
retrieve.ReturnDynamicEntities = true;
RetrieveMultipleResponse retrieved = (RetrieveMultipleResponse)crmService.Execute(retrieve);
}
return ActivityExecutionStatus.Closed;
}
And almost same for QueryExpression
QueryExpression phCallsQuery = new QueryExpression();
ColumnSet cols = new ColumnSet(new string[] { "activityid", "regardingobjectid" });
phCallsQuery.EntityName = EntityName.phonecall.ToString();
phCallsQuery.ColumnSet = cols;
phCallsQuery.Criteria = new FilterExpression();
phCallsQuery.Criteria.FilterOperator = LogicalOperator.And;
phCallsQuery.Criteria.AddCondition("statuscode", ConditionOperator.NotEqual, "1");
phCallsQuery.Criteria.AddCondition("regardingobjectid", ConditionOperator.Equal, context.PrimaryEntityId.ToString();
I always get something like Soap exception or "Server was unable to proceed the request" when debugging.
To get exception details try to use following code:
RetrieveMultipleResponse retrieved = null;
try
{
retrieved = (RetrieveMultipleResponse)crmService.Execute(retrieve);
}
catch(SoapException se)
{
throw new Exception(se.Detail.InnerXml);
}