Need help on using documentdb-lumenize on .net documentdb client sdk - azure-cosmosdb

I have a problem using the aggregate storedproc lumenize https://github.com/lmaccherone/documentdb-lumenize with the .net client. I get error when try passing in the parameter and query into the storedproc. Below is my code
public async static void QuerySP() {
using (client = new DocumentClient(new Uri(endpointUrl), authorizationKey))
{
//Get the Database
var database = client.CreateDatabaseQuery().Where(db => db.Id == databaseId).ToArray().FirstOrDefault();
//Get the Document Collection
var collection = client.CreateDocumentCollectionQuery(database.SelfLink).Where(c => c.Id == collectionId).ToArray().FirstOrDefault();
StoredProcedure storedProc = client.CreateStoredProcedureQuery(collection.StoredProceduresLink).Where(sp => sp.Id == "cube").ToArray().FirstOrDefault();
dynamic result = await client.ExecuteStoredProcedureAsync<dynamic>(storedProc.SelfLink, "{cubeConfig: {groupBy: 'publication', field: 'pid', f: 'count'}, filterQuery: 'SELECT pid, publication FROM c'}");
Console.WriteLine("Result from script: {0}\r\n", result.Response);
}
}
I am getting the following error when execute the code
Message: {"Errors":["Encountered exception while executing Javascript. Exception = Error: cubeConfig or savedCube required\r\nStack trace: Error: cubeConfig or savedCube required\n at fn (cube.js:1803:7)\n at __docDbMain (cube.js:1844:5)\n at Unknown script code (cube.js:1:2)"]}
Not sure what I had done wrong. I would really appreciate the help. Thanks.

You almost have it. The problem is that you are sending in the cubeConfig as a string. It needs to be an object. Here is code that does that:
string cubeConfigString = #"{
cubeConfig: {
groupBy: 'publication',
field: 'pid',
f: 'count'
},
filterQuery: 'SELECT * FROM c'
}";
Object cubeConfig = JsonConvert.DeserializeObject<Object>(cubeConfigString);
Console.WriteLine(cubeConfig);
dynamic result = await client.ExecuteStoredProcedureAsync<dynamic>("dbs/dev-test-database/colls/dev-test-collection/sprocs/cube", cubeConfig);
Console.WriteLine(result.Response);

my working code
public async static Task QuerySP2()
{
using (client = new DocumentClient(new Uri(endpointUrl), authorizationKey))
{
//Get the Database
var database = client.CreateDatabaseQuery().Where(db => db.Id == databaseId).ToArray().FirstOrDefault();
//Get the Document Collection
var collection = client.CreateDocumentCollectionQuery(database.SelfLink).Where(c => c.Id == collectionId).ToArray().FirstOrDefault();
StoredProcedure storedProc = client.CreateStoredProcedureQuery(collection.StoredProceduresLink).Where(sp => sp.Id == "cube").ToArray().FirstOrDefault();
string filterQuery = string.Format(#"SELECT * from c");
string cubeConfigString = #"{
cubeConfig: {
groupBy: 'publication',
field: 'id',
f: 'count'
},
filterQuery: '" + filterQuery + "'}";
dynamic cubeConfig = JsonConvert.DeserializeObject<dynamic>(cubeConfigString);
Console.WriteLine(cubeConfig);
string continuationToken = null;
dynamic result=null;
do
{
var queryDone = false;
while (!queryDone)
{
try
{
result = await client.ExecuteStoredProcedureAsync<dynamic>(storedProc.SelfLink, cubeConfig);
cubeConfig = result.Response;
continuationToken = cubeConfig.continuation;
queryDone = true;
}
catch (DocumentClientException documentClientException)
{
var statusCode = (int)documentClientException.StatusCode;
if (statusCode == 429 || statusCode == 503)
System.Threading.Thread.Sleep(documentClientException.RetryAfter);
else
throw;
}
catch (AggregateException aggregateException)
{
if (aggregateException.InnerException.GetType() == typeof(DocumentClientException))
{
var docExcep = aggregateException.InnerException as DocumentClientException;
var statusCode = (int)docExcep.StatusCode;
if (statusCode == 429 || statusCode == 503)
System.Threading.Thread.Sleep(docExcep.RetryAfter);
else
throw;
}
}
}
} while (continuationToken != null);
Console.WriteLine("Result from script: {0}\r\n", result.Response);
}
}

Related

how we can return a status code for the serialized JSON object using Newtonsoft.net

I have this Action method which act as an API end point inside our ASP.NET MVC-5, where it search for a username and return the username Phone number and Department from Active Directory (we are serializing the object using Newtonsoft.net):-
public ActionResult UsersInfo2()
{
DomainContext result = new DomainContext();
try
{
// create LDAP connection object
DirectoryEntry myLdapConnection = createDirectoryEntry();
string ADServerName = System.Web.Configuration.WebConfigurationManager.AppSettings["ADServerName"];
string ADusername = System.Web.Configuration.WebConfigurationManager.AppSettings["ADUserName"];
string ADpassword = System.Web.Configuration.WebConfigurationManager.AppSettings["ADPassword"];
using (var context = new DirectoryEntry("LDAP://mydomain.com:389/DC=mydomain,DC=com", ADusername, ADpassword))
using (var search = new DirectorySearcher(context))
{
// create search object which operates on LDAP connection object
// and set search object to only find the user specified
// DirectorySearcher search = new DirectorySearcher(myLdapConnection);
// search.PropertiesToLoad.Add("telephoneNumber");
search.Filter = "(&(objectClass=user)(sAMAccountName=test.test))";
SearchResult r = search.FindOne();
ResultPropertyCollection fields = r.Properties;
foreach (String ldapField in fields.PropertyNames)
{
// cycle through objects in each field e.g. group membership
// (for many fields there will only be one object such as name)
string temp;
// foreach (Object myCollection in fields[ldapField])
// {
// temp = String.Format("{0,-20} : {1}",
// ldapField, myCollection.ToString());
if (ldapField.ToLower() == "telephonenumber")
{
foreach (Object myCollection in fields[ldapField])
{
result.Telephone = myCollection.ToString();
}
}
else if (ldapField.ToLower() == "department")
{
foreach (Object myCollection in fields[ldapField])
{
result.Department = myCollection.ToString();
}
}
// }
}
string output = JsonConvert.SerializeObject(result);
return Json(output,JsonRequestBehavior.AllowGet);
}
}
catch (Exception e)
{
Console.WriteLine("Exception caught:\n\n" + e.ToString());
}
return View(result);
}
now the return JSON will be as follow:-
"\"DisplayName\":null,\"Telephone\":\"123123\",\"Department\":\"IT\",\"Name\":null,\"SamAccountName\":null,\"DistinguishedName\":null,\"UserPrincipalName\":null}"
but in our case we need to return a status code beside the return json data. for example inccase there is an exception we need to return an error code,also if we are able to get the user's info we need to pass succes code 200, and so on.. so how we can achieve this?
you can try something like this
var statusCode=200;
string output = JsonConvert.SerializeObject( new { result = result, StatusCode = statusCode);
but nobody usually do this. When users call API they can check status code that HTTP Client returns, using code like this
var response = await client.GetAsync(api);
//or
var response = await client.PutAsJsonAsync(api, data);
var statusCode = response.StatusCode.ToString();
//or usually
if (response.IsSuccessStatusCode) {...}
else {...}

Modify BuildApiResponse in ASP.Net Web Api

I am new in ASP.NET MVC Web API. I am trying to modified the return JSon to this format
{
"Error": false,
"Status": 200,
"Response": []
}
Now I able to do that by follow this post https://www.devtrends.co.uk/blog/wrapping-asp.net-web-api-responses-for-consistency-and-to-provide-additional-information . But the problem is I not able to show ModelState error like 'First name is required' because the code only show the first hit error.
if (error != null)
{
content = null;
//only show the first error
errorMessage = error.Message;
}
So I did some modification, now the code is written as below:
if (error != null)
{
content = null;
foreach(var e in error)
{
//if the error's type is ModelState
if (e.Key.Equals("ModelState"))
{
var allErrors = e.Value;
foreach (var modelError in (IEnumerable<KeyValuePair<string, object>>)allErrors)
{
var msg = modelError;
errorMessage = string.Concat(errorMessage, ", ", ((String[]) modelError.Value)[0]);
}
}
else
{
errorMessage = e.Value.ToString();
}
}
}
Now it's able to show all errors but the code is messy. I am writing this questions to find out what is the proper way to write this kind of code.
You can iterate over all the errors and concatenate them using StringBuilder. String.Join is much faster than Append for less than 1000 items (it is unlikely you will have so many errors in the modelstate object):
public static ValidationResult CheckValid(ModelStateDictionary modelState, string httpName = null)
{
if (!modelState.IsValid)
{
var sb = new StringBuilder();
sb.AppendLine(httpName + " failed: Invalid Json:");
foreach (var pair in modelState)
{
var error = String.Join(";", pair.Value.Errors.Select
(
i =>
{
if (!String.IsNullOrEmpty(i.ErrorMessage))
return i.ErrorMessage;
return i.Exception.Message;
}
));
sb.AppendLine($"Property: {pair.Key} Errors: ({error})");
}
return new ValidationResult(false, sb.ToString());
}
else
return new ValidationResult(true, "");
}

Retrieve an Entity value to add to return parameter

I have this method:
private List < ReportParameter > ParametrosReporte() {
List < ReportParameter > Parametros = new List < ReportParameter > ();
try {
int ? int_Ejercicio = this.radcmbEjercicio.SelectedItem == null ? 0 : this.radcmbEjercicio.SelectedValue.ToInt();
int ? int_Periodo = this.radcmbPeriodo.SelectedItem == null ? 0 : this.radcmbPeriodo.SelectedValue.ToInt();
int ? int_BSC = this.radcmbBSC.SelectedItem == null ? 0 : this.radcmbBSC.SelectedValue.ToInt();
Parametros.Add(Reportes.ParametrosReporte("pe_Ejercicio", int_Ejercicio.ToString()));
Parametros.Add(Reportes.ParametrosReporte("pe_Mes", int_Periodo.ToString()));
Parametros.Add(Reportes.ParametrosReporte("pe_BSC", int_BSC.ToString()));
catBSC _catBSC = new catBSC() {
mdUsuarioCaptura = new Entidades.Usuario() {
UsuarioID = ((Usuario) Session["User"]).UsuarioID,
}
};
Parametros.Add(Reportes.ParametrosReporte("pe_Usuario", UsuarioID ));
return Parametros;
} catch (Exception ex) {
throw new System.ArgumentException("Error en ParametrosReporte", ex);
}
}
As you can see I have logic to retrieve user who is logged in as:
catBSC _catBSC = new catBSC() {
mdUsuarioCaptura = new Entidades.Usuario() {
UsuarioID = ((Usuario) Session["User"]).UsuarioID,
}
};
but before retrieve it, I want to call it into Parametros.Add like:
Parametros.Add(Reportes.ParametrosReporte("pe_Usuario", UsuarioID ));
But I can´t because UsuarioID is out of scope and it throws me
UsuarioID does not exist in the current context
How can I call it and attach to my Parametros.Add?
You can just create a local variable above your creation of catBSC and then use that in both locations:
var usuarioID = ((Usuario) Session["User"]).UsuarioID
catBSC _catBSC = new catBSC() {
mdUsuarioCaptura = new Entidades.Usuario() {
UsuarioID = usuarioID
}
};
Parametros.Add(Reportes.ParametrosReporte("pe_Usuario", usuarioID));
You can replace var with the actual type, but I didn't want to make an assumption about what this was in your scenario.

Cosmosdb store procedure get less documents than real

I am preparing store procedure on cosmosdb by Javascript, however, it gets less documents than the real number of documents in collection.
The sproc is called by C#, C# pass a parameter "transmitterMMSI" which is also the partition key of this collection.
First, the following query is executed in sproc:
var query = 'SELECT COUNT(1) AS Num FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
The result is output in response, and the value is 5761, which is the same as the real number of documents in collection.
However, when I change the query to the following:
var query = 'SELECT * FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
The documents.length is output as 5574, which is smaller than the real number.
I have already changed the pageSize: -1, which should mean unlimited.
I did some search with google and stack overflow, it seems that continuation can be help. However, I tried some examples, and they don't work.
Anyone familiar with this can help?
The following list the scripts.
The sproc js script is here, which is also the file "DownSampling.js" used in the C# code:
function DownSampling(transmitterMMSI, interval) {
var context = getContext();
var collection = context.getCollection();
var response = context.getResponse();
var receiverTime;
var tempTime;
var groupKey;
var aggGroup = new Object();
var query = 'SELECT * FROM AISData a WHERE a.TransmitterMMSI="' + transmitterMMSI + '"';
var accept = collection.queryDocuments(collection.getSelfLink(), query, { pageSize: -1},
function (err, documents, responseOptions) {
if (err) throw new Error("Error" + err.message);
// Find the smallest deviation comparting to IntervalTime in each group
for (i = 0; i < documents.length; i++) {
receiverTime = Date.parse(documents[i].ReceiverTime);
tempTime = receiverTime / 1000 + interval / 2;
documents[i].IntervalTime = (tempTime - tempTime % interval) * 1000;
documents[i].Deviation = Math.abs(receiverTime - documents[i].IntervalTime);
// Generate a group key for each group, combinated of TransmitterMMSI and IntervalTime
groupKey = documents[i].IntervalTime.toString();
if (typeof aggGroup[groupKey] === 'undefined' || aggGroup[groupKey] > documents[i].Deviation) {
aggGroup[groupKey] = documents[i].Deviation;
}
}
// Tag the downsampling
for (i = 0; i < documents.length; i++) {
groupKey = documents[i].IntervalTime;
if (aggGroup[groupKey] == documents[i].Deviation) {
documents[i].DownSamplingTag = 1;
} else {
documents[i].DownSamplingTag = 0;
}
// Remove the items that are not used
delete documents[i].IntervalTime;
delete documents[i].Deviation;
// Replace the document
var acceptDoc = collection.replaceDocument(documents[i]._self, documents[i], {},
function (errDoc, docReplaced) {
if (errDoc) {
throw new Error("Update documents error:" + errDoc.message);
}
});
if (!acceptDoc) {
throw "Update documents not accepted, abort ";
}
}
response.setBody(documents.length);
});
if (!accept) {
throw new Error("The stored procedure timed out.");
}
}
And the C# code is here:
private async Task DownSampling()
{
Database database = this.client.CreateDatabaseQuery().Where(db => db.Id == DatabaseId).ToArray().FirstOrDefault();
DocumentCollection collection = this.client.CreateDocumentCollectionQuery(database.SelfLink).Where(c => c.Id == AISTestCollectionId).ToArray().FirstOrDefault();
string scriptFileName = #"..\..\StoredProcedures\DownSampling.js";
string scriptId = Path.GetFileNameWithoutExtension(scriptFileName);
var sproc = new StoredProcedure
{
Id = scriptId,
Body = File.ReadAllText(scriptFileName)
};
await TryDeleteStoredProcedure(collection.SelfLink, sproc.Id);
sproc = await this.client.CreateStoredProcedureAsync(collection.SelfLink, sproc);
IQueryable<dynamic> query = this.client.CreateDocumentQuery(
UriFactory.CreateDocumentCollectionUri(DatabaseId, AISTestCollectionId),
new SqlQuerySpec()
{
//QueryText = "SELECT a.TransmitterMMSI FROM " + AISTestCollectionId + " a",
QueryText = "SELECT a.TransmitterMMSI FROM " + AISTestCollectionId + " a WHERE a.TransmitterMMSI=\"219633000\"",
}, new FeedOptions { MaxItemCount = -1, EnableCrossPartitionQuery = true, MaxDegreeOfParallelism = -1, MaxBufferedItemCount = -1 });
List<dynamic> transmitterMMSIList = query.ToList(); //TODO: Remove duplicates
Console.WriteLine("TransmitterMMSI count: {0}", transmitterMMSIList.Count());
HashSet<string> exist = new HashSet<string>();
foreach (var item in transmitterMMSIList)
{
//int transmitterMMSI = Int32.Parse(item.TransmitterMMSI.ToString());
string transmitterMMSI = item.TransmitterMMSI.ToString();
if (exist.Contains(transmitterMMSI))
{
continue;
}
exist.Add(transmitterMMSI);
Console.WriteLine("TransmitterMMSI: {0} is being processed.", transmitterMMSI);
var response = await this.client.ExecuteStoredProcedureAsync<string>(sproc.SelfLink,
new RequestOptions { PartitionKey = new PartitionKey(transmitterMMSI) }, transmitterMMSI, 30);
string s = response.Response;
Console.WriteLine("TransmitterMMSI: {0} is processed completely.", transmitterMMSI);
}
}
private async Task TryDeleteStoredProcedure(string collectionSelfLink, string sprocId)
{
StoredProcedure sproc = this.client.CreateStoredProcedureQuery(collectionSelfLink).Where(s => s.Id == sprocId).AsEnumerable().FirstOrDefault();
if (sproc != null)
{
await client.DeleteStoredProcedureAsync(sproc.SelfLink);
}
}
I tried to comment the 2 loops in the JS codes, only the documents.length output, while the response number is still less. However, I changed the query to SELECT a.id, the documents.length is correct. Looks like it is the continuation issue.
The sproc is probably timing out. To use a continuation token in these circumstances, you will need to return it to your C# calling code then make another call to the sproc passing in your token. If you show us your sproc code we can help more.
You can use a continuation token to make repeated calls to queryDocuments() from within the sproc without additional roundtrips to the client. Keep in mind that if you do this too many times your sproc will eventually timeout, though. In your case, it sounds like you're already very close to getting all of the documents you're seeking so maybe you will be OK.
Here is an example of using a continuation token within a sproc to query multiple pages of data:
function getManyThings() {
var collection = getContext().getCollection();
var query = {
query: 'SELECT r.id, r.FieldOne, r.FieldTwo FROM root r WHERE r.FieldThree="sought"'
};
var continuationToken;
getThings(continuationToken);
function getThings(continuationToken) {
var requestOptions = {
continuation: continuationToken,
pageSize: 1000 // Adjust this to suit your needs
};
var isAccepted = collection.queryDocuments(collection.getSelfLink(), query, requestOptions, function (err, feed, responseOptions) {
if (err) {
throw err;
}
for (var i = 0, len = feed.length; i < len; i++) {
var thing = feed[i];
// Do your logic on thing...
}
if (responseOptions.continuation) {
getThings(responseOptions.continuation);
}
else {
var response = getContext().getResponse();
response.setBody("RESULTS OF YOUR LOGIC");
}
});
if (!isAccepted) {
var response = getContext().getResponse();
response.setBody("Server rejected query - please narrow search criteria");
}
}
}

Why am I getting Http error 500 when I return too many rows as json in my ajax call?

Here is my code:
var context = _contextFactory.Create(GetClient());
var r = from p in context.MyTable.ToList()
select p;
int tot;
var firstPageData = PagedResult(r, 0, 200000, rows => new { rows.Id }, true, out tot);
return Json(new { result = "ok", rows = firstPageData }, JsonRequestBehavior.AllowGet);
The first and second parameter to PagedResult() (0 and 200000) are the pagenumber and number of rows to return. This is returning http error 500.
But when I change the 2nd parameter to PagedResult() to return only 20 rows, it's working fine. Like this:
var context = _contextFactory.Create(GetClient());
var r = from p in context.MyTable.ToList()
select p;
int tot;
var firstPageData = PagedResult(r, 0, 20, rows => new { rows.Id }, true, out tot);
return Json(new { result = "ok", rows = firstPageData }, JsonRequestBehavior.AllowGet);
Is it I am getting this error because I am returning too many rows that http can't handle? Or are there configurations I need to make so I can return these too many rows?
Thanks,
use a custom JsonResult Class for ASP.Net MVC to avoid MaxJsonLength Exceeded Exception.
public class LargeJsonResult : JsonResult
{
const string JsonRequest_GetNotAllowed = "This request has been blocked because sensitive information could be disclosed to third party web sites when this is used in a GET request. To allow GET requests, set JsonRequestBehavior to AllowGet.";
public LargeJsonResult()
{
MaxJsonLength = 1024000;
RecursionLimit = 100;
}
public int MaxJsonLength { get; set; }
public int RecursionLimit { get; set; }
public override void ExecuteResult( ControllerContext context )
{
if( context == null )
{
throw new ArgumentNullException( "context" );
}
if( JsonRequestBehavior == JsonRequestBehavior.DenyGet &&
String.Equals( context.HttpContext.Request.HttpMethod, "GET", StringComparison.OrdinalIgnoreCase ) )
{
throw new InvalidOperationException( JsonRequest_GetNotAllowed );
}
HttpResponseBase response = context.HttpContext.Response;
if( !String.IsNullOrEmpty( ContentType ) )
{
response.ContentType = ContentType;
}
else
{
response.ContentType = "application/json";
}
if( ContentEncoding != null )
{
response.ContentEncoding = ContentEncoding;
}
if( Data != null )
{
JavaScriptSerializer serializer = new JavaScriptSerializer() { MaxJsonLength = MaxJsonLength, RecursionLimit = RecursionLimit };
response.Write( serializer.Serialize( Data ) );
}
}
}
and use it as
return new LargeJsonResult { Data = Your Data, JsonRequestBehavior = `System.Web.Mvc.JsonRequestBehavior.AllowGet };`

Resources