Here is my problem ..
We have a User table with over 50 million users. Now I want to retrieve some fbIds that I need to test a method (i.e. readUsersByFacebookIds())
My naive first attempt was to scan some records having a facebookId that is not null and then use those fbIds. This question timed out and my colleague mentioned that DataStore could have a hard time with NOT_EQUAL. I.e. can't use the index. (Sounds reasonable ...)
Attempt #1 Times out i.e. > 60 secs
public String testFbFriends(int count) {
try {
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Query q = new Query("User");
q.setFilter(new FilterPredicate("facebookId", Query.FilterOperator.NOT_EQUAL, null));
PreparedQuery pq = ds.prepare(q);
List<String> fbIds = new ArrayList<String>();
for (Entity userE : pq.asIterable(FetchOptions.Builder.withLimit(count).chunkSize(100))) {
User u = new User(1, userE, false);
fbIds.add(u.getFacebookId());
if (fbIds.size() >= count)
break;
}
List<User> users = UserModule.getInstance().readUsersByFacebookIds(1, fbIds, 0, 0);
return jsn.toJson(users);
} catch (Exception e) {
return STR.getStackTrace(e);
}
}
My second attempt was to scan a number of users and skip those where facebookId is null.
Attempt #2 - Also times out ...
public String testFbFriends(int count) {
try {
DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
Query q = new Query("User");
// q.setFilter(new FilterPredicate("facebookId", Query.FilterOperator.NOT_EQUAL, null));
PreparedQuery pq = ds.prepare(q);
List<String> fbIds = new ArrayList<String>();
for (Entity userE : pq.asIterable(FetchOptions.Builder.withLimit(count * 4).chunkSize(100))) {
User u = new User(1, userE, false);
if (u.getFacebookId() != null) {
fbIds.add(u.getFacebookId());
if (fbIds.size() >= count)
break;
}
}
List<User> users = UserModule.getInstance().readUsersByFacebookIds(1, fbIds, 0, 0);
return jsn.toJson(users);
} catch (Exception e) {
return STR.getStackTrace(e);
}
}
So does anyone out there know how to scan some records from a HUGE table?
Big thanks in advance!
Have you tried using cursors?
...
Query q = new Query("User");
if (cursorString != null) {
Cursor cursor = Cursor.fromWebSafeString(cursorString);
Map<String, Object> extensionMap = new HashMap<String, Object>();
extensionMap.put(JDOCursorHelper.CURSOR_EXTENSION, cursor);
q.setExtensions(extensionMap);
}
q.setRange(0, range);
...retrieve entities...
Cursor cursor = JDOCursorHelper.getCursor(results);
cursorString = cursor.toWebSafeString();
...
With cursors you can set the number of entities retrieved in each call, and avoid timeout.
Related
I have this Action method which act as an API end point inside our ASP.NET MVC-5, where it search for a username and return the username Phone number and Department from Active Directory (we are serializing the object using Newtonsoft.net):-
public ActionResult UsersInfo2()
{
DomainContext result = new DomainContext();
try
{
// create LDAP connection object
DirectoryEntry myLdapConnection = createDirectoryEntry();
string ADServerName = System.Web.Configuration.WebConfigurationManager.AppSettings["ADServerName"];
string ADusername = System.Web.Configuration.WebConfigurationManager.AppSettings["ADUserName"];
string ADpassword = System.Web.Configuration.WebConfigurationManager.AppSettings["ADPassword"];
using (var context = new DirectoryEntry("LDAP://mydomain.com:389/DC=mydomain,DC=com", ADusername, ADpassword))
using (var search = new DirectorySearcher(context))
{
// create search object which operates on LDAP connection object
// and set search object to only find the user specified
// DirectorySearcher search = new DirectorySearcher(myLdapConnection);
// search.PropertiesToLoad.Add("telephoneNumber");
search.Filter = "(&(objectClass=user)(sAMAccountName=test.test))";
SearchResult r = search.FindOne();
ResultPropertyCollection fields = r.Properties;
foreach (String ldapField in fields.PropertyNames)
{
// cycle through objects in each field e.g. group membership
// (for many fields there will only be one object such as name)
string temp;
// foreach (Object myCollection in fields[ldapField])
// {
// temp = String.Format("{0,-20} : {1}",
// ldapField, myCollection.ToString());
if (ldapField.ToLower() == "telephonenumber")
{
foreach (Object myCollection in fields[ldapField])
{
result.Telephone = myCollection.ToString();
}
}
else if (ldapField.ToLower() == "department")
{
foreach (Object myCollection in fields[ldapField])
{
result.Department = myCollection.ToString();
}
}
// }
}
string output = JsonConvert.SerializeObject(result);
return Json(output,JsonRequestBehavior.AllowGet);
}
}
catch (Exception e)
{
Console.WriteLine("Exception caught:\n\n" + e.ToString());
}
return View(result);
}
now the return JSON will be as follow:-
"\"DisplayName\":null,\"Telephone\":\"123123\",\"Department\":\"IT\",\"Name\":null,\"SamAccountName\":null,\"DistinguishedName\":null,\"UserPrincipalName\":null}"
but in our case we need to return a status code beside the return json data. for example inccase there is an exception we need to return an error code,also if we are able to get the user's info we need to pass succes code 200, and so on.. so how we can achieve this?
you can try something like this
var statusCode=200;
string output = JsonConvert.SerializeObject( new { result = result, StatusCode = statusCode);
but nobody usually do this. When users call API they can check status code that HTTP Client returns, using code like this
var response = await client.GetAsync(api);
//or
var response = await client.PutAsJsonAsync(api, data);
var statusCode = response.StatusCode.ToString();
//or usually
if (response.IsSuccessStatusCode) {...}
else {...}
I am implementing a multi-tenant application using cosmosDB. I am using partition keys to separate multiple users data. Following best practices i am trying to allow each tenant to have its own db access token.
I create a user and permission and use the created token to access the partition. But I get the following error:
Partition key provided either doesn't correspond to definition in the collection or doesn't match partition key field values specified
in the document.
ActivityId: 1659037a-118a-4a2d-8615-bb807b717fa7, Microsoft.Azure.Documents.Common/1.22.0.0, Windows/10.0.17134
documentdb-netcore-sdk/1.9.1
My code goes as follows:
Constructor Initiates the client
public Projects (CosmosDbConfig cosmosConfig)
{
config = cosmosConfig;
client = new DocumentClient(new Uri(config.Endpoint), config.AuthKey);
collectionUri = UriFactory.CreateDocumentCollectionUri(config.Database, config.Collection);
config.AuthKey = GetUserToken().Result;;
client = new DocumentClient(new Uri(config.Endpoint), config.AuthKey);
}
The get user function creates the user and retrieves the token. User Ids are partition keys.
private async Task<string> GetUserToken()
{
User user = null;
try
{
try
{
user = await client.ReadUserAsync(UriFactory.CreateUserUri(config.Database, config.PartitionKey));
var permission = await GetorCreatePermission(user, config.Collection, config.PartitionKey);
return permission.Token;
}
catch (Exception ex) {
Console.WriteLine(ex.Message);
}
if (user == null)
{
user = new User
{
Id = config.PartitionKey
};
user = await client.CreateUserAsync(UriFactory.CreateDatabaseUri(config.Database), user);
var permission = await GetorCreatePermission(user, config.Collection, config.PartitionKey);
return permission.Token;
}
else
{
throw new Exception("");
}
}
catch (Exception ex)
{
throw ex;
}
}
Permission are done per collections and holds the collection name as ID since Ids are unique per user.
private async Task<Permission> GetorCreatePermission(User user,
string collection,
string paritionKey)
{
var permDefinition = new Permission
{
Id = collection,
PermissionMode = PermissionMode.All,
ResourceLink = collectionUri.OriginalString,
ResourcePartitionKey = new PartitionKey(paritionKey),
};
var perms = client.CreatePermissionQuery(user.PermissionsLink).AsEnumerable().ToList();
var perm = perms.FirstOrDefault(x => x.Id == collection);
if (perm != null)
{
return perm;
}
else
{
var result = await client.CreatePermissionAsync(user.SelfLink, permDefinition);
perm = result.Resource;
return perm;
}
}
The create function utilizes the new client and this where the error occurs.
public async Task<string> Create(Project p)
{
var result = await client.CreateDocumentAsync(collectionUri, p, new RequestOptions()
{ PartitionKey = new PartitionKey(config.PartitionKey),
});
var document = result.Resource;
return document.Id;
}
Since error says that partition key is incorrect i can suggest you try define partition key pathes while creating collection:
var docCollection = new DocumentCollection();
docCollection.Id = config.CollectionName;
docCollection.PartitionKey.Paths.Add(string.Format("/{0}", config.PartitionKey );
collectionUri = UriFactory.CreateDocumentCollectionUri(config.Database, docCollection);
I'm trying to retrieve all phone calls related to opportunity, which statecode isn't equal 1. Tried QueryByAttribute, QueryExpression and RetrieveMultipleRequest, but still has no solution.
Here some code i wrote.
IContextService contextService = (IContextService)executionContext.GetService(typeof(IContextService));
IWorkflowContext context = contextService.Context;
ICrmService crmService = context.CreateCrmService(true);
if (crmService != null)
{
QueryByAttribute query = new Microsoft.Crm.Sdk.Query.QueryByAttribute();
query.ColumnSet = new Microsoft.Crm.Sdk.Query.AllColumns();
query.EntityName = EntityName.phonecall.ToString();
query.Attributes = new string[] { "regardingobjectid" };
query.Values = new string[] { context.PrimaryEntityId.ToString() };
RetrieveMultipleRequest retrieve = new RetrieveMultipleRequest();
retrieve.Query = query;
retrieve.ReturnDynamicEntities = true;
RetrieveMultipleResponse retrieved = (RetrieveMultipleResponse)crmService.Execute(retrieve);
}
return ActivityExecutionStatus.Closed;
}
And almost same for QueryExpression
QueryExpression phCallsQuery = new QueryExpression();
ColumnSet cols = new ColumnSet(new string[] { "activityid", "regardingobjectid" });
phCallsQuery.EntityName = EntityName.phonecall.ToString();
phCallsQuery.ColumnSet = cols;
phCallsQuery.Criteria = new FilterExpression();
phCallsQuery.Criteria.FilterOperator = LogicalOperator.And;
phCallsQuery.Criteria.AddCondition("statuscode", ConditionOperator.NotEqual, "1");
phCallsQuery.Criteria.AddCondition("regardingobjectid", ConditionOperator.Equal, context.PrimaryEntityId.ToString();
I always get something like Soap exception or "Server was unable to proceed the request" when debugging.
To get exception details try to use following code:
RetrieveMultipleResponse retrieved = null;
try
{
retrieved = (RetrieveMultipleResponse)crmService.Execute(retrieve);
}
catch(SoapException se)
{
throw new Exception(se.Detail.InnerXml);
}
LINQ to Entities does not recognize the method 'System.String ToString()' method, and this method cannot be translated into a store expression.
public ActionResult PopulateFromDB(string sidx, string sord, int page, int rows)
{
var context = new NerdDinnerEntities();
var jsonData = new
{
total = 1,
page = page,
sord =sord,
records = context.Authors.Count(),
rows = (from n in context.Authors
select new
{ AuthorId = n.AuthorId ,
cell = new string[] { n.AuthorId.ToString(), n.Name.ToString(), n.Location.ToString() }
}).ToList()
};
return Json(jsonData, JsonRequestBehavior.AllowGet);
}
I am writting ToList or Toarray is it not working the error comes :
public ActionResult PopulateFromDB(string sidx, string sord, int page, int rows)
{
var context = new NerdDinnerEntities();
var jsonData = new
{
total = 1,
page = page,
sord =sord,
records = context.Authors.Count(),
rows = (from n in context.Authors
select new
{ AuthorId = n.AuthorId ,
cell = new string[] { n.AuthorId.ToString(), n.Name.ToString(), n.Location.ToString() }
}).ToList()
};
return Json(jsonData,JsonRequestBehavior.AllowGet);
}
From your code I assume your adding a custom property cell for display/storage purposes on the client-side. I would avoid this as your essentially coupling your API call to one particular client. I would suggest you simply return the data required & deal with it at the client-side specifically e.g.
Server
...
select new
{
Id = n.AuthorId,
Name = n.Name,
Location = n.Location
}).ToList();
...
Client
var response = ...
foreach (var author in response)
{
var cell = new string[] { author.Id.ToString(), author.Name, author.Location };
// do something with cell
}
You should try SqlFunctions.StringConvert to convert this, There is no overload for int so you should cast your number to a double or a decimal.
public ActionResult PopulateFromDB(string sidx, string sord, int page, int rows)
{
var context = new NerdDinnerEntities();
var jsonData = new
{
total = 1,
page = page,
sord =sord,
records = context.Authors.Count(),
rows = (from n in context.Authors
select new
{ AuthorId = n.AuthorId ,
cell = new string[] { SqlFunctions.StringConvert((double)n.AuthorId), n.Name, n.Location }
}).ToList()
};
return Json(jsonData,JsonRequestBehavior.AllowGet);
}
You are not using LinqToSql Classes, if you were using that your code should work, but as you mention that you are using LinqToEntity then You should use SqlFunctions.StringConvert to convert to string.
I have the following bits of code in my accountRepository
public string[] GetRolesForUser(string email)
{
// User rolesUser = FindByMail(email);
IEnumerable<UserRole> RoleList = context.UserRolesSet.Where(u => u.user_id == 1).AsEnumerable();
string[] arr1 = new string[RoleList.Count()];
int i = 0;
foreach (UserRole r in RoleList)
{
arr1[i] = r.roles.name;
i++;
}
return arr1;
}
This should work but it doesn't. When it looping through the foreach loop it throws me this error:
Exception Details: MySql.Data.MySqlClient.MySqlException: There is already an open DataReader associated with this Connection which must be closed first.
Is my foreach loop wrong?
You could simplify the method:
IEnumerable<UserRole> RoleList = context.UserRolesSet.Where(u => u.user_id == 1);
return RoleList.Select(x => x.roles.name).ToArray();
Try the following code. Using ToArray makes sure it populates all of the UserRoles before hand so it'll be finished with the DataReader.
public string[] GetRolesForUser(string email)
{
// User rolesUser = FindByMail(email);
IEnumerable<UserRole> RoleList = context.UserRolesSet.Where(u => u.user_id == 1).ToArray();
string[] arr1 = new string[RoleList.Count()];
int i = 0;
foreach (UserRole r in RoleList)
{
arr1[i] = r.roles.name;
i++;
}
return arr1;
}