Core Data: fetching items is slow with predicate - sqlite

For my iPhone application I set up a data model for Core Data. It contains one entity Words and its attributes are language : String, length : Integer16 and word : String.
I prefilled my model's SQLite database with a word list (200k items) writing a separate iPhone application using the identical data model and coping the filled database to the main application.
Now using NSFetchedRequest I can query for managed objects as I like, but the results come in slow. I use the following method:
- (NSString *)getRandomWordLengthMin:(int)minLength max:(int)maxLength
{
NSString *word = #"";
MyAppDelegate *appDelegate = [[UIApplication sharedApplication] delegate];
NSManagedObjectContext *context = [appDelegate managedObjectContext];
NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] init];
NSEntityDescription *entity = [NSEntityDescription entityForName:#"Words"
inManagedObjectContext:context];
[fetchRequest setEntity:entity];
NSString *predicateString = #"length >= %d AND length <= %d";
NSPredicate *predicate = [NSPredicate predicateWithFormat:predicateString,
minLength, maxLength];
[fetchRequest setPredicate:predicate];
NSError *error = nil;
int entityCount = [context countForFetchRequest:fetchRequest error:&error];
[fetchRequest setFetchLimit:1];
if(entityCount != 0)
{
[fetchRequest setFetchOffset:arc4random()%entityCount];
}
NSArray *fetchedObjects = [context executeFetchRequest:fetchRequest error:&error];
if([fetchedObjects count] != 0)
{
Words * test = [fetchedObjects objectAtIndex:0];
word = [NSString stringWithFormat:#"%#", [test word]];
}
return word;
}
Using an SQLite editor I already set an index manually on column zLength, but this didn't bring any speedup. Where is the bottleneck?
EDIT:
I figured out that getting int entityCount = ... is slow. But even getting all objects and then selecting one random word is slow:
Words * test = [fetchedObjects objectAtIndex:arc4random()%[fetchedObjects count]];

You are effectively running two fetches here, one to get the fetch count and then one to fetch the actual object. That will slow things down.
Your predicate is "backwards." Compound predicates evaluate the first expression e.g. length >= %d and then evaluate the second e.g. length <= %d only against the results of the first. Therefore you should put the test that eliminates the most objects first. In this case, length <= %d probably eliminates more objects so it should come first in the predicate.
Since you don't actually need the entire Words managed object but just the word string, you can set the fetch return type to NSDictionaryResultType and then set the property to fetch to just the word attribute. That will speed things up considerably.
Part of your problem here is that Core Data is designed to managed a structured object graph and you are using a random/unstructured graph so you are cutting against the grain of Core Data's optimizations.

Do not use the SQLite editor to edit the SQLite backing store for a Core Data storage. The internals of the database is private and subject to change.
Instead go the the model editor in Xcode and simply put a checkmark on the "indexed" option for the entity attribute you want indexed.
Not sure but maybe this predicate is easier to optimize:
NSString *predicateString = #"length BETWEEN (%d, %d)";

Related

Why does this type conversion of NSString to NSNumber returns invalid value of -1?

This is the line of code in question:
bks.quantity = [NSNumber numberWithInteger: [[arrayOfSplitStrings[i] objectAtIndex:[[sd.dictionaryOfUserIndexes objectForKey: #"29"] intValue]] intValue]-1];
sd.dictionaryOfUserIndexes objectForKey: #"29" contains a string (representing a quantity) that has to be converted to NSNumber. When the statement is executed with a valid string quantity (0-10) it always returns -1, when it is supposed to return the NSNumber for the value.
What am I doing wrong?
This is not a "straight forward" answer (since the solution is just a silly one), it's more a suggestion on work methods, that's why I post an answer.
It's not always good to put it various lines in a single line.
Especially when in your case you encounter an issue. It's better to split each command, one by one, and to debug, check the value of each ones.
In your case:
bks.quantity = [NSNumber numberWithInteger: [[arrayOfSplitStrings[i] objectAtIndex:[[sd.dictionaryOfUserIndexes objectForKey: #"29"] intValue]] intValue]-1];
==>
NSInteger userOfIndexes = [[sd.dictionaryOfUserIndexes objectForKey: #"29"] intValue];
NSLog(#"userOfIndexes: %d", userOfIndexes);
NSInteger n = [arrayOfSplitStrings[i] objectAtIndex:userOfIndexes] intValue];
NSLog(#"n: %d", n);
bks.quantity = [NSNumberWithInteger:n-1];
I added NSLog(), but the values could be check with breakpoints and debugger. I could have also add a check on arrayOfSplitStrings with
NSArray *splitStrings = arrayOfSplitString[i];
NSLog(#"splitStrings: %#", splitStrings);
and replace n with:
NSInteger n = [splitStrings objectAtIndex:userOfIndexes] intValue];
That way, you would have check that apparently (according to your comment), your issue was were to put the "-1.
NSInteger n = [[arrayOfSplitStrings[i] objectAtIndex: userIndex-1] intValue];

Firebase FQuery how do you detect when at the end of a list of nodes

How do I detect when I have finished processing all found nodes when doing a query? In the following example, I do some processing on each encountered node. When I reach the "end" of the list I would like to be able to detect this so I know it's finished.
FQuery* messageListQuery = [m_firebaseRef queryLimitedToNumberOfChildren:100];
[messageListQuery observeEventType:FEventTypeChildAdded andPreviousSiblingNameWithBlock:^(FDataSnapshot *snapshot, NSString *prevNodeName) {
// 1. Do interesting stuff with the snapshot data
// 2. I want to detect when I'm at the end of the list so I know when I'm done processing the list.
}];
Here is the example use case. I would like to load the latest 100 messages in the background. Once the messages have been loaded, I would like to update the UI. However, I'm not sure how I know all the messages have been loaded given there might be less then 100 messages in the list.
I figured out how to read all the messages up front by using the observeSingleEventOfType and then iterating over the children.
[m_firebaseRef observeSingleEventOfType:FEventTypeValue withBlock:^(FDataSnapshot *snapshot) {
NSLog( #"Name %# with %d children.", snapshot.name, snapshot.childrenCount );
for( FDataSnapshot *child in snapshot.children )
{
NSDictionary *msgData = child.value;
NSString *message = msgData[kFirebaseLiveChatFieldMessage];
NSString *gamerTag = msgData[kFirebaseLiveChatFieldGamerTag];
NSString *gameCenterId = msgData[kFirebaseLiveChatFieldGameCenterId];
NSLog( #"Preload = %# (%#): %#", gamerTag, gameCenterId, message );
}
}];

Is the Expression.* namespace the only way to create expression trees for EF 4.3 + ODAC?

I am having an issue with ODAC (Oracle Data Access Components), Entity Framework 4.3.1, and expression trees. We have a legacy database (don't we all?) that we are mapping in Entity Framework. The table has millions of records and over one hundred columns (sad face).
Here is an example query on an indexed column:
int myId = 2;
var matchingRecord = context.MyLargeTable.Where(v=>v.Id == myId).ToList(); //Super slow (5+ minutes, sometimes Out of Memory exception)
int myId = 2;
Expression<Func<bool>> myLambda = v => v.Id == myId; //Shouldn't this work now?
var matchingRecord = context.MyLargeTable.Where(myLambda).ToList(); //Still super slow (5+ minutes, sometimes Out of Memory exception)
var elementName = Expression.Parameter(typeof(LargeTable), "v");
var propertyName = Expression.Parameter(elementName, "Id");
var constantValue = Expression.Constant(myId);
var comparisonMethod = Expression.Call(
propertyName,
typeof(int).GetMethod("Equals", new[] { typeof(int) }),
constantValue
)
var finalTree = Expression.Lambda<Func<LargeTable, bool>>(comparisonMethod, elementName);
var matchingRecord = context.MyLargeTable.Where(finalTree).ToList(); //Super fast
I've read things like this that explain the different between Func<> and Expression> and how Expression> actually gets passed to the database for the query and that's why it is faster.
http://www.fascinatedwithsoftware.com/blog/post/2011/12/02/Falling-in-Love-with-LINQ-Part-7-Expressions-and-Funcs.aspx - Whole thing is good, but if in a rush, just read the section titled “Unintended Consequences” for the main takeaway
http://fascinatedwithsoftware.com/blog/post/2012/01/10/More-on-Expression-vs-Func-with-Entity-Framework.aspx
Why would you use Expression<Func<T>> rather than Func<T>? - No set of links is complete without a corresponding SO question
My question is this: Are people really sitting there constructing expression trees using Expression.* classes? Any query beyond simple comparisons get really complicated and is almost impossible to read. What am I missing about passing the Expression> to the database? Who do I go punch in the face for this manually constructed expression tree solution? Oracle? EF? What am I missing?

NSDictionary and plist

I have a plist file with string keys and number values. I think I managed to fill an NSDictionary object with the plist contents just fine with the code:
NSBundle* bun = [NSBundle mainBundle];
NSString* path = [bun pathForResource:#"Tempos" ofType:#"plist"];
tempos = [NSDictionary dictionaryWithContentsOfFile:path];
Now, here's the part I don't understand.
id object = [tempos objectForKey:#"Allegro"];
NSLog(#"bpm: %#", object);
Outputs the desired number, 168.
NSInteger beats = (NSInteger)[tempos objectForKey:#"Allegro"];
NSLog(#"bpm: %ld", beats);
Instead outputs, 43203.
More importantly, when I try
bpm = (NSInteger)[tempos objectForKey:#"Allegro"];
I get 43203 assigned to bpm. How do I get 168 assigned to bpm instead??
I think using this should work:
int beats = [[tempos objectForKey:#"Allegro"] intValue];
NSLog(#"bpm: %i", beats);

CoreData - NSPredicate formated with time range (NSTimeInterval)

I'm trying to find out how to go through my CoreData information and find objects that have a createdAt (part of my object as an NSDate) that is within a NSTimeInterval. How do I set this up?
I've looked on the documentation at:
http://developer.apple.com/documentation/Cocoa/Conceptual/Predicates/predicates.html
But I'm not finding anything there.
Do I need to create two time stamps and use SQL's BETWEEN?
Any help would be wonderful.
First of all, it doesn't make sense to check if an NSDate is within an NSTimeInterval, because NSTimeInterval just specifies a length of time, not its location. Instead, you want to use two separate NSDates specifying the beginning and end of your intervals.
Here's what it would look like (beginningTime and endTime are NSDates).
NSFetchRequest *request = [[NSFetchRequest alloc] init];
request.entity = [NSEntityDescription entityForName:#"YourEntityName" inManagedObjectContext:yourContext];
NSPredicate *beginningPredicate = [NSPredicate predicateWithFormat:#"createdAt >= %#", beginningTime];
NSPredicate *endPredicate = [NSPredicate predicateWithFormat:#"createdAt <= %#", endTime];
request.predicate = [NSCompoundPredicate andPredicateWithSubpredicates:[NSArray arrayWithObjects:beginningPredicate, endPredicate, nil]];
NSArray *results = [yourContext executeFetchRequest:request error:NULL];

Resources