I am developping and Android application and want to find the slowest/more expansive sqlite queries running on my device to fine tune them.
Which query can I use and how can I run it on my device database ?
There is no built-in mechanism that already times your queries.
You have to add your own logging:
long t1 = System.currentTimeMillis();
Cursor c = db.query(...);
try {
while (c.moveToNext()) {
...
}
} finally {
c.close();
}
long t2 = System.currentTimeMillis();
Log.d(TAG, "query time: " + (t2 - t1));
And please note that SQLite computes result rows on demand, so you cannot easily isolate the query execution time from the time that your own code needs to handle the result rows.
Related
I'm using System.Data.SQLite ADO.NET provider for SQLite and the following Powershell code to execute queries (and nonqueries) against a Sqlite3 DB:
Function Invoke-SQLite ($DBFile,$Query) {
try {
Add-Type -Path ".\System.Data.SQLite.dll"
}
catch {
write-warning "Unable to load System.Data.SQLite.dll"
return
}
if (!$DBFile) {
throw "DB Not Found" R
Sleep 5
Exit
}
$conn = New-Object System.Data.SQLite.SQLiteConnection
$conn.ConnectionString="Data Source={0}" -f $DBFile
$conn.Open()
$cmd = $Conn.CreateCommand()
$cmd.CommandText = $Query
#$cmd.CommandTimeout = 10
$ds = New-Object system.Data.DataSet
$da = New-Object System.Data.SQLite.SQLiteDataAdapter($cmd)
[void]$da.fill($ds)
$cmd.Dispose()
$conn.Close()
write-host ("{0} Row(s) returned " -f ($ds.Tables[0].Rows|Measure-Object|Select -ExpandProperty Count))
return $ds.Tables[0]
}
The problem is: while it is trivial to know how many rows have been SELECTed in a query operation, the same is not true if the operation is an INSERT,DELETE or UPDATE (nonqueries)
I know I could use the ExecuteNonQuery method, but i need a generic wrapper which returns number of affected rows while being agnostic about the query it executed (as Invoke-SQLCmd would do, for example)
Is that possible?
Thanks!
A few comments before the answer:
System.data.Sqlite supports executing multiple SQL statements for one command, as long as the CommandText has each valid statements delimited by a semicolon (;). This means that there could be a mixture of queries and DML statements (i.e. INSERT, UPDATE, DELETE). The fact that you do not want to distinguish between the type of statement in $Query tells me that you are likely just passing statements blindly, so it could contain any combination of statements. Simply getting only one value (whether from a query or DML) seems too limiting.
Using a DataAdapter to fill a dataset just to get counts is inefficient. Instead, it may be better to just get a DataReader object and count the returned rows. This also allows a separate count for each query statement to be retrieved, something that gets obscured by using the DataAdapter object. (Perhaps enumerating all tables in the resultant dataset could get the same number, but I'm not certain that would always be equivalent.)
One good thing is that if you insist on using a DataAdapter, it will still execute DML statements (even though the expected result is query that returns rows). The dataset will not be changed (filled), but all statements in the command text will still affect changes in the database, so the following solution will still be useful.
Even if the code had works, I assume that the line which prints "{0} Rows returned" is meant to get a simple count, but $ds.Tables[0].Rows needs to be $ds.Tables[0].Rows.Count.
Notes about this particular solution:
The key is to call either of the sqlite SQL functions changes() or total_changes(). These can be retrieved using SQL: SELECT total_changes();. I recommend getting total_changes() before and after a command, then subtracting the difference. That will get changes for multiple statements executed by one command.
I'm not a PowerShell guru, so I tested everything in C#. Treat the code below more as pseudo code since it may need tweaking.
The code:
$conn = New-Object System.Data.SQLite.SQLiteConnection
try {
$conn.ConnectionString="Data Source={0}" -f $DBFile
$conn.Open()
$cmdCount = $Conn.CreateCommand()
$cmd = $Conn.CreateCommand()
try {
$cmdCount.CommandText = "SELECT total_changes();"
$beforeChanges = $cmdcount.ExecuteScalar()
$cmd.CommandText = $Query
$ds = New-Object System.Data.DataSet
$da = New-Object System.Data.SQLite.SQLiteDataAdapter($cmd)
$rows = 0
try {
[void]$da.fill($ds)
foreach ($tbl in $ds.Tables) {
$rows += $tbl.Rows.Count;
}
} catch {}
$afterChanges = $cmdcount.ExecuteScalar()
$DMLchanges = $afterChanges - $beforeChanges
$totalRowAndChanges = $rows + $DMLchanges
# $ds.Tables[0] may or may not be valid here.
# If query returned no data, no tables will exist.
} finally {
$cmdCount.Dispose()
$cmd.Dispose()
}
} finally {
$conn.Dispose()
}
Alternatively, you could eliminate the DataAdapter:
$cmd.CommandText = $Query
$rdr = $cmd.ExecuteReader()
$rows = 0
do {
while ($rdr.Read()) {
$rows++
}
} while ($rdr.NextResult())
$rdr.Close();
I have a question regarding on how to retrieve the records that I have selected in a form, to a report.
Currently, I am able to select multiple records, but when it comes to the report, it keep on processing the same value. However the number of the records that it processed is correct, only the value is repeating.
I am not sure on how to fix this, therefore your help is kindly appreciated.
Below is the part that i get the record:
if (element.args() && element.args().dataset())
{
switch(args.dataset())
{
case tablenum(LedgerJournalTrans) :
ledgerJournalTrans = element.args().record();
info(ledgerJournalTrans.Voucher);
break;
case tablenum(LedgerJournalTable) :
ledgerJournalTable = args.record();
break;
}
}
The element.args().record() only points to the last selected record. Its datasource comes to rescue. The usual approach to process multi-selected records applies:
Common record;
FormDataSource fds;
fds = element.args().record().dataSource();
for (record = fds.getFirst(1) ? fds.getFirst(1) : fds.cursor(); record; record = fds.getNext())
{
// Do the printing using record
}
You often see this approach used in main methods of functions capable of processing multi-selected records.
The FormLetter.getFormRecord uses this pattern as well.
According to this blog post, firebase array keys are created using a timestamp:
It does this by assigning a permanent, unique id based on the current timestamp (offset to match server time).
Is there a way to recover this timestamp for use later, given the key?
As I said in my comment, you should not rely on decoding the timestamp from the generated id. Instead of that, you should simply store it in a property in your Firebase.
That said, it turns out to be fairly easy to get the timestamp back:
// DO NOT USE THIS CODE IN PRODUCTION AS IT DEPENDS ON AN INTERNAL
// IMPLEMENTATION DETAIL OF FIREBASE
var PUSH_CHARS = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz";
function decode(id) {
id = id.substring(0,8);
var timestamp = 0;
for (var i=0; i < id.length; i++) {
var c = id.charAt(i);
timestamp = timestamp * 64 + PUSH_CHARS.indexOf(c);
}
return timestamp;
}
var key = prompt("Enter Firebase push ID");
if (key) {
var timestamp = decode(key);
console.log(timestamp+"\n"+new Date(timestamp));
alert(timestamp+"\n"+new Date(timestamp));
}
I'll repeat my comment, just in case somebody thinks it is a good idea to use this code for anything else than as an exercise in reverse engineering:
Even if you know how to retrieve the timestamp from the key, it would be a bad idea to do this in production code. The timestamp is used to generate a unique, chronologically ordered sequence. If somebody at Firebase figures out a more efficient way (whichever subjective definition of efficiency they happen to choose) to accomplish the same goal, they might change the algorithm for push. If your code needs a timestamp, you should add the timestamp to your data; not depend on it being part of your key.
Update
Firebase documented the algorithm behind Firebase push IDs. But the above advice remains: don't use this as an alternative to storing the date.
Here's a version of Frank's code re-written in Swift (4.2 at the time of writing.)
Just to be clear, my use case for this was to patch my old models with no timestamps (createdAt, updatedAt.) I could just throw in random dates in them just to save me some headaches. But then that wouldn't be relevant to their models. I knew that there's an element of time baked into these auto-ids based on what I've read from other articles.
let PUSH_CHARS = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"
func decode(autoId: String) -> TimeInterval {
let substring = autoId.substring(toIndex: 8)
var timestamp = 0
for i in 0..<substring.length {
let c = Character(substring[i])
timestamp = (timestamp * 64) + PUSH_CHARS.firstIndex(of: c)!.encodedOffset
}
return TimeInterval(exactly: timestamp)!
}
Grab the Playground-ready code here: https://gist.github.com/mkval/501c03cbb66cef12728ed1a19f8713f7.
And in python
PUSH_CHARS = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"
def get_timestamp_from_id(id):
timestr = id[0:8]
timestamp = 0
for idx, ch in enumerate(timestr):
timestamp = timestamp * 64 + PUSH_CHARS.index(ch)
return timestamp/1000
I'm adding sqlite support to a my Google Chrome extension, to store historical data.
When creating the database, it is required to set the maximum size (I used 5MB, as suggested in many examples)
I'd like to know how much memory I'm really using (for example after adding 1000 records), to have an idea of when the 5MB limit will be reached, and act accordingly.
The Chrome console doesn't reveal such figures.
Thanks.
You can calculate those figures if you wanted to. Basically, the default limit for localStorage and webStorage is 5MB where the name and values are saved as UTF16 therefore it is really half of that which is 2.5 MB in terms of stored characters. In webStorage, you can increase that by adding "unlimited_storage" within the manifest.
Same thing would apply in WebStorage, but you have to go through all tables and figure out how many characters there is per row.
In localStorage You can test that by doing a population script:
var row = 0;
localStorage.clear();
var populator = function () {
localStorage[row] = '';
var x = '';
for (var i = 0; i < (1024 * 100); i++) {
x += 'A';
}
localStorage[row] = x;
row++;
console.log('Populating row: ' + row);
populator();
}
populator();
The above should crash in row 25 for not enough space making it around 2.5MB. You can do the inverse and count how many characters per row and that determines how much space you have.
Another way to do this, is always adding a "payload" and checking the exception if it exists, if it does, then you know your out of space.
try {
localStorage['foo'] = 'SOME_DATA';
} catch (e) {
console.log('LIMIT REACHED! Do something else');
}
Internet Explorer did something called "remainingSpace", but that doesn't work in Chrome/Safari:
http://msdn.microsoft.com/en-us/library/cc197016(v=VS.85).aspx
I'd like to add a suggestion.
If it is a Chrome extension, why not make use of Web SQL storage or Indexed DB?
http://html5doctor.com/introducing-web-sql-databases/
http://hacks.mozilla.org/2010/06/comparing-indexeddb-and-webdatabase/
Source: http://caniuse.com/
We'll soon be embarking on the development of a new mobile application. This particular app will be used for heavy searching of text based fields. Any suggestions from the group at large for what sort of database engine is best suited to allowing these types of searches on a mobile platform?
Specifics include Windows Mobile 6 and we'll be using the .Net CF. Also some of the text based fields will be anywhere between 35 and 500 characters. The device will operate in two different methods, batch and WiFi. Of course for WiFi we can just submit requests to a full blown DB engine and just fetch results back. This question centres around the "batch" version which will house a database loaded with information on the devices flash/removable storage card.
At any rate, I know SQLCE has some basic indexing but you don't get into the real fancy "full text" style indexes until you've got the full blown version which of course isn't available on a mobile platform.
An example of what the data would look like:
"apron carpenter adjustable leather container pocket waist hardware belt" etc. etc.
I haven't gotten into the evaluation of any other specific options yet as I figure I'd leverage the experience of this group in order to first point me down some specific avenues.
Any suggestions/tips?
Just recently I had the same issue. Here is what I did:
I created a class to hold just an id and the text for each object (in my case I called it a sku (item number) and a description). This creates a smaller object that uses less memory since it is only used for searching. I'll still grab the full-blown objects from the database after I find matches.
public class SmallItem
{
private int _sku;
public int Sku
{
get { return _sku; }
set { _sku = value; }
}
// Size of max description size + 1 for null terminator.
private char[] _description = new char[36];
public char[] Description
{
get { return _description; }
set { _description = value; }
}
public SmallItem()
{
}
}
After this class is created, you can then create an array (I actually used a List in my case) of these objects and use it for searching throughout your application. The initialization of this list takes a bit of time, but you only need to worry about this at start up. Basically just run a query on your database and grab the data you need to create this list.
Once you have a list, you can quickly go through it searching for any words you want. Since it's a contains, it must also find words within words (e.g. drill would return drill, drillbit, drills etc.). To do this, we wrote a home-grown, unmanaged c# contains function. It takes in a string array of words (so you can search for more than one word... we use it for "AND" searches... the description must contain all words passed in... "OR" is not currently supported in this example). As it searches through the list of words it builds a list of IDs, which are then passed back to the calling function. Once you have a list of IDs, you can easily run a fast query in your database to return the full-blown objects based on a fast indexed ID number. I should mention that we also limit the maximum number of results returned. This could be taken out. It's just handy if someone types in something like "e" as their search term. That's going to return a lot of results.
Here's the example of custom Contains function:
public static int[] Contains(string[] descriptionTerms, int maxResults, List<SmallItem> itemList)
{
// Don't allow more than the maximum allowable results constant.
int[] matchingSkus = new int[maxResults];
// Indexes and counters.
int matchNumber = 0;
int currentWord = 0;
int totalWords = descriptionTerms.Count() - 1; // - 1 because it will be used with 0 based array indexes
bool matchedWord;
try
{
/* Character array of character arrays. Each array is a word we want to match.
* We need the + 1 because totalWords had - 1 (We are setting a size/length here,
* so it is not 0 based... we used - 1 on totalWords because it is used for 0
* based index referencing.)
* */
char[][] allWordsToMatch = new char[totalWords + 1][];
// Character array to hold the current word to match.
char[] wordToMatch = new char[36]; // Max allowable word size + null terminator... I just picked 36 to be consistent with max description size.
// Loop through the original string array or words to match and create the character arrays.
for (currentWord = 0; currentWord <= totalWords; currentWord++)
{
char[] desc = new char[descriptionTerms[currentWord].Length + 1];
Array.Copy(descriptionTerms[currentWord].ToUpper().ToCharArray(), desc, descriptionTerms[currentWord].Length);
allWordsToMatch[currentWord] = desc;
}
// Offsets for description and filter(word to match) pointers.
int descriptionOffset = 0, filterOffset = 0;
// Loop through the list of items trying to find matching words.
foreach (SmallItem i in itemList)
{
// If we have reached our maximum allowable matches, we should stop searching and just return the results.
if (matchNumber == maxResults)
break;
// Loop through the "words to match" filter list.
for (currentWord = 0; currentWord <= totalWords; currentWord++)
{
// Reset our match flag and current word to match.
matchedWord = false;
wordToMatch = allWordsToMatch[currentWord];
// Delving into unmanaged code for SCREAMING performance ;)
unsafe
{
// Pointer to the description of the current item on the list (starting at first char).
fixed (char* pdesc = &i.Description[0])
{
// Pointer to the current word we are trying to match (starting at first char).
fixed (char* pfilter = &wordToMatch[0])
{
// Reset the description offset.
descriptionOffset = 0;
// Continue our search on the current word until we hit a null terminator for the char array.
while (*(pdesc + descriptionOffset) != '\0')
{
// We've matched the first character of the word we're trying to match.
if (*(pdesc + descriptionOffset) == *pfilter)
{
// Reset the filter offset.
filterOffset = 0;
/* Keep moving the offsets together while we have consecutive character matches. Once we hit a non-match
* or a null terminator, we need to jump out of this loop.
* */
while (*(pfilter + filterOffset) != '\0' && *(pfilter + filterOffset) == *(pdesc + descriptionOffset))
{
// Increase the offsets together to the next character.
++filterOffset;
++descriptionOffset;
}
// We hit matches all the way to the null terminator. The entire word was a match.
if (*(pfilter + filterOffset) == '\0')
{
// If our current word matched is the last word on the match list, we have matched all words.
if (currentWord == totalWords)
{
// Add the sku as a match.
matchingSkus[matchNumber] = i.Sku.ToString();
matchNumber++;
/* Break out of this item description. We have matched all needed words and can move to
* the next item.
* */
break;
}
/* We've matched a word, but still have more words left in our list of words to match.
* Set our match flag to true, which will mean we continue continue to search for the
* next word on the list.
* */
matchedWord = true;
}
}
// No match on the current character. Move to next one.
descriptionOffset++;
}
/* The current word had no match, so no sense in looking for the rest of the words. Break to the
* next item description.
* */
if (!matchedWord)
break;
}
}
}
}
};
// We have our list of matching skus. We'll resize the array and pass it back.
Array.Resize(ref matchingSkus, matchNumber);
return matchingSkus;
}
catch (Exception ex)
{
// Handle the exception
}
}
Once you have the list of matching skus, you can iterate through the array and build a query command that only returns the matching skus.
For an idea of performance, here's what we have found (doing the following steps):
Search ~171,000 items
Create list of all matching items
Query the database, returning only the matching items
Build full-blown items (similar to SmallItem class, but a lot more fields)
Populate a datagrid with the full-blow item objects.
On our mobile units, the entire process takes 2-4 seconds (takes 2 if we hit our match limit before we have searched all items... takes 4 seconds if we have to scan every item).
I've also tried doing this without unmanaged code and using String.IndexOf (and tried String.Contains... had same performance as IndexOf as it should). That way was much slower... about 25 seconds.
I've also tried using a StreamReader and a file containing lines of [Sku Number]|[Description]. The code was similar to the unmanaged code example. This way took about 15 seconds for an entire scan. Not too bad for speed, but not great. The file and StreamReader method has one advantage over the way I showed you though. The file can be created ahead of time. The way I showed you requires the memory and the initial time to load the List when the application starts up. For our 171,000 items, this takes about 2 minutes. If you can afford to wait for that initial load each time the app starts up (which can be done on a separate thread of course), then searching this way is the fastest way (that I've found at least).
Hope that helps.
PS - Thanks to Dolch for helping with some of the unmanaged code.
You could try Lucene.Net. I'm not sure how well it's suited to mobile devices, but it is billed as a "high-performance, full-featured text search engine library".
http://incubator.apache.org/lucene.net/
http://lucene.apache.org/java/docs/