I use in a Windows 8 project (js/html) the SQLite3-WinRT library https://github.com/doo/SQLite3-WinRT.
I create a function that is called in a for loop.
I have this error:
SQLiteError: 0x800700aa: eachAsync("INSERT INTO home (id, url, cksum)VALUES (16, 'main_page_2.jpg', 'e0d046ca3421a3c2df328b293ad5981a');", ) database is locked
I think the error is because I create a new connection every iteration of loop, but I don't understand another method. Who can help me?
This is the function:
function insertInDB(dbPath, tbName, arrayCol, arrayVal) {
SQLite3JS.openAsync(dbPath).then(function (db) {
var query = "INSERT INTO " + tbName;
var column = " (";
var values = "VALUES (";
for (var i = 0; i < arrayCol.length; i++) {
if (i == arrayCol.length - 1) {
column = column + arrayCol[i] + ")";
} else {
column = column + arrayCol[i] + ", ";
}
}
for (var i = 0; i < arrayVal.length; i++) {
if (i == arrayCol.length - 1) {
values = values + arrayVal[i] + ");";
} else {
values = values + arrayVal[i] + ", ";
}
}
query = query + column + values;
return db.eachAsync(query).done(function () {
console.log("Ok");
db.close();
},
function (error) { console.log(error); },
function (progress) { });
});
}
and this is the loop that call a previous function:
listHome.forEach(function(value, index, array){
var valconfig = new Array(value.id, "'" + value.url + "'", "'" + value.cksum + "'");
console.log("id=" + value.id + " url=" + value.url + " ck=" + value.cksum);
insertInDB(sqlPath, "home", colconfig, valconfig);
})
If I'm reading this correctly, your calling code is iterating over a list of values synchronously. listHome.forEach will call insertInDB for each item in listHome ... but it doesn't wait for insertInDB to return before making the next call to insertInDB.
Inside insertInDB you have call to SQLite3JS.openAsync and db.eachAsync - both asynchronous methods. After perusing SQLite3JS a little bit (which looks pretty cool), both of those methods return promises, where internally they call into a WinRT component. Great design.
So this is what I suspect is happening: one of the asynchronous calls in insertInDB puts a lock on the database. However, insertInDB returns control back to the listHome.forEach loop as soon as it hits the first asynchronous method call. If the lock on the database remains once forEach gets to the next item in listHome, then the operation will attempt to write to a locked database. Hence the error.
I'll think about this a little bit and see if I can come up with a solution.
-- edit --
Okay, I have a solution that might work for you. You might want to create a "DataBaseHelper" class that will queue up the transactions that you need to make in the database.
Here's a rough prototype that I threw together:
[Replaces your foreach loop]
DBHelper.queueUpdates(listHome);
[DBHelper module definition]
(function () {
var _queue;
function queueUpdates(array) {
_queue = array;
scheduleUpdates();
}
function scheduleUpdates() {
if (_queue.length > 0) {
var transaction = _queue.pop();
insertInDB("path", "table", "column", transaction);
}
}
function insertInDB(dbPath, tbName, arrayCol, arrayVal) {
return SQLite3JS.openAsync(dbPath).then(function (db) {
// Construct your SQL query ...
return db.eachAsync(query).done(function () {
db.close();
scheduleUpdates();
},
function (error) { console.log(error); },
function (progress) { });
});
}
WinJS.Namespace.define("DBHelper", {
queueUpdates: queueUpdates
})
})();
Related
I hope that someone could please help me. I have setup a simple test to test query performance for a basic graph. I have included the code below. Basically what I do is create 30000 vertices and then create 20000 edges on one of the vertices.
public async Task<bool> runTests(Models.Person person)
{
try
{
var gremlinServer = new GremlinServer(_hostname, _port, enableSsl: true,
username: "/dbs/" + _dataBaseId + "/colls/" + _collectionId,
password: _authKey);
using (var gremlinClient = new GremlinClient(gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(), GremlinClient.GraphSON2MimeType))
{
await gremlinClient.SubmitAsync<dynamic>("g.E().drop()");
await gremlinClient.SubmitAsync<dynamic>("g.V().drop()");
var counter = 30000;
while (counter != 0)
{
await gremlinClient.SubmitAsync<dynamic>("g.addV('partition" + counter + "').property('id', 'partition" + counter + "').property('profilePicture', '" + person.ProfilePicture + "').property('name', 'Person " + counter + "').property('partitionKey', 'partition" + counter + "')");
counter--;
}
var counter = 20000;
while (counter != 0)
{
int num = counter + 1;
var personToLink = "partition" + num;
await gremlinClient.SubmitAsync<dynamic>("g.V('partition1').addE('friendsWith').to(g.V('partition" + num + "'))");
counter--;
}
var searchResults = await gremlinClient.SubmitAsync<dynamic>("g.V().hasId('partition1').out('friendsWith').order().by('name', incr).valueMap('name', 'profilePicture').range(0,2)");
return true;
}
}
catch (Exception ex)
{
throw ex;
}
}
When I run the following query results are returned quickly:
g.V().hasId('partition1').out('friendsWith').valueMap('name', 'profilePicture').range(0,2)
However, as soon as I add an order clause the query takes much, much longer. Over a minute to complete:
g.V().hasId('partition1').out('friendsWith').order().by('name', incr).valueMap('name', 'profilePicture').range(0,2)
Is there a way to index the graph to speed this type of query up?
I have another question as well, I have set the throughput to 5000 RU, however when I run a query that run very quickly I get the following:
Data Explorer Query Stats Result
What is this value supposed to represent (RU's?) if so why is it so high?
Also when I try to run a simple query:
g.V().hasId('partition1').out('friendsWith').hasId('partition20001')
I get "Request rate is large" even though this is such a simple query. Even more concerning is when I increase the throughput to 5000 RU's I do get a result back but its really slow, takes about 5-6 seconds for what should be a really simple query.
Using cassandra with meteor.
let client = new cassandra.Client({contactPoints: [cassandraHost]});
var cassandraExecSync = Meteor.wrapAsync(client.execute, client);
MyProject.Feed.CassandraMeteorWrap = {
insertNewPost: function (userId, postContentJson, relevance) {
var insertCommand = insert(userId, postContentJson, relevance);
try {
return cassandraExecSync(insertCommand);
} catch (err) {
console.log("error inserting: " + insertCommand);
console.log(err);
}
}
So I wrapped Cassandra.client.execute (which has last arg as callback) with Meteor.wrapAsync
The first few inserts work, but after a few inserts (the insert is being called periodically) I get a:
[Error: Meteor code must always run within a Fiber. Try wrapping callbacks that you pass to non-Meteor libraries with Meteor.bindEnvironment.]
Update: Debug meteor showed the stack trace and the exception starts from the npm package I use "cassandra-driver" on the .onTimeout():
function listOnTimeout() {
var msecs = this.msecs;
var list = this;
debug('timeout callback ' + msecs);
var now = Timer.now();
debug('now: %d', now);
var first;
while (first = L.peek(list)) {
// If the previous iteration caused a timer to be added,
// update the value of "now" so that timing computations are
// done correctly. See test/simple/test-timers-blocking-callback.js
// for more information.
if (now < first._monotonicStartTime) {
now = Timer.now();
debug('now: %d', now);
}
var diff = now - first._monotonicStartTime;
if (diff < msecs) {
list.start(msecs - diff, 0);
debug(msecs + ' list wait because diff is ' + diff);
return;
} else {
L.remove(first);
assert(first !== L.peek(list));
if (!first._onTimeout) continue;
// v0.4 compatibility: if the timer callback throws and the
// domain or uncaughtException handler ignore the exception,
// other timers that expire on this tick should still run.
//
// https://github.com/joyent/node/issues/2631
var domain = first.domain;
if (domain && domain._disposed) continue;
try {
if (domain)
domain.enter();
var threw = true;
**first._onTimeout();**
if (domain)
domain.exit();
threw = false;
} finally {
if (threw) {
// We need to continue processing after domain error handling
// is complete, but not by using whatever domain was left over
// when the timeout threw its exception.
var oldDomain = process.domain;
process.domain = null;
process.nextTick(function() {
list.ontimeout();
});
process.domain = oldDomain;
}
}
}
}
debug(msecs + ' list empty');
assert(L.isEmpty(list));
list.close();
delete lists[msecs];
}
I confess that I do not get along very well with the Deferred object. I'm making a query to the database on several "Stores" and as a result I want to do a series of operations. This troubles me because the results are returned asynchronously and I have no way to perform the corresponding operation on the "store" you should. In short, the problem is that this piece of code always executes the same function on the same "Store"
for (var i = 0; i < schema['stores'].length; i++) {
storeName = schema['stores'][i].name;
var objeto = db.executeSql('SELECT MAX(date_upd) FROM ' + '"' + storeName + '"').done(
function(result, a){
//saveDataSynce(db, storeName, result);
console.log(result);
}
);
}
Whenever there is a loop on async operation, be very careful about function scope. In your example code, storeName inside the function will always be the last executed value. Use function scope as follow:
var getMax = function(storeName) {
db.executeSql('SELECT MAX(date_upd) FROM ' + '"' + storeName + '"').done(
function(result){
//saveDataSynce(db, storeName, result);
console.log(storeName, result);
}
);
}
for (var i = 0; i < schema['stores'].length; i++) {
getMax(schema['stores'][i].name);
}
However, preferred coding pattern for YDN-DB is NoSQL style as follow:
var getMax = function(storeName) {
var indexName = 'date_upd';
var key_range = null; // whole store
var limit = 1;
var offset = 0;
var reverse = true;
db.values(storeName, indexName, key_range, limit, offset, reverse).done(
function(results) {
var max_key = results[0]; // may be undefined. OK.
//saveDataSynce(db, storeName, max_key);
console.log(storeName, max_key);
}
);
}
Note that keys (primary or index) are always sorted by ascending order. Max key is the first key in reverse order.
At the end of my submit button click handler, Resharper warns that, "Not all code paths return a value."
What value would it be expecting from an event handler?
In deference to full disclosure, this is that event handler:
$("#submit_button").click(function() {
// http://stackoverflow.com/questions/18192288/how-can-i-compare-date-time-values-using-the-jqueryui-datepicker-and-html5-time
var begD = $.datepicker.parseDate('mm/dd/yy', $('#BeginDate').val());
var endD = $.datepicker.parseDate('mm/dd/yy', $('#EndDate').val());
if (begD > endD) {
alert('Begin date must be before End date');
$('#BeginDate').focus();
return false;
}
else if (begD.toString() == endD.toString()) {
var dteString = begD.getFullYear() + "/" + (begD.getMonth() + 1) + "/" + begD.getDate();
var begT = new Date(dteString + " " + $('#BeginTime').val());
var endT = new Date(dteString + " " + $('#EndTime').val());
if (begT > endT) {
alert('Begin date must be before End date');
$('#BeginTime').focus();
return false;
}
}
$("#NumberOfResults").css("visibility", "visible");
$("#NumberOfResults").html("Please wait...");
EnableButton("submit_button", false);
// If all are selected, don't enumerate them; just set it at "All" (change of case shows that the logic did execute)
var deptsList = $('#depts').checkedBoxes();
if (deptsList.length < deptsArray.length) {
$('#deptHeader span').html(deptsList.join(", "));
}
else if (deptsList.length == deptsArray.length) {
$('#deptHeader span').html("All");
}
// " "
var sitesList = $('#sites').checkedBoxes();
$('#sitesHeader span').html(sitesList.join(", "));
if (sitesList.length < sitesArray.length) {
$('#sitesHeader span').html(sitesList.join(", "));
}
else if (sitesList.length == sitesArray.length) {
$('#sitesHeader span').html("All");
}
$('#hiddenDepts').val(deptsList);
$('#hiddenSites').val(sitesList);
var UPCs = $('#UPC').val();
if (UPCs == "All") {
$('#UPC').val("1"); // take everything (1 and greater)
}
var resultsText = jQuery.trim($("#spanNumberOfResults").text());
if (resultsText != "") {
$("#NumberOfResults").css("visibility", "visible");
if (resultsText == "0") {
$("#NumberOfResults").css("color", "red");
} else {
var href = '/#ConfigurationManager.AppSettings["ThisApp"]/CCRCriteria/LoadReport';
// report_parms (sic) is referenced from LoadReport
var report_parms = {
GUID: "#Model.GUID",
SerialNumber: "#Model.SerialNumber",
ReportName: "#Model.ReportName"
};
window.open(href, "report_window", "resizable=1, width=850, left=" + (screen.width / 2 - 425));
}
}
}); // end of submit button click
Resharper isn't aware of event handlers.
It sees that your function will sometimes return false and sometimes won't return anything, and it complains.
It doesn't realize that this pattern is perfectly fine for event handlers.
Ignore it. Click handlers "can" return a boolean value indicating whether to process the click normally (true) or ignore it (false).
Resharper sees any return in the function as a clue that it should always return something.
I am a little stuck and need some advice/help.
I have a progress bar:
<mx:ProgressBar id="appProgress" mode="manual" width="300" label="{appProgressMsg}" minimum="0" maximum="100"/>
I have two listener functions, one sets the progress, and one sets the appProgressMsg:
public function incProgress(e:TEvent):void {
var p:uint = Math.floor(e.data.number / e.data.total * 100);
trace("Setting Perc." + p);
appProgress.setProgress(p, 100);
}
public function setApplicationProgressStep(e:TEvent):void {
trace("Setting step:" + e.data);
appProgressMsg = e.data;
}
I want to reuse this progress bar alot. And not necessarily for ProgressEvents, but when going through steps.
For instance, I loop over a bunch of database inserts, and want to undate the progress etc.
Here is a sample:
public function updateDatabase(result:Object):void {
var total:int = 0;
var i:int = 0;
var r:SQLResult;
trace("updateDatabase called.");
for each (var table:XML in this.queries.elements("table")) {
var key:String = table.attribute("name");
if (result[key]) {
send(TEvent.UpdateApplicationProgressStep, "Updating " + key);
i = 1;
total = result[key].length;
for each (var row:Object in result[key]) {
//now, we need to see if we already have this record.
send(TEvent.UpdateApplicationProgress, { number:i, total: total } );
r = this.query("select * from " + key + " where server_id = '" + row.id + "'");
if (r.data == null) {
//there is no entry with this id, make one.
this.query(table.insert, row);
} else {
//it exists, so let's update.
this.update(key, row);
}
i++;
}
}
}
}
Everything works fine.
That is, the listener functions are called and I get trace output like:
updateDatabase called.
Setting step:Updating project
Setting Perc 25
Setting Perc 50
Setting Perc 75
Setting Perc 100
The issue is, only the very last percent and step is shown. that is, when it's all done, the progress bar jumps to 100% and shows the last step label.
Does anyone know why this is?
Thanks in advance for any help,
Jason
The new code, which works awesomely I might add:
public function updateDatabase(result:Object, eindex:int = 0, sindex:int = 0 ):void {
var total:int = 0;
var i:int = 0;
var j:int;
var r:SQLResult;
var table:XML;
var key:String;
var elems:XMLList = this.queries.elements("table");
var startTime:int = getTimer();
var row:Object;
for (i = eindex; i < elems.length(); i++) {
table = elems[i];
key = table.attribute("name");
if (!result[key])
continue;
total = result[key].length;
send(TEvent.UpdateApplicationProgressStep, "Updating " + key);
for (j = sindex; j < result[key].length; j++) {
if (getTimer() - startTime > 100) {
setTimeout(updateDatabase, 100, result, i, j);
send(TEvent.UpdateApplicationProgress, { number:j, total: total } );
return;
}
row = result[key][j];
r = this.query("select * from " + key + " where server_id = '" + row.id + "'");
if (r.data == null) {
//there is no entry with this id, make one.
this.query(table.insert, row,false);
} else {
//it exists, so let's update.
this.update(key, row,false);
}
}
send(TEvent.UpdateApplicationProgress, { number:1, total: 1 } );
}
}
Flash is single threaded. The display will not update until your function returns. For this reason, you should never have any code that runs for longer than about 100ms (1/10th of a second), otherwise the UI (or even the entire browser) will appear to be locked up.
The general solution is to split up your work over multiple frames, here is some pseudo-code:
function doWork(arg1:Obj, arg2:Obj, start:int=0) {
var startTime = getTimer(); // store starting time
for(i=start; i<length; i++) {
if(getTimer() - startTime > 100) { // see if we've been working too long
trace("Current progress: "+(i/length * 100)+"%");
updateProgress( i / length );
setTimeout(doWork, 100, arg1, arg2, i); // schedule more work to be done
return; // stop current loop
}
trace("Working on item "+i);
// processing here
}
trace("All work done");
}
doWork(data1, data2); // start the work
Your pseudo-code works for updating the progress bar however in my case my "work" was copying of files from DVD to the appStorageDirectory which seem to reintroduce the same issue that your work around resolved - the progress bar now does not update
Here is my hack of your solution
function doWork(arg1:int, arg2:int, start:int=0) {
var startTime = getTimer(); // store starting time
for(var i:int=start; i<arg2; i++) {
if(getTimer() - startTime > 100 ) { // see if we've been working too long
trace("Current progress: "+(i/arg2 * 100)+"%");
setTimeout(doWork, 100, i, arg2, i); // schedule more work to be done
return; // stop current loop
}
trace("Working on item "+i);
dispatchEvent(new progressMadeEvent("incrementChange",i,arg2))
var dir:String = copyRes[i].nativePath.toString().split(OSSep).pop()
copyRes[i].copyTo(appStore.resolvePath(dir)) // copies dir from DVD to appStorageDir
}
trace("All work done");
}