What is the difference between
void QSqlQuery::clear ()
and
void QSqlQuery::finish ()
Based on the documentation, I don't see what the diff is. What is the difference? I'd like to know specifically when to use one over the other.
EDIT - Some more elaboration and info from documentation.
clear()
-Clears the result set and releases any resources held by the query.
Sounds like finish() does the same...
-Sets the query state to inactive.
Finish does the same.
finish()
-Instruct the database driver that no more data will be fetched from this query until it is re-executed.
What does this mean specifically? What is the consequence of this?
-It may be helpful in order to free resources such as locks or cursors if you intend to re-use the query at a later time.
Doesn't clear do the same? Doesn't clear release locks, cursors, etc?
-Sets the query to inactive.
clear does the same I believe.
-Bound values retain their values.
What is the point of this?
Qt comes with source code, you can see what's the difference by simply looking into the qsqlquery.cpp file
So according to the source code:
clear - clears and resets the QSqlQuery object;
finish - resets the result member of the current query into inactive state;
hope this helps, regards
The language used to describe these functions is similar so it can definitely be a little confusing and I hope this explanation helps. Here's how I interpret and use these methods.
void QSqlQuery::finish ()
I think of this as a way of saying I'm done with the query I just requested (eg no more reading/iterating) but I still plan on using that QSqlQuery object to do more work. You're just releasing any memory/resources used to get the values from the previous query. This really only makes a big, noticeable difference when you're dealing with large datasets over and over again, but I view it as good practice to use none the less.
void QSqlQuery::clear ()
This is my way of saying that I'm done with the QSqlQuery object and want to guarantee that none of the resources/memory I was using gets left around while I'm disposing of the object. I rarely, if ever, use this as I've found that it's effectiveness can vary widely depending on the database you use and if you're using modern C++ features, it doesn't do a lot for you.
It's easier to understand the difference if you look at them as being written to solve a similar problem for two different time periods (eg old C code as opposed to modern C++).
They do very similar things but I'd recommend you just use finish().
For all, like me, that are wondering which method to invoke. I will share my research.
NOTE: I read the sources of the SQLite driver, so other databases drivers can be different.
finish() resets the statement; in SQLite context it calls sqlite3_reset;
clear() resets the whole QSqlQuery object; it clears bound values, prepared statement, lastError(), lastQuery() ..., sets the default options for all object's parameters; in SQLite context I think that sqlite3_finalize is also called;
So I should visualize it like that finish < clear. After finish() you could call exec() to reexecute the query, but after clear() you must prepere the query again and bind its values before you can successfully reexecute the query.
Related
I'll definitely need to update this based on feedback so I apologize in advance.
The problem I'm trying to solve is roughly this.
The graph shows Disk utilization in the Windows task manager. My sqlite application is a webserver that takes in json requests with timestamps, looks up the existing entry in a 2 column key/value table, merges the request into the existing item (they don't grow over time), and then writes it back to the database.
The db is created as follows. I've experimented with and without WAL without difference.
createStatement().use { it.executeUpdate("CREATE TABLE IF NOT EXISTS items ( key TEXT NOT NULL PRIMARY KEY, value BLOB );") }
The write/set is done as follows
try {
val insertStatement = "INSERT OR REPLACE INTO items (key, value) VALUES (?, ?)"
prepareStatement(insertStatement).use {
it.setBytes(1, keySerializer.serialize(key))
it.setBytes(2, valueSerializer.serialize(value))
it.executeUpdate()
}
commit()
} catch (t: Throwable) {
rollback()
throw t
}
I use a single database connection the entire time which seems to be ok for my use case and greatly improves performance relative to getting a new one for each operation.
val databaseUrl = "jdbc:sqlite:${System.getProperty("java.io.tmpdir")}/$name-map-v2.sqlite"
if (connection?.isClosed == true || connection == null) {
connection = DriverManager.getConnection(databaseUrl)
}
I'm effectively serializing access to the db. I'm pretty sure the default threading mode for the sqlite driver is to serialize and I'm also doing some serializing in kotlin coroutines (via actors).
I'm load testing the application locally and I notice that disk utilization spikes around the one minute mark but I can't determine why. I know that throughput plummets when that happens though. I expect the server to chug along at a more or less constant rate. The db in these tests is pretty small too, hardly reaches 1mb.
Hoping people can recommend some next steps or set me straight as far as performance expectations. I'm assuming there is some sqlite specific thing that happens when throughput is very high for too long, but I would have thought it would be related to WAL or something (which I'm not using).
I have a theory but it's a bit farfetched.
The fact that you hit a performance wall after some time makes me think that either a buffer somewhere is filling up, or some other kind of data accumulation threshold is being reached.
Where exactly the culprit is, I'm not sure.
So, I'd run the following tests.
// At the beginning
connection.setAutoCommit(true);
If the problem is in the driver side of the rollback transaction buffer, then this will slightly (hopefully) slow down operations, "spreading" the impact away from the one-minute mark. Instead of getting fast operations for 59 seconds and then some seconds of full stop, you get not so fast operations the whole time.
In case the problem is further down the line, try
PRAGMA JOURNAL_MODE=MEMORY
PRAGMA SYNCHRONOUS=OFF disables the rollback journal synchronization
(The data will be more at risk in case of a catastrophic powerdown).
Finally, another possibility is that the page translation buffer gets filled after a sufficient number of different keys has been entered. You can test this directly by doing these two tests:
1) pre-fill the database with all the keys in ascending order and a large request, then start updating the same many keys.
2) run the test with only very few keys.
If the slowdown does not occur in the above cases, then it's either TLB buffer management that's not up to the challenge, or database fragmentation is a problem.
It might be the case that issuing
PRAGMA PAGE_SIZE=32768
upon database creation might solve or mitigate the problem. Conversely, PRAGMA PAGE_SIZE=1024 could "spread" the problem avoiding performance bottlenecks.
Another thing to try is closing the database connection and reopening it when it gets older than, say, 30 seconds. If this works, we'll still need to understand why it works (in this case I expect the JDBC driver to be at fault).
First of all, I want to say that I do not use exactly your driver for sqlite, and I use different devices in my work. (but how different are they really?)
From what I see, correct me if im wrong, you use one transaction, for one insert statement. You get request, you use the disc, you use the memory, open, close etc... every time. This can't work fast.
The first thing I do when I have to do inserts in sqlite is to group them, and use a single transaction to do it. That way, you are using your resources in batches.
One transaction, many insert statements, single commit. If there is a problem with a batch, handle the valid separately, log the faulty, move the next batch of requests.
I understand that sessionless operations are the preferred method of using gremlin. I'm wondering when is the sessioned approach better?
So I might be doing something like...
graph.addVertex("foo").property("name","bar")
graph.traversal().V().has("name","bar").as("f").addV("foo").property("name","baz").as("g").addE("test").from("f").to("g")
I'm doing this type of operation a lot. Often there's also a query (usually involving a coalesce) beforehand to check if a node (g in my example) exists, and create it if not.
So I'm wondering if a session might be better because I could hold a handle to the previous vertices and just attach new nodes to them without the expense of the lookup.
Feel free to tell me why I'm wrong in anything else that I'm doing.. Just trying to make things faster.
First of all, I would avoid use of addVertex() and stick to addV() - see more details here.
As to your question, I think the only time to leverage sessions is if you have some sort of loading operation that requires explicit control over transactions and you're not using a JVM based language. Even then, I might consider other options for dealing with that and just avoid sessions completely. You end up with a less portable solution as there are a number of graph systems which don't even support them directly (e.g. Neptune).
The cost to do a T.id based lookup should be really fast, so saving a vertex between requests in a session really shouldn't vastly improve performance. Even if you keep the vertex between requests you will still need to pass the vertex into your traversal so you still have the lookup anyway - I'm not sure I see the difference in cost there.
// first request
v = g.addV(...).property(...).next()
// second request
g.V(v).addE(....
// third request
g.V(v).addE(....
The above should not be that much faster than:
// first request - returns id=1
g.addV(...).property(...).id().next()
// second request - where "1" is just passed in on the next request as a parameter
g.V(1).addE(....
// third request
g.V(1).addE(....
I have a program that reads 173 (c) data structures from a memory map that need to be converted to Go. The value of the type is stored as a string in those structures. The structures are received 60 times per second.
I'm now using reflection (FieldByName) to get a reference to the go struct field and set the received data. But because there a many fields (173) and they get updated a lot this ads a lot of overhead and that function call is the slowest part of my program (jay go prof!).
What is the best way to make this faster? As far as I can see I have three options:
cache the reflect.Value's in a map and make a function that receives data, use a template struct tied to the cache map, fill that struct and return a copy of that template-struct
go generate all the setters and a giant switch statement for each received field
Just code all the different setters
What would be the "best" option? Is there an option I'm overlooking?
With #1, to be concurrency-safe you'd need a pool of those "template-struct" or at least a mutex protecting it. That adds some overhead and can be tricky to debug.
#3 is a nightmare to maintain.
I would go with #2. The running code will be fast, concurrency-safe and easy to debug.
Once your tool is setup, a change in your struct only requires running a command line to update the setters.
for a pseudo function like
void transaction(Account from, Account to, double amount){
Semaphore lock1, lock2;
lock1 = getLock(from);
lock2 = getLock(to)
wait(lock1);
wait(lock2);
withdraw(from, amount);
deposit(to, amount);
signal(lock2);
signal(lock1);
}
deadlock happens if you run transaction(A,B,50) transaction(B,A,10)
how can this be prevented?
would this work?
A simple deadlock prevention strategy when handling locks is to have strict order on the locks in the application and always grab the locks according to this order. Assuming all accounts have a number, you could change your logic to always grab the lock for the account with the lowest account number first. Then grab the lock for the one with the highest number.
Another strategy for preventing deadlocks is to reduce the number of locks. In this case it might be better to have one lock that locks all accounts. It would definitely make the lock structure far more simple. If the application shows performance problems under heavy load and profiling shows that lock congestion is the problem - then it is time to invent a more fine grained locking strategy.
By making the entire transaction a critical section? That's only one possible solution, at least.
I have a feeling this is homework of some sort, because it's very similar to the dining philosophers problem based on the example code you give. (Multiple solutions to the problem are available at the link provided, just so you know. Check them out if you want a better understanding of the concepts.)
I have the feeling that I should not care about thread safe accessing / writing to an
public static int MyVar = 12;
in ASP .NET.
I read/write to this variable from various user threads. Let's suppose this variable will store the numbers of clicks on a certain button/link.
My theory is that no thread can read/write to this variable at the same time. It's just a simple variable of 4 bytes.
I do care about thread safe, but only for refference objects and List instances or other types that take more cycles to read/update.
I am wrong with my presumption ?
EDIT
I understand this depend of my scenario, but wasn't that the point of the question. The question is: it is right that can be written thread safe code with an (static int) variable without using lock keyword ?
It is my problem to write correct code. The answer seems to be: Yes, if you write correct and simple code, and not to much complicated, you can create thread safe functions without the need of lock keyword.
If one thread simply sets the value and another thread reads the value, then a lock is not necessary; the read and write are atomic. But if multiple threads might be updating it and are also reading it to do the update (e.g., increment), then you definitely do need some kind of synchronization. If only one thread is ever going to update it even for an increment, then I would argue that no synchronization is necessary.
Edit (three years later) It might also be desirable to add the volatile keyword to the declaration to ensure that reads of the value always get the latest value (assuming that matters in the application).
The concept of thread 'safety' is too vague to be meaningful unfortunately. If you're asking whether you can read and write to it from multiple threads without the program crashing during the operation, the answer is almost certainly yes. If you're also asking if the variable is guaranteed to either be the old value or the new value without ever storing any broken intermediate values, the answer for this data type is again almost certainly yes.
But if your question is "will my program work correctly if I access this from multiple threads", then the answer depends entirely on what your program is doing. For example, if you run the following pseudo code in 2 threads repeatedly in most programming languages, eventually you'll hit the assertion.
if MyVar >= 1:
MyVar = MyVar - 1
assert MyVar >= 0
Primitives like int are thread-safe in the sense that reads/writes are atomic. But as with most any type, it's left to you to do proper checking with more complex operations. For example, if (x > 0) x--; would be problematic in a multi-threaded scenario because x might change in between the if condition check and decrement.
A simple read or write on a field of 32 bits or less is always atomic. But you should provide your read/write code to make sure that it is thread safe.
Check out this post: http://msdn.microsoft.com/en-us/magazine/cc163929.aspx
It explains why you need to synchronize access to the integers in this scenario
Try Interlocked.Increment() or Interlocked.Add() and you'll be right. Your code complexity will be the same but you truly won't have to worry. If you're not worried about losing a few clicks in your counter, you can continue as you are.
Reading or writing integers is atomic. However, reading and then writing is not atomic. So, if you have one thread that writes and many that read, you may be able to get away without locks.
However, even though the operations are atomic, there are still potential multi-threading issues. In order for one thread to be guaranteed that another thread can see values it writes, you need a memory barrier. Otherwise, the compiler can optimize the code so that the variable stays in a register (or even optimize the operation away completely), so changes would be invisible from one thread to another.
You can establish a memory barrier explicitly (volatile or Thread.MemoryBarrier), or with the Interlocked class -- or with the lock statement (Monitor).