In RocksDB writes are stored in an in-memory buffer namely memtable. The memtable gets flushed into disk (SSTable sorted static table) once it gets full. Is it possible to programmatically enforce flushing the memtable to disk before its gets full? What Rocksdb APIs does it if any?
Thank you.
DB::Flush() is the function to use: https://github.com/facebook/rocksdb/blob/6.23.fb/include/rocksdb/db.h#L1264-L1282
Related
My app implements a web cache using Core Data with an SQLite store (a bit like NSURLCache but with more control over what is cached and when it can be purged from the cache). I can purge old data from the cache when it gets too big, but of course this does not actually reduce the amount of storage space used in the file system.
I know that NSPersistentStoreCoordinator can be passed the NSSQLiteManualVacuumOption option to do an SQLite vacuum on opening the store, and I thought this might be a good way to reduce the file size (even if I have to temporarily close and re-open the store).
However, I am managing the Core Data stack through an NSPersistentContainer. Is there any way to do an Apple-approved SQLite vacuum with an NSPersistentContainer?
Or do I have to establish the Core Data stack manually to do this?
Or otherwise how else should I reclaim SQLite storage space when using an NSPersistentConainer?
CORRECTION:
I was partly wrong about the space not being reclaimed automatically. Even without setting NSSQLiteManualVacuumOption, the disk space is being reclaimed automatically - sometimes. I'm not sure how this is working. It doesn't happen immediately and it doesn't happen every time I save the context, but when it does happen, it is very quickly after deleting managed objects from the context and saving the context.
If I do a series of delete/save commands, the storage space is not freed in between them. But if I check again shortly afterwards, the space has been reclaimed (sometimes).
Perhaps Core Data is using the 'auto_vacuum' SQLite pragma by default? Although if I log the sqlite pragmas, it just prints an empty dictionary.
I think it should work like this; I've never done it myself:
Create an NSPersistentStoreDescription and set the options to include the NSSQLiteManualVacuumOption. Assign this description to the the NSPersistentContainer's persistentStoreDescriptions property before loading the store.
"Bitcask keeps all keys in memory at all times, this means that your system must have enough memory to contain your entire keyspace"
I read this in the description of Bitcask, and I'm worried the same issue may affect Riak, as Riak is implemented on Bitcask. Is this the case?
If you use the Bitcask backend, yes.
Bitcask is the defualt backend, but should you expect your list of keys to exceed your server's memory, you can use the eleveldb backend instead.
In all optimization guides people talk about pragma's like JOURNAL_MODE or SYNCHRONOUS, but I never read anything about the TEMP_STORE pragma? I would expect it has a large impact, so why is it never mentioned?
It's purpose is to move all SQLite's internal temporary tables from disk (temp directory) to memory, which seems a lot faster than hitting the disk on every SELECT?
SQLite locks the entire database when doing writes, so I would imagine that it is better to get the data onto the platters before proceeding with your next task.
Putting the data in memory is most likely reserved for those occasions when you would only want a temporary data store (as the TEMP_STORE name implies); you would still need to provide a method for periodically flushing the data to disk (if you want to save it), and since the locking is not granular, you would have to flush the entire database.
In other words, TEMP_STORE is not a caching mechanism.
Where does cached data is stored in Asp.Net application (heap or ram)
Eoin is absolutely correct! Caching only means storing the data from your secondary memory or hard disk (database, files, etc) into the primary memory or application memory. It speeds up the execution because reading from app mem is faster than reading from disks. So if say a file is stored in Cache, you can read it quicker than if it wasn't and you had to read it from disk.
For more details on caching in asp.net visit this link
General idea on cache memory can be found here
Well (and this is completely simplified)
Classes (i.e. Reference Types) are stored on the heap, with a pointer to that reference type stored on that stack.
Structs/Simple Types (i.e. Value Types) are stored directly on the stack.
But with regards caching, the idea is that the value you are storing is stored in Application Memory full stop.
The benefit would be that if you have some value that you regulary use, that's stored in the Database, then you could retrieve it once, place it in cache memory, and retrieve it directly from memory on each subsequent use, instead of having to go back to your Database (or FileSystem or other relatively slow-retrieval storage medium)
Where do you think the heap is? It all ends up in RAM or the Pagefile :)
Was there something else specific to this question, access times etc?
I'm investigating SQLite as a storage engine, and am curious to know whether SQLite locks the database file on reads.
I am concerned about read performance as my planned project will have few writes, but many reads. If the database does lock, are there measures that can be taken (such as memory caching) to mitigate this?
You can avoid locks when reading, if you set database journal mode to Write-Ahead Logging (see: http://www.sqlite.org/wal.html).
From its Wikipedia page:
Several computer processes or threads may access the same database without problems. Several read accesses can be satisfied in parallel.
More precisely, from its FAQ:
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
A single write to the database however, does lock the database for a short time so nothing can access it at all (not even reading). Details may be found in File Locking And Concurrency In SQLite Version 3. Basically reading the database is no problem unless someone wants to write to the database immediately. In that case the DB is locked exclusively for the time it takes to execute that transaction and the lock is released afterwards. However, details are scarce on what exactly does with read operations on the datapase in the time of a PENDING or EXCLUSIVE lock. My guess is that they either return SQLITE_BUSY or block until they can read. In the first case, it shouldn't be too hard to just try again, especially if you are expecting few writes.
Adding more info for this answer:
Q: Does SQLite lock the database file on reads?
A: No and Yes
Ref: https://www.sqlite.org/atomiccommit.html#_acquiring_a_read_lock
The first step toward reading from the database file is obtaining a shared lock on the database file. A "shared" lock allows two or more database connections to read from the database file at the same time. But a shared lock prevents another database connection from writing to the database file while we are reading it