So I have a 150 GB database that I want to import to Google Cloud SQL (it doesn't matter if its PSQL or MySQL.) The reason I want to do this is to have more flexible space, and faster computation.
However, I find no easy intro on how this is done. It seems like the way to do this is to also make a Google Cloud Storage instance, then dump my SQLite database to an SQL file, upload it to the Cloud Storage bucket, then import it to Google Cloud SQL. Is this the best and quickest way?
Dumping a 150 GB database would probably require lots of space and lots and lots of time.
Ultimately, what you need to do is to:
make sql dump of your sqlite database
convert it to be mysql- or postgresql- compatible (either manually or using some tool)
upload it to Google
import into your CloudSQL instance
You can try to minimize intermediate steps by using something like https://github.com/dimitri/pgloader. It seems like you can use this tool to connect your sqlite database directly to CloudSQL instance. It will take time - there's no getting around trasferring ~150G worth of data to Google
Where is your sqlite database stored now? If it's already in GCE VM, running pgloader from the same region as your CloudSQL instance should make it much faster.
Related
I want to recreate SQLite but when I modify underlying storage engine from v3.38.2 (B-tree, tightly coupled with pager and VDBE modules), I found it hard to separate. I am not sure if there is a common internal storage engine interface. First step is to separate the B-tree engine and write another version of it.
I noticed assert statements inside every function to check integrity of the database. Is it possible to call the internal functions without opening a database connection? sqlite4 mentions interchangeable storage engines. Is it possible to develop my own storage engine for SQLite?
I need to store information in the mongoDB database on the phone to be used while off-line. The app will download the data while online, and store it in the DB, to be used while off-line. Then when the user is online again, I will send the mongoDB info collected, using my API.
I don't want the mongoDB to be synced with the server while online, either. I want to keep the data on the individual phone. I want to use the data in mongoDB while offline. I need the app to be able to quit/restart, without losing the data on the phone locally.
What is the best way to go about doing this?
There are some options to consider.
1) Create a local mongo database - this is client only storage with no server publication (not sure if it persists between app invocations)
2) sqlite can do the job, but only on Android (not IOS)
3) LokiJS is a fast JS only database that promises to be useful - haven't explored it, but it would be good to hear some feedback
4) If the data is small, you could use LocalStorage, it's pretty simple, you just need to look after serialising and de-serialising it yourself
I'm writing a WP8 app that needs local storage on the phone itself (due to poor internet connectivity) but also the ability to push to the Azure cloud when there is internet available.
I'm just looking to see what everyone thinks is the best method for local storage?
SQLite? JSON string saved to local storage? Or maybe I'm missing something. Ideally I'd like something that mirrored/used the classes I need that represent the tables in the cloud.
Hope this makes sense.
Cheers
You can use the local database for Windows Phone, which allows you to mark up your classes with attributes to define the tables, columns, relationships, etc. You can use SQLite, but I prefer this approach since it's straightforward and easy to implement.
Here's the information on MSDN about it: http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202860(v=vs.105).aspx
And this walk through example: http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202860(v=vs.105).aspx
i was wondering if there is a way to enable cloud features for a SQLite database application.
Should i save the whole database to the cloud each time ? For example when i quit the application is it required to save the whole database to the cloud.
What do you suggest ?
Should i drop SQLite and use another database for cloud programming .
iCloud supports SQLite databases.
When properly setup it will only sync change logs instead of the entire database. In theory it's pretty nice. I haven't however had the best of luck using it yet, it seems to be a little too buggy to actually use in ios 5, hopefully it's better in 6.
To be most efficient you could manage a changelog of objects that are modified by the app. Then when its time to sync (while closing the app for instance), you can make operational requests to the Cloud. For add and update you can send the entire object, while for delete just the oid should suffice.
This is a very simple sync scenario. Things can get complicated fast if you are looking to send changes that happen in the Cloud down to the device. That is a scenario for a different thread.
Based on your question, you just need to sync from the device to the Cloud.
Is anyone running SQLite in-memory db as their main database for their web app? I'm trying to get a handle on the feasibility/stupidity of such a setup.
If I have a web application with fewer than 500 concurrent users and a database that's smallish in size (between 0 and 4 GB), what's the feasibility of running SQLite in-memory with the entire database in it as the main database for the app?
Running in-memory would obviously make the "Durable" aspect of ACID difficult, but with the SQLite backup API, it seems like it's possible to keep an in-memory db in sync with a file-based db. I'm thinking that when a user executes a Save/Update/Delete command, it could instantly update the in-memory copy and then be queued to make it to the file-based db. Whenever the app gets cycled, it could just be loaded from the file-based db via the backup API, right?
According to the SQLite documentation, a single connection must be kept open in order to keep the in-memory db up and running, so is this a problem if it's kept open for hours or days?
You should only use an in-memory database for data processing, not to store data. Because you need to store the data on disk anyway, it's much simpler and more effective to store the database on disk in the first place.
If you need to sync the database to disk, why use an in-memory database? The advantage of an in-memory database is that data never needs to be written to a filesystem. Since you need data to be written to disk, you've turned the sole advantage into a disadvantage. So why do it? Just crank up the cache as large as you can.