Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I have an application that use SQLite for storage, and I'm wondering whether it is safe to use Windows DFS Replication to backup the database file to a second server which has a cold standby instance of the application installed.
Potentially relevant details:
Although DFS supports two-way replication, in this case it is only the master DB file that is written to, so the replication is effectively one-way.
The master DB file is located on the same server as the process that is writing to it.
Currently SQLite is configured to use the standard Rollback Journal, but I could switch to Write-Ahead Log if necessary.
If DFS locks the master DB file during replication then I think this approach could work as long as the lock isn't held for too long. However I can't find sufficient information on how DFS is implemented.
UPDATE: I've implemented this in a test environment, and have had it running for several days. During that time I have not encountered any problems, so I am tempted to go with this solution.
Considering that DFS Replication is oriented towards files and folders:
DFS Replication is an efficient, multiple-master replication engine
that you can use to keep folders synchronized between servers across
limited bandwidth network connections.
I would probably try to avoid it if you care about consistency and keeping all your data, as stated by the SQLite backup documentation:
Historically, backups (copies) of SQLite databases have been created
using the following method:
Establish a shared lock on the database file using the SQLite API (i.e. the shell tool).
Copy the database file using an external tool (for example the unix 'cp' utility or the DOS 'copy' command).
Relinquish the shared lock on the database file obtained in step 1.
This procedure works well in many scenarios and is usually very fast.
However, this technique has the following shortcomings:
Any database clients wishing to write to the database file while a backup is being created must wait until the shared lock is relinquished.
It cannot be used to copy data to or from in-memory databases.
If a power failure or operating system failure occurs while copying the database file the backup database may be corrupted
following system recovery.
In the case of DFS, it wouldn't even lock the database prior to copying.
I think your best bet would be to use some kind of hot replication, you might want to use the SQLite Online Backup API, you could check this tutorial on creating a hot backup with the Online Backup API.
Or if you want something simpler, you might try with SymmetricDS, an open source database replication system, compatible with SQLite.
There are other options (like litereplicator.io), but this one went closed source and is limited to old SQLite versions and ~50MB size databases
ps. I would probably move away from SQLite if you really need HA, replication or this kind of features. Depending on your programming language of choice, most probably, you already have the DB layer abstracted and you could use MySQL or PosgreSQL.
Related
I can't seem to find an application to monitor SQLite DB performance. Currently I have a test server that uses SQLite. I'm primarily concerned with obtaining a benchmark of storage requirements and performance for scaling this server to production.
I know for MySQL there is the standard Nagios for monitoring (changing to mySQL is not an option at this point). Is there anything analogous for SQLite?
SQLite has functions like sqlite3_status() and sqlite3_db_status(), but those do not really give you the information you want, and might not even be available in all languages.
Anyway, SQLite is an embedded library, so you'd have to monitor your actual application. Tools like Nagios allow to monitor a server's CPU load and disk usage, but you can also use any other tool of your OS.
I use Informix DBMS in all my web applications. My question has two parts:
Does the DBMS have a big effect on the performance of my applications
and if the answer is yes what about Informix and `MS SQL Server in this
issue?
I want some GUI tools to facilitate my job when writing queries,
creating database, relationships, ERD, etc. The Informix client
is so bad. There are no facilities at all. I want some tools
like SQL Server Management Studio
As a GUI tool for Informix you can use Aqua Data Studio from Aquafold. It also supports MS SQL Server.
As of the performance: it depends. How well is your Database design. Do you use indexes, is your query well-written, etc. etc. Very hard to answer your question, we just don't know enough.
To design a solution that would perform the best, one needs to know the nature of the application you are building. For example, if you are building a system that needs to process and compute large volume of data and computations can be distributed, a "traditional" relational database is not a good option no matter what vendor you choose. You would be better off with an option that supports sharding, Hadoop and will likely be based on some kind of NoSQL solution.
If you are sticking with RDMS and building something that has a lot of reads and not a lot of writes, go for a database that supports Snapshot Isolation which will allow your readers to not be blocked by writers.
Cost also plays into this - some RDMS systems are free, some are not. Your question is way to general to be answered specifically.
Aqua Data Studio is good but quite expensive. An open source tool SQL Workbench/J is also an effective tool for informix.
Informix have its own charm but i guess it should not be said that MS-Sql Server is slower or not good in performance. You may decide DBMS according to your nature of application. There are many techniques to optimize Database performance like, Applying Indexes/ Not too many Joins/ Queries can be optimize too/ Stored Procedure can also be used/ Multi-DBs level etc.
Once i need to develop Social Media site, i used MySQL in this project but only for POSTs i installed MongoDB.
Regards,
Salik
I have a large (several Gb) berkeley db that I am thinking of migrating from windows (2K) to Linux (either Redhat or Ubuntu). I am not sure how to go about this. Can I merely move the db files accross, or do I need a special conversion utility?
Database and log files are portable across different endian systems. Berkeley DB will recognize the kind of system it is on and swap bytes accordingly for data structures it manages that make up the database itself. Berkeley DB's region files, which are memory mapped, are not portable. That's not such a big deal because they region files hold the cache and locks which, because your application will not be running during the transition, will be re-created on the new system.
But, be careful, Berkeley DB doesn't know anything about the byte-order or types in your data (in your keys and values, stored at "DBTs"). Your application code is responsible for knowing what kind of system it is running on, how it stored the data (big or little endian) and how to transition it (or simple re-order on access). Also, pay close attention to your btree comparison function. That too may be different depending on your system's architecture.
Database and log files are also portable across operating systems with the same caveat as with byte-ordering -- the application's data is the application's responsibility.
You might consider reviewing the following:
Selecting a Byte Order
DB->set_lorder()
Berkeley DB's Getting Started Guide for Transactional Applications
Berkeley DB's Reference Guide
Voice-over presentation about Berkeley DB/DS (Data Store)
Voice-over presentation about Berkeley DB/CDS (Concurrent Data Store)
Berkeley DB's Documentation
Disclosure: I work for Oracle as a product manager for Berkeley DB products. :)
There's a cross-platform file transfer utility described here.
You may also need to be concerned about the byte order on your machine, but that's discussed a little here.
If you're using Java Berkeley though it shouldn't matter?
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I am trying to plan a way for 5 developers to use Visual Studio 2005/2008 to collaboratively develop an ASP.NET web app on a development web server against an Oracle 8i(soon to be 10g) Database.
The developers are either on the local network or coming in over a vpn (not a very fast connection),
I evaluated the latest Visual SourceSafe, but ran into the following gotchas:
1) We can't use decentralized development because we can't replicate a development oracle database to all developers computers. Also, the vpn is too slow to let their local app instances connect to the database server.
2) Since VSS source code not on the file system, the only way to debug it is to build the app and run debugger, which only one developer can do at a time on a centralized development server. This is unacceptable. We tried using shadow folders so that every time a file is checked in it gets published to the app instance on the development server, but this failed for remote developers on the vpn.
3) Since the developers do a lot of web code, it is important for productivity reasons that when they SAVE a file, they should be able to immediately see the change working on the development server.
4) No easy way to implement a controlled process for pushing files to the production server.
Any suggestions on a source control solution that would work under these contraints?
Update: I guess since development is forced to be on the server, we need to go with a "Lock and Check In" model. So which source control solution would work best for "Lock and Check In' scenarios?
Update: Does Visual SVN support developing centrally against a development server? As in, the dev can immediately see his update on the development server after saving in VS?
I have used Subversion and TortoiseSVN and was very pleased.
Is point 1 due to an issue with your database schema (or data) ?
We can't use decentralized development because we can't replicate a development oracle database to all developers computers.
If not, I strongly suggest that every developer has its own environment (Visual Studio, Oracle...) and use your development server for integration purposes. Maybe you could just give them a subset of the data, or maybe just the schema scripts.
Oracle Express Edition is perfectly fit for this scenario. Besides, sharing the same database violates rule #1 for database work, which in my experience should be enforced anywhere possible.
As Guy suggested, have an automated build allowing any developer to recreate its database schema at any time.
More very useful guidelines can be found here (include rule #1 above).
Define your development process so that parallel development is possible, and only use locks as a last resort.
I'm sorry if you already envisioned these solutions and found them unfit to your situation, but I really felt the urge to express them just in case...
Visual Source Safe is the spawn of Satan.
Look at Subversion, and Visual SVN (with Tortise SVN). Sure, Visual SVN costs a bit - $49 per seat - but it is a great tool. We have a development team of 6 programmers, and it has been a great boon to us.
If you can spend the money, then Team Foundation Server is the one that works best in a Visual Studio dev environment.
And based on personal experience, it works beautifully over VPN connections. And you can of course have automated builds going on it.
I would say SVN on price (free), Perforce on ease of integration.
You will undoubtedly hear about GIT and CVS as well and there are good reasons to look at them.
Interesting -- it sounds you are working on a web site project on the server, and everyone is working on the same physical files. I agree that SVN is far superior to VSS and really good to work with, but in my experience it's really geared toward developers working on a copy of the code locally.
VSS is a "lock and check in" type of source control, while SVN and TFS and most others are "edit and merge" -- devs all get copies of the source, edit the files as needed, and later merge their changes in to source control, and if someone else has edited the file in the meantime they merge the changes together.
From a database standpoint, I assume you are checking in your database scripts, then have some automated build packaging and running them (or maybe just a dev or DBA running them manually every so often). In this case, having the developers have a local copy of the scripts that they can edit and merge using SVN or TFS makes sense.
For a team working on a shared copy of the source code on a development server, though, you may get into problems using edit and merge -- a "lock and check in" model of source control may work better for you. Just not VSS, from a corruption and stability standpoint.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
There are quite a few Sqlite GUI applications listed here:
http://www.sqlite.org/cvstrac/wiki?p=ManagementTools
some appear to be incomplete, buggy, not maintained, etc. Do you have any recommendations?
I use the SQLite manager plugin for Firefox.
It seems pretty stable to me.
Here's another free option. It has been pretty stable for me. SQLiteStudio
For Windows: I've been looking for functionality and a comfortable GUI - it's been particularly hard to satisfy the latter requirement, but these two picks are both fine:
a) Good enough: SQLite Expert
Less expensive, fully featured manager. The author is very responsive to comments and bug reports, and publishes updates frequently. The flip side, if you look at version history, is that new releases seem to introduce new bugs, which are then fixed in sunsequent builds. The GUI is fine, very good for quickly designing new databases; a little less so for designing queries and working with large amounts of data. Main gripe: you can't see the schema while editing a query (without flipping tabs in the program).
b) Nearly perfect: SQLite Maestro
Pricier. Rich UI, easy access to all features, nice visual query builder and automatic SQL formatter for readability, lots of eye candy. There is a separate, more specialized query builder (SQLite Code Factory), but you can make do with just the main Maestro application. Main gripe: can't seem to be able to change font size for table display and the default is a little too small; at the beginning it's easy to get lost in the thicket of tabs, though overall I find the GUI very productive.
Both solutions are very stable in my experience, and both seem to offer occasional discounts, if you can afford to wait.
For just browsing data, try SQLite Spy - free and lets you execture queries, but no or little GUI support for editing. Very convenient for quick lookups though.
Spent the morning looking for a good Sqlite Database Manager/Browser and have settled on SqliteStudio, currently v2.20.28, which I'm running on Ubuntu Linux 10.04.
The download is one 4.3MB uncompressed executable file. Period!
It's fast, well-behaved, and uses modest resources.
The interface is clean and attractive with logical functionality.
I don't say this often, but it's just a "A Joy to Use".
It was developed and is actively maintained by Pawel Salawa who is is to be commended for producing a very nice program.
I just used it to merge two Firefox FloatNotes databases (Sqlite v3), which just use a single table...
The target database is on the local host where SqliteStudio is running (i.e. /0/LX02)
The source database file is on a remote host mounted via SSHFS to /0/LX04
Opened the two database files using Add Database.
Used the SQL Editor to execute an INSERT to the LX02 database from a SELECT on the LX04.
Click the Commit icon when all goes well or Rollback if there are errors to fix.
Refresh the table data to see the inserted rows.
There's no Import GUI functionality but merging is pretty simple if you know SQL. I supplied NULL for the first column which is the unique-id primary-key so that Sqlite would autoincrement, thus renumbering the rows being merged in...
INSERT INTO [floatnotes.sqlite].floatnotes
SELECT NULL,
url,
protocol,
content,
x,
y,
w,
h,
color,
status,
guid,
creation_date,
modification_date
FROM [floatnotes-LX04.sqlite].floatnotes;
I like SQLite Administrator
I've been using SQLite Professional and it's been reliable. The only downside was that changes I made to the db via the iOS simulator didn't resolve in the app (I had to close it and reopen for changes to show) but that feature was added in a recent version. So now it does everything I need. I'm happy with the support. It's good for testing and I use it to build out the db structure. Never been buggy.
I've used Sqliteman in the past. Quite nice.
SQLiteSpy is a good choice
Sqlite.org has a list of management tools available here:
http://www.sqlite.org/cvstrac/wiki?p=ManagementTools
If your using OSX you may like SQLPro for SQLite (App Store).
The app has a few neat features such as:
Versions Integration (rollback to previous versions).
Inline data filtering.
Exporting options to CSV, JSON, XML and MySQL.
Column reordering.
Full screen support.
I've been using SQL Explorer. The firefox plugin is awesome, but it couldn't handle BIGINTs properly (it truncated them). I have noticed that the .jar driver doesn't seem to support FTS4 (but it does support FTS3) and doesn't show indexes in the data structure.
I like SQL explorer as you can use other providers like mySQL too with the one client.
You didn't mention a platform, so here's a great comparison of Mac OS X SQLite tools. I personally found MesaSQLite to be most like my preference for database tools, which was CocoaMySQL incidentally. (For Windows, I just used the Firefox add-on mentioned above.)
If you're within Visual Studio most of the time then System.Data.SQLite is good, and as a plus handles encrypted databases.
Navicat SQLite is very good and they support Windows, OS X and Linux too.
You might want to check MYZSQLExplorer, here.
Unlike the other tools, it is running on iOS devices (or in the simulator). It is a Viewer, not a management tool, and is not as feature-complete as some of the other tools, but is convenient as you can launch it from within your code and browse your databse from within your app.
Or you can "Open In..." it sqlite stores, by sending them as email attachments.
I developed it in order to help myself during the development and testing phases, and it did help.
If you have any feedback on it, I'd love to hear it.
If you want just CRUD operations on the sqlite database file, then SQuirreL is a very option as it has an auto complete feature which drastically improve the speed and efficiency of typing the sql queries.
To use the SQLite database in SQuirreL first download the JDBC driver of the SQLite from here then drop the jar in the lib folder of the SQuirreL folder. Now open SQuirreL and choose Create a New Driver.
In the Example URL field put
jdbc:sqlite:$file_url
and in Class Name put
org.sqlite.JDBC
After that choose Create a New Alias and choose the driver that you just added and replace $file_url with the actual location of the sqlite file then click ok and you are done.