I have two SQLite databases, and I want to do some operation to them (such as INNER JOIN). How can I do that?
Thanks,
Vincent
If you have 2 separate databases there are 2 main ways of tackling that:
1) Insert all (copy) the data into a single database, then query that.
2) Load all data into memory and manipulate it there.
Unless I'm missing it, there does not seem to be a "Linked Server" function (like in SQL Server) which would allow you to query them in a distributed manner.
Related
We have several pairs of databases that relate to each other, i.e.
db1a and db1b
db2a and db2b
db3a and db3b
etc.
where there are cross-db joints between db1a and db1b, 2a and 2b, etc.
Having an open sqlite database connection, they can be attached, i.e.
ATTACH 'db1a' as a; ATTACH 'db1b' as b;
and later detached and replaced with other pairs when needed.
How should the database "connection" be created, though, as there is originally no real database to attach to? Using the first one as the main database gives it much more significance which is not meaningful - and as it is not detachable it's a hinderance later on.
The only way I can see is opening a :memory: or a temporary ('') database connection. Is there some better option available?
In the absence of any better alternative, :memory: is the best choice. (:temp: is a normal file name, and invalid in many OSes; a temp DB would have an empty file name.)
If you have some meta database that lists all other databases, you could use that.
Please note that when you have multiple attached databases, any change to one of them will involve a multi-database transaction.
So if the various database pairs do not have any relationship with other databases with different numbers, consider using a new connection for each pair.
I'm in the process of developing an application that allows users to select tables and columns, and choose join methods ( like inner, outer..) as well as aggregate functions sql (sum, avg..etc) for the purpose of generating reports from those selections.
What I did is append strings to build a request to an sql server. I think i'm wrong doing it this way because users can choose a lot of columns and that throws unexpected exceptions. Are there some ideas on a better way to go about this (not source code)?
As you have asked, what is best way to manage query ( can contain sum,avg etc..) generated on server side and made those to execute on SQL server, I am writing best possible suggestion for this. Consider that and follow your path.
If you have adhoc queries generated in server side, then definitely you can write those into SQL group of statements known as Stored Procedure . And this has following advantages.
More manageable, if same set of SQL blocks need in multiple places, then use this single procedure everywhere.
Secure - you don't have to set permission( like select) for multiple tables (joining tables), instead just have execute permission for procedure.
Good Performance - Execution Plan Caching and Reuse
Source - http://weblogs.asp.net/fbouma/38178
Stored proc - http://msdn.microsoft.com/en-us/library/ms190782.aspx
If you are new to stored procedure and want to know, how to use that in asp.net, here is tutorial for this - http://www.dbtutorials.com/advanced/using-stored-procedures-cs/
Afaik, SQLite stores a single database in a single file. Since this would decrease the performance when working with large databases, is it possible to explicitly tell SQLite not to store the whole DB in a single file and store different tables in different files instead?
I found out, that it is possible.
Use:
sqlite3.exe MainDB.db
ATTACH DATABASE 'SomeTableFile.db' AS stf;
Access the table from the other database file:
SELECT * FROM stf.SomeTable;
You can even join over several files:
SELECT *
FROM MainTable mt
JOIN stf.SomeTable st
ON (mt.id = st.mt_id);
https://www.sqlite.org/lang_attach.html
tameera said there is a limit of 62 attached databases but I never hit that limit so I can't confirm that.
The big advantage besides some special cases is that you limit the fragmentation in the database files and you can use the VACUUM command separately on each table!
If you don't need a join between these tables you can manually split the DB and say which tables are in which DB (=file).
I don't think that it's possible to let SQLite split your DB in multiple files, because you connect to a DB by telling the filename.
SQLite database files can grow quite large without any performance penalties.
The things that might degrade performance are:
file-locking contention
table size (if using indexes and issuing write queries)
Also, by default, SQLite limits the number of attached databases to 10.
Anyway, try partition your tables. You'll see that SQLite can grow enormously this way.
we have two sqlite DB's , we have a requirement to "attach" one to other and perform some joins. we have some questions/concerns as below:
say we have attached DB1 with DB2 and performing some SELECT's , can some other thread concurrently UPDATE/INSERT on DB2 or DB1 with a different connection ?
is there a separate C API to attach or we need to use "sqlite3_step"
how is the performance with ATTACH.
Thanks in Advance
DEE
Another thread can concurrently alter either database, but this will mean that at some point the database can be locked for the querying thread. See here about concurrency with SQLite.
ATTACH is a one step operation, you can us sqlite3_exec.
Performance is a tough thing to predict and will vary greatly with schema, indexing, usage, and data stored (and some other factors too like page size). In some cases, ATTACH can be slower than if all data is in one database. My personnal experience was that separating large datasets was faster for inserts and affected final query output minimally/imperceptibly. Your mileage may vary.
I am re-designing an application for a ASP.NET CMS that I really don't like. I have made som improvements in performance only to discover that not only does this CMS use MS SQL but some users "simply" use MS Access database.
The problem is that I have some tables which I inner join, that with the MS Access version are in two different files. I am not allowed to simply move the tables to the other mdb file.
I am now trying to figure out a good way to "inner join" across multiple access db files?
It would really be a pity if I have fetch all the data and the do it programmatically!
Thanks
You don't need linked tables at all. There are two approaches to using data from different MDBs that can be used without a linked table. The first is to use "IN 'c:\MyDBs\Access.mdb'" in the FROM clause of your SQL. One of your saved queries would be like:
SELECT MyTable.*
FROM MyTable IN 'c:\MyDBs\Access.mdb'
and the other saved query would be:
SELECT OtherTable.*
FROM OtherTable IN 'c:\MyDBs\Other.mdb'
You could then save those queries, and then use the saved queries to join the two tables.
Alternatively, you can manage it all in a single SQL statement by specifying the path to the source MDB for each table in the FROM clause thus:
SELECT MyTable.ID, OtherTable.OtherField
FROM [c:\MyDBs\Access.mdb].MyTable
INNER JOIN [c:\MyDBs\Other.mdb].OtherTable ON MyTable.ID = OtherTable.ID
Keep one thing in mind, though:
The Jet query optimizer won't necessarily be able to use the indexes from these tables for the join (whether it will use them for criteria on individual fields is another question), so this could be extremely slow (in my tests, it's not, but I'm not using big datasets to test). But that performance issue applies to linked tables, too.
If you have access to the MDBs, and are able to change them, you might consider using Linked Tables. Access provides the ability to link to external data (in other MDBs, in Excel files, even in SQL Server or Oracle), and then you can perform your joins against the links.
I'd strongly encourage performance testing such an option. If it's feasible to migrate users of the Access databases to another system (even SQL Express), that would also be preferable -- last I checked, there are no 64-bit JET drivers for ODBC anymore, so if the app is ever hosted in a 64-bit environment, these users will be hosed.
Inside one access DB you can create "linked tables" that point to the other DB. You should (I think) be able to query the tables as if they both existed in the same DB.
It does mean you have to change one of the DBs to create the virtual table, but at least you're not actually moving the data, just making a pointer to it
Within Access, you can add remote tables through the "Linked Table Manager". You could add the links to one Access file or the other, or you could create a new Access file that references the tables in both files. After this is done, the inner-join queries are no different than doing them in a single database.