I am inserting a large number of records into a SQLite database on Android. To improve insert times, I am considering creating the index on the table after data has been fully added.
My question is, at what point does the database actually build the index against values on the table? Does it happen as soon as I issue the SQL statement (create index index_name on table ...), or can the database defer it until the first query arrives?
Thanks,
Ranjit
It creates the index immediately you issue the create index command. The relevant code is in sqlite3CreateIndex and this will create the index and write it to disk (except for the special case where it's called as part of a database open operation but that's not the case when a user creates an index).
Related
I have an excel spreadsheet with multiple entries that I want to insert into an SQLite DB from UIPath. How do I do this?
You could do it one of two ways. For both methods, you will need to use the Excel Read Range to put the excel into a table.
Scenario 1: You could read the table in a for each loop, line by line, converting each row to SQL and use a Execute non-query activity. This is too long, and if you like O notation, this is an O(n) solution.
Scenario 2: You could upload the entire table (as long as its compatible with the DB Table) to the database.
you will need Database > Insert activity
You will need to provide the DB Connection (which I answer in another post how to create)
Then enter the sqlite database table you want to insert into in Quotes
And then enter the table name that you have created or pulled from another resource in the last field
Output will be an integer (Affected Records)
For O Notation, this is an O(1) solution. At least from our coding perspective
A user has a large table (3+ billion rows).
To speed up queries for the next few months, an index on the remote database must be created.
Assuming there is a connection called conn - what is the best way to create an index and make it persist after disconnection from the database.
e.g.,
library(DBI)
sql<-'CREATE INDEX idx_pmid ON medcit (pmid ASC);'
dbExecute(conn,sql)
dbDisconnect(conn)
The code above seem to work but how can the index be somehow verified (make sure it truly exist and speeds up future queries)? In other words - how can a user check the existence of the index? Also, do I need to issue COMMIT command somehow?
To create an index on a table:
dbGetQuery(conn, "CREATE INDEX index_name ON public.db_name USING btree (variable_name)")
I have a massive database (~800 GB) with several indexed tables. I need to copy one table (including indexes) to a new database. Copying the table itself is pretty straightforward.
$ sqlite3 newDB
> attach database 'oldDB.db' as oldDB
> create table newTable as select * from oldDB.oldTable
But I can't seem to find any information on a way to also copy over an index. Is there any way to do this? Since the tables are so large I'd really like to avoid having to re-index them.
SQLite has no mechanism to copy index contents.
If this particular table would be the majority of the data in the database, the fastest way to copy it would be to copy the database file and then to drop all other tables.
But otherwise, there you cannot avoid the reindex operation.
Please note that CREATE TABLE ... AS ... does copy only the contents of the table, but not the complete table definition (such as column types or constraints).
Copying large table in a single transaction is not a good idea. If you really have to you should turn off journaling first (destination database):
PRAGMA journal_mode=OFF;
As the others have stated, the index cannot be broken out. I suspect that time spent copying the database and then dropping a very large table would be longer than just -> 1. creating the new destination database, 2. determining the original CREATE TABLE statement (from the SQLITE_MASTER table of the source database) and recreating the table in the destination database. Then 3. just ATTACH your destination database to the source database and INSERT INTO destinationdb.tablename SELECT * FROM sourcedb.tablename;* to get the copy rolling.
I am using a global application user account to access database A. This user account does not have permissions to modify database A's schema (ie, create tables, modify tables, etc). This user also has access to database B, but only views. I need to run SQL to feed data from a view in database B into a table in database A.
In a perfect world, I would be able to use this SQL:
create database_a.mytable as (select * from database_b) with no data
However, the user can't create tables in database A. If I could get the DDL of the select statement then I could log in under my personal account (which doesn't have any access to database B) and run the DDL in database A to create the table.
The only other option is to manually write the SQL, but I don't want to do that, especially since this view I am wanting to copy has many columns of varying data types and sizes.
Edit: I may be getting closer. I just experimented with this:
show (select * from database_b.myview)
However, it generated the DLL of every single table that is used in the view itself, as well as the definition for the view. This doesn't really help me since I just want the schema of the select statement itself. In other words, I need what would be generated if I were to use the create table as statement mentioned above.
Edit for Rob: Perhaps "DDL" was the wrong term to use. Using show view db.myview just shows the definition of the view, not the schema it represents. In my above example of create table as, I show how you can create a table that mimics the schema of a result set returned in a select. It generates a DDL on the back end for creating a table and then executes that DDL to actually create the table. You can then say show table db.newtable and see the new table's DDL. I want to get that DDL directly from a select statement so that I can copy it, log out of the app account, into my personal account, and then execute the DDL to create the table.
This is only to save me the headache of having to type out the DDL manually by hand to save time and reduce typing errors, especially since the source view has so many columns. That said, I think hitting up the DBA or writing some snazzy stored procedure to do dynamic stuff would be a bit over the top for my needs. I think there has to be a way to get the DDL for creating a table schema directly from a select statement.
Generate DDL Statements for objects:
SHOW TABLE {DatabaseB}.{Table1};
SHOW VIEW {DatabaseB}.{View1};
Breakdown of columns in a view:
HELP VIEW {DatabaseB}.{View1};
However, without the ability to create the object in the target database DatabaseA your don't have much leverage. Obviously, if the object already existed INSERT INTO SELECT ... FROM DatabaseB.Table1 or MERGE INTO would be options that you already explored.
Alternative Solution
Would it be possible to have a stored procedure created that dynamically created the table based on the view name that is provided? The global application account would simply need privilege to execute the procedure. Generally the user creating the stored procedure would need the permissions to perform the actions contained within the stored procedure. (You have some additional flexibility with this in Teradata 13.10.)
There are some caveats with this approach. You are attempting to materialize views that could reference anywhere from hundreds to billions of records. These aren't simple 1:1 views that are put on top of the target tables. Trying to determine the required space in the target database to materialize the view will be difficult. Performance can and will vary depending on the complexity of the view and the data volumes. This will not be a fast-path or data block optimized operation.
As a DBA, I would be concerned with this approach being taken on by a global application account without fully understanding the intent. I trust you have an open line of communication with the DBA(s) involved for supporting this system. I'm sure there are reasons for your madness that can't be disclosed here.
Possible Solution - VOLATILE TABLE
Unless the implicit privilege for CREATE TABLE has been revoked from the global application account this solution should work.
Volatile tables do not require perm space. There table definitions persist for the duration of the session and any data inserted into them relies on the spool space of the user who instantiated it.
CREATE VOLATILE TABLE {Global Application UserID}.{TableA_Copy} AS
(
SELECT *
FROM {DatabaseB}.{TableA}
)
WITH NO DATA
NO PRIMARY INDEX
ON COMMIT PRESERVE ROWS;
SHOW TABLE {Global Application UserID}.{TableA_Copy};
I opted to use a Teradata 13.10 feature called NO PRIMARY INDEX. By default, CREATE TABLE AS will take the first column of the SELECT statement and make it the PRIMARY INDEX of the table. This could lead to skewing and perm space issues in your testing depending on the data demographics. You can specify an explicit PRIMARY INDEX on your own as you understand the underlying data. (See the DDL manuals for details on the syntax if you're uncertain.)
The use of ON COMMIT PRESERVE ROWS for the intent of this example is probably extraneous. But in reality if you popped any data into that table for testing this clause would be beneficial in Teradata mode as the data would otherwise be lost immediately after the CREATE TABLE or any other data manipulation was performed against the volatile table.
I have a .sqlite db which contains only one table. That table contains three columns and I am interested in indexing one column ONLY.
The problem is, when I perform the indexing, I got an empty index table !
I am using SQLite Manager add-ons for Firefox. This is the syntax that appears before I confirm the indexing:
CREATE INDEX "main"."tableIndex" ON "table" ("column1" ASC)
I don't know what is the problem here. I tried this tool - long time ago - with another database and it works fine.
Any suggestion ?
You cannot "see" the contents of a database index. No table or table-like structure is created that corresponds to the index. So there is nothing to look at that could be empty.
If the CREATE INDEX command ran without error, you can be confident that the index was created and will continue to be maintained by SQLite as you add, remove, and update data.
As per the comments, below, #iturki is actually trying to index for full text search. SQLite supports several extensions for full text search but they are not implemented through the stanard CREATE INDEX command. See this reference.
Try use VACUUM query. It will completely rebuild sqlite database file and will rebuild all indices and reset all ROWID etc.