Is there any way to query physical database size per table in SQLite?
In MySQL, there's a meta-table called information_schema.TABLES that lists physical table sizes in bytes. PostgreSQL has a similar meta-table called pg_tables. Is there anything like this in SQLite?
If you know where the database is, it's just a file. For example:
$ wc -c db/development.sqlite3
2338816 db/development.sqlite3
The sqlite3_analyzer tool outputs lots of information about a database, among them the amount of storage used by each table.
Related
I've a clustered partitioned table exported from GA 360. Attached is the image. I would like to create exact replica of the same. Using Web UI it's not possible. I created backup table using bq command line tool, still no luck.
Also, whenever we check preview it has a day filter. It looks like this:
Whenever data is appended to the backup table, I don't find this filter there even though this option is set to true while creating a table.
If you can give more context about handling this kind of table it would be beneficial.
Those are indeed sharded tables. As explained by #N. L they follow a time-based naming approach: [PREFIX]_YYYYMMDD. They then get grouped together. The explained procedure to backup them seems correct. Anyhow, I would recommend to use partitioned tables as it will be easier to backup them and they perform better in general.
This is not a cluster / partitioned table. This one is a sharded non-partitioned table having one common prefix. Once you start creating multiple tables with same prefix we can see them under the same prefix.
Ex:
ga_session_20190101
ga_session_20190102
both these tables will be grouped together.
To take backup of these tables you need to create a script to copy source to destination table with same name and execute that script using bq command line tool under the same project.
I want to delete multiple rows from Berkeley DB using where condition. For example the query would be like this:
DELETE FROM table_name
WHERE [condition];
Is it possible to do?
Yes, you can use the BDB SQL Interface, which is compatible with SQLite's API, take a look at BDB SQL Is Nearly Identical to SQLite
Is it possible to execute an Array DML INSERT or UPDATE statement passing a BLOB field data in the parameter array ? And the more important part of my question, if it is possible, will Array DML command containing BLOB data still be more efficient than executing commands one by one ?
I have noticed that TADParam has a AsBlobs indexed property so I assume it might be possible, but I haven't tried this yet because there's no mention of performance nor example showing this and because the indexed property is of type RawByteString which is not much suitable for my needs.
I'm using FireDAC and working with SQLite database (Params.BindMode = pbByNumber, so I'm using native SQLite INSERT with multiple VALUES). My aim is to store about 100k records containing pretty small BLOB data (about 1kB per record) as fast as possible (in cost of the FireDAC's abstraction).
The main point in your case is that you are using a SQLIte3 database.
With SQLite3, Array DML is "emulated" by FireDAC. Since it is a local instance, not a client-server instance, there is no need to prepare a bunch of rows, then send them at once to avoid network latency (as with Oracle or MS SQL).
Using Array DML may speed up your insertion process a little bit with SQLite3, but I doubt it will very high. Good plain INSERT with binding per number will work just fine.
The main tips about performance in your case will be:
Nest your process within a single transaction (or even better, use one transaction per 1000 rows of data);
Prepare an INSERT statement, then re-execute it with a bound parameter each time;
By default, FireDAC initialize SQLite3 with the fastest options (e.g. disabling LOCK), so let it be.
SQlite3 is very good about BLOB process.
From my tests, FireDAC insertion timing is pretty good, very close to direct SQlite3 access. Only reading is slower than a direct SQLite3 link, due to the overhead of the Delphi TDataSet class.
I'm trying to read a .sql file into SQLite, but I'm getting syntax errors because the file was dumped from MySQL, which can add multiple entries at once, but I'm using SQLite v3.7.7, which can't read more than one entry to a table at a time with the VALUES command.
My understanding is that I either need to upgrade SQLite, or somehow modify the file to read in one entry at a time into the tables. Please note I'm dealing with tens of thousands of entries, so inserting the UNION SELECT command probably won't be very easy.
You need at least SQLite 3.7.11 to use the VALUES syntax you're interested in. But mysqldump has about 100 command-line options. And one of them, --skip-extended-insert, can disable extended inserts. (So you get one INSERT statement per row.) Read the mysqldump documentation, and run the dump again with options that better fit your target.
Or better yet, look at the list of SQLite converter tools.
Afaik, SQLite stores a single database in a single file. Since this would decrease the performance when working with large databases, is it possible to explicitly tell SQLite not to store the whole DB in a single file and store different tables in different files instead?
I found out, that it is possible.
Use:
sqlite3.exe MainDB.db
ATTACH DATABASE 'SomeTableFile.db' AS stf;
Access the table from the other database file:
SELECT * FROM stf.SomeTable;
You can even join over several files:
SELECT *
FROM MainTable mt
JOIN stf.SomeTable st
ON (mt.id = st.mt_id);
https://www.sqlite.org/lang_attach.html
tameera said there is a limit of 62 attached databases but I never hit that limit so I can't confirm that.
The big advantage besides some special cases is that you limit the fragmentation in the database files and you can use the VACUUM command separately on each table!
If you don't need a join between these tables you can manually split the DB and say which tables are in which DB (=file).
I don't think that it's possible to let SQLite split your DB in multiple files, because you connect to a DB by telling the filename.
SQLite database files can grow quite large without any performance penalties.
The things that might degrade performance are:
file-locking contention
table size (if using indexes and issuing write queries)
Also, by default, SQLite limits the number of attached databases to 10.
Anyway, try partition your tables. You'll see that SQLite can grow enormously this way.