I tried adapting a solution for MySQL, but it turns out information_schema.innodb_table_stats is empty. SHOW INDEX FROM schema_name.table_name doesn't cut it, by only showing cardinality.
mysql.innodb_table_stats is not available until MySQL 5.6 and MariaDB 10.0. Before that...
SHOW TABLE STATUS LIKE 'tablename';
will provide Data_length, which is the amount of space taken by all indexes except the PRIMARY KEY (in the case of InnoDB). There was no way to get the sizes of individual secondary indexes. The PK is "clustered" with the data, so there is really very little space taken in addition to the data.
It is general inadvisable to have a Slave running an older version than the Master, if that is what you are doing.
What are you really looking for?
Related
There are some groups of queries that include creating tables, fields, etc. How to implement a mechanism by which a group of requests should be executed all, or if somewhere the error was canceled all? That is, the principle of transactionality with ALTER TABLE queries for example (which are committed implicitly)
You are talking about Commit and Rollback.
However if you are mixing normal SQL with DML (CREATE/ALTER/DROP etc) the DML commands have an implied COMMIT as part of there execution, you cannot avoid it. So this will likely cause you problems if mixed with your normal insert/update/delete type queries
Transactional DML is not possible until MySQL 8.0. In that version, there is a "Data Dictionary" that is stored in InnoDB tables. (I don't want to think about the bootstrap required!) The DD allows, at least in theory, the ROLLBACK of DDL actions such as ALTER and DROP TABLE. Study the 8.0 docs to see if it does enough for your needs.
In the past (pre-8.0), Many DML operations were at least reasonable "crash safe". For example, an ALTER used to copy the table over, then quickly swap in the new table. This provided a reasonably good way to recover from, a crash during an ALTER.
There have been a lot of major improvements to ALTER since 5.6. Before then, the only really "instant" alter was adding an option to an ENUM, and that had caveats. There are still things that mandate a complete, time-consuming, rebuild of the table -- such as any change to the PRIMARY KEY.
DML operations should be minimized; they are not designed for frequent use.
I tried to use clickhouse to store 4 billion data, deployed on a single machine, 48-core cpu and 256g memory, mechanical hard disk.
My data has ten columns, and I want to quickly search any column through SQL statements, such as:
select * from table where key='mykeyword'; or select * from table where school='Yale';
I use order by to establish a sort key, order by (key, school, ...)
But when I search, only the first field ordered by key has very high performance. When searching for other fields, the query speed is very slow or even memory overflow (the memory allocation is already large enough)
So ask every expert, does clickhouse support such high-performance search for each column index similar to mysql? I also tried to create a secondary index for each column through index, but the performance did not improve.
You should try to understand how works sparse primary indexes
and how exactly right ORDER BY clause in CREATE TABLE help your query performance.
Clickhouse never will works the same way as mysql
Try to use PRIMARY KEY and ORDER BY in CREATE TABLE statement
and use fields with low value cardinality on first order in PRIMARY KEY
don't try to use ALL
SELECT * ...
it's really antipattern
moreover, maybe secondary data skip index may help you (but i'm not sure)
https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-data_skipping-indexes
We heavily use mariadb sequences for key based naming series generation. We eventually realised that we have hit 5,000+ sequences and many more (40k-50K) to come.
Till now we do not see any major impact on performance, however, knowing that every sequence internally creates a table, will this cause any major impact in future?
We use desc <table> command a lot which scans information_schema.
I don't have specifics, but...
The OS probably has troubles with 50K tables -- each table is one or more file in the oS.
AUTO_INCREMENT is extremely well optimized; use that whenever practical.
Consider MariaDB's third sequencing object: pseudo tables like seq_1_to_10, which probably takes very little overhead.
I find that SHOW CREATE TABLE is more descriptive than desc. But why do you need it "a lot"? Once an hour is "rather often" for that query. (I am looking at the STATUS value Com_create_table; I suspect DESCRIBE increments that.)
Could someone explain what does the above error message mean? How can it be fixed?
Thanks
There appears to be two main causes of this error:
Bugs in the client software
The query is too large
Bugs:
Make sure that you have the latest tools installed.
I have seen this error when incompatible versions of different TTU
software components are installed, especially CLI.
Please install (or-reinstall) the latest and greatest patches of CLI.
-- SteveF
Steve Fineholtz Taradata Employee
The other reference is from the comments to the original post:
Could be the driver. I had a similar issue with JDBC drivers, which went away
when I simply switched to a different version. – access_granted
Query is too large:
This is the root of the problem, even if it is caused by the above bugs.
Check your actual SQL query size sent to the server. Usually OBDC logs or debug files will let you examine the actual SQL generated.
Some SQL generators include charsets and collations to each field, increasing the query length.
You may want to create your own SQL Query from scratch.
Avoid the following, since they can be added by using additional queries.
Indexes
Default Values
Constraints
Non-ASCII characters as Column Names.
Also, remove all whitespace except a single space.
Do not attempt to add data while creating a table; Unless, the total size of the SQL statement is less than 1 MB.
From the first reference, the maximum query size is 1MB.
On the extreme side, you can name all of your fields a single letter(or double letters...). You can rename them with Alter Table queries later.
The same goes for type; you can set the type for all of the columns as CHAR, and modify it later(before any data is added to the table).
Even after reading a lot about the fulltext index of SQLite and a question arises that I didn't see answered anywhere:
I already have a table that I want to search with the fulltext index. I would just create an extra virtual table USING FTS3 or USING FTS4 and then INSERT my data into it.
Does that then use the double storage in total? Can I use such a virtual table just like a normal table and thus preventing storing the data twice?
(I am working with SQLite on Android but this question may apply to usage on any SQLite compatible platform.)
Despite the fact you did found some details I'll try to provide detailed answer:
1. Does that then use the double storage in total?
Yes it does. Moreover it might use event more space. For example, for widely known Enron E-Mail Dataset and FTS3 example, just feel the difference:
The FTS3 table consumes around 2006 MB on disk compared to just
1453 MB for the ordinary table
The FTS3 table took just under 31
minutes to populate, versus 25 for the ordinary table
Which makes the situation a bit unpleasant, but still full-text search worth it.
2. Can I use such a virtual table just like a normal table?
The short answer no, you can't. Virtual table is just a some kind of a View with several limitations. You've noticed several already.
Generally saying you should not use any feature which is seems to be unnatural for a View. Just a bare minimum required to let your application fully utilize the power of full-text search. So there will be no surprises later, with newer version of the module.
There is no magic behind this solution, it is just a trade-off between performance, required disk space and functionality.
Final conclusion
I would highly recommend to use FTS4, because it is faster and the only drawback is additional storage space needed.
Anyway, you have to carefully design virtual table taking into account a supplementary and highly specialized nature of such solution. In the other words, do not try to replace your initial table with the virtual one. Use both with a great care.
Update
I would recommend to look through the following article: iOS full-text search with Core Data and SQLite. Several interesting moments:
The virtual table is created in the same SQLite database in wich the Core Data content resides. To keep this table as light as possible
only object properties relevant to the search query are inserted.
SQLite implementation offers something Core Data does not: full-text search. Next to that, it performs almost 10% faster and at least
660% more (memory) efficiently than a comparable Core Data query.
I just found out the main differences of virtual tables and it seems to depend on your usage whether a single table suffices for you.
One cannot create a trigger on a virtual table.
One cannot create additional indices on a virtual table. (Virtual tables can have indices but that must be built into the virtual table
implementation. Indices cannot be added separately using CREATE INDEX
statements.)
One cannot run ALTER TABLE ... ADD COLUMN commands against a virtual table.
So if you need another index on the table, you need to use two tables.