We have a cluster of 8 nodes running MariaDB 10.5.
Why would this statement produce different results for 2 users?
select count(*) from schemaA.table where col in (select col from schemaB.table where col = 'abc');
When I run this as root I get 0 rows.
But when I run this as userX with all privileges I get 100 rows.
Does anyone have an explanation for this?
Thanks
Related
we have centos 7 machine with mariadb installed.
When I run:
SELECT h.id,
h.name,
group_concat(distinct d.name ORDER BY d.name SEPARATOR " ") AS descriptions
FROM inventar h
LEFT JOIN descriptions d ON(FIND_IN_SET(d.id, h.description_id) > 0) GROUP BY h.id,h.description_id
ORDER BY h.name asc;
ERROR 5 (HY000): Out of memory (Needed 65535816 bytes)
I read that it probably limit of the size of temporary table.
I checked the size:
MariaDB [wexac_hosts]> show variables like "%table_size%";
Variable_name
Value
max_heap_table_size
1048576000
tmp_disk_table_size
18446744073709551615
tmp_memory_table_size
12572426240
tmp_table_size
12572426240
it's bigger then 65535816 bytes.
Which mariadb variable should I increase?
If it's GROUP_CONCAT that's running out of memory, you need to increase group_concat_max_len.
From the GROUP_CONCAT documentation:
The maximum returned length in bytes is determined by the
group_concat_max_len server system variable, which defaults to 1M (>=
MariaDB 10.2.4) or 1K (<= MariaDB 10.2.3).
(be Kind, this is my first question and I did extensive Research here and on the net beforehand. Question Oracle ROWID for Sqoop Split-By Column did not really solve this issue, as the original Person asking resorted to using another column)
I am using sqoop to copy data from an Oracle 11 DB.
Unfortunately, some tables have no index, no Primary key, only partitions (date). These tables are very large, hundreds of millions if not billions of rows.
so far, I have decided to Access data in the source by explicitly adressing the partitions. That works well and Speeds up the process nicely.
I need to do the splits by data that resides in each and every table in order to avoid too many if- branches in my bash script. (we're talking some 200+ tables here)
I notice that a split by 8 Tasks results in very uneven spread of workload among the Tasks. I considered using Oracle ROWID to define the split.
To do this, I must define a boundary-query. In a Standard query 'select * from xyz' the rowid is not part of the result set. therefore, it is not an option to let Sqoop define the boundary-query from --query.
Now, when I run this, I am getting the error
ERROR tool.ImportTool: Encountered IOException running import job:
java.io.IOException: Sqoop does not have the splitter for the given SQL
data type. Please use either different split column (argument --split-by)
or lower the number of mappers to 1. Unknown SQL data type: -8
samples of ROWID :
AAJXFWAKPAAOqqKAAA
AAJXFWAKPAAOqqKAA+
AAJXFWAKPAAOqqKAA/
it is static and unique once it is created for any row.
I cast this funny datatype into something else in my boundary-query
sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect
jdbc:oracle:thin:#127.0.0.1:port:mydb --username $USER --P --m 8
--split-by ROWID --boundary-query "select cast(min(ROWID) as varchar(18)), cast
( max(ROWID)as varchar(18)) from table where laufbzdt >
TO_DATE('2019-02-27', 'YYYY-MM-DD')" --query "select * from table
where laufbzdt > TO_DATE('2019-02-27', 'YYYY-MM-DD') and \$CONDITIONS "
--null-string '\\N'
--null-non-string '\\N'
But then I get ugly ROWIDs that are rejected by Oracle:
select * from table where laufbzdt > TO_DATE('2019-02-27', 'YYYY-MM-DD')
and ( ROWID >= 'AAJX6oAG聕聁AE聉N:' ) AND ( ROWID < 'AAJX6oAH⁖⁁AD䁔䀷' ) ,
Error Msg = ORA-01410: invalid ROWID
how can I resolve this properly?
I am a LINUX-Embryo and have painfully chewed myself through the Topics of bash-shell-scripting and Sqooping so far, but I would like to make better use of evenly spread mapper-task workload - it would cut sqoop-time in half, I guess, saving some 5 to 8 hours.
TIA!
wahlium
You can try ROWNUM, but I think sqoop import does not work with pseudocolumn.
I'm writing an SQLite select statement and want to pick out the first hit only that satisfy my criterion.
My problem is that I'm writing code inside a simulation framework that wraps my SQLite code before sending it to the database, and this wrapping already adds 'LIMIT 100' to the end of the code.
What I want to do:
SELECT x, y, z FROM myTable WHERE a = 0 ORDER BY y LIMIT 1
What happens when this simulation development framework has done its job:
SELECT x, y, z FROM myTable WHERE a = 0 ORDER BY y LIMIT 1 LIMIT 100
exec error near "LIMIT": syntax error
So my question is: How do I work around this limitation? Is there any way to still limit my results to give only one hit back despite that the statement will end in 'LIMIT 100'? I'm thinking something like creating a temporary table, add an index and filter on that, but my knowledge is limited to simple database queries.
Using SQLite 3.19.2, I'm running into an odd situation.
One of the queries my application performs takes an enormously long time (100+ seconds) when running inside my app. Using the sqlite3 shell, the same query takes 0.5s.
I'm using a custom build of SQLite, statically linked into my app. The version of the shell is from my custom compilation, so this isn't an issue with the compilation.
I was using multiple threads however I've since managed to reproduce this issue single-threaded.
Using perf I've determined that the majority of the CPU time is spent in sqlite3VdbeExec and not in any of my code (for instance code that reads the fields of each returned row.)
The query is sqlite3_prepare_v2'd with bound parameters. I've reproduced the query below as well as a similar query that doesn't exhibit the performance problem.
Has anyone else seen anything like this?
SLOW QUERY (100+s in the app, 0.5s in the shell):
SELECT DISTINCT
Track.*
FROM
TrackGenres,
TrackFirstArtist,
Track
WHERE
TrackFirstArtist.id = Track.id AND
TrackGenres.id = Track.id AND
TrackGenres.genreID = 328
ORDER BY
(CASE WHEN 1 = 1 THEN TrackFirstArtist.artistName COLLATE ENGLISH END) ASC,
(CASE WHEN 1 != 1 THEN TrackFirstArtist.artistName COLLATE ENGLISH END) DESC
LIMIT 50
OFFSET 0;
QUERY PLAN (the plans are identical for whether run in the app or in the shell):
2 0 0 SEARCH TABLE TrackGenre USING COVERING INDEX sqlite_autoindex_TrackGenre_1 (genreID=?)
3 0 1 SEARCH TABLE WorkGenre USING COVERING INDEX sqlite_autoindex_WorkGenre_1 (genreID=?)
3 1 0 SEARCH TABLE TrackWork USING COVERING INDEX sqlite_autoindex_TrackWork_1 (workID=?)
1 0 0 COMPOUND SUBQUERIES 2 AND 3 USING TEMP B-TREE (UNION)
4 0 0 SCAN TABLE TrackArtist USING INDEX idx_TrackArtist_trackID
4 1 1 SEARCH TABLE Artist USING INTEGER PRIMARY KEY (rowid=?)
0 0 0 SCAN SUBQUERY 1
0 1 2 SEARCH TABLE Track USING INTEGER PRIMARY KEY (rowid=?)
0 2 1 SEARCH SUBQUERY 4 USING AUTOMATIC COVERING INDEX (id=?)
0 0 0 USE TEMP B-TREE FOR DISTINCT
0 0 0 USE TEMP B-TREE FOR ORDER BY
SIMILAR QUERY (0.5s in app, 0.5s in shell):
SELECT
COUNT (Track.id)
FROM
TrackGenres,
TrackFirstArtist,
Track
WHERE
TrackFirstArtist.id = Track.id AND
TrackGenres.id = Track.id AND
TrackGenres.genreID = 328 AND
( TrackFirstArtist.artistName >= 'a' AND TrackFirstArtist.artistName < 'b' );
I believe this is a bug in SQLite. I tracked the problem down to the CASE statements in the ORDER BY clause when using the query in a prepared statement.
When I removed one or the other of the statements, the query ran quickly again. The collation didn't make any difference.
This doesn't seem to be a general issue with prepared statements, I have other queries with a similar structure and they work properly. As such, I've been unable to create a simple, reproducible example that illustrates the problem.
Ultimately I had to solve this by replacing the variables in the query manually to avoid the use of a prepared statement.
I have a dev environment running sqlite 3.7.16.2 and a production environment running sqlite 3.7.9 and I am running into some unexpected backwards incompatability.
I have a table that looks like this:
sqlite> select * from calls;
ID|calldate|calltype
1|2013-10-01|monthly
1|2013-11-01|3 month
1|2013-12-01|monthly
2|2013-07-11|monthly
2|2013-08-11|monthly
2|2013-09-11|3 month
2|2013-10-11|monthly
2|2013-11-11|monthly
3|2013-04-22|monthly
3|2013-05-22|monthly
3|2013-06-22|3 month
3|2013-07-22|monthly
4|2013-10-04|monthly
4|2013-11-04|3 month
4|2013-12-04|monthly
5|2013-10-28|monthly
5|2013-11-28|monthly
With the newer version of sqlite (3.7.16.2) I can use this:
SELECT ID, MIN(calldate), calltype FROM calls WHERE calldate > date('NOW') GROUP BY ID;
which gives me:
ID|MIN(calldate)|calltype
1|2013-11-01|3 month
2|2013-11-11|monthly
4|2013-11-04|3 month
5|2013-10-28|monthly
However when I run this same code on the older version of sqlite (3.7.9) I get this:
ID|MIN(calldate)|calltype
1|2013-11-01|monthly
2|2013-11-11|monthly
4|2013-11-04|monthly
5|2013-10-28|monthly
I looked through the changes here, but could not figure out why this is still happening. Any suggestions on how to work around this or how to rewrite my query?
You are using an extension that was added in SQLite 3.7.11.
In standard SQL, it is not allowed to use columns that appear neither in the GROUP BY clause nor are wrapped in an aggregate function.
(SQLite accepts this silently for compatibility with MySQL, but returns the data from some random record in the group.)
To get other columns from a record with the minimum value, you have to search the minimum values for each group first, and then to join these with the original table:
SELECT calls.ID,
calls.calldate,
calls.calltype
FROM calls
JOIN (SELECT ID,
MIN(calldate) AS calldate
FROM calls
WHERE calldate > date('now')
GROUP BY ID
) AS earliest
ON calls.ID = earliest.ID AND
calls.calldate = earliest.calldate