I have a little database of a few hundreds of millions of rows for storing call detail records. I setup partitioning as per:
http://www.postgresql.org/docs/9.1/static/ddl-partitioning.html
and it seemed to work pretty well until now. I have master table "acmecdr" which has rules for inserting into the correct partition and check constraints to make sure the correct table is used when selecting data. Here is an example of one of the partitions:
cdrs=> \d acmecdr_20130811
Table "public.acmecdr_20130811"
Column | Type | Modifiers
-------------------------------+---------+------------------------------------------------------
acmecdr_id | bigint | not null default
...snip...
h323setuptime | bigint |
acmesipstatus | integer |
acctuniquesessionid | text |
customers_id | integer |
Indexes:
"acmecdr_20130811_acmesessionegressrealm_idx" btree (acmesessionegressrealm)
"acmecdr_20130811_acmesessioningressrealm_idx" btree (acmesessioningressrealm)
"acmecdr_20130811_calledstationid_idx" btree (calledstationid)
"acmecdr_20130811_callingstationid_idx" btree (callingstationid)
"acmecdr_20130811_h323setuptime_idx" btree (h323setuptime)
Check constraints:
"acmecdr_20130811_h323setuptime_check" CHECK (h323setuptime >= 1376179200 AND h323setuptime < 1376265600)
Inherits: acmecdr
Now as one would expect with SET constraint_exclusion = on the correct partition should automatically be preferred and since there is an index on it there should only be one index scan.
However:
cdrs=> explain analyze select * from acmecdr where h323setuptime > 1376179210 and h323setuptime < 1376179400;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Result (cost=0.00..1435884.93 rows=94 width=1130) (actual time=138857.660..138858.778 rows=112 loops=1)
-> Append (cost=0.00..1435884.93 rows=94 width=1130) (actual time=138857.628..138858.189 rows=112 loops=1)
-> Seq Scan on acmecdr (cost=0.00..1435863.60 rows=1 width=1137) (actual time=138857.584..138857.584 rows=0 loops=1)
Filter: ((h323setuptime > 1376179210) AND (h323setuptime < 1376179400))
-> Index Scan using acmecdr_20130811_h323setuptime_idx on acmecdr_20130811 acmecdr (cost=0.00..21.33 rows=93 width=1130) (actual time=0.037..0.283 rows=112 loops=1)
Index Cond: ((h323setuptime > 1376179210) AND (h323setuptime < 1376179400))
Total runtime: 138859.240 ms
(7 rows)
So, I can see it's not scanning all the partitions, only the relevant one (which in index scan and pretty quick) and also the master table (which seems to be normal from the examples I've seen). But the high cost of the seq scan on the master table seems to be abnormal. I would love for that to come down and I see no reason for it, especially since the master table does not have any records in it:
cdrs=> select count(*) from only acmecdr;
count
-------
0
(1 row)
Unless I'm missing something obvious, this query should be quick. But it's not - it takes about 2 minutes? This does not seem normal at all (even for a slow server).
I'm out of ideas of what to try next, so if anyone has any suggestions or pointers in the right direction, it would be very much appreciated.
Related
Suppose I have a table, I run a query like below :-
let data = orders | where zip == "11413" | project timestamp, name, amount ;
inject data into newOrdersInfoTable in another cluster // ==> How can i achieve this?
There are many ways to do it. If it is a manual task and with not too much data you can simply do something like this in the target cluster:
.set-or-append TargetTable <| cluster("put here the source cluster url").database("put here the source database").orders
| where zip == "11413" | project timestamp, name, amount
Note that if the dataset is larger you can use the "async" flavor of this command. If the data size is bigger then you should consider exporting the data and importing it to the other cluster.
Join operator documentation says:
Tip
For best performance, if one table is always smaller than the
other, use it as the left (piped) side of the join.
The purpose of leftsemi in most cases is to filter a bigger set on the left by a smaller set on the right. Is the quote above still applicable to leftsemi flavor of join operator?
At least at this point, Tables' order does matter.
Here is a quick test results, executed on my dev cluster:
Setup
.set-or-replace L100M <| range i from 1 to 100000000 step 1
.set-or-replace S1M <| range i from 1 to 1000000 step 1 | project i = tolong(rand(100000000))
rightsemi (small table first)
S1M | join kind=rightsemi L100M on i | consume
Query completes in around 3 seconds
leftsemi (large table first)
L100M | join kind=leftsemi S1M on i | consume
Query runs about 20 seconds and then fails with the following exception:
Query execution lacks memory resources to complete (80DA0007): Partial
query failure: Low memory condition (E_LOW_MEMORY_CONDITION).
(message: 'bad allocation', details: '').
We have a WordPress website using MariaDB where a wp_options table keeps growing due to a rogue plugin writing thousands of records to the table. The issue has not been resolved by the plugin maintainer yet and I keep having to remove these 'transient' (temp) records manually via DELETE statement. The problem is the ibd file keeps growing and now 35GB in size. Once this is resolved, I plan to do an OPTIMIZE TABLE on the table to cleanup. Is that the best approach to reclaim all that space? I assume I'll need as much as 40GB free space to do this and how long should the OPTIMIZE TABLE take? Since this table is used quite a bit by WordPress, it seems it will be best to take the website offline while optimizing to avoid locks. I'll looking for the quickest way to resolve.
At least I think these rogue records are the cause of the table growing. Below is a list of the top 10 type of entries in the table:
MariaDB [wmnf_www]> SELECT substr(`wp_options`.`option_name`, 1, 18) AS `option_name`, count(`wp_options`.`option_value`) AS `cnt` FROM `wp_options` GROUP BY substr(`wp_options`.`option_name`, 1, 18) ORDER BY `cnt` DESC LIMIT 10;
+--------------------+-------+
| option_name | cnt |
+--------------------+-------+
| _transient_timeout | 21186 |
| _transient_ee_ssn_ | 12628 |
| _transient_jpp_li_ | 222 |
| _transient_externa | 125 |
| _transient_wc_rela | 63 |
| jpsq_sync-14716436 | 50 |
| wpmf_current_folde | 35 |
| _wc_session_expire | 34 |
| jpsq_sync-14716465 | 29 |
| jpsq_sync-14716417 | 25 |
+--------------------+-------+
10 rows in set (0.17 sec)
The _transient_ee_ssn_ and _transient_timeout_ee_ are the issue and keep growing, the only ones in the set above that has grown since last night and was initially found with 800K records. I keep removing the records as the plugin maintainer said was safe. But is this the cause of the ibd file growing?
---UPDATE---
Oddly enough, the issue is not resolved and transient records keep getting generated by the thousands, but this ibd index file has stopped growing for the moment. After steadily growing over the weekend from 20GB to now 39GB, it has not grown in a couple of hours. Perhaps there's a limit or this file was growing for other reasons?
I think it would be a better solution to recreate the table using Percona pt-online-schema-change tool. This will recreate the table and move all the data to the new table then drop the old table. This will avoid locking the database for a long time.
I am just new to PL/SQL.
I wrote a block to calculate the radius and circumference of a circle like below:
SET SERVEROUTPUT ON;
CREATE OR REPLACE PROCEDURE cal_circle AS
-- DECLARE
pi CONSTANT NUMBER := 3.1415926;
radius NUMBER := 3;
-- to make it more dynamic I can set
-- radius NUMBER := &enter_value;
circumference DECIMAL(4,2) := radius * pi * 2;
area DECIMAL(4,2) := pi * radius ** 2;
BEGIN
-- DBMS_OUTPUT.PUT_LINE('Enter a valur of radius: '|| radius);
dbms_output.put_line('For a circle with radius '
|| radius
|| ',the circumference is '
|| circumference
|| ' and the area is '
|| area
|| '.');
END;
/
Here you could see that I've commented the declare code radius NUMBER := &enter_value;, however, when I run my scripts in SQL*Plus or SQL Developer, I always got popup message like please enter the value for enter_value.
Oppositely, what if I delete this comment in the declare and just put it outside of it, then there will be no prompt anymore.
SET SERVEROUTPUT ON;
/* to make it more dynamic, I can set
radius NUMBER := &enter_value;
*/
CREATE OR REPLACE PROCEDURE cal_circle AS
-- DECLARE
pi CONSTANT NUMBER := 3.1415926;
radius NUMBER := 3;
circumference DECIMAL(4,2) := radius * pi * 2;
area DECIMAL(4,2) := pi * radius ** 2;
BEGIN
......
Here I want to clarify that does it mean the DECLARE block cannot accept comment when I try to comment a dynamic variable?
Thanks.
You can use a parameter instead of a substitution variable to allow different users to call the procedure with different values of pi.
I'd recommend using a FUNCTION instead of a PROCEDURE for this, but here's an example (Also using a parameter for radius). :
CREATE OR REPLACE PROCEDURE CAL_CIRCLE(P_RADIUS IN NUMBER, P_PI IN NUMBER) AS
CIRCUMFERENCE DECIMAL(4, 2) := P_RADIUS * P_PI * 2;
AREA DECIMAL(4, 2) := P_PI * P_RADIUS ** 2;
BEGIN
DBMS_OUTPUT.put_line('For a circle with radius '
|| P_RADIUS
|| ',the circumference is '
|| CIRCUMFERENCE
|| ' and the area is '
|| AREA
|| '.' || 'Calculated with Pi=: ' || P_PI);
END;
/
Then try it out:
BEGIN
CAL_CIRCLE(3, 3.14);
END;
/
For a circle with radius 3,the circumference is 18.84 and the area is
28.26.Calculated with Pi=: 3.14
BEGIN
CAL_CIRCLE(3, 3.14159);
END;
/
For a circle with radius 3,the circumference is 18.85 and the area is
28.27.Calculated with Pi=: 3.14159
If you really need to actually COMPILE the procedure with different values for its constants (not recommended) with a substituted value of pi, you can set the substitution variable first with DEFINE. like DEFINE pi_value = 3.1415;, then using &pi_value later.
Update: Why do SQLPlus and SQL Developer detect the Substitution Variable and request a value for it, even when it is in a comment?
TLDR: SQL Clients must deliver comments to the server. Preprocessing substitutions in comments gives greater flexibility and keeps the SQL Clients simpler. The clients have good support for controlling substitution behavior. There is not much of a reason to have orphan substitution variables in finalized code.
Longer-Version:
These tools are database clients--they have lots of features but first and foremost their first job is to gather input SQL, deliver it to the database server and handle fetched data.
Comments need to be delivered to the database server with their accompanying SQL statements. There are reasons for this -- so users can save comments on their compiled SQL code in the database, of course, but also for compiler hints.
Substitution Variables are not delivered with the SQL to the server like comments are. Instead they are evaluated first, and the resultant SQLText is sent to the server. (You can see the SQL that gets into the server has its Substitution Variables replaced with real values. See V$SQLTEXT).
Since the server "makes use" of the comments, it makes things more flexible and simplifies things for SQLPlus to replace the Substitution Variables even in comments. (If needed this can be overridden. I'll show that below). SQLPlus,SQLDeveloper, etc could have been designed to ignore Substitution Variables in comments, but that would make them less flexible and perhaps require more code since they would need to recognize comments and change their behavior accordingly, line-by-line. I'll show some example of this flexibility further below.
There is not much of a drawback to the tools working this way.
Suppose one just wants to ignore a chunk of code for a minute during development and quickly run everything. It would be annoying if one had DEFINE everything even though it wasn't used, or to delete all the commented code just so it could run. So these tools instead allow you to SET DEFINE OFF; and ignore the variables.
For example, this runs fine:
SET DEFINE OFF;
--SELECT '&MY_FIRST_IGNORED_VAR' FROM DUAL;
-- SELECT '&MY_SECOND_IGNORED_VAR' FROM DUAL;
SELECT 1919 FROM DUAL;
If one needs to use '&' itself in a query, SQLPlus lets you choose another character as the substitution marker. There are lots of options to control things.
If one has finished developing one's final query or procedure, it isn't a valid situation to have leftover "orphan" comments with undefined-substitutions. When development is complete, orphan substitutions should all be removed, and anything remaining should reference valid DEFINEd variables.
Here's an example that make use of processing substitutions in comments.
Suppose you wanted to tune some poor-performing SQL. You could use a substitution variable in the HINTs (in a comment) to allow for quickly changing which index is used, or execution mode, etc. without needing to actually change the query script.
CREATE TABLE TEST_TABLE_1(TEST_KEY NUMBER PRIMARY KEY,
TEST_VALUE VARCHAR2(128) NOT NULL,
CONSTRAINT TEST_VALUE_UNQ UNIQUE (TEST_VALUE));
INSERT INTO TEST_TABLE
SELECT LEVEL, 'VALUE-'||LEVEL
FROM DUAL CONNECT BY LEVEL <= 5000;
Normally a query predicating against TEST_VALUE here would normally use its UNIQUE INDEX when fetching the data.
SELECT TEST_VALUE FROM TEST_TABLE WHERE TEST_VALUE = 'VALUE-1919';
X-Plan:
------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 66 | 1 (0)| 00:00:01 |
|* 1 | INDEX UNIQUE SCAN| TEST_VALUE_UNQ | 1 | 66 | 1 (0)| 00:00:01 |
------------------------------------------------------------------------------------
But one can force a full-scan via a hint. By using a substitution variable in the hint (in a comment), one can allow the values of the Substitution Variable to direct query-execution:
DEFINE V_WHICH_FULL_SCAN = 'TEST_TABLE';
SELECT /*+ FULL(&V_WHICH_FULL_SCAN) */ TEST_VALUE FROM TEST_TABLE WHERE TEST_VALUE = 'VALUE-1919';
Here the Substitution Variable (in its comment) has changed the query-execution.
X-Plan:
--------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 23 | 1518 | 9 (0)| 00:00:01 |
|* 1 | TABLE ACCESS FULL| TEST_TABLE | 23 | 1518 | 9 (0)| 00:00:01 |
--------------------------------------------------------------------------------
If there were a bunch of tables here instead of one, a person could DEFINE different targets to full-scan, and evaluate each impact on the query quickly.
I have a MariaDB 10.2.8 database which I am using to store the results of a crawl of all files beneath a particular root directory. So a file (in the file table) has a parent directory (in the directory table). This parent directory may have its own parents and so on up to the original point at which the directory crawl began.
So if I did a crawl from /home, the file /home/tim/projects/foo/bar.py would have a parent directory foo, which would have a parent directory projects and so on. /home (the root of the crawl) would have a null parent.
I've got the following recursive CTE:
with recursive tree as (
select id, name, parent from directory where id =
(select parent from file where id = #FileID)
union
select d.id, d.name, d.parent from directory d, tree t
where t.parent = d.id
) select name from tree;
which (as expected) returns the results in order, where #FileID is the primary key of the file. e.g.
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 17
Server version: 10.2.8-MariaDB-10.2.8+maria~jessie-log mariadb.org binary distribution
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use inventory;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [inventory]> with recursive tree as (
-> select id, name, parent from directory where id =
-> (select parent from file where id = 3790)
-> union
-> select d.id, d.name, d.parent from directory d, tree t
-> where t.parent = d.id
-> ) select name from tree;
+----------+
| name |
+----------+
| b8 |
| objects |
| .git |
| fresnel |
| Projects |
| metatron |
+----------+
6 rows in set (0.00 sec)
MariaDB [inventory]> Bye
tim#merlin:~$
So in this case, file ID 3790 corresponds a file in the directory /metatron/Projects/fresnel/.git/objects/b8 (/metatron is, of course, the root of the crawl).
Is there a reliable way of reversing the order of the output (as I want to concatentate it together to produce the full path). I can order by id but this doesn't feel reliable as even though I know, in this case, that children will have a higher ID than their parents I can't guarantee this will always be the case in every circumstance I want to use a CTE.
(
your-current-query
) ORDER BY ...;
(If you have trouble with that, then stick SELECT ... in front, too.