How can I increase memory for ORE?
My server have 10G free memory, but I can't use all of them for ORE.
These commands have error:
begin
sys.rqscriptDrop('clustering_test');
sys.rqscriptcreate('clustering_test','function(n){
rm(list=ls())
cnt=30000
d=dist(data.frame(x=rnorm(cnt),
y=rnorm(cnt)))
}');
end;
(for run above R script you can run this):
select * from table(rqtableeval(cursor(select 1 n from dual),
cursor(select 1 "ore.connect" from dual),
'select 1 x, 2 y from dual','clustering_test'))
But these commands could run on R in server:
cnt=30000
d=dist(data.frame(x=rnorm(cnt),y=rnorm(cnt)))
They are same code, but one of them ran directly on R and another through ORE.
cnt depends to your free system RAM.
error in ORE has not any description. but if you decrease cnt that could run.
Related
I’m trying to make the move from SQL Server to Mariadb and I’m running into a lot of little quirks that drive me crazy at times. Chief among them are the apparent differences between executing a script from a file and pasting it into a console window. I was used to using tabs for formatting code in MSSQL, so I’m still trying to get used to the tab behavior in the Mariadb console, but at least that makes sense. The difference between var and #var is odd to me as well. The code below runs fine when execute through Pycharm, but fails when pasted into a console window. I’m running Mariadb (##version: 10.5.18-MariaDB-0+deb11u1) on a Pi Raspberry accessing it through SSH from a Windows 11 box.
All the script is doing is populating a table with random combinations of last names and male/female first names from U.S. census data for testing. Any help with the error would be greatly appreciated. I really want to understand why the difference between pasting and executing the file.
SET #loops := 10;
WHILE #loops > 0 DO
INSERT person(last_name, first_name, iso_country)
VALUES(census.random_name(0), census.random_name(FLOOR(1 + RAND() * (2 - 1 +1))), 'US');
SET #loops := #loops - 1;
END WHILE;
SELECT * FROM person;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'END WHILE' at line 1
I executed the file from the Pycharm IDE where it runs fine, and pasted the same code into the Mariadb console where the error was raised, and the insertions did not occur.
The mariadb command line client has it's own small parser, by default a semicolon is interpreted as end of statement.
With default delimiter ; your statement is splitted after first semicolon into WHILE #loops > 0 DO INSERT person(last_name, first_name, iso_country) VALUES(census.random_name(0), census.random_name(FLOOR(1 + RAND() * (2 - 1 +1))), 'US'); SET #loops := #loops - 1; and END WHILE. When sending both statements to the server, server will return 2 errors, but only last error is displayed.
Instead you should use a different delimiter:
DELIMITER $$
WHILE #loops > 0 DO
INSERT person(last_name, first_name, iso_country)
VALUES(census.random_name(0), census.random_name(FLOOR(1 + RAND() * (2 - 1 +1))), 'US');
SET #loops := #loops - 1;
END WHILE;$$
See also Delimiters
we have centos 7 machine with mariadb installed.
When I run:
SELECT h.id,
h.name,
group_concat(distinct d.name ORDER BY d.name SEPARATOR " ") AS descriptions
FROM inventar h
LEFT JOIN descriptions d ON(FIND_IN_SET(d.id, h.description_id) > 0) GROUP BY h.id,h.description_id
ORDER BY h.name asc;
ERROR 5 (HY000): Out of memory (Needed 65535816 bytes)
I read that it probably limit of the size of temporary table.
I checked the size:
MariaDB [wexac_hosts]> show variables like "%table_size%";
Variable_name
Value
max_heap_table_size
1048576000
tmp_disk_table_size
18446744073709551615
tmp_memory_table_size
12572426240
tmp_table_size
12572426240
it's bigger then 65535816 bytes.
Which mariadb variable should I increase?
If it's GROUP_CONCAT that's running out of memory, you need to increase group_concat_max_len.
From the GROUP_CONCAT documentation:
The maximum returned length in bytes is determined by the
group_concat_max_len server system variable, which defaults to 1M (>=
MariaDB 10.2.4) or 1K (<= MariaDB 10.2.3).
I was wondering if anybody had any ideas on how to set up a SQL Server Job to run r scripts? Here is what I have so far in terms of the SQL code: I have to extract data out of a database (ETL) and want that data to be aggregated/analyzed by R by a specified date. After that, the database would run automatically. Does anybody have any ideas where the SQL ETL Code (from the database) will go and where the R script procedure would go that would eventually run automatically? Thanks!
DATABASE -> ETL Code from database (generating own dataset) -> Using ONLY that dataset where an R script can then manipulate/transform it.
DECLARE #job_name NVARCHAR(128),
#description NVARCHAR(512),
#owner_login_name NVARCHAR(128),
#database_name NVARCHAR(128);
SET #job_name = N'Some Title';
SET #description = N'Periodically do something';
SET #owner_login_name = N'login';
SET #database_name = N'DATABASE';
-- Delete job if it already exists:
IF EXISTS(SELECT job_id FROM msdb.dbo.sysjobs WHERE (name = #job_name))
BEGIN
EXEC msdb.dbo.sp_delete_job
#job_name = #job_name;
END
EXEC msdb.dbo.sp_add_job
#job_name=#job_name,
#enabled=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name;
-- Add server:
EXEC msdb.dbo.sp_add_jobserver #job_name=#job_name;
-- Add step to execute SQL:
EXEC msdb.dbo.sp_add_jobstep
#job_name=#job_name,
#step_name=N'Execute SQL',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=1,
#on_fail_action=2,
#retry_attempts=0,
#retry_interval=0,
#os_run_priority=0,
#subsystem=N'TSQL',
#command=N'EXEC my_stored_procedure; -- OR ANY SQL STATEMENT',
#database_name=#database_name,
#flags=0;
-- Update job to set start step:
EXEC msdb.dbo.sp_update_job
#job_name=#job_name,
#enabled=1,
#start_step_id=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name,
#notify_email_operator_name=N'',
#notify_netsend_operator_name=N'',
#notify_page_operator_name=N'';
-- Schedule job:
EXEC msdb.dbo.sp_add_jobschedule
#job_name=#job_name,
#name=N'Daily',
#enabled=1,
#freq_type=4,
#freq_interval=1,
#freq_subday_type=1,
#freq_subday_interval=0,
#freq_relative_interval=0,
#freq_recurrence_factor=1,
#active_start_date=20170101, --YYYYMMDD
#active_end_date=99991231, --YYYYMMDD (this represents no end date)
#active_start_time=010000, --HHMMSS
#active_end_time=235959; --HHMMSS
Sql-server supports running R and python since sql-server 2014. Take a look on this.
Method before sql-server 2014
However if you are using version before 2014, you need to have some other way else. My suggestion is using R to kick start the sql (if you can).
Using library like taskscheduleR. You first make a schedule task of R for computing your data. And then connect to your sql-server by some library like obdc, dbi, etc. to update what you want.
ps. I didn't try this before, but I can make some test if you still have further problem about it.
I am collecting data via SQL query though R. I have a loop to pull small chunks of a large table, save the chunk and drop the chunk, on repeat for an hour or so till the whole table is in flat files in my RSQL directory.
However, R shoots a Cputime limit exceeded: 24 error every so often.
I am running Mountain Lion.
I have tried
nice -19n R CMD BATCH myscript.R
and the OS continues to kill the process at odd intervals. I do not believe the script to be getting stuck on a particular operation, it just takes a while to plough through the loop.
The loop looks like so..
for (i in 1:64){
foobyte <- NULL
for (j in 0:7){
max id = 1000000
rows = 1e5
to = max_id * (rows * j) - (i * 7 * rows)
from = max_id * (rows * (j-1)) - (1 * 7 * rows)
foobit <- queryDB(paste("SELECT * FROM foobar where id <= ', to,' and id > ',from,';")
foobyte <- rbind(foobit, foobyte)
}
filename <- paste("/my/data/dir/foobyte", j, ".csv", sep="")
write.table(foobyte, filename)
}
It runs for 30-90 minutes before crashing. I will try firing up R from a shell script calling ulimit in only that terminal session, and see how this works.
Tried ulimit... Appears I do not have access, even via sudo. I get the same output from
ulimit -a -H
before and after giving
ulimit -t 12000 # sets cputime limit to 12000 seconds from 600 seconds
SOLVED via Debian Virtual Machine. If someone has a Mountain Lionic solution, please let us know.
A cursory google search for "Cputime limit exceeded: 24" shows me that this is not an R specific error.
Based on the loop you've posted, I'm guessing its exceeding the cpu time limit on the queryDB call, due to the size of the chunks you're retrieving from the database.
I'm not sure if your from and to math checks out: At rows = 1e5, you're loading 1e11 ids, if you reduce it so rows = 1, you're loading 1e6 ids from the table.
Either way, try reducing size of the chunks you're loading from the database, and see if that helps
SQL. I have created 1 procedure but I am not getting the desired output. My procedure is below:
--/
CREATE OR REPLACE procedure Update_TB_INTERACTLOG
IS
BEGIN
FOR records in (select TNAME from tab where TNAME like 'TB_INTERACTLOG%' and TABTYPE = 'TABLE')
LOOP
dbms_output.put_line(records.TNAME||' modified');
END LOOP;
END;
/
There are 7 records I am getting from select query.
This I am getting in Log Output.
13:10:02 [CREATE - 0 row(s), 0.031 secs] Command processed. No rows were affected
... 1 statement(s) executed, 0 row(s) affected, exec/fetch time: 0.031/0.000 sec [0 successful, 1 warnings, 0 errors]
It looks as if you have created the procedure but not executed it. To execute it, run the following code:
exec Update_TB_INTERACTLOG;
Furthermore, you will need to turn on DBMS output in the tool you're using to run it (unless it's SQL*plus).
And please not that the procedure wasn't properly compiled (1 warnings). The procedure should probably end with:
END Update_TB_INTERACTLOG;
instead of:
END;