I am trying to insert values into an existing table in Oracle database from R via ODBC by running the following command,
conODBC<-odbcConnect(dsn="xxxx", uid="xxxxx", pwd="xxxx", readOnly=FALSE)
sqls<-sprintf("INSERT INTO R_AAAA (AAA,AAB,AAC,AAD) VALUES (%s)",
apply(R_AAAA, 1, function(i) paste(i, collapse=",")))
lapply(sqls, function(s) sqlQuery(conODBC, s))
And I got this error,
"HY000 984 [Oracle][ODBC][Ora]ORA-00984: 列在此处不允许\n"
列在此处不允许 is chinese meaning 'Column is not allowed here'. And the variable sqls appears as follow,
>sqls
"INSERT INTO R_AAAA (AAA,AAB,AAC,AAD) VALUES
(H3E000000000000,344200402050,12, 2.347826e+01)"
Column AAA in R_AAAA is a string column. It appears to me Oracle database needs single quotes ' around a string value, 'H3E000000000000' for instance, in an insert statement. Anyone knows how to add single marks? I would like to insert rows into an existing table instead of creating a new table in Oracle with sqlSave.
Thanks
you can preprocess R_AAAA by adding ' to character columns before calling the code
R_AAAA <- data.frame(AAA="H3E000000000000", AAB=344200402050, AAC=12, AAD=2.347826e+01)
as.data.frame(lapply(R_AAAA, function(x) {
if (class(x) == "character") {
x <- paste0("'",x,"'")
}
x
}))
what R package are you for the sqlSave? There should be an option to insert new rows e.g. append=TRUE.
Preprocessing works. I use RODBC package. If I create an empty R_AAAA in oracle database and run the command in R
sqlSave(conODBC,R_AAAA,tablename="R_AAAA",append=TRUE)
R-studio crashes. There wasnt even an error message:)
Related
Basic setup is that I connect to a database A, get some data back to R, write it to another connection, database B.
The database is SQL_Latin1_General_CP1_CI_AS and I'm using encoding = "windows-1252" in connection A and B.
Display in RStudio is fine, special characters show as they should.
When I try to write the data, I get a "Cannot insert the value NULL into column".
I narrowed it down to at least one offending field: a cell with a PHI symbol, which causes the error.
How do I make it so the PHI symbol and presumably other special characters are kept the same from source to destination?
conA <- dbConnect(odbc(),
Driver = "ODBC Driver 17 for SQL Server",
Server = "DB",
Database = "serverA",
Trusted_connection = "yes",
encoding = "1252")
dbWriteTable(conB,SQL("schema.table"),failing_row, append = T)
#This causes the "cannot insert null value" error
I suggest working around this problem without dbplyr. As the overwhelming majority of encoding questions have nothing to do with dbplyr (the encoding tag has 23k questions, while the dbplyr tag has <400 questions) this may be a non-trivial problem to resolve without dbplyr.
Here are two work-arounds to consider:
Use a text file as an intermediate step
R will have no problem writing an in-memory table out to a text file/csv. And SQL server has standard ways of reading in a text file/csv. This gives you the added advantage of validating the contents of the text file before loading it into SQL.
Documentation for SQL Server BULK INSERT can be found here. This answer gives instructions for using UTF-8 encoding: CODEPAGE = '65001'. And this documentation gives instructions for unicode: DATAFILETYPE = 'widechar'.
If you want to take this approach entirely within R, it will likely look something like:
write.csv(failing_row, "output_file.csv")
query_for_creating_table = "CREATE TABLE schema_name.table_name (
col1 INT,
col2 NCHAR(10),
)"
# each column listed along with suitable data types
query_for_bulk_insert = "BULK INSERT schema_name.table_name
FROM 'output_file.csv'
WITH
(
DATAFILETYPE = 'widechar',
FIRSTROW = 2,
ROWTERMINATOR = '\n'
)"
DBI::dbExecute(con, query_for_creating_table)
DBI::dbExecute(con, query_for_bulk_insert)
Load all the non-error rows and append the final row after
I have has some success in the past using the INSERT INTO syntax. So would recommend loading the failing row using this approach.
Something like the following:
failing_row = local_df %>%
filter(condition_to_get_just_the_failing_row)
non_failing_rows = local_df %>%
filter(! condition_to_get_just_the_failing_row)
# write non-failing rows
dbWriteTable(con, SQL("schema.table"), non_failing_rows, append = T)
# prep for insert failing row
insert_query = "INSERT INTO schema.table VALUES ("
for(col in colnames(failing_row)){
value = failing_row[[col]]
if(is.numeric(value)){
insert_query = paste0(insert_query, value, ", ")
} else {
insert_query = paste0(insert_query, "'", value, "', ")
}
}
insert_query = paste0(insert_query, ");")
# insert failing row
dbExecute(con, insert_query)
other resources
If you have not seen them already, here are several related Q&A that might assist: Arabic characters, reading UTF-8, encoding to MySQL, and non-Latin characters as question marks. Though some of these are for reading data into R.
I have a Shiny app with which the user selects a table in a SQL database and then selects some columns of this table. Then the app runs this function to get the table:
selectAllGetQuery <- function(conn, columns, table){
columns <- toString(sprintf("[%s]", columns))
query <- sprintf("select %s from %s", columns, table)
print(columns) # ok
print(query) # ok
dbGetQuery(conn, query)
}
One table has a column named ACT_TEMP_°C_. When I select it, the dbGetQuery statement fails: Invalid column name 'ACT_TEMP_°C_'. However, as you can see, I included print(columns) and print(query) in the function, and these statements correctly print ACT_TEMP_°C_ in the R console. So I'm really lost. The function receives the correct name but the name changes to the wrong name in dbGetQuery.
Solved with
dbGetQuery(conn, stringi::stri_enc_tonative(query))
I am new to RSQLite.
I have an input document in text format in which values are seperately by '|'
I created a table with the required variables (dummy code as follows)
db<-dbconnect(SQLite(),dbname="test.sqlite")
dbSendQuery(conn=db,
"CREATE TABLE TABLE1(
MARKS INTEGER,
ROLLNUM INTEGER
NAME CHAR(25)
DATED DATE)"
)
However I am struck at how to import values into the created table.
I cannot use INSERT INTO Values command as there are thousands of rows and more than 20+ columns in the original data file and it is impossible to manually type in each data point.
Can someone suggest an alternative efficient way to do so?
You are using a scripting language. The deal of this is literally to avoid manually typing each data point. Sorry.
You have two routes:
1: You have corrected loaded a database connection and created an empty table in your SQLite database. Nice!
To load data into the table, load your text file into R using e.g. df <-
read.table('textfile.txt', sep='|') (modify arguments to fit your text file).
To have a 'dynamic' INSERT statement, you can use placeholders. RSQLite allows for both named or positioned placeholder. To insert a single row, you can do:
dbSendQuery(db, 'INSERT INTO table1 (MARKS, ROLLNUM, NAME) VALUES (?, ?, ?);', list(1, 16, 'Big fellow'))
You see? The first ? got value 1, the second ? got value 16, and the last ? got the string Big fellow. Also note that you do not enclose placeholders for text in quotation marks (' or ")!
Now, you have thousands of rows. Or just more than one. Either way, you can send in your data frame. dbSendQuery has some requirements. 1) That each vector has the same number of entries (not an issue when providing a data.frame). And 2) You may only submit the same number of vectors as you have placeholders.
I assume your data frame, df contains columns mark, roll, and name, corrsponding to the columns. Then you may run:
dbSendQuery(db, 'INSERT INTO table1 (MARKS, ROLLNUM, NAME) VALUES (:mark, :roll, :name);', df)
This will execute an INSERT statement for each row in df!
TIP! Because an INSERT statement is execute for each row, inserting thousands of rows can take a long time, because after each insert, data is written to file and indices are updated. Insert, enclose it in an transaction:
dbBegin(db)
res <- dbSendQuery(db, 'INSERT ...;', df)
dbClearResult(res)
dbCommit(db)
and SQLite will save the data to a journal file, and only save the result when you execute the dbCommit(db). Try both methods and compare the speed!
2: Ah, yes. The second way. This can be done in SQLite entirely.
With the SQLite command utility (sqlite3 from your command line, not R), you can attach a text file as a table and simply do a INSERT INTO ... SELECT ... ; command. Alternately, read the text file in sqlite3 into a temporary table and run a INSERT INTO ... SELECT ... ;.
Useful site to remember: http://www.sqlite.com/lang.html
A little late to the party, but DBI provides dbAppendTable() which will write the contents of a dataframe to an SQL table. Column names in the dataframe must match the field names in the database. For your example, the following code would insert the contents of my random dataframe into your newly created table.
library(DBI)
db<-dbConnect(RSQLite::SQLite(),dbname=":memory")
dbExecute(db,
"CREATE TABLE TABLE1(
MARKS INTEGER,
ROLLNUM INTEGER,
NAME TEXT
)"
)
df <- data.frame(MARKS = sample(1:100, 10),
ROLLNUM = sample(1:100, 10),
NAME = stringi::stri_rand_strings(10, 10))
dbAppendTable(db, "TABLE1", df)
I don't think there is a nice way to do a large number of inserts directly from R. SQLite does have a bulk insert functionality, but the RSQLite package does not appear to expose it.
From the command line you may try the following:
.separator |
.import your_file.csv your_table
where your_file.csv is the CSV (or pipe delimited) file containing your data and your_table is the destination table.
See the documentation under CSV Import for more information.
I can't figure out how to update an existing DB2 database in R or update a single value in it.
I can't find much information on this topic online other than very general information, but no specific examples.
library(RJDBC)
teachersalaries=data.frame(name=c("bob"), earnings=c(100))
dbSendUpdate(conn, "UPDATE test1 salary",teachersalaries[1,2])
AND
teachersalaries=data.frame(name=c("bob",'sally'), earnings=c(100,200))
dbSendUpdate(conn, "INSERT INTO test1 salary", teachersalaries[which(teachersalaries$earnings>200,] )
Have you tried passing a regular SQL statement like you would in other languages?
dbSendUpdate(conn, "UPDATE test1 set salary=? where id=?", teachersalary, teacherid)
or
dbSendUpdate(conn,"INSERT INTO test1 VALUES (?,?)",teacherid,teachersalary)
Basically you specify the regular SQL DML statement using parameter markers (those question marks) and provide a list of values as comma-separated parameters.
Try this, it worked for me well.
dbSendUpdate(conn,"INSERT INTO test1 VALUES (?,?)",teacherid,teachersalary)
You just need to pass a regular SQL piece in the same way you do in any programing langs. Try it out.
To update multiple rows at the same time, I have built the following function.
I have tested it with batches of up to 10,000 rows and it works perfectly.
# Libraries
library(RJDBC)
library(dplyr)
# Function upload data into database
db_write_table <- function(conn,table,df){
# Format data to write
batch <- apply(df,1,FUN = function(x) paste0("'",trimws(x),"'", collapse = ",")) %>%
paste0("(",.,")",collapse = ",\n")
#Build query
query <- paste("INSERT INTO", table ,"VALUES", batch)
# Send update
dbSendUpdate(conn, query)
}
# Push data
db_write_table(conn,"schema.mytable",mydataframe)
Thanks to the other authors.
I have a data frame data_frm which has following columns:
emp_id | emp_sal | emp_bonus | emp_desig_level
| | |
| | |
And I want to insert all the records present in this data frame into the database table tab1.I executed this query but I got an error:
for(record in data_frm)
{
write_sql <- paste("Insert into tab1 (emp_id,emp_sal,emp_bonus,emp_desig_level) values (",data_frm[,"emp_id"],",",data_frm[,"emp_sal"],",",data_frm[,"emp_bonus"],",",data_frm[,"emp_desig_level"],")",sep="")
r <- dbSendQuery(r,write_sql)
}
I get error as:
Error in data_frm[, "emp_id"] : incorrect number of dimensions
How do I insert all the records from the data frame into database?
NOTE: I want to insert all the records of the data frame using insert statement.
dbWriteTable(conn, "RESULTS", results2000, append = T) # to protect current values
dbWriteTable(conn, "RESULTS", results2000, append = F) # to overwrite values
From the RDBI homepage at sourceforge. Hope that helps...
In your for loop, you need to put:
data_frm[record,"column_name"]
Other wise your loop is trying to insert the entire column instead of just the particular record.
for(record in data_frm)
{
write_sql <- paste("Insert into tab1 (emp_id,emp_sal,emp_bonus,emp_desig_level) values (",data_frm[record,"emp_id"],",",data_frm[record,"emp_sal"],",",data_frm[record,"emp_bonus"],",",data_frm[record,"emp_desig_level"],")",sep="")
r <- dbSendQuery(r,write_sql)
}
Answered here
Copied one more time:
Recently I had similar issue.
Problem description: MS Server data base with scheme. The task is to save an R data.frame object to a predefined data base table without dropping it.
Problems I faced:
Some packages functions does not support schemes or require github development version installation
You can save data.frame only after drop (delete table) operation (I needed just "clear table" operation)
How I solved the issue
Using simple RODBC::sqlQuery, writing a data.frame row by row.
The solution (couple of functions) is available here or here