I have a problem with importing data from edb postgres into R (with RPostgres). I using this code:
library(RPostgres)
conn <- dbConnect(Postgres(), dbname = 'xx', host = 'xx', port = 'xx',
user = 'xx', password = 'xx',
options="-c search_path=xx")
a=dbGetQuery(conn, "SELECT * FROM xx")
After importing data into R is all Ok exept all column with date which become corruped. A few exemples:
postgres value: 2009-11-02 after importing in R: 0231-08-11
postgres value: 2009-08-12 after importing in R: 1206-01-15
data type in postgres for this column is date. If I read from postgres column (date) with query:
".. to_char(column_with_date, 'DD-MM-YYYY') as column_with_date .."
then I get ok data in R but data type in R is not date but character.
I tried the same with importing the same data from mysql base into R (with RMySQL) and in this case data are noncorrupted . Used code:
library(RMySQL)
conn <- dbConnect(MySQL(), user = "xx", password = "xx", dbname = "xx", host = "xx" )
a=dbGetQuery(conn,"select* from xx ")
Thank you
Related
I have a table in Postgres database with some columns and values. I have imported this table into local memory, performed some computation on these columns and have a data frame with new values . Now I want this updated data frame to be placed back in the database in the same table.
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, host = "*****", port = "****",
dbname = "sai", user = "sai", password = "password")
saix_account_demo <- dbReadTable(con = con, name = "saix_account")
...
dbWriteTable(con, name = "saix_account", value = saix_account_demo,
row.names=FALSE, append=TRUE)`
I have performed dbWrtiteTable() with append==T and overwrite ==F. But I am facing an error saying primary key constraint violated. I understood the problem that I was trying to insert data instead of updating.
I have table 'df1' in mysql , now i am trying to append next set of rows (df2) to the df1,
I am trying to append the df2 to df1 as below
connection <- dbConnect(MySQL(), user = 'root', password = 'pass',
host = 'localhost', dbname = 'data')
dbWriteTable(connection,"df1",df2,row.names=T,append=T)
ERROR:could not run statement: Table 'df1' already exists
please suggest Any modifications in the above code
Thanks in advance
The following works fine for me:
library(RMySQL)
library(stringi)
con <- dbConnect(MySQL(), user = 'test', password = '****',
host = 'localhost', dbname = 'test')
dbf1 <- data.frame(x=round(runif(500, 1, 1000),0),
y=rnorm(500, mean=100, sd=21),
z=stri_rand_strings(500, 5),
stringsAsFactors = FALSE)
dbWriteTable(con, "test1", dbf1)
dbDisconnect(con)
and then:
library(RMySQL)
library(stringi)
con <- dbConnect(MySQL(), user = 'test', password = '****',
host = 'localhost', dbname = 'test')
dbf2 <- data.frame(x=round(runif(300, 10000, 11000),0),
y=rnorm(300, mean=100, sd=21),
z=stri_rand_strings(300, 5),
stringsAsFactors = FALSE)
dbWriteTable(con, "test1", dbf2, append = TRUE)
dbGetQuery(con, "SELECT count(x) FROM test1")
dbDisconnect(con)
The query returns:
count(x)
1 800
showing that the second set of 300 rows has been appended as expected.
You need to give a working example of the error that we can run. The immediate thing that springs to mind is that using T and F as abbreviations for TRUE and FALSE is bad practice! Whilst TRUE and FALSE are reserved words in R and cannot be changed, T and F are ordinary variables and can be modified. So, without seeing your whole script, there is no guarantee that row.names=T,append=T actually means that append=TRUE!
Using R, I tried to insert a data frame. My script looks like below:
con <- dbConnect(RMySQL::MySQL(), username = "xxxxxx", password = "xxxxxx",host = "127.0.0.1", dbname = "xxxxx")
dbWriteTable(conn=con,name='table',value=as.data.frame(df), append = TRUE, row.names = F)
dbDisconnect(con)
WHen the script hit below line:
dbWriteTable(conn=con,name='table',value=as.data.frame(df), append = TRUE, row.names = F)
I got an error like below:
Error in .local(conn, statement, ...) : could not run statement: Invalid utf8 character string: 'M'
I am not sure why this error occurred. This is a part of script that has been run well on another machine.
Please advise
You should create proper connection, then only can insert data frame to your DB.
for creating connection username & password, host name & data base name should correct. same code only but i removed some parameter
try this:
mydb = dbConnect(MySQL(), user='root', password='password', dbname='my_database', host='localhost')
i just insert Iris data in my_database
data(iris)
dbWriteTable(mydb, name='db', value=iris)
i inserted iris data frame in the name of db in my_database
When I read table from Postgre SQL database I get encoded polish letters. The database has encoding UTF8 and collation Polish_Poland.1250.
Code:
library(PostgreSQL)
# make connection
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, host="localhost", port=xxxx, user="user", dbname="myBase")
# read tables and assign to variables
table <- dbReadTable(con, c("public","table"))
Results:
"BYDGOSZCZ" "wĹ‚ocĹ‚awski" "WROCĹAW"
"gostyĹ„ski" "nowotarski" "MOGILEĹSKI" "myĹ›liborski"
Should be:
"BYDGOSZCZ" "włocławski" "WROCŁAW"
"gostyński" "nowotarski" "MOGILEŃSKI" "myśliborski"
I found the solution!!! This line should be added after the connection:
postgresqlpqExec(con, "SET client_encoding = 'windows-1250'").
I'm trying to upload a dataframe to a SQL Server table, I tried breaking it down to a simple SQL query string..
library(RODBC)
con <- odbcDriverConnect("driver=SQL Server; server=database")
df <- data.frame(a=1:10, b=10:1, c=11:20)
values <- paste("(",df$a,",", df$b,",",df$c,")", sep="", collapse=",")
cmd <- paste("insert into MyTable values ", values)
result <- sqlQuery(con, cmd, as.is=TRUE)
..which seems to work but does not scale very well. Is there an easier way?
[edited] Perhaps pasting the names(df) would solve the scaling problem:
values <- paste( " df[ , c(",
paste( names(df),collapse=",") ,
")] ", collapse="" )
values
#[1] " df[ , c( a,b,c )] "
You say your code is "working".. I would also have thought one would use sqlSave rather than sqlQuery if one wanted to "upload".
I would have guessed this would be more likely to do what you described:
sqlSave(con, df, tablename = "MyTable")
This worked for me and I found it to be simpler.
library(sqldf)
library(odbc)
con <- dbConnect(odbc(),
Driver = "SQL Server",
Server = "ServerName",
Database = "DBName",
UID = "UserName",
PWD = "Password")
dbWriteTable(conn = con,
name = "TableName",
value = x) ## x is any data frame
Since insert INTO is limited to 1000 rows, you can dbBulkCopy from rsqlserver package.
dbBulkCopy is a DBI extension that interfaces the Microsoft SQL Server popular command-line utility named bcp to quickly bulk copying large files into table. For example:
url = "Server=localhost;Database=TEST_RSQLSERVER;Trusted_Connection=True;"
conn <- dbConnect('SqlServer',url=url)
## I assume the table already exist
dbBulkCopy(conn,name='T_BULKCOPY',value=df,overwrite=TRUE)
dbDisconnect(conn)