I am trying to write data into a non-public schema using function st_write from sf package
I cannot change the following way I connect to the db, as it will break all other functions -
create_db_connection <- function(host, dbuser, dbpassword){
drv = RPostgreSQL::PostgreSQL()
DBuser = dbuser
DBhost = host
DBport = "5432"
DBpassword = dbpassword
db = RPostgreSQL::dbConnect(drv, dbname = "DIFM", user = DBuser, host = DBhost, port = DBport, password = DBpassword)
return(db)
}
Using the above connection, I have tried the following -
1. sf::st_write(obj = obj_geom, dsn = db, layer = c(schema_name, "temp_geometrytable"), row.names = FALSE, append = TRUE)
2. sf::st_write(obj = obj_geom, dsn = db, DBI::Id(schema=schema_name, table = "temp_geometrytable"), row.names = FALSE, append = TRUE)
But all of these keep adding into the public schema, but i want to dynamically provide schema name and then add the object into the database into a non-public schema
Related
I want to add a new user in user table from R here is my way of doing it:
con<-dbConnect(odbc::odbc(),
Driver = "SQL Server",
Server = "my server adresse",
Database = "name_DataBase",
UID = "ID",
PWD = "pwd"
)
formData <- data.frame(CLE =as.integer(198),LOG ="user",PASSWORD="pwd",PROFIL = as.integer(2),ETB =as.integer(3),CREA =as.POSIXct(Sys.time(),format="%Y-%m-%d %H:%M:%S"),ETAT = "A")
Here are the different column types of my user table at the sql server level:
CLE = int identity,LOG= nvarchar,PASSWORD = varbinary,PROFIL= int,ETB = int ,CREA= smalldatime,ETAT= nvarchar
add a new line:
dbAppendTable(conn = con,
name = Id(schema = "x", table = "User"),
value = formData,
row.names = NULL)
here is the error:
Error in result_insert_dataframe(rs#ptr, values, batch_rows) :
nanodbc/nanodbc.cpp:1655: 22018: \[Microsoft\]\[ODBC SQL Server Driver\]Valeur de caractère non valide pour la spécification de la casse
I'm waiting for your help
I don't know if this is a bad configuration set up by I am testing connections to different database network settings.
Whilst I am testing the set up I want the app to be able to run with three different set ups:
The development one I have on a local machine.
On a standalone docker container on the deployment space using a docker network.
In the deployment space using the host machine network
2/3 but in a shinyproxy container setup
These all have marginally different setups, but as I am testing out which of these connection settings. The first three work, but 2 and 3 do not.
Also, 2, 3, and 4 share the same .Renviron file as I want to be able to create the set up and get it to use the first one that works according to the preferential order of set-up. This will make handover easier to clients as they can choose how secure to make it.
I am using RPostgres for the connection.
I want to be able to use the same app code for every setup, but be able to name the environmental variables differently to allow for two options in connection to be left for deployment.
Here is my plan but it seems long winded.
First, test all the connections using the different credentials and assigning a true false to a variable. Code stolen from https://github.com/brry/berryFunctions/blob/master/R/is.error.R for error testing.
admin_connect_error <-
inherits(try(dbConnect(
drv = RPostgres::Postgres(),
dbname = Sys.getenv("ADMINDBNAME"),
host = Sys.getenv("ADMINHOSTNAME"),
port = Sys.getenv("ADMINPORTNAME"),
user = Sys.getenv("ADMINUSERNAME"),
password = Sys.getenv("ADMINPASSNAME")
),
silent = TRUE)
,
"try-error")
fake_connect_error <-
inherits(try(dbConnect(
drv = RPostgres::Postgres(),
dbname = "db_name",
host = "host_name",
port = "port_name",
user = "user_name",
password = "pass_name"
),
silent = TRUE)
,
"try-error")
local_connect_error <-
inherits(tryCatch(
dbConnect(
drv = RPostgres::Postgres(),
dbname = Sys.getenv("USERDBNAME"),
host = Sys.getenv("LOCHOSTNAME"),
port = Sys.getenv("LOCPORTNAME"),
user = Sys.getenv("USERUSERNAME"),
password = Sys.getenv("USERPASSNAME")
),
silent = TRUE
),
"try-error")
docker_connect_error <-
inherits(try(dbConnect(
drv = RPostgres::Postgres(),
dbname = Sys.getenv("USERDBNAME"),
host = Sys.getenv("DOCKHOSTNAME"),
port = Sys.getenv("DOCKPORTNAME"),
user = Sys.getenv("USERUSERNAME"),
password = Sys.getenv("USERPASSNAME")
),
silent = TRUE)
,
"try-error")
Second, check which are correct and use that to set up the dbPool for the app to use.
if (local_connect_error == F) {
print("connection on local machine as host network")
pool <- dbPool(
drv = RPostgres::Postgres(),
dbname = Sys.getenv("USERDBNAME"),
user = Sys.getenv("USERUSERNAME"),
password = Sys.getenv("USERPASSNAME"),
host = Sys.getenv("LOCHOSTNAME"),
port = Sys.getenv("LOCPORTNAME")
)
} else if (docker_connect_error == F) {
print("Connecting on docker network")
pool <- dbPool(
drv = RPostgres::Postgres(),
dbname = Sys.getenv("USERDBNAME"),
host = Sys.getenv("DOCKHOSTNAME"),
port = Sys.getenv("DOCKPORTNAME"),
user = Sys.getenv("USERUSERNAME"),
password = Sys.getenv("USERPASSNAME")
)
} else if (fake_connect_error == F) {
print("connection on fake local machine as fake host network")
pool <- dbConnect(
drv = RPostgres::Postgres(),
dbname = "db_name",
host = "host_name",
port = "port_name",
user = "user_name",
password = "pass_name"
)
} else if (admin_connect_error == F) {
print("connection on admin machine as host admin")
pool <- dbConnect(
drv = RPostgres::Postgres(),
dbname = Sys.getenv("ADMINDBNAME"),
host = Sys.getenv("ADMINHOSTNAME"),
port = Sys.getenv("ADMINPORTNAME"),
user = Sys.getenv("ADMINUSERNAME"),
password = Sys.getenv("ADMINPASSNAME")
)
}else{
print("DB connection options not working - please debug")
}
This seems very longwinded. Is there a better way?
I am trying to use a pool to connect to my database in R, but I get the error:
Schema must be specified when session schema is not set
How does one specify a schema ? It seems like I need to specify it inside the pool. If that's the case, what's the parameter name for a schema?
pool <- dbPool(
drv = RJDBC::JDBC(
"xxx",
"dir_to_jar", "`"
),
dbname = "db",
schema = "schema" # this didn't work
url = url,
user = user,
password = password,
SSL = 'true'
)
pool %>% tbl("schema.table")
I tried several other methods using DBI::dbConnect combined with Id and it worked:
pool <- DBI::dbConnect(
drv = RJDBC::JDBC(
"xxx",
"dir_to_jar", "`"
),
url = url,
user = user,
password = password,
SSL = 'true'
)
# Didn't work
pool %>% tbl(dbplyr::in_schema("catalog.schema", "table"))
# Works!
s <- Id(catalog = "catalog", schema = "schema", table = "table")
df <- dbReadTable(pool, s)
I have a PostgreSQL database set up with a table and columns already defined. The primary key for the table is a combination of (Id, datetime) column. I need to periodically INSERT data for different Ids from R data.table into the database. However, if data for a particular (Id, datetime) combination already exists it should be UPDATED (overwritten). How can I do this using RPostgres or RPostgreSQL packages?
When I try to insert a data.table where some (Id, datetime) rows already exist I get an error saying the primary key constraint is violated:
dbWriteTable(con, table, dt, append = TRUE, row.names = FALSE)
Error in connection_copy_data(conn#ptr, sql, value) :
COPY returned error: ERROR: duplicate key value violates unique constraint "interval_data_pkey"
DETAIL: Key (id, dttm_utc)=(a0za000000CSdLoAAL, 2018-10-01 05:15:00+00) already exists.
CONTEXT: COPY interval_data, line 1
You can use my pg package that has upsert functionality, or just grab code for upsert from there: https://github.com/jangorecki/pg/blob/master/R/pg.R#L249
It is basically what others said in comments. Write data into temp table and then insert into destination table using on conflict clause.
pgSendUpsert = function(stage_name, name, conflict_by, on_conflict = "DO NOTHING", techstamp = TRUE, conn = getOption("pg.conn"), .log = getOption("pg.log",TRUE)){
stopifnot(!is.null(conn), is.logical(.log), is.logical(techstamp), is.character(on_conflict), length(on_conflict)==1L)
cols = pgListFields(stage_name)
cols = setdiff(cols, c("run_id","r_timestamp")) # remove techstamp to have clean column list, as the fresh one will be used, if any
# sql
insert_into = sprintf("INSERT INTO %s.%s (%s)", name[1L], name[2L], paste(if(techstamp) c(cols, c("run_id","r_timestamp")) else cols, collapse=", "))
select = sprintf("SELECT %s", paste(cols, collapse=", "))
if(techstamp) select = sprintf("%s, %s::INTEGER run_id, '%s'::TIMESTAMPTZ r_timestamp", select, get_run_id(), format(Sys.time(), "%Y-%m-%d %H:%M:%OS"))
from = sprintf("FROM %s.%s", stage_name[1L], stage_name[2L])
if(!missing(conflict_by)) on_conflict = paste(paste0("(",paste(conflict_by, collapse=", "),")"), on_conflict)
on_conflict = paste("ON CONFLICT",on_conflict)
sql = paste0(paste(insert_into, select, from, on_conflict), ";")
pgSendQuery(sql, conn = conn, .log = .log)
}
#' #rdname pg
pgUpsertTable = function(name, value, conflict_by, on_conflict = "DO NOTHING", stage_name, techstamp = TRUE, conn = getOption("pg.conn"), .log = getOption("pg.log",TRUE)){
stopifnot(!is.null(conn), is.logical(.log), is.logical(techstamp), is.character(on_conflict), length(on_conflict)==1L)
name = schema_table(name)
if(!missing(stage_name)){
stage_name = schema_table(stage_name)
drop_stage = FALSE
} else {
stage_name = name
stage_name[2L] = paste("tmp", stage_name[2L], sep="_")
drop_stage = TRUE
}
if(pgExistsTable(stage_name)) pgTruncateTable(name = stage_name, conn = conn, .log = .log)
pgWriteTable(name = stage_name, value = value, techstamp = techstamp, conn = conn, .log = .log)
on.exit(if(drop_stage) pgDropTable(stage_name, conn = conn, .log = .log))
pgSendUpsert(stage_name = stage_name, name = name, conflict_by = conflict_by, on_conflict = on_conflict, techstamp = techstamp, conn = conn, .log = .log)
}
I am trying to use dplyr with a snowflake db, using the dplyr.snowflakedb package (on GitHub). I am able to install and load the libraries, then set the classpath pointing to the latest JDBC driver (snowflake-jdbc-3.0.9.jar).
# need to load RJDBC, or error 'could not find function ".jinit"' is thrown
library(RJDBC)
library(dplyr)
library(dplyr.snowflakedb)
options(dplyr.jdbc.classpath = "drivers/snowflake-jdbc-3.0.9.jar")
When trying to setup the connection object with src_snowflakedb(), I get the following error message (I removed the account details, but they are correct in the actual code):
> nike_db <- src_snowflakedb(user = "user",
password = "user",
account = "acme",
opts = list(warehouse = "my_wh",
db = "my_db",
schema = "my_schema"))
URL: jdbc:snowflake://acme.snowflakecomputing.com:443/?account=acme&warehouse=my_wh&my_db=db&schema=my_schema
Error: 'src_sql' is not an exported object from 'namespace:dplyr'
Indeed the current version of dplyr doesn't export nor include any src_sql() function:
> dplyr:::src_sql
Error in get(name, envir = asNamespace(pkg), inherits = FALSE) :
object 'src_sql' not found
Is there any way to fix this?
Please try the below sample code :
Sys.getenv("JAVA_HOME")
Sys.setenv(JAVA_HOME="C:\\Program Files\\Java\\jdk-1.8\\jre")
Sys.getenv("JAVA_HOME")
install.packages(c("rJava"))
install.packages(c("RJDBC", "DBI", "dplyr"))
install.packages("devtools")
devtools::install_github("snowflakedb/dplyr-snowflakedb")
library(RJDBC)
library(dplyr)
library(dplyr.snowflakedb)
options(dplyr.jdbc.classpath = "C:\\Driver\\snowflake-jdbc-3.11.1.jar")
my_db <- src_snowflakedb(user = "USERNAME" , password = "PASSWORD", account = "test",host = 'test.us-east-1.snowflakecomputing.com',opts = list(warehouse = "WAREHOUSE_NAME",db='DATABASE_NAME',schema='SCHEMA_NAME'))
tbl(my_db, "TABLE_NAME")
Note :
a) If your Snowflake Account URL is like "https://test.snowflakecomputing.com" use below format :
my_db <- src_snowflakedb(user = "USERNAME" , password = "PASSWORD", account = "test", opts = list(warehouse = "WAREHOUSE_NAME",db='DATABASE_NAME',schema='SCHEMA_NAME'))
b) If your Snowflake Account URL is like "https://test.us-east-1.snowflakecomputing.com" use below format :
my_db <- src_snowflakedb(user = "USERNAME" , password = "PASSWORD", account = "test", host = 'test.us-east-1.snowflakecomputing.com', opts = list(warehouse = "WAREHOUSE_NAME",db='DATABASE_NAME',schema='SCHEMA_NAME'))
I had the same issue, and had to go back to resort to the new version of the JDBC connection via SF: you can see link here:
https://support.snowflake.net/s/article/ka131000000O5Jr/connecting-r-to-snowflake-using-the-jdbc-driver-mac-os-x
all you really need though is this:
result <- dbGetQuery(jdbcConnection, "select current_timestamp() as now")
print(result)