I'm using tbl_sql object in my Shiny app to have access to a database table. I've noticed that sometimes dplyr close this connection. It might be because garbage collector calls db_disconnector. Is there any way to stop this? I could close the connection on the shiny close event.
It seems like, if you d <- src_mysql(...) (I guess that's the backend you're using, and how you're connecting to the data base?) then the garbage collector will only run if d goes out of scope. Maybe its the database that is timing out connections as a way to manage load?
One way to test this is to write your own wrapper (rather than src_mysql()) that does not disconnect
src_yoursql <-
function (dbname, host = NULL, port = 0L, user = "root", password = "",
...)
{
if (!requireNamespace("RMySQL", quietly = TRUE)) {
stop("RMySQL package required to connect to mysql/mariadb",
call. = FALSE)
}
con <- DBI::dbConnect(RMySQL::MySQL(), dbname = dbname, host = host,
port = port, username = user, password = password, ...)
info <- DBI::dbGetInfo(con)
src_sql("mysql", con, info = info)
}
d = src_yoursql(...)
Close it manually with
DBI::dbDisconnect(d$con)
Related
I have an R Shiny app connecting to SQL database with the following code:
ui <- fluidPage(
column(12, align = 'center', textInput("userName", "User Name")),
column(12, align = 'center', actionButton("dbConnect", "Connect to Database")))
server <- function(session, input, output) {
observeEvent(input$dbConnect(odbc::odbc(),
driver = "ODBC Driver 17 for SQL Server",
Server = '123......',
Database = 'example',
UID = input$userName,
PWD = askpass("Enter Database Password"))
}
The code properly works for entry of username and then prompts password. However, it does not connect to the database and prompts Login failed for user 'username'.
I have ensured that the username and password are correct and connect to the database when entering SQL.
Are you trying to return a connection?
conn = eventReactive(
input$dbConnect,
{
PWD = askpass::askpass("Enter Database Password")
conn = DBI::dbConnect(
odbc::odbc(),
driver = "ODBC Driver 17 for SQL Server",
Server = '123.......',
Database = 'example',
port=1433,
UID = input$userName,
PWD = PWD
)
return(conn)
}
)
You can then use conn() in subsequent calls, for example
output$tables=renderText({
req(conn())
DBI::dbListTables(conn())
})
Your sample code is a syntax error,
observeEvent(input$dbConnect(odbc::odbc(),
driver = "ODBC Driver 17 for SQL Server",
Server = '123......',
Database = 'example',
UID = input$userName,
PWD = askpass("Enter Database Password"))
should probably be
observeEvent(input$userName, {
dbConnect(odbc::odbc(),
driver = "ODBC Driver 17 for SQL Server",
Server = '123......',
Database = 'example',
UID = input$userName,
PWD = askpass("Enter Database Password"))
})
but even that won't work.
You never store that anywhere, so no query can use it. The DBI methods such as DBI::dbGetQuery all require the object as their first argument, so you need to put it in a real object. #langtang's suggestion to use eventReactive is the right start.
You cannot use askpass in a shiny app that you intend to deploy. You'll need to use environment variables, config.yml (via the config R package), or some other ways to pass secrets to the app. (I'm going to ignore this part for the sample code below, but keep it in mind.)
My suggestion
Store the credentials in a list (as opposed to a database connection object) and include a function that connects, runs the query, then disconnects. While there is a very little overhead added for each query, it is very small.
But the bigger reason is this: if you ever plan on providing even simple async operations in your shiny app by including promise+future, the connection object cannot be transferred to future processes, so any query in a promise will fail. However, a list of parameters can transfer.
So you might try:
myGetQuery <- function(..., cred) {
tmpcon <- do.call(DBI::dbConnect, c(list(odbc::odbc()), cred))
on.exit(DBI::dbDisconnect(tmpcon), add = TRUE)
DBI::dbGetQuery(tmpcon, ...)
}
# similar funcs needed if you need things like: dbExecute,
# dbListTables, dbListFields, etc, elsewhere in your server component
cred <- list(
driver = "ODBC Driver 17 for SQL Server",
Server = '123......',
Database = 'example'
)
And somewhere in your shiny app, something like:
cred_with_userpass <- reactive({
c(cred, UID = input$userName,
PWD = askpass("Enter Database Password"))
})
somedat <- reactive({
myGetQuery("select 1 as a", cred = cred_with_userpass())
})
I am using odbc in my R to get data from SQL server.
Recently, I had an issue: For some unknown reason. My query may take hours to get the result from the SQL server. It was fine before. The return data is only 10,000 rows. My data team colleagues haven't figure out the reason. My old code was:
getSqlData = function(server, sqlstr){
con = odbc::dbConnect(odbc(),
Driver = "SQL Server",
Server = server,
Trusted_Connection = "True")
result = odbc::dbGetQuery(con, sqlstr)
dbDisconnect(con)
return(result)
}
At first, I was trying to find a timeout parameter for dbGetQuery(). Unfortunately, there is no such parameter for this function. So I decide to monitor the runtime by myself.
getSqlData = function(server, sqlstr){
con = odbc::dbConnect(odbc(),
Driver = "SQL Server",
Server = server,
Trusted_Connection = "True")
result = tryCatch(
{
a = withTimeout(odbc::dbGetQuery(con, sqlstr), timeout = 600, onTimeout = "error")
return(a)
},
error=function(cond) {
msg <- "The query timed out:"
msg <- paste(msg,sqlstr,sep = " ")
error(logger,msg)
return(NULL)
},finally={
dbDisconnect(con)
}
)
return(result)
}
I force the function to stop if dbGetQuery() didn't finish in 10 mins. However, I get warning message as
In connection_release(conn#ptr) : There is a result object still in use.
The connection will be automatically released when it is closed
My understanding is this means the query is still running and the connection is not closed.
Is there a way to force the connection to be closed and force the query to stop?
The other thing I notice is even I set timeout = 1, it will not raise the error in 1s, it will run for around 1mins and then raise the error. Does anyone know why it behaved like this?
Thank you.
I am using RStudio Server and ODBC to connect to a redshift database. I can connect easily using:
conn <- dbConnect(odbc::odbc(), Driver="redshift",
Server = SERVER_URL,
Port = "5439",
Database = DB_NAME,
PWD = PASSWORD,
UID = CREDENTIALS,
timeout = 10,
Trusted_Connection = "True")
When connected in shows up in the sidebar "connections" where I have an UI to look through the database. That is exactly what i want.
The problem is that if i call the same code inside a function, then I get the database connection but no UI?!? How do i get the UI to appear when calling this code from inside a function?
C
onnection_odbc_profile <- function(INPUT){
conn <- dbConnect(odbc::odbc(), Driver="redshift",
Server = SERVER_URL,
Port = "5439",
Database = DB_NAME,
PWD = PASSWORD,
UID = CREDENTIALS,
timeout = 10,
Trusted_Connection = "True")
return(conn)
}
I think the issue is that the connection pane only gets updated when the code is run at top-level. Is there any way to force a line of code in a function to run at top-level (or directly in the console)
I solved the problem by adding:
code <- c(match.call()) # This saves what was typed into R
odbc:::on_connection_opened(conn, paste(c(paste("con <-", gsub(", ", ",\n\t", code))), collapse = "\n"))
As I am new to developing with Shiny, I am interested in the best practices for automated database queries. At the time of writing there are a number of different sources with different information.
If I am to query my postgres database every 10 minutes as in the example below, I want to make sure that there are no issues with a) closing the connection on session exit and b) not being able to connect due to too many open connections. My dashboard will in the future have at most a dozen users at one time.
Having done some research, I am convinced that the best way to do this is not necessarily to use a pool but to use the "one connection per query" method documented by Shiny here
Is using reactivePoll() as I have below the correct way to implement a query that will refresh the rendered table every 10 minutes? The database I will be querying will definitely return different data with every call. Does that mean that checkFunc and valueFunc should be the same or can checkFunc be left as an empty function altogether ?
library(shiny)
library(DBI)
args <- list(
drv = dbDriver("PostgreSQL"),
dbname = "shinydemo",
host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
username = "guest",
password = "guest"
)
ui <- fluidPage(
textInput("ID", "Enter your ID:", "5"),
tableOutput("tbl"),
numericInput("nrows", "How many cities to show?", 10),
plotOutput("popPlot")
)
server <- function(input, output, session) {
output$tbl <- renderTable({
conn <- do.call(DBI::dbConnect, args)
on.exit(DBI::dbDisconnect(conn))
sql <- "SELECT * FROM City WHERE ID = ?id;"
query <- sqlInterpolate(conn, sql, id = input$ID)
data <- reactivePoll(10000, session,
checkFunc = function() {}
valueFunc = function() {
dbGetQuery(conn, query)
})
})
}
shinyApp(ui, server)
I recommend creating your db connection 'conn' out of any output objects.
args <- list(
drv = dbDriver("PostgreSQL"),
dbname = "shinydemo",
host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
username = "guest",
password = "guest"
)
conn <- do.call(DBI::dbConnect, args)
it could be a global environment object, like the 'args' list in your sample code, or inside the server function, queries within rendered output objects will all access the same 'conn' db connection. In my experience, disconnect was not necessary to include, after the Rsession with the Shiny app is closed the database disconnects too.
I have a postgresql database connection and want to get a table from the database. Presumably it's good practice to keep the connection info in a different file?
I have two files just now:
#getthetable.R
library(tidyverse)
library(dbplyr)
## connect to db
con <- src_postgres(dbname = "thedbname",
host = "blablabla.amazonaws.com",
port = NULL,
user = "myname",
password = "1234")
thetable <- tbl(con, "thetable") %>% select(id, apples, carrots) %>% collect
And then:
#main.R
library(tidyverse)
## get data from getthetable script with connection
source("rscripts/getthetable.R")
This now makes both con and thetable variables available in main.R. I just want the variable thetable from getthetable.R. How do I do that? Leaving out con variable?
Also, is there a best practice here when working with db connections in r? Is my thinking logical? Are there drawbacks to what I'm doing or do most people just put the connection in together with the main scripts?
I also like to capture such things (like connections) in a different file, but also in an designated environment like this:
ConnectionManager <- local({
con <- src_postgres(dbname = "thedbname",
host = "blablabla.amazonaws.com",
port = NULL,
user = "myname",
password = "1234")
collectTable <- function() {
tbl(con, "thetable") %>% select(id, apples, carrots) %>% collect
}
list(collectTable = collectTable)
})
This way you have only one object ConnectionManager after sourcing the file and can get the table with ConnectionManager$collectTable(). Additionally you can easily extend it to fetch other tables or to include some connection utility functions.