I'm running the database query by using sqldf in Shiny in R. But getting error.
ui.R:
observeEvent (input$uploadForTest_1, {
inFile=input$uploadForTest_1
inFileName=input$uploadForTest_1$name
file <-"tss.txt"
tmp = paste("audio/street", inFileName, sep = "/")
res <- read.csv.sql(file,header=FALSE,sql = "select * from file where V1=tmp",sep="\t")
print(res)
})
I'm successfully running the following query:
res <- read.csv.sql(file,header=FALSE,sql = "select * from file where V1='audio/street/b098.wav'",sep="\t")
But, if I run the query that is mentioned in ui.R it is giving me error that tmp column doesn't exists:
Warning: Error in result_create: no such column: tmp 86:
I dont want to use string in my query. I want to use variable name. Because I don't want to hard code string in query. Can I use variable name in query instead of string. If yes, then how can I do this? I didn't find solution to my problem in Internet. Thanks.
Preface read.csv.sql with fn$, and use '$tmp' in the SQL statement.
fn$read.csv.sql(file, sql = "select * from file where V1 = '$tmp'",
header = FALSE, sep = "\t")
See ?fn and the gsubfn vignette for more info. Note that sqldf automatically loads the gsubfn package so it will already be available.
You could use sprintf. Another option would be to paste together a string, but I find sprintf far more elegant for this task.
> tmp <- "audio/street/somefile.txt"
> tmp <- "audio/street/somefile.txt"
> "select * from file where V1=tmp"
[1] "select * from file where V1=tmp"
> sprintf("select * from file where V1='%s'", tmp)
[1] "select * from file where V1='audio/street/somefile.txt'"
Related
Having issues with pasting a variable into the query string for RMariaDB. I can return a query without paste and find the proper where statement I am looking for within the dataframe I query (ex. MIN). When I try to use a variable in the query it fails. I have searched stackoverflow up and down and read the dbgetquery docs but nothing seems to be working. I am sure it is something simple, just can't seem to find it.
library(RMariaDB)
team <- "MIN"
# This returns entire database with MIN in tm column.
filename <- dbGetQuery(conn, "select * from nhl_lab_lines_today")
# These will all give me a [1054] error.
test <- paste("select * from nhl_lab_lines_today WHERE tm = ",paste(team,collapse=", "),sep ="")
test <- paste("select * from nhl_lab_lines_today WHERE tm = team")
test <- paste("select * from nhl_lab_lines_today WHERE tm =", team,sep=" ")
filename <- dbGetQuery(conn, test)
dbGetQuery(con, paste0("select * from nhl_lab_lines_today WHERE tm = '", team ,"'"))
I'm having some trouble to read big csv files with my R, so i'm trying to use the package sqldf to read just some column or lines from the csv.
I tried this:
test <- read.csv.sql("D:\\X17065382\\Documents\\cad\\2016_mar\\2016_domicilio_mar.csv", sql = "select * from file limit 5", header = TRUE, sep = ",", eol = "\n")
but i got this problem:
Error in connection_import_file(conn#ptr, name, value, sep, eol, skip) : RS_sqlite_import: D:\X17065382\Documents\cad\2016_mar\2016_domicilio_mar.csv line 198361 expected 1 columns of data but found 2
If you're not too fussy about which package you use, data.table has a great function for doing just what you need
library(data.table)
file <- "D:\\X17065382\\Documents\\cad\\2016_mar\\2016_domicilio_mar.csv"
fread(file, nrows = 5)
Like Shinobi_Atobe said, the fread() function from data.table is working really well. If you prefer to use base R you could also use : read.csv() or read.csv2().
i.e.:
read.csv2(file_path, nrows = 5)
Also what do you mean by "big files" ? 1GB, 10GB, 100GB?
This works for me.
require(sqldf)
df <- read.csv.sql("C:\\your_path\\CSV1.csv", "select * from file where Name='Asher'")
df
I'm new at using Postgresql, and I'm having trouble populating a table I created with multiple *.csv files. I was working first in pgAdmin4, then I decide to work on RPostgreSQL as R is my main language.
Anyway, I am dealing (for now) with 30 csv files located in one folder. All have the same headers and general structure, for instance:
Y:/Clickstream/test1/video-2016-04-01_PARSED.csv
Y:/Clickstream/test1/video-2016-04-02_PARSED.csv
Y:/Clickstream/test1/video-2016-04-03_PARSED.csv
... and so on.
I tried to load all csv files by using a following the RPostgresql specific answer from Parfait. Sadly, it didn't work. My code is specified below:
library(RPostgreSQL)
dir = list.dirs(path = "Y:/Clickstream/test1")
num = (length(dir))
psql.connection <- dbConnect(PostgreSQL(),
dbname="coursera",
host="127.0.0.1",
user = "postgres",
password="xxxx")
for (d in dir){
filenames <- list.files(d)
for (f in filenames){
csvfile <- paste0(d, '/', f)
# IMPORT USING COPY COMMAND
sql <- paste("COPY citl.courses FROM '", csvfile , "' DELIMITER ',' CSV ;")
dbSendQuery(psql.connection, sql)
}
}
# CLOSE CONNNECTION
dbDisconnect(psql.connection)
I'm not understanding the error I got:
Error in postgresqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not Retrieve the result : ERROR: could not open file
" Y:/Clickstream/test1/video-2016-04-01_PARSED.csv " for reading: Invalid
argument
)
If I'm understanding correctly, there is an invalid argument in the name of my first file. I'm not very sure about it, but again I am recently using PostgreSQL and this RPostgreSQL in R. Any help will be much appreciated.
Thanks in advance!
Edit: I found the problem, but cannot solve it for some reason. When I copy the path while in the for loop:
# IMPORT USING COPY COMMAND
sql <- paste("COPY citl.courses FROM '",csvfile,"' DELIMITER ',' CSV ;")
I have the following result:
sql
[1] "COPY citl.courses FROM ' Y:/Clickstream/test1/video-2016-04-01_PARSED.csv ' DELIMITER ',' CSV ;"
This means that the invalid argument is the blank space between the file path. I've tried to change this unsuccessfully. Any help will be deeply appreciated!
Try something like this
Files <- list.files("Y:/Clickstream/test1", pattern = "*.csv", full.names = TRUE)
CSVs <- lapply(Files, read.csv)
psql.connection <- dbConnect(PostgreSQL(),
dbname="coursera",
host="127.0.0.1",
user = "postgres",
password="xxxx")
for(i in 1:length(Files)){
dbWriteTable(psql.connection
# schema and table
, c("citl", "courses")
, CSVs[i]
, append = TRUE # add row to bottom
, row.names = FALSE
)
}
Background:
I can successfully pull a particular dataset (shown in the code below) from the internet using the read.csv() function. However, when I try to utilize the sqldf package to speed up the process using read.csv.sql() it produces errors. I've tried various solutions but can't seem to solve this problem.
I can successfully pull the data and create the data frame that I want with read.csv() using the following code:
ce_data <- read.csv("http://download.bls.gov/pub/time.series/cx/cx.data.1.AllData",
fill=TRUE, header=TRUE, sep="")
To test the functionality of sqldf on my machine, I successfully tested read.csv.sql() by reading in the data as 1 variable rather than the 5 desired using the following code:
library(sqldf)
ce_data_sql1 <- read.csv.sql("http://download.bls.gov/pub/time.series/cx/cx.data.1.AllData",
sql = "select * from file")
To produce the result that I got using read.csv() but utilizing the speed of read.csv.sql(), I tried this code:
ce_data_sql2 <- read.csv.sql("http://download.bls.gov/pub/time.series/cx/cx.data.1.AllData",
fill=TRUE, header=TRUE, sep="", sql = "select * from file")
Unfortunately, it produced this error:
trying URL
'http://download.bls.gov/pub/time.series/cx/cx.data.1.AllData' Content
type 'text/plain' length 24846571 bytes (23.7 MB) downloaded 23.7 MB
Error in sqldf(sql, envir = p, file.format = file.format, dbname =
dbname, : unused argument (fill = TRUE)
I have tried various methods to address the errors, using sqldf documentation and have been unsuccessful.
Question:
Is there a solution where I can read in this table with 5 variables desired using read.csv.sql()?
The reason you are reading it in as a single variable is because you did not correctly specify the separator for the original file. Try the following, where sep = "\t", for tab-separated:
ce_data_sql2 <- read.csv.sql("http://download.bls.gov/pub/time.series/cx/cx.data.1.AllData",
sep = "\t", sql = "select * from file")
.
The error you are getting in the final example:
Error in sqldf(sql, envir = p, file.format = file.format, dbname =
dbname, : unused argument (fill = TRUE)
Is due to the fact that read.csv.sql does not accept the fill argument.
I have a query in R that extracts data from SQL Server that takes column values as command line arguments. But it does not result in any output.
library(RODBC)
argv <- commandArgs(TRUE)
dbhandle <- odbcDriverConnect('driver={SQL Server};server=<srvr_nm>; database=<db>; trusted_connection=true')
res <- sqlQuery(dbhandle, 'select * from table where col = \'argv[1]\'')
This is how I am calling it
C:\Users\uid>"C:\Program Files\R\R-3.1.0\bin\x64\Rscript.exe" --slave --vanilla "c:\R\script.R" "abc"
(even if I remove the quotation from command line argument while passing that also dies not help)
The output that I get is:
<0 rows> (or 0-length row.names)
When I saw what was getting passed had quotation mark... e.g. "abc"... The value stored in table is abc (without quotation). I tried to remove the quotation with
as.name(argv[1])
but it also did not help...
Then i inserted a value in table with quotation like "abc" (instead of abc)... but still it is not getting selected.
Can you help me in the query.
Try fn$ in the gsubfn package:
library(gsubfn)
argv <- gsub('"', '', commandArgs(TRUE)) # remove double quotes
# ...
res <- fn$sqlQuery(dbhandle, "select * from table where col = '`argv[1]`' ")
or replace last line with:
argv1 <- argv[1]
res <- fn$sqlQuery(dbhandle, "select * from table where col = '$argv1' ")
Unless sqlQuery has some special internal query parsing, the string 'argv[1]' will not evaluate to the value of argv[1] but remain unparsed.
Try any of these instead
res <- sqlQuery(dbhandle,
paste('select * from table where col = \'', argv[1], '\'', sep=""))
res <- sqlQuery(dbhandle,
sprintf('select * from table where col = \'%s\'', argv[1]))