I've established a successful connection to a Google BigQuery database in R using the following code
library(tidyverse)
library(bigrquery)
con <- DBI::dbConnect(bigrquery::bigquery(),
project = project,
dataset = dataset,
billing = billing)
But when I try to run a simply dplyr query such as:
tbl(con, "table_name") %>%
summarise(total = n())
I receive the following error and warning:
Error: (L4:10): GROUP BY cannot refer to an empty field name
In addition: Warning message:
Translator is missing window variants of the following aggregate
functions:
* %||%
I figured out if I add the argument use_legacy_sql = F to the dbConnect function, I can solve for the error but I still receive the warning
total
<int>
1 6259914
Warning message:
Translator is missing window variants of the following aggregate
functions:
* %||%
I've never seen any information about needing to specify a legacy sql argument in the connection for dplyr to function, and the fact that I'm still getting a warning tells me something is still wrong with my connection. I've received the same warning when I establish a connection with the src_bigquery function in the bigrquery package.
Related
I'm trying to upload a data frame into SQL, but I keep receiving errors. I've tried the following funciont
DBI::dbWriteTable(rposgre_rsql, name = "my_schema.the_table", value = base, row.names = FALSE, append=TRUE)
that functions returns the error
RPosgreSQL error: could not Retrieve the result : ERROR: syntax error at or near "STDIN"
so i tried:
insert\<- "INSERT INTO my_schema.the_table VALUES base"
M2_results\<- RPostgreSQL::dbSendQuery(conn= rposgre_rsql,statement= insert)
but it returns
RPosgreSQL error: could not Retrieve the result : ERROR: syntax error at or near "base"
I'm positive the connection works, since I can either select the table or use "dbExistsTable.R", but I don't understand why it doesn't work with INSERT INTO. The connection is for a corporate enviroment, maybe it's a permission issue? Also, I don't quite understand what's "STDIN" is.
I have bigquery some table. bq_table_download functions work for most of the table, however, it shows failed to parse errors with 1 table.
bigrquery::bq_auth()
sql_query=paste0("select * from `project_id.dataset.table_name`")
bq_table_ext=bq_table_download(bq_project_query(project_id, sql_query))
In order to solve this I have also used as suggested bq_table_download have download quota limit.
options(scipen = 20)
However, still, I am unable to figure what is an issue.
Here is snapshot of error code
Error in bq_parse_files(schema_path, c(path_first_chunk, chunk_plan$dat$path), :
Failed to parse '/tmp/Rtmpjd5eQN/bq-download-8650ad590.json'
Calls: ... master_table_base -> bq_table_download -> bq_parse_files
In addition: Warning messages:
1: In writeBin(result$content, con) : problem writing to connection
2: In writeBin(result$content, con) : problem writing to connection
Execution halted
I am using Blogdown to create a new post and I am getting the following error when trying to preview.
The code works well in my Rmarkdown file but I cannot update it to my blog. Do anyone know where the problem is?
Quitting from lines 36-47
Error in UseMethod("select_") :
no applicable method for 'select_' applied to an object of class "function"
Calls: local ... freduce -> -> select -> select.default -> select_
Execution halted
Here is my code in lines 36-47;
library(corrplot)
library(RColorBrewer)
library(tidyverse)
corrplot(cor(df %>% select(Sales, Customers, Store,
Open, SchoolHoliday,
DayOfWeek, month, year,
CompetitionDistance,
Promo, Promo2_active) %>%
filter(!is.na(Sales), !is.na(CompetitionDistance))),
type="upper", order="original",
col=brewer.pal(n=8, name="RdYlBu"))
Thanks a lot.
I think you're getting this error because you don't have an object called df in your global environment. Either your data frame hasn't been created yet or it is called something else. There is a little-known function called df in the stats package, which is on the search path when you start an R session. You can check this by starting a new R session and typing df into the console. You will see the body of the function stats::df.
You are therefore getting the error because you are trying to subset a function, not a data frame. To resolve the error, make sure you create a data frame called df before your call to corrplot
I am trying to transfer data from the Thingspeak API into a postgres database. The API limits each request to 8000 observations, but I need to pull millions! I'm using R to iteratively pull from the API, do a bunch of wrangling, and then submit the results as a data.frame to my table within the db.
The current way I am doing this relies on the dbWriteTable() function from the RPostgres package. However, this method does not account for existing observations in the db. I have to manually DELETE FROM table_name before running the script or I'll end up writing duplicate observations each time I try to update the db. I'm still wasting time re-writing observations that I deleted, and the script takes ~2 days to complete because of this.
I would prefer a script that incorporates the functionality of postgres-9.5' ON CONLFICT DO NOTHING clause, so I don't have to waste time re-uploading observations that are already within the db. I've found the st_write() and st-read() functions from the sf packages to be useful for running SQL queries directly from R, but have hit a roadblock. Currently, I'm stuck trying to upload the 8000 observations within each df from R to my db. I am getting the following error:
Connecting to database:
# db, host, port, pw, and user are all objects in my R environment
con <- dbConnect(drv = RPostgres::Postgres()
,dbname = db
,host = host
,port = port
,password = pw
,user = user)
Current approach using RPostgres:
dbWriteTable(con
,"table_name"
,df
,append = TRUE
,row.names = FALSE)
New approach using sf:
st_write(conn = conn
,obj = df
,table = 'table_name'
,query = "INSERT INTO table_name ON CONFLICT DO NOTHING;"
,drop_table = FALSE
,try_drop = FALSE
,debug = TRUE)
Error message:
Error in UseMethod("st_write") :
no applicable method for 'st_write' applied to an object of class "c('tbl_df', 'tbl', 'data.frame')"
Edit:
Converting to strictly a dataframe, i.e. df <- as.data.frame(df) or attributes(df)$class <- "data.frame", resulted in a similar error message, only without the tbl_df or tbl classes.
Most recent approach with sf:
I'm making some progress with using st_write() by changing to the following:
# convert geom from WKT to feature class
df$geom <- st_as_sfc(df$geom)
# convert from data.frame to sf class
df <- st_as_sf(df)
# write sf object to db
st_write(dsn = con # changed from drv to dsn argument
,geom_name = "geom"
,table = "table_name"
,query = "INSERT INTO table_name ON CONFLICT DO NOTHING;"
,drop_table = FALSE
,try_drop = FALSE
,debug = TRUE
)
New Error:
Error in result_create(conn#ptr, statement) :
Failed to fetch row: ERROR: type "geometry" does not exist at character 345
I'm pretty sure that this is because I have not yet installed the PostGIS extension within my PostgreSQL database. If anyone could confirm I'd appreciate it! Installing PostGIS a pretty lengthy process, so I won't be able to provide an update for a few days. I'm hoping I've solved the problem with the st_write() function though!
I'm pulling in data from a api and keep getting the following error.
I put together the sql query and am connecting to the instance for pulling the data. However, when I run collect, it given me an error.
soql_query = paste("SELECT Id, subject FROM Table")
myDF2 <- read.df(sqlContext, source="...", username=sf_username, password=sf_password, version=apiVersion, soql=soql_query)
temp2 <- SparkR::collect(myDF2)
Error in rawToChar(string) :
embedded nul in string: 'VOID: \xe5,nq\b\x92ƹ\xc8Y\x8b\n\nAdd a new comment by Asako:\0\xb3\xe1\xf3Ȓ\xfd\xa0\bE\xe4\t06/29 09:23'
In addition: Warning message:
closing unused connection 6 (col)
I've gone through and identified what column it is. It contains a lot of string data and sentences, so the error partially makes sense.
I was wondering if there was any way to get around this issue.