I'm trying to run sparklyr from my local environment to replicate a production environment. However, I can't even get started. I successfully installed the latest version of Spark using spark_install(), but when trying to run spark_connect() I get this vague and unhelpful error.
> library(sparklyr)
> spark_installed_versions()
spark hadoop dir
1 2.3.1 2.7 C:\\Users\\...\\AppData\\Local/spark/spark-2.3.1-bin-hadoop2.7
> spark_connect(master = "local")
Error in if (is.na(a)) return(-1L) : argument is of length zero
Here is what my session info looks like.
> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] sparklyr_0.8.4.9003
loaded via a namespace (and not attached):
[1] Rcpp_0.12.17 dbplyr_1.2.1 compiler_3.5.0 pillar_1.2.3 later_0.7.3
[6] plyr_1.8.4 bindr_0.1.1 base64enc_0.1-3 tools_3.5.0 digest_0.6.15
[11] jsonlite_1.5 tibble_1.4.2 nlme_3.1-137 lattice_0.20-35 pkgconfig_2.0.1
[16] rlang_0.2.1 psych_1.8.4 shiny_1.1.0 DBI_1.0.0 rstudioapi_0.7
[21] yaml_2.1.19 parallel_3.5.0 bindrcpp_0.2.2 stringr_1.3.1 dplyr_0.7.5
[26] httr_1.3.1 rappdirs_0.3.1 rprojroot_1.3-2 grid_3.5.0 tidyselect_0.2.4
[31] glue_1.2.0 R6_2.2.2 foreign_0.8-70 reshape2_1.4.3 purrr_0.2.5
[36] tidyr_0.8.1 magrittr_1.5 backports_1.1.2 promises_1.0.1 htmltools_0.3.6
[41] assertthat_0.2.0 mnormt_1.5-5 mime_0.5 xtable_1.8-2 httpuv_1.4.3
[46] config_0.3 stringi_1.1.7 lazyeval_0.2.1 broom_0.4.4
Well, with a bit of guessing I was able to solve my problem. I had to specify the "SPARK_HOME" environment manually.
spark_installed_versions()[1, 3] %>% spark_home_set()
Related
sessionInfo() prints the below:
R version 3.4.3 (2017-11-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] ggmap_2.7.900 gdtools_0.1.7 bindrcpp_0.2 crosstalk_1.0.0
[5] leaflet_1.1.0 networkD3_0.4 d3heatmap_0.6.1.2 dygraphs_1.1.1.4
[9] ggiraph_0.4.2 plotly_4.7.1 ggplot2_2.2.1.9000 highcharter_0.5.0
[13] formattable_0.2.0.1 DT_0.4 kableExtra_0.7.0 stringr_1.3.0
[17] tidyr_0.8.0 dplyr_0.7.4 markdown_0.8 bookdown_0.7
[21] rmarkdown_1.9 knitr_1.20
loaded via a namespace (and not attached):
[1] nlme_3.1-131.1 bitops_1.0-6 xts_0.10-2 devtools_1.13.5
[5] lubridate_1.7.3 RColorBrewer_1.1-2 httr_1.3.1 rprojroot_1.3-2
[9] tools_3.4.3 backports_1.1.2 R6_2.2.2 lazyeval_0.2.1
[13] colorspace_1.3-2 sp_1.2-7 withr_2.1.2 mnormt_1.5-5
[17] git2r_0.21.0 curl_3.1 compiler_3.4.3 rvest_0.3.2
[21] xml2_1.2.0 officer_0.2.2 labeling_0.3 scales_0.5.0.9000
[25] psych_1.7.8 readr_1.1.1 digest_0.6.15 foreign_0.8-69
[29] R.utils_2.6.0 jpeg_0.1-8 base64enc_0.1-3 pkgconfig_2.0.1
[33] htmltools_0.3.6 maps_3.2.0 highr_0.6 rvg_0.1.8
[37] htmlwidgets_1.0 rlang_0.2.0.9001 TTR_0.23-3 quantmod_0.4-12
[41] xaringan_0.6 shiny_1.0.5 bindr_0.1.1 zoo_1.8-1
[45] jsonlite_1.5 zip_1.0.0 R.oo_1.21.0 magrittr_1.5
[49] rlist_0.4.6.1 geosphere_1.5-7 Rcpp_0.12.16 munsell_0.4.3
[53] proto_1.0.0 R.methodsS3_1.7.1 stringi_1.1.7 yaml_2.1.18
[57] plyr_1.8.4 grid_3.4.3 parallel_3.4.3 lattice_0.20-35
[61] mapproj_1.2-5 hms_0.4.2 pillar_1.2.1 igraph_1.2.1
[65] uuid_0.1-2 rjson_0.2.15 reshape2_1.4.3 glue_1.2.0
[69] evaluate_0.10.1 data.table_1.10.4-3 png_0.1-7 httpuv_1.3.6.2
[73] RgoogleMaps_1.4.1 gtable_0.2.0 purrr_0.2.4 assertthat_0.2.0
[77] xfun_0.1 mime_0.5 xtable_1.8-2 broom_0.4.3
[81] viridisLite_0.3.0 tibble_1.4.2 memoise_1.1.0
This is too much information. How can I skip the last category (loaded via a namespace (and not attached):) ?
You can remove an item from sessionInfo(), which is a list with a class attribute then use the relevant method to print the resulting object:
getS3method("print","sessionInfo")(sessionInfo()[-7])
R version 3.4.1 (2017-06-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 LC_MONETARY=French_France.1252
[4] LC_NUMERIC=C LC_TIME=French_France.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.10.4 tidytext_0.1.8 bindrcpp_0.2 dplyr_0.7.4 readr_1.1.1 tidyr_0.7.2
[7] tibble_1.3.4 ggplot2_2.2.1 tidyverse_1.1.1 purrr_0.2.4 grofit_1.1.1-1 swirl_2.4.3
[13] lobstr_0.0.0.9000 Rcpp_0.12.16 rlang_0.2.0
This also works with same output:
utils:::print.sessionInfo(sessionInfo()[-7]) # thanks #r2evans
or
`class<-`(sessionInfo()[-7],"sessionInfo")
To know which part to remove from output, check:
names(sessionInfo())
[1] "R.version" "platform" "locale" "running" "basePkgs" "otherPkgs" "loadedOnly" "matprod" "BLAS"
[10] "LAPACK"
If you read ?sessionInfo, you will see that it returns an object of "sessionInfo", which is really just a named list with a lot of "stuff". Unfortunately, the only argument (package=NULL) does not limit to what you want.
When you realize that it returns a classed list, then you can start looking for print.sessionInfo, R's mechanism for class-specific function. I looked for it and didn't find it, but if you look closer into its package, you'll see utils:::print.sessionInfo. It's source is actually quite readable (not long and obfuscated).
I suggest you copy that function, remove 4 of the last 5 lines, and use it. For instance:
mySIprint <- function(x, locale = TRUE, ...) {
mkLabel <- function(L, n) {
vers <- sapply(L[[n]], function(x) x[["Version"]])
pkg <- sapply(L[[n]], function(x) x[["Package"]])
paste(pkg, vers, sep = "_")
}
# ... don't forget this code that I'm snipping out for brevity
if (!is.null(x$otherPkgs)) {
cat("\nother attached packages:\n")
print(mkLabel(x, "otherPkgs"), quote = FALSE, ...)
}
#if (!is.null(x$loadedOnly)) {
# cat("\nloaded via a namespace (and not attached):\n")
# print(mkLabel(x, "loadedOnly"), quote = FALSE, ...)
#}
invisible(x)
}
mySIprint(sessionInfo())
# R version 3.4.3 (2017-11-30)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows 10 x64 (build 16299)
#
# Matrix products: default
#
# locale:
# [1] LC_COLLATE=English_United States.1252
# [2] LC_CTYPE=English_United States.1252
# [3] LC_MONETARY=English_United States.1252
# [4] LC_NUMERIC=C
# [5] LC_TIME=English_United States.1252
#
# attached base packages:
# [1] stats graphics grDevices utils datasets methods base
#
# other attached packages:
# [1] ggplot2_2.2.1 microbenchmark_1.4-4 readr_1.1.1
# [4] bindrcpp_0.2 dplyr_0.7.4 tidyr_0.7.2
# [7] testthat_2.0.0 plumber_0.4.4 r2_0.6.0
I am trying to run a flexdashboard with a couple plotly graphs, but I receive the error:
Error in switch(options$dev, pdf = ".pdf", jpeg = ".jpeg", ".png") :
EXPR must be a length 1 vector Calls: <Anonymous> ... fun ->
withVisible -> knit_print -> html_screenshot
The example code I am running is a set of ggplotly example code.
Session info:
R version 3.3.1 (2016-06-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] grid stats graphics grDevices utils datasets methods base
other attached packages:
[1] maps_3.2.0 bindrcpp_0.2 highcharter_0.5.0 quantmod_0.4-12 TTR_0.23-3
[6] xts_0.10-0 zoo_1.8-0 formattable_0.2.0.1 officer_0.2.1 flextable_0.4.2
[11] gridExtra_2.3 knitr_1.17 kableExtra_0.7.0 dplyr_0.7.4 lubridate_1.6.0
[16] rstudioapi_0.7 RODBC_1.3-15 plotly_4.7.1 ggplot2_2.2.1
loaded via a namespace (and not attached):
[1] httr_1.3.1 tidyr_0.7.2 jsonlite_1.5 viridisLite_0.2.0 R.utils_2.6.0
[6] shiny_1.0.5 assertthat_0.2.0 yaml_2.1.16 gdtools_0.1.6 backports_1.1.1
[11] lattice_0.20-35 glue_1.1.1 rlist_0.4.6.1 uuid_0.1-2 digest_0.6.12
[16] rvest_0.3.2 colorspace_1.3-2 htmltools_0.3.6 httpuv_1.3.5 R.oo_1.21.0
[21] plyr_1.8.4 psych_1.7.8 pkgconfig_2.0.1 broom_0.4.2 flexdashboard_0.5.1
[26] purrr_0.2.3 xtable_1.8-2 scales_0.5.0 tibble_1.3.4 lazyeval_0.2.0
[31] mnormt_1.5-5 magrittr_1.5 mime_0.5 evaluate_0.10.1 R.methodsS3_1.7.1
[36] nlme_3.1-131 xml2_1.1.1 foreign_0.8-69 tools_3.3.1 data.table_1.10.4
[41] hms_0.3 stringr_1.2.0 munsell_0.4.3 zip_1.0.0 rlang_0.1.2
[46] htmlwidgets_1.0 crosstalk_1.0.0 igraph_1.1.2 base64enc_0.1-3 labeling_0.3
[51] rmarkdown_1.6 gtable_0.2.0 curl_3.0 reshape2_1.4.2 R6_2.2.2
[56] bindr_0.1 rprojroot_1.2 readr_1.1.1 stringi_1.1.5 parallel_3.3.1
[61] Rcpp_0.12.12 tidyselect_0.2.0
A similar error resulted in the case of this question on github, but I am not using shiny nor do I set a working directory in the code. Any ideas?
Overview
From All apps with htmlwidgets failing to render on Mac after #150, Yihui identified this error with version 1.17 of the knitr package.
This is the same version you currently have installed (see 'other attached packages' section in sessionInfo()). To squash the error message, please run the following code in r to update the knitr package to version 1.19:
# install package from CRAN
install.packages( pkgs = 'knitr' )
I have been struggling with encoding in twitteR.
The code below is the one I tried.
query <- list()
query <- c(query, "xxx")
query <- c(query, "xxx2")
for(i in 1:length(query)){
result<-searchTwitter(query[i],n=500, resultType ="recent",lang="en")
seo_tweets <- c(tweets,result)
seo_tweets <- unique(tweets)
}
But it ended up fetching the garbage data such as: Motor at #IAA2017 \xed��\xed�� https://t.c.
I have tried this code as well, but it did not work at all.
Sys.setlocale(category = "LC_ALL", locale = "English_United States.1252")
How can I solve this issue? Here's sessioninfo
sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] wordcloud_2.5 RColorBrewer_1.1-2 tm_0.7-1 NLP_0.1-11 stringr_1.2.0 h2o_3.10.5.3 rvest_0.3.2
[8] xml2_1.1.1 ggmap_2.6.1 ggplot2_2.2.1 twitteR_1.1.8 base64enc_0.1-3 ROAuth_0.9.6 ROCR_1.0-7
[15] gplots_3.0.1
loaded via a namespace (and not attached):
[1] gtools_3.5.0 slam_0.1-40 reshape2_1.4.2 lattice_0.20-35 colorspace_1.3-2 rlang_0.1.1 withr_2.0.0
[8] DBI_0.7 selectr_0.3-1 sp_1.2-5 bit64_0.9-7 jpeg_0.1-8 plyr_1.8.4 munsell_0.4.3
[15] gtable_0.2.0 devtools_1.13.3 caTools_1.17.1 RgoogleMaps_1.4.1 mapproj_1.2-5 memoise_1.1.0 curl_2.8.1
[22] parallel_3.4.1 proto_1.0.0 Rcpp_0.12.12 KernSmooth_2.23-15 geosphere_1.5-5 openssl_0.9.6 scales_0.4.1
[29] gdata_2.18.0 jsonlite_1.5 bit_1.1-12 rjson_0.2.15 png_0.1-7 digest_0.6.12 stringi_1.1.5
[36] grid_3.4.1 tools_3.4.1 bitops_1.0-6 magrittr_1.5 maps_3.2.0 lazyeval_0.2.0 RCurl_1.95-4.8
[43] tibble_1.3.3 httr_1.3.1 R6_2.2.2 git2r_0.19.0 compiler_3.4.1
Thank you.
I'm running something like this: (sorry it is not 100% reproducible, unless you have postgres running in your machine, with mydb and tables created, and the given user and password)
library(RPostgreSQL)
library(tidyverse)
library(dbplyr)
pg_conn <- RPostgreSQL::dbConnect(
drv = "PostgreSQL", dbname = "mydb",
user = "postgres", password = "postgres"
)
table1_pg <- dplyr::tbl(pg_conn, "table1")
table2_pg <- dplyr::tbl(pg_conn, "table2")
table_join <- table1_pg %>%
left_join(table2_pg, by = c("x" = "x"))
And I get the following error:
Error in nlevels(object) : argument "object" is missing, with no default
And I have no clue what is going on (I am 100% sure the tables exists and each of them has the x column; I can query them, both using the tbl or directly sending an sql with RPostgreSQL::dbGetQuery).
I've googled the problem and searched GitHub and SO, but I found no solution, nor anybody reporting the problem. The closest thing I found is this issue that left_join duplicates join variables. I tried #hadley's dx example there using SQLite, but that works all right on my machine, so perhaps this is a postgres-specific issue?
Here's my sessionInfo()
> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dbplyr_1.0.0 dplyr_0.7.1 purrr_0.2.2.2 readr_1.1.1 tidyr_0.6.3 tibble_1.3.3
[7] ggplot2_2.2.1 tidyverse_1.1.1 RPostgreSQL_0.4-1 DBI_0.6-1
loaded via a namespace (and not attached):
[1] Rcpp_0.12.11 cellranger_1.1.0 compiler_3.4.0 plyr_1.8.4 bindr_0.1 forcats_0.2.0 tools_3.4.0
[8] lubridate_1.6.0 jsonlite_1.5 nlme_3.1-131 gtable_0.2.0 lattice_0.20-35 pkgconfig_2.0.1 rlang_0.1.1
[15] psych_1.7.5 parallel_3.4.0 haven_1.0.0 bindrcpp_0.2 xml2_1.1.1 httr_1.2.1 stringr_1.2.0
[22] hms_0.3 grid_3.4.0 glue_1.1.1 R6_2.2.1 readxl_1.0.0 foreign_0.8-68 modelr_0.1.0
[29] reshape2_1.4.2 magrittr_1.5 scales_0.4.1 rvest_0.3.2 assertthat_0.2.0 mnormt_1.5-5 colorspace_1.3-2
[36] stringi_1.1.5 lazyeval_0.2.0 munsell_0.4.3 broom_0.4.2
I'm using caret to train earth models like so:
test <- train(Y ~ X1+X2,table.needed,method = "earth")
which worked fine previously. However, now I am getting the following error:
Error in .C("ForwardPassR", fullset = as.integer(fullset), bx = matrix(0, :
"ForwardPassR" not available for .C() for package "earth"
This also happens when I call earth directly:
earth(Y ~ X1+X2,data = table.needed)
From other questions, I surmise that this is something with the underlying C - does anyone know how to fix the error and get the package running again?
I've attached the sessionInfo() call below:
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server >= 2012 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base
other attached packages:
[1] randomForest_4.6-12 broom_0.4.2 timekit_0.3.0 readr_1.1.1 hts_5.0
[6] matrixcalc_1.0-3 Matrix_1.2-9 SparseM_1.77 forecast_8.0 earth_4.5.0
[11] plotmo_3.3.3 TeachingDemos_2.10 plotrix_3.6-5 doParallel_1.0.10 iterators_1.0.8
[16] foreach_1.4.3 prophet_0.1.1 Rcpp_0.12.11 zoo_1.8-0 caretEnsemble_2.0.0
[21] caret_6.0-76 ggplot2_2.2.1 lattice_0.20-35 reshape2_1.4.2 dtplyr_0.0.2
[26] data.table_1.10.4 dplyr_0.5.0
loaded via a namespace (and not attached):
[1] tidyr_0.6.3 psych_1.7.5 assertthat_0.2.0 digest_0.6.12 lmtest_0.9-35
[6] R6_2.2.1 plyr_1.8.4 MatrixModels_0.4-1 stats4_3.4.0 rlang_0.1.1
[11] lazyeval_0.2.0 minqa_1.2.4 car_2.1-4 fracdiff_1.4-2 TTR_0.23-1
[16] nloptr_1.0.4 splines_3.4.0 lme4_1.1-13 foreign_0.8-68 stringr_1.2.0
[21] munsell_0.4.3 compiler_3.4.0 rstan_2.15.1 mnormt_1.5-5 mgcv_1.8-17
[26] nnet_7.3-12 tibble_1.3.3 gridExtra_2.2.1 quadprog_1.5-5 codetools_0.2-15
[31] MASS_7.3-47 ModelMetrics_1.1.0 grid_3.4.0 nlme_3.1-131 gtable_0.2.0
[36] DBI_0.6-1 magrittr_1.5 StanHeaders_2.15.0-1 scales_0.4.1 quantmod_0.4-9
[41] stringi_1.1.5 pbapply_1.3-2 tseries_0.10-41 timeDate_3012.100 xts_0.9-7
[46] tools_3.4.0 hms_0.3 pbkrtest_0.4-7 inline_0.3.14 colorspace_1.3-2
[51] quantreg_5.33
There is no version information about the earth package.
Anyway, I encounter with the same error with earth_4.5.0 and the error disappears when I update it to earth_4.5.1.
So I suggest you update or reinstall the package.