Calling mclapply from Rscript - r

When calling a function which uses mclapply() with Rscript myFuction.R --json=config.json the mclapply functions fails with message
all scheduled cores encountered errors in user code
However when I run the code within RStudio it runs fine. I'm developing on RStudio AWS AMI and testing in RStudio, and executing Rscript from the terminal of RStudio AWS machine, and the environments are the same between RStudio and the terminal.
Does anyone have an idea of extra parameters I might need to give to mclapply or other environment parameters I need to define when running mclapply with Rscript?
I've tried changing all the mclapply() arguments without success
Here is my environment.
R version 3.5.1 (2018-07-02)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Amazon Linux 2
Matrix products: default
BLAS/LAPACK: /anaconda3/envs/r35p27/lib/R/lib/libRblas.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats4 parallel stats graphics grDevices utils datasets
[8] methods base
other attached packages:
[1] rlist_0.4.6.1 stringi_1.4.3
[3] stringr_1.3.1 seqinr_3.4-5
[5] rBLAST_0.99.2 ShortRead_1.40.0
[7] GenomicAlignments_1.18.1 SummarizedExperiment_1.12.0
[9] DelayedArray_0.8.0 matrixStats_0.54.0
[11] Biobase_2.42.0 Rsamtools_1.34.0
[13] GenomicRanges_1.34.0 GenomeInfoDb_1.18.1
[15] Biostrings_2.50.2 XVector_0.22.0
[17] IRanges_2.16.0 S4Vectors_0.20.1
[19] BiocParallel_1.16.6 BiocGenerics_0.28.0
[21] aws.s3_0.3.12 jsonlite_1.5
[23] configr_0.3.3 optparse_1.6.2
loaded via a namespace (and not attached):
[1] Rcpp_1.0.1 RColorBrewer_1.1-2 compiler_3.5.1
[4] base64enc_0.1-3 bitops_1.0-6 tools_3.5.1
[7] zlibbioc_1.28.0 digest_0.6.20 lattice_0.20-35
[10] Matrix_1.2-14 yaml_2.2.0 GenomeInfoDbData_1.2.1
[13] hwriter_1.3.2 httr_1.3.1 xml2_1.2.0
[16] ade4_1.7-13 grid_3.5.1 getopt_1.20.2
[19] data.table_1.12.2 glue_1.3.1 R6_2.2.2
[22] latticeExtra_0.6-28 magrittr_1.5 MASS_7.3-51.4
[25] aws.signature_0.5.0 ini_0.3.1 RCurl_1.95-4.12
[28] RcppTOML_0.1.6
Although this is a "Warning message" mclapply quits
Warning messages:
1: In mclapply(1:dim(this.split)[1], BLAST.loop, mc.preschedule = TRUE) :
all scheduled cores encountered errors in user code

Related

skimr(iris) generates error in check_dots_used when printing in linux RStudio console

When I run the default skimr command in the console on a linux RStudio server I get the following partial output and error:
library(skimr)
skim(iris)
── Data Summary ────────────────────────
Values
Name iris
Number of rows 150
Number of columns 5
Column type frequency:
factor 1
numeric 4
Group variables None
Error in check_dots_used(action = warn) : unused argument (action = warn)
However, the same code will run just fine when I knit it in an RMarkdown document.
The same code will also run fine on my Mac OSX laptop instance of RStudio, both in the console and an RMarkdown document.
I can assign the output of the skimr command and View the assigned output object just fine on the server instance:
out <- skim(iris)
View(out)
class(out)
1 "skim_df" "tbl_df" "tbl" "data.frame"
but print(out) generates the same error again
Here's the sessionInfo.
sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /usr/lib64/R/lib/libRblas.so
locale:
1 LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
1 stats graphics grDevices utils datasets methods base
other attached packages:
1 skimr_2.1.2
loaded via a namespace (and not attached):
1 Rcpp_1.0.2 rstudioapi_0.13 knitr_1.31 magrittr_1.5 tidyselect_1.1.0 R6_2.3.0 rlang_0.4.10 fansi_0.4.0 stringr_1.4.0
[10] dplyr_1.0.4 tools_3.5.1 xfun_0.20 utf8_1.1.4 cli_2.3.0 DBI_1.0.0 withr_2.4.1 htmltools_0.3.6 ellipsis_0.2.0.1
[19] yaml_2.2.0 rprojroot_1.3-2 assertthat_0.2.0 digest_0.6.18 tibble_3.0.6 lifecycle_0.2.0 crayon_1.3.4 tidyr_1.1.2 purrr_0.3.4
[28] repr_1.1.3 base64enc_0.1-3 vctrs_0.3.6 evaluate_0.12 glue_1.4.2 rmarkdown_1.10 stringi_1.2.4 compiler_3.5.1 pillar_1.4.7
[37] backports_1.1.2 generics_0.1.0 jsonlite_1.6 pkgconfig_2.0.2
You need to use options(skimr_strip_metadata = FALSE). It's due to something related to the {pilar} package.
https://github.com/ropensci/skimr/issues/641
Also please update the ellipsis package. See this answer.
Basic dyplr functions give an error: "check_dots_used"
I don't know why doing it in rmarkdown would work when the console doesn't, unless somehow a different version of the package is loading.
I used to have a similar situation where skim_without_chart will return an error could not find function "skim_without_charts".
The solution I used is to first call skimr before calling skimm_without_charts like skimr::skim_without_charts(penguins)
That solves the problem for me.

How to install rgdal and/or upload raster on google-collaboration

I am trying to upload several rasters on tif format with google-collaboration. However I always had the same error message:
"Cannot create RasterLayer object from this file; perhaps you need to install rgdal first"
So, I tried to install rgdal...and another error appeared:
"Installation of package had non-zero exit status."
I tried to install using a local file, and apparently, its run ok, however, when i required the package, another error appeared:
"Error in library(rgdal) : there is no package called ‘rgdal’
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/init.py:146: RRuntimeWarning: 4:
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/init.py:146: RRuntimeWarning: 5:
warnings.warn(x, RRuntimeWarning)
/usr/local/lib/python3.6/dist-packages/rpy2/rinterface/init.py:146: RRuntimeWarning: 6: package ‘rgdal’ is not available (for R version 3.6.1)"
Could anyone help me to fix these errors?
I provide my session info and a fragment of code
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas/liblapack.so.3
Random number generation:
RNG: Mersenne-Twister
Normal: Inversion
Sample: Rounding
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel tools stats graphics grDevices utils datasets
[8] methods base
other attached packages:
[1] snowfall_1.84-6.1 snow_0.4-3 biomod2_3.3-7.1 ggplot2_3.2.1
[5] reshape_0.8.8 raster_2.9-23 sp_1.3-1
loaded via a namespace (and not attached):
[1] gbm_2.1.5 zoo_1.8-6 tidyselect_0.2.5
[4] purrr_0.3.2 splines_3.6.1 lattice_0.20-38
[7] colorspace_1.4-1 viridisLite_0.3.0 survival_2.44-1.1
[10] rlang_0.4.0 hexbin_1.27.3 pillar_1.4.2
[13] glue_1.3.1 withr_2.1.2 RColorBrewer_1.1-2
[16] plotmo_3.5.5 plyr_1.8.4 mda_0.4-10
[19] munsell_0.5.0 gtable_0.3.0 codetools_0.2-16
[22] latticeExtra_0.6-28 class_7.3-15 Rcpp_1.0.2
[25] scales_1.0.0 plotrix_3.7-6 abind_1.4-5
[28] gridExtra_2.3 TeachingDemos_2.10 dplyr_0.8.3
[31] dismo_1.1-4 rasterVis_0.46 grid_3.6.1
[34] magrittr_1.5 lazyeval_0.2.2 tibble_2.1.3
[37] PresenceAbsence_1.1.9 randomForest_4.6-14 Formula_1.2-3
[40] crayon_1.3.4 pkgconfig_2.0.2 MASS_7.3-51.4
[43] Matrix_1.2-17 pROC_1.15.3 assertthat_0.2.1
[46] earth_5.1.1 R6_2.4.0 rpart_4.1-15
[49] nnet_7.3-12 compiler_3.6.1
%load_ext rpy2.ipython
from google.colab import drive
drive.mount('/content/drive/')
%%R
install.packages("rgdal", dependencies=TRUE,repos='http://cran.rstudio.com/')
library(rgdal)
library(raster)
myExpl.EUR <- stack(list.files("/content/drive/My Drive/Sycios/",pattern=".tif",full.names=TRUE))
Thank you very much in advance
Try this from the command line
sudo apt-get update
sudo apt-get install libgdal-dev libproj-dev
And then in R
install.packages("rgdal")

Shiny Update Windows 10 : Failed to load resource img src from local

I'm working on a windows 10 with local R/RStudio installed.
A few days again I updated R to v3.5.3 and RStudio to v1.2.1330.
I didn't check my app immediately, only a few days later. Meanwhile I had to install many Julia/Atom-Juno packages for a training and now when I run (locally) an app I'm working on, all the images are gone.
https://github.com/phileas-condemine/carto_indicateurs
The app looked like that : https://drees.shinyapps.io/Cartographie_des_indicateurs/ (I didn't try to redeploy, even if, being another OS it might still work)
Here is a simple reprex
within app.R (ran within an RStudio project)
library(shiny)
download.file(url = "https://www.rstudio.com/wp-content/uploads/2019/03/RStudio-Logo-Blue-Gray-125.png",
destfile = "www/rstudio.png",mode = "wb")#I have to add this for windows 10
shinyApp(
ui = fluidPage(
tags$img(src="https://www.rstudio.com/wp-content/uploads/2019/03/RStudio-Logo-Blue-Gray-125.png"),
tags$img(src="rstudio.png")
),
server = function(input, output) { }
)
The LOCAL image doesn't render, instead I have a broken image. (the one I get from URL is OK)
When I open the Chrome inspector and check the JS console I have the following error
Failed to load resource: the server responded with a status of 404 (Not Found) rstudio.png:1
I tried to check the reprex with the latest rocker/rstudio image but I can't reproduce it. Probably because it's a Windows 10 related issue.
Here is my sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)
Matrix products: default
locale:
[1] LC_COLLATE=French_France.1252
[2] LC_CTYPE=French_France.1252
[3] LC_MONETARY=French_France.1252
[4] LC_NUMERIC=C
[5] LC_TIME=French_France.1252
attached base packages:
[1] stats graphics
[3] grDevices utils
[5] datasets methods
[7] base
other attached packages:
[1] shiny_1.3.1
loaded via a namespace (and not attached):
[1] compiler_3.5.3
[2] magrittr_1.5
[3] R6_2.4.0
[4] promises_1.0.1
[5] later_0.8.0
[6] htmltools_0.3.6
[7] tools_3.5.3
[8] Rcpp_1.0.1
[9] jsonlite_1.6
[10] digest_0.6.18
[11] xtable_1.8-3
[12] httpuv_1.5.1
[13] mime_0.6
I tried re-installing R v3.5.3 & latest RStudio v1.2.1335, it doesn't fix it.
What troubles me most is that some of the apps render all the images without any problem.
For example this one works alright : https://github.com/phileas-condemine/trafic_dataviz.drees
Is it because of the dependencies that somehow fix it ?
Back to shiny 1.2.0 & R 3.5.2 fixes the problem, all images are displayed correctly. So the "bug" either comes from shiny 1.3.1 or R 3.5.3
R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17134)
Matrix products: default
locale:
[1] LC_COLLATE=French_France.1252
[2] LC_CTYPE=French_France.1252
[3] LC_MONETARY=French_France.1252
[4] LC_NUMERIC=C
[5] LC_TIME=French_France.1252
attached base packages:
[1] stats graphics
[3] grDevices utils
[5] datasets methods
[7] base
other attached packages:
[1] bsplus_0.1.1
[2] shinycssloaders_0.2.0
[3] markdown_0.9
[4] stringr_1.3.1
[5] shinyBS_0.61
[6] shinydashboard_0.7.1
[7] shinyjs_1.0
[8] V8_2.2
[9] shinyWidgets_0.4.4
[10] DT_0.5
[11] data.table_1.12.0
[12] dplyr_0.7.8
[13] shiny_1.2.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.0
[2] pillar_1.3.1
[3] compiler_3.5.2
[4] later_0.8.0
[5] bindr_0.1.1
[6] tools_3.5.2
[7] digest_0.6.18
[8] packrat_0.5.0
[9] lubridate_1.7.4
[10] jsonlite_1.6
[11] tibble_2.0.1
[12] pkgconfig_2.0.2
[13] rlang_0.3.1
[14] rstudioapi_0.9.0
[15] crosstalk_1.0.0
[16] yaml_2.2.0
[17] curl_3.3
[18] bindrcpp_0.2.2
[19] htmlwidgets_1.3
[20] tidyselect_0.2.5
[21] glue_1.3.0
[22] R6_2.3.0
[23] purrr_0.3.0
[24] magrittr_1.5
[25] scales_1.0.0
[26] promises_1.0.1
[27] htmltools_0.3.6
[28] assertthat_0.2.0
[29] mime_0.6
[30] xtable_1.8-3
[31] colorspace_1.4-0
[32] httpuv_1.5.1
[33] stringi_1.2.4
[34] munsell_0.5.0
[35] crayon_1.3.4

'filter' from dplyr package causes error message

I run queries to ClickHouse database (connected with Rserver by RClickhouse package). Queries run smoothly unless I use filter function - which generates error message about wrong object type
Important detail: this problem appears only in Rstudio project that is located in common server folder. Same code works fine (without errors) in similar project that shares parent folder (/users/boris/) with R
> a <- con %>% tbl("test_sample") %>% select(domain) %>% collect()
> show(a)
# A tibble: 140,000 x 1
domain
* <chr>
1 allforchildren.ru
> a <- con %>% tbl("test_sample") %>% filter(domain == "wildberries.ru") %>% collect()
Error in storage.mode(x) <- "double" :
(list) object cannot be coerced to type 'double'
Any guesses what is the reason for such reaction on filter function?
P.S. session info
> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)
Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.7.0
LAPACK: /usr/lib/lapack/liblapack.so.3.7.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] urltools_1.7.1 dplyr_0.7.6 getPass_0.2-2
[4] DBI_1.0.0 RClickhouse_0.4.0
loaded via a namespace (and not attached):
[1] Rcpp_0.12.18 dbplyr_1.2.2 crayon_1.3.4 assertthat_0.2.0
[5] R6_2.2.2 magrittr_1.5 pillar_1.3.0 rlang_0.2.1
[9] rstudioapi_0.7 bindrcpp_0.2.2 tools_3.5.1 glue_1.3.0
[13] triebeard_0.3.0 purrr_0.2.5 compiler_3.5.1 yaml_2.2.0
[17] pkgconfig_2.0.1 bindr_0.1.1 tidyselect_0.2.4 tibble_1.4.2

Facing difficulties in regression with mxnet in R

I was following this tutorial http://mxnet.io/tutorials/r/fiveMinutesNeuralNetwork.html#regression
Everything worked accordingly but when I changed:
fc1 <- mx.symbol.FullyConnected(data, num_hidden=1)
to
fc1 <- mx.symbol.FullyConnected(data, num_hidden=2)
And among the stacks of error logs I thought may be this is the most interesting:
Error in exec$update.arg.arrays(arg.arrays, match.name, skip.null) :
[20:22:59] src/ndarray/ndarray.cc:239: Check failed: from.shape() == to->shape()
shape mismatchfrom.shape = (20,) to.shape=(20,2)
How do I diagnose this problem?
Here is the output of sessionInfo():
R version 3.3.3 RC (2017-02-27 r72279)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.5 LTS
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] mlbench_2.1-1 mxnet_0.9.5
loaded via a namespace (and not attached):
[1] igraph_1.0.1 Rcpp_0.12.10 rstudioapi_0.6 magrittr_1.5 munsell_0.4.3 colorspace_1.3-2
[7] viridisLite_0.2.0 R6_2.2.0 brew_1.0-6 stringr_1.2.0 plyr_1.8.4 dplyr_0.5.0
[13] visNetwork_1.0.3 Rook_1.1-1 tools_3.3.3 grid_3.3.3 gtable_0.2.0 DBI_0.6
[19] influenceR_0.1.0 DiagrammeR_0.9.0 htmltools_0.3.5 lazyeval_0.2.0 digest_0.6.12 assertthat_0.1
[25] tibble_1.2 gridExtra_2.2.1 RColorBrewer_1.1-2 ggplot2_2.2.1 codetools_0.2-8 htmlwidgets_0.8
[31] viridis_0.4.0 rgexf_0.15.3 stringi_1.1.3 scales_0.4.1 XML_3.98-1.6 jsonlite_1.3
The thing is that below the fc1 <- mx.symbol.FullyConnected(data, num_hidden=1) line, tutorial uses linear regression for the output lro <- mx.symbol.LinearRegressionOutput(fc1).
LinearRegressionOutput is used to compute the l2-loss between it's input symbol and the labels provided to it. It assumes 1 label per example, and passing 2 breaks it. In my case it is a little bit different from your message, maybe because the difference in versions:
Error in symbol$infer.shape(list(...)) :
Error in operator linearregressionoutput5: Shape inconsistent, Provided=(20,), inferred shape=(20,2)
Fixing of this depends of what exactly you want to achieve. If you are solving classification task and want to receive probabilities for both classes, then you need to use Softmax:
fc1 <- mx.symbol.FullyConnected(data, num_hidden=2)
lro <- mx.symbol.SoftmaxOutput(fc1)

Resources