Not able to read MNIST dataset (from dslabs library) due to a C stack limit. R console output is:
> mnist <- read_mnist()
Error: C stack usage 15923024 is too close to the limit
Unable to implement a solution from forum searches. Closest I can determine is to:
(preferable) increase the stack size in terminal and unable to find correct syntax for command line (Windows terminal) or within RStudio terminal.
reduce size of raw file (loaded from gz download rather than read_mnist() from the dslabs package.
System: Windows 10, 64bit, RStudio 3.6.3
> Cstack_info()["size"]
size
15922790
... seem to have changed limit in RTools:
Dav#DESKTOP-V8I63M1 MINGW64 ~
$ ulimit -s
2032
Dav#DESKTOP-V8I63M1 MINGW64 ~
$ ulimit -s 16384
Dav#DESKTOP-V8I63M1 MINGW64 ~
$ ulimit -s
16384
... with no effect on the Cstack size in RStudio.
Any thoughts?
Related
I get a new error, when I try to compile an R Markdown file int appears the next message:
Error: C stack usage 7971408 is too close to the limit
Execution halted
I did some research and I found some people with the same error:
Error: C stack usage is too close to the limit
C stack usage 7970960 is too close to the limit
GenomicRanges: C stack usage ... is too close to the limit
R mapping (C stack usage 7971616 is too close to the limit)
C stack usage 7972356 is too close to the limit #335
But these guys have problems with some function or something like that.
The actions I did in orden to try to solve this situation:
Uninstall R and RStudio, reinstall de last versions of both, reboot my computer... nothing.
Try to change ulimit -s, and this point is interesting because this is my ulimit -a on R terminal:
geomicrobio-mac:~ geomicrobio$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 10240
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1392
virtual memory (kbytes, -v) unlimited
When I try to change de ulimit -s for unlimited or 65532 on R terminal, it doesn't change.
The ulimit -a of my terminal (macOS Monterey v12.0.1) is:
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 65532
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 1392
-n: file descriptors 2560
This just happen with R Markdown, I can do Shinny apps, and run scripts, etc. but I can`t compile any R Markdown despite it contains only text.
This is the info when I put base::Cstack_info() on console:
size current direction eval_depth
7969177 14032 1 2
My version of R:
platform x86_64-apple-darwin17.0
arch x86_64
os darwin17.0
system x86_64, darwin17.0
status
major 4
minor 1.2
year 2021
month 11
day 01
svn rev 81115
language R
version.string R version 4.1.2 (2021-11-01)
nickname Bird Hippie
If you know how to solve this I really appreciate your help.
Thank you.
I just delete the .Rprofile .-.
For research purposes, I am trying to run the sqlite3 full test suite.
I am running under Linux for this test.
I believe I have installed the correct libraries
sudo apt-get install tcl-dev tk-dev
And when I run the configure command, it finds tcl
checking whether to use an in-ram database for temporary tables... no
checking if executables have the .exe suffix... unknown
checking for Tcl configuration... found /usr/lib/tclConfig.sh
checking for existence of /usr/lib/tclConfig.sh... loading
checking for library containing readline... no
checking for library containing tgetent... no
checking for readline in -lreadline... no
checking readline.h usability... no
Then when I run make test it ends with:
Time: orderby8.test 5080 ms
Time: orderby9.test 291 ms
Makefile:1201: recipe for target 'tcltest' failed
make: *** [tcltest] Killed
It is hard to tell because your question misses important details:
What *nix exactly?
More importantly, which Tcl versions are installed by apt?
How do you obtain the sqlite3 sources incl. test harnesses?
I just tested fine the following:
Ubuntu Xenial
apt-get install tcl tcl-dev (turns out to be 8.6.7)
wget https://www.sqlite.org/2018/sqlite-src-3250200.zip
unzip sqlite-src-3250200.zip
cd sqlite-src-3250200
./configure && make test
... and it succeeds to complete the make test run:
...
Time: orderby7.test 6 ms
Time: orderby8.test 115 ms
Time: orderby9.test 6 ms
Time: oserror.test 49 ms
...
Time: zipfile2.test 10 ms
SQLite 2018-09-25 19:08:10 fb90e7189ae6d62e77ba3a308ca5d683f90bbe633cf681865365b8e92792d1c7
0 errors out of 147094 tests on builda Linux 64-bit little-endian
All memory allocations freed - no leaks
Maximum memory usage: 9278848 bytes
Current memory usage: 0 bytes
Number of malloc() : -1 calls
It seems that oserror.test crashes whatever Tcl in your case.
I have the TSP package installed and working.
I downloaded all the files from the Concorde (TSP/Waterloo) websites. I tried different versions. Even extracted all the files.
I put the files in my R working directory.
Finally, when running concorde_path() it was able to pick up that the files are found.
However, when I run concorde_help() I receive an error.
I got a cygwin1.dll not found error. So I installed cygwin.
I still get an error.
I tried putting all the concorde files in the bin folder of cygwin (where cygwin1.dll lives) and pointing the R working directory and the concorde_path() there and I get a status 123 error.
I also have the Concorde windows app downloaded and it does work. I found another post suggesting that it has to work in order for it to work within R.
Running R/Rstudio under Windows
Thank you for any suggestions and help you may have.
I just got this working for TSPMap so hopefully it helps someone out.
Concorde for windows doesn't appear to have a command line interface that works with the TSP package.. This is where Cygwin comes in because that version of concorde can work on the command line and interface with the TSP package.
You really need to get it working in cygwin first.. So you need to get cygwin console up and running.
If you've got the console working you can download and gunzip the concorde binary and test that.. Just running ./concorde.exe within cygwin should display the help for the program.
Another test is to use the following test file and see if that works through concorde.
NAME: TEST
TYPE: TSP
DIMENSION: 6
EDGE_WEIGHT_TYPE: EXPLICIT
EDGE_WEIGHT_FORMAT: FULL_MATRIX
EDGE_WEIGHT_SECTION:
0 1 2 1 1 2
1 0 1 2 2 1
2 1 0 1 2 1
1 2 1 0 1 2
1 2 2 1 0 1
2 1 1 2 1 0
Once you get that working in cygwin, it is time to try from the command line to see if cygwin is integrated with windows. If that is fine R shouldn't have an issue with it.
This command should display the concorde command line help if your system can recognise it.
t1<- try(system("c:/cygwin64/home/davisst5/concorde.exe"))
This one tests that concorde_path should work..
concorde_path("c:/cygwin64/home/davisst5/")
found: concorde.exe
If you've done all of this already and still having issues, its possible that there might be a 32/64 bit version problem where either R or cygwin is running in a different mode and cannot call the other properly (which might be a source of the dll issues).. I've got RGui in 64 bit and Cygwin in 64 bit.. One tutorial I read said that it was critically important to install 32 bit cygwin to make it work, so thats possibly why.
I followed the answer given by "Stephen Davison" above. However, I still got the cygwin1.dll file missing error. I had cygwin 32 bit installed in C drive. What I did is that I get the cygwin1.dll file from the C:\cygwin\bin folder and paste it in my R working directory, which is E:\RA\Concorde_Code
Then in my R-studio (64 bit installed), I ran the following code to check whether the Concorde is working or not
concordePath = "E:/RA/Concorde_Code/"
concorde_path(concordePath)
It says the following
found: concorde concorde.exe
Then I ran the following code
concorde_help()
It gave me following output
The following options can be specified in solve_TSP with method "concorde" using clo in control:
/Concorde_Code/concorde
Usage: /Concorde_Code/concorde [-see below-] [dat_file]
-B do not branch
-C # maximum chunk size in localcuts (default 16)
-d use dfs branching instead of bfs
-D f edgegen file for initial edge set
-e f initial edge file
-E f full edge file (must contain initial edge set)
-f write optimal tour as edge file (default is tour file)
-F f read extra cuts from file
-g h be a grunt for boss h
-h be a boss for the branching
-i just solve the blossom polytope
-I just solve the subtour polytope
-J # number of tentative branches
-k # number of nodes for random problem
-K h use cut server h
-M f master file
-m use multiple passes of cutting loop
-n s problem location (just a name or host:name, not a file name)
-o f output file name (for optimal tour)
-P f cutpool file
-q do not cut the root lp
-r # use #x# grid for random points, no dups if #<0
-R f restart file
-s # random seed
-S f problem file
-t f tour file (in node node node format)
-u v initial upperbound
-U do not permit branching on subtour inequalities
-v verbose (turn on lots of messages)
-V just run fast cuts
-w just subtours and trivial blossoms
-x delete files on completion (sav pul mas)
-X f write the last root fractional solution to f
-y use simple cutting and branching in DFS
-z # dump the #-lowest reduced cost edges to file xxx.rcn
-N # norm (must specify if dat file is not a TSPLIB file)
0=MAX, 1=L1, 2=L2, 3=3D, 4=USER, 5=ATT, 6=GEO, 7=MATRIX,
8=DSJRAND, 9=CRYSTAL, 10=SPARSE, 11-15=RH-norm 1-5, 16=TOROIDAL
17=GEOM, 18=JOHNSON
This confirmed that Concorde is properly installed and working.
After the installation, I ran the TSP code to check the Concorde working using following code
tour_test <- solve_TSP(tsp_test, method = "concorde")
It is working fine now. I got the following output
Used control parameters:
clo =
exe = E:\RA\Concorde_Code\/concorde
precision = 6
verbose = TRUE
keep_files = FALSE
/Concorde_Code/concorde -x -o file225841777aa0.sol file225841777aa0.dat
Host: Pasha Current process id: 1193
Using random seed 1586547969
Problem Name: TSP
Generated by write_TSPLIB (R-package TSP)
Problem Type: TSP
Number of Nodes: 6
Explicit Lengths (CC_MATRIXNORM)
Optimal Solution: 60000.00
Total Running Time: 0.01 (seconds)
Using RHEL7.3
Using R 3.3.2
Installed Rmpi_0.6-6.tar.gz and doMPI_0.2.1.tar.gz
Installed mpich-3.0-3.0.4-10.el7 RPM for x86_64
I created a cluster of three machines (aml1,2,3). I can run the /examples/cpi example from the mpich installation and the processes run without issue on all three machines.
I can also run an R script that needs to be run multiple times, which is discussed on the doMPI documentation -- so the script runs on all clusters.
My problem is when my R script has code prior to the %dopar% that needs to be run once on the master(aml1), and have the %dopar% run on the cluster (aml2,aml3). It only runs on the master. And doMPI says Size of MPI universe: 0 and doesn't recognize aml2 or aml3.
For example:
Run: mpirun -np 1 --hostfile ~/projects/hosts R --no-save -q < example6.R
(and my ~/projects/hosts file is defined to use 8 cores)
example6.R:
library(doMPI) #load doMPI library
cl <- startMPIcluster(verbose=TRUE)
#load data
#clean data
#perform some functions
#let's say I want to have this done in the script and only parallelize this
x <- foreach(seed=c(7, 11, 13), .combine="cbind") %dopar% {
set.seed(seed)
rnorm(3)
}
x
closeCluster(cl)
Output of example6.R:
Master processor name: aml1; nodename: aml1
Size of MPI universe: 0
Spawning 2 workers using the command:
/usr/lib64/R/bin/Rscript /usr/lib64/R/library/doMPI/RMPIworker.R WORKDIR=/home/spark LOGDIR=/home/spark MAXCORES=1 COMM=3 INTERCOMM=4 MTAG=10 WTAG=11 INCLUDEMASTER=TRUE BCAST=TRUE VERBOSE=TRUE
2 slaves are spawned successfully. 0 failed.
If I define cl <- startMPIcluster(count=34, verbose=TRUE) I still get the following but at least I can run 34 slaves:
Master processor name: aml1; nodename: aml1
Size of MPI universe: 0
34 slaves are spawned successfully. 0 failed.
How can I troubleshoot this? I would like to run the R script so it runs the first portion once on the master, and then do %dopar% on the cluster.
Thanks!!
Update 1
Since the last update, I tried running an older version of OpenMPI:
[spark#aml1 ~]$ which mpirun
/opt/openmpi-1.8.8/bin/mpirun
Per #SteveWeston, I created the following script and ran it:
[spark#aml1 ~]$ cat sanity_check.R
library(Rmpi)
print(mpi.comm.rank(0))
mpi.quit()
With the following output:
[spark#aml1 ~]$ mpirun -np 3 --hostfile ~/projects/hosts R --slave -f sanity_check.R
FIPS mode initialized
master (rank 0, comm 1) of size 3 is running on: aml1
slave1 (rank 1, comm 1) of size 3 is running on: aml1
slave2 (rank 2, comm 1) of size 3 is running on: aml1
[1] 0
Here it just hangs -- and nothing happens.
I've already accepted #SteveWeston's answer as it helped me in better understanding my original question.
I commented to his answer that I was still having issues with my R script hanging; the scripts would run, but it would never finish on its own or close its own clusters and I would have to kill it with ctrl-C.
I ultimately set up an nfs environment, build and installed openmpi-1.10.5 there, and installed my R libraries there as well. R is installed separately on both machines, but they share the same library in my nfs directory. Previously I had installed and managed everything under root, including the R libraries (I know). I'm not sure if this what caused complications, but my issues seem to be resolved.
[master#aml1 nfsshare]$ cat sanity_check.R
library(Rmpi)
print(mpi.comm.rank(0))
mpi.quit(save= "no")
[master#aml1 nfsshare]$ mpirun -np 3 --hostfile hosts R --slave -f sanity_check.R
FIPS mode initialized
[1] 1
[1] 0
[1] 2
# no need to ctrl-C here. It no longer hangs
I'm trying to install riak on my OSX 10.8.5, but when using the command riak-admin test it always fail. I can't find a solution for it!
Also using sudo riak-admin test doesn't help it.
I have installed riak(1.4.2) through brew.
>riak start
!!!!
!!!! WARNING: ulimit -n is 256; 4096 is the recommended minimum.
!!!!
>riak ping
pong
>riak-admin test
Failed to write test value: {error,timeout}%
I have installed riak(1.4.2) precompiled tarball using wget
>curl -O http://s3.amazonaws.com/downloads.basho.com/riak/1.4/1.4.2/osx/10.8/riak-1.4.2-OSX-x86_64.tar.gz
>tar xzvf riak-1.4.2-osx-x86_64.tar.gz
>cd riak-1.4.2
>bin/riak start
!!!!
!!!! WARNING: ulimit -n is 256; 4096 is the recommended minimum.
!!!!
>bin/riak ping
pong
>bin/riak-admin test
Failed to write test value: {error,timeout}%
I have install riak(1.4.1) precompiled tarball using wget
>curl -O http://s3.amazonaws.com/downloads.basho.com/riak/1.4/1.4.1/osx/10.8/riak-1.4.1-OSX-x86_64.tar.gz
>tar xzvf riak-1.4.1-osx-x86_64.tar.gz
>cd riak-1.4.1
>bin/riak start
!!!!
!!!! WARNING: ulimit -n is 256; 4096 is the recommended minimum.
!!!!
>bin/riak ping
pong
>bin/riak-admin test
Failed to read test value: {error,{insufficient_vnodes,0,need,1}}%
Solution
Following this procedure http://docs.basho.com/riak/... solved my issue.
It has to do with the Open Files Limit on mac OSX.
Before
To check the current limits on your Mac OS X system, run:
>launchctl limit maxfiles
maxfiles 256 unlimited
Edit (or create) /etc/launchd.conf
Edit (or create) /etc/launchd.conf and increase the limits. Add lines
that look like the following (using values appropriate to your
environment):
limit maxfiles 16384 32768
Restart the system
Save the file, and restart the system for the new limits to take
effect. After restarting, verify the new limits with the launchctl
limit command:
>launchctl limit maxfiles
maxfiles 16384 32768