I'm trying to use the hugo-future-imperfect theme but I cannot make new Rmarkdown posts using: Addins > New Post > <>.
When I click done there is no file in the editor or in the filesystem, and the Console reports:
Loading required package: shiny
Warning: package ‘shiny’ was built under R version 3.3.3
Listening on http://127.0.0.1:7789
Error: open : The system cannot find the file specified.
Warning: running command '"C:\Users\jrl\AppData\Roaming\Hugo\hugo.exe" new "post/2017-06-26-test.Rmd" -k default' had status 65535
Error: open : The system cannot find the file specified.
Warning: running command '"C:\Users\jrl\AppData\Roaming\Hugo\hugo.exe" new "post/2017-06-26-test.Rmd" -k default' had status 65535
The system cannot find the path specified.
Warning: running command 'C:\WINDOWS\system32\cmd.exe /c "C:\Users\jrl\AppData\Roaming\Hugo\hugo.exe" new "post/2017-06-26-test.Rmd" -k default' had status 1
Warning: Error in shell: '"C:\Users\jrl\AppData\Roaming\Hugo\hugo.exe" new "post/2017-06-26-test.Rmd" -k default' execution failed with error code 1
Stack trace (innermost first):
82: shell
81: shell2
80: system2
79: hugo_cmd
78: new_content
77: blogdown::new_post
76: observeEventHandler
12: shiny::runApp
11: shiny::runGadget
10: eval
9: eval
8: eval
7: eval
6: eval.parent
5: local
4: eval
3: eval
2: sys.source
1: blogdown:::new_post_addin
The steps I used to build the site were:
blogdown::new_site(theme = 'jpescador/hugo-future-imperfect')
This sounds like a bug that has been fixed in Hugo 0.24.1 the other day. Please restart your R session, try to reinstall blogdown, and update Hugo:
devtools::install_github('rstudio/blogdown')
blogdown::update_hugo()
Related
After some trials, I was able to install Rmpi package on my computer using the following code:
R CMD INSTALL -l /storage/home/***/.R Rmpi_0.6-7.tar.gz --configure-args="--with-Rmpi-type=OPENMPI --disable-dlopen --with-Rmpi-include=/gpfs/group/RISE/sw7/openmpi_4.1.4_gcc-9.3.1/include --with-Rmpi-libpath=/gpfs/group/RISE/sw7/openmpi_4.1.4_gcc-9.3.1/lib"
I tried to run the following test code:
# Load the R MPI package if it is not already loaded.
if (!is.loaded("mpi_initialize")) {
library("Rmpi")
}
ns <- mpi.universe.size() - 1
mpi.spawn.Rslaves(nslaves=ns)
#
# In case R exits unexpectedly, have it automatically clean up
# resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function(){
if (is.loaded("mpi_initialize")){
if (mpi.comm.size(1) > 0){
print("Please use mpi.close.Rslaves() to close slaves.")
mpi.close.Rslaves()
}
print("Please use mpi.quit() to quit R")
.Call("mpi_finalize")
}
}
# Tell all slaves to return a message identifying themselves
mpi.bcast.cmd( id <- mpi.comm.rank() )
mpi.bcast.cmd( ns <- mpi.comm.size() )
mpi.bcast.cmd( host <- mpi.get.processor.name() )
mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size()))
# Test computations
x <- 5
x <- mpi.remote.exec(rnorm, x)
length(x)
x
# Tell all slaves to close down, and exit the program
mpi.close.Rslaves(dellog = FALSE)
mpi.quit()
On my HPC I run the following:
qsub -A open -l walltime=6:00:00 -l nodes=4:ppn=4:stmem -I
module use /gpfs/group/RISE/sw7/modules
module load openmpi/4.1.4-gcc.9.3.1 r/4.0.3
mpirun -np 4 Rscript "codes/test/test4.R"
But then I get the following error indicating that I only have 1 number of slaves:
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default. The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.
Local host: comp-sc-0222
Local adapter: mlx4_0
Local port: 1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default. The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.
Local host: comp-sc-0222
Local adapter: mlx4_0
Local port: 1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default. The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.
Local host: comp-sc-0222
Local adapter: mlx4_0
Local port: 1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default. The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.
Local host: comp-sc-0222
Local adapter: mlx4_0
Local port: 1
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: comp-sc-0222
Local device: mlx4_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: comp-sc-0222
Local device: mlx4_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: comp-sc-0222
Local device: mlx4_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.
Local host: comp-sc-0222
Local device: mlx4_0
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
Choose a positive number of slaves.
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
Choose a positive number of slaves.
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
Choose a positive number of slaves.
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
Choose a positive number of slaves.
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
I have tried specifying different number of np's but still get the same error. What could be the cause here?
============================================================
(EDIT)
It seems that my original command to load the modules also load intel/19.1.2 and mkl/2020.3. If I unload them, I do see that OMPI_UNIVERSE_SIZE=4.
[****#comp-sc-0220 work]$ module purge
[****#comp-sc-0220 work]$ module load openmpi/4.1.4-gcc.9.3.1 r/4.0.3
[****#comp-sc-0220 work]$ module list
Currently Loaded Modules:
1) openmpi/4.1.4-gcc.9.3.1 2) intel/19.1.2 3) mkl/2020.3 4) r/4.0.3
[****#comp-sc-0220 work]$ mpirun -np 4 env | grep OMPI_UNIVERSE_SIZE
[****#comp-sc-0220 work]$ type mpirun; mpirun --version; mpirun -np 1 env | grep OMPI
mpirun is /opt/aci/intel/compilers_and_libraries_2020.2.254/linux/mpi/intel64/bin/mpirun
Intel(R) MPI Library for Linux* OS, Version 2019 Update 8 Build 20200624 (id: 4f16ad915)
Copyright 2003-2020, Intel Corporation.
LMOD_FAMILY_COMPILER_VERSION=19.1.2
LMOD_FAMILY_COMPILER=intel
[****#comp-sc-0220 work]$ module purge
[****#comp-sc-0220 work]$ module load openmpi/4.1.4-gcc.9.3.1 r/4.0.3
[****#comp-sc-0220 work]$ module unload intel mkl
[****#comp-sc-0220 work]$ module list
Currently Loaded Modules:
1) openmpi/4.1.4-gcc.9.3.1 2) r/4.0.3
[****#comp-sc-0220 work]$ mpirun -np 4 env | grep OMPI_UNIVERSE_SIZE
OMPI_UNIVERSE_SIZE=4
OMPI_UNIVERSE_SIZE=4
OMPI_UNIVERSE_SIZE=4
OMPI_UNIVERSE_SIZE=4
[****#comp-sc-0220 work]$ type mpirun; mpirun --version; mpirun -np 1 env | grep OMPI
mpirun is /gpfs/group/RISE/sw7/openmpi_4.1.4_gcc-9.3.1/bin/mpirun
mpirun (Open MPI) 4.1.4
Report bugs to http://www.open-mpi.org/community/help/
OMPI_MCA_pmix=^s1,s2,cray,isolated
OMPI_COMMAND=env
OMPI_MCA_orte_precondition_transports=954e2ae0a9569e46-2223294369d728a3
OMPI_MCA_orte_local_daemon_uri=4134338560.0;tcp://10.102.201.220:58039
OMPI_MCA_orte_hnp_uri=4134338560.0;tcp://10.102.201.220:58039
OMPI_MCA_mpi_oversubscribe=0
OMPI_MCA_orte_app_num=0
OMPI_UNIVERSE_SIZE=4
OMPI_MCA_orte_num_nodes=1
OMPI_MCA_shmem_RUNTIME_QUERY_hint=mmap
OMPI_MCA_orte_bound_at_launch=1
OMPI_MCA_ess=^singleton
OMPI_MCA_orte_ess_num_procs=1
OMPI_COMM_WORLD_SIZE=1
OMPI_COMM_WORLD_LOCAL_SIZE=1
OMPI_MCA_orte_tmpdir_base=/tmp
OMPI_MCA_orte_top_session_dir=/tmp/ompi.comp-sc-0220.26954
OMPI_MCA_orte_jobfam_session_dir=/tmp/ompi.comp-sc-0220.26954/pid.8212
OMPI_NUM_APP_CTX=1
OMPI_FIRST_RANKS=0
OMPI_APP_CTX_NUM_PROCS=1
OMPI_MCA_initial_wdir=/storage/work/k/****
OMPI_MCA_orte_launch=1
OMPI_MCA_ess_base_jobid=4134338561
OMPI_MCA_ess_base_vpid=0
OMPI_COMM_WORLD_RANK=0
OMPI_COMM_WORLD_LOCAL_RANK=0
OMPI_COMM_WORLD_NODE_RANK=0
OMPI_MCA_orte_ess_node_rank=0
OMPI_FILE_LOCATION=/tmp/ompi.comp-sc-0220.26954/pid.8212/0/0
But if I run the same test4.R again, I get the following error:
/gpfs/group/RISE/sw7/R-4.0.3-intel-19.1.2-mkl-2020.3/R-4.0.3/../install/lib64/R/bin/exec/R: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/gpfs/group/RISE/sw7/R-4.0.3-intel-19.1.2-mkl-2020.3/R-4.0.3/../install/lib64/R/bin/exec/R: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
/gpfs/group/RISE/sw7/R-4.0.3-intel-19.1.2-mkl-2020.3/R-4.0.3/../install/lib64/R/bin/exec/R: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
/gpfs/group/RISE/sw7/R-4.0.3-intel-19.1.2-mkl-2020.3/R-4.0.3/../install/lib64/R/bin/exec/R: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[63743,1],0]
Exit code: 127
--------------------------------------------------------------------------
============================================================
(EDIT 2)
I changed my module load command again to module load openmpi/4.1.4-gcc.9.3.1 r/4.0.5-gcc-9.3.1. With this newer version of R I ran my test4.R script again with mpirun -np 4 Rscript "codes/test/test4.R". It is now returning a new error message as follows:
[1] "/storage/home/k/kxk5678/.R"
[2] "/gpfs/group/RISE/sw7/R-4.0.5-gcc-9.3.1/install/lib64/R/library"
[1] "/storage/home/k/kxk5678/.R"
[2] "/gpfs/group/RISE/sw7/R-4.0.5-gcc-9.3.1/install/lib64/R/library"
[1] "/storage/home/k/kxk5678/.R"
[2] "/gpfs/group/RISE/sw7/R-4.0.5-gcc-9.3.1/install/lib64/R/library"
[1] "/storage/home/k/kxk5678/.R"
[2] "/gpfs/group/RISE/sw7/R-4.0.5-gcc-9.3.1/install/lib64/R/library"
[1] 4
[1] 4
[1] 4
[1] 4
--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
MPI_ERR_SPAWN: could not spawn processes
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
MPI_ERR_SPAWN: could not spawn processes
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
MPI_ERR_SPAWN: could not spawn processes
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
MPI_ERR_SPAWN: could not spawn processes
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn
Execution halted
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[62996,1],1]
Exit code: 1
--------------------------------------------------------------------------
Install the package pbdMPI in an R session on the login node and run the following translation of the Rmpi test code into the use of pbdMPI:
library(pbdMPI)
ns <- comm.size()
# Tell all R sessions to return a message identifying themselves
id <- comm.rank()
ns <- comm.size()
host <- system("hostname", intern = TRUE)
comm.cat("I am", id, "on", host, "of", ns, "\n", all.rank = TRUE)
# Test computations
x <- 5
x <- rnorm(x)
comm.print(length(x))
comm.print(x, all.rank = TRUE)
finalize()
You run it the same way you used for the Rmpi version: mpirun -np 4 Rscript your_new_script_file.
Spawning MPI (as in the Rmpi example) was appropriate when running on clusters of workstations but on an HPC cluster the prevalent way to program with MPI is SPMD - single program multiple data. SPMD means that your code is a generalization of a serial code that is able to have several copies of itself cooperate with each other.
In the above example, cooperation happens only with printing (the comm... functions). There is no manager/master, just several R sessions running the same code (usually computing something different based on comm.rank()) and cooperating/communicating via MPI. This is the prevalent way of large scale parallel computing on HPC clusters.
I am having the error described here:
Chef::Exceptions::ChecksumMismatch when installing nginx-1.7.8 from source
However, that fix does not work for me.
Following the suggestions on that page, I created a shasum:
shasum -a 256 nginx-1.4.4.tar
ed744d916a19e426fd6885e14036d64156f4b6f6aca1d22a1b9307de8d46e8d3 nginx-1.4.4.tar
then I opened this file:
vi /opt/chef/cookbooks/nginx/recipes/source.rb
and I found this block:
remote_file nginx_url do
source nginx_url
checksum node['nginx']['source']['checksum']
path src_filepath
backup false
end
which I changed to this:
remote_file nginx_url do
source nginx_url
checksum 'ed744d916a19e426fd6885e14036d64156f4b6f6aca1d22a1b9307de8d46e8d3'
path src_filepath
backup false
end
but I continue to get this error:
Recipe: nginx::source
* yum_package[pcre-devel] action install (up to date)
* yum_package[openssl-devel] action install (up to date)
* remote_file[http://nginx.org/download/nginx-1.4.4.tar.gz] action create
================================================================================
Error executing action `create` on resource 'remote_file[http://nginx.org/download/nginx-1.4.4.tar.gz]'
================================================================================
Chef::Exceptions::ChecksumMismatch
----------------------------------
Checksum on resource (ed744d) does not match checksum on content (7c989a)
Resource Declaration:
---------------------
# In /opt/chef/cookbooks/nginx/recipes/source.rb
58: remote_file nginx_url do
59: source nginx_url
60: checksum 'ed744d916a19e426fd6885e14036d64156f4b6f6aca1d22a1b9307de8d46e8d3'
61: path src_filepath
62: backup false
63: end
64:
Compiled Resource:
------------------
# Declared in /opt/chef/cookbooks/nginx/recipes/source.rb:58:in `from_file'
remote_file("http://nginx.org/download/nginx-1.4.4.tar.gz") do
provider Chef::Provider::RemoteFile
action [:create]
retries 0
retry_delay 2
default_guard_interpreter :default
source ["http://nginx.org/download/nginx-1.4.4.tar.gz"]
use_etag true
use_last_modified true
declared_type :remote_file
cookbook_name :nginx
recipe_name "source"
checksum "ed744d916a19e426fd6885e14036d64156f4b6f6aca1d22a1b9307de8d46e8d3"
path "/var/chef/cache/nginx-1.4.4.tar.gz"
atomic_update true
end
Running handlers:
[2016-02-16T15:40:44+00:00] ERROR: Running exception handlers
Running handlers complete
[2016-02-16T15:40:44+00:00] ERROR: Exception handlers complete
Chef Client failed. 5 resources updated in 19 seconds
[2016-02-16T15:40:44+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2016-02-16T15:40:44+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2016-02-16T15:40:44+00:00] ERROR: remote_file[http://nginx.org/download/nginx-1.4.4.tar.gz] (nginx::source line 58) had an error: Chef::Exceptions::ChecksumMismatch: Checksum on resource (ed744d) does not match checksum on content (7c989a)
[2016-02-16T15:40:44+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Does anyone have any advice?
I am running this inside of vagrant, on my local machine (a Mac).
It sounds like this was resolved in comments above, but for the record you need to make sure the checksum exactly the file being downloaded. This can be confusing in cases with gzip compression.
Run sha256sum command against the file and use the output as checksum. Seems like you're using md5 or sha1 hash algorithm as checksum
Good afternoon,
I have installed the Generic enablers Cosmos, following the manual BigData Analysis - Installation and Administration Guide. When I have come to 'Step 7: applying Puppet' and executed the commands, in the file puppet.err has appeared the following errors:
Error: Could not prefetch yumrepo provider ' inifilé: Section 'openvz-utils' is already defined, cannot re-defines in/etc/yum.repos.d/openvz.repo
Description: There is a conflict with the titles (indicated in bold type) of the file /etc/yum.repos.d/cosmos-openvz.repo and /etc/yum.repos.d/openvz.repo .
cat /etc/yum.repos.d/cosmos-openvz.repo
[openvz-utils]
...
[openvz-kernel-rhel6]
...
cat /etc/yum.repos.d/openvz.repo
[openvz-utils]
...
[openvz-kernel-rhel6]
...
[openvz-kernel-rhel6-testing]
...
Solution: I have realized a change in the titles of the file /etc/yum.repos.d/openvz.repo, example: [openvz-utils_1]
Error: Could not prefetch database_grant provider 'mysql': Execution of '/usr/bin/mysql mysql -Be describe user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Description: in the folder /var/lib/mysql/ was not found the file mysql.sock.
Solution: I have installed mysql-server.x86_64:
yum install mysql-server.x86_64
At the end of the installations, I restarted the service:
/etc/init.d/mysqld stop
/etc/init.d/mysqld start
Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y list vzstats' returned 1: Error: Cannot retrieve repository metadata (repomd.xml) for repository: ambari. Please verify its path and try again
Description: This error appears in the machine of the Master node, this one is provoked by the configuration of the file [COSMOS_TMP_PATH]/puppet/modules/cosmos/manifests/hieradata/my_environment/common.yaml, indicated in 'Step 6: Puppet configuration'. Concretely, the URL that use the IP: 130.206.81.65
Solution: in the file [COSMOS_TMP_PATH]/puppet/modules/cosmos/manifests/hieradata /my_environment/common.yaml to change the line:
ambari::params::repo_url: 'http:// 130.206.81.65/cosmos/ambari/'
(without blank space)
for
ambari::params::repo_url: 'http:// public-repo-1.hortonworks.com/ambari/centos6/1.x/GA'
(without blank space)
Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y list vzstats' returned 1: Error: Cannot retrieve repository metadata (repomd.xml) for repository: cosmos-libvirt. Please verify its path and try again
Description: it is the same problem as the previous error. The difficulty in this one is that I cannot modify the file [COSMOS_TMP_PATH]/puppet/modules/cosmos/manifests/hieradata /my_environment/common.yaml in the line:
cosmos::params::cosmos_repo_deps_url: 'http:// 130.206.81.65/cosmos/rpms/cosmos-deps'
(without blank space)
Because it is line is used in several files:
cat /etc/yum.repos.d/cosmos-libvirt.repo
[cosmos-libvirt]
name=Cosmos LibVirt with OpenVZ - v1.0.5 - NO PolKIT
baseurl=http:// 130.206.81.65/cosmos/rpms/cosmos-deps//libvirt
gpgcheck=0
priority=10
enabled=1
cat /etc/yum.repos.d/cosmos-openvz.repo
[openvz-utils]
name=OpenVZ utilities
baseurl=http:// 130.206.81.65/cosmos/rpms/cosmos-deps//OpenVZ/openvz-utils
enabled=1
gpgcheck=0
priority=1
[openvz-kernel-rhel6]
name=OpenVZ RHEL6-based kernel
baseurl=http:// 130.206.81.65/cosmos/rpms/cosmos-deps//OpenVZ/openvz-kernel- > rhel6
enabled=1
gpgcheck=0
priority=1
It does not also allow to modify the file previous, in the moment to execute the command (indicated in 'Step 7: applying Puppet'):
puppet apply --debug --verbose \
--modulepath [COSMOS_TMP_PATH]/puppet/modules/:[COSMOS_TMP_PATH]/puppet/modules_third_party/ \
--environment my_environment --hiera_config [COSMOS_TMP_PATH]/puppet/modules/cosmos/manifests/hiera.yaml \
--manifestdir [COSMOS_TMP_PATH]/puppet/modules/cosmos/manifests/ [COSMOS_TMP_PATH]/puppet/modules/cosmos/manifests/site.pp \
> puppet.out 2> puppet.err
It will erase the modified.
Solution: https://github.com/telefonicaid/fiware-cosmos-platform/issues/4
I need help with the error:
Error: /Stage[main]/Ambari::Server::Config/Augeas[ambari-config-repoinfo]: Could not evaluate: Saving failed, see debug
Might they to throw me a hand with these last error?
Thank you in advance.
PD: Forgive for if it is written badly
I can't get a public key for my Moodle network, the xlrpc is installed on Ubuntu. Can anyone help?
Moodle: 1.9.4
Apache: 2.2.8
PHP: 5.2.4-2
Error on debug mode:
Warning: openssl_csr_new() [function.openssl-csr-new]: dn: add_entry_by_NID 17 ->
Moodle (failed) in /var/www/moodle_teste/mnet/lib.php on line 361
Warning: openssl_csr_sign() [function.openssl-csr-sign]: cannot get CSR from
parameter 1 in /var/www/moodle_teste/mnet/lib.php on line 362
Warning: openssl_x509_export() [function.openssl-x509-export]: cannot get cert
from parameter 1 in /var/www/moodle_teste/mnet/lib.php on line 380
Warning: openssl_x509_free() expects parameter 1 to be resource, boolean given
in /var/www/moodle_teste/mnet/lib.php on line 381
I have a R script which works perfectly fine in R Colsole ,but when I am running in Hadoop streaming it is failing with the below error in Map phase .Find the Task attempts log
The Hadoop Streaming Command I have :
/home/Bibhu/hadoop-0.20.2/bin/hadoop jar \
/home/Bibhu/hadoop-0.20.2/contrib/streaming/*.jar \
-input hdfs://localhost:54310/user/Bibhu/BookTE1.csv \
-output outsid -mapper `pwd`/code1.sh
stderr logs
Loading required package: class
Error in read.table(file = file, header = header, sep = sep, quote = quote, :
no lines available in input
Calls: read.csv -> read.table
Execution halted
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
syslog logs
2013-07-03 19:32:36,080 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2013-07-03 19:32:36,654 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2013-07-03 19:32:36,675 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2013-07-03 19:32:36,835 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2013-07-03 19:32:36,835 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2013-07-03 19:32:36,899 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/home/Bibhu/Downloads/SentimentAnalysis/Sid/smallFile/code1.sh]
2013-07-03 19:32:37,256 INFO org.apache.hadoop.streaming.PipeMapRed: Records R/W=0/1
2013-07-03 19:32:38,509 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done
2013-07-03 19:32:38,509 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed!
2013-07-03 19:32:38,557 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
2013-07-03 19:32:38,631 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the task
write hadoopStreamming jar with full version like hadoop-streaming-1.0.4.jar
specify separate file path for mapper & reducer with -file option
tell hadoop which is your mapper & reducer code with -mapper & -reducer option
for more ref see Running WordCount on Hadoop using R script
You need to find the logs from your mappers and reducers, since this is the place where the job is failing (as indicated by java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1). This says that your R script crashed.
If you are using the Hortonworks Hadoop distribuion, the easiest way is to open your jobhistory. It should be at http://127.0.0.1:19888/jobhistory . It should be possible to find the log in the filesystem using the command line as well, but I haven't yet found where.
Open http://127.0.0.1:19888/jobhistory in your web browser
Click on the Job ID of the failed job
Click the number indicating the failed job count
Click an attempt link
Click the logs link
You should see a page which looks something like
Log Type: stderr
Log Length: 418
Traceback (most recent call last):
File "/hadoop/yarn/local/usercache/root/appcache/application_1404203309115_0003/container_1404203309115_0003_01_000002/./mapper.py", line 45, in <module>
mapper()
File "/hadoop/yarn/local/usercache/root/appcache/application_1404203309115_0003/container_1404203309115_0003_01_000002/./mapper.py", line 37, in mapper
for record in reader:
_csv.Error: newline inside string
This is an error from my Python script, the errors from R look a bit different.
source: http://hortonworks.com/community/forums/topic/map-reduce-job-log-files/
I received this same error tonight, while also developing Map Reduce Streaming jobs with R.
I was working on a 10 node cluster, each with 12 cores, and tried to supply at submission time:
-D mapred.map.tasks=200\
-D mapred.reduce.tasks=200
The job completed successfully though when I changed these to
-D mapred.map.tasks=10\
-D mapred.reduce.tasks=10
This was a mysterious fix, and perhaps more context will arise this evening. But if any readers can elucidate, please do!