Environment:
CentOS 7 64bit
a) created system user ds:
useradd -r -s /sbin/nologin ds
b) downloaded julia executable tar and extracted to /opt/julia:
tar -xvf "/tmp/downloads/julia/julia-1.3.1-linux-x86_64.tar.gz" -C "/opt/julia"
c) made ds as owner of folders and files of /opt/julia and its sub folders:
chown -R "ds:ds" "/opt/julia"
I would like to setup shared julia environment for all users.
Please guide me in setting up the following, in the shared environment for all the users:
a) history
b) JULIA_HOME
c) JULIA_LOAD_PATH
d) JULIA_DEPOT_PATH
e) JULIA_PKG_DEVDIR
f) JULIA_PROJECT
g) PATH
h) security
Note: this proposed solution is valid for Ubuntu. Paths may be different in CentOS, but the process should be the same.
The solution I ended up for a similar scenario was to share only the Julia binaries (those downloaded from Julialang.org) expanding them in /lib/julia-1.x/ (with a symlink to /lib/julia.x/bin/julia in /bin) and let each user to have its own ~/.julia folder.
In this way you avoid the mess of having package versions that are not compatible between the global package repository and the local one.
If you go trough this configuration, you can add a script to /usr/local/sbin/adduser.local that will be executed any time you add a new user with adduser.
For example I added:
su $1 -c "julia /usr/bin/initJuliaRepository.jl" where in turn I pre-install on each users a set of commonly used packages:
import Pkg
Pkg.update()
Pkg.add("IJulia")
Pkg.add("DataFrames")
Pkg.add("Plots")
Pkg.build("IJulia")
Related
I am trying to create a completely portable version of R for Mac that I can send to users with no R on their system and they can essentially double click a command file and it launches a Shiny application. I'll need to be able to install packages including some built from source (and some from GitHub).
I am using the script from this GitHub repository (https://github.com/dirkschumacher/r-shiny-electron/blob/master/get-r-mac.sh) as a starting point (it's also pasted below), creating a version of R, but (A) I find that when I try to launch R it gives me an error not finding etc/ldpaths and (B) when I try to launch Rscript it runs my system version -- I run `Rscript -e 'print(R.version)' and it prints out 4.0 which is my system version of R rather than the version 3.5.1 which the shell script has downloaded and processed.
I've experimented with editing the "R" executable and altering R_HOME and R_HOME_DIR but it still runs into issues when I try to install packages to the 3.5.1 directory.
Can anyone provide some guidance?
(By the way docker is not an option, this needs to be as simple as possible end-users with limited technical skills. So having them install docker etc won't be an option)
#!/usr/bin/env bash
set -e
# Download and extract the main Mac Resources directory
# Requires xar and cpio, both installed in the Dockerfile
mkdir -p r-mac
curl -o r-mac/latest_r.pkg \
https://cloud.r-project.org/bin/macosx/R-3.5.1.pkg
cd r-mac
xar -xf latest_r.pkg
rm -r r-1.pkg Resources tcltk8.pkg texinfo5.pkg Distribution latest_r.pkg
cat r.pkg/Payload | gunzip -dc | cpio -i
mv R.framework/Versions/Current/Resources/* .
rm -r r.pkg R.framework
# Patch the main R script
sed -i.bak '/^R_HOME_DIR=/d' bin/R
sed -i.bak 's;/Library/Frameworks/R.framework/Resources;${R_HOME};g' \
bin/R
chmod +x bin/R
rm -f bin/R.bak
# Remove unneccessary files TODO: What else
rm -r doc tests
rm -r lib/*.dSYM
Happy to help you get this working for your shiny app. You can use this github repo for Electron wrapping R/Shiny... just clone, and replace the app.R (for your other packages you need to install them in the local R folder after cloning and then running R from the command line out of the R-Portable-Mac/bin folder...
Try it with the Hello World app.R that is included first
https://github.com/ColumbusCollaboratory/electron-quick-start
And, then installing your packages in the local R-Portable-Mac folder runtime. Included packages by default...
https://github.com/ColumbusCollaboratory/electron-quick-start/tree/master/R-Portable-Mac/library
Your packages will show up here after install.packages() from the command line using the local R-Mac-Portable runtime.
We have been working on a R Addin for this also...
https://github.com/ColumbusCollaboratory/photon
But, note the add-in is still a work in progress and doesn't work with compiled R packages; still have to go into the local R folder and runtime on the command line and install the packages directly into the local R folder libpath as discussed above.
Give it a try and let us know through Github issues if you have any questions and issues. And, if you've already posted out there, sorry we haven't responded as of yet. Would love to communicate through the photon Add-In for this to get it working with compiling packages (into the libPath)--if you have the time to help. Thanks!
I have followed the blog (Below mentioned) here and downloaded the parcel and put as per required.
Please let me know if any one has installed and the steps.
(https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html)
/opt/cloudera/csd/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658-el5.parcel
But service cloudera-scm-server restart is not executing.
To use Cloudera Express (free), run:
sudo /home/cloudera/cloudera-manager --express
This requires at least 8 GB of RAM and at least 2 virtual CPUs.
SPARK 2.2 Installation Setup on Cloudera VM
Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids.
Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.
Step 3: Please open the terminal and switch to root user as:
su root
password: cloudera
Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
(a). Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
(b). Switch to /usr/java/ directory with “cd /usr/java/” command.
(c). cp the java download tar file to the /usr/java/ directory.
(d). Untar the directory with “tar –zxvf jdk-8u31-linux-x64.tar.gz”
(e). Open the profile file with the command “vi ~/.bash_profile”
(f). export JAVA_HOME to the new java directory.
“export JAVA_HOME=/usr/java/jdk1.8.0_131”
Save and Exit.
(g). In order to reflect the above change, following command needs to be executed on the shell:
source ~/.bash_profile
Step 5: The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.
(a). Switch to /opt/ directory with the command:
“cd /opt/”
(b). Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
(c). Untar the spark tar with the following command:
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz
(d). We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
Save and exit
(e). We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home :
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/
(f). Change the permissions of the directory:
chmod 777 -R /tmp/hive
(g). Try “spark-shell”, it should work.
Please follow below video it has all the necessary step required in order to install Sprak2 in Clouedra VM.
youtubue link - https://www.youtube.com/watch?v=lQxlO3coMxM
Also for for starting Cloudera Express (free) your VM should have at-least 8Gb RAM allocated or if you have default 4GB RAM allocated then you can forcefullly start ysing below command and then follow the above video.
sudo /home/cloudera/cloudera-manager --force --express
Try this command
sudo /home/cloudera/cloudera-manager --express --force
I gave up on this, nothing works well with parcel and non-parcel installation.
As soon as cloudera express is started numerous errors and Java 7 instead of Java 8.
I got a mapr VM install with Spark 2.x. No issues. Works first time.
That works well. This is my advice # 1.
If you want KUDU, then I would install centos and install things oneself. This is advice # 2. OK, you may miss Impala, but if for pure research and development then not so much of an issue.
With following two command my spark2.2 was automatically updated to spark 2.4:
(i) sudo yum update
It might be that your java, home path is screwed, in that case please export the java home path in bash file.
(a) vi ~/.bash_profile
(b)
(c) source ~/.bash_profile
Just download the right version of spark that you need say 'spark-2.2.0-bin-hadoop2.6'
open bashrc_profile through vi editor
vi ~/.bash_profile. Paste the below 2 lines
SPARK_HOME=/home/cloudera/Downloads/spark-2.2.0-bin-hadoop2.6
PATH=$PATH:$HOME/bin:$SPARK_HOME/bin
Save it
Then run the command : source ~/.bash_profile
Now start spark-shell .
Note : Make sure you have JDK 1.8 installed
SnPARK 2.2 Installation Setup on Cloudera VM
Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids.
Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.
Step 3: Please open the terminal and switch to root user as:
su root
password: cloudera
Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
(a). Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
(b). Switch to /usr/java/ directory with “cd /usr/java/” command.
(c). cp the java download tar file to the /usr/java/ directory.
(d). Untar the directory with “tar –xvzf jdk-8u31-linux-x64.tar.gz”
(e). Open the profile file with the command “vi ~/.bash_profile”
(f). export JAVA_HOME to the new java directory.
“export JAVA_HOME=/usr/java/jdk1.8.0_131”
Save and Exit.
(g). In order to reflect the above change, following command needs to be executed on the shell:
source ~/.bash_profile
Step 5: The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.
(a). Switch to /opt/ directory with the command:
“cd /opt/”
(b). Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
(c). Untar the spark tar with the following command:
tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz
(d). We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
SPARK_LOCAL_IP=127.0.0.1
Save and exit
(e). We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home :
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/
(f). Change the permissions of the directory:
chmod 777 -R /tmp/hive
(g). Try “spark-shell”, it should work.
Same answeras swapnil shashank with small modification below
SPARK_LOCAL_IP=127.0.0.1
tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz
As the title states: is it possible to install multiple git branches of the same package side-by-side in the same R environment? I want to do some benchmarking and it would be easier to compare the two branches in the same session. I think one workaround is to change the package name in the DESCRIPTION file in the new branch, but is there a more clever way to do this with devtools?
Sample code:
devtools::install_github("mkoohafkan/RAStestR", ref = "master")
# overwrites the prior install
devtools::install_github("mkoohafkan/RAStestR", ref = "hdf5r_transition")
In short, no. At least not without an extra layer. Read on.
While git (the protocol, as well as the client) support "branches" akin to a virtual filesystem allowing you to switch easily, R does not.
For every package you install, one and only one version can be installed.
But don't despair, because the file system can be used as a backend, and R can then switch by adjusting the library path. This is all in help(Startup) but it may help to be explicit.
What you can do (and I mock this here)
mkdir master; cd master; installFromBranch.R master; cd ..
mkdir featureA; cd featureA; installFromBranch.R featureA; cd ..
mkdir featureB; cd featureA; installFromBranch.R featureB; cd ..
and then in R use, say,
.libPaths("master"); library("mypackage")
or if you want a feature
.libPaths("featureA"); library("mypackage")
You can also use R_LIB_USER=featureA Rscript -e '.....someCommandHere...'
So in short: map the branches to directories into which you install and tell R about those directories.
I have a package that previously only targeted RPM based distros for which I am now building .deb packages for Debian based distros.
The aim is to simulate a test installation from user-space that is isolated from the system you are building on. It may be multi-user and you do not want to require root access just to build the software. Many of our tests simulate the installation directory structure already. This is for the next step up to simulate an actual installation using packages built.
For the RPM packages I was able to create test installations using:
WSDIR=/where/I/want/my/tests/to/run
rpmdb --initdb --dbpath "$WSDIR"/rpmdb
rpm --relocate /opt="$WSDIR"/opt --dbpath $WSDIR/rpmdb -i <package>.rpm
The equivalent in the Debian world is something like:
dpkg --force-not-root --admindir=$WSDIR/dpkg --root=$WSDIR/install --install "$DEB"
However, I am stuck over the equivalent to the rpmdb --initdb step.
Note that I can just unpack the archive using:
dpkg-deb -x "$DEB" $WSDIR/install
But I would prefer to be closer to how a real package is installed.
Also I don't think this will run preinstall and postinstall scripts.
Similar questions have suggested using deboostrap to create a chroot environment but this creates a complete new installation. As well as being overkill it is too slow for an automated test. I intend to use this for quick tests of the installation package prior to further testing in actual test environments.
My experiments so far:
(cd $WSDIR/dpkg && mkdir alternatives info parts triggers updates)
cp /var/lib/dpkg/status $WSDIR/dpkg/status
have at best resulted in:
dpkg: error: unable to access dpkg status area: No such file or directory
which does not indicate clear what is wrong.
So how do you create a dpkg admin directory?
Cross posted as https://superuser.com/questions/1271145/how-do-you-create-a-dpkg-admin-directory
Update 24/11/2017
I've tried copying using the dpkg dir from an environment created by [cowdancer][1] (which uses deboostrap under the hood) or copying the real one from /var/lib/dpkg but I still get the same error message so perhaps the error (and/or the --admindir option) doesn't mean quite what I think it means.
Note that:
sudo dpkg --force-not-root --root=$WSDIR/install --admindir=/var/lib/dpkg --install "$DEB"
does work. So it is something to do with the admin dir.
I've also retitled the question as "How do you create a dpkg admin directory" is interesting question but the answer is not necessarily the solution to my problem.
The minimal way to create a dpkg database is something like this:
$ mkdir -p db/{updates,info}
$ touch db/{status,diversions,statoverride}
If you want to use that as non-root, currently the best way is to use fakeroot.
$ mkdir -p fsys
$ PATH=/sbin:/usr/sbin:$PATH fakeroot dpkg --log=/dev/null --admindir=db --instdir=fsys -i pkg.deb
But take into account that passing --root after --admindir or --instdir will reset those paths, which is I think the problem you have been having here.
Also using sudo and --force-not-root does not make much sense? :) And is definitely less confined than using just fakeroot. In the near future it will be possible to run dpkg fully unprivileged in some local tree.
I eventually found an answer for this. Thanks to Guillem Jover for some of this.
Pasting a copy of it here:
mkdir fake
mkdir fake/install
mkdir -p fake/dpkg/info
mkdir -p fake/dpkg/updates
touch fake/dpkg/status
PATH=/sbin:/usr/sbin:$PATH fakeroot dpkg --force-script-chrootless --log=`pwd`/fake/dpkg.log --root=`pwd`/fake --instdir `pwd`/fake --admindir=`pwd`/fake/dpkg --install *.deb
Some points to note:
--force-not-root is not enough. fakeroot is required.
ldconfig and start-stop-daemon must be on the path.
(hence PATH=/sbin:/usr/sbin:$PATH)
The log file needs to be relocated from the default /var/log/dpkg.log
The order of arguments is significant. If used --root must be before --instdir and --admindir.
The admindir is supposed to have a the installation dir as a prefix.
If the package contains any pre or post installation scripts (preinst,postinst) then --force-script-chrootless is required as these scripts are normally run via chroot() which gives operation not permitted when attempted under fakeroot.
For a quick test of trivial dependencies, you can directly install on the system using 'dpkg -i' then 'dpkg -P' and 'apt-get autoremove' to purge the package and clean the dependencies.
An other more secure but slower solution could be to use the autopkgtest package:
https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html
I am trying to install ODBC driver for Debian arrording to these instructions: https://blog.afoolishmanifesto.com/posts/install-and-configure-the-ms-odbc-driver-on-debian/
However trying to run:
sqlcmd -S localhost
I get the error
libcrypto.so.10: cannot open shared object file: No such file or
directory
What could be the cause?
So far I have tried
1.
$ cd /usr/lib
$ sudo ln -s libssl.so.0.9.8 libssl.so.10
$ sudo ln -slibcrypto.so.0.9.8 libcrypto.so.10
2.
/usr/local/lib64 to the /etc/ld.so.conf.d/doubango.conf file
3.
sudo apt-get update
sudo apt-get install libssl1.0.0 libssl-dev
cd /lib/x86_64-linux-gnu
sudo ln -s libssl.so.1.0.0 libssl.so.10
sudo ln -s libcrypto.so.1.0.0 libcrypto.so.10
4. Sudo apt-get install libssl0.9.8:i386
None of these have helped.
As I'm quite familiar with Debian and programming, here is some advice:
if you have questions about setting up your system, ask on SuperUser and/or (if your question is specific to a Un*x flavour) on Unix&Linux
when fuddling around with symlinks to shared-libraries, you should have a thorough understanding of what you are doing. these files are named for a reason - and the reason is to protect you (the user of the system) from weird crashes, because an application is using a wrong/incompatible library.
a tutorial that tells you to do so, should give proper warning and explanation about what you are to do.
So, why are these instructions in the tutorial you are following?
The application you are trying to run, has been linked against libcrypto.so.
On the developer machine (that was used to produce the application binary), libcrypto.so was a symlink to libcrypto.so.10, but this is missing on Debian: maybe because the library has been removed (and replaced by a new and incompatible version), or because Debian uses a different naming scheme as compared to the system that was used to compile the application.
If it is the former, then you cannot solve the issue by using symlinks.
You have to get the right library (or the application linked against the correct libraries).
If it is the latter, you may get away with symlinking the expected library name with the correct library files found on your system. (This is assuming that the only difference between the two systems is indeed the so-naming scheme).
So, how to do it?
first of all, you should find out, against which libraries your application was really linked, and which of these libraries are missing.
$ ldd /path/to/my/app | grep -i "not found"
libfoo.so.10 => not found
then find out, whether you have a (hopefully compatible) library on your system. A good place to start is /usr/lib/. but not-so-recently, Debian has started moving the libraries to /usr/lib/<host-triplet>, with <host-triplet> describing a target architecture. You can find out the default value if your application was indeed built for the architecture you are running (e.g. for linux-amd64) you can get the string by running something like:
$ gcc -print-multiarch
Imagine you discover that you have /usr/lib/x86_64-linux-gnu/libfoo.so.1.0.0.
if you have good reason to believe that this can act as a replacement for libfoo.so.10, you can go make the found library available to your application by means of a symlink, e.g.
# cd /usr/local/lib/
# ln -s /usr/lib/x86_64-linux-gnu/libfoo.so.1.0.0 libfoo.so.10
Finally, you might need to refresh the cache of the dynamic linker so it starts using the new library, by running ldconfig as root/superuser.