Unable to install spark 2.2 in Cloudera Quickstart VM (5.10) - cloudera

I have followed the blog (Below mentioned) here and downloaded the parcel and put as per required.
Please let me know if any one has installed and the steps.
(https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html)
/opt/cloudera/csd/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658-el5.parcel
But service cloudera-scm-server restart is not executing.
To use Cloudera Express (free), run:
sudo /home/cloudera/cloudera-manager --express
This requires at least 8 GB of RAM and at least 2 virtual CPUs.

SPARK 2.2 Installation Setup on Cloudera VM
Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids.
Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.
Step 3: Please open the terminal and switch to root user as:
su root
password: cloudera
Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
(a). Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
(b). Switch to /usr/java/ directory with “cd /usr/java/” command.
(c). cp the java download tar file to the /usr/java/ directory.
(d). Untar the directory with “tar –zxvf jdk-8u31-linux-x64.tar.gz”
(e). Open the profile file with the command “vi ~/.bash_profile”
(f). export JAVA_HOME to the new java directory.
“export JAVA_HOME=/usr/java/jdk1.8.0_131”
Save and Exit.
(g). In order to reflect the above change, following command needs to be executed on the shell:
source ~/.bash_profile
Step 5: The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.
(a). Switch to /opt/ directory with the command:
“cd /opt/”
(b). Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
(c). Untar the spark tar with the following command:
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz
(d). We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
Save and exit
(e). We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home :
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/
(f). Change the permissions of the directory:
chmod 777 -R /tmp/hive
(g). Try “spark-shell”, it should work.

Please follow below video it has all the necessary step required in order to install Sprak2 in Clouedra VM.
youtubue link - https://www.youtube.com/watch?v=lQxlO3coMxM
Also for for starting Cloudera Express (free) your VM should have at-least 8Gb RAM allocated or if you have default 4GB RAM allocated then you can forcefullly start ysing below command and then follow the above video.
sudo /home/cloudera/cloudera-manager --force --express

Try this command
sudo /home/cloudera/cloudera-manager --express --force

I gave up on this, nothing works well with parcel and non-parcel installation.
As soon as cloudera express is started numerous errors and Java 7 instead of Java 8.
I got a mapr VM install with Spark 2.x. No issues. Works first time.
That works well. This is my advice # 1.
If you want KUDU, then I would install centos and install things oneself. This is advice # 2. OK, you may miss Impala, but if for pure research and development then not so much of an issue.

With following two command my spark2.2 was automatically updated to spark 2.4:
(i) sudo yum update
It might be that your java, home path is screwed, in that case please export the java home path in bash file.
(a) vi ~/.bash_profile
(b)
(c) source ~/.bash_profile

Just download the right version of spark that you need say 'spark-2.2.0-bin-hadoop2.6'
open bashrc_profile through vi editor
vi ~/.bash_profile. Paste the below 2 lines
SPARK_HOME=/home/cloudera/Downloads/spark-2.2.0-bin-hadoop2.6
PATH=$PATH:$HOME/bin:$SPARK_HOME/bin
Save it
Then run the command : source ~/.bash_profile
Now start spark-shell .
Note : Make sure you have JDK 1.8 installed

SnPARK 2.2 Installation Setup on Cloudera VM
Step 1: Download a quickstart_vm from the link:
Prefer a vmware platform as it is easy to use, anyways all the options are viable.
Size is around 5.4gb of the entire tar file. We need to provide the business email id as it won’t accept personal email ids.
Step 2: The virtual environment requires around 8gb of RAM, please allocate sufficient memory to avoid performance glitches.
Step 3: Please open the terminal and switch to root user as:
su root
password: cloudera
Step 4: Cloudera provides java –version 1.7.0_67 which is old and does not match with our needs. To avoid java related exceptions, please install java with the following commands:
(a). Downloading Java:
wget -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
(b). Switch to /usr/java/ directory with “cd /usr/java/” command.
(c). cp the java download tar file to the /usr/java/ directory.
(d). Untar the directory with “tar –xvzf jdk-8u31-linux-x64.tar.gz”
(e). Open the profile file with the command “vi ~/.bash_profile”
(f). export JAVA_HOME to the new java directory.
“export JAVA_HOME=/usr/java/jdk1.8.0_131”
Save and Exit.
(g). In order to reflect the above change, following command needs to be executed on the shell:
source ~/.bash_profile
Step 5: The Cloudera VM provides spark 1.6 version by default. However, 1.6 API’s are old and do not match with production environments. In that case, we need to download and manually install Spark 2.2.
(a). Switch to /opt/ directory with the command:
“cd /opt/”
(b). Download spark with the command:
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
(c). Untar the spark tar with the following command:
tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz
(d). We need to define some environment variables as default settings:
Please open a file with the following command:
vi /opt/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh
Paste the following configurations in the file:
SPARK_MASTER_IP=192.168.50.1
SPARK_EXECUTOR_MEMORY=512m
SPARK_DRIVER_MEMORY=512m
SPARK_WORKER_MEMORY=512m
SPARK_DAEMON_MEMORY=512m
SPARK_LOCAL_IP=127.0.0.1
Save and exit
(e). We need to start spark with the following command:
/opt/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
Export spark_home :
export SPARK_HOME=/opt/spark-2.2.0-bin-hadoop2.7/
(f). Change the permissions of the directory:
chmod 777 -R /tmp/hive
(g). Try “spark-shell”, it should work.
Same answeras swapnil shashank with small modification below
SPARK_LOCAL_IP=127.0.0.1
tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz

Related

karaf: JAVA_HOME is not valid: /usr/lib/jvm/java-11-openjdk-amd64/bin/java [duplicate]

I am working in Ubuntu 16.04. I need to install gradle and the gradle is installed when i checked with sudo apt list --installed command but when i use gradle -version command it shows the following error,
JAVA_HOME is set to an invalid directory: /usr/lib/jvm/java-8-oracle/jre/bin/java
In sudo vim /etc/environment file,
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/"
http_proxy="http://username:password#IP:port no/"
https_proxy="https://IP:port no/"
ftp_proxy="ftp://IP:port no/"
I don't know where i made mistakes. Please help me.
Thanks.
On a 64bit openSuse 64 42.1 box;
readlink -f $(which java)
provided;
/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/bin/java
But;
export JAVA_HOME=/usr/lib64/jvm/jre-1.8.0-openjdk
is the path that worked and allowed java emulator to run.
So i think we have to manually browse our file system and see what path to choose.
Today I faced this problem. I am using the default java that comes with your linux distro (so in my case, linux mint).
$ whereis java
This command gave me
java: /usr/bin/java /usr/share/java
So, I opened /user/bin. There was a link to Java. I right clicked it and selected follow original link. This lead me to /usr/lib/jvm/java-11-openjdk-amd64/bin/java.
So now that I know where this java is, I opened my .bashrc file, and edited the JAVA_HOME.
So for my case,
## My Custom variables
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
This solved the problem.
Now if you are using some other java (say you downloaded from oracle and extracted the zip file ...), then you have to add that location. So for example, if your java is in /home/user/.sdkman/candidates/java/current, then
export JAVA_HOME=/home/user/.sdkman/candidates/java/current
export PATH=$JAVA_HOME/bin:$PATH
I see a mismatch. In your enviornment file the JAVA_HOME is set to "/usr/lib/jvm/java-8-openjdk-amd64/" and your mentioned that the error that you got relates to the JAVA_HOME as "/usr/lib/jvm/java-8-oracle/jre/bin/java"
If you JAVA is really installed in /usr/lib/jvm/java-8-oracle directory, then you need to ensure that the JAVA_HOME is set to that directory. And also your PATH reflects $JAVA_HOME/bin in it.
I typically install Oracle JDK/JRE separately in a separate directory such as /usr/local/jdk1.8.0 etc.
check the jvm installtion folder from Files
eg : /usr/lib/jvm/java-12-oracle
then in terminal run sudo nano /etc/environment and add the line
JAVA_HOME="/usr/lib/jvm/java-12-oracle"
Then open terminal and run
export JAVA_HOME="/usr/lib/jvm/java-12-oracle"

how can I check if Leiningen is successfully installed and working at full capacity?

Java JDK is installed, everything works great, also maven works great.
I followed the instructions from the site, downloaded the bat file, set the path and at the command prompt started the installation with the command lein self-install.
what i got in response is downloading leiningen now ....
after a while, C: \ windows \ system32> appeared below that line and it stays that way for several hours.
What more should I do?
The command line normally uses a prompt like > or similar to indicate it is waiting for a command. Then you type something followed by the <return> key:
> lein --version
Leiningen 2.9.8 on Java 1.8.0_322 OpenJDK 64-Bit Server VM
>
Here I typed lein --version, then <ret>, and the computer responded with a 1-liner about the version of Leiningen and Java that are installed. Then it displayed the prompt again, waiting for the next command.
You can use the cd (change directory) command to move around. With no prompt it will send you back to your "home" directory. See docs for the BASH or ZSH shell.
For Windows, I highly recommend installing the BASH Shell and git from this site:
https://gitforwindows.org/
IMHO the bash shell is greatly superior to the built-in Windows shell from CMD.EXE

How to install and use mirthSync on MacOS?

Setup
I'm following the installation directions in the mirthSync readme, which is to clone the repo. The next indication of usage that I can see is in the Examples section, which via CLI is to "pull Mirth Connect code from a Mirth Connect instance":
java -jar mirthsync.jar -s https://localhost:8443/api -u admin -p admin pull -t /home/user/
I'm assuming that after cloning the repo, one should cd into that directory and then run the java -jar... command with all the appropriate flag values (server, username, password, etc).
Error
After running the CLI command, I get this error:
Error: Unable to access jarfile mirthsync.jar
Question
Where is this mirthsync.jar file supposed to come from? Is there something I need to do in order to generate the mirthsync.jar file?
Generate it via lein uberjar (which creates target/uberjar/*-standalone.jar) or download it from a release.

How do I enable python35 from Software Collections at login?

I followed the Software Collections Quick Start and I now have Python 3.5 installed. How can I make it always enabled in my ~/.bashrc, so that I do not have to enable it manually with scl enable rh-python35 bash?
Use the scl_source feature.
Create a new file in /etc/profile.d/ to enable your collection automatically on start up:
$ cat /etc/profile.d/enablepython35.sh
#!/bin/bash
source scl_source enable python35
See How can I make a Red Hat Software Collection persist after a reboot/logout? for background and details.
This answer would be helpful to those who have limited auth access on the server.
I had a similar problem for python3.5 in HostGator's shared hosting. Python3.5 had to be enabled every single damn time after login. Here are my 10 steps for the resolution:
Enable the python through scl script python_enable_3.5 or scl enable rh-python35 bash.
Verify that it's enabled by executing python3.5 --version. This should give you your python version.
Execute which python3.5 to get its path. In my case, it was /opt/rh/rh-python35/root/usr/bin/python3.5. You can use this path to get the version again (just to verify that this path is working for you.)
Awesome, now please exit out of the current shell of scl.
Now, lets get the version again through this complete python3.5 path /opt/rh/rh-python35/root/usr/bin/python3.5 --version.
It won't give you the version but an error. In my case, it was
/opt/rh/rh-python35/root/usr/bin/python3.5: error while loading shared libraries: libpython3.5m.so.rh-python35-1.0: cannot open shared object file: No such file or directory
As mentioned in Tamas' answer, we gotta find that so file. locate doesn't work in shared hosting and you can't install that too.
Use the following command to find where that file is located:
find /opt/rh/rh-python35 -name "libpython3.5m.so.rh-python35-1.0"
Above command would print the complete path (second line) of the file once located. In my case, output was
find: `/opt/rh/rh-python35/root/root': Permission denied
/opt/rh/rh-python35/root/usr/lib64/libpython3.5m.so.rh-python35-1.0
Here is the complete command for the python3.5 to work in such shared hosting which would give the version,
LD_LIBRARY_PATH=/opt/rh/rh-python35/root/usr/lib64 /opt/rh/rh-python35/root/usr/bin/python3.5 --version
Finally, for shorthand, append the following alias in your ~/.bashrc
alias python351='LD_LIBRARY_PATH=/opt/rh/rh-python35/root/usr/lib64 /opt/rh/rh-python35/root/usr/bin/python3.5'
For verification, reload the .bashrc by source ~/.bashrc and execute python351 --version.
Well, there you go, now whenever you login again, you have got python351 to welcome you.
This is not just limited to python3.5, but can be helpful in case of other scl installed softwares.

Completely remove openstack from system after installation from devstack script

I am installing OpenStack on my local machine via this link. But I am having trouble in completely removing installed components from my local machine. I ran following command:-
$ sudo ./unstack.sh
tgtadm: can't send the request to the tgt daemon, Transport endpoint is not connected
tgtd seems to be in a bad state, restarting...
stop: Unknown instance:
tgt start/running, process 14629
tgt stop/waiting
Volume group "stack-volumes" not found
Skipping volume group stack-volumes
And file are still present in /opt/stack and /usr/local/bin/. But manually removing these file will not be a good option.
The unstack.sh script only stops the services without removing them.
Devstack's folder contains a clean.sh script that removes openstack and dependencies so you can run something like this:
cd path/to/devstack
# There's no need to call unstack.sh explicitly
# clean.sh invokes that script itself.
./clean.sh
Follow the following 3 steps:
./clean.sh
rm -rf /opt/stack
rm -rf /usr/local/bin (careful, this will remove everything installed to your local bin folder, which might include previously installed applications).
For more info of all the impacted files and directories this link.
unstack doesn't clean out /opt/stack. or purge all dependency packages. or clean all eggs out of python.
I recommend running devstack in a VM. It's easy enough to simply remove the VM and rebuild from scratch.
Example shell script for creating a devstack VM for kvm:
#!/bin/sh
/usr/bin/vmbuilder kvm ubuntu -v --suite=oneiric --libvirt=qemu:///system --flavour=server --arch=amd64 --cpus=2 --mem=4096 --swapsize=2048 --rootsize=30480 --ip=192.168.122.236 --hostname=devstack --user=stack --name=stack --pass=stack --addpkg=git --addpkg=screen --addpkg=vim --addpkg=strace --addpkg=lsof --addpkg=nmap --addpkg=subversion --addpkg=acpid --addpkg=tcpdump --addpkg=python-pip --addpkg=wget --addpkg=htop --mirror=http://us.archive.ubuntu.com/ubuntu --components='main,universe' --addpkg=openssh-server --dns=8.8.8.8 --dest=/virts/devstack

Resources