I am trying to connect R to a Hadoop cluster using R. The cluster has HDFS, Map Reduce, Hive, Pig and Sqoop installed on it.
R will be running on in the Windows environment. I know that rhdfs, rhadoop and rmr exist for Linuix, but I can't find anything on Windows.
Does anyone know of a library to use?
Thank you
Revolution Analytrics is trying to make a name for themselves in this space. They have a couple of nice packages (some of which are open-source and/or free for non-commercial use) which allow you to interact with Hadoop from R in a Windows environment fluidly.
Related
I am new to cloud computing and GCP. I am trying to create a notebook instance running R with a GPU. I got a basic instance with 1 core and 0 GPUs to start and I was able to execute some code which was cool. When I try to create an instance with a GPU I keep getting all sorts of errors about something called live migration, or that there are no resources available, etc. Can someone tell me how to start an R notebook instance with a GPU? It can't be this difficult.
The CRAN (The Comprehensive R Archive Network) doesn't support GPU. However, you can follow this link might help you to install a Notebook instance running R with a GPU. You need a machine with Nvidia GPU drivers installed then install R and Jupyter Lab. After that compile those R packages which require it for use with GPU's.
I have a cluster having Cloudera CDH running. I need to connect my R programs (running on my Laptop) to the Spark running in cluster.
However If I try to connect the local R through Sparklyr, it is giving Error. As it is searching the Spark home on the laptop itself means i have to install spark on my laptop but i can't do it.
I googled and found we can install SparkR and use R with Spark. However, for using sparkR what i have to do:
install SparkR on all nodes of cluster?
How to configure to use SparkR?
I am new in R and I am working with a datasets that has more than 5 millions of observations. So I thought that it would be a good idea to use RStudio on a virtual machine instead of using it on my local machine.
I am reading the documentation about virtual machines and RServer but it is still not clear to me if I have to use Microsoft R Server to create a VIM and then just install Rstudio as I would do in my local machine or if I can create a generic VIM and then install RStudio. Which is the correct way? Why?
If both of these options are possible, which one is the best?
Please help me. Sorry for my confusion.
You can do either. If you are using Azure (which I think you are given that you mention Microsoft R Server), there is also the Data Science VM, which will come preinstalled with RStudio and many other useful programs.
R Server is more for production workloads with R, so unless you are planning that you could probably stick with the Data Science VM. If you end up choosing this option, you can connect directly to an RStudio instance on the R Server from the Azure portal.
I know how to access packages in R scripts in Azure machine learning by either using the Azure supported ones or by zipping up the packages.
My problem now is that Azure machine learning does not support the h2o package and when I tried using the zipped file - it gave an error.
Has anyone figured out how to use h2o in R in Azure machine learning?
So since there was no reply to my question, I made some research and came up with the following:
H2O cannot be ran in a straightforward manner in Azure machine learning embedded R scripts. A workaround the problem is to consider using an Azure created environment - specially for H2O. The options available are:
Spinning up an H2O Artificial Intelligence Virtual Machine solution
Using an H2O application for HDInsight
For more reading, you can go to: http://docs.h2o.ai/h2o/latest-stable/h2o-docs/azure.html
I am going to setup a local cluster where i am planning to use CDH5. With this i will have all the inbuilt eco-system of hadoop on the fly however i need Rscript also in my cluster for some hadoop streaming and data analytics work. So just wondering is it possible to use CDH5 and install R script. Thanks
Yes but you will have to use RHadoop (https://github.com/RevolutionAnalytics/RHadoop/wiki)
It runs on CDH. You can find more information about Cloudera and RHadoop here :
http://www.cloudera.com/content/cloudera/en/solutions/partner/Revolution-analytics.html