Can anyone Please clarify me, i have a only 4gb Ram laptop with windows 7 installed ,currently i have working with Apache distributed hadoop1.x in a vmware,i want to practice Cloudera distributed hadoop with cloudera manager ,can you please tell how to install cloudera manager in 2gb ram allocated vmware.is it possible to install cloudera using cloudera manager in 2gb Ram allocated VMWARE with Redhat linux 6 installed?if yes can anyone tell me steps to install it in vmware with only 2gb Ram Size.
Thanks in advance.
In short: While running a CM cluster instance in pseudo-distributed mode on a 2GB VM is theoretically possible, due to the resource constraints it may lead to a sub-par user experience and is therefore not recommended. It's strongly advised to consider either installing CM or using the existing Cloudera Quickstart VM on a machine with no less than 4GB RAM available (after OS overhead).
Reasoning:
The 2GB of RAM will have to be divvied up between all of the selected components chosen at the time of installation (e.g. HDFS, YARN). Given a barebones configuration (HDFS + YARN), this will require the 2GB to be spread among the following services: NameNode, Secondary NameNode, DataNode, ResourceManager, NodeManager,JobHistoryServer, Cloudera Manager Web UI, PostgreSQL or whatever DB backend was chosen at the time of install, and Cloudera Manager Management services (if configured).
This would yield approx. 8-9 applications/services that would be constricted to using anywhere from 128MB - 256MB each, which depending on usage, could lead to utilization challenges such as GC thrashing, OOMs, or even CPU and RAM contention.
Related
If we install.packages("dplyr") on a GCP 'RStudio Server Pro Standard' VM, it takes around 3 minutes to install (on instance with 4 cores / 15 gb ram)
This seems unusual, as installation would typicaly take ~20 seconds on a laptop with equivalent specs.
Why so slow, and is there a quick and easy way of speeding this up?
Notes
I use the RStudio Server Pro Standard image from GCP marketplace to start the instance
Keen to know if there are any 'startup scripts' or similar I can set to run after the instance starts, e.g. to install a collection of commonly used packages
#user5783745 you can also adjust the Makevars to allow multithreaded compilation, which will help speed up compilations.
I followed this RStudio community post, and dropped MAKEFLAGS = -j4 into ~/.R/Makevars.
This basically halved the amount of time it took to install dplyr from scratch on the RStudio Server Pro Standard for GCP instance I spun up. (same as yours, 4 vCPU, 15GB ram)
I have downloaded the Cordapp example provided in the Corda website. I follow all the steps (to run it from the console) in
https://docs.corda.net/tutorial-cordapp.html
without any problem until "Running the example CorDapp". Here i get to errors one way or another.
First, when running
workflows-kotlin/build/nodes/runnodes
one or more of the nodes would not start. I was using a virtual machine with 2 cores and 4GB of RAM. Eventually, i noticed it seemed to be an issue with the RAM, so i changed the VM condig to 4 cpus and 10 GB of RAM.
Now, i can run
workflows-kotlin/build/nodes/runnodes
and get all 4 nodes working but, as soon as I run the following instruction
/gradlew runPartyXServer
Where X=[A,B,C] for each of the possible nodes, after 20-30 seconds as much, the machine repently slows down and aborts.
The VM has Fedora 30, 4 cores and 10GB of RAM. It is empty except for what i downloaded for the tutorial. I cannot believe those are not enough resources to run the tutorial, Am i wrong? Do i need more? may it be another thing?
Any help is welcome.
== Solved ==
The issue were the resources. I jumped to 8 cores and 32GB and it ran. I will try at some point with 16GB. In any case, the problem, from my point of view, is that having those large hardware requirements, the tutorial should include a section describing the minimum setup needed to run it.
From the given information, I believe you had ran into a Memory issue.
According to our documentation, Corda has a suggested minimal requirement of 1GB of Heap and 2-3GB of Host RAM per node.
https://docs.corda.net/docs/corda-enterprise/4.4/node/sizing-and-performance.html#sizing
I would suggest either reduce the number of nodes hosted on a single machine or expand your RAM size of the VM
I installed all prerequisites as mentioned for ubuntu machine
(JDK-8, intellij IDEA, git).
I clone the sample code from the git. It's running but system got struct too much.
My system configurations are operating system : ubuntu 16.04, RAM :4Gb , Hard Disk:500 Gb. Is there any alternative solution for this problem any one can help me out.
Thanks in advance.
This question is answered at: Cordapp tutorial crashing in a Fedora VirtualBox Machine
For similar reason: your RAM is too small to handle multiple nodes.
According to our documentation, Corda has a suggested minimal requirement of 1GB of Heap and 2-3GB of Host RAM per node. https://docs.corda.net/docs/corda-enterprise/4.4/node/sizing-and-performance.html#sizing
I have a server machine with the following configuration.
1) Dual Quad Core Xeon
2) 24 GB Memory
3) 500 GB Sata
4) 256 * 2 RAID 1
The machine just arrived and we want to install CDH5 in it. We want create a sand box / dev cluster.
I am looking for some expert advice on
A) How many nodes we can create? We are targeting 4-5 nodes. Is that advisable.
B) I read Cloudera manager should be residing in the node with solid configuration. Based on our configuration how much resource should be allocated for the same.
C) We will install Ubuntu 12.4.
We are fairly new to this process. Any help would be really helpful.
Thanks,
Amit
The answer for A and B would be up to the purpose of the cluster. Let me answer in case the cluster is just for test purpose.
A: 4-5 nodes should be reasonable and Cloudera Manager can manage more than hundreds nodes.
B: Cloudera Manager server can be located with the other services (HDFS, MapReduce, or so). Please see the resource requirements here
C: Ubuntu 12.04 is supported.
A) You can start with 2 Master and 4 datanodes
B) Cloudera Manager will reside on one of the master nodes and run the supervising services and will give you a UI for managing services.
C) 16.04 LTS (Xenial), 14.04 LTS (Trusty), 12.04 LTS (Precise) are supported for ubuntu
I have some questions:
Is it possible to install openstack on a Notebook with a 4GB DD3 Ram? Because the website says it needs atleast 8GB of RAM.
They say it requirs a double-QuadCore , I assue that means Octacore. Can we install that on a Quadcore?
They say that there is no possibility to install it on a NAS . Did you find any where if there is a possibility to do?. I dint find any even after asking our friend(google).
All in all, is it at-all possible to install on it a notebook/Desktop?
That advice is for production environments,
so 1)If you just want to play around your notebook will do fine. I had a succesful test-run on a 1.2 Ghz 1GB Netbook. It became incredibly slow when it launched it's first instance...
With a Double Quadcore they actually mean two seperate Quad-cores, as in two quad-core xeon processors on a single motherboard
So 2) yes you can install it on a quad-core.
3) a NAS device running openstack an openstack storage service seems to be unlikely indeed. You will most likely need more computing power.However If your NAS supports NFS or SSH or sth you can probably mount this drive and use it for storage.
4) You can perfectly build a all-in-one openstack test setup on your notebook. Performance will be low, but acceptable for testing.
It depends on what you mean by "install OpenStack". OpenStack itself is an extremely modular framework consisting on many services (Compute, Networking, Image service, Block Storage, Object Storage, Orchestration, Telemetry, ...). On top of that, a typical production deployment of OpenStack also requires several components, like load balancers, caching systems, firewalls, web servers and others. It is definitely possible to install a minimal openstack system, even on an average laptop.
The simplest way to run OpenStack on a laptop/desktop is to use Devstack, a shell script that installs all services from source and run them (by default) on a single machine. It is customizable enough to provide very good testing ground; it's used by OpenStack developers as well as the OpenStack QA team to test latest developments against "real" systems.
To avoid messing up your system, it's generally recommended to install OpenStack in a VM. From devstack doc:
DevStack should run in any virtual machine running a supported Linux release. It will perform best with 2Gb or more of RAM.
As of the time of this writing (Jan 2015), supported distros are:
Ubuntu (latest LTS)
Fedora
CentOS
Regarding NAS: you can of course use it, but "outside" Openstack apis, by providing mount points to your vms. It's even mandatory if you want to support live migration.