we use Spark2 Thrift in order to run Hive queries.
Thrift comes as part of the HDP 2.6 and our spark version is 2.1.0.2.6.0.3-8.
The more queries we run simultaneously, the faster we encounter OOM in the driver. These queries also contain JOINs and UNIONs.
from the jstat is seems there's no memory leak, however no matter how much memory is given to the driver, it seems it's never enough. The more queries that are run simultaneously, the faster Thrift driver starts to perform full GC until it crashes, since the full GC can't clean the old memory (since it's being used).
The OOM never occurs in the executors, only in the driver.
Does anyone work with Thrift over spark and encounters this problem? and if so - how can the Thrift driver be configured not to crash on OOM when running several queries simultaneously?
These are the configurations we use:
Thrift spark driver:
spark.driver.memory=15g
Thrift spark executors:
spark.executor.memory=10g
num cores = 7
config params from /usr/hdp/current/spark2-thriftserver/conf/spark-thrift-sparkconf.conf:
spark.broadcast.blockSize 32m
spark.driver.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.driver.maxResultSize 0
spark.dynamicAllocation.enabled true
spark.dynamicAllocation.executorIdleTimeout 45s
spark.dynamicAllocation.initialExecutors 2
spark.dynamicAllocation.maxExecutors 15
spark.dynamicAllocation.minExecutors 0
spark.dynamicAllocation.schedulerBacklogTimeout 1s
spark.eventLog.dir hdfs:///spark2-history/
spark.eventLog.enabled true
spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
spark.executor.memory 10g
spark.files.maxPartitionBytes 268435456
spark.files.openCostInBytes 33554432
spark.hadoop.cacheConf false
spark.history.fs.logDirectory hdfs:///spark2-history/
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
spark.kryoserializer.buffer.max 2000m
spark.master yarn-client
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 104857600
spark.scheduler.allocation.file /usr/hdp/current/spark2-thriftserver/conf/spark-thrift-fairscheduler.xml
spark.scheduler.mode FAIR
spark.shuffle.service.enabled true
spark.sql.autoBroadcastJoinThreshold 1073741824
spark.sql.shuffle.partitions 100
spark.storage.memoryMapThreshold 8m
Try changing the scheduler mode to FIFO.
Also, dont forget there are 2 different zones in memory :
- storage
- execution
Storage will use by défaut 60% of driver memory, so if you never cache data, decrease it to give more memory where its needed (they say its done automatically but ...).
Try decreasing spark shuffle partition to 100 then 10 if possible.
Try offheap (never tested but could help).
Related
What are the minimum hardware requirements for setting up an Apache Airflow cluster.
Eg. RAM, CPU, Disk etc for different types of nodes in the cluster.
I have had no issues using very small instances in pseudo-distributed mode (32 parallel workers; Postgres backend):
RAM 4096 MB
CPU 1000 MHz
VCPUs 2 VCPUs
Disk 40 GB
If you want distributed mode, you should be more than fine with that if you keep it homogenous. Airflow shouldn't really do heavy lifting anyways; push the workload out to other things (Spark, EMR, BigQuery, etc).
You will also have to run some kind of messaging queue, like RabbitMQ. I think they take Redis too. However, this doesn't really dramatically impact how you size.
We are running the airflow in AWS with below config
t2.small --> airflow scheduler and webserver
db.t2.small --> postgres for metastore
The parallelism parameter in airflow.cfg is set to 10 and there are around 10 users who access airflow UI
All we do from airflow is ssh to other instances and run the code from there
I'm trying to set up a H2O cloud on a 4 data nodes hadoop spark cluster using R in a Zeppelin notebook. I found that I have to give each executor at least 20Gb of memory before my R paragraph stops complaining about running out of memory (java error message of GC out of memory).
Is it expected that I need 20Gb of memory per executor for running an H2O cloud? Or are there any configuration entries that I can change to reduce the memory requirement?
There isn't enough information in this post to give specifics. But I will say that the presence of Java GC messages is not necessarily a problem, especially at startup. It's normal to see a flurry of GC messages at the beginning of a Java program's life as the heap expands from nothing to it's steady-state working size.
A sign that Java GC really is becoming a major problem is when you see back-to-back full GC cycles that have a real wall-clock time of seconds or more.
I'm checking a server that has 32gb of ram and I see 99% memory usage.
The machine is used with IIS, MongoDB and ElasticSearch.
None of the processes seemed to be that big. The largest was MongoDB at about 1gb.
So, I shut down everything.. and now that memory usage is 88%
After a reboot, with all services running, the memory usage is 23%
Those are the largest processes on the system, with everything being shutdown. As you can see, everything is very small, but most of the ram remains gone.
How can I track what is eating up all the ram? I tried process explorer, but it doesn't give me any more useful info.
Try to use RAMMAP from sysinternals it will give you more details about memory usage. Like metaFile for example.
Elasticsearch generally a lot of the available RAM to cache search results aggregation. This is to avoid memory swapping. It's very evident and observable in LINUX servers. Thus it's recommended using ES in separate server in production with heavy usage.
So please try and check cache memory once.
Have a look at the Heap size allotted to Elasticsearch. You could check the values of -Xms and -Xmx in jvm.options file. Usually, 50% of physical RAM is allotted to ES and with bootstrap.memory_lock set to true, it locks the RAM. Ideally, as another answer mentions, Elasticsearch should be run in its own machine.
Environment :
machines : 2.1 xeon, 128 GB ram, 32 cpu
os : centos 7.2 15.11
cassandra version : 2.1.15
opscenter version : 5.2.5
3 keyspaces : Opscenter (3 tables), OpsCenter (10 tables), application`s keyspace with (485 tables)
2 Datacenters, 1 for cassandra (5 machines )and another one DCOPS to store up opscenter data (1 machine).
Right now the agents on the nodes consume on average ~ 1300 cpu (out of 3200 available). The only transactioned data being ~ 1500 w/s on the application keyspace.
Any relation between number tables and opscenter? Is it behaving alike, eating a lot of CPU because agents are trying to write the data from too many metrics or is it some kind of a bug!?
Note, same behaviour on previous version of opscenter 5.2.4. For this reason i first tried to upg opscenter to newest version available.
From opscenter 5.2.5 release notes :
"Fixed an issue with high CPU usage by agents on some cluster topologies. (OPSC-6045)"
Any help/opinion much appreciated.
Thank you.
Observing with the awesome tool you provided Chris, on specific agent`s PID noticed that the heap utilisation was constant above 90% and that triggered a lot of GC activity with huge GC pauses of almost 1 sec. In this period of time i suspect the pooling threads had to wait and block my cpu alot. Anyway i am not a specialist in this area.
I took the decision to enlarge the heap for the agent from default 128 to a nicer value of 512 and i saw that all the GC pressure went off and now any thread allocation is doing nicely.
Overall the cpu utilization dropped from values of 40-50% down to 1-2% for the opscenter agent. And i can live with 1-2% since i know for sure that the CPU is consumed by the jmx-metrics.
So my advice is to edit the file:
datastax-agent-env.sh
and alter the default 128 value of Xmx
-Xmx512M
save the file, restart the agent, and monitor for a while.
http://s000.tinyupload.com/?file_id=71546243887469218237
Thank you again Chris.
Hope this will help other folks.
I am using Cloudify 2.7 with OpenStack Icehouse.
I developed a tomcat recipe and deployed it. In the orchestrator log of the cloudify console, I read the following WARNING:
2015-06-04 11:05:01,706 ESM INFO [org.openspaces.grid.gsm.strategy.ScaleStrategyProgressEventState] - [tommy.tomcat] machines SLA enforcement is in progress.; Caused by: org.openspaces.grid.gsm.machines.exceptions.ExpectedMachineWithMoreMemoryException: Machines SLA Enforcement is in progress: Expected machine with more memory. Machine <Public_IP>/<Public_IP> has been started with not enough memory. Actual total memory is 995MB. Which is less than (reserved + container) = (0MB+3800MB) = 3800MB
The Flavor of the VM is: 4GB RAM, 2vCPU, 20GB Disk
Into the cloud driver I commented the following line:
//reservedMemoryCapacityPerMachineInMB 1024
and configured the compute section related to the flavor as following:
computeTemplate
{
imageId <imageID>
machineMemoryMB 3900
hardwareId <hardwareId>
...
}
Can someone help me to pointing out the error?
Thanks.
The error message states that the actual available memory is only 995MB, which is considerably less than the expected 4GB. To clarify that:
do you run multiple services on the same machine?
maybe the VM really has less memory than expected. please run 'cat /proc/meminfo' on the started VM to verify the exact memory it has
In principle, you should not comment out any setting of reserved memory because Cloudify must take that into account - this setting is supposed to represent the memory used by the OS and other processes. additionally, the orchestrator (ESM) takes into account ~100 MB for cloudify to run freely.
So, please update machineMemoryMB to the value calculated this way:
(the number returned by 'cat /proc/meminfo') - 1024 - 100