I've got a strange problem with opencpu. I'am setting up a webinterface to use xcms (R-package) and when reading an mzxml file with the webinterface I get the error : cannot allocate vector of size 207.2 Mb. This looks like an error from R, but if I open R on the server and try to open the file myself it works and R creates and object of 435Mb. Is there somewhere a memory limit set (apache, opencpu, R)? If I use ulimit -a the max memory size is set to unlimited.
Cheers, Rico
Yes, the OpenCPU cloud server sets a memory limit RLIMIT_AS on every request using the RAppArmor package. This is to prevent a single user from consuming all resources on a server. The default limit is 1GB per process. You can configure these limits by editing:
/etc/opencpu/server.conf
After editing the file, restart the server:
sudo service opencpu restart
For more information on rlimits, read the JSS paper on RAppArmor.
Related
I have a Golang process that runs SQL queries on a 400MB SQLite file.
I am using https://github.com/mattn/go-sqlite3 with the connection string:
file:mydb.sqlite?mode=ro&_journal=DELETE
When run on my dev machine on Docker it only needs 20MB of RAM, but on Google Run any instance smaller than 512MB will return HTTP code 500 with a memory exceeded limit in the logs.
docker diff x shows that the DB file is not modified (which I assume would cause gVisor to copy the whole binary SQLite db file to RAM to modify it).
How the docker image is built
I am copying the SQLite DB file into the image with the source code:
FROM golang:latest
...
COPY . /go/src/api
I have a global var in my Golang file: var db *sqlx.DB
This gets set in the main fn, before ListenAndServe:
conStr := fmt.Sprintf("file:%s?mode=ro&_journal=DELETE", *fileName)
dbConn, err := sqlx.Open("sqlite3", conStr)
db = dbConn
I query the db within a HTTP request:
err := db.Selectv(&outDataSet, "SELECT...", arg1, arg2)
Why this must be an issue with the Cloud Run environment
docker stats never goes above 20MB when run locally.
Limiting docker run to 20MB RAM also runs fine on my dev machine:
docker run \
--memory=20m \
--memory-swap=20m \
The Cloud Run "Container Memory Allocation" metric also stays well below 128M:
https://console.cloud.google.com/monitoring/metrics-explorer
Thanks.
According to the official documentation:
Configuring Memory Limits
Cloud Run container instances that exceed their allowed memory limit
are terminated.
The following count towards the available memory of your container
instance:
a.running the application executable (as the executable must be loaded
to memory)
b.allocating memory in your application process
c.writing files
to the filesystem
The size of the deployed container image does not
count towards the available memory.
Also I would suggest to consider:
Are your container instances exceeding memory?
Your container instances might be exceeding the available memory. To
determine if this is the case, look for such errors in the
varlog/system logs. If the instances are exceeding the available
memory, consider increasing the memory limit.
Note that the Cloud Run container instances run in an environment
where the files written to the local filesystem count towards the
available memory. This also includes any log files that are not
written to /var/log/* or /dev/log.
It seems that your container file systems is using the memory.
In the Cloud Run (fully managed) environment disk storage is an in-memory filesystem. link
I have a Java server from which I am calling R functions using c.eval.
I use Rserve to do this and preload all libraries using the following call
R CMD Rserver --RS-conf Rserve.conf
The file Rserve.conf has all my loaded libraries. The total memory requirement for this is around 127MB.
The challenge I have is that every time I call a function from within my Java server, a new process is spawned but it looks like the process requires the full 127MB of memory. So with 32GB RAM, roughly, 240 concurrent calls are sufficient to max out the memory usage and brings the server crashing down.
I found this link: Rserve share library code but it talks about exactly what I have been doing. Any help in understanding how to make Rserve work without it loading all libraries for every call would be much appreciated.
R requires CPU more than anything else so it is recommended to pick one of the newer generation compute optimized instance types, preferably with a SSD disk.
I've recently run into a problem with high memory usage (quickly raising to 100%) during load testing. To reproduce: there is an R package for which processing time is UP TO 0.2 in no-stress conditions. If I'm trying to query one of the endpoints using curl for 1000 jsons on 3 machines in parallel all of the memory is suddenly used which results in 'cannot fork' or:
cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory' In call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
The setup is 2x AWS 8GB CPU-optimized servers + load balancer all in private network. HTTPS is enabled and my main usage is online processing of requests so I'm mostly querying /json endpoints.
Do you happen to have any suggestions on how to approach this issue? The plan is to have more packages installed (more online processes requesting result from various functions) and don't want to end up having 32GB RAM per box.
All of the packages are deployed with such options:
LazyData: false
LazyLoad: false
They are also added into serverconf.yml.j2 - preload section.
RData files are loaded within an onLoad function by calling utils::data.
Also, keeping in mind that I'm using OpenCPU without github and only one-way communication (from backend to ocpu box) which options do you suggest to turn on/optimize? It's not clearly stated in the docs yet.
It mostly depends on which packages you are using and what you are doing. Can you run the same functionality that you are invoking through opencpu locally (in the command line) without running out of memory?
Apache2 prefork creates worker processes to handle concurrent requests. Each of these workers contains an R process with all preloaded
packages. So if one request would take 500mb, the total memory
consumption on the server is n * 500 where n is the number of workers
that are loaded.
Depending on how many concurrent requests you expect, you could try
lowering StartServers or MaxRequestWorkers in your apache2 config.
Also try raising (or lowering) the option rlimit.as in the file /etc/opencpu/server.conf which limits the amount of memory (address space) a single process is allowed to consume.
I am trying to run a kaa server on a raspberry pi, and have successfully compiled it from source on the ARM processor, and installed the resulting .deb package.
However when i try to start the kaa-node i get the following error.
Starting Kaa Node...
Invalid maximum heap size: -Xmx4G
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I have tried to search through the /etc/kaa-node/conf directory, and the bin files, but I can't see where the "4G" setting is actually set, so that I might change it to something smaller and launch this on the Pi which has 1G of RAM.
Can someone point me to the correct place to make this modification, while still making use of launching the server as a service using the built in utilities? I know i could just run it with java, and passit my own JAVA_OPTIONS.
I think you can try to find the "kaa-node" file in /etc/default/ and modify the JAVA_OPTIONS in it.
We try to modify it to config heap size and GC for our Kaa server.
You can try starting kaa-node service with
service kaa-node start -Xmx500M
To limit heap size by 500mb.
If it won't work, try
export _JAVA_OPTIONS=-Xmx500m
To set global JVM heap size limit.
regarding the variable TMOUT. I have used
export TMOUT=3600
but my SSH terminal still gets time out even less than 10minutes without keyboard activity. How can I change this TMOUT value to 3600?
Also, if I use sftp to transfer a very large local file (in GB) to the server, will I get timed out as well?
thanks
This might be the solution to your problem :
http://www.daveperrett.com/articles/2006/08/17/stop-ssh-and-sftp-timeout/
A transfer won't timeout, it's activity.