I have a Golang process that runs SQL queries on a 400MB SQLite file.
I am using https://github.com/mattn/go-sqlite3 with the connection string:
file:mydb.sqlite?mode=ro&_journal=DELETE
When run on my dev machine on Docker it only needs 20MB of RAM, but on Google Run any instance smaller than 512MB will return HTTP code 500 with a memory exceeded limit in the logs.
docker diff x shows that the DB file is not modified (which I assume would cause gVisor to copy the whole binary SQLite db file to RAM to modify it).
How the docker image is built
I am copying the SQLite DB file into the image with the source code:
FROM golang:latest
...
COPY . /go/src/api
I have a global var in my Golang file: var db *sqlx.DB
This gets set in the main fn, before ListenAndServe:
conStr := fmt.Sprintf("file:%s?mode=ro&_journal=DELETE", *fileName)
dbConn, err := sqlx.Open("sqlite3", conStr)
db = dbConn
I query the db within a HTTP request:
err := db.Selectv(&outDataSet, "SELECT...", arg1, arg2)
Why this must be an issue with the Cloud Run environment
docker stats never goes above 20MB when run locally.
Limiting docker run to 20MB RAM also runs fine on my dev machine:
docker run \
--memory=20m \
--memory-swap=20m \
The Cloud Run "Container Memory Allocation" metric also stays well below 128M:
https://console.cloud.google.com/monitoring/metrics-explorer
Thanks.
According to the official documentation:
Configuring Memory Limits
Cloud Run container instances that exceed their allowed memory limit
are terminated.
The following count towards the available memory of your container
instance:
a.running the application executable (as the executable must be loaded
to memory)
b.allocating memory in your application process
c.writing files
to the filesystem
The size of the deployed container image does not
count towards the available memory.
Also I would suggest to consider:
Are your container instances exceeding memory?
Your container instances might be exceeding the available memory. To
determine if this is the case, look for such errors in the
varlog/system logs. If the instances are exceeding the available
memory, consider increasing the memory limit.
Note that the Cloud Run container instances run in an environment
where the files written to the local filesystem count towards the
available memory. This also includes any log files that are not
written to /var/log/* or /dev/log.
It seems that your container file systems is using the memory.
In the Cloud Run (fully managed) environment disk storage is an in-memory filesystem. link
Related
R requires CPU more than anything else so it is recommended to pick one of the newer generation compute optimized instance types, preferably with a SSD disk.
I've recently run into a problem with high memory usage (quickly raising to 100%) during load testing. To reproduce: there is an R package for which processing time is UP TO 0.2 in no-stress conditions. If I'm trying to query one of the endpoints using curl for 1000 jsons on 3 machines in parallel all of the memory is suddenly used which results in 'cannot fork' or:
cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory' In call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
The setup is 2x AWS 8GB CPU-optimized servers + load balancer all in private network. HTTPS is enabled and my main usage is online processing of requests so I'm mostly querying /json endpoints.
Do you happen to have any suggestions on how to approach this issue? The plan is to have more packages installed (more online processes requesting result from various functions) and don't want to end up having 32GB RAM per box.
All of the packages are deployed with such options:
LazyData: false
LazyLoad: false
They are also added into serverconf.yml.j2 - preload section.
RData files are loaded within an onLoad function by calling utils::data.
Also, keeping in mind that I'm using OpenCPU without github and only one-way communication (from backend to ocpu box) which options do you suggest to turn on/optimize? It's not clearly stated in the docs yet.
It mostly depends on which packages you are using and what you are doing. Can you run the same functionality that you are invoking through opencpu locally (in the command line) without running out of memory?
Apache2 prefork creates worker processes to handle concurrent requests. Each of these workers contains an R process with all preloaded
packages. So if one request would take 500mb, the total memory
consumption on the server is n * 500 where n is the number of workers
that are loaded.
Depending on how many concurrent requests you expect, you could try
lowering StartServers or MaxRequestWorkers in your apache2 config.
Also try raising (or lowering) the option rlimit.as in the file /etc/opencpu/server.conf which limits the amount of memory (address space) a single process is allowed to consume.
I am trying to run a kaa server on a raspberry pi, and have successfully compiled it from source on the ARM processor, and installed the resulting .deb package.
However when i try to start the kaa-node i get the following error.
Starting Kaa Node...
Invalid maximum heap size: -Xmx4G
The specified size exceeds the maximum representable size.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
I have tried to search through the /etc/kaa-node/conf directory, and the bin files, but I can't see where the "4G" setting is actually set, so that I might change it to something smaller and launch this on the Pi which has 1G of RAM.
Can someone point me to the correct place to make this modification, while still making use of launching the server as a service using the built in utilities? I know i could just run it with java, and passit my own JAVA_OPTIONS.
I think you can try to find the "kaa-node" file in /etc/default/ and modify the JAVA_OPTIONS in it.
We try to modify it to config heap size and GC for our Kaa server.
You can try starting kaa-node service with
service kaa-node start -Xmx500M
To limit heap size by 500mb.
If it won't work, try
export _JAVA_OPTIONS=-Xmx500m
To set global JVM heap size limit.
I am using Cloudify 2.7 with OpenStack Icehouse.
I developed a tomcat recipe and deployed it. In the orchestrator log of the cloudify console, I read the following WARNING:
2015-06-04 11:05:01,706 ESM INFO [org.openspaces.grid.gsm.strategy.ScaleStrategyProgressEventState] - [tommy.tomcat] machines SLA enforcement is in progress.; Caused by: org.openspaces.grid.gsm.machines.exceptions.ExpectedMachineWithMoreMemoryException: Machines SLA Enforcement is in progress: Expected machine with more memory. Machine <Public_IP>/<Public_IP> has been started with not enough memory. Actual total memory is 995MB. Which is less than (reserved + container) = (0MB+3800MB) = 3800MB
The Flavor of the VM is: 4GB RAM, 2vCPU, 20GB Disk
Into the cloud driver I commented the following line:
//reservedMemoryCapacityPerMachineInMB 1024
and configured the compute section related to the flavor as following:
computeTemplate
{
imageId <imageID>
machineMemoryMB 3900
hardwareId <hardwareId>
...
}
Can someone help me to pointing out the error?
Thanks.
The error message states that the actual available memory is only 995MB, which is considerably less than the expected 4GB. To clarify that:
do you run multiple services on the same machine?
maybe the VM really has less memory than expected. please run 'cat /proc/meminfo' on the started VM to verify the exact memory it has
In principle, you should not comment out any setting of reserved memory because Cloudify must take that into account - this setting is supposed to represent the memory used by the OS and other processes. additionally, the orchestrator (ESM) takes into account ~100 MB for cloudify to run freely.
So, please update machineMemoryMB to the value calculated this way:
(the number returned by 'cat /proc/meminfo') - 1024 - 100
I've got a strange problem with opencpu. I'am setting up a webinterface to use xcms (R-package) and when reading an mzxml file with the webinterface I get the error : cannot allocate vector of size 207.2 Mb. This looks like an error from R, but if I open R on the server and try to open the file myself it works and R creates and object of 435Mb. Is there somewhere a memory limit set (apache, opencpu, R)? If I use ulimit -a the max memory size is set to unlimited.
Cheers, Rico
Yes, the OpenCPU cloud server sets a memory limit RLIMIT_AS on every request using the RAppArmor package. This is to prevent a single user from consuming all resources on a server. The default limit is 1GB per process. You can configure these limits by editing:
/etc/opencpu/server.conf
After editing the file, restart the server:
sudo service opencpu restart
For more information on rlimits, read the JSS paper on RAppArmor.
We have been using micro instance till our development phase. But now, as we are about to go live, we want to upgrade our instance to type medium.
I followed these simple steps: stop the running instance, change instance type to medium and then start the instance again. I can see the instance is upgraded in terms of the memory. But the storage still shows to be 8GB. But according to the configuration mentioned, a m1.medium type instance should have 1x410GB storage.
Am I doing anything wrong or missing out something? Please help!
Keep in mind, EBS storage (which you are currently using) and Instance storage (which is what you are looking for) are two different things in EC2.
EBS storage is similar to a SAN volume. It exists outside of the host. You can create multiple EBS volumes of up to 1TB and attach them to any instance size. Smaller instances have lower available bandwidth to EBS volumes so they will not be able to effectively take advantage of all that many volumes.
Instance storage is essentially hard drives attached to the host. While its included in the instance cost, it comes with some caveats. It is not persistent. If you stop your instance, or the host fails for any reason, the data stored on the instance store will be lost. For this reason, it has to be explicitly enabled when the instance is first launched.
Generally, its not recommended that to use instance storage unless you are conformable with and have designed your infrastructure around the non-persistance of instance storage.
The sizes mentioned for the instance types are just these defaults. If you create an image from a running micro instance, it will get that storage size as default, even if this image later is started as medium.
But you can change the storage size when launching the instance:
You also can change the default storage size when creating an image:
WARNING: This will resize the storage size. It will not necessarily resize the partition existing on it nor will it necessarily resize the file system on that partition. On Linux it resized everything automagically (IIRC), on a Windows instance you will have to resize your stuff yourself. For other OSes I have no idea.
I had a similar situation. I created a m2.medium instance of 400 GB, but when I log into the shell and issue the command
df -h
... it shows an 8 GB partition.
However, the command
sudo fdisk -l
showed that the device was indeed 400 GB. The problem is that Amazon created a default 8 GB partition on it, and that partition needs to be expanded to the full size of the device. The command to do this is:
sudo resize2fs -f /dev/xvda1
where /dev/xvda1 is the mounted root volume. Use the 'df -h' command to be sure you have the right volume name.
Then simply reboot the instance, log in again, and you'll see the fdisk command now says there's nearly 400 GB available space. Problem solved.