I'm running R to handle a file which is about 1Gb in size, filtering it into several smaller files, and then trying to print them out. I'm getting errors like this at different points throughout the process:
**Error: cannot allocate vector of size 79.4 Mb**
A vector of this size should be a non-issue, with how much memory I /should/ be working with. My machine has 24Gb of memory, and the overwhelming majority of that is still free, even when the R environment with these large objects in it is up and running, and I'm seeing the error above.
free -m
total used free shared buffers cached
Mem: 24213 2134 22079 0 55 965
-/+ buffers/cache: 1113 23100
Swap: 32705 0 32705
here is R's response to gc():
corner used (Mb) gc trigger (Mb) max used (Mb)
Ncells 673097 18.0 1073225 28.7 956062 25.6
Vcells 182223974 1390.3 195242849 1489.6 182848399 1395.1
I'm working in Ubuntu 12.04.1 LTS
here are some specs from the machine I'm using:
i7-3930K 3.20 GHz Hexa-core (6 Core)12MB Cache
ASUS P9X79 s2011 DDR3-1333MHZ,1600 upto 64GB
32GB DDR3 ( 8x4GB Module )
128GB SSD drive
Asus nVidia 2GB GTX660 TI-DC2O-2GD5 GeForce GTX 660 Ti i
this is the object I'm attempting to write to file:
dim(plant)
[1] 10409404 13
the 'plant' object is of the class "data.frame". here is one of the lines of code that prompts the error:
write.table(plant, "file.txt", quote=F, sep="\t", row.names=T, col.names=F)
Any help on solving this issue would be much appreciated.
Tries with memory.limit() function!
-
memory.limit(2000)
-
Related
> adj = as.dist(adj)
Error: cannot allocate vector of size 3.0 Gb
> system("free -h")
total used free shared buff/cache available
Mem: 14G 2.2G 12G 88M 420M 12G
Swap: 0B 0B 0B
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 7799796 416.6 16746142 894.4 16746142 894.4
Vcells 41146681 314.0 520945812 3974.6 1665927139 12710.1
Why can't I allocate a vector of 3.0GB if I have 12GB free? I'm working on Linux CentOS, so I don't have access to memory.limit() and memorize.size().
This general question has been asked a lot. I don't see an answer that addresses my specific question, though. These questions are kind of the same, but the only answers I see are related to reducing the size of the object, using memory.limit() and memory.size(), or using bsub -q server_name -R. I don't know how to use the latter inside a script, and it doesn't address my question of why I can't allocate a vector when (it appears) I have the memory to do so. I've also tried including Sys.setenv('R_MAX_VSIZE'=32000000000) but that did not fix it.
Edit: I added gc() output. Does it matter that the available 12GB are virtual memory??
I am trying to stress a ubuntu container's memory. Typing free in my command terminal provides the following result:
free -m
total used free shared buff/cache available
Mem: 7958 585 6246 401 1126 6743
Swap: 2048 0 2048
I want to stress exactly 10% of the total available memory. Per stress-ng manual:
-m N, --vm N
start N workers continuously calling mmap(2)/munmap(2) and writing to the allocated
memory. Note that this can cause systems to trip the kernel OOM killer on Linux
systems if not enough physical memory and swap is not available.
--vm-bytes N
mmap N bytes per vm worker, the default is 256MB. One can specify the size as % of
total available memory or in units of Bytes, KBytes, MBytes and GBytes using the
suffix b, k, m or g.
Now, on my target container I run two memory stressors to occupy 10% of my memory:
stress-ng -vm 2 --vm-bytes 10% -t 10
However, the memory usage on the container never reaches 10% no matter how many times I run it. I tried different timeout values, no result. The closet it gets is 8.9% never approaches 10%. I inspect memory usage on my container this way:
docker stats --no-stream kind_sinoussi
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
c3fc7a103929 kind_sinoussi 199.01% 638.4MiB / 7.772GiB 8.02% 1.45kB / 0B 0B / 0B 7
In an attempt to understand this behaviour, I tried running the same command with an exact unit of bytes. In my case, I'll opt for 800 mega since 7958m * 0.1 = 795,8 ~ 800m.
stress-ng -vm 2 --vm-bytes 800m -t 15
And, I get 10%!
docker stats --no-stream kind_sinoussi
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
c3fc7a103929 kind_sinoussi 198.51% 815.2MiB / 7.772GiB 10.24% 1.45kB / 0B 0B / 0B 7
Can someone explain why this is happening?
Another question, is it possible for stress-ng to stress memory usage to 100%?
stress-ng --vm-bytes 10% will use sysconf(_SC_AVPHYS_PAGES) to determine the available memory. This sysconf() system call will return the number of pages that the application can use without hindering any other process. So this is approximately what the free command is returning for the free memory statistic.
Note that stress-ng will allocate the memory with mmap, so it may be that during run time mmap'd pages may not necessarily be physically backed at the time you check how much real memory is being used.
It may be worth trying to also use the --vm-populate option; this will try and ensure the pages are physically populated on the mmap'd memory that stress-ng is exercising. Also try --vm-madvise willneed to use the madvise() system call to hint that the pages will be required fairly soon.
(New to GNU Parallel)
My aim is to run the same Rscript, with different arguments, over multiple cores. My first problem is to get this working on my laptop (2 real cores, 4 virtual), then I will port this over to one with 64 cores.
Currently:
I have a Rscript, "Test.R", which takes in arguments, does a thing (say adds some numbers then writes it to a file), then stops.
I have a "commands.txt" file containing the following:
/Users/name/anaconda3/lib/R/bin/Rscript Test.R 5 100 100
/Users/name/anaconda3/lib/R/bin/Rscript Test.R 5 100 1000
/Users/name/anaconda3/lib/R/bin/Rscript Test.R 5 100 1000
/Users/name/anaconda3/lib/R/bin/Rscript Test.R 5 100 1000
/Users/name/anaconda3/lib/R/bin/Rscript Test.R 50 100 1000
/Users/name/anaconda3/lib/R/bin/Rscript Test.R 50 200 1000
So this tells GNU parallel to run Test.R using R (I have installed this using anaconda)
In the terminal (after navigating to the desktop which is where Test.R and commands.txt are) I use the command:
parallel --jobs 2 < commands.txt
What I want this to do, is to use 2 cores, and run the commands, from commands.txt, until all tasks are complete. (I have tried variations on this command, such as changing the 2 to a 1, in this case, 2 of the cores run at 100%, and the other 2 run around 20-30%).
When I run this, all of the 4 cores go to 100% (as seen from htop), and the first 2 jobs complete, and no more jobs get complete, despite all 4 cores still being at 100%.
When I run the same command on the 64 core compute, all 64 cores go to 100%, and I have to cancel the jobs.
Any advice on resources to look at, or what I am doing wrong would be greatly appreciated.
Bit of a long question, let me know if I can clarify anything.
The output from htop as requested, during running the above command (sorted by CPU%:
1 [||||||||||||||||||||||||100.0%] Tasks: 490, 490 thr; 4 running
2 [|||||||||||||||||||||||||99.3%] Load average: 4.24 3.46 4.12
3 [||||||||||||||||||||||||100.0%] Uptime: 1 day, 18:56:02
4 [||||||||||||||||||||||||100.0%]
Mem[|||||||||||||||||||5.83G/8.00G]
Swp[|||||||||| 678M/2.00G]
PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command
9719 user 16 0 4763M 291M ? 182. 3.6 0:19.74 /Users/user/anaconda3
9711 user 16 0 4763M 294M ? 182. 3.6 0:20.69 /Users/user/anaconda3
7575 user 24 0 4446M 94240 ? 11.7 1.1 1:52.76 /Applications/Utilities
8833 user 17 0 86.0G 259M ? 0.8 3.2 1:33.25 /System/Library/StagedF
9709 user 24 0 4195M 2664 R 0.2 0.0 0:00.12 htop
9676 user 24 0 4197M 14496 ? 0.0 0.2 0:00.13 perl /usr/local/bin/par
Based on the output from htop the script /Users/name/anaconda3/lib/R/bin/Rscript uses more than one CPU thread (182%). You have 4 CPU threads and since you run 2 Rscripts we cannot tell if Rscript would eat all 4 CPU threads if it ran by itself. Maybe it will eat all CPU threads that are available (your test on the 64 core machine suggests this).
If you are using GNU/Linux you can limit which CPU threads a program can use with taskset:
taskset 9 parallel --jobs 2 < commands.txt
This should force GNU Parallel (and all its children) to only use CPU thread 1 and 4 (9 in binary: 1001). Thus running that should limit the two jobs to run in two threads only.
By using 9 (1001 binary) or 6 (0110 binary) we are reasonably sure that the two CPU threads are on two different cores. 3 (11 binary) might refer to the two threads on the came CPU core and would therefore probably be slower. The same goes for 5 (101 binary).
In general you want to use as many CPU threads as possible as that will typically make the computation faster. It is unclear from your question why you want to avoid this.
If you are sharing the server with others a better solution is to use nice. This way you can use all the CPU power that others are not using.
My GPU seems to allow 562% use of global memory and 133% use of local memory for a simple PyOpenCL matrix addition kernel. Here is what my script prints:
GPU: GeForce GTX 670
Global Memory - Total: 2 GB
Global Memory - One Buffer: 3.750000 GB
Number of Global Buffers: 3
Global Memory - All Buffers: 11.250000 GB
Global Memory - Usage: 562.585844 %
Local Memory - Total: 48 KB
Local Memory - One Array: 32.000000 KB
Number of Local Arrays: 2
Local Memory - All Arrays: 64.000000 KB
Local Memory - Usage: 133.333333 %
If I increase global memory use much above this point, I get the error: mem object allocation failure
If I increase local memory use above this point, I get the error: invalid work group size
Why doesn't my script fail immediately when memory use of local or global exceeds 100%?
Global size is multiplied by 32, thats the error.
When clearly a float32 has 4bytes, this makes a and b arrays 4 bytes each. Not 32.
So the proper results for you would be:
Global Memory - Total: 2 GB
Global Memory - One Buffer: 0.4687500 GB
Number of Global Buffers: 3
Global Memory - All Buffers: 1.40625 GB
Global Memory - Usage: 70.3125 %
Local Memory - Total: 48 KB
Local Memory - One Array: 4.000000 KB
Number of Local Arrays: 2
Local Memory - All Arrays: 8.000000 KB
Local Memory - Usage: 16.6666666 %
I have a memory leak and I am having a really hard time trying to figure out where the problem is. The ASP.NET process is raising to 1GB every now and then. I have followed the instructions on this page (http://humblecoder.co.uk/tag/windbg) and the !gcroot command returns the following (last x lines). I have looked at all my OracleConnections and OracleCommands and they appear to be closed and disposed correctly:
6523dfd4 282 28200 System.Data.SqlClient.SqlParameter
0e90d850 548 28496 System.IO.MemoryStream
67b71a0c 1461 29220 System.Transactions.SafeIUnknown
7a5ee588 1924 30784 System.Collections.Specialized.ListDictionary+NodeKeyValueCollection
648c91f4 665 31920 System.Configuration.ConfigurationValues
7a5e5d04 1342 32208 System.Threading.Semaphore
652410f8 670 34840 System.Data.ProviderBase.DbConnectionPool+PoolWaitHandles
6613228c 1319 36932 System.Web.Security.FileSecurityDescriptorWrapper
66106948 2449 39184 System.Web.UI.AttributeCollection
0e8ff780 2021 40420 Microsoft.Win32.SafeHandles.SafeLsaPolicyHandle
01e34730 336 43008 Oracle.DataAccess.Client.OracleDataReader
648c9434 2218 44360 System.Configuration.ConfigurationValue
7a5ea0e4 1918 46032 System.Collections.Specialized.ListDictionary+NodeKeyValueCollection+NodeKeyValueEnumerator
7a5eaaa8 3088 49408 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
652435c4 1138 59176 System.Data.SqlClient.SqlBuffer
0e912c9c 2491 59784 System.Collections.ArrayList
0e9132c0 1236 69216 System.Collections.Hashtable
6614bf64 45 69660 System.Web.Caching.ExpiresEntry[]
0e8ff7d8 4042 80840 Microsoft.Win32.SafeHandles.SafeLsaMemoryHandle
66105ff4 5434 86944 System.Web.UI.StateBag
01e364c8 5686 90976 Oracle.DataAccess.Client.OpoSqlValTimeoutCtx
0e912e08 1007 91556 System.Int32[]
7a5ee300 3942 94608 System.Collections.Specialized.ListDictionary+NodeEnumerator
01e35ef8 7918 95016 Oracle.DataAccess.Client.OpoSqlRefCtx
01e353bc 6043 96688 Oracle.DataAccess.Client.MetaData
0e8f83e8 5017 100340 Microsoft.Win32.SafeHandles.SafeLocalAllocHandle
7a5ef738 6284 125680 System.Collections.Specialized.HybridDictionary
7a5ef7f4 5143 144004 System.Collections.Specialized.ListDictionary
661060d0 10908 174528 System.Web.UI.StateItem
0e91189c 533 184492 System.Char[]
6610d15c 2426 203784 System.Web.UI.WebControls.TableCell
01e362ec 7918 221704 Oracle.DataAccess.Client.OracleXmlQueryProperties
7a5ef8b4 11231 224620 System.Collections.Specialized.ListDictionary+DictionaryNode
65242390 1814 232192 System.Data.SqlClient._SqlMetaData
0e8f832c 12124 242480 Microsoft.Win32.SafeHandles.SafeTokenHandle
01e36444 7918 253376 Oracle.DataAccess.Client.OracleXmlSaveProperties
0e8f7ca8 13394 267880 Microsoft.Win32.SafeHandles.SafeWaitHandle
0e9133bc 1255 267912 System.Collections.Hashtable+bucket[]
0e8f7a98 12048 289152 System.Threading.ManualResetEvent
0e8e443c 7886 385508 System.Object[]
01e34b60 6456 387360 Oracle.DataAccess.Client.OpoConRefCtx
01e33860 6432 668928 Oracle.DataAccess.Client.OracleConnection
01e34f9c 6439 824192 Oracle.DataAccess.Client.OpoConCtx
01e34038 7918 1171864 Oracle.DataAccess.Client.OracleCommand
000dfbe0 70 5839608 Free
0e9136dc 2622 17492932 System.Byte[]
0e910c6c 56049 19472876 System.String
Total 283875 objects
If mem usage drops to 200 MB after a time, this shows your memory is beeing collected, but you still might have a memory missuse issue.
this dump doesn't show alot, but if it was taken when the process is 1GB as you said, you can still use it:
1) use !gcroot on several objects, to see how they are attached to the memory (i would check the DB usage, it seems you have a large amount of oracle connections (6432), and alot of other DB stuff floating around.)
like this:
!dumpheap -MT <mt = the left most number>
objects will show with memory addresses
!gcroot <address>
an object stack will show displaying how the object is attached to the memory tree.
a sample of this process
2) check performance counters (start->run->perfmon) add these counters:
- .Net Clr Memory-> #bytes all heaps
- Process->private bytes
calculate the difference between them - this is the memory consumed by unmanaged resources (like DB client objects)
check this in low memory and in high memory scenarios, and you will see if the memory consumption is mostly due to Managed memory (all heaps) or unmanaged.
3) if the memory is unmanaged, it's still likely to be held by managed objects, as the main application is managed, so making sure you free unmanaged resources after you are done with them is key. (close DBConnections, dispose DBCommands, close file handles, free COMObjects etc.)
Hope this helps,
Amit.