I am dropped into the ldb with the message
Heap exhausted during garbage collection: 0 bytes available, 16
requested.
The message itself is clear. Though I have a hard time figuring out where my memory is going to. I tried to use the ldb command dump but I always get
invalid Lisp-level address:
How can I use dump and is there a way to print a large block of used memory in a human readable way to improve the understanding of what is cluttering the memory?
E.g.
ldb> dump magicKeyWords
#0xFFFF OBJECTTYPEA
#0xFFFE OBJECTTYPEA
....
Related
Problem Background
From the past few months we found issue in my online Asp.net application. Application working fine but 1 or 2 times in a day it suddenly crash on different modules on live server, but their is no as such issue in code at all and find not such types of issues on local server.
After some research i found that process that run my application on IIS on live server its memory increase continuously and when it reaches at certain level its begin to crash.
Temporary solution:
when ever we found such issue we restart application on IIS.
which can end this process and start new process,then after that application start working.
In a day some time we need to restart our application 2 or 3 times some times more.
Problem that i find: memory leak.
Find some solution after some research:
create a dump file from my application process from task manager when application is crashes.
Tool use: Windbg
open in Windbg tool for analysis.
write command
.load by clr
dumpheap -stat
shows a tons of references of datatypes.
now i stuck at thses point.
i share with you in image section.
Question:
1. I am on the right direction in finding memory leaks issue?
2. if my path is right where whats my next step?
3. Windbg tool is good for finding such kind of issue?
dump file link for detail review, i take this dump file when server stop to response
create a dump file from my application process from task manager when application is crashes
That's not a good choice, because
you don't have much time to do so. You can only do that as long as the crash dialog is displayed. If you're too late, the application is gone.
in that state, you'll have difficulties debugging it. Instead of the original exception, it will show a breakpoint, which is used by the OS to show the dialog and collect diagnostic data
Use WER local dumps to automatically create a crash dump when it crashes instead of doing it manually. It's much more reliable and gives you the original exception. See How do I take a good crash dump for .NET
I am on the right direction in finding memory leaks issue?
Sorry, you're on the wrong track already.
Starting with !dumpheap -stat is not a good idea. Usually one would start at the lowest level, which is !address -summary. It will give you an indicator whether it's a managed memory leak or a native memory leak. If it's a managed leak, you could continue with !dumpheap -stat
if my path is right where whats my next step?
Even if it's not the right path, it's a good idea that you learn how figure out that you're on the wrong path. So, how do I know?
Looking at your output of !dumpheap -stat, you can see
[...]
111716 12391360 System.String.
This tells you there are 110.000 different strings, using 12 MB of memory. It also tells you that everything else takes less than 12 MB. Look at the other sizes and you'll find out that .NET is not the reason for your OutOfMemoryException. They use less than 50 MB.
If there were a managed leak, you would look for paths where objects are connected to, so that the garbage collector thinks it cannot be freed. The command is !gcroot.
Windbg tool is good for finding such kind of issue?
It is possible, but WinDbg is not the best tool. Use a memory profiler instead. That's a dedicated tool for memory leaks. Typically it has a much better usability. Unfortunately you'll need to decide whether you need a managed memory profiler, native memory profiler or both.
I once wrote how to use WinDbg to track down .NET OutOfMemoryException. You'll find a chart there which gives you ideas on how to proceed in different situations.
In your dump I see 2 TB of <unknown> memory, which could be .NET, but needn't be. Still, these 2 TB are likely the cause of the OOM, because the rest is less than 350 MB in size.
Since clr is in the list of loaded modules, we can check !dumpheap -stat as you did. But there are not many objects using memory.
!eeheap -gc shows that there are 8 heaps, corresponding to the 8 processors of your machine, for parallel garbage collection. The largest individual heap is 45 MB, the total 249 MB. This roughly matches the sum of !dumpheap. conclusion: .NET is not the culprit.
Let's check the special cases:
Presence of MSXML
Bitmaps
Calls to HeapAlloc() which are so large that they are directly forwarded to VirtualAlloc().
Direct calls to VirtualAlloc()
MSXML is not present: lm m msxml* does not produce output.
There are no Bitmaps: !dumpheap -stat -type Bitmap
Heap allocations larger than 512 kB: !heap -stat. Here's a truncated part of the output:
0:000> !heap -stat
_HEAP 0000018720bd0000
Segments 00000006
Reserved bytes 0000000001fca000
Committed bytes 0000000001bb3000
VirtAllocBlocks 00000002
VirtAlloc bytes 00000312cdc4b110
_HEAP 0000018bb0fe0000
Segments 00000005
Reserved bytes 0000000000f0b000
Committed bytes 0000000000999000
VirtAllocBlocks 00000001
VirtAlloc bytes 0000018bb0fe0110
As you can see, there are 3 blocks that went to VirtualAlloc. The size is somewhat unrealistic:
0:000> ? 00000312cdc4b110
Evaluate expression: 3379296514320 = 00000312`cdc4b110
0:000> ? 0000018bb0fe0110
Evaluate expression: 1699481518352 = 0000018b`b0fe0110
That would be a total of 3.3TB + 1.7TB = 6TB and not 2TB. Now, it may happen that this is a bug of !address, but 4TB is not a common overflow point.
With !heap -a 0000018720bd0000 you can see the 2 virtual allocs:
Virtual Alloc List: 18720bd0110
0000018bac70c000: 00960000 [commited 961000, unused 1000] - busy (b), tail fill
0000018bad07b000: 00960000 [commited 961000, unused 1000] - busy (b), tail fill
And with !heap -a 0000018bb0fe0000 you can see the third one:
Virtual Alloc List: 18bb0fe0110
0000018bb1043000: 00400000 [commited 401000, unused 1000] - busy (b), tail fill
These are all relatively small blocks of 4.1MB and 9.8 MB.
For the last part, direct calls to VirtualAlloc(), you need to get back to the level of !address. With !address -f:VAR -c:".echo %1 %3" you can see the address and size of all <unknown> regions. You'll find a lot of entries there, many of small sizes, some which could be the .NET heaps, a few 2GB ones and one really large allocation
The 2GB ones:
0x18722070000 0x2d11000
0x18724d81000 0x7d2ef000
0x187a2070000 0x2ff4000
0x187a5064000 0x7d00c000
0x18822070000 0x2dfe000
0x18824e6e000 0x7d202000
0x188a2070000 0x2c81000
0x188a4cf1000 0x7d37f000
0x18922070000 0x2d13000
0x18924d83000 0x7d2ed000
0x189a2070000 0x2f5a000
0x189a4fca000 0x7d0a6000
0x18a22070000 0x2c97000
0x18a24d07000 0x7d369000
0x18aa2070000 0x2d0c000
0x18aa4d7c000 0x7d2f4000
It is likely that these are the .NET heaps (committed part + reserved part).
The large one:
0x7df600f57000 0x1ffec56a000
The information about it:
0:000> !address 0x7df600f57000
Usage: <unknown>
Base Address: 00007df6`00f57000
End Address: 00007ff5`ed4c1000
Region Size: 000001ff`ec56a000 ( 2.000 TB)
State: 00002000 MEM_RESERVE
Protect: <info not present at the target>
Type: 00040000 MEM_MAPPED
Allocation Base: 00007df5`ff340000
Allocation Protect: 00000001 PAGE_NOACCESS
It looks like a 2TB memory mapped file which is unused (and therefore reserved).
I don't know what your application is doing. This is really where I need to stop the analysis. I hope this was helpful and you can draw your conclusions and fix the issue.
I currently have a kernel that processes a global buffer by reading
into local memory and doing calculations. Now, I would like to use registers
instead of local memory. How do I convert to registers?
Thanks!
Edit: project can be found here:
https://github.com/boxerab/ocldwt
Without seeing some code, it's impossible to give much more guidance than has already been given but I will try to elaborate on the comments.
Any variable declared without __local or __global is private, so if you remove the modifier the memory will only be visible to the single processing element running the work item. This will likely be stored in a register, although that will only happen if there is register space available. The compiler will already be putting some values into registers on your behalf, even if you haven't asked it to do so. You can see evidence of this if, for example, you are running on the NVIDIA platform and pass the -cl-nv-verbose flag when you build your kernels. You will see output like this:
ptxas info : Compiling entry function 'monte' for 'sm_35'
ptxas info : Function properties for monte
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 61 registers, 360 bytes cmem[0], 96 bytes cmem[2]
indicating that 61 registers are in use.
However, as pointed out by #DarkZeros, the decision to move from local memory to private memory is much more about the scope of variables. If your algorithm depends on all of the members of a compute unit having access to the same copy of a variable, then it isn't going to work any more.
OK, all you ASP.NET Experts: I have used reflector to look into ASP.NET Cache implementation (which sits on HttpRuntime.Cache and HttpContext.Current.Cache) uses a Hashtable internally to keep the cache.
However, the data gets stored in unmanaged memory. This is very strange since I could not see anywhere data getting stored in unmanaged memory. However, writing a very simple web application that inserts a chunk of byte array into cache, we can see this:
Private Bytes: 460MB
Bytes in all heaps: 150MB
=>
Managed Memory: 150 MB
Unmanaged Memory: 310 MB
So basically I am calling the application many times (each increase is 1000x requests each putting 64KB empty buffer byte[] into cache). So the one that has grown the most is private bytes (total memory) instead of bytes in all heaps (managed memory). However I am expecting managed memory to grow in line with total memory since I am adding objects to the managed heap using Hashtable.
Can you please explain this behaviour?
UPDATE
As Simon said, the bytes in all heaps value only changes after a garbage collection - I changed the code to induce garbage collection and it update the counters. Increase in Gen 2 Heap memory is EXACTLY the same as the amount of memory added. However, unmanaged memory is still much higher. In this example, Heap 2 was only 96MB while total memory 231 MB.
The # Bytes in all Heaps is only updated when the garbage collection is executed, while the Private Bytes is available at much faster update rate. (I'm not sure where that number comes from, internally, and how often it's updated.)
The amount of Private Bytes increases just after 17:42:45. This amount does seem to match the value jump of # Bytes in all Heaps at about 17:43:10. It looks like it took 20-25 seconds before any garbage collection was done and updated the # Bytes in all Heapscounter.
It's hard to work out how memory allocations work from a few minutes worth of performance counters presented in a screenshot. ;) Keep running your test and see how your expectations work out over a longer time period.
TL;DR: The amount of managed bytes should correlate with private bytes, but the managed counter will only update during a garbage collection.
Small note from the OP: As this response says, the lagging in the memory can be fully explained by lagging GC. The fact that unmanaged memory also rises was not my question. So thanks #Simon.
I'm trying to write a memory test program for the NXT, since I have several with burned memory cells and would like to identify which NXTs are unusable. This program is intended to test each byte in memory for integrity by:
Allocating 64 bits to an Linear Feedback Shift Register randomizer
Adding another byte to a memory pointer
Writing random data to the selected memory cell
Verifying the data is read back correctly
However, I then discovered through these attempts that the NXT doesn't actually support pointer operations. Thus, I can't simply iterate the pointer byte and read its location to test.
How do I go about iterating over indexes in memory without pointers?
I think the problem is that you don't really get direct memory access in either NBC/NXC or RobotC.
From what I know, both run on an NXT firmware emulator; so the bad memory address[es] might change from your program's point of view (assuming the emulator does virtual memory).
To actual run bare metal, I would suggest using the NXTBINARY function of John Hansen's modified firmware as described here:
http://www.tau.ac.il/~stoledo/lego/nxt-native/
The enhanced fimware can be found at:
http://bricxcc.sourceforge.net/test_releases/
Sometimes, on various Unix architectures, recompiling a program while it is running causes the program to crash with a "Bus error". Can anyone explain under which conditions this happens? First, how can updating the binary on disk do anything to the code in memory? The only thing I can imagine is that some systems mmap the code into memory and when the compiler rewrites the disk image, this causes the mmap to become invalid. What would the advantages be of such a method? It seems very suboptimal to be able to crash running codes by changing the executable.
On local filesystems, all major Unix-like systems support solving this problem by removing the file. The old vnode stays open and, even after the directory entry is gone and then reused for the new image, the old file is still there, unchanged, and now unnamed, until the last reference to it (in this case the kernel) goes away.
But if you just start rewriting it, then yes, it is mmap(3)'ed. When the block is rewritten one of two things can happen depending on which mmap(3) options the dynamic linker uses:
the kernel will invalidate the corresponding page, or
the disk image will change but existing memory pages will not
Either way, the running program is possibly in trouble. In the first case, it is essentially guaranteed to blow up, and in the second case it will get clobbered unless all of the pages have been referenced, paged in, and are never dropped.
There were two mmap flags intended to fix this. One was MAP_DENYWRITE (prevent writes) and the other was MAP_COPY, which kept a pure version of the original and prevented writers from changing the mapped image.
But DENYWRITE has been disabled for security reasons, and COPY is not implemented in any major Unix-like system.
Well this is a bit complex scenario that might be happening in your case. The reason of this error is normally the Memory Alignment issue. The Bus Error is more common to FreeBSD based system. Consider a scenario that you have a structure something like,
struct MyStruct {
char ch[29]; // 29 bytes
int32 i; // 4 bytes
}
So the total size of this structure would be 33 bytes. Now consider a system where you have 32 byte cache lines. This structure cannot be loaded in a single cache line. Now consider following statements
Struct MyStruct abc;
char *cptr = &abc; // char point points at start of structure
int32 *iptr = (cptr + 1) // iptr points at 2nd byte of structure.
Now total structure size is 33 bytes your int pointer points at 2nd byte so you can 32 byte read data from int pointer (because total size of allocated memory is 33 bytes). But when you try to read it and if the structure is allocated at the border of a cache line then it is not possible for OS to read 32 bytes in a single call. Because current cache line only contains 31 bytes data and remaining 1 bytes is on next cache line. This will result into an invalid address and will give "Buss Error". Most operating systems handle this scenario by generating two memory read calls internally but some Unix systems don't handle this scenario. To avoid this, it is recommended take care of Memory Alignment. Mostly this scenario happen when you try to type cast a structure into another datatype and try reading the memory of that structure.
The scenario is bit complex, so I am not sure if I can explain it in simpler way. I hope you understand the scenario.