I am a creating a huge mesh object (some 900 megabytes in size).
Once I am done with analysing it, I would like to somehow delete it from the memory.
I did a bit of search on stackoverflow.com, and I found out that del will only delete the reference to mentioned mesh. Not the mesh object itself.
And that after some time, the mesh object will eventually get garbage collected.
Is gc.collect() the only way by which I could instantly release the memory, and there for somehow remove the mentioned large mesh from the memory?
I've found replies here on stackoverflow.com which state that gc.collect() should be avoided (at least when it comes to regular python, not specifically ironpython).
I've also find comments here on stackoverflow which claim that in IronPython it is not even guaranteed the memory will be released if nothing else is holding a reference.
Any comments on all these issues?
I am using ironpython 2.7 version.
Thank you for the reply.
In general managed enviroments relase there memory, if no reference is existing to the object anymore (from connection from the root to the object itself). To force the .net framework to release memory, the garbage collector is your only choice. In general it is important to know, that GC.Collect does not free the memory, it only search for objects without references and put the in a queue of objects, which will be released. If you want to free memory synchron, you also need GC.WaitForPendingFinalizers.
One thing to know about large objects in the .net framework is, that they are stored seperatly, in the Large Object Heap (LOH). From my point of few, it is not bad to free those objects synchron, you only have to know, that this can cause some performance issues. That's why in general the GC decide on it's own, when to collect and free memory and when not to.
Because gc.collect is implemented in Python as well as in IronPython, you should be able to use it. If you take a look at the implementation in IronPython, gc.collect does exactly what you want, call GC.Collect() and GC.WaitForPendingFinalizer. So in your case, i would use it.
Hope this helps.
Related
I was wondering if it is possible to create a programming language without explicit memory allocation/deallocation (like C, C++ ...) AND without garbage collection (like Java, C#...) by doing a full analysis at the end of each scope?
The obvious problem is that this would take some time at the end of each scope, but I was wondering if it has become feasible with all the processing power and multiple cores in current CPU's. Do such languages exist already?
I also was wondering if a variant of C++ where smart pointers are the only pointers that can be used, would be exactly such a language (or am I missing some problems with that?).
Edit:
Well after some more research apparently it's this: http://en.wikipedia.org/wiki/Reference_counting
I was wondering why this isn't more popular. The disadvantages listed there don't seem quite serious, the overhead should be that large according to me. A (non-interpreted, properly written from the ground up) language with C family syntax with reference counting seems like a good idea to me.
The biggest problem with reference counting is that it is not a complete solution and is not capable of collecting a cyclic structure. The overhead is incurred every time you set a reference; for many kinds of problems this adds up quickly and can be worse than just waiting for a GC later. (Modern GC is quite advanced and awesome - don't count it down like that!!!)
What you are talking about is nothing special, and it shows up all the time. The C or C++ variant you are looking for is just plain regular C or C++.
For example write your program normally, but constrain yourself not to use any dynamic memory allocation (no new, delete, malloc, or free, or any of their friends, and make sure your libraries do the same), then you have that kind of system. You figure out in advance how much memory you need for everything you could do, and declare that memory statically (either function level static variables, or global variables). The compiler takes care of all the accounting the normal way, nothing special happens at the end of each scope, and no extra computation is necessary.
You can even configure your runtime environment to have a statically allocated stack space (this one isn't really under the compiler's control, more linker and operating system environment). Just figure out how deep your function call chain goes, and how much memory it uses (with a profiler or similar tool), an set it in your link options.
Without dynamic memory allocation (and thus no deallocation through either garbage collection or explicit management), you are limited to the memory you declared when you wrote the program. But that's ok, many programs don't need dynamic memory, and are already written that way. The real need for this shows up in embedded and real-time systems when you absolutely, positively need to know exactly how long an operation will take, how much memory (and other resources) it will use, and that the running time and the use of those resources can't ever change.
The great thing about C and C++ is that the language requires so little from the environment, and gives you the tools to do so much, that smart pointers or statically allocated memory, or even some special scheme that you dream up can be implemented. Requiring the use them, and the constraints you put on yourself just becomes a policy decision. You can enforce that policy with code auditing (use scripts to scan the source or object files and don't permit linking to the dynamic memory libraries)
The current way our application is written, involves creating a new connection and command object in every method that access our sqlite db. Considering we need it to run on a WM5 device, that is leading to hideous performance.
Our plan is to use just one connection object per-thread but it's also occurred to us to use one global command object per-thread too. The benefit of this is it reduces the overhead on the garbage collector created by instantiating objects all over the place.
I can't find any advice against doing this but wondered if anyone can answer definitively if this is a good or bad thing to do, and why?
While I'm not sure about reducing the number of command objects, reducing the number of connections is definitely a good plan. They're designed to be relatively expensive to set up (hey, they involve actually opening a disk file!) so keeping them around for a relatively long time is highly sensible. So do the first stage of your plan and retime to see if that makes things good enough, or if you need to do more work optimizing…
Note that it is quite possible that generating the command objects once per connection will be a saving too, since that will allow them to be compiled once and reused multiple times. Not that that matters until you're persisting the connection in the first place!
I know GC wasn't popular in the days when Ada was developed and for the main use case of embedded programming it is still not a good choice.
But considering that Ada is a general purpose programming language why wasn't a partial and optional (traces only explicitly tagged memory objects) garbage collector introduced in later revisions of the language and the compiler implementations.
I simply can't think of developing a normal desktop application without a garbage collector anymore.
Ada was designed with military applications in mind. One of the big priorities in its design was determinism. i.e. one wanted an Ada program to consistently perform exactly the same way every time, in any environment, under all operating systems... that kinda thing.
A garbage collector turns one application into two, working against one another. Java programs develop hiccups at random intervals when the GC decides to go to work, and if it's too slow about it there's a chance that an application will run out of heap sometimes and not others.
Simplified: A garbage collector introduces some variability into a program that the designers didn't want. You make a mess - you clean it up! Same code, same behavior every time.
Not that Ada became a raging worldwide success, mind you.
Because Ada was designed for use in defense systems which control weapons in realtime, and garbage collection interferes with the timing of your application. This is dangerous which is why, for many years, Java came with a warning that it was not to be used for healthcare and military control systems.
I believe that the reason there is no longer such a disclaimer with Java is because the underlying hardware has become much faster as well as the fact that Java has better GC algorithms and better control over GC.
Remember that Ada was developed in the 1970's and 1980's at a time when computers were far less powerful than they are today, and in control applications timing issues were paramount.
First off, there is nothing in the language really that prohibits garbage collection.
Secondly some implementations do perform garbage collection. In particular, all the implementations that target the JVM garbage collect.
Thirdly, there is a way to get some amount of garbage collection with all compilers. You see, when an access type goes out of scope, if you specifially told the language to set aside a certian amount of space for storage of its objects, then that space will be destroyed at that point. I've used this in the past to get some modicum of garbage collection. The declaration voodo you use is:
type Foo is access Blah;
for Foo'storage_size use 100_000_000; --// 100K
If you do this, then all (100K of) memory allocated to Blah objects pointed to by Foo pointers will be cleaned up when the Foo type goes out of scope. Since Ada allows you to nest subroutines inside of other subroutines, this is particularly powerful.
To see more about what storage_size and storage pools can do for you, see LRM 13.11
Fourthly, well-written Ada programs don't tend to rely on dynamic memory allocation nearly as much as C programs do. C had a number of design holes that practicioners learned to use pointers to paint over. A lot of those idioms aren't nessecary in Ada.
the answer is more complicated: Ada does not require a garbage collector, because of real-time constraints and such. however, the language have been cleverly designed so as to allow the implementation of a garbage collector.
although, many (almost all) compilers do not include a garbage collector, there are some notable implementation:
a patch for GNAT
Ada compilers targeting the Java Virtual Machine (i don't know if those projects are still supported). It used the garbage collector of the JVM.
there are plenty other sources about garbage collection in Ada around the web. this subject has been discussed at length, mainly because of the fierce competition with Java in the mid '90s (have a look at this page: "Ada 95 is what the Java language should have been"), when Java was "The Next Big Thing" before Microsoft drew C#.
First off, I'd like to know who's using Ada these days. I actually like the language, and there's even a GUI library for Linux/Ada, but I haven't heard anything about active Ada development for years. Thanks to its military connections, I'm really not sure if it's ancient history or so wildly successful that all mention of its use is classified.
I think there's a couple of reason for no GC in Ada. First, and foremost, it dates back to an era where most compiled languages used primarily stack or static memory, or in a few cases, explicit heap allocate/free. GC as a general philosophy really only took off about 1990 or so, when OOP, improved memory management algorithms and processors powerful enough to spare the cycles to run it all came into their own. What simply compiling Ada could do to an IBM 4331 mainframe in 1989 was simply merciless. Now I have a cell phone that can outperform that machine's CPU.
Another good reason is that there are people who think that rigorous program design includes precise control over memory resources, and that there shouldn't be any tolerance for letting dynamically-acquired objects float. Sadly, far too many people ended up leaking memory as dynamic memory became more and more the rule. Plus, like the "efficiency" of assembly language over high-level languages, and the "efficiency" of raw JDBC over ORM systems, the "efficiency" of manual memory management tends to invert as it scales up (I've seen ORM benchmarks where the JDBC equivalent was only half as efficient). Counter-intuitive, I know, but these days systems are much better at globally optimizing large applications, plus they're able to make radical re-optimizations in response to superficially minor changes.Including dynamically re-balancing algorithms on the fly based on detected load.
I'm afraid I'm going to have to differ with those who say that real-time systems can't afford GC memory. GC is no longer something that freezes the whole system every couple of minutes. We have much more intelligent ways to reclaim memory these days.
Your question is incorrect. It does. See the package ada.finalization which handles GC for you.
I thought I'd share a really simple example of how to implement a Free() procedure (which would be used in a way familiar to all C programmers)...
with Ada.Integer_Text_IO, Ada.Unchecked_Deallocation;
use Ada.Integer_Text_IO;
procedure Leak is
type Int_Ptr is access Integer;
procedure Free is new Ada.Unchecked_Deallocation (Integer, Int_Ptr);
Ptr : Int_Ptr := null;
begin
Ptr := new Integer'(123);
Free (Ptr);
end Leak;
Calling Free at the end of the program will return the allocated Integer to the Storage Pool ("heap" in C parlance). You can use valgrind to demonstrate that this does in fact prevent 4 bytes of memory being leaked.
The Ada.Unchecked_Deallocation (a generically defined procedure) can be used on (I think) any type that may be allocated using the "new" keyword. The Ada Reference Manual ("13.11.2 Unchecked Storage Deallocation") has more details.
I have been working on an Adobe Flex application for some months now, and the application is meant to run 24/7 for days (weeks!) continuously. However, I'm now seeing that after a few days of running nonstop the computer it runs on tells me that the system is low on virtual memory and gives me an error about Page File usage. Once I close the Flex app, the Page File usage goes down from 1.9 GB to 100 MB (or less). It seems that its using up all this memory and not freeing it although I have been very careful in my app to not keep huge arrays.
The app does some graphing and draws a lot of shapes (to greate a 'gauge') and then gets rid of them by re-declaring that object as another 'gauge'.
Any idea why my page file usage is climbing so high?!
You most probably have eventListeners that are not being removed. They keep references to objects and prevent them from being garbage collected.
You can use the profiler in Flex Builder professional to see where your memory usage is going. Like another poster mentioned, event listeners are alot of times the culprits in cases like this, but more generally, just because you think you are getting rid (destroying or deleting) a variable, doesn't mean that it is really getting taken care of by the garbage collector. If any reference (like an event listener) still exists to that variable (or object) it will not be collected. The profiler will point out these things.
I've heard rumors that putting anything on the Stage will create memory leaks. In other words, you can be as careful as possible with your code, but you'll still leak memory. This has not been validated by Adobe, as far as I know. A good test might be to instantiate a Shape and a Sprite and a MovieClip, add them to the display list, and then let the app run overnight. Would love to hear the results if you do end up testing this.
Is it possible to programmatically force a full garbage collection run in ActionScript 3.0?
Let's say I've created a bunch of Display objects with eventListeners and some of the DO's have been removed, some of the eventListeners have been triggered and removed etc... Is there a way to force garbage collection to run and collect everything that is available to be collected?
Yes, it's possible, but it is generally a bad idea. The GC should have a better idea of when is a good time to run than you should, and except for a very specific case, like you just used 500MB of memory and you need to get it back ASAP, you shouldn't call the GC yourself.
In Flash 10, there is a System.gc() method you can call (but please don't, see above) - keep in mind System.gc() only works in the debugging version of Flash player 10+.
In Flash 9, there is an unsupported way to force it via an odd LocalConnection command, but it may not work in all versions. See this post by Grant Skinner.
There is a new API for telling the GC that it might be a "relatively good moment" to collect.
See the Adobe API docs for
System.pauseForGCIfCollectionImminent
And also this Adobe blog post from shortly after the method was introduced in Player version 11
The method takes an "imminence" argument; basically, you feed in a low number (near 0.0) if you really want the collector to run, even if there has not been much activity (currently measured by bytes-allocated) since the last collection, and you feed in a large number (near 1.0) if you only want the collection pause to happen if we were already near the point where a collection would happen anyway.
The motivation here is for situations in e.g. games where you want to shift the point where GC's happen by a small amount, e.g. do the GC during a change of level in the game, rather than two seconds after the player started exploring the level.
One very important detail: This new API is supported by both the Release and the Debugger Flash Runtimes. This makes it superior to calling System.gc().
For all currently released versions, System.gc() only works in the debug version of the Flash player and ADL (the debug environment for AIR apps). Flash player 10 beta currently does work in all flavors.
I agree with Davr, it's a bad idea to do. The runtime will usually have a better idea than you do.
Plus, the specifics of how the garbage collector works is an implementation detail subject to change between flash player versions. So what works well today has no guarantee to work well in the future.
As others said: do not try to GC manually, there are hacks but it's not safe.
You should try recycling objects when you can - you'll save a lot of memory.
This can be applied for instance to BitmapDatas (clear and reuse), particles (remove from display and reuse).
I have a comment on those saying you should never do GC manually. I'm used to manual memory management in C++ and I prefer sharedptr a lot over GC, but anyway.
There is a specific case where I can't find another solution than do a GC. Please consider: I have a DataCache class, the way it work is it keeps result objects for certain method calls that send out updated events when refreshing/receiving data. The way the cache is refreshed is I just clean all results from it and send the event which causes any remaining listeners to re-request their data and listeners that went out of scope should not rerequest which cleans out not needed results. But apparently, if I can't force all listeners that still dangle waiting for GC to be cleaned up immediatly before sending out the "ask you data again" event, those dangling listeners will request data again unnecessarily. So since I can't removeEventListener because AS3 doesn't have destructors I can't see another easy solution than forcing a GC to make sure there's no dangling listeners anymore.
(Edit) On top of that I cannot use removeEventListener anyway for binding which were setup in mxml, for example (using my custom DataCacher class which handles remoteobj)
<mx:DataGrid id="mygrid" dataProvider="{DataCacher.instance().result('method').data}" ... />
When the popup window containing this datagrid is closed, you would expect the bindings to be destroyed. Apparently they live on and on. Hmm, shouldn't flex destroy all bindings (meaning eventlisteners) from an object when it's being marked for GC because the last reference is deleted. That would kinda solve the problem for me.
At least that's why I think, I'm still a beginner in Flex so any thoughts would be appreciated.
try {
new LocalConnection().connect('foo');
new LocalConnection().connect('foo');
} catch (e:*){
trace("Forcing Garbage Collection :"+e.toString());
}
If you have to, calling the gargabe collector could be useful... so, you have to be carefull how and when you do it, but there is no doubt that there are times when is neccesary.
for example, if you have an app that is modular, when you change from one view to the other, all the deleted objects could represent a large amount of memory that should be available as faster as possible, you just need to have control of the variables and references you are disposing.
recycling doesn't really help. I used one loader that repeatedly loaded the same jpg every 500ms. task manager still reported a non stop increase in memory.
tried and proven solution here.
http://simplistika.com/as3-garbage-collection/