asdagfsdgas memory leaks

[BwdYeti]

Okay, biggest problem that's always plagued FE7x (after reducing system load) is memory leaks. Every few months I have to come up with a bunch of debug code to find out why it's happening and fix things, but I never get everything, and have to come up with a more thorough way next time which still doesn't entirely work.
It's much better than it has been, in the past it could pick up 10MB or more per turn (about 3-10 minutes?), now it's less than 1. But it should be zero.
So I want to know how I can deal with it completely. Places to look for problems, ways to check if specific things are flooding the ram (you had 50 Sprites in memory and now you have 500, SPRITES MIGHT NOT BE GETTING DISPOSED), data types that I wouldn't expect to be the culprit, if any objects that aren't disposed still need to be freed from memory, anything.
Also an explanation of RPG::Cache, and if it can ever be part of the problem.

 

[Neo-Bahamut]

I hope I understood you problem right =o

Well, Ruby doesn't really have the usual "memory leak" which occures by bad code.
Disposing stuff should not do much except for Bitmaps.

Ruby does already clean up the unused objects with the Garbage Collector.
So, what you should be looking for isn't objects which aren't disposed. Except for Sprites there is no class which needs this and if you don't dispose a Sprite you will clearly see that mistake since the Sprite will stay on the screen even if it shouldn't. If you have problems with the memory, you have to check for variables which have a global scope. Those are global variables and constants.
You can also try calling GC.start which clears all objects which are in memory but unused.

The Cache:
RPG::Cache saves all Bitmaps you load. This provides faster access to those but of course higher RAM use. You can clear the Cache with RPG::Cache.clear.
It can be part of the problem but there isn't much you can do and there isn't much you should do since the cache is a good system.
Perhaps you might clear the cache before a new map is loaded so you only have the graphics in memory which you really need at the moment:

class Game_Map

 alias_method :bwdyeti_has_too_much_memory, :setup

 def setup(id)

  RPG::Cache.clear

  bwdyeti_has_too_much_memory(id)

 end

end

I hope I could help you :cute:

 

[BwdYeti]

It's the "unused things are not being freed" kind of memory leak, not the "memory is getting reserved and never closed" kind and goes away when the program is closed, so that's good I guess.

Hmmm. I think part of the problem might be assuming disposing things would get rid of them, when they need to not have references anymore so the garbage collector notices?
Are there any things to know about the garbage collector, and what it deems unused? For example, when the scene is changed how does it know to get rid of the old one and everything under it, does it see the old scene isn't tied to any variable reference, gets rid of it, and then repeats the process on each of its instance vars?
Global variables shouldn't be a problem since I rarely use those, constants I have a lot of to define data but I don't think I ever modify them.
Cache probably isn't part of the problem then, it increases memory use when loading new things but once they're in memory it levels off. I'm talking about when I run the same block of code repeatedly and the memory consumption slowly rises; most recent problem was creating a class reel that shows off the classes and animations when you sit on the title, and each time it started over the memory use was slightly higher. Confusing part was sometimes it was the same, so it's harder to pinpoint the problem >:

And undisposed sprites can sometimes be a problem when their bitmap is disposed/nil p:

 

[Neo-Bahamut]

I think part of the problem might be assuming disposing things would get rid of them, when they need to not have references anymore so the garbage collector notices? Are there any things to know about the garbage collector, and what it deems unused?

Disposing won't really help. It doesn't really frees the Sprites/Bitmaps/whatever but only makes them unusable.
The GC checks about every ten seconds for each object in memory if some variable is pointing on it. If it isn't it removes it.
And then, as you said, the object's instance variables' objects (lol nice expression) may also have no variable pointing on them.

Perhaps you should post your problematic code so we can have a look on it

If you are interested in the Garbage Collector maybe you want to read this article:
http://timetobleed.com/garbage-collecti ... onference/

 

[DeM0nFiRe]

Well, here's the thing, you have to remember this is not just Ruby. We have been assuming that Enterbrain has written the classes they've added with RGSS correctly, but remember there is still a C/C++ side of this thing and it's both possible and likely that there's a real memory leak on that side.

That being said, it is most likely (we can't say 100% sure since we can't see the source code) that disposing is more important than you think Neo-Bahamut. However, you also have to remember that disposing does not immediately free that memory, it just tells RGSS that it's OK to free that memory. If you are disposing and then reloading the same files via RPG::Cache, you will use more memory than if you had just not disposed it, because the old one is still going to be there. I would be able to recognize this sort of issue in code if I saw the code.

Unfortunately for you, BwdYeti, we are not going to be able to do much to help you without seeing the code that you are having trouble with.

 

[BwdYeti]

Alright, that helps. So say I have a Bitmap as an instance variable, dispose it, and don't intend to replace it or get rid of the object that holds it, I should set the instance variable to nil so the Bitmap has no reference and gets freed? That makes sense.

I think I just need to ensure that if any global variables or Modules or something are holding on to instance variables, if there's anything like that it would help to remove it. Also I should check if anything that's an instance of two things could just have the second object hold a reference and go through the first if it needs access.

Partially unrelated, but just another thought. If I have an array that periodically empties and fills with new data, should I use ' = []' or '.clear' ? Which is faster, which is 'better' programming, etc. I'm guessing '.clear', but I wanted to make sure.

@Demonfire, I've got at least as much code written as the default classes, maybe close to twice as much, so I doubt anyone wants to look through it for the problems, and I don't know exactly what's causing it. That's why I'm asking for general advice, so I can be more aware of if something looks wrong, and so anything new is done right. It's improved some just from the past couple days, so I think I'm on the right track.

 

[DeM0nFiRe]

Keep in mind that that in Ruby, it will automatically delete the object as long as all the references for it have fallen out of scope. That's why you don't see at the end of every scene:

@sprite.dispose
@sprite = nil

because once the scene changes to the new scene, the @sprite reference to the sprite object has fallen out of scope. Also, keep in mind that if you do something like

@scene = $scene
@scene = nil

$scene will still hold a reference to the object, so realistically speaking if you have to manually set a variable to nil, it probably means you've messed up the scopes somehow. If you are only using a variable once per frame, make it a local variable in the update function for that frame and then it will automatically be freed when the function exits each frame. In other words, make sure the scope of the variables you use are the absolute smallest they can be.

 

[Neo-Bahamut]

@ DeM0nFiRe: I just tried it. Neither Sprites nor Windows nor Bitmaps leave memory after the GC cleared even if you don't dispose them before.
I don't know much about C but as far as I know there are also destructors like in C# and C++ which seem to be called when the GC clears something.

@ BwdYeti: As DeM0nFiRe said, it is not necessary to use @array.clear, @array = nil or similar stuff.
When the variables are out of scope the GC will clear them anyway.
Only if you would have an object which uses e.g. 500 MB of memory you may want to clear it immediately:

class Test

 def initialize

  @ultra_object = Bitmap.new(15000, 15000)

 end

end

 

i = Test.new

# creating another instance of Test would make the game crash

# so you can free the memory directly:

i = nil

GC.start

i = Test.new

But hopefully you won't need this

 

[DeM0nFiRe]

@Neo-Bahamut: RGSS is not a language, RGSS is actually an engine that is wrapped for Ruby. What that means is that there's a whole C/C++ program where Enterbrain could have screwed up, and that stuff is not going to be automatically taken care of when the Ruby GC clears. The Ruby GC only takes care of the Ruby objects. In C/C++ there is no garbage collector, everything has to be taken care of manually. If Enterbrain did a poor job of that, memory leaks pop up like weeds.

EDIT: Also, unless Enterbrain has done better than I expect, your example will leak all of the memory for the bitmap data. You have to call bitmap.dispose, because the bitmap data is C/C++ data, not Ruby data. As I said, there is no GC in C/C++ that will take care of that for you.

 

[Neo-Bahamut]

I know, RGSS is a library and everything we are writing is Ruby
By the way, I think RGSS is written in Delphi but since it is compiled it doesn't really matter

Are you sure that Ruby handles it like this?
I think it is like this:
Ruby Object
-> Goes to Ruby GC
-> Calls the C (or whatever) classes destructor to free the memory

Otherwise every simple class like an Array or a Hash would probably result in memory leaks.

 

[DeM0nFiRe]

RGSS is not a library, it's an engine, there's a difference. Array and Hash are written as library classes, they are written in the specific way that Ruby manages things. An RGSS Bitmap is not a Ruby object. There is a significant amount of the bitmap that is not exposed to Ruby, and that's the difference. The Bitmap class we see is just a sort of "glue" between the engine and the script. There's actually a C/C++ (or maybe Delphi) object that is not actually connected to the Ruby GC.

 

[BwdYeti]

@ BwdYeti: As DeM0nFiRe said, it is not necessary to use @array.clear, @array = nil or similar stuff.

I meant when the object holding the array still needs an array, it just wants to get rid of the old one. For example, an array of targets that can be attacked or something that was set up with some method, and it needs updated; so, the old data isn't needed. Make a new array, or clear the old one out and refill it? It shouldn't matter, was just wondering if one was preferred.

 

[kyonides]

After checking Ruby code I noticed that ary.clear is faster since the C code is much shorter than ary.new and has less conditions to fulfill before it finishes execution but if you need to completely change an ary values soon, ary = [values] would be a proper solution. There's no real best way to code it, they're just options, alternatives.

 

[Neo-Bahamut]

@ DeM0nFiRe: I asked some people now, who know more about the C extensions in Ruby.
If Enterbrain wrote some code which is even halfway acceptable, the objects should work like I said :P
I copied a part of the official README.EXT:

3.3 Encapsulate C data into a Ruby object

To wrap and objectify a C pointer as a Ruby object (so called
DATA), use Data_Wrap_Struct().

Data_Wrap_Struct(klass, mark, free, ptr)

Data_Wrap_Struct() returns a created DATA object. The klass argument
is the class for the DATA object. The mark argument is the function
to mark Ruby objects pointed by this data. The free argument is the
function to free the pointer allocation. If this is -1, the pointer
will be just freed. The functions mark and free will be called from
garbage collector.

These mark / free functions are invoked during GC execution. No
object allocations are allowed during it, so do not allocate ruby
objects inside them.

And since Enterbrain had the "free" function do prevent all memory leaks, I don't think that this problem is their fault :biggrin:

@ BwdYeti (and also @kyonides): Ok, sorry, I didn't understand you correctly.
If you want to call Array#clear or Array#new only once a second or something nobody will ever notice the difference between them.
There is no "best" way to code, as kyonides said, but they do something different.
Rather than explaining that I will just give an example:

foo = []

bar = foo

 

foo << 1

foo << 2

foo << 3

 

p foo # [1, 2, 3]

p bar # [1, 2, 3]

 

foo.clear

foo << 1

 

p foo # [1]

p bar # [1]

foo = []

bar = foo

 

foo << 1

foo << 2

foo << 3

 

p foo # [1, 2, 3]

p bar # [1, 2, 3]

 

foo = []

foo << 1

 

p foo # [1]

p bar # [1, 2, 3]

To answer your question, what is better to use: you have to decide that for your particular case.
If we use you example with the Array of targets: an Array like this would be created / filled with new targets after each round in the battle so for this case I would use Array.new
If you want to clear e.g. $game_party.actors you should better use Array#clear since other Objects may have a reference to that Array.

 

[DeM0nFiRe]

Neo-Bahamut, I think you still don't understand the difference between wrapping code as a library and exposing code to a scripting language. What you showed is how you wrap code as a library, that's the way the array, hash, string etc. are all done. While it's possible that enterbrain did it that way, it's not likely because it would be really stupid, because you sacrifice a huge amount of speed. (Believe it or not, RMXP is probably too fast to have been 1:1 wrapped as a library)

Imstead, RMXP is probably just exposed to a certain degree to the scripting language. In fact, there probably is no Bitmap object on the C side at all. Instead, there is probably only an array of the bitmap data, or maybe a GDI image object.

Check out this page and scroll down to "Writing Ruby in C" http://ruby-doc.org/docs/ProgrammingRub ... _ruby.html

Notice how as they create the Test class in C, there is no C Test class, just an assortment of functions? Well, you can imagine how that assortment of functions could expose a GDI image to the script, and that assortment of functions could manipulate that GDI image, and you can also imagine how in the C side you have to explicitly manage the memory for that GDI image, because it's still entirely C. This becomes more likely because Ruby's destructors are nor actually destructors! Ruby does not have a destructor at all! When you delete a Ruby object, Ruby automatically checks to see what else should be deleted with it, because Ruby can easily count for itself how many references to an object there are.

Instead of destructors, Ruby has something called a finalizer which is a function that runs when the object is being destroyed. HOWEVER the behavior of that function is not guaranteed. For instance, say you have an array of GDI images on the C/C++ side and then when you are writing the Ruby class in C, you give the Ruby object an integer value that is an index to that GDI image. You would think that in your finalizer, you just remove the GDI image right? Well, unfortunately, Ruby does not guarantee when the finalizer will happen, and it can happen AFTER the object is destroyed, meaning that variable where you stored the index could be gone, so by the time you want to find that object's GDI image you can't!

There are obviously ways around it, but since Enterbrain explicitly put a dispose function, that tells me that they most likely decided not to try to find a way around it. Instead, they want you to dispose a bitmap when you are done with that bitmap. Note that to avoid permanent memory leaks, when the program exits, they probably clear up all memory on the C side because it's easy to do that.

 

[Neo-Bahamut]

Neo-Bahamut, I think you still don't understand the difference between wrapping code as a library and exposing code to a scripting language.

No, I don't. But I never cared about how C extensions are libraries are written before so state of non-understanding may stay for a while :P

Notice how as they create the Test class in C, there is no C Test class, just an assortment of functions? Well, you can imagine how that assortment of functions could expose a GDI image to the script, and that assortment of functions could manipulate that GDI image, and you can also imagine how in the C side you have to explicitly manage the memory for that GDI image, because it's still entirely C. This becomes more likely because Ruby's destructors are nor actually destructors! Ruby does not have a destructor at all! When you delete a Ruby object, Ruby automatically checks to see what else should be deleted with it, because Ruby can easily count for itself how many references to an object there are.

I get that point. Seems like I had a wrong sight of how those "hidden" classes work.

But still, I tried some code like this:

p 'Check task manager -> Game.exe RAM usage'

 

10000.times {Sprite.new}

GC.start

 

p 'Check again'

It did not leave any memory rests.

By the way, the user Kai told me some interesting thing which could also be the problem.
Some methods like Sprite#src_rect or Bitmap#font return a Rect or Font. It looks like they are usual Rects/Fonts which you created with Rect.new(100, 100, 100, 100), you know?

list = []

2000.times do

 list.push(Bitmap.new(100, 100).font))

end

GC.start

After this code, you may expect only a small increase on RAM since all the Font class stores is two Booleans, five Integers and one String ( = usually less than 50 bytes).
But the font for some reason keeps a reference to its Bitmap which causes that the GC doesn't clears it from memory.

 

[DeM0nFiRe]

Font needs a bitmap to draw on, that's why. Also, sprite.new doesn't create a bitmap. The sprite is probably a ruby only object, while the bitmap is probably C/C++. Try the test again with 1000.times (bitmap.new)

 

[Neo-Bahamut]

Font needs a bitmap to draw on, that's why.

No, a Bitmap needs a Font so something can be drawn on it
A font is nothing more than a struct with name, color, size, bold and italic.

The sprite is probably a ruby only object, while the bitmap is probably C/C++. Try the test again with 1000.times (bitmap.new)

I did, same result.

 

[DeM0nFiRe]

First of all, you misunderstood what I meant by "font needs a bitmap to draw on" I meant that a font is useless without a bitmap, since only a bitmap can use the font.

Second of all, you did the test wrong. When you create the bitmap, you have to give it a width and height, otherwise it creates a 0 x 0 bitmap, which is going to have no pixel data to store. I just did the test with

10000.times{bitmap.new(100,100)}

And it leaked about 300 MB of memory. 10000 (bitmaps) * 100 (width) * 100(height) * 3(channels, RGB) is 300MB (Actually, it's a bit less than 300MB)

 

[Neo-Bahamut]

First of all, you misunderstood what I meant by "font needs a bitmap to draw on" I meant that a font is useless without a bitmap, since only a bitmap can use the font.

Ok, but that's not a reason to give the font a reference to a Bitmap

Second of all, you did the test wrong. When you create the bitmap, you have to give it a width and height, otherwise it creates a 0 x 0 bitmap, which is going to have no pixel data to store.

No, I did it right. Don't worry, I know enough about RGSS so I can create a Bitmap without getting errors :P
I did it with 5000 (10000 crashes the game :/) and the increase of RAM is ~700 KB. This may be some memory leak, I don't know, but that would be 140 bytes each Bitmap so it doesn't really matter
I think you didn't call GC.start before checking the RAM again and that is why your process needed 300 MB.

3(channels, RGB)

4 channels, RGBA (*smart ass*