Possible Memory Leak

[tobiasflohre]

We did some performance tests and stumbled over a memory leak. If you take a look at the following file, you see that there are Thymeleaf objects held by a Groovy map, and there's no other Groovy in our application than the layout dialect.
Leak Suspects.pdf

I've looked into your code and have one assumption: There's that static FragmentMap which is filled for each IContext object, and I don't see any point in your code where you delete entries from that Map. It's just getting bigger and bigger.
Our 4GB heap application doesn't respond anymore after a number of requests that's not very high (like 10000 or less), so it's a very serious thing for us.

We can provide you with a heap dump, if you want.

[zhanhb]

Maybe ChildEventIterator line 70, ChildModelIterator line 68,70, IModelExtensions line 180,203,204 was to blame. These codes will create ExpandoMetaClass instance.

[tobiasflohre]

@ultraq Hm, when I look into the code of FragmentMap again: you just add the FragmentMap to each IContext, so if the IContext object is garbage collected like it should, there should be no problem. However, the problem exists - tell us if we can help identifying the root cause. There must be some Groovy map holding objects too long.

[ultraq]

Thanks for the analysis @tobiasflohre! I've got a YourKit license lying around here somewhere so it should show me the same things you've discovered once I turn it back on. I'll keep a look out for those objects you've identified in particular.

[ultraq]

Hey @tobiasflohre, that heap dump might be a good idea, although I'm not sure how to transfer something of that size around. Ideas?

I'm currently trying to generate a lot of load on a test site, so it might be a while before I can see things at the same magnitude as in the PDF you've shared.

[tobiasflohre]

You got mail!

[zhanhb]

Groovy is slower than java and takes more memory consumption. Add annotation CompileStatic or compiled with --indy can make that better.

[ultraq]

Thanks for the suggestions as well @zhanhb

@tobiasflohre, I've managed to simulate load on 2 projects (1 Groovy, 1 Java) but they both seem to clean-up very well. Even when I pick smaller heap sizes and squeeze the wiggle room for the memory allocations to work with, I end up with a memory profile that tends towards a large old generation, a series of predictable peaks of young generation objects being created and GC'ed between the heap limit and the old gen size, but never run out of space, eg:

The ExpandoMetaClass object does show up a lot just like in your initial analysis PDF, and that is a consequence of the meta-programming feature of Groovy I use quite a lot to extend Thymeleaf's standard objects with convenience methods and other things that, frankly, make my job easier 😄 Every instance of ExpandoMetaClass was eligible for GC though, so that shouldn't be the cause of the memory leak. It may be something for me to consider for future versions though that I might be over-relying on meta-programming, and it might be good for me to take a hard look at what I need or don't need out of it.

The memory dump you provided though is a bit more telling and, as well as everything I saw from the previous projects, I was able to see instances of layout dialect classes but they were all via weak/soft references. So yay I can see something, but I'm scratching my head as to why they're even in there if they can be GC'ed. The fact that I can't reproduce it either is also annoying me because then I don't know if what I can do will actually fix things.

I noticed that the Tomcat classloader shows up a lot in there, so I should ask: what version of Tomcat and Java are you running? If I can reproduce the leak then I'll be more confident in any fix I make. (My tests were all on Jetty, so that might prevent me from getting things the same as what you see.)

[tobiasflohre]

It's Tomcat 7.0.37 and Java 1.7.0_72. Thanks for the efforts.

[tobiasflohre]

We've looked into this a little bit further, and we also see that each object is held by one weak and one soft reference. The soft reference is only garbage collected when the JVM is running out of memory, and because there's a weak AND a soft reference to the objects, all objects are only garbage collected when the JVM is running out of memory. We saw that memory consumption is going up through the JVM's lifetime, which is 100% explicable with those soft references. Maybe our JVM was not perfectly configured, that could be one reason why it died in the end (when it should have just thrown away all the soft references), but anyway: Using soft references is wrong IMO. The objects held there are held for one request and not used again, they should be garbage collected as soon as possible.
Maybe you see something in the reference stack of the soft reference that explains to you why it is held (thread-bound meta programming stuff?):

For comparison, here the reference stack of the weak reference of the same object.

[ultraq]

Warning, long and rambly paragraph, mainly for my own benefit if I need to come back to this in future...

At first I was first going to try removing some closures that seemed responsible for the soft/weak references we've seen (our tools always showed layout dialect closures that had any kind of reference to a static as somewhere between the GC root and the large objects in memory), but I did some more digging to try find what code was creating those large objects and found that one of the meta-programming methods I added, findModel was being rather wasteful in that it called another meta-method, find (returns a Thymeleaf template event and creates a meta-class for every returned object which Groovy then goes and tracks in that gigantic meta-class cache that our tools are identifying as the single instance holding on to so many OpenElementTag objects), but it didn't even use the template event; only the index value of the model on the meta-class! (

thymeleaf-layout-dialect/Source/nz/net/ultraq/thymeleaf/models/extensions/IModelExtensions.groovy

Line 200 in f18a36d

def event = delegate.find(closure)

) I thought that was rather wasteful so I created a new method, findIndexOf, for returning the index of an event in the model without all of the overhead, changed findModel and some other places in the layout dialect to use that (I found 2 other places making the same wasteful choices), re-ran my load tests and saw some promising improvements.

When comparing load tests between 2.0.0 and a 2.0.1-SNAPSHOT with the above changes:

• the memory impact of findModel has been cut by anywhere from 1/2 to 1/10th, depending on the caller and the way it was used
• the over-time memory profile even allowed for the old generation heap to decrease in size at several points, especially after a forced GC I perform at the end of every test to see only the leaking objects
• 10 million less objects were created
• the time spent on GCs was cut in half
• and it performed 20% less GCs

Here are some telling screenshots:

Before (2.0.0):

After (2.0.1-SNAPSHOT):

@tobiasflohre, would you be able to give the 2.0.1-SNAPSHOT version a test and see if it helps with the memory issues you're seeing at all?