Memory leaks while using

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory leaks while using

Nate Weiss
Hi--

Been working with MyBatis on a project for about 6 months and love it.
However, and maybe this is really a testament to the strength of the
framework, I am still somewhat of a noob when it comes to handling
connections from connection pools in Java environments. So please be
easy on me... :)

This app runs in a servlet context. What I've noticed in my app is
that I appear to leak memory when using any type of pooled connection
with MyBatis (3.02). Whether I'm using the built-in "POOLED" type of
datasource, or a JNDI datasource that uses pooling (I've tried under
vanilla Tomcat 6, vanilla Tomcat 7, and vanilla Glassfish 3),

Having read the iBatis user guide, especially the "lifecycle" part, I
*believe* I am doing the right thing... basically I instantiate a
factory builder and throw it away, instantiate a session factory and
*dont* throw it away, and use it to instantiate new SqlSessions for
each servlet request. At the end of each request, I call close() on
the session. I have confirmed that close() is actualy being called for
each connection. I do *not* explicitly close the underlying
connection.

To test, I have created a few different <environments> in my mybatis
configuration file (I use the xml method). Then I run some test
classes for my app and observe memory usage using eclipse profiling.
For testing purposes I then trigger a System.gc() in my app and see
whether connections (and JDBC4PreparedStatements, and database
metadata classes, and associated byte arrays, bit arrays, etc etc) are
able to be collected.

What I've found is that when I use an environment that uses
type="UNPOOLED", I see my resources being freed up after the gc()
rather cleanly, and there is nothing in memory that I wouldn't really
expect. But if I use any type of connection pool, then my resources
are never freed (the number of certain JDBC-related objects that are
alive now always exactly equals the number of that have been spawned
since startup).

I know I'm not giving all the details, basically I am writing to
confirm whether I am using the classes as intended, at least at the
highest level. Any hints as to whether I should investigate further on
the mybatis side, or think about it more abstractly as a general jdbc
usage issue? Should one currently avoid pooling w/ MyBatis as a
practical matter? Anything you would try in particular?

I would like to return to pooling if possible, because it saves enough
ms per request to have an impact.

Thanks in advance for any thoughts or advice.

nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Raj Nagappan
Hi Nate,

What exactly do you mean by leaking memory? If you mean that after a
profiling test the connections are still being held open, well this is
the basic idea of connection pooling - hold a set of open connections
ready to use so you don't need to acquire a new connection each time.

If you mean that you're getting a memory leak over an extended
timeframe this is a different question. I have experienced the
situation where the connection pool "dried up" and the number of
available connections dwindled down to zero over time. To fix this I
had to:
1. Adjust the idle time settings on both the JDBC side and the
database side. Basically you want JDBC to evict the idle connection
before the database does.
2. Change my connection pool to C3P0 as this was the most stable for
my purposes (web server).

Hope this helps,
Raj.

On Sep 30, 12:43 am, Nate Weiss <[hidden email]> wrote:

> Hi--
>
> Been working with MyBatis on a project for about 6 months and love it.
> However, and maybe this is really a testament to the strength of the
> framework, I am still somewhat of a noob when it comes to handling
> connections from connection pools in Java environments. So please be
> easy on me... :)
>
> This app runs in a servlet context. What I've noticed in my app is
> that I appear to leak memory when using any type of pooled connection
> with MyBatis (3.02). Whether I'm using the built-in "POOLED" type of
> datasource, or a JNDI datasource that uses pooling (I've tried under
> vanilla Tomcat 6, vanilla Tomcat 7, and vanilla Glassfish 3),
>
> Having read the iBatis user guide, especially the "lifecycle" part, I
> *believe* I am doing the right thing... basically I instantiate a
> factory builder and throw it away, instantiate a session factory and
> *dont* throw it away, and use it to instantiate new SqlSessions for
> each servlet request. At the end of each request, I call close() on
> the session. I have confirmed that close() is actualy being called for
> each connection. I do *not* explicitly close the underlying
> connection.
>
> To test, I have created a few different <environments> in my mybatis
> configuration file (I use the xml method). Then I run some test
> classes for my app and observe memory usage using eclipse profiling.
> For testing purposes I then trigger a System.gc() in my app and see
> whether connections (and JDBC4PreparedStatements, and database
> metadata classes, and associated byte arrays, bit arrays, etc etc) are
> able to be collected.
>
> What I've found is that when I use an environment that uses
> type="UNPOOLED", I see my resources being freed up after the gc()
> rather cleanly, and there is nothing in memory that I wouldn't really
> expect. But if I use any type of connection pool, then my resources
> are never freed (the number of certain JDBC-related objects that are
> alive now always exactly equals the number of that have been spawned
> since startup).
>
> I know I'm not giving all the details, basically I am writing to
> confirm whether I am using the classes as intended, at least at the
> highest level. Any hints as to whether I should investigate further on
> the mybatis side, or think about it more abstractly as a general jdbc
> usage issue? Should one currently avoid pooling w/ MyBatis as a
> practical matter? Anything you would try in particular?
>
> I would like to return to pooling if possible, because it saves enough
> ms per request to have an impact.
>
> Thanks in advance for any thoughts or advice.
>
> nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Nate Weiss
Hi Raj...

Thanks for the reply. Yes, I understand that the connections would
still be hanging around, since yes that's the purpose of the pool in
the first place. What I mean by leaking memory is that I see instances
of jdbc result sets, callable statements, and especially byte arrays
which all trace back to jdbc objects when I profile the app. All of
these (aside from the connections themselves and perhaps any prepared
statements) should, as far as I understand the suggested usage, be
elibible for garbage collection after I close the SqlSession that
spawned them.

But instead memory usage grows with each new db interaction, and is
never regained by gc. From reading up elsewhere it seems likely that
for some reason the underlying jdbc result sets etc are not being
closed when I coose the sessions. It doesn't seeem to me that I am
meant to close them myself... from looking at source and the docs it
seems that mybatis is meant to do that for me.

I will look into the connection timeouts as you mention, thanks...
though I am concerned to rely on idle timeouts because it is possible
that the app will be using all of the connections in the pool
relatively often. In any case I will also look intoC3P0.

Any other input from you or anyone else greatly appreciated.

Nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Matt Passell
Hi Nate,

With regard to connection pools, I've been using Tomcat jdbc-pool (http://people.apache.org/~fhanik/jdbc-pool/ ) in a product as a replacement for C3P0 and have been very happy with it.  It has a rather informal-looking URL, but is worked on by several of the core Tomcat committers and is shipped with SpringSource's commercial Tomcat-based server.  C3P0 is decent, but not very actively maintained.  I occasionally ran into deadlocks with it and wasn't very reassured with the suggested workaround, "just allocate more threads to the pool", which feels like putting on a bandaid instead of actually fixing the problem.

--Matt


On Wed, Sep 29, 2010 at 10:17 PM, Nate Weiss <[hidden email]> wrote:
Hi Raj...

Thanks for the reply. Yes, I understand that the connections would
still be hanging around, since yes that's the purpose of the pool in
the first place. What I mean by leaking memory is that I see instances
of jdbc result sets, callable statements, and especially byte arrays
which all trace back to jdbc objects when I profile the app. All of
these (aside from the connections themselves and perhaps any prepared
statements) should, as far as I understand the suggested usage, be
elibible for garbage collection after I close the SqlSession that
spawned them.

But instead memory usage grows with each new db interaction, and is
never regained by gc. From reading up elsewhere it seems likely that
for some reason the underlying jdbc result sets etc are not being
closed when I coose the sessions. It doesn't seeem to me that I am
meant to close them myself... from looking at source and the docs it
seems that mybatis is meant to do that for me.

I will look into the connection timeouts as you mention, thanks...
though I am concerned to rely on idle timeouts because it is possible
that the app will be using all of the connections in the pool
relatively often. In any case I will also look intoC3P0.

Any other input from you or anyone else greatly appreciated.

Nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Nate Weiss
Cool Matt, thanks for the tip. Do you use this with Tomcat 7?

nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Matt Passell
You're welcome.  No, not yet.  We're using it with Tomcat 6.0.x, but it's a completely self contained library, so you should be able to use it in any Java app.  If you want to read a bit more about it, Filip Hanik has a few articles about it on TomcatExpert - http://www.tomcatexpert.com/tags/connection-pool

--Matt


On Wed, Sep 29, 2010 at 11:08 PM, Nate Weiss <[hidden email]> wrote:
Cool Matt, thanks for the tip. Do you use this with Tomcat 7?

nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Nate Weiss
Thanks Matt, currently testing the Tomcat jdbc-pool. Was easy to
switch to, and I like the additional configuration parameters.

So far this does seem to improve things, I can put a "maxAge" on the
connections and just watch them die in the memory profiler. When the
connections die the associated resultsets and whatnot do seem to get
cleaned up as well.

Still doesn't seem like (to me, but I'm not that knowledgeable in
these matters, for real) the right behavior from the mybatis side, as
I would expect that resultsets would be eligible for GC as sessions
close... not only when connections close. Not sure if I am
misunderstanding what is supposed to happen or not. In any case this
new pool does seem to help (if only, perhaps, because of the maxAge
setting which lets the pool discard connections even if they aren't
going idle). Question mark question mark.

Anyway thanks again for your help and suggestion, I'll post more if I
find anything else additional in case it helps anyone else.

nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Clinton Begin
Administrator
In reply to this post by Nate Weiss
That's very odd.  MyBatis internally is very strict about closing all resources explicitly (unless of course I missed one -- which would be a bug). 

It's possible that the driver is not actually releasing resources until the connection itself is truly closed... which would be a driver bug.

Can anyone else confirm if they see this behavior?

Cheers,
Clinton



On Wed, Sep 29, 2010 at 8:17 PM, Nate Weiss <[hidden email]> wrote:
Hi Raj...

Thanks for the reply. Yes, I understand that the connections would
still be hanging around, since yes that's the purpose of the pool in
the first place. What I mean by leaking memory is that I see instances
of jdbc result sets, callable statements, and especially byte arrays
which all trace back to jdbc objects when I profile the app. All of
these (aside from the connections themselves and perhaps any prepared
statements) should, as far as I understand the suggested usage, be
elibible for garbage collection after I close the SqlSession that
spawned them.

But instead memory usage grows with each new db interaction, and is
never regained by gc. From reading up elsewhere it seems likely that
for some reason the underlying jdbc result sets etc are not being
closed when I coose the sessions. It doesn't seeem to me that I am
meant to close them myself... from looking at source and the docs it
seems that mybatis is meant to do that for me.

I will look into the connection timeouts as you mention, thanks...
though I am concerned to rely on idle timeouts because it is possible
that the app will be using all of the connections in the pool
relatively often. In any case I will also look intoC3P0.

Any other input from you or anyone else greatly appreciated.

Nate

Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Matt Passell
In reply to this post by Nate Weiss
Hi Nate,

I'm glad that jdbc-pool is working well for you so far.  As Clinton said in his recent reply, I'm surprised that you're seeing problems on the MyBatis end, aside from whether the connection pool is helping out with cleanup.  Looking over your notes in this thread, I don't see which DB/driver you're using.  Clinton's theory about it being a problem with the driver seems like a plausible one.

--Matt


On Thu, Sep 30, 2010 at 1:04 AM, Nate Weiss <[hidden email]> wrote:
Thanks Matt, currently testing the Tomcat jdbc-pool. Was easy to
switch to, and I like the additional configuration parameters.

So far this does seem to improve things, I can put a "maxAge" on the
connections and just watch them die in the memory profiler. When the
connections die the associated resultsets and whatnot do seem to get
cleaned up as well.

Still doesn't seem like (to me, but I'm not that knowledgeable in
these matters, for real) the right behavior from the mybatis side, as
I would expect that resultsets would be eligible for GC as sessions
close... not only when connections close. Not sure if I am
misunderstanding what is supposed to happen or not. In any case this
new pool does seem to help (if only, perhaps, because of the maxAge
setting which lets the pool discard connections even if they aren't
going idle). Question mark question mark.

Anyway thanks again for your help and suggestion, I'll post more if I
find anything else additional in case it helps anyone else.

nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Nate Weiss
In reply to this post by Clinton Begin
Hi Clinton... yes I was touring around the source code and it does
certainly seem that the code really tries to close everything. So
maybe there is a bug in the driver, I supposd that's possible. I am
using the latest mysql connector/j driver which is 5.1.13. I am using
autocommit if you feel that is relevant or brings its own gotchas in
terms of releasing object references. I see the same behavior on
windows or ubuntu, which I would expect.

What puzzles me is that it appears that resources such as result sets
appear to share the lifecycle of the connection, rather that the
session. I wish I unsderstood what kind of app practice might cause
that symptom. No offense to either of us but I suspect my code first
by a long shot, then mybatis, then the driver (just because i am not
pushing this very hard at all so I figure this would be be widely
noticed by any mysql user using any pooling at all).

Thanks again and any hints appreciated.
Nate
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

François Schiettecatte
Can you tell us which driver you are using, I can set up a long running test here, I am running the latest mysql connector/j driver as well and C3P0 and have had no issues, even with apps running for days at a time.

François

On Oct 1, 2010, at 9:09 AM, Nate Weiss wrote:

> Hi Clinton... yes I was touring around the source code and it does
> certainly seem that the code really tries to close everything. So
> maybe there is a bug in the driver, I supposd that's possible. I am
> using the latest mysql connector/j driver which is 5.1.13. I am using
> autocommit if you feel that is relevant or brings its own gotchas in
> terms of releasing object references. I see the same behavior on
> windows or ubuntu, which I would expect.
>
> What puzzles me is that it appears that resources such as result sets
> appear to share the lifecycle of the connection, rather that the
> session. I wish I unsderstood what kind of app practice might cause
> that symptom. No offense to either of us but I suspect my code first
> by a long shot, then mybatis, then the driver (just because i am not
> pushing this very hard at all so I figure this would be be widely
> noticed by any mysql user using any pooling at all).
>
> Thanks again and any hints appreciated.
> Nate

Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Clinton Begin
Administrator
In reply to this post by Nate Weiss
The later JDBC specs do have an interesting characteristic in that a new part of the spec specifically states that connections, statements and resultsets should close anything they create.  Thus connections should close their statements, and statements should close their resultsets.  This was added because so many applications weren't closing JDBC resources correctly (it's actually quite a lot of code to do correctly).

The downside of this is that it means that the connection has to keep a reference to all of its statements, and the statements keep a reference to their resultsets.  Furthermore, a reverse relationship must also be maintained to ensure that if I close the resultset or statement explicitly (which I should anyway), it is "de-registered" from the statement or connection respectively.

What this means is that the connection could easily be keeping references to old resultsets for the life of the connection -- which in a connection pool is quite some time, or possibly the same as the application lifecycle. 

You might want to grab the MySQL driver source and step through the JDBC resource lifecycle, to see if things are being closed --  AND to ensure that the driver is releasing the resources.

You would only need to step through a single instance to find this, no load testing or concurrency testing would be required to prove this hypothesis. 

Cheers,
Clinton

On Fri, Oct 1, 2010 at 7:09 AM, Nate Weiss <[hidden email]> wrote:
Hi Clinton... yes I was touring around the source code and it does
certainly seem that the code really tries to close everything. So
maybe there is a bug in the driver, I supposd that's possible. I am
using the latest mysql connector/j driver which is 5.1.13. I am using
autocommit if you feel that is relevant or brings its own gotchas in
terms of releasing object references. I see the same behavior on
windows or ubuntu, which I would expect.

What puzzles me is that it appears that resources such as result sets
appear to share the lifecycle of the connection, rather that the
session. I wish I unsderstood what kind of app practice might cause
that symptom. No offense to either of us but I suspect my code first
by a long shot, then mybatis, then the driver (just because i am not
pushing this very hard at all so I figure this would be be widely
noticed by any mysql user using any pooling at all).

Thanks again and any hints appreciated.
Nate

Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks while using

Nate Weiss
Hi Clinton--

Thanks for that, I will do as you suggest (step through driver source)
as soon as I have a chance. What you describe seems to contradict what
Francoise experiences (one post above yours), which perhaps means that
the driver is not implementing the policy you mentioned, not sure. Or
perhaps there is some kind of driver-level setting which controls this
behavior. Will try to look into it. At least now I know what I should
be expecting from the various layers... even if it means, in the end,
that I should just make sure that the connection pool closes its
connections relatively frequently (which is still better than opening/
closing w/ every servlet request).

Thanks, nate