Most bizarre bug ever...

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Most bizarre bug ever...

Larry Meadors
OK, I can't reproduce this in a test...but I can tell you what happened today.

I'm using mybatis and apache camel with jms with global transactions
(atomikos) - so if a database update fails the jms messages roll back,
too.

Working really well.

...until we added caching.

We turned on caching in 3 sqlmaps and all of a sudden, our datasource
just went crazy and started using up all the connections.

Turning off caching made it work again.

:-/

Too tired to look at the code on this one now, but sharing in case
someone else has a similar issue. It took us 3-4 hours to find this
one, and it worked in our development and test environments, so we got
to find it in production. No pressure. :-P

mybatis.version = 3.0.5
mybatis-guice.version = 3.0
camel.version = 2.8.0
activemq-core.version = 5.5.0
atomikos.version = 3.6.6

Larry
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Eduardo Macarron
O_o

No idea but in 3.0.x cached selects use connections, maybe a throwable
is arising from the cache (OutOfMemory?) and connections are not being
closed.


On 9 feb, 05:21, Larry Meadors <[hidden email]> wrote:

> OK, I can't reproduce this in a test...but I can tell you what happened today.
>
> I'm using mybatis and apache camel with jms with global transactions
> (atomikos) - so if a database update fails the jms messages roll back,
> too.
>
> Working really well.
>
> ...until we added caching.
>
> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
> just went crazy and started using up all the connections.
>
> Turning off caching made it work again.
>
> :-/
>
> Too tired to look at the code on this one now, but sharing in case
> someone else has a similar issue. It took us 3-4 hours to find this
> one, and it worked in our development and test environments, so we got
> to find it in production. No pressure. :-P
>
> mybatis.version = 3.0.5
> mybatis-guice.version = 3.0
> camel.version = 2.8.0
> activemq-core.version = 5.5.0
> atomikos.version = 3.6.6
>
> Larry
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Thijs
Just a side note. Why is this?
Why does it need a connection to cache things?

I'm running queries that have a high cache hit rate and that are called
a lot of times (some batch process). Getting the connection is taking so
much time, in comparison to the rest of the processing, that I've
completely disabled the myBatis (we still use 2.3.5) caching in some
cases and used a custom ehCache in front that doesn't rely on the
connection to get to the cache.
We do use a connection pool (BoneCP). So it doesn't have to 'create' the
connection every time. Yet even then it's slow.

Thijs


On 9-2-2012 8:03, Eduardo wrote:

> O_o
>
> No idea but in 3.0.x cached selects use connections, maybe a throwable
> is arising from the cache (OutOfMemory?) and connections are not being
> closed.
>
>
> On 9 feb, 05:21, Larry Meadors<[hidden email]>  wrote:
>> OK, I can't reproduce this in a test...but I can tell you what happened today.
>>
>> I'm using mybatis and apache camel with jms with global transactions
>> (atomikos) - so if a database update fails the jms messages roll back,
>> too.
>>
>> Working really well.
>>
>> ...until we added caching.
>>
>> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
>> just went crazy and started using up all the connections.
>>
>> Turning off caching made it work again.
>>
>> :-/
>>
>> Too tired to look at the code on this one now, but sharing in case
>> someone else has a similar issue. It took us 3-4 hours to find this
>> one, and it worked in our development and test environments, so we got
>> to find it in production. No pressure. :-P
>>
>> mybatis.version = 3.0.5
>> mybatis-guice.version = 3.0
>> camel.version = 2.8.0
>> activemq-core.version = 5.5.0
>> atomikos.version = 3.6.6
>>
>> Larry

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Larry Meadors
In reply to this post by Eduardo Macarron

Does 3.1.x do it differently?

I can test it real easily. :-)

Larry

On Feb 9, 2012 12:03 AM, "Eduardo" <[hidden email]> wrote:
O_o

No idea but in 3.0.x cached selects use connections, maybe a throwable
is arising from the cache (OutOfMemory?) and connections are not being
closed.


On 9 feb, 05:21, Larry Meadors <[hidden email]> wrote:
> OK, I can't reproduce this in a test...but I can tell you what happened today.
>
> I'm using mybatis and apache camel with jms with global transactions
> (atomikos) - so if a database update fails the jms messages roll back,
> too.
>
> Working really well.
>
> ...until we added caching.
>
> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
> just went crazy and started using up all the connections.
>
> Turning off caching made it work again.
>
> :-/
>
> Too tired to look at the code on this one now, but sharing in case
> someone else has a similar issue. It took us 3-4 hours to find this
> one, and it worked in our development and test environments, so we got
> to find it in production. No pressure. :-P
>
> mybatis.version = 3.0.5
> mybatis-guice.version = 3.0
> camel.version = 2.8.0
> activemq-core.version = 5.5.0
> atomikos.version = 3.6.6
>
> Larry
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Eduardo Macarron
@Larry

Yes, that was fixed.

@Thijs

3.1.0 would be highly recommended in your case

On 9 feb, 14:21, Larry Meadors <[hidden email]> wrote:

> Does 3.1.x do it differently?
>
> I can test it real easily. :-)
>
> Larry
> On Feb 9, 2012 12:03 AM, "Eduardo" <[hidden email]> wrote:
>
>
>
>
>
>
>
> > O_o
>
> > No idea but in 3.0.x cached selects use connections, maybe a throwable
> > is arising from the cache (OutOfMemory?) and connections are not being
> > closed.
>
> > On 9 feb, 05:21, Larry Meadors <[hidden email]> wrote:
> > > OK, I can't reproduce this in a test...but I can tell you what happened
> > today.
>
> > > I'm using mybatis and apache camel with jms with global transactions
> > > (atomikos) - so if a database update fails the jms messages roll back,
> > > too.
>
> > > Working really well.
>
> > > ...until we added caching.
>
> > > We turned on caching in 3 sqlmaps and all of a sudden, our datasource
> > > just went crazy and started using up all the connections.
>
> > > Turning off caching made it work again.
>
> > > :-/
>
> > > Too tired to look at the code on this one now, but sharing in case
> > > someone else has a similar issue. It took us 3-4 hours to find this
> > > one, and it worked in our development and test environments, so we got
> > > to find it in production. No pressure. :-P
>
> > > mybatis.version = 3.0.5
> > > mybatis-guice.version = 3.0
> > > camel.version = 2.8.0
> > > activemq-core.version = 5.5.0
> > > atomikos.version = 3.6.6
>
> > > Larry
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Thijs
Good to hear.

Just not looking forward to doing the upgrade with close to a thousand
mapping files.
It's planned in near future...




On 9-2-2012 14:23, Eduardo wrote:

> @Larry
>
> Yes, that was fixed.
>
> @Thijs
>
> 3.1.0 would be highly recommended in your case
>
> On 9 feb, 14:21, Larry Meadors<[hidden email]>  wrote:
>> Does 3.1.x do it differently?
>>
>> I can test it real easily. :-)
>>
>> Larry
>> On Feb 9, 2012 12:03 AM, "Eduardo"<[hidden email]>  wrote:
>>
>>
>>
>>
>>
>>
>>
>>> O_o
>>> No idea but in 3.0.x cached selects use connections, maybe a throwable
>>> is arising from the cache (OutOfMemory?) and connections are not being
>>> closed.
>>> On 9 feb, 05:21, Larry Meadors<[hidden email]>  wrote:
>>>> OK, I can't reproduce this in a test...but I can tell you what happened
>>> today.
>>>> I'm using mybatis and apache camel with jms with global transactions
>>>> (atomikos) - so if a database update fails the jms messages roll back,
>>>> too.
>>>> Working really well.
>>>> ...until we added caching.
>>>> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
>>>> just went crazy and started using up all the connections.
>>>> Turning off caching made it work again.
>>>> :-/
>>>> Too tired to look at the code on this one now, but sharing in case
>>>> someone else has a similar issue. It took us 3-4 hours to find this
>>>> one, and it worked in our development and test environments, so we got
>>>> to find it in production. No pressure. :-P
>>>> mybatis.version = 3.0.5
>>>> mybatis-guice.version = 3.0
>>>> camel.version = 2.8.0
>>>> activemq-core.version = 5.5.0
>>>> atomikos.version = 3.6.6
>>>> Larry

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Larry Meadors
In reply to this post by Eduardo Macarron
On Thu, Feb 9, 2012 at 6:23 AM, Eduardo <[hidden email]> wrote:
> @Larry
>
> Yes, that was fixed.

Thanks Edu, I'll try it today and see if it passes all the tests in
the project (including one that consistently fails with this
scenario).

The problem with this particular test is that it's using everything
under the sun (jms, atomikos, oracle xa data source, caching, xsl,
camel...), so I can't share it.

Larry
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Clinton Begin
Administrator
In reply to this post by Thijs
When you do your upgrade, consider some options....

* Write a converter (there are some partially finished out there somewhere)
* Write an adapter to allow iBATIS 2.x code to work against 3.x with minimal changes
* Both or a hybrid of the two... 

iBATIS 2.x had this kind of backward compatibility with 1.x.  I believe it's possible to create a 2x -> 3x adapter/converter set to deal with 80 - 90% of cases.  I just never had the time to get to it.

The unit test suite has some beginning steps toward this though, as all of the iBATIS 2.x unit tests and JPetStore unit tests were migrated to MyBatis 3.0.  

It comes down to who is most invested in 2.x maps and code to justify the effort to build such an adapter/converter.  :-)

Clinton



On Thu, Feb 9, 2012 at 8:00 AM, Thijs <[hidden email]> wrote:
Good to hear.

Just not looking forward to doing the upgrade with close to a thousand mapping files.
It's planned in near future...





On 9-2-2012 14:23, Eduardo wrote:
@Larry

Yes, that was fixed.

@Thijs

3.1.0 would be highly recommended in your case

On 9 feb, 14:21, Larry Meadors<[hidden email]>  wrote:
Does 3.1.x do it differently?

I can test it real easily. :-)

Larry
On Feb 9, 2012 12:03 AM, "Eduardo"<[hidden email]>  wrote:







O_o
No idea but in 3.0.x cached selects use connections, maybe a throwable
is arising from the cache (OutOfMemory?) and connections are not being
closed.
On 9 feb, 05:21, Larry Meadors<[hidden email]>  wrote:
OK, I can't reproduce this in a test...but I can tell you what happened
today.
I'm using mybatis and apache camel with jms with global transactions
(atomikos) - so if a database update fails the jms messages roll back,
too.
Working really well.
...until we added caching.
We turned on caching in 3 sqlmaps and all of a sudden, our datasource
just went crazy and started using up all the connections.
Turning off caching made it work again.
:-/
Too tired to look at the code on this one now, but sharing in case
someone else has a similar issue. It took us 3-4 hours to find this
one, and it worked in our development and test environments, so we got
to find it in production. No pressure. :-P
mybatis.version = 3.0.5
mybatis-guice.version = 3.0
camel.version = 2.8.0
activemq-core.version = 5.5.0
atomikos.version = 3.6.6
Larry


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Most bizarre bug ever...

Eduardo Macarron
Have a look at this page contributed by Herman Bovens and  Peter
Köhler (code). It may help a lot

http://code.google.com/p/mybatis/wiki/DocUpgrade3
Loading...