Most bizarre bug ever...

[Larry Meadors]

OK, I can't reproduce this in a test...but I can tell you what happened today.

I'm using mybatis and apache camel with jms with global transactions
(atomikos) - so if a database update fails the jms messages roll back,
too.

Working really well.

...until we added caching.

We turned on caching in 3 sqlmaps and all of a sudden, our datasource
just went crazy and started using up all the connections.

Turning off caching made it work again.

:-/

Too tired to look at the code on this one now, but sharing in case
someone else has a similar issue. It took us 3-4 hours to find this
one, and it worked in our development and test environments, so we got
to find it in production. No pressure. :-P

mybatis.version = 3.0.5
mybatis-guice.version = 3.0
camel.version = 2.8.0
activemq-core.version = 5.5.0
atomikos.version = 3.6.6

Larry

[Eduardo Macarron]

O_o

No idea but in 3.0.x cached selects use connections, maybe a throwable
is arising from the cache (OutOfMemory?) and connections are not being
closed.


On 9 feb, 05:21, Larry Meadors <[hidden email]> wrote:

> OK, I can't reproduce this in a test...but I can tell you what happened today.
>
> I'm using mybatis and apache camel with jms with global transactions
> (atomikos) - so if a database update fails the jms messages roll back,
> too.
>
> Working really well.
>
> ...until we added caching.
>
> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
> just went crazy and started using up all the connections.
>
> Turning off caching made it work again.
>
> :-/
>
> Too tired to look at the code on this one now, but sharing in case
> someone else has a similar issue. It took us 3-4 hours to find this
> one, and it worked in our development and test environments, so we got
> to find it in production. No pressure. :-P
>
> mybatis.version = 3.0.5
> mybatis-guice.version = 3.0
> camel.version = 2.8.0
> activemq-core.version = 5.5.0
> atomikos.version = 3.6.6
>
> Larry

[Thijs]

Just a side note. Why is this?
Why does it need a connection to cache things?

I'm running queries that have a high cache hit rate and that are called
a lot of times (some batch process). Getting the connection is taking so
much time, in comparison to the rest of the processing, that I've
completely disabled the myBatis (we still use 2.3.5) caching in some
cases and used a custom ehCache in front that doesn't rely on the
connection to get to the cache.
We do use a connection pool (BoneCP). So it doesn't have to 'create' the
connection every time. Yet even then it's slow.

Thijs


On 9-2-2012 8:03, Eduardo wrote:

> O_o
>
> No idea but in 3.0.x cached selects use connections, maybe a throwable
> is arising from the cache (OutOfMemory?) and connections are not being
> closed.
>
>
> On 9 feb, 05:21, Larry Meadors<[hidden email]>  wrote:
>> OK, I can't reproduce this in a test...but I can tell you what happened today.
>>
>> I'm using mybatis and apache camel with jms with global transactions
>> (atomikos) - so if a database update fails the jms messages roll back,
>> too.
>>
>> Working really well.
>>
>> ...until we added caching.
>>
>> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
>> just went crazy and started using up all the connections.
>>
>> Turning off caching made it work again.
>>
>> :-/
>>
>> Too tired to look at the code on this one now, but sharing in case
>> someone else has a similar issue. It took us 3-4 hours to find this
>> one, and it worked in our development and test environments, so we got
>> to find it in production. No pressure. :-P
>>
>> mybatis.version = 3.0.5
>> mybatis-guice.version = 3.0
>> camel.version = 2.8.0
>> activemq-core.version = 5.5.0
>> atomikos.version = 3.6.6
>>
>> Larry

[Larry Meadors]

Does 3.1.x do it differently?

I can test it real easily. :-)

Larry

On Feb 9, 2012 12:03 AM, "Eduardo" <[hidden email]> wrote:

O_o

No idea but in 3.0.x cached selects use connections, maybe a throwable
is arising from the cache (OutOfMemory?) and connections are not being
closed.


On 9 feb, 05:21, Larry Meadors <[hidden email]> wrote:
> OK, I can't reproduce this in a test...but I can tell you what happened today.
>
> I'm using mybatis and apache camel with jms with global transactions
> (atomikos) - so if a database update fails the jms messages roll back,
> too.
>
> Working really well.
>
> ...until we added caching.
>
> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
> just went crazy and started using up all the connections.
>
> Turning off caching made it work again.
>
> :-/
>
> Too tired to look at the code on this one now, but sharing in case
> someone else has a similar issue. It took us 3-4 hours to find this
> one, and it worked in our development and test environments, so we got
> to find it in production. No pressure. :-P
>
> mybatis.version = 3.0.5
> mybatis-guice.version = 3.0
> camel.version = 2.8.0
> activemq-core.version = 5.5.0
> atomikos.version = 3.6.6
>
> Larry

[Eduardo Macarron]

@Larry

Yes, that was fixed.

@Thijs

3.1.0 would be highly recommended in your case

On 9 feb, 14:21, Larry Meadors <[hidden email]> wrote:

> Does 3.1.x do it differently?
>
> I can test it real easily. :-)
>
> Larry
> On Feb 9, 2012 12:03 AM, "Eduardo" <[hidden email]> wrote:
>
>
>
>
>
>
>
> > O_o
>
> > No idea but in 3.0.x cached selects use connections, maybe a throwable
> > is arising from the cache (OutOfMemory?) and connections are not being
> > closed.
>
> > On 9 feb, 05:21, Larry Meadors <[hidden email]> wrote:
> > > OK, I can't reproduce this in a test...but I can tell you what happened
> > today.
>
> > > I'm using mybatis and apache camel with jms with global transactions
> > > (atomikos) - so if a database update fails the jms messages roll back,
> > > too.
>
> > > Working really well.
>
> > > ...until we added caching.
>
> > > We turned on caching in 3 sqlmaps and all of a sudden, our datasource
> > > just went crazy and started using up all the connections.
>
> > > Turning off caching made it work again.
>
> > > :-/
>
> > > Too tired to look at the code on this one now, but sharing in case
> > > someone else has a similar issue. It took us 3-4 hours to find this
> > > one, and it worked in our development and test environments, so we got
> > > to find it in production. No pressure. :-P
>
> > > mybatis.version = 3.0.5
> > > mybatis-guice.version = 3.0
> > > camel.version = 2.8.0
> > > activemq-core.version = 5.5.0
> > > atomikos.version = 3.6.6
>
> > > Larry

[Thijs]

Good to hear.

Just not looking forward to doing the upgrade with close to a thousand
mapping files.
It's planned in near future...




On 9-2-2012 14:23, Eduardo wrote:

> @Larry
>
> Yes, that was fixed.
>
> @Thijs
>
> 3.1.0 would be highly recommended in your case
>
> On 9 feb, 14:21, Larry Meadors<[hidden email]>  wrote:
>> Does 3.1.x do it differently?
>>
>> I can test it real easily. :-)
>>
>> Larry
>> On Feb 9, 2012 12:03 AM, "Eduardo"<[hidden email]>  wrote:
>>
>>
>>
>>
>>
>>
>>
>>> O_o
>>> No idea but in 3.0.x cached selects use connections, maybe a throwable
>>> is arising from the cache (OutOfMemory?) and connections are not being
>>> closed.
>>> On 9 feb, 05:21, Larry Meadors<[hidden email]>  wrote:
>>>> OK, I can't reproduce this in a test...but I can tell you what happened
>>> today.
>>>> I'm using mybatis and apache camel with jms with global transactions
>>>> (atomikos) - so if a database update fails the jms messages roll back,
>>>> too.
>>>> Working really well.
>>>> ...until we added caching.
>>>> We turned on caching in 3 sqlmaps and all of a sudden, our datasource
>>>> just went crazy and started using up all the connections.
>>>> Turning off caching made it work again.
>>>> :-/
>>>> Too tired to look at the code on this one now, but sharing in case
>>>> someone else has a similar issue. It took us 3-4 hours to find this
>>>> one, and it worked in our development and test environments, so we got
>>>> to find it in production. No pressure. :-P
>>>> mybatis.version = 3.0.5
>>>> mybatis-guice.version = 3.0
>>>> camel.version = 2.8.0
>>>> activemq-core.version = 5.5.0
>>>> atomikos.version = 3.6.6
>>>> Larry

[Larry Meadors]

On Thu, Feb 9, 2012 at 6:23 AM, Eduardo <[hidden email]> wrote:
> @Larry
>
> Yes, that was fixed.

Thanks Edu, I'll try it today and see if it passes all the tests in
the project (including one that consistently fails with this
scenario).

The problem with this particular test is that it's using everything
under the sun (jms, atomikos, oracle xa data source, caching, xsl,
camel...), so I can't share it.

Larry

[Clinton Begin]

When you do your upgrade, consider some options....

* Write a converter (there are some partially finished out there somewhere)

* Write an adapter to allow iBATIS 2.x code to work against 3.x with minimal changes

* Both or a hybrid of the two... 

iBATIS 2.x had this kind of backward compatibility with 1.x.  I believe it's possible to create a 2x -> 3x adapter/converter set to deal with 80 - 90% of cases.  I just never had the time to get to it.

The unit test suite has some beginning steps toward this though, as all of the iBATIS 2.x unit tests and JPetStore unit tests were migrated to MyBatis 3.0.  

It comes down to who is most invested in 2.x maps and code to justify the effort to build such an adapter/converter.  :-)

Clinton

On Thu, Feb 9, 2012 at 8:00 AM, Thijs <[hidden email]> wrote:

Good to hear.

Just not looking forward to doing the upgrade with close to a thousand mapping files.
It's planned in near future...

On 9-2-2012 14:23, Eduardo wrote:

@Larry

Yes, that was fixed.

@Thijs

3.1.0 would be highly recommended in your case

On 9 feb, 14:21, Larry Meadors<[hidden email]>  wrote:

Does 3.1.x do it differently?

I can test it real easily. :-)

Larry
On Feb 9, 2012 12:03 AM, "Eduardo"<[hidden email]>  wrote:

O_o
No idea but in 3.0.x cached selects use connections, maybe a throwable
is arising from the cache (OutOfMemory?) and connections are not being
closed.
On 9 feb, 05:21, Larry Meadors<[hidden email]>  wrote:

OK, I can't reproduce this in a test...but I can tell you what happened

today.

I'm using mybatis and apache camel with jms with global transactions
(atomikos) - so if a database update fails the jms messages roll back,
too.
Working really well.
...until we added caching.
We turned on caching in 3 sqlmaps and all of a sudden, our datasource
just went crazy and started using up all the connections.
Turning off caching made it work again.
:-/
Too tired to look at the code on this one now, but sharing in case
someone else has a similar issue. It took us 3-4 hours to find this
one, and it worked in our development and test environments, so we got
to find it in production. No pressure. :-P
mybatis.version = 3.0.5
mybatis-guice.version = 3.0
camel.version = 2.8.0
activemq-core.version = 5.5.0
atomikos.version = 3.6.6
Larry

[Eduardo Macarron]

Have a look at this page contributed by Herman Bovens and  Peter
Köhler (code). It may help a lot

http://code.google.com/p/mybatis/wiki/DocUpgrade3