Long running open connections badly leak memory

Hi,
I wrote a batch import process that uses connection pooling, i.e. the
connections are never closed until the process is finished. I am writing
large chunks of data (byte[]) into the database, and I made absolutely
sure that my application does not hold any references to the byte[]
objects. However, the process increasingly consumes memory until an
OutOfMemoryError, and I tracked it down to the Microsoft JDBC driver
still holding references to the byte[] objects. The workaround is to
clear the pool "from time to time", closing its connections. When I do
that, the memory is properly reclaimed by the GC.
This "from time to time" involves some unsatisfying heuristics, though
(apart from the inconvenience of having to find the pool and access it).
If the driver is doing any caching on purpose, it should be documented
somewhere, and it should be configurable. My guess is that it's not on
purpose, though. I can't see much sense in holding on to written data
after a transaction is finished.
In case any driver developer should be interested, using a profiling
tool I found references from the following driver classes to a byte[]
object that should normally be reclaimed by the GC:
com.microsoft.jdbc.sqlserver.SQLServerPacketizingD ataConsumer and
com.microsoft.jdbc.sqlserver.tds.TDSConnection
Regards,
Jrg.
Joerg von Frantzius wrote:
> Hi,
> I wrote a batch import process that uses connection pooling, i.e. the

> connections are never closed until the process is finished. I am
writing
> large chunks of data (byte[]) into the database, and I made
absolutely
> sure that my application does not hold any references to the byte[]
> objects. However, the process increasingly consumes memory until an
> OutOfMemoryError, and I tracked it down to the Microsoft JDBC driver
> still holding references to the byte[] objects. The workaround is to
> clear the pool "from time to time", closing its connections. When I
do
> that, the memory is properly reclaimed by the GC.
> This "from time to time" involves some unsatisfying heuristics,
though
> (apart from the inconvenience of having to find the pool and access
it).
> If the driver is doing any caching on purpose, it should be
documented
> somewhere, and it should be configurable. My guess is that it's not
on
> purpose, though. I can't see much sense in holding on to written data

> after a transaction is finished.
> In case any driver developer should be interested, using a profiling
> tool I found references from the following driver classes to a byte[]

> object that should normally be reclaimed by the GC:
> com.microsoft.jdbc.sqlserver.SQLServerPacketizingD ataConsumer and
> com.microsoft.jdbc.sqlserver.tds.TDSConnection
The MS driver does indeed do a lot of unnecessary caching (such as
caching the whole ResultSet in direct mode), but I don't think this is
your problem here.
You are closing all JDBC resources (i.e. connections, statements and
result sets), aren't you? Even if SQLExceptions are thrown...
Alin,
The jTDS Project.
|||Alin Sinpalean schrieb:

>The MS driver does indeed do a lot of unnecessary caching (such as
>caching the whole ResultSet in direct mode), but I don't think this is
>your problem here.
>You are closing all JDBC resources (i.e. connections, statements and
>result sets), aren't you? Even if SQLExceptions are thrown...
>Alin,
>The jTDS Project.
>
When I close the connections, memory is reclaimed properly. What I
wanted to say is that memory is eaten up indefinitely as long as the
connection *remains open* and is used to repeatedly write large chunks
of data into the DB.
|||Joerg von Frantzius wrote:
> When I close the connections, memory is reclaimed properly. What I
> wanted to say is that memory is eaten up indefinitely as long as the
> connection *remains open* and is used to repeatedly write large
chunks
> of data into the DB.
I understand that. What I meant to say is "are you sure you are
explicitly closing ALL Statements and ResultSets when you're done with
them?". Leaving them open could and will cause the effects you now see.
Alin.
|||<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Alin Sinpalean schrieb:
<blockquote
cite="mid1109058703.898619.117040@.f14g2000cwb.goog legroups.com"
type="cite">
<pre wrap="">Joerg von Frantzius wrote:
</pre>
<blockquote type="cite">
<pre wrap="">When I close the connections, memory is reclaimed properly. What I
wanted to say is that memory is eaten up indefinitely as long as the
connection *remains open* and is used to repeatedly write large
</pre>
</blockquote>
<pre wrap=""><!-->chunks
</pre>
<blockquote type="cite">
<pre wrap="">of data into the DB.
</pre>
</blockquote>
<pre wrap=""><!-->
I understand that. What I meant to say is "are you sure you are
explicitly closing ALL Statements and ResultSets when you're done with
them?". Leaving them open could and will cause the effects you now see.
Alin.
</pre>
</blockquote>
Alright, now I got the point about the Statements and ResultSets,
sorry. To be honest, I can't be absolutely sure about that, because I'm
relying on a JDO implementation to take care of it.<br>
</body>
</html>
|||Joerg von Frantzius wrote:
> Alright, now I got the point about the Statements and ResultSets,
sorry.
> To be honest, I can't be absolutely sure about that, because I'm
relying
> on a JDO implementation to take care of it.
You could try jTDS (disclaimer: I'm a jTDS developer). I'm not saying
it will definitely make the problem disappear, but at least I will be
able to help more.
Alin.