Kelly's Space

Coding 'n stuff

Monthly Archives: February 2011

Diagnosing a Resource Leak

We had a customer report an issue with their NeoKicks converted application, where NeoData was apparently exhausting pooled SQL connections.  It certainly seemed odd, given the fact that their load was relatively low.  To help us debug, we had them create a crash dump of IIS using the Debug Diagnostic Tool from Microsoft

Once I got a crash dump, I fired up Windbg, attached to the crash dump, and loaded SOS by issuing:

.loadby sos mscorwks

Since this is a .NET solution and the error was related to running out of pooled connections, I could dump the objects with references (that the GC won’t collect) using !dumpheap

Glancing through the output, I found the following:

image

The highlighted number above is the count of the object for each type.  The reason this one in particular is odd is that the WorkerSession object represents a current, running transaction.  The WorkerSession object contains things like open file handles, etc.  If there are 525 active transactions with each containing a few NeoData files, then it is certainly possible the SQL connection pool is exhausted.  Since this server wasn’t under a lot of load (just had been running for a while), 525 seems way too high.  I would expect this to be in the 1-10 range.  Luckily, we can dig in a little deeper to see who has a reference to these objects in Windbg. 

The next step is to get more information about each one of these objects.  You can use:

!dumpheap -type Alchemy.NeoKicks.WorkerSession

to dump out the references to that object type. 

image

I’ve highlighted the address for one of the objects in the listing from !dumpheap.  I just picked one at random.  I want to find out what has references to this object, to see what might be preventing this object from being garbage collected.  To do this, I use the following:

!gcroot 000000016fa53470

image

I looked at a few other objects (again, using !gcroot) and found that they were all being ultimately held by a System.Threading._TimerCallback object. 

This was getting interesting!  Why in the world would a TimerCallback object have a reference to my WorkerSession object?  I decided it was time to go looking through our NeoKicks code to see where a TimerCallback was used.  It turns out, a WorkerSession object can contain a Timer object that is used when a CICS START is issued so that the transaction will execute on a thread pool thread (and optionally on a time interval).  These objects should have gotten cleaned up by the GC, since these tasks run immediately and return.

I decided to look through the documentation for Timer objects in the MSDN documentation and found the following:

“When a timer is no longer needed, use the Dispose method to free the resources held by the timer.”

Now we’re getting somewhere.  Sure enough, although we implement IDisposable for the WorkerSession object, we were not calling Dispose on the Timer object.  I added the Dispose() call, and re-ran a test that reproduced the problem and the issue went away.

The moral of this story – if you are using System.Threading.Timer objects in your code, you must call Dispose() on the object when you’re done.  Otherwise, you will end up with a memory leak!

Advertisements

Querying Census Data

I was building a tool to generate some random data for testing with the goal that the data is readable.  I wanted to get actual city/state/zip code information so I began looking at the census.gov website for the type of data that is available.  I stumbled across several “names files” that contain first and last names and their frequency.  The last name file was about 2MB in size – perfect for generating some random customer names.

I opened the file in notepad and started looking through the file to find how common my last name is in comparison to others.  Sure, the Find… feature in notepad works – but a SQL query would be better at pulling out the data that I wanted.

Since the format of the file was fixed record size (not comma or tab delimited), I realized that the easiest solution would be to use NeoData to create a map for the fields in the file and import directly into a SQL server table.

Each record contains a record like this:

       01 NAME-RECORD.
          05 NAME     PIC X(15).
          05 FREQ     PIC 9.999.
          05 FILLER   PIC X.
          05 CUM-FREQ PIC ZZ.999.
          05 FILLER   PIC X(2).
          05 RANK     PIC 9(5).

 

So I fired up NeoData and created a sequential file mapping using the COBOL description above, and added a line sequential endpoint pointing to the dist.all.txt file.

image

Next, I created a SQL mapping with the name field set as the primary key (since it is unique in this file) and added a SQL Endpoint to point to a test database:

image

Now it is just a matter of running the converter, to convert from the line sequential input to the SQL database:

image

I now have 88,799 records in my SQL table!  Now for the fun part – fire up SQL Management studio and run some queries.  I wanted to see how some of the names of my coworkers compare to each other.  I used the following query:

SELECT [NAME]
      ,[FREQ]
      ,[CUM-FREQ]
      ,[RANK]
  FROM [VidRent].[dbo].[NAME-RECORD]
  WHERE [NAME] IN ( 'HOLLIS', 'ANDERSON', 'COOK', 'PAULSON', 'ROBSON', 'LANGER' )
  ORDER BY [RANK] ASC

And it gave me the following results:

image

It turned out to be a trivial exercise to map this Census data into SQL Server using the data conversion feature in NeoData, and now the data is in a form that is easy to consume for my data generator.  So when you have a text file that can easily be described with a COBOL record, you can easily convert it into SQL Server using NeoData following steps similar to these.