Friday, March 19, 2010

JRun / Apache / jrApache Malloc Errors

I was chatting with a friend to today and this issue was showing up in his Apache logs for an application running on ColdFusion with JRun running on Windows:

[Fri Mar 19 18:01:06 2010] [notice] jrApache[1720: 12042] MEM apache_malloc: 0x2e74af0: 72 bytes
[Fri Mar 19 18:01:06 2010] [notice] jrApache[1720: 12042] MEM apache_free: 0x2e74b70

Believe it or not a quick search of Google "jrApache apache_malloc" or "jrApache apache_free" show a big zero results (which is not common these days on Google).

We figured out after reading about malloc on Apache that it had to do with this setting in the JRunConfig in Apache:

JRunConfig Apialloc true

Apialloc has to do if the memory allocation is going to be handled by the Apache subsystem or the operating system.  Setting it to "true" tells Apache to handle it and setting it to "false" tells the module to ask the OS.  Setting apialloc to false solved the problem:

JRunConfig Apialloc false

We gleamed this information from Steven Erat's blog :

The JRun connector setting for Apialloc governs from where the connector allocates memory. The connector stub is written in C and when it needs to allocate memory it can either ask the operating system for memory (apialloc=false) or ask Apache for it (apialloc=true). Asking Apache for memory was probably quite safe in Apache 1.3x with just one instance of a JRun server, but its best to always set apialloc to false and let the stub work with the OS to manage memory.


I hope this blog entry filling the zero search results for Google and helps somebody else in the future.

Changing CVS Username and Password in an Eclipse Project

I'm posting this for my future reference since I could find any reference to changing a saved CVS username and password for an already created Eclipse project.  This post has nothing to changing CVS credentials on a CVS server.  I needed to do this because I had commits to make to an open source project however the project I originally setup was using the anonymous read only access.

1. You need to set a new connection string for the CVS repository you want to access by using the CVS Repository view.

Goto the "Window" menu -> "Show View" -> "Other" -> find "CVS" section and expand -> "CVS Repositories"

2. Next setup the new connection string for your repository

Click on the mini icon of this view (a cylinder with the word "CVS" and a "+" sign) -> Add your new CVS repository

3. Associate the new repository (could be the same repository with different login credentials) with your project

Right click on the project in the "Navigator" -> select "Properties" -> in the properties tree select "CVS" -> hit the "Change Sharing..." button -> select your new repository and "OK" -> hit "Apply" back on the properties page

I hope this helps somebody in the future.

Thursday, March 4, 2010

Fixing a Mysterious Memory Leak on ColdFusion 8

Last year I noticed a strange problem while developing the Mach-II Dashboard on an Adobe CF8 based application.   If I reloaded my entire Mach-II application through an URL argument which called the bootstrapper loadFramework() method everything was fine.  However, if I reloaded my application via the Dashboard I would see a slow memory leak of 30MB of Java heap space per reload.  Suffice it to say that in normal development, you can reload an application many times every day.  Thus this was very bad.  I do remind you that my application is fairly large (200-250MB of heap space to load up).

The only difference between the two methods of reloading was the bootstrapper method instantiated a Mach-II AppLoader CFC and replaced the current one in the application scope.  The dashboard on the other hand would reload the application by reusing the current AppLoader and calling an method on it.  I was able to replicate the memory leak using the bootstrapper by calling the same method that the Dashboard was using instead instantiating a new AppLoader CFC.

I immediately tried to replicate this memory leak on OpenBD and Railo test beds however I did not observe any memory leaks.  So the problem only existed on CF8 and probably CF7 based on another blog post posted by the great development team at Dave Ramsey.  To be fair, I personally haven't tested on CF9 but have heard reports of this from another Mach-II user.

Basically, it appears that CF8 holds on to a strong reference of some type of object in memory and is never released  from the heap for reuse.  I checked a lot stuff in our code base for accidental circular references or something strange like that, but I was unable to figure out the problem.  I even devised a stripped down application based on the Mach-II skeleton and nothing else. This was attempt #1 where I spent about four hours.  Frustrated, I went to bed.

It took a couple of months of just ignoring the problem for me to release all the frustration that I have had about this issue.  Plus, I was out of ideas even after discussing this Kurt Wiersma and Matt Woodward.  So on attempt #2, I decided that I needed to use some more high powered tools available to me.  So I fired up VisualVM which is bundled with Java 6 these days (before it was separate download).  VisualVM allows you to get detailed information about Java applications that are running in a JVM (local or remote) such as monitoring memory consumption, profile applications and take/display heap or thread dumps.  Since ColdFusion runs on Java, this is perfect tool to figure out the problem.

I'll spare you my frustrations, but about four hours in I was able to see that CF8 was holding on to a strong reference to the MachII.framework.AppLoader component which does have a circular reference to another AppLoader when Mach-II modules are in use.  However, my frustrations were not of what was the problem, but how to fix the issue.  So I Googled and Googled for blog posts and after a while I came upon this post (thanks Dave Ramsey guys - great blog name):

ColdFusion 7 Memory Leak (via Web Monkeys With Laser Beams)

So, what the hell?  If I leave off a "output" attribute on a <cffunction> there could be a memory leak?  That seems like a totally stupid bug.  So I did a code base search of Mach-II for missing "output" attributes on <cffunction> tags.  I knew there were a few because CF doesn't treat the "output" attribute like a true boolean.

1. output="true" - output everything
2. output="false - suppresses all output
3. no output attribute defined-  suppresses all output except if you have something wrapped in a <cfoutput> tag

Option number three is used in a couple of places because it is actually quite usual.  Now, this "feature" of the "output" is not documented, but is known to I guess a more advanced CFML programmers.  So I added in cfoutput="true" for all the places in all the places where we had left off the "output" attribute.  I thought I would worry about whitespace suppression later if this work.  So, restarted my instance of Tomcat and did a few reloads of my application via the Dashboard.  I took a heap space dump before I started to get a baseline and did a few more snapshots after a few reloads to see what was happening.

I think after seeing the first results I yelled some expletive along the lines of Holy Cow!  This small change was like a tourniquet on the memory leak.  It was now down to 6MB memory leak per reload.  So I knew there was something else up.  A quick search of ColdSpring showed a ton of missing "output" attributes on <cffunction> tags.  A RegEx replace of those was quickly performed.

After restarting my Tomcat instance, I repeated my analysis process of heap dumps and reloading of my application.  Viola! The memory leak was gone.  Literally, the JVM would recover all the memory -- down the almost 1kb of heap space after a full GCC was requested (which is why I wrote a "request garbage collection" feature in the Dashboard that night).

Summary: Leaving off the "output" attribute of <cffunction> tags can cause memory leaks in certain situations on CF7 and CF8 (even with CHF 3 installed which supposedly fixes memory leak issues with CFCs and memory scopes).

Solution: Never leaver off the "output" attribute!

Cases in the wild: I'm sure everything had a few of these in their code base.

I can't confirm how CF9 handles this stuff since aren't running that on any of our applications at work, but I suspect the same problem exists.  So if anybody wants check, feel free to comment.


Update: March 6, 2010

Because I have been asked this a few times.  The CF8 server I was running my tests against was 8.01 with CHF3 installed.

I have filed a ticket for this issue with Adobe as ColdFusion Bug 82362.