Out of Memory errors can crop up in a number of different places. Certain ANT tasks, like javac, java, and jar can use a lot of the VM's available memory, especially if many tasks are being run in the same VM as ANT. If you are merging the xml log files generated by cruise control (not advised) cruise control will report an Out of Memory error after merge becomes unable to handle the size of the log file, for example.
It's possible to increase the amount of memory allotted to ANT's VM by adding a few <jvmarg> elements to the AntBuilder element in your config.xml file, like this:
<ant buildfile="..." target="build" multiple="1">
<jvmarg arg="-server" />
<jvmarg arg="-Xms64m" />
<jvmarg arg="-Xmx256m" />
</ant>
In addition, you can try setting an ANT_OPTS environmental variable to "-Xmx256m". Sometimes, the solution may be to fork an all-new instance of the VM for the really intensive tasks. Try to reproduce the error outside cruise control. If it only shows up when running cruise control, check the logs to see where & when things are going awry.
An OutOfMemoryError can also happen due to the amount of information being logged during a build. See UseLogger for directions on reducing the amount of information logged.
An OutOfMemoryError can also happen due to the amount of classes being loaded over and over again. This can happen because ANT uses its own class loader and even uses a new class loader instance each time it invokes an ANT project (which is at every <ant/> and <antcall/> task invocation). Therefore classes, for example in <taskdef/>, are loaded over and over again.
Strikingly, the OutOfMemoryError occurred far before the heap size was reached!
Possible solution: add the classes (for example your own ANT tasks) to the classpath used when CC starts ANT, for example by setting the CLASSPATH environmental variable. Or use the AntScriptAttribute
Hope this helps, cheers - PatrickLisser
Tony Cook reports on his out of memory problem:
From: cruisecontrol-user-admin@lists.sourceforge.net cruisecontrol-user-admin@lists.sourceforge.net On Behalf Of Tony Cook
Sent: Tuesday, January 06, 2004 7:40 AM
To: cruisecontrol-user@lists.sourceforge.net
Subject: [Cruisecontrol-user] Re: Re: Memory resourse issues with Cruise Control
In November 2003 I reported that for my configuration, Cruise Control would hog about 400Mb of memory while idling and there was some speculation that a forced garbage collect might be needed. (Memory eaten up and held via the HTMLEmailPublisher [Xalan is culprit actually]). I've dug into this deeply and eventually found the underlying issues and can report that I fixed my problem via the following steps.
1) The main hog was due to a custom recursive XSLT template used to insert <br/> tags in place of a log file new lines. When the log files were large (more lines) more memory was hogged (marked for deletion but not yet available to other processes). I converted to an iterative line breaking XSLT template and most of the HTMLEmailPublisher memory hogging issues simply disappeared.
2) More memory/speed streamlining was achieved by switching to running Ant in a separate JVM process via the ant builder antscript attribute. When the out-of-process Ant finishes, you naturally get all its memory back before CC goes to sleep. As an aside this switch dropped my build times from 18 minutes to 6 minutes (a fragmented heap issue?), so I highly recomment this approach 
. Also this method has allowed me to painlessly switch to Ant 1.6.0 today.
So in summary - there is really no need to force garbage collection (at least for now), its better to find out why so much peak memory was being used in the first place and fix that.
Cheers, Tony
Dan Penhurst reports that changing JVM solved his OOM problems:
To: <cruisecontrol-devel@lists.sourceforge.net>
Sent: Wednesday, January 12, 2005 5:59 PM
Subject: Re: [Cruisecontrol-devel] Memory leak
> I suffered from OOM exceptions every 4 days but they were down to a JVM bug.
> I now run IBMs JVM (1.4.2) on multiple platforms building many projects
> very frequently (multiple times per hour) and CC 2.2 is rock solid.
>
> Dan
WARNING...WARNING....WARNING...
Well, not so much of a warning but a GOTCHA!
The <jvmargs> tags are IGNORED when using the antscript attribute. So, if you need more memory for that new JVM, you need to place it in the wrapper.
Out of memory for class loading.
The best thing you can do is to use antscript, run cruise control in a profiler to where the problem is. If you see that there isn't much allocated but you go out of memory, you'll know that you have problem with class loading.
Adding all classes tio the system class path does not seem to help. However use the following command:
java ... org.apache.tools.ant.launch.Launcher -lib <path to ant jars> -lib <path to other ant jars> <etc...>
This solved the outofmemory problems for med. This is tested only one ant 1.6.2
/erik
From: Gisbert Amm
Sent: Tuesday, March 22, 2005 4:43 AM
Cc: Cruisecontrol-user@lists.sourceforge.net
Subject: Re: [Cruisecontrol-user] StackOverFlowError on Test Results
We run CC on a machine with 2 GB RAM with -Xms 1664m and -Xmx 1664m.
Above this value I get OutOfMemoryErrors. As far as I see, the CC
processes allocate not so much physical memory but the virtual memory
they consume can grow really huge.
Recently we discovered the -XX:+UseParallelGC option which decreases
runtime because of parallel garbage collection. This could also help to
lower the needed memory a little because the memory is freed a bit earlier.
Another option I've just found out about and that might help you is
-XX:+AggressiveHeap, which causes the VM to try to get nearer to the
physical limits of the provided RAM (see
http://java.sun.com/j2se/1.4.2/1.4.2_whitepaper.html, "AggressiveHeap -
Server Performance Option").
Then there are the options -XX:PermSize and -XX:MaxPermSize which are
good explained on
http://www.unixville.com/~moazam/stories/2004/05/17/maxpermsizeAndHowItRelatesToTheOverallHeap.html
and might also help.
Probably setting some of these options could help you to fix your
problem. More experienced Java experts on the list please correct me if
I said something erroneous.
Regards,
Gisbert
From: Pascal Giard (QC/EMC)
Sent: Tuesday, March 22, 2005 8:38 AM
To: 'Cruisecontrol-user@lists.sourceforge.net'
Subject: RE: [Cruisecontrol-user] StackOverFlowError on Test Results
Thanks alot Gisbert!!
It works just great with:
-Xms512M -Xmx512M -XX:+UseParallelGC
"-XX:+UseParallelGC" seems to help alot.
-Pascal
When you still having problems ... try Ivan's fix (committed to 2.6)
As many users, I experienced OutOfMemory, StackOverflow, etc. troubles with log files. And recently when thigs get out of hands I start digging things out. First I noticed Xalan, it is usual source of pain, after I checked xslt templates and they appear to be bad also.
But before blaming anything or anyone let's do some testing.
I took dual cpu linux server with 2G of ram, that does not run anything, except, ssh, apache + tomcat + CC
I set -Xmx10240m -Xms10240m -server and run Tomcat with CC.
Since I need to test only html page generation I disable project build triggering, so build won't start.
Logs dir has 6m test results file that hasn't been processed yet (no html for this log file has been produced).
Now I start the browser and ask for logs page. At the same time I have `top` opened on another monitor to see what server is doind. For a few minutes I observed how Xalan eat up all memory and began swapping, in few minutes I simply got StackOverflow and jvm crashed with ~8G of allocated memory. I repeated this test multiple time with different memory settings - result is consistent - StackOverflow or OutOfMemory
As been expected Xalan can't do the job, throw it away, and replace with Saxon8 (no classpath adjustments has been made)
Runnung the same test again - memory usage is stable and does not go beyond 256m, in 5-8 minutes I have my page, that proves that Saxon is well written and performs as expected, but still 5-8 minutes is unacceptable.
Since xslt template are bad, we have to rewrite it, also I noticed some `smart` ideas that scared me.
<!--
Write output and err into a JavaScript data structure.
This is based on the original idea by Erik Hatcher (ehatcher@apache.org)
-->
I haven't seen such insane idea in my life, load few megs of text into browser memory...
No wonder why some browsers were crashing on this page.
As a fast fix I replaced it with hidden div's, and single javascript function. But to do it properly it should be written into separate file and loaded only when needed.
After replacing xslt template with the new one, test result page appearing in 3-5 seconds. Simple performance tests show 1-2Mb/sec processing speed. So if you have 10M log file it might take up to 10 seconds (depending on your server).
 | Now lets summ all this into simple instructions
- Remove WEB-INF/lib/xalan-2.6.0.jar and place there saxon8.jar
- Replace xsl/testdetails.xsl with testdetails.xsl file
- Edit css/cruisecontrol.css and add following style:
.testresults-output-div {border:solid 1px; font-size: 9pt; font-family:monospace}
- Optional you can modify classpath and put saxon instead of xalan
- (note) I had to add
in order to get CC to use the Saxon transformer, otherwise it was trying to use the bundled xalan transformer in the JDK (com.sun.org.apache.xalan ...) - without good results (ArrayOutOfBoundsException).
- Don't forget to restart your web server and CC after
|
--
Ivan Latysh
Ivan's fix (committed to 2.6) worked perferctly - just follow the summary instructions. The one comment, however, is that replacing xalan with saxon in the classpath isn't optional - it needs to go in there. Thanks again for the suggestion!
I think there is an issue with the posted testresults.xsl. It uses a javascript function "displayMessage", but this function is not defined in the xsl file. This causes all failure and error messages to be lost. Click on a failure or error and nothing happens. It was defined in the original testresults.xsl, but was removed. System.out, System.err and Properties display were changed from a javascript function to a hidden div instead. Looks like the test result were missed.