At my work we have multiple environments in our Application Development department. They are separated by clusters in each environment with a load balancer on the front. Most of the individual servers run httpd with mod_jk to talk to Tomcat. One of the issues we experience is the PermGen space running out and all the cores hitting 100%. Recently, this was happening on our production server. We realized a few things.
Our Tomcat servers have several grails applications. Our JDBC connection through Tomcat uses a single entry in server.xml for each database. This means that all connections to the Tomcat server are funneled through this one tunnel back to the database. As people open and close their connections, the garbage collection sees these objects and does not collect them. Moreover, Grails tends to not optimize the queries it runs against the database and these “ad-hoc” queries are seen as new objects in Grails. They are not collected. Another thing to note is builds that are done from Jenkins or another build product. Each new build is not collected completely by the garbage collector. Over time, the PermGen space will be used up.
On our servers we noticed that when enough people were logged on to the application, the CPU would run high and eventually no more people could log on. The best way to solve this issue is to add the following to the startup environment settings in Tomcat:
As a side note, always make sure you put the following in your startup as well:
-server -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
These settings help make better use of the memory and initiate garbage collection with the newer generation of settings. If you look closely, you will see that the GenSweepingEnabled is actually an older, depreciated entry, but still useful and since Tomcat doesn’t complain, I like to leave it in there for peace of mind.
However, other tuning adjustments should be made to both the system and server.xml file to actually affect the PermGen space issue. First, let’s look at how Tomcat uses memory. I won’t purport to be an expert at memory utilization, but Tomcat tends to use swap space quite a bit. In fact, because of its use of swap space, it can greatly slow down the server in terms of thread execution. On Oracle systems, we try to adjust the swap in such a way that the server uses physical memory first. It’s faster. To that end, you can tell the server how much weight should be given to runtime memory as opposed to using page cache. To see what yours is, type this:
My systems, and I believe the Centos default, is 30. We want to reduce that. Simply type:
echo 5 | sudo tee /proc/sys/vm/swappiness
This will change the weight of cache to 5 in favor of runtime memory. Now run:
again to double-check the change. Finally, we will need to open up /etc/sysctl.conf and add the following to it so the setting holds on reboot:
We also want to change the server.xml. There are two settings that I directly look at on ours. First are the maxThreads and the minSpareThreads. The default looks like the following:
<Executor name=”tomcatThreadPool” namePrefix=”catalina-exec-”