JVM Tuning - Live Data Size

Another critical concept to understand is live data size. Charlie Hunt and Binu John define this as the occupancy of the old generation and the permanent generation after a full collection, when the application is running in steady state.  Our application doesn't have a steady state, but it does have predictable peak usage hours every day. Here's a snippet from one of our GC logs in production during peak hours:

[Full GC
[PSYoungGen: 32045K->0K(4118912K)]
[PSOldGen: 6684267K->4483995K(8388608K)] 6716313K->4483995K(12507520K)
[PSPermGen: 716799K->716799K(716800K)], 10.6217130 secs
]

We can see from the bold figures that our old gen occupancy is 4.3GB and our perm gen occupancy is 700MB (and it is full). It's also interesting to note that we see about the same live data size in a 12GB heap and in an 8GB heap. Our third on-site Red Hat consultant (we'll call him Boris), who has seen a lot of different operations and seems to know his stuff well, was surprised that our live data size could be so big. Why so big? Well, it turns out our secondary Hibernate cache is 3.9GB - 90% of live data size. And our StandardQueryCache alone is 3.4GB. So we have a big investment there. Is it beneficial? The best way to know is to test - preferably in a performance lab, not directly in production. However, a meaningful performance lab is not cheap and requires real organizational commitment (including money and time). We have, anyway, figured out that some of our StandardQueryCache is counterproductive. SQC uses a very blunt invalidation - if any change is made to any row of a table, then any SQC for that table is invalidated. So SQC should only be used for read-only or read-mostly tables. We'll be cleaning that up.

Hunt and John recommend, as a general guideline, setting the heap size to be 3 to 4 times the live data size of the old generation. For us, that would be 12.8GB to 17.1GB. So, even after bumping up our heap from 8GB to 12GB, we are still running a lean and hungry heap, by this measure. They also recommend setting perm gen size to 1.2 to 1.5 times the live data size of the perm gen. Here, that would be 840MB to 1050MB.