Performance Analysis of Java Middleware on Linux

I routinely have to look after some reasonably complex Java middleware deployments, deployed variously on container technology provided by Tomcat or Oracle WebLogic. The hardest, and generally the most useful, thing to determine is identifying which resource is being constrained (commonly not CPU or OS memory, but often things like number of threads [dedicated to something like database connection pool]).

This is a post that I intend on maintaining as I document (and discover, hopefully) more tips and tricks; because sometimes its just not so great being the Go To Guy when it comes to engaging your head against a brick wall.

Tip: top(1) is more useful than you think

Getting the most out of top(1) requires a bit of knowledge about adding fields and changing options. I'm very thankful for Jordan Sissel's post (from 2010) on Debugging java threads with top and jstack, because it showed be something useful I didn't realise about top(1): that you can show threads. But, contrary to what Jordan indicated [back in 2010 and likely on a different OS release], top(1) can indeed show Parent PID.

Here's a quick run-down on some keystroke sequences of the useful things you can try with top, you can get to this display from within top(1) using the ? key:

Help for Interactive Commands - procps version 3.2.8
Window 1:Def: Cumulative mode Off.  System: Delay 3.0 secs; Secure mode Off.

  Z,B       Global: 'Z' change color mappings; 'B' disable/enable bold
  l,t,m     Toggle Summaries: 'l' load avg; 't' task/cpu stats; 'm' mem info
  1,I       Toggle SMP view: '1' single/separate states; 'I' Irix/Solaris mode

  f,o     . Fields/Columns: 'f' add or remove; 'o' change display order
  F or O  . Select sort field
  <,>     . Move sort field: '<' next col left; '>' next col right
  R,H     . Toggle: 'R' normal/reverse sort; 'H' show threads
  c,i,S   . Toggle: 'c' cmd name/line; 'i' idle tasks; 'S' cumulative time
  x,y     . Toggle highlights: 'x' sort field; 'y' running tasks
  z,b     . Toggle: 'z' color/mono; 'b' bold/reverse (only if 'x' or 'y')
  u       . Show specific user only
  n or #  . Set maximum tasks displayed

  k,r       Manipulate tasks: 'k' kill; 'r' renice
  d or s    Set update interval
  W         Write configuration file
  q         Quit
          ( commands shown with '.' require a visible task display window ) 
Press 'h' or '?' for help with Windows,
any other key to continue 

Here's a useful key sequence to commit to memory (or save with W): xybH1 to show current row and sorted field, as well as show threads and individual CPU cores; you might also add utomcat<CR> if you want to limit the display to just the tomcat. fb will toggle the Parent PID (PPID) column, and you can use > or < repeatedly to select which field to sort on. c can (or perhaps not-so-much with java's very long commands) be useful to show the entire command. Finally, i can be quite useful if you only care about tasks that are currently running.

Tip: identify which installation of Java is being used

Before we get into using the tools that come with the JVM (such as jstack, etc.), we need to determine which JVM is in use; because the tools generally need to match; and there may be multiple deployments and implementations of Java sitting around (eg. Oracle Java, OpenJDK, Oracle JRockit, etc.).

Assuming you've been thrown in the deep-end of the Java pool, it's useful to quickly determine which Java is in use; I tend to use ps -eo command to get the full command-names, and then isolate the particular Java instance I want (such as by looking for a WebLogic managed server name, or for the work 'tomcat', etc.). Then I take the /path/to/bin/java and I should be able to use /path/to/bin/jstack etc.

# ps -eo command | grep -o '^.*/java '
grep -o ^.*/java 


Popular posts from this blog

ORA-12170: TNS:Connect timeout — resolved

Getting MySQL server to run with SSL

From DNS Packet Capture to analysis in Kibana