Wednesday, May 23, 2012

Where and How to Set JVM Parameters in Oracle SOA 11G


Let's say you experience some performance issues of your Oracle SOA 11G servers, and after some research you know that you need to tune your JVM heap size, and turn on some JVM debugging to monitor such as Garbage Collection behaviour. After some due diligence of reading WebLogic documents and Internet postings, you found the right flags to use, and the right values to set. It typically looks something like

-Xmx4096M -Xms4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M -Xloggc:/tmp/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

But the problem now is: WHERE and HOW to set these JVM arguments and debug flags? Should I do it through <server-start/> via Console or config.xml? Should I do it at one of those scripts under DOMAIN_HOME/bin? Exactly which script and which variables?


And to go further, what should I do if I want to set JVM differently for different servers within the same domain?

Here is what you could do in a typical Oracle SOA 11G installation.

First, choose the right way to specify JVM arguments


There are two options as the places for you to specify the JVM arguments:
  1. Via the "Server Start" tab at Administration Console , which is equivalent to the <server-start/> tag in config.xml
  2. Via the startXXX.sh and setXXX.sh scripts under DOMAIN_HOME/bin/ of each server

The problem of Option 1 is that the JVM parameters you put in <server-start/> via either Console or directly via config.xml will take effect only if you start the server from NodeManager. If you start the server via scripts, e.g.,  startWebLogic.sh, the parameters in <server-start/> in config.xml will be ignore.

For Option 2, if you start servers via scripts under DOMAIN_HOME/bin, specifying JVM parameters will of course work. And as long as you set the StartScriptEnabled=true and StartScriptName=startWebLogic.sh, as required by the Enterprise Deployment Guide, starting servers via NodeManager will also call the same startWebLogic.sh scripts, and hence will pick up the JVM arguments you set there. So you are covered both ways.

Therefore, I would like to recommend Option 2 to specify the JVM arguments in the scripts under DOMAIN_HOME/bin/"

Second, choose the right script and right variables


But the problem is that there are many scripts under the DOMAIN_HOME/bin. And those scripts even reference the scripts outside of the bin/ directory. Meanwhile, there are a lot of variables that are related to JVM arguments: MEM_ARGS, USER_MEM_ARGS, JAVA_OPTIONS, JAVA_PROPERTIES ... to just name a few.

Among these scripts and variables, I recommend you define the JVM arguments via the USER_MEM_ARGS and JAVA_OPTIONS variables in setSOADomainEnv.sh.

The USER_MEM_ARGS and JAVA_OPTIONS variables


When you examine the contents of startWebLogic.sh/setDomainEnv.sh/setSOADomainEnv.sh, you can conclude that you should customize the JVM memory settings (i.e. max and min heap size, PermGen size, etc) via variables USER_MEM_ARGS, and specifying any other JVM arguments (i.e. debug flags, system properties, etc) via JAVA_OPTIONS:

# from setDomainEnv.sh

# IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values


# from setDomainEnv.sh:

# JAVA_OPTIONS    - Java command-line options for running the server. (These

#                   will be tagged on to the end of the JAVA_VM and

#                   MEM_ARGS)



The setSOADomainEnv.sh script


In a typical SOA installation, you find that startWebLogic.sh under DOMAIN_HOME/bin sources setDomainEnv.sh, and the setDomainEnv.sh source setSOADomainEnv.sh.

Further, if you go to the end of the setSOADOmainEnv.sh, you will notice the following:

echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} 
echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"


These lines print the values of JVM memory settings to the *.out file. So it makes sense to insert the settings of the JVM heap size and debug flags right above them  Otherwise, the actual settings may be different from what the setSOADomainEnv.sh print to the *.out file and would cause confusions.

Therefore, it is obvious that the proper place to set the USER_MEM_ARGS and JAVA_OPTIONS is right above the afore mentioned code snippet in the setSOADomainEnv.sh

Here is an example of how to set the JVM heap size and debug flags at setSOADomainEnv.sh:

USER_MEM_ARGS="-Xmx4096M -Xms4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M"

JAVA_OPTIONS="${JAVA_OPTIONS} -Xloggc:${DOMAIN_HOME}/servers/${SERVER_NAME}/logs/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"


Third, to make it a bit more robust.

Now, what if I want to have different settings for different servers? For examples, I may want my managed servers to run with a bigger heap size than the AdminServer, while turning on the debug flags on all of servers:

In such case you can script your JVM settings based on the server names:

if [ "${SERVER_NAME}" = "WLS_SOA1" -o "${SERVER_NAME}" = "WLS_SOA2" ] ; then
    USER_MEM_ARGS="-Xms4096M -Xmx4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M"
fi

JAVA_OPTIONS="${JAVA_OPTIONS} -Xloggc:${DOMAIN_HOME}/servers/${SERVER_NAME}/logs/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"


echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"

If each server has its own DOMAIN_HOME/bin directory, then you need to make sure the same code snippet are copied to the same setSOADomainEnv.sh script for all servers.

Forth, make the settings future proof.


So far so good with the modification of the setSOADomainEnv.sh, until you found the following comments at the top of the file:

# WARNING: This file is created by the Configuration Wizard.
# Any changes to this script may be lost when adding extensions to this configuration.


That means the changes you put in may be gone. Actually I have experienced this first hand when you upgrade the servers from the a new minor release (such as a patch set).

So, to be safe, it is better to move the settings to an external script (e.g. setCustomEnv.sh), and source this external script from setSOADomainEnv.sh:

setSOADomainEnv.sh (please note the line starts with a dot ('.'):
. ${DOMAIN_HOME}/bin/setCustomEnv.sh

echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"



setCustomEnv.sh:
#!/bin/sh

echo "Settings from setCustomEnv.sh"
if [ "${SERVER_NAME}" = "WLS_SOA1" -o "${SERVER_NAME}" = "WLS_SOA2" ] ; then
    echo "Customizing USER_MEM_ARGS for SERVER_NAME ${SERVER_NAME}"
    USER_MEM_ARGS="-Xms4096M -Xmx4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M"
fi

JAVA_DEBUG_FLAGS="-Xloggc:${DOMAIN_HOME}/servers/${SERVER_NAME}/logs/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"
JAVA_OPTIONS="${JAVA_OPTIONS} ${JAVA_DEBUG_FLAGS}"
echo "Setting JAVA_OPTIONS from setCustomEnv.sh: JAVA_OPTIONS ${JAVA_OPTIONS}"
echo "End settings from setCustomEnv.sh"

In the event when the setDomainEnv.sh was overwritten by product patching or upgrades, you can still very easily put back that single line of sourcing the external script.

In the case where each server has its own copy of DOMAIN_HOME/bin directory, you can put the setCustomEnv.sh at a shared file system. So you keep only one copy of setCustomEnv.sh where all the JVM settings are centrialized. This single copy of setCustomEnv.sh can be referenced by the individual setSOADomainEnv.sh for each server.

Now, you are good to go!

5 comments:

  1. Many thanks.. it's certainly useful.

    ReplyDelete
  2. Its very useful... Thanks a lot for sharing this info...

    ReplyDelete
  3. that's really an exhaustive coverage of the topic, really appreciated

    ReplyDelete