Capture thread dumps on WebLogic server or on Metasolv application servers

At many instances one might notice that the WebLogic/Metasolv servers (Weblogic Admin or Managed servers) are displayed in WARNING state in the admin console, one of the possible reason for the WARNING is stuck threads or JMS bridges not coming up properly or similar such issues.

In such situations it becomes very important to debug(using thread dumps) to find out which application(API)/code is causing these warnings/issues.

Lets look at the ways of capturing the thread dumps.

1. WebLogic console

This is one of very easy way to capture thread dumps but it is not the preferred method due to the limited details of its output.

To capture thread dump please follow below steps.

Login to WebLogic Admin Console —> Servers —> Monitoring —> Threads

Then click Dump Thread Stack.

2.  Command Line or shell

WebLogic 8.1 does not provide a way to invoke a THREAD DUMP using the "Dump Thread Stacks" feature from the WebLogic Administration Console as provided in 10.3.

For Weblogic 8.1 thread dumps can be captured from below command from a command line or shell.

WebLogic command (java weblogic.Admin -url t3://serverhost:serverport -username <USER> -password <PASSWORD> THREAD_DUMP)

NOTE: The weblogic.jar must be in the CLASSPATH to avoid a "java.lang.NoClassDefFoundError: weblogic/Admin" exception (see below).

3. Kill -3 <PID> command

Use the "ps -ef | grep"  or "/usr/ucb/ps auxwwwww|grep"(shows memory parameters too) command

to identify the WebLogic Server's java process ID or <PID>.

Grep argument could be your server name but the <PID> will be the java process ID.

See below for exact steps.

THREAD DUMP data is logged to the ${MSLV_HOME}/${SERVER_NAME}/appserver/logs/${SERVER_NAME}.mss.log file.

4.  Using a WLST script

Below script describes how to take thread dump using WLST.

1. Establish a connection to the Admin Server.
2. Provide Server Name where you want to take thread dump i.e. managed server test_srvr_1.1 and provide 5 number(any quantity) of thread dumps and take each dump after 10 secs interval.
3. Save the file as "ThreadDump.py" at /opt/xyz/abc/Test_domain(choose your location).
4. Now to execute the above file and go to weblogic server bin directory(as per your directory structure), run cd /opt/xyz/abc/bea/wli10gR3MP1/wlserver_10.3/server/bin.
5. Run ". ./setWLSEnv.sh" ---to set up class path.
6. Run "java weblogic.WLST /opt/xyz/abc/Test_domain/ThreadDump.py"
7. As the result of the script, you will get some dmp files that contain the thread dumps in the same directory where ThreadDump.py is placed.

File: ThreadDump.py

# Purpose: To generate thread dump at specific interval
# Author: Alok Mishra
# Contact: alokmishra188@gmail.com

connect('weblogic','weblogic1','localhost:7001')

from time import strftime
from java.text import SimpleDateFormat
serverName = 'test_srvr_1.1'
counter = 0
sleepTime = 10000
threadNumber = 5

for counter in range(threadNumber):
        currentDate = java.util.Date().toString()
        myDate = currentDate.split(' ');
        finalDate = myDate[3]
        java.lang.Thread.sleep(sleepTime)
        fileName = 'Thread dump' + '_' + serverName + '_' + finalDate + '.dmp'
        threadDump('true', fileName, serverName)

disconnect()

Hope the article helps you in understanding how to take thread dumps, please leave your feedback or query.

Applications showing warning due to MessageBufferBean and MessageBufferTopicBean while migrating projects from WebLogic 8.1 to 10.3

PROBLEM

While deploying WebLogic applications to cluster, the following warning messages are displayed in server logs.

<Jun 9, 2015 3:37:43 PM EDT> <Error> <Deployer> <BEA-149202> <Encountered an exception while attempting to commit the 1 task for the application 'cluster-nur'.>
<Jun 9, 2015 3:43:42 PM EDT> <Warning> <EJB> <BEA-010061> <The Message-Driven EJB: MessageBufferTopicBean is unable to connect to the JMS destination: weblogic.test1.WlwRuntimeAppScopedJMS#MSG_B
UFFER_TOPIC. The Error was:
Can not get distribute destination information. The destination JNDI name is weblogic.test1.WlwRuntimeAppScopedJMS#MSG_BUFFER_TOPIC, the provider URL is null>
<Jun 9, 2015 3:43:42 PM EDT> <Warning> <EJB> <BEA-010061> <The Message-Driven EJB: MessageBufferBean is unable to connect to the JMS destination: weblogic.test1.WlwRuntimeAppScopedJMS#MSG_BUFFER
_QUEUE. The Error was:
Can not get distribute destination information. The destination JNDI name is weblogic.test1.WlwRuntimeAppScopedJMS#MSG_BUFFER_QUEUE, the provider URL is null>

Also the applications display Warning in console, to confirm the reason for warning go to click on application and go to monitoring tab as below.

CAUSE 

weblogic-application.xml  in the WebLogic applications has a reference to the library weblogic-controls-10.0. Because of which we get a additional MDBs from a jar file in this library (weblogic-
messagebuffer-mdb) that listen on a specific application-scoped queue and topic and as our application does not define these queues, the WebLogic will report a warning that these MDBs could not connect to their destination.

Below is weblogic-application.xml(inside TestEar.ear\META-INF\) of a sample test application which is using weblogic-controls-10.0. See highlighted.

<?xml version="1.0" encoding="UTF-8"?>

<wls:weblogic-application xmlns:wls="http://www.bea.com/ns/weblogic/weblogic-application" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/javaee_5.xsd http://www.bea.com/ns/weblogic/weblogic-application http://www.bea.com/ns/weblogic/weblogic-application/1.0/weblogic-application.xsd">

    <!-- server-version: 10.3 -->

    <wls:ejb>

        <wls:start-mdbs-with-application>false</wls:start-mdbs-with-application>

    </wls:ejb>

    <wls:application-param>

        <wls:param-name>webapp.encoding.default</wls:param-name>

        <wls:param-value>UTF-8</wls:param-value>

    </wls:application-param>

    <wls:classloader-structure>

        <wls:classloader-structure>

            <wls:module-ref>

                <wls:module-uri>TestWeb.war</wls:module-uri>

            </wls:module-ref>

            <wls:classloader-structure>

                <wls:module-ref>

                    <wls:module-uri>TestWeb_WLI_ProjectBeans</wls:module-uri>

                </wls:module-ref>

            </wls:classloader-structure>

            <wls:classloader-structure>

                <wls:module-ref>

                    <wls:module-uri>TestWeb_WLI_ComponentBeans</wls:module-uri>

                </wls:module-ref>

            </wls:classloader-structure>

        </wls:classloader-structure>

    </wls:classloader-structure>

    <wls:listener>

        <wls:listener-class>com.bea.wli.management.WLIAppListener</wls:listener-class>

    </wls:listener>

    <wls:library-ref>

        <wls:library-name>beehive-controls-1.0.1-10.0</wls:library-name>

        <wls:specification-version>1.0</wls:specification-version>

        <wls:exact-match>true</wls:exact-match>

    </wls:library-ref>

    <wls:library-ref>

        <wls:library-name>sb-transport-control-10.0</wls:library-name>

        <wls:specification-version>10.0</wls:specification-version>

        <wls:exact-match>true</wls:exact-match>

    </wls:library-ref>

    <wls:library-ref>

        <wls:library-name>weblogic-controls-10.0</wls:library-name>

        <wls:specification-version>10.0</wls:specification-version>

        <wls:exact-match>true</wls:exact-match>

    </wls:library-ref>

    <wls:library-ref>

        <wls:library-name>wls-commonslogging-bridge</wls:library-name>

        <wls:specification-version>1.0</wls:specification-version>

        <wls:exact-match>true</wls:exact-match>

    </wls:library-ref>

</wls:weblogic-application>

These JMS destinations are created by the application when the MessageBuffer annotation is used and our applications are not using com.bea.control.annotations.MessageBuffer annotation which is why above error is thrown.  These created JMS elements are used in the following library:

-{BEA_HOME}\workshop_10.3\common\deployable-libraries\weblogic-controls-10.2.ear
-in APP-INF/lib/weblogic-controls.jar
-in weblogic-messagebuffer-mdb.jar, in META-INF, weblogic-ejb-jar.xml

where the following are noted:
WlwRuntimeAppScopedJMS#MSG_BUFFER_QUEUE
WlwRuntimeAppScopedJMS#MSG_BUFFER_TOPIC

Below is  weblogic-ejb-jar.xml of weblogic-controls-10.2.ear.See highlighted.

<?xml version="1.0" encoding="UTF-8"?>

<weblogic-ejb-jar xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xsi:schemaLocation="http://www.bea.com/ns/weblogic/90

    http://www.bea.com/ns/weblogic/90/weblogic-ejb-jar.xsd"

    xmlns="http://www.bea.com/ns/weblogic/90"

    xmlns:j2ee="http://java.sun.com/xml/ns/j2ee">

  <weblogic-enterprise-bean>

    <ejb-name>MessageBufferBean</ejb-name>

    <message-driven-descriptor>

      <pool>

        <initial-beans-in-free-pool>1</initial-beans-in-free-pool>

      </pool>

    </message-driven-descriptor>

    <transaction-descriptor>

        <trans-timeout-seconds>300</trans-timeout-seconds>

    </transaction-descriptor>

    <dispatch-policy>controls_buffering_mdb_execute_queue</dispatch-policy>

  </weblogic-enterprise-bean>  

  <weblogic-enterprise-bean>

    <ejb-name>MessageBufferTopicBean</ejb-name>

    <message-driven-descriptor>

      <pool>

        <initial-beans-in-free-pool>1</initial-beans-in-free-pool>

      </pool>

      <generate-unique-jms-client-id>true</generate-unique-jms-client-id>

      <durable-subscription-deletion>true</durable-subscription-deletion>

    </message-driven-descriptor>

    <transaction-descriptor>

        <trans-timeout-seconds>300</trans-timeout-seconds>

    </transaction-descriptor>

    <dispatch-policy>controls_buffering_mdb_execute_topic</dispatch-policy>

  </weblogic-enterprise-bean>  

  

  <message-destination-descriptor>

    <message-destination-name>MessageBufferInvokeQueue</message-destination-name>

    <destination-resource-link>WlwRuntimeAppScopedJMS#MSG_BUFFER_QUEUE</destination-resource-link>

  </message-destination-descriptor>

  <message-destination-descriptor>

    <message-destination-name>MessageBufferInvokeTopic</message-destination-name>

    <destination-resource-link>WlwRuntimeAppScopedJMS#MSG_BUFFER_TOPIC</destination-resource-link>

  </message-destination-descriptor>

</weblogic-ejb-jar>

Above is a known Oracle bug.

SOLUTION

Controls Message Buffering in a cluster requires submodule targeting.

If your application enables message buffering on controls use the annotation
'com.bea.control.annotations.MessageBuffer', you may have the following error when you try to deploy it on a cluster:"While attempting to create destination MSG_BUFFER_QUEUE in module
<name_of_your_ear>!WlwRuntimeAppScopedJMS the JMSServer of name <name_of_your_cluster> could not be found".

Workaround: 

In this case, you must target the MSG_BUFFER_QUEUE to a specific JMS Server with the following submoduletargets option to your weblogic.Deployer command:

-submoduletargets cgJMSServer@WlwRuntimeAppScopedJMS@WseeJmsServer_auto_1

Hope above helps in understanding the WebLogic behavior behind these warnings, please leave your feedback or query.