Weblogic Integration MQ Event Generators not polling and retrieving messages from MQ in a Weblogic clustered environment

PROBLEM

A WLI Event Generator appears not to be consuming messages from an MQ queue.

When there is just one managed server running in a cluster, the Event Generator is retrieving the
messages from MQ.
But as soon as the second managed server is started in the cluster, the Event Generator gets redeployed (suspended for the time the second server is in process of starting and becomes running once the second server is running) and after that it stops picking the messages until the Event Generator is suspended/resumed again at which point it will perform a one time retrieval of message from the queue.

Essentially in a clustered environment, as soon as a new managed server is added (beyond the very first one) into the cluster Event Generator seems to start having problems and it stops polling messages from queue.

CAUSE

This is due to the fact that Server Affinity was disabled in the Connection Factory thus preventing the cluster from knowing which server would process the messages.

Changing the Server Affinity to "true" resolved this issue.

SOLUTION

1. Enable the server affinity for this connection factory or create a new connection factory with this
enabled
2. Create a new MQ Event Generator that uses this new connection factory
3. Restart admin and managed servers
4. Test the scenario again and make sure messages get processed.

Hope above helps in fixing above mentioned issue related to Event Generators in WLI, please leave your feedback or query.

Send messages to IBM WebSphere MQ from Unix server using Shell script and JAVA

A standalone shell script to send messages to MQ on an IBM WebSphere MQ server using JAVA program.

The application mostly helps in testing connectivity from Metasolv M6 server to IBM WebSphere connectivity in MQ event generators in WLI.

In order to build this utility please follow below steps.

STEP 1. 

Create a alok_tmp(choose any name) in any directory on your unix box and create src and bin folder and below files.

Folders created inside directory i.e. /opt/app/script are as below

1. alok_tmp
2. alok_tmp/src
3. alok_tmp/bin

Files created inside directory src and bin

1. alok_tmp/sendMQ.sh
2. alok_tmp/src/MQSendUtility.java
3. alok_tmp/bin/requestMessage.xml
4. alok_tmp/bin/mq.properties

Below should be directory structure.

Class file will automatically be created after each time sh file is run.

STEP 2. 

Copy below to Java file to send the messages to MQ on IBM WebSphere server. Code below is self explanatory.

File: MQSendUtility.java

/** 
* This class provides access to the MQSeries queue facilities.  Once 
*
* instantiated, the main application program need only call the 'send'
*
* method to put message on request queue. The send method returns an MQMessage. 
*
* reply queue is still kept in class as earlier it was also used to receive the reply.
*
* but now it is now utilised to send a MQ message only. 
*
* @author Alok Mishra
*
* @version 1.0
*/

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.PrintWriter;
import java.util.Properties;
import com.ibm.mq.MQC;
import com.ibm.mq.MQEnvironment;
import com.ibm.mq.MQException;
import com.ibm.mq.MQMessage;
import com.ibm.mq.MQPutMessageOptions;
import com.ibm.mq.MQQueue;
import com.ibm.mq.MQQueueManager; // Include the MQ package

public class MQSendUtility {
private static Properties props = new Properties();
private MQQueueManager qMgr;
private MQQueue requestQueue;
private MQQueue replyQueue;
public String replyQueueName;
private int openOptions;
private MQMessage storedMessage;

@SuppressWarnings("unchecked")
public MQSendUtility(String hostname, String channel, String qManager,
String aRequestQueue, String aReplyQueue) throws MQException {

try {
MQEnvironment.hostname = hostname;
MQEnvironment.channel = channel;
MQEnvironment.port = Integer.valueOf(props
.getProperty("mq.listen.port"));
MQEnvironment.userID = props
.getProperty("mq.server.security.principal");
MQEnvironment.password = props
.getProperty("mq.server.security.credentials");
MQEnvironment.properties.put(MQC.TRANSPORT_PROPERTY,
MQC.TRANSPORT_MQSERIES);
try {
qMgr = new MQQueueManager(qManager);
} catch (MQException e) {
print("An MQException occurred trying to connect to the QManager.");
Object o = e.exceptionSource;
print("MQException originated from object '" + o.toString());
print("Completion code = " + e.completionCode);
System.exit(1);
}
print("Connected to QManager " + qManager);

replyQueueName = aReplyQueue;

if (aRequestQueue.equals(aReplyQueue)) {
print("Open Options from IF");
openOptions = MQC.MQOO_INPUT_AS_Q_DEF | MQC.MQOO_OUTPUT;
} else {
print("Open Options from ELSE");
openOptions = MQC.MQOO_OUTPUT;
}
print("Open Options"+openOptions);
requestQueue = qMgr.accessQueue(aRequestQueue, openOptions, 
null, // default queue manager
null, // no dynamic q name
null); // no alternate user id

print("Got access to request queue "+requestQueue);

} catch (MQException ex) {
print("MQCommunicator.constructor - MQException occurred : Completion code "
+ ex.completionCode
+ "\n>MQStatus: Reason code "
+ ex.reasonCode + " mesage" + ex.getClass());
System.exit(1);
}

catch (Exception e) {
print("MQCommunicator.constructor - Exception occurred - "
+ e.toString());
System.exit(1);
}
}

public MQMessage send(String buffer) {

try {
MQMessage sendMessage = null;

try {
// Create new MQMessage object
sendMessage = new MQMessage();
} catch (NullPointerException e) {
print("Unable to create new MQMessage");
return null;
}
print("MQMessage created");

sendMessage.format = MQC.MQFMT_STRING;
sendMessage.messageType = MQC.MQMT_REQUEST;
sendMessage.replyToQueueName = replyQueueName;
sendMessage.writeString(buffer);
MQPutMessageOptions pmo = new MQPutMessageOptions();
try {
requestQueue.put(sendMessage, pmo);
} catch (NullPointerException e) {
print("Request Q is null - cannot put message");
return null;
}
print("Message placed on queue");
storedMessage = new MQMessage();
storedMessage.correlationId = sendMessage.messageId;
print("Message ID for sent message = "
+ sendMessage.messageId.toString());
print("Correlation ID stored = "
+ storedMessage.correlationId.toString());

return storedMessage;

} catch (MQException ex) {
print("MQCommunicator.send - MQException occurred : Completion code "
+ ex.completionCode + " Reason code " + ex.reasonCode);
return null;
} catch (java.io.IOException ex) {
print("MQCommunicator.send - IOException occurred: " + ex);
return null;
} catch (Exception ex) {
print("MQCommunicator.send - General Exception occurred: " + ex);
return null;
}
}

public void finalise() {
try {
// Closing the queues
requestQueue.close();

if (requestQueue != replyQueue)
replyQueue.close();

// Disconnect from the queue manager
qMgr.disconnect();

} catch (MQException ex) {
print("MQCommunicator.finalise - MQException occurred : Completion code "
+ ex.completionCode
+ ">MQStatus: Reason code "
+ ex.reasonCode);
}
}

public static void print(String msg) {
PrintWriter message = new PrintWriter(System.out, true);
message.println(">MQStatus: " + msg);
}

private static String getFileContent(String fileName) throws Exception {
File file = new File(fileName);
StringBuffer strBuffer = new StringBuffer();
try {
FileInputStream fileIn = new FileInputStream(file);
int lengthOfFile = (int) file.length();
byte[] outputByte = new byte[lengthOfFile];
// copy binary content to output stream
while (fileIn.read(outputByte) != -1) {
strBuffer.append(new String(outputByte));
}
fileIn.close();
} catch (Exception e) {
print("Exception FileInputStream" + e.getMessage());
e.printStackTrace();
throw e;
}
return strBuffer.toString();
}

public static void main(String args[]) throws Exception {


InputStream file = null;
String propertyFile = "mq.properties";
print("Load property File");
try {
file = MQSendUtility.class.getClassLoader().getResourceAsStream(
propertyFile);
props.load(file);
print("Loaded property File");
String filePath="requestMessage.xml";
String message = MQSendUtility.getFileContent(filePath);
String reqQueueName = props.getProperty("mq.request.queue");
String repQueueName = props.getProperty("mq.reply.queue");
String queueManager = props.getProperty("mq.queue.manager");
String channel = props.getProperty("mq.channel.name");
String qManagerUrl = props.getProperty("mq.provider.url");
print("Calling Constructor");
MQSendUtility util = new MQSendUtility(qManagerUrl, channel,
queueManager, reqQueueName, repQueueName);
MQMessage replyMsg = util.send(message);


} catch (IOException io) {
if (file == null)
print("No input file specified");
} catch (Exception e) {
print("Exception occured in main " + e.getMessage());
} finally {
if (file != null) {
try {
file.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

}

}

STEP 3. 

Copy below to shell script.

File: sendMQ.sh

#!/bin/ksh

#Execute Domain variables
. /opt/app/domains/my_domain/bin/setDomainEnv.sh

# Assuming domain variables don't include com.ibm.mq.jar so include it
CLASSPATH=$CLASSPATH:/opt/app/domains/my_domain/mq-jars/com.ibm.mq.jar
export CLASSPATH

cd /opt/app/script/alok_tmp
javac -d bin -sourcepath src src/MQSendUtility.java

cd /opt/app/script/alok_tmp/bin
java -Dbea.home=$BEA_HOME  MQSendUtility

STEP 4. 

Below is property file & request message file(XML).

File: mq.properties 

mq.provider.url=#IP Address of the IBM WebSphere MQ server i.e. 172.0.0.1
mq.channel.name=#Channel Name.This must be a valid channel on the IBM WebSphere MQ server.
mq.request.queue=# MQ Name where msg need to be put.This should be a valid queue for the specified queue manager.
mq.reply.queue=# reply queue name.This should be a valid queue for the specified queue manager.
mq.queue.manager=# This should be a valid queue manager on the IBM WebSphere MQ server i.e. qmgr123.
mq.listen.port=# Specifies the port of the IBM WebSphere MQ server.
mq.server.security.principal=#Specifies the username who has permission to use this channel.
mq.server.security.credentials=# Specifies the password for the given user.

File: requestMessage.xml

Hello, this is a test message being sent to MQ on IBM websphere server.

STEP 5. 

Now run sendMQ.sh and see output as below.

Hope above helps in testing Metasolv to WebSphere MQ connectivity by checking if M6 application server is able to send a message to MQ , please leave your feedback or query.

Metasolv 6.2.1- Multiple STUCK Threads due to HashMap in com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagementBean.java:1191) in WebLogic 10.3.0

PROBLEM

Multiple STUCK threads seen in Weblogic servers caused by custom java applications that calls the 'getOrderByKeyRequest' API - the STUCK thread seems to be tied to internal API code where the root of the issue may be improper use of HashMap versus something thread safe like ConcurrentHashMap

The issue is happening on the "getOrderByKeyRequest" API. Refer below stack trace.

[STUCK] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000108da6000 nid=0x665d runnable [0xfffffffdfd8f9000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:737)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(ControlBeanContext.java:420)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForBean(JpdControlContainerContext.java:78)
at com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.setDelegateMapForBean(AnnotationOverrideHelper.java:49)
at com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.getBeanAnnotationMap(AnnotationOverrideHelper.java:58)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.getBeanAnnotationMap(JpdControlContainerContext.java:83)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.getAnnotationMap(ControlBeanContext.java:379)
at org.apache.beehive.controls.runtime.bean.ClientInitializer.getAnnotationMap(ClientInitializer.java:61)
at com.metasolv.api.control.OrderManagementImplClientInitializer.initializeFields(OrderManagementImplClientInitializer.java:221)
at com.metasolv.api.control.OrderManagementImplClientInitializer.initialize(OrderManagementImplClientInitializer.java:279)
at com.metasolv.api.control.OrderManagementImplInitializer.initControls(OrderManagementImplInitializer.java:26)
at org.apache.beehive.controls.runtime.bean.ImplInitializer.initialize(ImplInitializer.java:35)
at org.apache.beehive.controls.runtime.bean.ControlBean.ensureControl(ControlBean.java:321)
- locked <0xfffffffebb181348> (a com.metasolv.api.control.OrderManagementBean)
at com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagementBean.java:1191)
at sun.reflect.GeneratedMethodAccessor1441.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:433)
at com.sun.proxy.$Proxy169.getOrderByKeyRequest(Unknown Source)
at com.fairpoint.om.workflow.orderprocessing.OrderStatusWorkFlow.orderManagementGetOrderByKeyRequest(OrderStatusWorkFlow.java:109)
at sun.reflect.GeneratedMethodAccessor1444.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.SyncReceive.messageDelivery(SyncReceive.java:63)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at sun.reflect.GeneratedMethodAccessor1363.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:281)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:15)
at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
at com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:30)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
at com.sun.proxy.$Proxy167.invoke(Unknown Source)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:62)
at com.bea.wli.knex.runtime.core.dispatcher.Dispatcher.remoteDispatch(Dispatcher.java:165)
at com.bea.wli.bpm.proxy.ProxyDispatcherBean.invoke(ProxyDispatcherBean.java:132)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl.invoke(ProxyDispatcher_9it87k_EOImpl.java:134)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl_1030_WLStub.invoke(Unknown Source)
at com.bea.wli.bpm.proxy.JpdProxyImpl.invoke(JpdProxyImpl.java:182)
at com.sun.proxy.$Proxy165.clientRequestwithReturn(Unknown Source)
at com.fairpoint.omintegration.integration.digidata.DigiDataUtility.getOrderDetails(DigiDataUtility.java:654)
at com.fairpoint.omintegration.integration.digidata.DigiDataHandler.processDigiDataRequest(DigiDataHandler.java:294)
at com.fairpoint.omintegration.integration.common.OutBoundEventHandler.decideDigidataCall(OutBoundEventHandler.java:416)
at com.fairpoint.omintegration.integration.mdbs.Digidata.onMessage(Digidata.java:76)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

IMPACT

1. The STUCK threads issue in the WebLogic tier is contributing to “slow task processing”:Tasks/Gateway Events are being fired off by M6 for processing and the custom Gateway Event handlers are sometimes getting STUCK mid-process when a Core M6 API is called.

2. Application servers will go in Warning state frequently.

CAUSE

The reason for STUCK threads seems to be improper use of HashMap vs. ConcurrentHashMap for multi-threaded applications.

The pattern on the stuck threads appears to be related to the HashMap conncurrency:

[STUCK] ExecuteThread: '129' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000114856800 nid=0x2bf6 runnable [0xfffffffcef4fa000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:737)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(ControlBeanContext.java:420)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForBean(JpdControlContainerContext.java:78)

The WLI platform implementation for JPD processing in version 8.1 used the “com.bea.wlw” package whereas the WLI 10.3.1 platform implementation was changed to use the “com.bea.wli.knex” package.

It seems 10.3 package implementation (one of the two packages listed above) is not thread-safe (it should use ConcurrentHashMap instead of HashMap).

Significant change between WLI 8.1 and WLI 10.3:

 Weblogic 8.1 packages 

    com.bea.wlw.runtime.jws.compiler.wsdl.WsdlDefinitions.compileSchemas(WsdlDefinitions.java:1336)    com.bea.xbean.schema.SchemaTypeSystemCompiler.compileImpl(SchemaTypeSystemCompiler.java:197)

Weblogic 10.3.1 packages

    com.bea.wli.knex.runtime.jws.compiler.wsdl.WsdlDefinitions.compileSchemas(WsdlDefinitions.java:1372)
org.apache.xmlbeans.impl.schema.SchemaTypeSystemCompiler.compileImpl(SchemaTypeSystemCompiler.java:308)

SOLUTION

To resolve this issue of corrupt HashMap caused by asynchronous concurrent access is to do either of the following:

1. Synchronize objects using the HashMap

OR

2. Implement ConcurrentHashMap instead of HashMap.

MetaSolv Support rebuilt/recompiled the mss_integration.ear after applying the WLW 10.3 patch ANM6 and instructed to follow as below:

1. Deploy new mss_integration.ear
2. Extract the MetaSolvSolutionInterfaces.jar from the mss_integration.ear
3. Replace old occurrence of MetaSolvSolutionInterfaces.jar in Workshop project
4. Start workshop with -clean and rebuild custom application

Also do below on your development/PROD environment.

1. Ensure you have ALL of the below patches as we have in the environment.

• Patch 8504591: SU Patch [43DL]: 8504591 - wlw9.2 - sqlparser.parse access thread unsafe method
• Patch 8305952: SU Patch [A486]: _providers map is synchronized
• Patch 9940818: SU Patch [ANM6]: Bug9940818 -> PSE for base bug9336580
• Patch 8184769: SU Patch [CQVL]: Use concurrent HashMap to stroe request attribtues to avoid infinite loops in multi-proc environments

2. After installing the patches, you must execute the following command – it will take a few seconds to complete:

workshop.exe -clean -initialize

3. Then launch the Workshop GUI and begin importing and compiling each and every custom custom application which is using below files.

The Weblogic patches will patch the following files

1. MetaSolvSolutionInterfaces.jar.
2. beehive-jdbc-control.jar 
3. beehive-controls.jar 
4. beehive-controls-1.0.2.2.war 
5. beehive-controls-1.0.2.2.ear 
6. weblogic-controls.jar 
7. weblogic-controls-10.2.war 
8. weblogic-controls-10.2.ear 

Deploy all the custom applications to Weblogic servers and now observe STUCK threads.

Note: Ref Bug for above issue Bug 21539933

Bug 21539933 - Stuck Thread due to HashMap in com.metasolv.api.control.OrderManagementBeanSev

Product 2267 - Oracle Communications MetaSolv Solution

Problem Description
===================
Abstract: Stuck Thread due to HashMap in com.metasolv.api.control.OrderManagementBean

[STUCK] ExecuteThread: '50' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=3 tid=0x0000000104c5a800 nid=0x1ff9 runnable
[0xfffffffcfaffa000]
   java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
at
org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(Con
trolBeanContext.java:420)
at
com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForB
ean(JpdControlContainerContext.java:78)
at
com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.setDelegateMapForBea
n(AnnotationOverrideHelper.java:49)
at
com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.getBeanAnnotationMap
(AnnotationOverrideHelper.java:58)
at
com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.getBeanAnnotationM
ap(JpdControlContainerContext.java:83)
at
org.apache.beehive.controls.runtime.bean.ControlBeanContext.getAnnotationMap(C
ontrolBeanContext.java:379)
at
org.apache.beehive.controls.runtime.bean.ClientInitializer.getAnnotationMap(Cl
ientInitializer.java:61)
at
com.metasolv.api.control.OrderManagementImplClientInitializer.initializeFields
(OrderManagementImplClientInitializer.java:179)
at
com.metasolv.api.control.OrderManagementImplClientInitializer.initialize(Order
ManagementImplClientInitializer.java:279)
at
com.metasolv.api.control.OrderManagementImplInitializer.initControls(OrderMana
gementImplInitializer.java:26)
at
org.apache.beehive.controls.runtime.bean.ImplInitializer.initialize(ImplInitia
lizer.java:35)
at
org.apache.beehive.controls.runtime.bean.ControlBean.ensureControl(ControlBean
.java:321)
- locked <0xfffffffd80be5518> (a
com.metasolv.api.control.OrderManagementBean)
at
com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagem
entBean.java:1191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.
java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:433)
at com.sun.proxy.$Proxy242.getOrderByKeyRequest(Unknown Source)
at
com.metasolv.api.COProvisioning.workflow.common.ProvisioningChangeEventHandler
.mslvOmGetOrderByKeyRequest11(ProvisioningChangeEventHandler.java:2467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:148)
at
com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at
com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
at
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
at
com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2ho
zgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at
com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatch
erBean.java:187)
at
com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBea
n.java:54)
at
com.bea.wli.knex.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatch
erBean.java:259)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.j
ava:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkMana
gerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Key WebLogic bugs related to the issue:

Bug 9336580 : JDBCCONTROL INIT HANGS PERIODICALLY RESULTING IN HIGH CPU

Product 5309 - Oracle Workshop for Weblogic

Problem Description
===================

This bug is related to a race condition in HashMap. Race conditions are often very difficult to reproduce.
The customer experienced this condition in production, performance throughput of the server increased, and they encountered the looping (High CPU) problem in HashMap.

Apply patch if encountered below STUCK threads stack trace.

[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010618c000 nid=0x41 runnable [0xffffffff67af7000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:745)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.getMethodPropertySet(ControlBeanContext.java:465)
at org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(JdbcControlImpl.java:258)
at org.apache.beehive.controls.system.jdbc.JdbcControlImpl.invoke(JdbcControlImpl.java:225)
at com.fairpoint.integration.activation.control.DSLServiceActivationBean.getWLTYPCount(DSLServiceActivationBean.java:5013)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:446)
at com.sun.proxy.$Proxy510.getWLTYPCount(Unknown Source)
at com.fairpoint.integration.activation.workflow.handler.ActivationOrderManager_DSL.isDSLAMReuse(ActivationOrderManager_DSL.java:699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.SyncReceive.messageDelivery(SyncReceive.java:63)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:62)
at com.bea.wli.knex.runtime.core.dispatcher.Dispatcher.remoteDispatch(Dispatcher.java:165)
at com.bea.wli.knex.runtime.core.dispatcher.ServiceHandleImpl.invoke(ServiceHandleImpl.java:460)
at com.bea.wli.knex.runtime.core.call.JavaCall.invoke(JavaCall.java:56)
at com.bea.wli.bpm.runtime.SubFlowCall.invoke(SubFlowCall.java:112)
at com.bea.wli.knex.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.java:1318)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.java:833)
at com.bea.wli.knex.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.java:1185)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.java:768)
at com.fairpoint.integration.activation.workflow.handler.ActivationOrderManager_DSLPControlBean.clientRequestwithReturn(ActivationOrderManager_DSLPControlBean.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:446)
at com.sun.proxy.$Proxy444.clientRequestwithReturn(Unknown Source)
at com.fairpoint.integration.activation.workflow.common.MSSOutboundEventHandler.activationOrderManager_DSLPControlClientRequestwithReturn1(MSSOutboundEventHandler.java:1512)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:148)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatcherBean.java:259)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Bug 20415750 : Stuck thread in beehive.controls.runtime.bean.ControlBean.getAnnotationMap

Product 5323 - Oracle WebLogic Integration

Problem Description
===================

Customer has  applied patch 43DL for Bug 8504591 but it only partially solved
the problem.

Indeed thread Stuck coming from "SQLParser" are now OK. Customer no longer
see this issue:

Stack trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.system.jdbc.parser.SqlParser.parse(SqlParser.java:
66)
   
org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(
JdbcControlImpl.java:273)

But regarding ThreadStuck coming from "getAnnotationMap" are still present:

<7 janv. 2015 15 h 39 CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "2 261" seconds working on the request "weblo
gic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@14dd41f", which is more
than the configured time (StuckThreadMaxTime) of "1 800" seconds. Stack
trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
   
org.apache.beehive.controls.runtime.bean.ControlBean.<init>(ControlBean.java:1
31)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:482)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:465)
       sun.reflect.GeneratedConstructorAccessor355.newInstance(Unknown
Source)
   
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
       java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   
org.apache.beehive.controls.spi.bean.JavaControlFactory.instantiate(JavaContro
lFactory.java:119)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.instantiate(DispCont
rolField.java:167)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.getInitialValue(Disp
ControlField.java:105)
   
com.bea.wli.knex.runtime.core.container.Container.initializeTarget(Container.j
ava:413)
   
com.bea.wli.knex.runtime.core.container.Container.createTarget(Container.java:
350)
   
com.bea.wli.knex.runtime.core.container.Container.getTarget(Container.java:332
)
   
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1113)
       com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
   
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
       com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
   
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
   
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
       sun.reflect.GeneratedMethodAccessor455.invoke(Unknown Source)
   
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
       java.lang.reflect.Method.invoke(Method.java:597)
   
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUs
ingReflection(AopUtils.java:281)
   
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocati
on.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

Diagnostic analysis confirming the issue is a bug
=================================================

Customer has  applied patch 43DL for Bug 8504591 but it only partially solved
the problem.

Indeed thread Stuck coming from "SQLParser" are now OK. Customer no longer
see this issue:

]", which is more than the configured time (StuckThreadMaxTime) of "1 800"
seconds. Stack trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.system.jdbc.parser.SqlParser.parse(SqlParser.java:
66)
   
org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(
JdbcControlImpl.java:273)

But regarding ThreadStuck coming from "getAnnotationMap" are still present:

<7 janv. 2015 15 h 39 CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "2 261" seconds working on the request "weblo
gic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@14dd41f", which is more
than the configured time (StuckThreadMaxTime) of "1 800" seconds. Stack
trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
   
org.apache.beehive.controls.runtime.bean.ControlBean.<init>(ControlBean.java:1
31)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:482)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:465)
       sun.reflect.GeneratedConstructorAccessor355.newInstance(Unknown
Source)
   
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
       java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   
org.apache.beehive.controls.spi.bean.JavaControlFactory.instantiate(JavaContro
lFactory.java:119)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.instantiate(DispCont
rolField.java:167)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.getInitialValue(Disp
ControlField.java:105)
   
com.bea.wli.knex.runtime.core.container.Container.initializeTarget(Container.j
ava:413)
   
com.bea.wli.knex.runtime.core.container.Container.createTarget(Container.java:
350)
   
com.bea.wli.knex.runtime.core.container.Container.getTarget(Container.java:332
)
   
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1113)
       com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
   
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
       com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
   
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
   
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
       sun.reflect.GeneratedMethodAccessor455.invoke(Unknown Source)
   
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
       java.lang.reflect.Method.invoke(Method.java:597)
   
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUs
ingReflection(AopUtils.java:281)
   
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocati
on.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

Bug 14169920 : [MENTORING] BEEHIVE STUCK THREAD IN HASHMAP AND HIGH CPU IN WLP APP

Product 5309 - Oracle Workshop for Weblogic

Problem Description
===================
Customer are having a performance issue in their production environment
where within half an hour of starting the server, they see the stuck thread:

<29/05/2012#08:43:31.268#+0200> <Error> <WebLogicServer> <BEA-000337>
<[STUCK] ExecuteThread: '194' for queue: 'weblogic.kernel.Default
(self-tuning)' has been busy for "659" seconds working on the request "Http
Request: /ASPFront/appmanager/ASPFront/front", which is more than the
configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.util.HashMap.containsKey(HashMap.java:383)
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(Control
Bean.java:737)
org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(Co
ntrolBeanContext.java:420)
weblogic.controls.container.WlsServletBeanContext.setDelegateMapForBean(WlsSe
rvletBeanContext.java:112)

CPU utilization goes upto 100% and their application becomes very slow.This
issue started happening after they made a few changes to their portal
application and deployed it. After this, they have started seeing this
issue.The Portal team would upload the details on the changes made in their
application.Also the other point to note is this happens only in the
production environment only under high load.The customer informed that they
were not able to replicate it consistently.



Hope above helps in fixing the STUCK thread issues on your M6 application, please leave your feedback or query