Metasolv 6.2.1- Multiple STUCK Threads due to HashMap in com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagementBean.java:1191) in WebLogic 10.3.0

PROBLEM

Multiple STUCK threads seen in Weblogic servers caused by custom java applications that calls the 'getOrderByKeyRequest' API - the STUCK thread seems to be tied to internal API code where the root of the issue may be improper use of HashMap versus something thread safe like ConcurrentHashMap

The issue is happening on the "getOrderByKeyRequest" API. Refer below stack trace.

[STUCK] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000108da6000 nid=0x665d runnable [0xfffffffdfd8f9000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:737)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(ControlBeanContext.java:420)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForBean(JpdControlContainerContext.java:78)
at com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.setDelegateMapForBean(AnnotationOverrideHelper.java:49)
at com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.getBeanAnnotationMap(AnnotationOverrideHelper.java:58)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.getBeanAnnotationMap(JpdControlContainerContext.java:83)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.getAnnotationMap(ControlBeanContext.java:379)
at org.apache.beehive.controls.runtime.bean.ClientInitializer.getAnnotationMap(ClientInitializer.java:61)
at com.metasolv.api.control.OrderManagementImplClientInitializer.initializeFields(OrderManagementImplClientInitializer.java:221)
at com.metasolv.api.control.OrderManagementImplClientInitializer.initialize(OrderManagementImplClientInitializer.java:279)
at com.metasolv.api.control.OrderManagementImplInitializer.initControls(OrderManagementImplInitializer.java:26)
at org.apache.beehive.controls.runtime.bean.ImplInitializer.initialize(ImplInitializer.java:35)
at org.apache.beehive.controls.runtime.bean.ControlBean.ensureControl(ControlBean.java:321)
- locked <0xfffffffebb181348> (a com.metasolv.api.control.OrderManagementBean)
at com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagementBean.java:1191)
at sun.reflect.GeneratedMethodAccessor1441.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:433)
at com.sun.proxy.$Proxy169.getOrderByKeyRequest(Unknown Source)
at com.fairpoint.om.workflow.orderprocessing.OrderStatusWorkFlow.orderManagementGetOrderByKeyRequest(OrderStatusWorkFlow.java:109)
at sun.reflect.GeneratedMethodAccessor1444.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.SyncReceive.messageDelivery(SyncReceive.java:63)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at sun.reflect.GeneratedMethodAccessor1363.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:281)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:15)
at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
at com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:30)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
at com.sun.proxy.$Proxy167.invoke(Unknown Source)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:62)
at com.bea.wli.knex.runtime.core.dispatcher.Dispatcher.remoteDispatch(Dispatcher.java:165)
at com.bea.wli.bpm.proxy.ProxyDispatcherBean.invoke(ProxyDispatcherBean.java:132)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl.invoke(ProxyDispatcher_9it87k_EOImpl.java:134)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl_1030_WLStub.invoke(Unknown Source)
at com.bea.wli.bpm.proxy.JpdProxyImpl.invoke(JpdProxyImpl.java:182)
at com.sun.proxy.$Proxy165.clientRequestwithReturn(Unknown Source)
at com.fairpoint.omintegration.integration.digidata.DigiDataUtility.getOrderDetails(DigiDataUtility.java:654)
at com.fairpoint.omintegration.integration.digidata.DigiDataHandler.processDigiDataRequest(DigiDataHandler.java:294)
at com.fairpoint.omintegration.integration.common.OutBoundEventHandler.decideDigidataCall(OutBoundEventHandler.java:416)
at com.fairpoint.omintegration.integration.mdbs.Digidata.onMessage(Digidata.java:76)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

IMPACT

1. The STUCK threads issue in the WebLogic tier is contributing to “slow task processing”:Tasks/Gateway Events are being fired off by M6 for processing and the custom Gateway Event handlers are sometimes getting STUCK mid-process when a Core M6 API is called.

2. Application servers will go in Warning state frequently.

CAUSE

The reason for STUCK threads seems to be improper use of HashMap vs. ConcurrentHashMap for multi-threaded applications.

The pattern on the stuck threads appears to be related to the HashMap conncurrency:

[STUCK] ExecuteThread: '129' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000114856800 nid=0x2bf6 runnable [0xfffffffcef4fa000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:737)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(ControlBeanContext.java:420)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForBean(JpdControlContainerContext.java:78)

The WLI platform implementation for JPD processing in version 8.1 used the “com.bea.wlw” package whereas the WLI 10.3.1 platform implementation was changed to use the “com.bea.wli.knex” package.

It seems 10.3 package implementation (one of the two packages listed above) is not thread-safe (it should use ConcurrentHashMap instead of HashMap).

Significant change between WLI 8.1 and WLI 10.3:

 Weblogic 8.1 packages 

    com.bea.wlw.runtime.jws.compiler.wsdl.WsdlDefinitions.compileSchemas(WsdlDefinitions.java:1336)    com.bea.xbean.schema.SchemaTypeSystemCompiler.compileImpl(SchemaTypeSystemCompiler.java:197)

Weblogic 10.3.1 packages

    com.bea.wli.knex.runtime.jws.compiler.wsdl.WsdlDefinitions.compileSchemas(WsdlDefinitions.java:1372)
org.apache.xmlbeans.impl.schema.SchemaTypeSystemCompiler.compileImpl(SchemaTypeSystemCompiler.java:308)

SOLUTION

To resolve this issue of corrupt HashMap caused by asynchronous concurrent access is to do either of the following:

1. Synchronize objects using the HashMap

OR

2. Implement ConcurrentHashMap instead of HashMap.

MetaSolv Support rebuilt/recompiled the mss_integration.ear after applying the WLW 10.3 patch ANM6 and instructed to follow as below:

1. Deploy new mss_integration.ear
2. Extract the MetaSolvSolutionInterfaces.jar from the mss_integration.ear
3. Replace old occurrence of MetaSolvSolutionInterfaces.jar in Workshop project
4. Start workshop with -clean and rebuild custom application

Also do below on your development/PROD environment.

1. Ensure you have ALL of the below patches as we have in the environment.

• Patch 8504591: SU Patch [43DL]: 8504591 - wlw9.2 - sqlparser.parse access thread unsafe method
• Patch 8305952: SU Patch [A486]: _providers map is synchronized
• Patch 9940818: SU Patch [ANM6]: Bug9940818 -> PSE for base bug9336580
• Patch 8184769: SU Patch [CQVL]: Use concurrent HashMap to stroe request attribtues to avoid infinite loops in multi-proc environments

2. After installing the patches, you must execute the following command – it will take a few seconds to complete:

workshop.exe -clean -initialize

3. Then launch the Workshop GUI and begin importing and compiling each and every custom custom application which is using below files.

The Weblogic patches will patch the following files

1. MetaSolvSolutionInterfaces.jar.
2. beehive-jdbc-control.jar 
3. beehive-controls.jar 
4. beehive-controls-1.0.2.2.war 
5. beehive-controls-1.0.2.2.ear 
6. weblogic-controls.jar 
7. weblogic-controls-10.2.war 
8. weblogic-controls-10.2.ear 

Deploy all the custom applications to Weblogic servers and now observe STUCK threads.

Note: Ref Bug for above issue Bug 21539933

Bug 21539933 - Stuck Thread due to HashMap in com.metasolv.api.control.OrderManagementBeanSev

Product 2267 - Oracle Communications MetaSolv Solution

Problem Description
===================
Abstract: Stuck Thread due to HashMap in com.metasolv.api.control.OrderManagementBean

[STUCK] ExecuteThread: '50' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=3 tid=0x0000000104c5a800 nid=0x1ff9 runnable
[0xfffffffcfaffa000]
   java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
at
org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(Con
trolBeanContext.java:420)
at
com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForB
ean(JpdControlContainerContext.java:78)
at
com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.setDelegateMapForBea
n(AnnotationOverrideHelper.java:49)
at
com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.getBeanAnnotationMap
(AnnotationOverrideHelper.java:58)
at
com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.getBeanAnnotationM
ap(JpdControlContainerContext.java:83)
at
org.apache.beehive.controls.runtime.bean.ControlBeanContext.getAnnotationMap(C
ontrolBeanContext.java:379)
at
org.apache.beehive.controls.runtime.bean.ClientInitializer.getAnnotationMap(Cl
ientInitializer.java:61)
at
com.metasolv.api.control.OrderManagementImplClientInitializer.initializeFields
(OrderManagementImplClientInitializer.java:179)
at
com.metasolv.api.control.OrderManagementImplClientInitializer.initialize(Order
ManagementImplClientInitializer.java:279)
at
com.metasolv.api.control.OrderManagementImplInitializer.initControls(OrderMana
gementImplInitializer.java:26)
at
org.apache.beehive.controls.runtime.bean.ImplInitializer.initialize(ImplInitia
lizer.java:35)
at
org.apache.beehive.controls.runtime.bean.ControlBean.ensureControl(ControlBean
.java:321)
- locked <0xfffffffd80be5518> (a
com.metasolv.api.control.OrderManagementBean)
at
com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagem
entBean.java:1191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.
java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:433)
at com.sun.proxy.$Proxy242.getOrderByKeyRequest(Unknown Source)
at
com.metasolv.api.COProvisioning.workflow.common.ProvisioningChangeEventHandler
.mslvOmGetOrderByKeyRequest11(ProvisioningChangeEventHandler.java:2467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:148)
at
com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at
com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
at
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
at
com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2ho
zgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at
com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatch
erBean.java:187)
at
com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBea
n.java:54)
at
com.bea.wli.knex.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatch
erBean.java:259)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.j
ava:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkMana
gerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Key WebLogic bugs related to the issue:

Bug 9336580 : JDBCCONTROL INIT HANGS PERIODICALLY RESULTING IN HIGH CPU

Product 5309 - Oracle Workshop for Weblogic

Problem Description
===================

This bug is related to a race condition in HashMap. Race conditions are often very difficult to reproduce.
The customer experienced this condition in production, performance throughput of the server increased, and they encountered the looping (High CPU) problem in HashMap.

Apply patch if encountered below STUCK threads stack trace.

[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010618c000 nid=0x41 runnable [0xffffffff67af7000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:745)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.getMethodPropertySet(ControlBeanContext.java:465)
at org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(JdbcControlImpl.java:258)
at org.apache.beehive.controls.system.jdbc.JdbcControlImpl.invoke(JdbcControlImpl.java:225)
at com.fairpoint.integration.activation.control.DSLServiceActivationBean.getWLTYPCount(DSLServiceActivationBean.java:5013)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:446)
at com.sun.proxy.$Proxy510.getWLTYPCount(Unknown Source)
at com.fairpoint.integration.activation.workflow.handler.ActivationOrderManager_DSL.isDSLAMReuse(ActivationOrderManager_DSL.java:699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.SyncReceive.messageDelivery(SyncReceive.java:63)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:62)
at com.bea.wli.knex.runtime.core.dispatcher.Dispatcher.remoteDispatch(Dispatcher.java:165)
at com.bea.wli.knex.runtime.core.dispatcher.ServiceHandleImpl.invoke(ServiceHandleImpl.java:460)
at com.bea.wli.knex.runtime.core.call.JavaCall.invoke(JavaCall.java:56)
at com.bea.wli.bpm.runtime.SubFlowCall.invoke(SubFlowCall.java:112)
at com.bea.wli.knex.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.java:1318)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.java:833)
at com.bea.wli.knex.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.java:1185)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.java:768)
at com.fairpoint.integration.activation.workflow.handler.ActivationOrderManager_DSLPControlBean.clientRequestwithReturn(ActivationOrderManager_DSLPControlBean.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:446)
at com.sun.proxy.$Proxy444.clientRequestwithReturn(Unknown Source)
at com.fairpoint.integration.activation.workflow.common.MSSOutboundEventHandler.activationOrderManager_DSLPControlClientRequestwithReturn1(MSSOutboundEventHandler.java:1512)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:148)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatcherBean.java:259)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Bug 20415750 : Stuck thread in beehive.controls.runtime.bean.ControlBean.getAnnotationMap

Product 5323 - Oracle WebLogic Integration

Problem Description
===================

Customer has  applied patch 43DL for Bug 8504591 but it only partially solved
the problem.

Indeed thread Stuck coming from "SQLParser" are now OK. Customer no longer
see this issue:

Stack trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.system.jdbc.parser.SqlParser.parse(SqlParser.java:
66)
   
org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(
JdbcControlImpl.java:273)

But regarding ThreadStuck coming from "getAnnotationMap" are still present:

<7 janv. 2015 15 h 39 CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "2 261" seconds working on the request "weblo
gic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@14dd41f", which is more
than the configured time (StuckThreadMaxTime) of "1 800" seconds. Stack
trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
   
org.apache.beehive.controls.runtime.bean.ControlBean.<init>(ControlBean.java:1
31)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:482)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:465)
       sun.reflect.GeneratedConstructorAccessor355.newInstance(Unknown
Source)
   
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
       java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   
org.apache.beehive.controls.spi.bean.JavaControlFactory.instantiate(JavaContro
lFactory.java:119)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.instantiate(DispCont
rolField.java:167)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.getInitialValue(Disp
ControlField.java:105)
   
com.bea.wli.knex.runtime.core.container.Container.initializeTarget(Container.j
ava:413)
   
com.bea.wli.knex.runtime.core.container.Container.createTarget(Container.java:
350)
   
com.bea.wli.knex.runtime.core.container.Container.getTarget(Container.java:332
)
   
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1113)
       com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
   
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
       com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
   
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
   
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
       sun.reflect.GeneratedMethodAccessor455.invoke(Unknown Source)
   
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
       java.lang.reflect.Method.invoke(Method.java:597)
   
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUs
ingReflection(AopUtils.java:281)
   
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocati
on.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

Diagnostic analysis confirming the issue is a bug
=================================================

Customer has  applied patch 43DL for Bug 8504591 but it only partially solved
the problem.

Indeed thread Stuck coming from "SQLParser" are now OK. Customer no longer
see this issue:

]", which is more than the configured time (StuckThreadMaxTime) of "1 800"
seconds. Stack trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.system.jdbc.parser.SqlParser.parse(SqlParser.java:
66)
   
org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(
JdbcControlImpl.java:273)

But regarding ThreadStuck coming from "getAnnotationMap" are still present:

<7 janv. 2015 15 h 39 CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "2 261" seconds working on the request "weblo
gic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@14dd41f", which is more
than the configured time (StuckThreadMaxTime) of "1 800" seconds. Stack
trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
   
org.apache.beehive.controls.runtime.bean.ControlBean.<init>(ControlBean.java:1
31)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:482)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:465)
       sun.reflect.GeneratedConstructorAccessor355.newInstance(Unknown
Source)
   
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
       java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   
org.apache.beehive.controls.spi.bean.JavaControlFactory.instantiate(JavaContro
lFactory.java:119)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.instantiate(DispCont
rolField.java:167)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.getInitialValue(Disp
ControlField.java:105)
   
com.bea.wli.knex.runtime.core.container.Container.initializeTarget(Container.j
ava:413)
   
com.bea.wli.knex.runtime.core.container.Container.createTarget(Container.java:
350)
   
com.bea.wli.knex.runtime.core.container.Container.getTarget(Container.java:332
)
   
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1113)
       com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
   
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
       com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
   
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
   
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
       sun.reflect.GeneratedMethodAccessor455.invoke(Unknown Source)
   
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
       java.lang.reflect.Method.invoke(Method.java:597)
   
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUs
ingReflection(AopUtils.java:281)
   
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocati
on.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

Bug 14169920 : [MENTORING] BEEHIVE STUCK THREAD IN HASHMAP AND HIGH CPU IN WLP APP

Product 5309 - Oracle Workshop for Weblogic

Problem Description
===================
Customer are having a performance issue in their production environment
where within half an hour of starting the server, they see the stuck thread:

<29/05/2012#08:43:31.268#+0200> <Error> <WebLogicServer> <BEA-000337>
<[STUCK] ExecuteThread: '194' for queue: 'weblogic.kernel.Default
(self-tuning)' has been busy for "659" seconds working on the request "Http
Request: /ASPFront/appmanager/ASPFront/front", which is more than the
configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.util.HashMap.containsKey(HashMap.java:383)
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(Control
Bean.java:737)
org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(Co
ntrolBeanContext.java:420)
weblogic.controls.container.WlsServletBeanContext.setDelegateMapForBean(WlsSe
rvletBeanContext.java:112)

CPU utilization goes upto 100% and their application becomes very slow.This
issue started happening after they made a few changes to their portal
application and deployed it. After this, they have started seeing this
issue.The Portal team would upload the details on the changes made in their
application.Also the other point to note is this happens only in the
production environment only under high load.The customer informed that they
were not able to replicate it consistently.



Hope above helps in fixing the STUCK thread issues on your M6 application, please leave your feedback or query