Metasolv M6 GUI white screen issue after logging with tbs.exe caused by insufficient permission

PROBLEM

After logging in to tbs.exe, M6 GUI opens with white screen  as below with the following error logged to mss logs.

Exception:

####<Aug 5, 2015 11:45:44 AM EDT> <Info> <ServletContext-/main> <nh05> <GUI_srvr_1.1> <nh05_gateway_Main> <weblogic> <> <> <1438789544667> <BEA-000000> <action: MSLVActionServlet: Unable to determine global name.java.rmi.AccessException: [EJB:010160]Security Violation: User: 'weblogic' has insufficient permission to access EJB: type=<ejb>, application=cluster-nur, module=System-ejb.jar, ejb=Security, method=getGlobalName, methodInterface=Remote, signature={}.>

CAUSE

MSS 6.2.x requires weblogic user to be assigned 'MSSRole' global role. If this role is missing or weblogic user is missing this role then this issue will occur.
This role is created by M6 installer in addition to 'MSSJMSRole' & 'APP_JMS' user.

Note: The JMS user, APP_JMS, has been created for security purposes. Do not modify or remove this user.

SOLUTION

Use the WebLogic Administration Console & steps listed below to configure the missing configurations listed above.
1. Create Global Role 'MSSRole' & add weblogic user to it.
Security Realms > myrealm > Roles and Policies > Realm Roles > Global Roles > Click Roles > New > Name (MSSRole) > Add (User: xxxxxxx)
Note: xxxxxxx is the actual name of the weblogic admin user. Here in above case it is weblogic.

2. Create Global Role 'MSSJMSRole' & add 'IntegrationAdministrators' group to it.
Security Realms > myrealm > Roles and Policies > Realm Roles > Global Roles > Click Roles > New > Name (MSSJMSRole) > Add (Group:  IntegrationAdministrators)

3. Create User APP_JMS & assign it to 'IntegrationAdministrators' group.
Security Realms > myrealm >Users and Groups > Users > New > Name (APP_JMS) , Password (mss62jms) > Group 'IntegrationAdministrators'

Hope above helps in fixing the GUI white screen issues on your M6 application, please leave your feedback or query

Metasolv 6.2.1- Multiple STUCK Threads due to HashMap in com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagementBean.java:1191) in WebLogic 10.3.0

PROBLEM

Multiple STUCK threads seen in Weblogic servers caused by custom java applications that calls the 'getOrderByKeyRequest' API - the STUCK thread seems to be tied to internal API code where the root of the issue may be improper use of HashMap versus something thread safe like ConcurrentHashMap

The issue is happening on the "getOrderByKeyRequest" API. Refer below stack trace.

[STUCK] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000108da6000 nid=0x665d runnable [0xfffffffdfd8f9000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:737)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(ControlBeanContext.java:420)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForBean(JpdControlContainerContext.java:78)
at com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.setDelegateMapForBean(AnnotationOverrideHelper.java:49)
at com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.getBeanAnnotationMap(AnnotationOverrideHelper.java:58)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.getBeanAnnotationMap(JpdControlContainerContext.java:83)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.getAnnotationMap(ControlBeanContext.java:379)
at org.apache.beehive.controls.runtime.bean.ClientInitializer.getAnnotationMap(ClientInitializer.java:61)
at com.metasolv.api.control.OrderManagementImplClientInitializer.initializeFields(OrderManagementImplClientInitializer.java:221)
at com.metasolv.api.control.OrderManagementImplClientInitializer.initialize(OrderManagementImplClientInitializer.java:279)
at com.metasolv.api.control.OrderManagementImplInitializer.initControls(OrderManagementImplInitializer.java:26)
at org.apache.beehive.controls.runtime.bean.ImplInitializer.initialize(ImplInitializer.java:35)
at org.apache.beehive.controls.runtime.bean.ControlBean.ensureControl(ControlBean.java:321)
- locked <0xfffffffebb181348> (a com.metasolv.api.control.OrderManagementBean)
at com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagementBean.java:1191)
at sun.reflect.GeneratedMethodAccessor1441.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:433)
at com.sun.proxy.$Proxy169.getOrderByKeyRequest(Unknown Source)
at com.fairpoint.om.workflow.orderprocessing.OrderStatusWorkFlow.orderManagementGetOrderByKeyRequest(OrderStatusWorkFlow.java:109)
at sun.reflect.GeneratedMethodAccessor1444.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.SyncReceive.messageDelivery(SyncReceive.java:63)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at sun.reflect.GeneratedMethodAccessor1363.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:281)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:15)
at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
at com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:30)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
at com.sun.proxy.$Proxy167.invoke(Unknown Source)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:62)
at com.bea.wli.knex.runtime.core.dispatcher.Dispatcher.remoteDispatch(Dispatcher.java:165)
at com.bea.wli.bpm.proxy.ProxyDispatcherBean.invoke(ProxyDispatcherBean.java:132)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl.invoke(ProxyDispatcher_9it87k_EOImpl.java:134)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at com.bea.wli.bpm.proxy.ProxyDispatcher_9it87k_EOImpl_1030_WLStub.invoke(Unknown Source)
at com.bea.wli.bpm.proxy.JpdProxyImpl.invoke(JpdProxyImpl.java:182)
at com.sun.proxy.$Proxy165.clientRequestwithReturn(Unknown Source)
at com.fairpoint.omintegration.integration.digidata.DigiDataUtility.getOrderDetails(DigiDataUtility.java:654)
at com.fairpoint.omintegration.integration.digidata.DigiDataHandler.processDigiDataRequest(DigiDataHandler.java:294)
at com.fairpoint.omintegration.integration.common.OutBoundEventHandler.decideDigidataCall(OutBoundEventHandler.java:416)
at com.fairpoint.omintegration.integration.mdbs.Digidata.onMessage(Digidata.java:76)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

IMPACT

1. The STUCK threads issue in the WebLogic tier is contributing to “slow task processing”:Tasks/Gateway Events are being fired off by M6 for processing and the custom Gateway Event handlers are sometimes getting STUCK mid-process when a Core M6 API is called.

2. Application servers will go in Warning state frequently.

CAUSE

The reason for STUCK threads seems to be improper use of HashMap vs. ConcurrentHashMap for multi-threaded applications.

The pattern on the stuck threads appears to be related to the HashMap conncurrency:

[STUCK] ExecuteThread: '129' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000114856800 nid=0x2bf6 runnable [0xfffffffcef4fa000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:737)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(ControlBeanContext.java:420)
at com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForBean(JpdControlContainerContext.java:78)

The WLI platform implementation for JPD processing in version 8.1 used the “com.bea.wlw” package whereas the WLI 10.3.1 platform implementation was changed to use the “com.bea.wli.knex” package.

It seems 10.3 package implementation (one of the two packages listed above) is not thread-safe (it should use ConcurrentHashMap instead of HashMap).

Significant change between WLI 8.1 and WLI 10.3:

 Weblogic 8.1 packages 

    com.bea.wlw.runtime.jws.compiler.wsdl.WsdlDefinitions.compileSchemas(WsdlDefinitions.java:1336)    com.bea.xbean.schema.SchemaTypeSystemCompiler.compileImpl(SchemaTypeSystemCompiler.java:197)

Weblogic 10.3.1 packages

    com.bea.wli.knex.runtime.jws.compiler.wsdl.WsdlDefinitions.compileSchemas(WsdlDefinitions.java:1372)
org.apache.xmlbeans.impl.schema.SchemaTypeSystemCompiler.compileImpl(SchemaTypeSystemCompiler.java:308)

SOLUTION

To resolve this issue of corrupt HashMap caused by asynchronous concurrent access is to do either of the following:

1. Synchronize objects using the HashMap

OR

2. Implement ConcurrentHashMap instead of HashMap.

MetaSolv Support rebuilt/recompiled the mss_integration.ear after applying the WLW 10.3 patch ANM6 and instructed to follow as below:

1. Deploy new mss_integration.ear
2. Extract the MetaSolvSolutionInterfaces.jar from the mss_integration.ear
3. Replace old occurrence of MetaSolvSolutionInterfaces.jar in Workshop project
4. Start workshop with -clean and rebuild custom application

Also do below on your development/PROD environment.

1. Ensure you have ALL of the below patches as we have in the environment.

• Patch 8504591: SU Patch [43DL]: 8504591 - wlw9.2 - sqlparser.parse access thread unsafe method
• Patch 8305952: SU Patch [A486]: _providers map is synchronized
• Patch 9940818: SU Patch [ANM6]: Bug9940818 -> PSE for base bug9336580
• Patch 8184769: SU Patch [CQVL]: Use concurrent HashMap to stroe request attribtues to avoid infinite loops in multi-proc environments

2. After installing the patches, you must execute the following command – it will take a few seconds to complete:

workshop.exe -clean -initialize

3. Then launch the Workshop GUI and begin importing and compiling each and every custom custom application which is using below files.

The Weblogic patches will patch the following files

1. MetaSolvSolutionInterfaces.jar.
2. beehive-jdbc-control.jar 
3. beehive-controls.jar 
4. beehive-controls-1.0.2.2.war 
5. beehive-controls-1.0.2.2.ear 
6. weblogic-controls.jar 
7. weblogic-controls-10.2.war 
8. weblogic-controls-10.2.ear 

Deploy all the custom applications to Weblogic servers and now observe STUCK threads.

Note: Ref Bug for above issue Bug 21539933

Bug 21539933 - Stuck Thread due to HashMap in com.metasolv.api.control.OrderManagementBeanSev

Product 2267 - Oracle Communications MetaSolv Solution

Problem Description
===================
Abstract: Stuck Thread due to HashMap in com.metasolv.api.control.OrderManagementBean

[STUCK] ExecuteThread: '50' for queue: 'weblogic.kernel.Default
(self-tuning)'" daemon prio=3 tid=0x0000000104c5a800 nid=0x1ff9 runnable
[0xfffffffcfaffa000]
   java.lang.Thread.State: RUNNABLE
at java.util.HashMap.getEntry(HashMap.java:347)
at java.util.HashMap.containsKey(HashMap.java:335)
at
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
at
org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(Con
trolBeanContext.java:420)
at
com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.setDelegateMapForB
ean(JpdControlContainerContext.java:78)
at
com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.setDelegateMapForBea
n(AnnotationOverrideHelper.java:49)
at
com.bea.wli.knex.runtime.beehive.AnnotationOverrideHelper.getBeanAnnotationMap
(AnnotationOverrideHelper.java:58)
at
com.bea.wli.knex.runtime.beehive.JpdControlContainerContext.getBeanAnnotationM
ap(JpdControlContainerContext.java:83)
at
org.apache.beehive.controls.runtime.bean.ControlBeanContext.getAnnotationMap(C
ontrolBeanContext.java:379)
at
org.apache.beehive.controls.runtime.bean.ClientInitializer.getAnnotationMap(Cl
ientInitializer.java:61)
at
com.metasolv.api.control.OrderManagementImplClientInitializer.initializeFields
(OrderManagementImplClientInitializer.java:179)
at
com.metasolv.api.control.OrderManagementImplClientInitializer.initialize(Order
ManagementImplClientInitializer.java:279)
at
com.metasolv.api.control.OrderManagementImplInitializer.initControls(OrderMana
gementImplInitializer.java:26)
at
org.apache.beehive.controls.runtime.bean.ImplInitializer.initialize(ImplInitia
lizer.java:35)
at
org.apache.beehive.controls.runtime.bean.ControlBean.ensureControl(ControlBean
.java:321)
- locked <0xfffffffd80be5518> (a
com.metasolv.api.control.OrderManagementBean)
at
com.metasolv.api.control.OrderManagementBean.getOrderByKeyRequest(OrderManagem
entBean.java:1191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.
java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:433)
at com.sun.proxy.$Proxy242.getOrderByKeyRequest(Unknown Source)
at
com.metasolv.api.COProvisioning.workflow.common.ProvisioningChangeEventHandler
.mslvOmGetOrderByKeyRequest11(ProvisioningChangeEventHandler.java:2467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:148)
at
com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at
com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
at
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
at
com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2ho
zgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at
com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatch
erBean.java:187)
at
com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBea
n.java:54)
at
com.bea.wli.knex.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatch
erBean.java:259)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at
weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.j
ava:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkMana
gerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Key WebLogic bugs related to the issue:

Bug 9336580 : JDBCCONTROL INIT HANGS PERIODICALLY RESULTING IN HIGH CPU

Product 5309 - Oracle Workshop for Weblogic

Problem Description
===================

This bug is related to a race condition in HashMap. Race conditions are often very difficult to reproduce.
The customer experienced this condition in production, performance throughput of the server increased, and they encountered the looping (High CPU) problem in HashMap.

Apply patch if encountered below STUCK threads stack trace.

[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010618c000 nid=0x41 runnable [0xffffffff67af7000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlBean.java:745)
at org.apache.beehive.controls.runtime.bean.ControlBeanContext.getMethodPropertySet(ControlBeanContext.java:465)
at org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(JdbcControlImpl.java:258)
at org.apache.beehive.controls.system.jdbc.JdbcControlImpl.invoke(JdbcControlImpl.java:225)
at com.fairpoint.integration.activation.control.DSLServiceActivationBean.getWLTYPCount(DSLServiceActivationBean.java:5013)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:446)
at com.sun.proxy.$Proxy510.getWLTYPCount(Unknown Source)
at com.fairpoint.integration.activation.workflow.handler.ActivationOrderManager_DSL.isDSLAMReuse(ActivationOrderManager_DSL.java:699)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.SyncReceive.messageDelivery(SyncReceive.java:63)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcherBean.invoke(SyncDispatcherBean.java:168)
at com.bea.wli.knex.runtime.core.bean.SyncDispatcher_k1mrl8_EOImpl.invoke(SyncDispatcher_k1mrl8_EOImpl.java:62)
at com.bea.wli.knex.runtime.core.dispatcher.Dispatcher.remoteDispatch(Dispatcher.java:165)
at com.bea.wli.knex.runtime.core.dispatcher.ServiceHandleImpl.invoke(ServiceHandleImpl.java:460)
at com.bea.wli.knex.runtime.core.call.JavaCall.invoke(JavaCall.java:56)
at com.bea.wli.bpm.runtime.SubFlowCall.invoke(SubFlowCall.java:112)
at com.bea.wli.knex.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.java:1318)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.java:833)
at com.bea.wli.knex.runtime.core.control.ServiceControlImpl.invoke(ServiceControlImpl.java:1185)
at com.bea.control.ProcessControlImpl.invoke(ProcessControlImpl.java:768)
at com.fairpoint.integration.activation.workflow.handler.ActivationOrderManager_DSLPControlBean.clientRequestwithReturn(ActivationOrderManager_DSLPControlBean.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invokeBeehiveControl(JcsProxy.java:666)
at com.bea.wli.knex.runtime.jcs.container.JcsProxy.invoke(JcsProxy.java:446)
at com.sun.proxy.$Proxy444.clientRequestwithReturn(Unknown Source)
at com.fairpoint.integration.activation.workflow.common.MSSOutboundEventHandler.activationOrderManager_DSLPControlClientRequestwithReturn1(MSSOutboundEventHandler.java:1512)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.bea.wli.bpm.runtime.Perform.invoke(Perform.java:39)
at com.bea.wli.bpm.runtime.Perform.execute(Perform.java:50)
at com.bea.wli.bpm.runtime.Receive.messageDelivery(Receive.java:148)
at com.bea.wli.bpm.runtime.ProcessState.processMessage(ProcessState.java:237)
at com.bea.wli.bpm.runtime.ProcessState.dispatchRequest(ProcessState.java:261)
at com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:1131)
at com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
at com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
at com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
at com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerBean.java:224)
at com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.java:136)
at com.bea.wlwgen.StatelessContainer_2hozgx_ELOImpl.invoke(StatelessContainer_2hozgx_ELOImpl.java:137)
at com.bea.wlwgen.SLSBContAdpt.invokeOnBean(SLSBContAdpt.java:29)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.runAsInvoke(BaseDispatcherBean.java:187)
at com.bea.wli.knex.runtime.core.bean.BaseDispatcherBean.invoke(BaseDispatcherBean.java:54)
at com.bea.wli.knex.runtime.core.bean.AsyncDispatcherBean.onMessage(AsyncDispatcherBean.java:259)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Bug 20415750 : Stuck thread in beehive.controls.runtime.bean.ControlBean.getAnnotationMap

Product 5323 - Oracle WebLogic Integration

Problem Description
===================

Customer has  applied patch 43DL for Bug 8504591 but it only partially solved
the problem.

Indeed thread Stuck coming from "SQLParser" are now OK. Customer no longer
see this issue:

Stack trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.system.jdbc.parser.SqlParser.parse(SqlParser.java:
66)
   
org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(
JdbcControlImpl.java:273)

But regarding ThreadStuck coming from "getAnnotationMap" are still present:

<7 janv. 2015 15 h 39 CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "2 261" seconds working on the request "weblo
gic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@14dd41f", which is more
than the configured time (StuckThreadMaxTime) of "1 800" seconds. Stack
trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
   
org.apache.beehive.controls.runtime.bean.ControlBean.<init>(ControlBean.java:1
31)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:482)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:465)
       sun.reflect.GeneratedConstructorAccessor355.newInstance(Unknown
Source)
   
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
       java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   
org.apache.beehive.controls.spi.bean.JavaControlFactory.instantiate(JavaContro
lFactory.java:119)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.instantiate(DispCont
rolField.java:167)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.getInitialValue(Disp
ControlField.java:105)
   
com.bea.wli.knex.runtime.core.container.Container.initializeTarget(Container.j
ava:413)
   
com.bea.wli.knex.runtime.core.container.Container.createTarget(Container.java:
350)
   
com.bea.wli.knex.runtime.core.container.Container.getTarget(Container.java:332
)
   
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1113)
       com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
   
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
       com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
   
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
   
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
       sun.reflect.GeneratedMethodAccessor455.invoke(Unknown Source)
   
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
       java.lang.reflect.Method.invoke(Method.java:597)
   
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUs
ingReflection(AopUtils.java:281)
   
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocati
on.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

Diagnostic analysis confirming the issue is a bug
=================================================

Customer has  applied patch 43DL for Bug 8504591 but it only partially solved
the problem.

Indeed thread Stuck coming from "SQLParser" are now OK. Customer no longer
see this issue:

]", which is more than the configured time (StuckThreadMaxTime) of "1 800"
seconds. Stack trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.system.jdbc.parser.SqlParser.parse(SqlParser.java:
66)
   
org.apache.beehive.controls.system.jdbc.JdbcControlImpl.execPreparedStatement(
JdbcControlImpl.java:273)

But regarding ThreadStuck coming from "getAnnotationMap" are still present:

<7 janv. 2015 15 h 39 CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK]
ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)' has
been busy for "2 261" seconds working on the request "weblo
gic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@14dd41f", which is more
than the configured time (StuckThreadMaxTime) of "1 800" seconds. Stack
trace:
       java.util.HashMap.getEntry(HashMap.java:347)
       java.util.HashMap.containsKey(HashMap.java:335)
   
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(ControlB
ean.java:737)
   
org.apache.beehive.controls.runtime.bean.ControlBean.<init>(ControlBean.java:1
31)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:482)
       controls.portage.PortageFacadeBean.<init>(PortageFacadeBean.java:465)
       sun.reflect.GeneratedConstructorAccessor355.newInstance(Unknown
Source)
   
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
rAccessorImpl.java:27)
       java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   
org.apache.beehive.controls.spi.bean.JavaControlFactory.instantiate(JavaContro
lFactory.java:119)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.instantiate(DispCont
rolField.java:167)
   
com.bea.wli.knex.runtime.core.dispatcher.DispControlField.getInitialValue(Disp
ControlField.java:105)
   
com.bea.wli.knex.runtime.core.container.Container.initializeTarget(Container.j
ava:413)
   
com.bea.wli.knex.runtime.core.container.Container.createTarget(Container.java:
350)
   
com.bea.wli.knex.runtime.core.container.Container.getTarget(Container.java:332
)
   
com.bea.wli.bpm.runtime.JpdContainer.dispatchProcessRequest(JpdContainer.java:
1113)
       com.bea.wli.bpm.runtime.JpdContainer.preInvoke(JpdContainer.java:1095)
   
com.bea.wli.knex.runtime.core.container.Invocable.invoke(Invocable.java:248)
       com.bea.wli.bpm.runtime.JpdContainer.invoke(JpdContainer.java:827)
   
com.bea.wli.knex.runtime.core.bean.BaseContainerBean.invokeBase(BaseContainerB
ean.java:224)
   
com.bea.wli.knex.runtime.core.bean.SLSBContainerBean.invoke(SLSBContainerBean.
java:136)
       sun.reflect.GeneratedMethodAccessor455.invoke(Unknown Source)
   
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
ava:25)
       java.lang.reflect.Method.invoke(Method.java:597)
   
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUs
ingReflection(AopUtils.java:281)
   
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocati
on.invokeJoinpoint(ReflectiveMethodInvocation.java:187)

Bug 14169920 : [MENTORING] BEEHIVE STUCK THREAD IN HASHMAP AND HIGH CPU IN WLP APP

Product 5309 - Oracle Workshop for Weblogic

Problem Description
===================
Customer are having a performance issue in their production environment
where within half an hour of starting the server, they see the stuck thread:

<29/05/2012#08:43:31.268#+0200> <Error> <WebLogicServer> <BEA-000337>
<[STUCK] ExecuteThread: '194' for queue: 'weblogic.kernel.Default
(self-tuning)' has been busy for "659" seconds working on the request "Http
Request: /ASPFront/appmanager/ASPFront/front", which is more than the
configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.util.HashMap.containsKey(HashMap.java:383)
org.apache.beehive.controls.runtime.bean.ControlBean.getAnnotationMap(Control
Bean.java:737)
org.apache.beehive.controls.runtime.bean.ControlBeanContext.setDelegateMap(Co
ntrolBeanContext.java:420)
weblogic.controls.container.WlsServletBeanContext.setDelegateMapForBean(WlsSe
rvletBeanContext.java:112)

CPU utilization goes upto 100% and their application becomes very slow.This
issue started happening after they made a few changes to their portal
application and deployed it. After this, they have started seeing this
issue.The Portal team would upload the details on the changes made in their
application.Also the other point to note is this happens only in the
production environment only under high load.The customer informed that they
were not able to replicate it consistently.



Hope above helps in fixing the STUCK thread issues on your M6 application, please leave your feedback or query

Metasolv Gateway Events not completing with 100004=4,Error occured in MetaSolv Solution

PROBLEM

Below error shows up in both the application log files and M6 GUI. The same is also being stored in the GATEWAY_EVENT_ERROR(GATEWAY_EVENT_ERR_MSG column) table.

Error in the GUI's Gateway Event Error List window (will be multiple entries) 

100004=4,Error occured in MetaSolv Solution:
100004=4,Error occured in MetaSolv Solution:

Errors in ASAP.GATEWAY_EVENT_ERROR.GATEWAY_EVENT_ERR_MSG (two separate rows) 

100004=4,Error occured in MetaSolv Solution:
Exception during SQL update: Event select for update failed, no row returned.
at com.mslv.core.global.MSLVException.createMSLVException(MSLVException.java:169)
at com.mslv.core.global.MSLVException.throwException(MSLVException.java:91)
at com.mslv.integration.integrationServer.SrsiEvent.selectForUpdate(SrsiEvent.java:202)
at com.mslv.integration.handlers.DefaultEventHandler.startOutboundProcess(DefaultEventHandler.java:22)
at com.mslv.integration.IntegrationManager.processOutbound(IntegrationManager.java:183)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.executeEvent(IntegrationServerConsumer.java:268)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.handleEvents(IntegrationServerConsumer.java:204)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.run(IntegrationServerConsumer.java:175)
at com.metasolv.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:725)
at java.lang.Thread.run(Thread.java:662)at com.mslv.

The above error can be captured from any of the rows fetched using below query:

select count(*) from GATEWAY_EVENT_ERROR
where gateway_event_err_msg = '100004=4,Error occured in MetaSolv Solution:'
and to_char(last_modified_date,'YYYY-MM-DD') > 'YYYY-MM-DD'; --change later 'YYYY-MM-DD'

Errors in appserverlog.xml:

<log4j:event logger="cmm.INTEGRATION_MANAGER" timestamp="1438353562960" level="FATAL" thread="Thread-43">
<log4j:message><![CDATA[[machineName :: nh1m6p01][appServerName :: DEF_APPSERVER][userName :: app_api][productName :: cluster-nur][messageID :: 90029][moduleName :: cmm.INTEGRATION_MANAGER]

[className :: DEF_CLASS][debugCode :: 0]::Exception during SQL update: Event select for update failed, no row returned.
at com.mslv.core.global.MSLVException.createMSLVException(MSLVException.java:169)
at com.mslv.core.global.MSLVException.throwException(MSLVException.java:91)
at com.mslv.integration.integrationServer.SrsiEvent.selectForUpdate(SrsiEvent.java:202)
at com.mslv.integration.handlers.DefaultEventHandler.startOutboundProcess(DefaultEventHandler.java:22)
at com.mslv.integration.IntegrationManager.processOutbound(IntegrationManager.java:183)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.executeEvent(IntegrationServerConsumer.java:268)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.handleEvents(IntegrationServerConsumer.java:204)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.run(IntegrationServerConsumer.java:175)
at com.metasolv.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:725)
at java.lang.Thread.run(Thread.java:662)
::]]></log4j:message>
</log4j:event>

<log4j:event logger="cmm.INTEGRATION_MANAGER" timestamp="1438356529993" level="FATAL" thread="Thread-43">
<log4j:message><![CDATA[[machineName :: nh1m6p01][appServerName :: DEF_APPSERVER][userName :: app_api][productName :: cluster-nur][messageID :: 90029][moduleName :: cmm.INTEGRATION_MANAGER]

[className :: DEF_CLASS][debugCode :: 0]::Exception during SQL update: java.sql.SQLException: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:128)
at com.metasolv.jdbc.MSVPreparedStatement.executeQuery(MSVPreparedStatement.java:75)
at com.mslv.integration.integrationServer.SrsiEvent.selectForUpdate(SrsiEvent.java:200)
at com.mslv.integration.handlers.DefaultEventHandler.startOutboundProcess(DefaultEventHandler.java:22)
at com.mslv.integration.IntegrationManager.processOutbound(IntegrationManager.java:183)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.executeEvent(IntegrationServerConsumer.java:268)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.handleEvents(IntegrationServerConsumer.java:204)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.run(IntegrationServerConsumer.java:175)
at com.metasolv.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:725)
at java.lang.Thread.run(Thread.java:662)
::]]></log4j:message>
</log4j:event>

The above error is coming up for SERV_ITEM level Gateway Events.

The same is already reported to Oracle and has a work around for Order level Gateway Events against Bug
9152079 : INTEGRATION SERVER DEFINED GATEWAY EVENTS INTERMITTENTLY FAIL.

CAUSE

Integration Server defined Item Level gateway events fail with "Exception during SQL update: Event select for update failed, no row returned."  This behavior occurs in environments where there are multiple Integration Servers enabled (cluster or multi-single managed) which are polling for the same gateway event IDs.  Below is the error displayed in the GUI's Gateway Event Error List which is stored in the ASAP.GATEWAY_EVENT_ERROR table (GATEWAY_EVENT_ERR_MSG column) as well as the appserverlog.xml file.  Note that the error will be displayed in the GUI multiple times showing duplicate "Times Initiated" values with nearly identical "Date/Time Create" values as well that indicate the event was consumed by multiple Integration Servers.

SOLUTION

As part of Oracle bug
9152079 - INTEGRATION SERVER DEFINED GATEWAY EVENTS INTERMITTENTLY FAIL; the bug's solution was applied only to Order Level Gateway Events(ServiceRequestEvent class/serv_req_gateway_event) and not Item Level Gateway Events (SrsiEvent class/srsi_gateway_event).

A new Oracle Bug
21541135 - Integration Server Defined Item Level Gateway Events Fail; 100004=4 
is created for SERV_ITEM level Gateway Events.

Please follow Bug 21541135 for the behavior to be permanently fixed to allow multiple INTEGRATIONSERVERS to process Item Level Gateway Events. The development is still in progress as on 26th Aug 2015.

TEMPORARY WORKAROUND

Disable or suspend all but one integration server in a multiple server(clustered server) environment if the system load permits.

The only configuration change is commenting out the server with a preceding semi-colon character like

;INTEGRATIONSERVER=com.mslv.integration.integrationServer.S3Startup

in gateway.ini file.


Hope above helps in fixing the above mentioned exception related to Gateway Events, please leave your feedback or query.

MetaSolv Solution Application Server Startup Error - JDBC store "cgJMSStore_auto_9" failed to open table "cgJMSStore_921344767WLStore".weblogic.store.io.jdbc.JDBCStoreException:[Store:280064]Expected empty table

PROBLEM

After restarting WebLogic servers we see below exception in startup logs.

<Jul 16, 2015 4:40:15 AM EDT> <Error> <Store> <BEA-280072> <JDBC store "cgJMSStore_auto_9" failed to open table "cgJMSStore_921344767WLStore".
weblogic.store.io.jdbc.JDBCStoreException: [Store:280064]Expected empty table. (server="my_srvr_2.1" store="cgJMSStore_auto_9" table="cgJMSStore_921344767WLStore")
at weblogic.store.io.jdbc.JDBCStoreIO.initializeEmptyTable(JDBCStoreIO.java:657)
at weblogic.store.io.jdbc.JDBCStoreIO.open(JDBCStoreIO.java:370)

<Jul 16, 2015 4:40:15 AM EDT> <Error> <Store> <BEA-280061> <The persistent store "cgJMSStore_auto_9" could not be deployed: weblogic.store.io.jdbc.JDBCStoreException: open failed
weblogic.store.io.jdbc.JDBCStoreException: open failed
at weblogic.store.io.jdbc.JDBCStoreIO.open(JDBCStoreIO.java:442)

CAUSE

JMS queue that uses JMS Server with JDBC store, the store gets corrupted when
tablespace runs out.
When JDBC store tablespace runs out, the WLS server goes to FAILED state.
This is expected.

However, after increasing the tablespace see a number of errors which appear
to indicate the store is corrupted and the JMS Server fails on trying to
restart server.

SOLUTION

The recommended steps to avoid store corruption are:

1. Do force shutdown with a timeout set to equal or great than the JTA timeout, using WLST.
2. Use non-XA driver for JDBC Store.
3. Avoid abrupt DB server failures if at all any.
4. Enable two-phase commit for the non-XA Datasource

To rectify the JMSStore corruption below is an Oracle patch given to fix the issue forever.

Patch 8553975: SU Patch [HTRG]: Patch to correct data corruption when tablespace runs out

TEMPORARY WORKAROUND

The other known method to restore the service is to truncate the JDBC store
table.Please do as below in WebLogic console.

1. Go to Persistent Stores.
2. Look for JDBC store cgJMSStore_auto_9(as per your exception message).
3. Under JDBC store look for Data Source 'cgDataSource-nonXA' and Prefix.
4. Under JDBC Data Source look for user in Connection Pool for DataSource 'cgDataSource-nonXA'.
5. Login to M6 DB with user from point 4 and look for a table starting name same as Prefix from point 3. If the Prefix is 'cgJMSStore_921344767' then table name would be as shown in exception
message like cgJMSStore_921344767WLStore.
6. Truncate the table and restart the server.
7. Server will restart fine.

Hope above helps in fixing the JMSStore related exceptions on your WebLogic server startup, please leave your feedback or query.

Deployment Of Application Fails on Weblogic With 'Failed to deploy process XXX.jpd. There are running instances with incompatible serialization shape'

PROBLEM

Deployment of any custom application or Metasolv core application(e.g. mss_integration.ear) fails with below exception.

<Mar 3, 2015 5:24:49 AM EST> <Error> <Deployer> <BEA-149231> <Unable to set the activation state to true for the application 'mss_integration'. java.lang.RuntimeException: Failed to deploy process

/API/com/metasolv/api/workflow/customer/GetCustomerAccountSync.jpd. There are running instances with incompatible serialization shape.
        at com.bea.wli.management.WLIAppListener.checkProject(WLIAppListener.java:172)
        at com.bea.wli.management.WLIAppListener.postStart(WLIAppListener.java:418)
        at weblogic.application.internal.flow.BaseLifecycleFlow$PostStartAction.run(BaseLifecycleFlow.java:292)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(Unknown Source)
        Truncated. see log file for complete stacktrace

CAUSE

The problem is caused by aborted/running processes data in the APP_INT.WLI_PROCESS_INSTANCE_INFO table. There could be 2 reasons for the error.

1. It occurs when there are running / aborted instances of a JPD and an attempt is made to deploy a newer version of incompatible JPD (code changed) or same ear file with no in the old one's place
and as WLI keeps state of previous JPD's therefore old instances in running or aborted state does not allow new ear to be deployed.

2. If the ear file(no change at all) is exactly the same as before server restart/application deployment and the issue occurs if there are any running or aborted instances.

Here we will only discuss about the 2nd scenario.

The query below identifies running/aborted transaction processes:

SELECT * FROM APP_INT.WLI_PROCESS_INSTANCE_INFO WHERE PROCESS_TYPE  LIKE '%jpd_name' AND PROCESS_STATUS IN (0,1);

Description of various PROCESS_STATUS that can be found:
0=RUNNING
1=ABORTED
2=SUSPENDED
3=COMPLETED
4=FROZEN
5=TERMINATED
If the process tracking is set to NONE we can also see a status=7.

Note: In case you have multiple domains in M6 application and each domain may use different WLI user for same DB instance.
e.g. I have two diff domains where I may have installed M6 and their names are 'ENTERPRISE' and  'WHOLESALE'. I may create different WLI users 'ENTERPRISE_INT' and 'WHOLESALE_INT' respectively. In such cases login with such users instead of APP_INT.

SOLUTION

If the server is restarted and the application is deployed with 'Failed' status after server is started. Do as below


1. Go to database(with APP_INT) delete all the running/aborted instances for the particular jpd/process

DELETE FROM APP_INT.WLI_PROCESS_INSTANCE_INFO WHERE PROCESS_TYPE  LIKE '%jpd_name' AND PROCESS_STATUS IN (0,1);
COMMIT;

2. Also delete all the child records for PROCESS_TYPE in tables like WLI_PROCESS_EVENT to have clean and consistent data.


3. Redeploy the application ear.

OR

Shutdown WebLogic(where the ear is deployed) Server & truncate WLI_PROCESS_INSTANCE_INFO. Start the WebLogic server.

Hope above helps in fixing the deployment issues with applications having running instances in WLI schema, please leave your feedback or query.

Slow performance while completing PSR Due Date(DD) tasks on Metasolv 6.2.1.870(build)

Slow performance encountered when attempting to complete Due Date tasks.  Users will report that the Due Date task they're attempting to complete hangs for a long period of time.

PROBLEM

Some PSR order's DD task taking long time to complete. A long running query was observed to be running at that time and taking ~12 minutes per execution. The query below identified the ACTIVE SQL

processes consuming the highest amount of CPU on the DB. Many sessions found were running these processes.

The following SQL can help identify the problem by showing the process that is consuming the highest amount of CPU resources on the database:

select p.spid "Thread ID",
b.name "Background Process",
s.username "User Name",
s.machine,
s.terminal,
s.osuser "OS User",
s.client_identifier "MetaSolv Solution User",
s.program "OS Program",
s.status "STATUS",
s.sid "Session ID",
s.serial# "Serial No.",
sa.CPU_TIME/1000000,
sa.ELAPSED_TIME/1000000, sa.executions,
sa.sql_id,
sa.SQL_TEXT
from v$process p, v$bgprocess b, v$session s , v$SQL Sa
where s.paddr = p.addr
and b.paddr(+) = p.addr
and s.sql_address = sa.address
and s.sql_hash_value = sa.hash_value
and s.status='ACTIVE'
order by sa.CPU_TIME DESC;

Highlights of query:

BUSINESS IMPACT: STUCK THREAD on WebLogic server. Decreased system throughput.

CAUSE

The slow performance is caused by a poor performing query in the PACKAGE_PSR_DUE_DATE procedure. The problematic SQL(Function) call is made in the ASAP.PACKAGE_PSR_DUE_DATE.

Function of_disc_cust_site
(arg_document_number IN asap.serv_req.document_number %Type,
arg_serv_item_id in asap.serv_item.serv_item_id %type,
arg_cust_acct_id IN asap.cust_acct.cust_acct_id %type,
arg_return_text OUT VARCHAR2) Return Number IS

loc_return_code Number;
loc_return_text Varchar2(2000);
loc_proc_name varchar2(50) := 'of_disc_cust_site';
loc_cust_acct_id asap.cust_acct.cust_acct_id %type;
loc_serv_item_id asap.serv_item.serv_item_id %type;

cursor cur_serv_item (csr_doc_nbr asap.serv_req.document_number %type, csr_serv_item_id asap.serv_item.serv_item_id %type)
is select si.serv_item_id, n.ns_comp_id, cd.circuit_design_id
from asap.ns_comp_si n,
asap.serv_item si,
(select si.circuit_design_id
from asap.serv_item si, asap.circuit c
where serv_item_id in (select serv_item_id from asap.serv_req_si where document_number = csr_doc_nbr)
and si.serv_item_id in (select serv_item_id_rel from asap.serv_item_rel where serv_item_id = csr_serv_item_id)
and si.circuit_design_id = c.circuit_design_id
and c.nst_con_type = 1) cd
where n.ns_comp_id in
(select distinct nc.ns_comp_id_parent
from asap.ns_connection nc,
asap.nst_comp_type_con nctc,
asap.nst_config_comp_type ncct
where nc.circuit_design_id = cd.circuit_design_id
and nc.nst_comp_type_con_id = nctc.nst_comp_type_con_id
and ncct.nst_config_type_id = nctc.nst_nst_config_type_id_parent
and ncct.nst_comp_type = nctc.nst_comp_type_parent
and ncct.nst_net_ext_ind = 'Y'
union
select distinct nc.ns_comp_id_child
from asap.ns_connection nc,
asap.nst_comp_type_con nctc,
asap.nst_config_comp_type ncct
where nc.circuit_design_id = cd.circuit_design_id
and nc.nst_comp_type_con_id = nctc.nst_comp_type_con_id
and ncct.nst_config_type_id = nctc.nst_config_type_id_child
and ncct.nst_comp_type = nctc.nst_comp_type_child
and ncct.nst_net_ext_ind = 'Y')
and si.serv_item_id = n.serv_item_id
and si.status in ('1','6','S'); -- Bug 16901003

SOLUTION

To implement the solution, please execute the following steps:

1. Refer Bug 21573008 - Slow Query for PACKAGE_PSR_DUE_DATE of_disc_cust_site Function.
2. The problem should be fixed with an interim patch in the new M6.2.1 build6.2.1.b914(should come by end of Aug 15).

TEMPORARY WORKAROUND

If your company does not specifically use below piece of code in ASAP.PACKAGE_PSR_DUE_DATE in 6.0.15 and if the code is introduced in ASAP.PACKAGE_PSR_DUE_DATE in 6.2.1(M6 Upgrade process) then you can safely comment below code.

BEFORE-COMMENTED:

            --BUG 16901003 - Calling the new function here only for product bundles
            IF ins_item_type_cd IN ('SYSTEM', 'PRDBUNDLE')
            THEN
               --BUG 18415660 - Calling this function to move customer site to In-Service
               IF ins_serv_item_status IN ('1', '6')
               THEN
                  IF of_set_component_status (ins_document_number,
                                              ins_serv_item_id,
                                              ins_cust_acct_id,
                                              loc_return_text
                                             ) = -1
                  THEN
                     RAISE logic_error;
                  END IF;
               --BUG 18415660 END
               ELSIF ins_serv_item_status IN ('S', '7', '8')
               THEN
                  IF of_disc_cust_site (ins_document_number,
                                        ins_serv_item_id,
                                        ins_cust_acct_id,
                                        loc_return_text
                                       ) = -1
                  THEN
                     RAISE logic_error;
                  END IF;
               END IF;
            END IF;

COMMENTED:

/** Remove extra code from MSS 6.2.1
            --BUG 16901003 - Calling the new function here only for product bundles
            IF ins_item_type_cd IN ('SYSTEM', 'PRDBUNDLE')
            THEN
               --BUG 18415660 - Calling this function to move customer site to In-Service
               IF ins_serv_item_status IN ('1', '6')
               THEN
                  IF of_set_component_status (ins_document_number,
                                              ins_serv_item_id,
                                              ins_cust_acct_id,
                                              loc_return_text
                                             ) = -1
                  THEN
                     RAISE logic_error;
                  END IF;
               --BUG 18415660 END
               ELSIF ins_serv_item_status IN ('S', '7', '8')
               THEN
                  IF of_disc_cust_site (ins_document_number,
                                        ins_serv_item_id,
                                        ins_cust_acct_id,
                                        loc_return_text
                                       ) = -1
                  THEN
                     RAISE logic_error;
                  END IF;
               END IF;
            END IF;
*/

Hope above helps in fixing the slowness for completing DD tasks for your Metasolv applications, please leave your feedback or query.