Metasolv Gateway Events not completing with 100004=4,Error occured in MetaSolv Solution

PROBLEM

Below error shows up in both the application log files and M6 GUI. The same is also being stored in the GATEWAY_EVENT_ERROR(GATEWAY_EVENT_ERR_MSG column) table.

Error in the GUI's Gateway Event Error List window (will be multiple entries) 

100004=4,Error occured in MetaSolv Solution:
100004=4,Error occured in MetaSolv Solution:

Errors in ASAP.GATEWAY_EVENT_ERROR.GATEWAY_EVENT_ERR_MSG (two separate rows) 

100004=4,Error occured in MetaSolv Solution:
Exception during SQL update: Event select for update failed, no row returned.
at com.mslv.core.global.MSLVException.createMSLVException(MSLVException.java:169)
at com.mslv.core.global.MSLVException.throwException(MSLVException.java:91)
at com.mslv.integration.integrationServer.SrsiEvent.selectForUpdate(SrsiEvent.java:202)
at com.mslv.integration.handlers.DefaultEventHandler.startOutboundProcess(DefaultEventHandler.java:22)
at com.mslv.integration.IntegrationManager.processOutbound(IntegrationManager.java:183)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.executeEvent(IntegrationServerConsumer.java:268)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.handleEvents(IntegrationServerConsumer.java:204)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.run(IntegrationServerConsumer.java:175)
at com.metasolv.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:725)
at java.lang.Thread.run(Thread.java:662)at com.mslv.

The above error can be captured from any of the rows fetched using below query:

select count(*) from GATEWAY_EVENT_ERROR
where gateway_event_err_msg = '100004=4,Error occured in MetaSolv Solution:'
and to_char(last_modified_date,'YYYY-MM-DD') > 'YYYY-MM-DD'; --change later 'YYYY-MM-DD'

Errors in appserverlog.xml:

<log4j:event logger="cmm.INTEGRATION_MANAGER" timestamp="1438353562960" level="FATAL" thread="Thread-43">
<log4j:message><![CDATA[[machineName :: nh1m6p01][appServerName :: DEF_APPSERVER][userName :: app_api][productName :: cluster-nur][messageID :: 90029][moduleName :: cmm.INTEGRATION_MANAGER]

[className :: DEF_CLASS][debugCode :: 0]::Exception during SQL update: Event select for update failed, no row returned.
at com.mslv.core.global.MSLVException.createMSLVException(MSLVException.java:169)
at com.mslv.core.global.MSLVException.throwException(MSLVException.java:91)
at com.mslv.integration.integrationServer.SrsiEvent.selectForUpdate(SrsiEvent.java:202)
at com.mslv.integration.handlers.DefaultEventHandler.startOutboundProcess(DefaultEventHandler.java:22)
at com.mslv.integration.IntegrationManager.processOutbound(IntegrationManager.java:183)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.executeEvent(IntegrationServerConsumer.java:268)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.handleEvents(IntegrationServerConsumer.java:204)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.run(IntegrationServerConsumer.java:175)
at com.metasolv.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:725)
at java.lang.Thread.run(Thread.java:662)
::]]></log4j:message>
</log4j:event>

<log4j:event logger="cmm.INTEGRATION_MANAGER" timestamp="1438356529993" level="FATAL" thread="Thread-43">
<log4j:message><![CDATA[[machineName :: nh1m6p01][appServerName :: DEF_APPSERVER][userName :: app_api][productName :: cluster-nur][messageID :: 90029][moduleName :: cmm.INTEGRATION_MANAGER]

[className :: DEF_CLASS][debugCode :: 0]::Exception during SQL update: java.sql.SQLException: ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1493)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:128)
at com.metasolv.jdbc.MSVPreparedStatement.executeQuery(MSVPreparedStatement.java:75)
at com.mslv.integration.integrationServer.SrsiEvent.selectForUpdate(SrsiEvent.java:200)
at com.mslv.integration.handlers.DefaultEventHandler.startOutboundProcess(DefaultEventHandler.java:22)
at com.mslv.integration.IntegrationManager.processOutbound(IntegrationManager.java:183)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.executeEvent(IntegrationServerConsumer.java:268)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.handleEvents(IntegrationServerConsumer.java:204)
at com.mslv.integration.integrationServer.IntegrationServerConsumer.run(IntegrationServerConsumer.java:175)
at com.metasolv.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:725)
at java.lang.Thread.run(Thread.java:662)
::]]></log4j:message>
</log4j:event>

The above error is coming up for SERV_ITEM level Gateway Events.

The same is already reported to Oracle and has a work around for Order level Gateway Events against Bug
9152079 : INTEGRATION SERVER DEFINED GATEWAY EVENTS INTERMITTENTLY FAIL.

CAUSE

Integration Server defined Item Level gateway events fail with "Exception during SQL update: Event select for update failed, no row returned."  This behavior occurs in environments where there are multiple Integration Servers enabled (cluster or multi-single managed) which are polling for the same gateway event IDs.  Below is the error displayed in the GUI's Gateway Event Error List which is stored in the ASAP.GATEWAY_EVENT_ERROR table (GATEWAY_EVENT_ERR_MSG column) as well as the appserverlog.xml file.  Note that the error will be displayed in the GUI multiple times showing duplicate "Times Initiated" values with nearly identical "Date/Time Create" values as well that indicate the event was consumed by multiple Integration Servers.

SOLUTION

As part of Oracle bug
9152079 - INTEGRATION SERVER DEFINED GATEWAY EVENTS INTERMITTENTLY FAIL; the bug's solution was applied only to Order Level Gateway Events(ServiceRequestEvent class/serv_req_gateway_event) and not Item Level Gateway Events (SrsiEvent class/srsi_gateway_event).

A new Oracle Bug
21541135 - Integration Server Defined Item Level Gateway Events Fail; 100004=4 
is created for SERV_ITEM level Gateway Events.

Please follow Bug 21541135 for the behavior to be permanently fixed to allow multiple INTEGRATIONSERVERS to process Item Level Gateway Events. The development is still in progress as on 26th Aug 2015.

TEMPORARY WORKAROUND

Disable or suspend all but one integration server in a multiple server(clustered server) environment if the system load permits.

The only configuration change is commenting out the server with a preceding semi-colon character like

;INTEGRATIONSERVER=com.mslv.integration.integrationServer.S3Startup

in gateway.ini file.


Hope above helps in fixing the above mentioned exception related to Gateway Events, please leave your feedback or query.