MetaSolv Solution Application Server Startup Error - JDBC store "cgJMSStore_auto_9" failed to open table "cgJMSStore_921344767WLStore".weblogic.store.io.jdbc.JDBCStoreException:[Store:280064]Expected empty table

PROBLEM

After restarting WebLogic servers we see below exception in startup logs.

<Jul 16, 2015 4:40:15 AM EDT> <Error> <Store> <BEA-280072> <JDBC store "cgJMSStore_auto_9" failed to open table "cgJMSStore_921344767WLStore".
weblogic.store.io.jdbc.JDBCStoreException: [Store:280064]Expected empty table. (server="my_srvr_2.1" store="cgJMSStore_auto_9" table="cgJMSStore_921344767WLStore")
at weblogic.store.io.jdbc.JDBCStoreIO.initializeEmptyTable(JDBCStoreIO.java:657)
at weblogic.store.io.jdbc.JDBCStoreIO.open(JDBCStoreIO.java:370)

<Jul 16, 2015 4:40:15 AM EDT> <Error> <Store> <BEA-280061> <The persistent store "cgJMSStore_auto_9" could not be deployed: weblogic.store.io.jdbc.JDBCStoreException: open failed
weblogic.store.io.jdbc.JDBCStoreException: open failed
at weblogic.store.io.jdbc.JDBCStoreIO.open(JDBCStoreIO.java:442)

CAUSE

JMS queue that uses JMS Server with JDBC store, the store gets corrupted when
tablespace runs out.
When JDBC store tablespace runs out, the WLS server goes to FAILED state.
This is expected.

However, after increasing the tablespace see a number of errors which appear
to indicate the store is corrupted and the JMS Server fails on trying to
restart server.

SOLUTION

The recommended steps to avoid store corruption are:

1. Do force shutdown with a timeout set to equal or great than the JTA timeout, using WLST.
2. Use non-XA driver for JDBC Store.
3. Avoid abrupt DB server failures if at all any.
4. Enable two-phase commit for the non-XA Datasource

To rectify the JMSStore corruption below is an Oracle patch given to fix the issue forever.

Patch 8553975: SU Patch [HTRG]: Patch to correct data corruption when tablespace runs out

TEMPORARY WORKAROUND

The other known method to restore the service is to truncate the JDBC store
table.Please do as below in WebLogic console.

1. Go to Persistent Stores.
2. Look for JDBC store cgJMSStore_auto_9(as per your exception message).
3. Under JDBC store look for Data Source 'cgDataSource-nonXA' and Prefix.
4. Under JDBC Data Source look for user in Connection Pool for DataSource 'cgDataSource-nonXA'.
5. Login to M6 DB with user from point 4 and look for a table starting name same as Prefix from point 3. If the Prefix is 'cgJMSStore_921344767' then table name would be as shown in exception
message like cgJMSStore_921344767WLStore.
6. Truncate the table and restart the server.
7. Server will restart fine.

Hope above helps in fixing the JMSStore related exceptions on your WebLogic server startup, please leave your feedback or query.