IBM Websphere MQ Manager Stops Responding To JMS Requests - Middleware News

MQ Manager stops responding to JMS requests.

Symptom

SystemOut.log:
FreePool E J2CA0046E: Method createManagedConnctionWithMCWrapper caught an exception during creation of the ManagedConnection for resource JMS$cftestcf$JMSManagedConnection@1373738090, throwing ResourceAllocationException. Original exception: javax.resource.spi.ResourceAdapterInternalException: Failed to create session
at com.ibm.ejs.jms.JMSCMUtils.mapToResourceException(JMSCMUtils.java:125)
at com.ibm.ejs.jms.JMSManagedSession.(JMSManagedSession.java:213)
. . .
javax.jms.JMSException: MQJMS2005: failed to create MQQueueManager for 'xldn0384abc:XYZ123'
at com.ibm.mq.jms.services.ConfigEnvironment.newException

(ConfigEnvironment.java:546)
at com.ibm.mq.jms.MQConnection.createQM
(MQConnection.java:1450)
at com.ibm.mq.jms.MQConnection.createQMNonXA
(MQConnection.java:960)
at
. . .
com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009
at com.ibm.mq.MQManagedConnectionJ11.
(MQManagedConnectionJ11.java:172)
at com.ibm.mq.MQClientManagedConnectionFactoryJ11._createManagedConnection
(MQClientManagedConnectionFactoryJ11.java:270)
at com.ibm.mq.MQClientManagedConnectionFactoryJ11.createManagedConnection
(MQClientManagedConnectionFactoryJ11.java:290)

AMQERRO1.log
AMQ9513: Maximum number of channels reached.

Cause

The maximum number of channels that can be in use simultaneously has been reached. The number of permitted channels is a configurable parameter in the queue manager configuration file.

When an application connects to MQ a channel is started on the MQ side. If the application, for any reason, is unexpectedly disconnected (no proper disconnection takes place) then the channel will not get cleaned up on the MQ side. It will become 'orphaned' from its original parent connection. When the application reconnects it will get a new instance of the channel, so now there will be 2 instances of the channel, the new one and the old, orphaned instance.

MQ only allows a certain number of channels. If you build up enough channels you will get the MaxChannels error AMQ9513.

Channels may also be getting orphaned due to TCP/IP interruptions rather than an application disconnecting improperly from MQ.

Resolving the problem

How should you manage these orphaned channels?

If you can get these orphaned channels to clean up you will go a long way towards avoiding this issue.

1. Set up the operating system TCP/IP KeepAlive parameter.
You must enable KeepAlive at operating system (TCP/IP) level. How this is done depends entirely on the operating system you are using. The TCP/IP keep alive setting is an operating system parameter. It determines how long to keep a TCP/IP connection alive.

KeepAlive has a timeout option that is usually set to 2 hours. Recommend setting this to a much shorter interval, such as 10 minutes. Once this change has been made the OS will need to be rebooted for this to take effect.

2. In addition to enabling KeepAlive at the OS level, MQ must also be configured to use KeepAlive. This is done by adding the following stanza to the QM.INI file for this queue manager, as follows:
TCP:
KeepAlive=yes

See MQ TCP Stanza

Once this stanza has been added the queue manager must be restarted for this to take effect.

3. It is also highly recommend to change the MaxChannels value (also in the QM.INI file) to 3 times what you think may be needed. For instance from 100 to 300 MaxChannel. See the MQ Channels stanza. This will ensure that you have some flexibility in the event a contingency occurs. To determine the number of channels WebSphere Application Server requires, see this technote, Explanation of connection pool and session pool settings for JMS connection factories.