A large number of amqrmppa processes spawn very quickly in IBM Websphere MQ system - Middleware News

 You see a large number of amqrmppa channel processes that are spawned very quickly in a WebSphere MQ system

Symptom

A large number of amqrmppa processes are created in a short period of time. Their parent process is a runmqlsr process. Sometimes there are multiple runmqlsr processes that were spawned, which has this runmqlsr process as a parent process as well. The number of spawned processes might keep increasing and sometimes reaches the maxuproc limit of the operating system, hence it could cause a queue manager outage.

Cause

The cause of the large number of amqrmppa and runmqlsr processes is that these processes are attached to an old set of shared memory segments.

Diagnosing the problem

When a MQCONN request is received by the runmqlsr process it seems that it has no active amqrmppa processes listed in its old shared memory segments and so runmqlsr starts a new amqrmppa process. However it fails to communicate with the newly started amqrmppa process because the amqrmppa attaches to the new shared memory segments, and so it instead starts a thread and writes an error message. The amqrmppa process is left orphaned waiting for a connect request which will never arrive. This process is repeated for each MQCONN request, with a new amqrmppa process started for each client MQCONN.
As the initial trigger for the failure, one scenario could be that the last time the queue manager was ended the system's shared memory was manually cleaned-up using ipcrm, but the listener was missed and was left running. The queue manager was subsequently restarted and had no knowledge of the old listener process.

The recommended workaround solution for this issue is to kill the old runmqlsr process and bring up a new MQ listener process either by "runmqlsr" command " or "start listener" using runmqsc interface like below:

$ runmqlsr -t tcp -p -m
or
$ runmqsc
start listener('listener name')