IBM Websphere MQ : Diagnostic improvements to MQ SSL/TLS error AMQ9633 - Middleware News

From MQ 7.1.0.2 onwards, the dreaded AMQ9633 "bad certificate" error message was enhanced with some additional information to help you understand which certificate was rejected and why. This is the most common error seen for SSL/TLS handshake errors and so we'd like to explain how to use the new information to understand the cause of the error.

Example AMQ9633 error

Below is an example of the new style of message with the new section highlighted in bold text, taken from a queue manager error log:

23/10/2014 08:31:41 - Process(1234.2) User(Fred) Program(amqrmppa.exe)
                      Host(hostname.example.com) Installation(Installation1)
                      VRMF(7.5.0.3) QMgr(QMGR1)
                   
AMQ9633: Bad SSL certificate for channel '????'.

EXPLANATION:
A certificate encountered during SSL handshaking is regarded as bad for one of
the following reasons:
(a) it was formatted incorrectly and could not be validated
(b) it was formatted correctly but failed validation against the Certification
  Authority (CA) root and other certificates held on the local system
(c) it was found in a Certification Revocation List (CRL) on an LDAP server
(d) a CRL was specified but the CRL could not be found on the LDAP server
(e) an OCSP responder has indicated that it is revoked

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The remote host is 'host2.example.com (10.0.0.2)'. The channel
did not start.

The details of the certificate which could not be validated are
'[Class=]GSKVALMethod::X509[Issuer=]CN=Example Certificate Authority,O=IBM,C=US[#=]67322456fe4af6de[Subject=]CN=Example MQ Certificate,O=IBM,C=US'.

The certificate validation error was 575010.
ACTION:
Check which of the possible causes applies on your system. Correct the error,
and restart the channel.

This error might indicate that the remote end of the channel is configured to
send the wrong certificate. Check the certificate label configuration at the
remote end of the channel and ensure that the local key repository contains all
of the necessary CA certificates.

For completeness, we should also mention that MQ 8.0 is able to determine the channel name in many cases so you should see fewer errors for channel '????' (although there are still some cases where we can't do that yet). The error messages are improving over time!

Understanding this error

The bold text in the message above shows the details of the problematic certificate. The main fields are:

• [Issuer=] - the Issuer Distinguished Name (DN) of the certificate, in this case CN=Example Certificate Authority,O=IBM,C=US
• [Subject=] - the Subject DN of the certificate, in this case CN=Example MQ Certificate,O=IBM,C=US
• [#=] - the serial number of the certificate in hexadecimal string form, in this case 67322456fe4af6de

So now we know which certificate had a problem. But what went wrong?

For this, there's a numeric certificate validation error, which is usually a 6-digit number beginning with "575". The certificate validation errors are listed in the product documentation in Table 2 of the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) return codes topic. For convenience, here's a copy of the current table:

Return code (decimal)	Explanation
575001	Internal error
575002	ASN error due to a malformed certificate
575003	Cryptographic error
575004	Key database error
575005	Directory error
575006	Invalid implementation library
575008	No appropriate validator
575009	The root CA is not trusted
575010	No certificate chain was built
575011	Digital signature algorithm mismatch
575012	Digital signature mismatch
575013	X.509 version does not allow Key IDs
575014	X.509 version does not allow extensions
575015	Unknown X.509 certificate version
575016	The certificate validity range is invalid
575017	The certificate is not yet valid
575018	The certificate has expired
575019	The certificate contains unknown critical extensions
575020	The certificate contains duplicate extensions
575021	The issuers directory name does not match the issuer's issuer
575022	The Authority Key ID serial number value does not match the serial number of the issuer
575023	The Authority Key ID and Subject Key ID do not match
575024	Unrecognized issuer alternative name
575025	The certificate Basic Constraints forbid use as a CA
575026	The certificate has a non-zero Basic Constraints path length but is not a CA
575027	The certificate Basic Constraints maximum path length was exceeded
575028	The certificate is not permitted to sign other certificates
575029	The certificate is not signed by a CA
575030	Unrecognized Subject Alternative Name
575031	The certificate chain is invalid
575032	The certificate is revoked
575033	Unrecognized CRL distribution point
575034	Name chaining failed
575035	Certificate is not in a chain
575036	The CRL is not yet valid
575037	The CRL has expired
575038	The certificate version does not allow critical extensions
575039	Unknown CRL distribution points
575040	No CRLs for CRL distribution points
575041	Indirect CRLs are not supported
575042	Missing issuing CRL distribution point name
575043	Distribution points do not match
575044	No available CRL data source
575045	CA Subject name is null
575046	Distinguished names do not chain
575047	Missing Subject Alternative Name
575048	Unique ID mismatch
575049	Name not permitted
575050	Name excluded
575051	CA certificate is missing Critical Basic Constraints
575052	Name constraints are not critical
575053	Name constraints minimum subtree value if set is not zero
575054	Name constraints maximum subtree value if set is not allowed
575055	Unsupported name constraint
575056	Empty policy constraints
575057	Bad certificate policies
575058	Certificate policies not acceptable
575059	Bad acceptable certificate policies
575060	Certificate policy mappings are critical
575061	Revocation status could not be determined
575062	Extended key usage error
575063	Unknown OCSP version
575064	Unknown OCSP response
575065	Bad OCSP key usage extension
575066	Bad OCSP nonce
575067	Missing OCSP nonce
575068	No OCSP client available

So we can see that our example error 575010 means that no certificate chain was built. A certificate chain is a sequence of related certificates which are linked by digital signatures (each certificate is the signer of the next one, up to the root certificate which is self-signed).

Failure to build a certificate chain usually means that our key repository lacks one of the Certificate Authority (CA) certificates needed to validate the remote certificate. Other errors have different causes, but since a missing CA certificate is a common error then it's worth looking at the next steps.

How to fix this error

In this example, our queue manager which wrote the error message doesn't have one of the CA certificates required. The first thing to do is to check that the issuer of the certificate is present in the queue manager's key repository. In this case, the issuer has a DN of "CN=Example Certificate Authority,O=IBM,C=US" so check to see if the CA certificate is in the key repository. To do this, we must search for a certificate with a Subject DN of "CN=Example Certificate Authority,O=IBM,C=US"; this is because a CA certificate's Subject DN is equal to the Issuer DN of any certificates signed by that CA, so we must find the certificate with a Subject DN that matches the problematic certificate's Issuer DN.

To check for the CA certificate, either:

• Start the iKeyman GUI using the strmqikm command. Open the key repository file and then navigate to the Signer Certificates section using the drop-down list in the middle of the window. Double-click each certificate label in turn to view the certificate.
• Use the command-line to find out which certificates are in the key repository by listing their labels. Then display the details for each certificate in turn until you find the one with the correct subject:

runmqckm -cert -list -db key.kdb -pw password

runmqckm -cert -details -db key.kdb -pw password -label certlabel

Since this is a 575010 error, you'll probably find that the CA certificate is missing so this is the CA certificate you need to add. Follow the steps described in the product documentation Adding a CA certificate (or the public part of a self-signed certificate) into a key repository, on UNIX®, Linux or Windows systems topic to add the missing CA certificate.

However, it's possible that you might find the CA certificate is already present. In that case, check whether it's a root CA: generally a root CA certificate has Subject DN and Issuer DN fields set to the same value. If the CA certificate is not a root CA then take the Issuer DN and look for the certificate which has that as its Subject DN: add that certificate if it's missing. Repeat the process as many times as necessary until you reach a root CA certificate that's in the key repository.

Important final steps

Finally, once all certificates are in place, remember that MQ caches the key repository in memory so that changes don't take effect immediately. To clear the cache:

• For a client application, restart the application.
• For a queue manager, issue the REFRESH SECURITY TYPE(SSL) MQSC command. Note that this command will terminate any active SSL/TLS channels, so take care to run it at a suitable time.

People often forget to clear the cached key repository and then wonder why their changes don't take effect. If you change your key repository in any way, always remember to clear the cached data.

The ability to examine the certificates and clear the cached key repository state are useful for many SSL/TLS connectivity problems. Now you know how to examine the certificates, you can use this technique to fix other AMQ9633 errors using the numeric validation error and the certificate details.