One-to-many file transfers using IBM WebSphere MQ File Transfer Edition - Middleware News

This article describes how to set up so that you can perform one-to-many transfers using the queue monitoring, file-to-message and message-to-file functions, which are all new in V7.0.3.
A new feature in IBM WebSphere MQ File Transfer Edition V7.0.3 (hereafter called FTE) enables you to perform a single-source transfer that sends a file to multiple destination agents. Before V7.0.3, you could do a one-to-many file distribution by constructing an Ant script with all of the individual transfers and then execute the script as part of the transfer command. This article shows you how the new features in V7.0.3 make it much easier to send files to multiple destinations by running a single file transfer.
The article show you how to set up the file-to-message functionality to send a single file to a WebSphere MQ publish/subscribe topic using a WebSphere MQ alias queue. You will set up subscribers that will subscribe to the topic, and the messages published to the topic will be written to the local subscription queues. You will then use FTE queue monitoring to monitor the queues populated by the subscriptions. If new messages are present on the queues when each monitor interval passes, a message-to-file transfer will be triggered, delivering the original file to the associated remote agents. You will also use metadata so that the transfer captures the originating transfer ID, file name, and source agent, so that the original single file-to-message transfer can be linked to all of the resulting message-to-file transfers, which is useful if you are logging the transfers in a database logger.
Experienced WebSphere MQ system administrators may be more comfortable using an environment like this to distribute files, as it builds on WebSphere MQ components such as queues, topics, and publish/subscribe, which may be more familiar than coding complex Ant scripts.

Scenario and solution overview

An admin department in New York wants to send numerous files to offices around the world. They want to be able to place files in a directory on their server, and then at intervals during the day, have these files automatically transferred to multiple remote offices.
This article shows you how to set up an environment to achieve this by using directory monitoring, file-to-message transfers, WebSphere MQ publish/subscribe, queue monitoring, and message-to-file transfers. You will first create a WebSphere MQ topic and subscribers to that topic. The initial file-to-message transfer will populate the topic with the contents of the message, and each subscription will then populate a local queue when new messages appear on the topic. Queue monitors will be used to monitor the subscription queues and initiate message-to-file transfers to their associated remote agents.
For details of WebSphere MQ publish/subscribe setup, see the topic Configuring publish/subscribe messaging in the WebSphere MQ V7 information center.
The article demonstrates the technique using one source and two destination agents. You can use the concepts described here to do transfers to a greater number of destinations – the steps just have to be repeated for any new agents you wish to integrate into the setup.

Prerequisites

Before doing the V7.0.3 setup, you need to define the following FTE environment:

1. On a server, define and install the following components. A Microsoft® Windows® server is used as the "New York" admin server.
   1. Install WebSphere MQ V7.0.1 or later, including WebSphere MQ Explorer.
   2. Define a WebSphere MQ queue manager called CENTRAL.ADMIN, which will be used as the coordination, command, and agent queue manager.
   3. Define a listener on an unused port and start this listener. It will be used by the remote agents.
   4. Install WebSphere MQ FTE V7.0.3 Server Edition.
   5. As part of the server installation, define an agent called CENTRAL.AG1. As it is a local agent to the queue manager, it can be defined as a bindings connection agent. Once defined, the agent must be enabled for file-to-message transfers by editing the agent.properties file and adding enableQueueInputOutput=true. Restart the agent after editing his file.
   6. Install the WebSphere MQ FTE V7.0.3 WebSphere MQ Explorer plug-in from the Remote Tools and Documentation installation.
2. On the two servers that are acting as remote stores (Paris and London), install WebSphere MQ FTE Server or Client Edition and define the following items:
   1. A client connection agent on each server. One server should have an agent called PARIS.AGENT, and the other, LONDON.AGENT. These should be defined to client connect to the CENTRAL.ADMIN queue manager using the port that the listener defined earlier is listening on, and should use the CENTRAL.ADMIN queue manager as their agent, command, and coordination queue manager.
   2. Once defined, enable the agent for file-to-message and message-to-file transfers by editing the agent.properties file and adding enableQueueInputOutput=true. Restart the agent after editing this file.

If you are new to FTE and need information setting it up, see the article WebSphere MQ File Transfer Edition walkthrough. Once basic setup is complete, you are ready to start defining the new components for single file to multiple destination transfers.

Defining a topic

Create the topic you intend to populate with messages using the following command from a runmqsc session for the CENTRAL.ADMIN queue manager:

Listing 1. The define topic command

C:\>runmqsc CENTRAL.ADMIN 
5724-H72 (C) Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. 
Starting MQSC for queue manager CENTRAL.ADMIN. 
DEFINE TOPIC(DAILY.PRICING.INFO) TOPICSTR(DAILY.PRICING.INFO)

In WebSphere MQ V7, you can define an alias queue that points to a topic. Use the new file-to-message functionality to send an ungrouped single message to this alias queue and populate the topic with the contents of this message. You can then use WebSphere MQ publish/subscribe to distribute this message to anyone who subscribes to the topic. Because of a WebSphere MQ restriction, you cannot send a group of messages to the topic, which means that you cannot split source files using the –qs, -dqdt, or –dqdb options.

Creating the alias queue

Create an alias queue to point to this topic using the following command from a runmqsc session for the CENTRAL.ADMIN queue manager:

Listing 2. The define qalias command

C:\>runmqsc CENTRAL.ADMIN 
5724-H72 (C) Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. 
Starting MQSC for queue manager CENTRAL.ADMIN. 
DEFINE QALIAS(PRICING.QUEUE) TARGTYPE(TOPIC) TARGET(DAILY.PRICING.INFO) +
DEFPSIST(YES)

Defining the subscriptions

Next, create the local subscriptions to the topic. This example sets up two subscriptions. You can use this procedure as many times as necessary to set up subscriptions to any number of queues. For the London store:
1. Create a local queue called LONDON.PRICING.INFO.

Listing 3. Defining the local queue

C:\>runmqsc CENTRAL.ADMIN 5724-H72 (C) 
Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. Starting MQSC for queue 
manager CENTRAL.ADMIN. 
DEFINE QLOCAL(LONDON.PRICING.INFO)

2. From a runmqsc session for the CENTRAL.ADMIN queue manager, define a subscription:

Listing 4. The define subscription command

C:\>runmqsc CENTRAL.ADMIN 5724-H72 (C) 
Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. Starting MQSC for queue 
manager CENTRAL.ADMIN. 
DEFINE SUB(LONDON.PRICING.SUB) TOPICSTR('DAILY.PRICING.INFO/#') +
    TOPICOBJ(DAILY.PRICING.INFO) DEST(LONDON.PRICING.INFO) +
    DESTCLAS(PROVIDED) DESTQMGR(CENTRAL.ADMIN)

Repeat the above steps for the Paris subscription, replacing LONDON.PRICING.INFO with PARIS.PRICING.INFO.

Verifying the setup

To confirm that the setup is correct, use the amqsput MQ sample program to send a message to the alias queue. It should result in an identical message being delivered to the LONDON.PRICING.INFO and PARIS.PRICING.INFO queues. Run the amqsput program with QUEUE and QUEUE MANAGER as parameters, type some text, and press Enter to send it. Press Enter again to end input to the sample program.

Listing 5. Command to send a test message to the alias queue

C:\WMQFTE\bin>amqsput PRICING.QUEUE CENTRAL.ADMIN
Sample AMQSPUT0 start
target queue is PRICING.QUEUE
This is my test publication
Sample AMQSPUT0 end

Use the WebSphere MQ sample program amqsget to drain the queues and confirm that the messages have arrived:

Listing 6. Example commands to drain both subscription queues

C:\>amqsget LONDON.PRICING.INFO CENTRAL.ADMIN
Sample AMQSGET0 start
 
C:\>amqsget PARIS.PRICING.INFO CENTRAL.ADMIN
Sample AMQSGET0 start message 

Creating the queue monitors

The next step is to set up two monitors to monitor the queues populated by the subscriptions, and send the contents of the queues to files on remote FTE agents. You can use the message-to-file feature to do this and use FTE queue monitoring to monitor the queues for the arrival of new messages. The article will explain the ways you can set this up and how you can also link together the initial file-to-message publication of the file to the topic to the second message-to-file transfer, which is useful for auditing and tracking purposes when you want to identify the origin of a file.

Creating the message-to-file task XML

Use the command-line fteCreateTransfer utility to create two task XML files, one for each agent. You cannot use the WebSphere MQ Explorer plug-in to create the monitor, because the task generation using the MQ Explorer plug-in does not expose the option to select messages from a queue by groupId. Even though you are not using grouped messages on the publication side, you must enable the groupId selector on the receiving side so that multiple messages on the queue are received and written to multiple files. Create the task XML with the following options:

Parameter	Value	Explanation
-sa	CENTRAL.AG1	Source agent
-da	LONDON.AGENT	Destination agent
-df	/pricingInfo/${WMQFTEFileName{token=-1}{separator=/}}	The –df value can use variable substitution to create a unique file name. You can read values from the metadata in the WebSphere MQ properties of the message read and use them to create a file name at the destination, basing it on the name of the file that has been transferred in the initial file-to-message transfer. In this example, the messages being sent to the topic have metadata in the WebSphere MQ properties of the message, including a header called WMQFTEFileName, which is the name of the originating source file. You can use it to determine the name of the destination file to write the message data to. By defining this file as /pricingInfo/${WMQFTEFileName{token=-1}{separator=/}} the transfer will write the data in the message that has triggered the transfer into a file in the pricingInfo directory and into a file based on just the file name and not the full path of the originating source file. Therefore, if the source file was /menus/pricing/Christmas.txt, the destination would be /pricingInfo/Christmas.txt. The forward slash separator works on Windows because all directory separators are treated as forward slashes in FTE.
-t	text	The mode in which the messages are transferred from the queue into a file. This can be set to “binary” or “text”.
-gt	c:\transferXML\london.xml	Location to write the transfer XML file
-de	error	If overwrite is set and if a destination file of the same name already exists, then the destination file will be overwritten. If error is set and the destination file already exists, then the transfer will fail with the message "BFGIO0006E: File “" already exists."
-sqgi		The -sqgi option tells the monitor to process complete groups of messages only, one group at a time. Even though you will not be placing groups of messages onto the destination queue, single ungrouped messages will be treated as a single group of messages when this option is used, and therefore processed individually. If this option is omitted, then transfer defaults to reading all messages from the queue and writing them to the destination file, which could include messages contained in other groups and messages from incomplete groups, as well as the messages in the group that triggered the transfer!
-md	OriginatingSourceAgent=${WMQFTESourceAgent}, OriginatingTransferId=${WMQFTETransferId}, OriginatingFileName=${WMQFTEFileName}, OriginatingTransferMode=${WMQFTETransferMode}	Extra metadata can be passed to the transfer command. It will be permanently logged in the transfer XML, and you can use it to link the initial file-to-message transfer and the resulting message-to-file transfer. If you use the values below, you can log the originating agent, originating source file, originating transfer ID, and originating transfer mode details, which can be useful if you want to identify the origin of a file.
-sq	LONDON.PRICING.INFO	The queue to read messages from

Listing 7: Example fteCreateTransfer command with options

fteCreateTransfer -sa CENTRAL.AG1 
    –da LONDON.AGENT 
    -df “/pricingInfo/${WMQFTEFileName{token=-1}{separator=/}}" 
    -t text 
    –gt c:\transferXML\london.xml 
    –de error 
    –sqgi 
    –md OriginatingSourceAgent=${WMQFTESourceAgent}, 
     OriginatingTransferId=${WMQFTETransferId}, 
     OriginatingFileName=${WMQFTEFileName}, 
     OriginatingTransferMode=${WMQFTETransferMode} 
    –sq LONDON.PRICING.INFO

Repeat for the Paris transfer

Repeat the creation of a transfer XML file for the transfers to the Paris agent, replacing LONDON.PRICING.INFO with PARIS.PRICING.INFO, and the destination agent name with PARIS.AGENT.

Updating the transfer XML files

Before the XML files can be included in a monitor, you must edit them to include an option that isn't available from the command line. In order for the correct message to be processed after the monitor has triggered the transfer, you must edit the transfer XML that has been created by the above steps. The monitor must pass the groupId of the message that has caused it to trigger, to the transfer definition which is being executed by that monitor.
Adding groupId=”${groupId}” to the queue element ensures that the message that the monitor was triggered by will be the one read by the resulting message-to-file transfer. If it is not added into the transfer template XML, then instead of the transfer reading the message that triggered the monitor, the resulting message-to-file transfer will just read from the start of the queue and pull off the first message found on the queue, which might not be the message that you intended to consume!
Edit the XML file you have just created and change the following line:

Listing 8: XML changes

From:     LONDON.PRICING.INFO  
To:          LONDON.PRICING.INFO

Repeat for the PARIS monitors transfer XML file.

Defining the monitors

Define the monitors using the fteCreateMonitor command-line script. The WebSphere MQ Explorer plug-in does not let you reference a previously created XML file. One monitor is required per queue, and therefore you need to create two monitors, one to monitor the LONDON.PRICING.INFO queue and one for the PARIS.PRICING.INFO queue. Using the fteCreateMonitor command-line utility, set up the monitor using the following options:

Parameter	Value	Explanation
-ma	CENTRAL.AG1	Agent to define the monitor for
-mq	LONDON.PRICING.INFO	Queue to monitor
-mn	"London Monitor"	Name of monitor
-mt	c:\transferXML\london.xml	Full path to transfer XML file created in earlier steps
-tr	completeGroups	Although groups of messages will not be sent to the alias queue, you must set the monitor to trigger on complete groups. Trigger condition in this instance is completeGroups and would only allow the monitor to trigger the transfer XML file if a complete group of messages exists on the monitored queue. Single messages without the WebSphere MQ flag of Message in group are treated as a unique group of messages and processed individually.
-dv	WMQFTEFileName="UNKNOWN_FILE_NAME", WMQFTESourceAgent="UNKNOWN_SOURCE_AGENT", WMQFTETransferId="UNKNOWN_ORIGINAL_TRANSFER_ID", WMQFTETransferMode="UNKNOWN"	This example expects metadata in the WebSphere MQ properties of the first message in the group that triggers the transfer. If a message appears on the queue without any metadata, you can use the –dv option to pass default values to the transfer request as in the example we rely on four being set. This example expects the following metadata, all of which are added as default when a file-to-message transfer is initiated with the –qmp true option: ${WMQFTESourceAgent} ${WMQFTEFileName} ${WMQFTETransferId} ${WMQFTETransferMode} Therefore, you can set default values for these if they do not appear, which means that a default file called UNKNOWN_FILE_NAME is created if a message with metadata in the WebSphere MQ properties of the first message ever appears.
-pu	minutes	Units to poll the monitor in
-pi	5	Interval, related to the -pu option, to poll the monitor in

Listing 9: Example fteCreateMonitor command with options

fteCreateMonitor -ma CENTRAL.AG1 
    -mq LONDON.PRICING.INFO 
    -mn "London monitor" 
    -mt C:\transferXML\london.xml 
    -tr completeGroups –dv WMQFTEFileName="UNKNOWN_FILE_NAME", 
    WMQFTESourceAgent="UNKNOWN_SOURCE_AGENT",
    WMQFTETransferId="UNKNOWN_ORIGINAL_TRANSFER_ID",
    WMQFTETransferMode="UNKNOWN" 
    -pi 5 
    -pu minutes

Repeat this setup for the Paris queue monitor. The creation of the monitor should be repeated for the Paris monitor, using PARIS.PRICING.INFO queue and defining a new monitor name.

Setting up a directory monitor to trigger the file-to-message transfer

Creating the transfer XML

To complete the setup, create a directory monitor that will transfer any files placed into a directory to the alias queue. Use the command-line utilities to create the transfer XML and the directory monitor which will trigger the transfer. First, create the transfer XML file using the fteCreateTransfer command with the following options:

Parameter	Value	Explanation
-sa	CENTRAL.AG1	Source agent's name
-da	CENTRAL.AG1	Destination agent's name
-dq	PRICING.QUEUE	Destination queue name
-t	text	Transfer mode
-gt	c:\transferXML\outboundfiles.xml	Location to write the transfer XML file
-qmp	true	Enable WebSphere MQ message properties on the message
	${FilePath}	The file to transfer can be set to this variable, which means that the file that triggers the monitor will be passed to the XML and will be transferred.

Listing 10: Example fteCreateTransfer command with options

fteCreateTransfer -sa CENTRAL.AG1
    -da CENTRAL.AG1 
    -dq PRICING.QUEUE 
    -t text 
    -gt c:\transferXML\outboundfiles.xml 
    -qmp true 
    ${FilePath}

Creating the directory monitor:

After creating the transfer XML, define the monitor task using the following options:

Parameter	Value	Extra information
-ma	CENTRAL.AG1	Agent to define the monitor for
-md	c:\outboundFiles	Directory to monitor
-mn	Outbound files monitor	Name of the monitor
-mt	c:\transferXML\outboundfiles.xml	Full path to the transfer xml file created in the earlier steps.
-tr	match,*	Trigger condition should be set to match on any files placed into the directory

Listing 11: Example fteCreateMonitor command with options

fteCreateMonitor 
    -ma CENTRAL.AG1 
    -md c:\outboundFiles 
    -mn "Outbound files monitor" 
    -mt C:\transferXML\outboundfiles.xml 
    -tr match,*

Verifying the setup

1. Place a file into the local directory that is being monitored by the directory monitor. It should trigger after one minute and the file will be sent to the PRICING.QUEUE, which is an alias queue that will send the contents to the WebSphere MQ TOPIC it is pointing at. Each of the subscribing queues will then get a copy of this message.
2. After the poll interval defined for the queue monitors has elapsed, the subscription queues should be drained and each message should appear at the destination as a new file.

If you examine the properties of the transfer and look at the metadata associated with the transfer, you can associate it with the original file-to-message transfer:

Frequently asked questions

1. Why use metadata in the WebSphere MQ properties of the first message on the files you want to distribute, or what happens if messages end up on the queue with no metadata in the WebSphere MQ properties of the first message?
In this example, if metadata is not present in the WebSphere MQ Properties of the first message, then the default value substitution variables are used in their place (these are set in the monitor using the –dv option). If no metadata is present in the WebSphere MQ Properties of the first message that is created by the file-to-message transfer, when this message triggers the monitor, no details can be passed to the resulting message-to-file transfer and instead the default substitution values are used. In this example, this would result in a file called UNKNOWN_FILE_NAME being produced in the destination directory. In the XML of the message to file transfer it would also mean that an unknown originating transfer, an unknown originating agent and unknown originating source file would be logged and it would not be possible to associate this transfer with the originating file to message transfer. Using the metadata allows us to easily associate the originating source file and transfer ID with the destination file.
2. What happens if the destination file already exists -- is it overwritten?
The answer depends on the value used for the –de option when the transfer XML is generated. As explained above, when the fteCreateTransfer command is executed to create the transfer definition file that will be driven by the monitor task, you can use the –de option to set what action takes place if a file with the same name already exists. If overwrite is set and a destination file of the same name already exists, the destination file will be overwritten with the new data in the message that triggered the transfer. If error is specified, the transfer will fail with the message BFGIO0006E: File already exists. If the transfer fails, the monitorGroupRetryLimit variable determines how many times the transfer is retried. The monitorGroupRetryLimit needs to be set in the agent.properties file for the agent initiating the message-to-file transfer. By default, if the agent.properties file does not contain this value, then the retry limit is 10. Once this retry limit has been reached, the transfer will be triggered again only if the agent initiating the transfer is restarted. To change the value of monitorGroupRetryLimit, update the agent.properties file and restart the agent.
3. The example transfers text files -- can I use this procedure to transfer binary files?
If you want to transfer both binary and text files, there are several approaches you can take:

• You can set up an environment identical to the one for text transfers -- an alias queue pointing at a topic where binary files are published. You would then need a whole set of queues and subscriptions to the topic where the binary files are being published. You would then need a set of queue monitors for these queues and set the transfers up so they performed a message-to-file transfer in binary mode.
• Alternatively, you can use information in the metadata in the WebSphere MQ properties of the first message to determine the file transfer mode. Due to a defect in FTE V7.0.3, the product cannot achieve the setup in this step, but an interim fix can be provided to allow for the mode attribute in the item element. When files are sent to message queues using the file-to-message feature and when message headers are enabled, one of the default message headers is WMQFTETransferMode. Once the XML has been created, in the step where the groupId option is added, the mode="text" can be changed to "${WMQFTETransferMode}".
  For example, causes the resulting message-to-file transfer to insert the value from the metadata in the message that triggered the transfer into the XML for the transfer. This approach lets you send both binary and text files to the alias queue, and as long as they have metadata in the WebSphere MQ properties of the first message, they can be correctly processed by the message-to-file transfer.

4. When attempting a file-to-message transfer to the alias queue to publish to the topic, I receive the error "BFGIO0197E: An attempt to write to a queue was rejected by the destination agent. The agent must have enableQueueInputOutput=true set in the agent.properties file to support transferring to a queue."
File-to-message and message-to-file need to be enabled because they are disabled by default. To enable this functionality, you must have enableQueueInputOutput=true set in your agents agent.properties file. Restart the agent to pick up the changes.
5. When attempting to send files to the topic, the file-to-message transfer fails with the error "BFGIO0178E: A queue write failed due to a WMQApiException with message text cc=2 rc=2417 op=put - MQPUT to PRICING.QUEUE."
The MQ return code "2417 MQRC_MSG_NOT_ALLOWED_IN_GROUP" means that the queue that you have attempted to put messages to has rejected the attempt because you are attempting to put a group of messages onto the queue. When publishing to a topic, messages that form part of a group will be rejected. The most common cause for this error is when you attempt a file-to-message transfer with the –qs, -dqdt or –dqdb options to split the source file, which is not permitted when sending messages to a topic.

Restrictions

• Single file to single message transfers only, since WebSphere MQ V7 publish/subscribe does not support grouped messages. Therefore you cannot split source files using the –qs, -dqdt, or –dqdb options.
• In a default WebSphere MQ network, the maximum message size is 4 MB. Therefore, unless the MQ setup is altered, the maximum size of a file that can be sent is 4MB.
• The file-to-message transfer that takes place must use the "–qmp true" option to add metadata to the WebSphere MQ properties of the message.
• The setup will only accept one transfer mode, text or binary. To use variable substitution to enable either file type to be transferred using the same queues, you must apply the IBM interim fix.