Back to Blog Archive

Polling on Multiple Nodes in a Mule ESB Cluster

Posted on: March 6, 2018
Author:
Justin

Introduction

It is generally understood that a Mule ESB cluster makes sure that only the primary node polls for external resources. This typically only applies to polling transports or connectors – other transports or connectors are event driven and wait for incoming messages. However, there are some exceptions where polling transports act in an active-active fashion, where all nodes in the cluster poll for resources on a single data source. This can be seen in the behaviour of the SFTP transport.

The single method which drives this behaviour is called; isPrimaryPollingInstance() on the MuleContext. Internally, this queries a PollingController, which in turn queries the Hazelcast library to check whether the current node is the primary node in the cluster. The Mule transport receivers then use this method to check whether they should poll or not. At the end of the day, a Mule ESB cluster is an active-active solution, and this behaviour proves this even further. Technically speaking, all nodes are performing a poll, however the secondary nodes simply short circuit the polling process.

There are scenarios where you may want a Mule ESB node to execute a poll regardless of the cluster member state i.e. primary or otherwise. One example is when a queuing technology offers a way of retrieving messages from a queue via a REST API such as Azure ServiceBus or Amazon SQS. The obvious question which comes next is: why would you want to use a custom HTTP polling solution rather than MuleSoft’s OOTB connectors? This blog post was borne out of a requirement where I needed 100% control of message acknowledgements, and missing one message could possibly be disastrous. At the time this solution was implemented, MuleSoft’s connectors did not support this, however since then they’ve released support for manual handling of ACKs as can be seen here. If you can rely on MuleSoft’s OOTB solutions then stick with those rather than relying on custom code from the internet ;). However, one of Mule’s strengths is its configurability, so why not use it?

The following sections describe some scenarios of how to override Mule’s default behaviour and allow polling on all cluster nodes with different transports.

Quartz Transport

The Quartz transport is mainly used to perform polling operations against an endpoint, a resource, or even just to initiate a flow. When you create a job with Mule’s XML DSL, the job always extends from; org.mule.transport.quartz.jobs.AbstractJob, which is configured to run on just the primary polling instance:

If you’re wondering what PROPERTY_JOB_DYNAMIC is, it’s just a query parameter called jobDynamic which you can supply at the end of the Quartz endpoint’s “address” field:

If you want to override this, you can simply create a custom Job which overrides the execute method above, and  performs doExecute() on the concrete job:

Note: I have overridden EventGeneratorJob to keep in line with the XML snippet above. If you want more functionality or flexibility, you can always override AbstractJob instead.

Poll Message Processor

It is a little known fact that the poll scope is really an inbound endpoint (or message receiver) acting as a scope. The class which backs the element is called MessageProcessorPollingMessageReceiver, and as with any message receiver, there must be a connector to back it up. The connector’s class is called org.mule.transport.polling.MessageProcessorPollingConnector. With this knowledge, we can use a service-overrides configuration to override the default behaviour of the message receiver (i.e. the scope) to enable cluster-wide polling:

Note: Be aware that any elements within your Mule application will inherit this new configuration, rendering all your pollers to be non-cluster aware, so use this with caution.

The custom message receiver looks like this:

A service-overrides configuration allows a Mule developer to configure the way a Mule transport behaves. In Mule, connectors (or more accurately, transports) tend to follow a standard.

For example:

  • Message Receivers are classes used to retrieve messages from external sources (similar to inbound endpoints)
  • Message Dispatchers are classes used to send messages to other Mule flows, or external services via some protocol (HTTPS, SFTP, and so on)

Each transport has its own message receiver, message dispatcher, endpoint builder, and so on. Using this services-overrides configuration, we can either extend the behaviour of the existing implementation, alter it, or rebuild it entirely. MuleSoft have documented how to configure a transport for your use on their documentation, including a handy list of possible services that you can override on each transport.

Hope this helps!
Justin

Author:
Justin

Comments

Contact Us

Ricston Ltd.
Triq G.F. Agius De Soldanis,
Birkirkara, BKR 4850,
Malta
MT: +356 2133 4457
UK: +44 (0)2071935107

Send our experts a message

Need Help?
Ask our Experts!