Back to Blog Archive

Creating Dynamic Flows in Mule using Dataweave and Spring

Posted on: July 25, 2017
Author:
Sam

MuleSoft offers a number of connectors that can be configured and used as inbound endpoints for a flow. Each inbound endpoint is a message source which receives data and triggers the execution of a flow.

A common scenario is to perform similar point to point integrations for a number of inbound endpoints where most of the processing logic is common for all messages. Each message source could be one of a number of inbound endpoints and the data received could be in one of several data formats such as JSON, XML or CSV.

Consider the example shown in the image above where a number of message sources are shown on the left hand side. Each of these message sources could correspond to a third party who are uploading files in their preferred format like one of the following; FTP, SFTP or an Amazon S3 bucket. The file should be read by Mule and transformed to a common JSON format before being uploaded to a single Amazon S3 bucket. A third party could be added or removed at any time so the number of inbound endpoints may be changing frequently.

One possible solution is to create a flow for each third party, where it’s inbound endpoint is configured and the application is packaged and deployed each time a third party is added or removed. An alternative is to generate the flows dynamically, allowing third parties to be added and removed without the need to re-package or re-deploy the application.

Generating Flows Dynamically

The Dataweave language is a tool for querying and transforming data into a variety of formats, including XML. This Dataweave script generates Mule configuration as XML. The script creates a single flow with either a Poll or File inbound endpoint as the message source, followed by some loggers. You can see that the type of endpoint used is controlled by the message payload as well as some attributes, for example the path of the file inbound endpoint.

Note: While this example uses a single dataweave script that can generate either a Poll or File inbound endpoint, you may choose to use a separate dataweave script for each type of inbound endpoint to reduce complexity.

I am using JSON as the input payload for this message transformer, but XML or an Object could be used here to the same effect. This JSON example creates the flow configuration with a file inbound endpoint which polls a directory inside my user folder.


I’ve included the generated XML file and also copied it into a configuration file in an Anypoint Studio project. The graphical equivalent shows the flow was created with the File inbound endpoint and is showing no problems with the generated configuration.

Adding Flows to the Application

When the Mule server starts it adds Mule configuration XML files to a single Spring application context. We can use Java to create new contexts (org.mule.api.MuleContext) from a collection of XML files. Each new inbound endpoint and flow are created dynamically using Dataweave and added to a new MuleContext in the application.

A Spring Bean with Singleton scope is created from a Java class (DynamicFlowController.java, which is at the end of this post). The class exposes methods to create a new MuleContext, list contexts and dispose of a context.

I have created a Mule Application that exposes three HTTP endpoints which create, list and delete contexts, each corresponding to one of the Java methods. The create endpoint accepts a JSON payload which is used to create the Mule config in Dataweave. The XML configuration for these endpoints is also included.

Sharing Common Flows with each Endpoint

Each Mule context is created containing only the flow that is generated by Dataweave. It has no access to the other flows in the application. We’d like to be able to share a set of flows between all the endpoints that contain the common processing logic.

Two ways to achieve this are:
1.) Create the MuleContext with multiple configuration files, the one created by Dataweave, and the others from the classpath. Each MuleContext would have it’s own copy of the common flows.
2.) Create a new context with all the configuration files containing flows that need to be shared. Create each context with a reference to the common context.

I will look at option 2 and create a Mule context called common-flows to contain a flow with a file outbound endpoint.

I’ve created common-flows.xml in src/main/resources/ that contains the flow ‘outboundFlow’ as above.

We need to ensure this context is created before any of the dynamically generated contexts. The Java class extends org.mule.api.lifecycle.Initialisable interface and has an initialise() method. This method is run when the Spring bean is created. We use the initialisation method to create the common-flows context from common-flows.xml.

Now when we create new contexts for each dynamic flow, we can add the shared context as a domain context. This makes the single copy of outboundFlow available to all of the dynamic contexts.

I’ve added a flow reference at the end of flows generated by dataweave to call the outbound flow.

This time I’ve included the JSON for creating a Poll inbound endpoint. Note that you can now see a flow reference is included at the end of the generated flow.


If I now run the application I can add one dynamic flow with a File endpoint and one with a Poll.
The logs are generated as below:

Note: Flow configurations are not saved, so the JSON config should be saved to and loaded from some type of persistent storage if they are required to survive application restarts. A file system, database, or object store could be used for this.

Source Code:



Author:
Sam

4 Comments for “Creating Dynamic Flows in Mule using Dataweave and Spring”

  1. Joe Baldwin says:

    Great Blogpost, A worthwhile read!

  2. Bhavinkumar Mistry says:

    Great post, first time I saw this kind of blog for MuleSoft dataweave.

  3. supraja says:

    where is the the code?

Comments

Contact Us

Ricston Ltd.
Triq G.F. Agius De Soldanis,
Birkirkara, BKR 4850,
Malta
MT: +356 2133 4457
UK: +44 (0)2071935107

Send our experts a message

Need Help?
Ask our Experts!