Back to Blog Archive

MuleSoft and Amazon Web Services (S3, SQS)

Posted on: April 3, 2019
Author:
Poorna Goyal

In this blog, I’d like to walk you through a simple demo app to show how to use AWS S3 and SQS connectors in MuleSoft’s Anypoint Platform.

Amazon Web Services (AWS) is a secure cloud services platform, offering computing power, database storage, content delivery and other functionality to help businesses scale and grow. MuleSoft provides different AWS connectors to connect with Amazon Web Services such as Simple Storage Service, Simple Queueing Service, EC2, SNS, etc.

Before we start, let us first look into the use cases where these connectors can be used.

Use Cases

AWS S3 is a reliable object-based storage option, allowing integrated applications the ability to read, write, copy and delete data. Some key use cases are:

  • Application data storage. AWS S3 is ideal for fast, reliable writing and retrieval of data.
  • Store configuration data in S3. The file data can be read, cached and refreshed based upon some default time frame. More importantly, this is useful when the configuration data is changing frequently and can be refreshed without redeploying the MuleSoft application.

AWS SQS offers queue services similar to message brokers such as Active MQ and Anypoint MQ. Some use cases are as follows:

  • Use cases for using messaging brokers in general, such as decoupling systems, scalability and reliability.
  • AWS S3 can be configured to send a message to SQS when a file arrives in the bucket (like a trigger to start the Mule flow).

Now let us take a simple scenario on how to configure and use AWS connectors with MuleSoft.

Scenario

In this scenario, we have a simple MuleSoft API using HTTP Listener to receive requests which are published on a queue using Amazon SQS. AWS SQS queue receiver in MuleSoft receives the message and writes a file in Amazon S3.

By the end of this walk-through, you will know how to create the access keys needed for AWS, create S3 bucket, create SQS queue and configure the S3 and SQS connectors in Mule to interact with each other.

Prerequisites

  • AWS Account
  • AWS provides a free trial account for a year.

Setup AWS Account

First, to make use of AWS connectors, access key and secret key are required which can be retrieved from the AWS account.

AWS Console link: https://aws.amazon.com/console/.

After logging into AWS console, go to “My Security Credentials” as shown below:



Click on “Create New Access Key” under “Access Keys” (access key ID and secret access key) tab, a popup window will open as shown below:

Download or store the access key and secret key. Once you have the access key and secret key, you are good to go.

Amazon S3 Configuration

Go to Amazon S3 in AWS Console. Select/Create the bucket with (default) permissions as follows:



Amazon SQS Configuration

Go to Amazon SQS in AWS Console, Create a new queue with default settings as below:



Once the queue is created, note the queue name and URL as below:



For this Proof-of-concept (PoC), the queue should at least have read and write access. The permissions can be modified by clicking on the “Permissions tab”, “Edit Policy” document. Below is the (default) permissions setup for our queue:



A dedicated user for integration should be used instead of the root user. The user creation and access management are maintained by DevOps person. User credentials are not required for MuleSoft AWS connectors, as explained earlier we only require access key and secret key for a successful connection.

Implementation

Configure global elements for S3, SQS and HTTP:

Now let’s test the S3 Connector Configuration after populating access key and secret key as below:



Populate S3 bucket name and key as below:



Test out the SQS Connector Configuration after populating the necessary fields as below:



We have successfully tested S3 and SQS connections to confirm that the AWS setup is proper and accessible.

Flow

The Mule flows look like below:



Maven Dependencies

The following maven dependencies can be added in pom.xml:

The AWS S3 connector supports encryption of file content, limiting the length of file content along with other functionalities (refer to MuleSoft documentation for more details).

Additionally, it supports multiple operations, each having specific functionality. In this PoC, “Create object” is used to support file creation. Functions such as “Get object”, “Copy object”, “Delete object”, etc can be used in a similar way.

Test Results

Run the code in Anypoint Studio. Send any payload using postman.

Postman:



Anypoint Studio Console:



Check-in S3 AWS console – a file is created:



Quick Links

Please do let me know your thoughts on this blog in the comments section below.

Author:
Poorna Goyal

Comments

Send our experts a message

Need Help?
Ask our Experts!