Back to Blog Archive

Downloading files with Mule’s SFTP Connector

Posted on: March 22, 2018
Author:
Justin

Introduction

Mule’s SFTP connector allows you to download and upload files from/to an SFTP server. In most cases the SFTP transport does a pretty good job, however things start to get a bit tricky when you want to target a requirement which Mule does not support OOTB, such as:

  • Downloading a file mid-flow based on some criteria already present in your Mule message
  • Downloading files in a particular order (future blog post on this topic!)
  • Querying the list of files on an SFTP server before attempting to download any, and so on.

In this blog post we explore how to download a file, or a list of files, from an SFTP server while being conscious of Mule’s component lifecycle. This is especially important because we do not want to create SSH sessions arbitrarily, but rather we want to maintain a small connection pool as much as possible.

Reasons why this is necessary

In my years working at Ricston I’ve seen several solutions to the above requirements, most often using the Mule Module Requester or maintaining a Jsch session in a Java component (don’t do this, otherwise you need to maintain the connection lifecycle yourself). The Mule requester module does have its benefits but it leaves a bit to be desired. This is not because the module is badly written, actually I think Sebastian Beltramini (the author of the module) did a pretty good job of attempting to fill a gaping hole in Mule’s endpoint model. This is purely because the module delegates any processing to the MessageRequester classes, as described on the module’s documentation page. Moreover, the SftpMessageRequester makes some assumptions that I do not agree with, such as:

  • Arbitrarily choosing the first file when listing the available files in the SFTP server
  • Assuming that any post processing should happen on any files it downloads
  • Not being expressive enough to tie the current Mule message to the SftpNotifier

Thankfully, Mule 4 is adopting a more operation-based model for the connectors, hoping to move away from the previous endpoint model as Mariano Gonzalez describes in this blog post. This blog post centres more around the new File connector, but it seems the new SFTP connector will support the same set of operations.

Implementation

Downloading a file, or a list of files, from an SFTP server can all be driven by a single Java component, or a custom MessageProcessor (we only explore Java components here). With a Java component, we need to:

  • Implement a method which executes when the application is starting. This can be done by implementing the org.mule.api.lifecycle.Startable interface.
  • Implement a method which executes whenever a Mule message arrives at the Java component. This can be done by implementing the org.mule.api.lifecycle.Callable interface.

This component must be configured to be a singleton-object. This is because we want to tie this Java component to the Mule application’s lifecycle, that is, it should only be initialised once when the application is starting, and disposed of once when the application is stopping. When the application is starting, we want to build an inbound endpoint that allows us to query for files against the SFTP server. Note that this inbound endpoint does not poll for files automatically, because it is not tied to an SftpMessageReceiver.

Similarly, when a Mule message is passed to the Java component at runtime, we want to use certain information on the Mule message to download the file and return an InputStream with the contents of the file.

This component can be used in a Mule XML flow such as the following:

The flow above uses a file (called an INSTRUCT file) that tells Mule which files to download. You can consider this INSTRUCT file as a form of ACK file which verifies that some other files were sent to the SFTP server successfully.

This project is up on GitHub at this link: https://github.com/ricston-git/download-files-from-sftp.

The README file should be enough to get more information on how to run the project and try it out for yourself.

Hope this helps!

Author:
Justin

4 Comments for “Downloading files with Mule’s SFTP Connector”

  1. vijay dhanakodi says:

    Hi Justin,

    I have the same use case which i need to read file in mid of the flow. I set my autoDelete option to false but still it’s deleting the file.
    What could be the problem?

    Thanks,
    Vijay D

    • Justin says:

      Hi Vijay,

      It depends which file you’re referring to. Be aware that in my example, there are two endpoints defined:

      One which pulls the INSTRUCT file from the SFTP (i.e. the inbound endpoint)
      A generic endpoint, which is used to pull files from the SFTP in the middle of the flow.

      If you want to keep the file which you are reading in the middle of the flow, set the autoDelete flag on the , not the inbound endpoint. At the end of the blog, there’s a link to a GitHub repository with an example, you may find it useful. If you still cannot trace the problem, I would suggest that you open org.mule.transport.sftp.SftpStream and set a breakpoint on the postProcess() method. That is the method which is invoked when the file is deleted from the SFTP server.

      Cheers,
      Justin

      • vijay dhanakodi says:

        Thanks for the reply justin. I am not that much expert in digging into the source code.
        Can you help me what code should I need to change in order to restrict file delete after processing?

        Thanks,
        Vijay D

        • vijay dhanakodi says:

          In the sftpstream class by default autoDelete is set to true.
          private boolean autoDelete = true;
          How can I set this to false?

Comments

Contact Us

Ricston Ltd.
Triq G.F. Agius De Soldanis,
Birkirkara, BKR 4850,
Malta
MT: +356 2133 4457
UK: +44 (0)2071935107

Send our experts a message

Need Help?
Ask our Experts!