How to mock SFTP servers in JUnit 5 using Testcontainers and atmoz/sftp

Overview

JUnit 5 + Testcontainers + SFTP (atmoz/sftp)

If your application communicates with FTP / SFTP servers you might want to test the SFTP interactions as well and you might be wondering how to setup a Mock SFTP Server in your integration tests. This post is about how to use Testcontainers together with the atmoz/sftp Docker container and JUnit 5 to test against a fake SFTP server in your integration tests.

Under Windows, you need to enable WSL 2 in your Docker Engine to be able to mount the by @TempDir created Windows directory inside your Docker Container. Using Hyper-V will not work.

Dependencies

Using Gradle, we need the following dependencies to include JUnit 5 Jupiter and Testcontainers into our application. We are using testImplementation here to make those dependencies only available for tests.

dependencies{
    ...
  
    def junitVersion=5.6.2    
    testImplementation "org.junit.jupiter:junit-jupiter-params:${junitVersion}"
    testImplementation "org.junit.jupiter:junit-jupiter-engine:${junitVersion}"
    testImplementation "org.junit.jupiter:junit-jupiter-api:${junitVersion}"
    testImplementation "org.testcontainers:testcontainers:1.15.3"
    testImplementation "org.testcontainers:junit-jupiter:1.15.3"
}

Preparations using @TempDir

In theory, we would create multiple directories by using the @TempDir annotation, and we create a static sftpHomeDirectory that is available before the GenericContainer gets created, which is right after the constructor call.

Now this would have been the ideal implementation of the first try:

    @TempDir
    File sourceDirectory;

    @TempDir
    static File sftpHomeDirectory;

    @Container
    private GenericContainer sftp = new GenericContainer("atmoz/sftp")
                .withExposedPorts(PORT)
                .withFileSystemBind(sftpHomeDirectory.getAbsolutePath(), "/home/" + USER + REMOTE_PATH, BindMode.READ_WRITE)
                .withCommand(USER + ":" + PASSWORD + ":1001");

Sadly due to restrictions of Junit 5´s @TempDir implementation, it´s currently not possible to use the annotation on multiple fields without assigning the same directory to all fields. https://github.com/junit-team/junit5/issues/1967#issuecomment-821023920

We therefore need to create a single temp directory and use the @TempDir just a single time instead of multiple times and we have to create all other directories inside the temp directory manually.

We´d assume to just use the default setup() method annotated by @BeforeEach like this:

    @BeforeEach
    public void setup() {
        sourceDirectory = new File(directory.getAbsolutePath(), "source");
        sftpHomeDirectory = new File(directory.getAbsolutePath(), "sftp");
        sourceDirectory.mkdir();
        sftpHomeDirectory.mkdir();
    }

But there we have a problem. As I´ve mentioned above, the field initialization of the GenericContainer happens right after the constructor was called, but the setup() call happens after the TestClass instance has been initialized. We´d get a NullPointerException when our GenericContainer is being initialized, as the sftpHomeDirectory is still null untill the setup() method has been executed.

First Solution (read-permissions, but write-permissions issue)

If we now want to share our FTP server between all test runs within the class (by making it static), we need to initialize the directory statically as well. We could of course still initialized all other directories within the @BeforeEach method and delete them or clear their contents in the @AfterEach hook if we want a fresh basis before each test is started, but we sadly cannot rely on the auto-cleaning feature of JUnit´s TempDir in this case as we have declared it static.

    @TempDir
    static File directory;

    static File sourceDirectory;

    static File sftpHomeDirectory;

    //@Container
    private static GenericContainer sftp;

    @BeforeAll
    public static void staticSetup() {
        sourceDirectory = new File(directory.getAbsolutePath(), "source");
        sftpHomeDirectory = new File(directory.getAbsolutePath(), "sftp");
        sourceDirectory.mkdir();
        sftpHomeDirectory.mkdir();

        sftp = new GenericContainer("atmoz/sftp")
                .withExposedPorts(PORT)
                .withFileSystemBind(sftpHomeDirectory.getAbsolutePath(), "/home/" + USER + REMOTE_PATH, BindMode.READ_WRITE)
                .withCommand(USER + ":" + PASSWORD + ":1001");

        sftp.start();
    }

This solution works, but there is a problem with the permissions I haven´t figured out yet, so the user has only read permissions and can pull files you´ve written into the sftpHomeDirectory beforehand.

No success with trying to give full permissions to the directory with 1777, which is why I´m assuming there is likely something wrong with how testcontainers applies the command.

When I exec into the docker container (requires run with debugger and stopping at a breakpoint, otherwise the sftp server will be shut down again in no time), I can chmod the directory testwise and pushing also works then.

Working Solution (read + write permissions)

I´m creating the upload directory now manually by using ImageFromDockerfile, which allows to extend the base image with custom commands, as if you would create a Dockerfile yourself.

        private static final GenericContainer sftp = new GenericContainer(
                new ImageFromDockerfile()
                        .withDockerfileFromBuilder(builder ->
                                builder
                                        .from("atmoz/sftp:latest")
                                        .run("mkdir -p /home/" + USER + "/upload; chmod -R 007 /home/" + USER)
                                        .build()))
                .withExposedPorts(PORT)
                .withCommand(USER + ":" + PASSWORD + ":1001:::upload");

(Not sure, why I need to give "other" permission to executable as well, instead of user/group permissions for reading and writing 660 which I originally expected, but as this is just for testing purposes I don´t bother, but still interested in the reason if anyone can give a hint :) )

I´ve given up on the idea to mount the TempDirectory directly into the container, although it works now.

Instead, I´m using the provided withCopyFileToContainer method from Testcontainers directly, to copy the required files afterwards in the created user directory.

sftp.withCopyFileToContainer(MountableFile.forHostPath(dummyFile.getAbsolutePath()), "/home/" + USER + "/" + dummyFile.getName());

But be aware: you need to call withCopyFileToContainer before the container has started, otherwise the files are not copied. This is the reason why we can´t use the @Container annotation that auto-manages the container lifecycle. Instead we need to use

    @BeforeAll
    public static void staticSetup() throws IOException {
      //copy your files to the sftp
      sftp.withCopyFileToContainer(MountableFile.forHostPath(someFile), "/home/" + USER + "/upload/" + someFile.getName());
      sftp.start();
    }

    @AfterAll
    static void afterAll() {
        sftp.stop();
    }

Conclusion

To provide a full working example of the "fake" FTP server running with TestContainers and JUnit 5, I put everything together to as sample test "MyTest". The "withFileSystemBind" method is uncomment - I do not see the need for it anymore when you can also just use withCopyFileToContainer and therefore do not recommend using it. Anyway, I left it there:

import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.io.TempDir;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.images.builder.ImageFromDockerfile;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.MountableFile;

import java.io.File;
import java.io.IOException;

@Testcontainers
public class MyTest {

    private static final int PORT = 22;
    private static final String USER = "user";
    private static final String PASSWORD = "password";
    private static final String FILE_NAME = "test.txt";
    private static final String REMOTE_PATH = "/upload/";

    @TempDir
    private static File directory;

    private static final GenericContainer sftp = new GenericContainer(
            new ImageFromDockerfile()
                    .withDockerfileFromBuilder(builder ->
                            builder
                                    .from("atmoz/sftp:latest")
                                    .run("mkdir -p /home/" + USER + "/upload; chmod -R 007 /home/" + USER)
                                    .build()))
            //.withFileSystemBind(sftpHomeDirectory.getAbsolutePath(), "/home/" + USER + REMOTE_PATH, BindMode.READ_WRITE) //uncomment to mount host directory - not required / recommended
            .withExposedPorts(PORT)
            .withCommand(USER + ":" + PASSWORD + ":1001:::upload");


    @BeforeAll
    public static void staticSetup() throws IOException {
        File sftpTestFile = new File(directory.getAbsolutePath() + "/" + FILE_NAME);
        sftpTestFile.createNewFile();

        //copy your files to the sftp
        sftp.withCopyFileToContainer(MountableFile.forHostPath(sftpTestFile.getPath()), "/home/" + USER + "/upload/" + sftpTestFile.getName());
        sftp.start();
    }

    @AfterAll
    static void afterAll() {
        sftp.stop();
    }
}

Testcontainers will create a container that can be expresses by creating a Dockerfile that looks something like this:

FROM atmoz/sftp

RUN mkdir -p /home/myuser/upload; chmod -R 007 /home/myuser

EXPOSE 22

CMD ["myuser:mypassword:1001:::upload"]