{"id":136,"date":"2024-09-24T05:00:00","date_gmt":"2024-09-24T05:00:00","guid":{"rendered":"https:\/\/imcodinggenius.com\/?p=136"},"modified":"2024-09-24T05:00:00","modified_gmt":"2024-09-24T05:00:00","slug":"test-spring-boot-aws-s3-with-localstack-and-testcontainers","status":"publish","type":"post","link":"https:\/\/imcodinggenius.com\/?p=136","title":{"rendered":"Test Spring Boot AWS S3 with Localstack and Testcontainers"},"content":{"rendered":"

Welcome to the last article<\/strong> in a series dedicated to integrating a Spring Boot Kotlin app with AWS S3<\/strong> Object Storage, in which we will focus on integration testing with LocalStack and Testcontainers<\/strong>. And although we will focus on Object Storage, the approach we will use can be easily replicated with other AWS services.<\/p>\n

I can guarantee that you will benefit from this tutorial regardless of whether you saw previous articles about S3Client or S3Template, or not. But, I definitely encourage you to take a look at them, too: <\/p>\n

Spring Boot with AWS S3, S3Client and Kotlin<\/a><\/p>\n

Spring Boot with Kotlin, AWS S3, and S3Template<\/a><\/p>\n

Prerequisites<\/h2>\n

Before heading to the guide, I just wanted to emphasize that today we will be working with Testcontainers<\/strong>. And this means that we must have a supported Docker environment<\/strong>.<\/p>\n

So, if you do not have Docker configured on your local and want to follow this article, please check out their documentation<\/a>.<\/p>\n

Of course, you must have Java, IDE, and Spring Boot project too, but I believe this is quite obvious <\/p>\n

Testcontainers and LocalStack<\/h2>\n

Lastly, I would like to say a few words about the Testcontainers and LocalStack, which in my opinion are a great way to test Spring Boot S3 integration (and other AWS integrations, too). <\/p>\n

Testcontainers<\/h3>\n

Testcontainers <\/strong>is a library for providing throwaway, lightweight instances of Docker containers. They are an excellent approach whenever need to test behavior dependent on external services, like AWS, or some external databases. <\/p>\n

Long story short, instead of mocking, or manual set up of some test environment, we define test dependencies as code. Then, we can run our test code and disposable containers will be started and deleted after they finish. <\/p>\n

Let\u2019s take a look at the example from Spring Boot docs<\/a>: <\/p>\n

@Testcontainers
\n@SpringBootTest
\nclass MyIntegrationTests {<\/p>\n

@Test
\n fun myTest() {
\n \/\/ …
\n }<\/p>\n

companion object {
\n @Container
\n @JvmStatic
\n val neo4j = Neo4jContainer(«neo4j:5»);
\n }
\n}<\/p>\n

The above code runs a Neo4j docker container before the tests. Of course, this is just an example, so most of the time, we will need to add some more config. <\/p>\n

Nevertheless, we can clearly see that this Testcontainers JUnit integration allows us to achieve our goal in an easy and neat manner. <\/p>\n

Localstack <\/h3>\n

Localstack<\/strong>, on the other hand, is a cloud service emulator that runs in a single container. In other words, we can run AWS applications or Lambdas without connecting to the remote cloud provider<\/strong>. <\/p>\n

And thanks to the Testcontainers module for LocalStack, we can test various AWS integrations with just a few lines of code. <\/p>\n

Again, let\u2019s take a look at the example, but this time from the LocalStack documentation: <\/p>\n

DockerImageName localstackImage = DockerImageName.parse(«localstack\/localstack:3.5.0»);<\/p>\n

@Rule
\npublic LocalStackContainer localstack = new LocalStackContainer(localstackImage)
\n .withServices(S3);<\/p>\n

You will find links to both documentation at the end of this article. But for now, let\u2019s not distract ourselves and focus on what we came here for <\/p>\n

Configure Project<\/h2>\n

If you are following my S3 series, or you already have a Spring Boot project, then those are the necessary dependencies for us today: <\/p>\n

testImplementation(«org.springframework.boot:spring-boot-starter-webflux»)
\ntestImplementation(«org.testcontainers:localstack»)
\ntestImplementation(«org.springframework.boot:spring-boot-testcontainers»)<\/p>\n

As we can see, apart from LocalStack and TestContainers, we must provide the Spring Boot Starter WebFlux. <\/p>\n

But why? <\/p>\n

Well, this is necessary to work with WebTestClient- a client we will use to test our web servers (REST endpoints). <\/p>\n

On the other hand, if you would like to set up a project from scratch, then please navigate to the Spring Initializr<\/a> and select the following:<\/p>\n

\n<\/div>\n

However, please keep in mind that LocalStack is not provided out of the box in Spring, so we must add it manually. <\/p>\n

Moreover, as we have chosen the Spring Web, the WebFlux dependency is not present, too:<\/p>\n

testImplementation(«org.springframework.boot:spring-boot-starter-webflux»)
\ntestImplementation(«org.testcontainers:localstack»)<\/p>\n

Testcontainers Singleton Approach<\/h2>\n

With all of that being done, let\u2019s head to the practice part. <\/p>\n

When working with Testcontainers, we can configure them in various ways: <\/p>\n

we can use the JUnit extension<\/strong> (Jupiter integration)- which allows us to use @Testcontainers<\/em> and @Container<\/em> annotations and makes JUnit responsible for the automatic startup and stop of containers in our tests.<\/p>\n

we can configure them manually <\/strong>in every test case, <\/p>\n

or, alternatively, we can use the singleton approach<\/strong>\u2013 in which we control containers\u2019 lifecycle manually<\/a>. But, thanks to that we can easily reuse them across multiple test classes. <\/p>\n

Of course, these are not all approaches, and based on your needs you may want to configure Testcontainers differently. Nevertheless, in this tutorial, we will focus on the manual, reusable approach<\/strong>.<\/p>\n

Introduce Base Class<\/h3>\n

Firstly, let\u2019s introduce the LocalStackIntegrationTest:<\/p>\n

@SpringBootTest(webEnvironment = RANDOM_PORT)
\nclass LocalStackIntegrationTest { }<\/p>\n

As we can see, we mark our class with @SpringBootTest<\/em> \u2013 annotation necessary to run our integration tests and inject the instance of WebTestClient later in our subclasses. <\/p>\n

Add Testcontainer<\/h3>\n

Following, let\u2019s take a look at how to instantiate a LocalStack container:<\/p>\n

companion object {
\n val localStack: LocalStackContainer = LocalStackContainer(
\n DockerImageName.parse(«localstack\/localstack:3.7.2»)
\n )
\n}<\/p>\n

Right here, we create an instance of LocalStackContainer<\/strong> and we pass the name of a Docker image \u2013 localstack\/localstack:3.7.2\u2013 to its constructor. Alternatively, if we are working only with AWS S3 Buckets service, then we can use a dedicated image- localstack:s3-latest. But personally, I am not a big fan of the latest<\/strong> tag, which can easily break our code.<\/p>\n

Additionally, we put the LocalStackContainer instance in the companion object<\/strong>. Why? Because in the next steps, we will reference it in a function annotated with @DynamicPropertySource<\/em>\u2013 and it must be static.<\/p>\n

Note related to JUnit extension<\/strong>: <\/p>\n

This is not the case here, as we want to take care of the container lifecycle manually, but, when using the Jupiter integration, containers declared as static fields will be shared between test methods. They will be started only once before any test method is executed and stopped after the last test method has executed. So, if in your case you pick the JUnit extension and don\u2019t want that to happen, then you must not put the localstack in the companion object. <\/p>\n

Control Testcontainer lifecycle<\/h3>\n

As we already know, with this approach we are responsible<\/strong> for the container lifecycle control. And although this may sound complicated, it basically means that without the extension we must start the container manually.<\/p>\n

So the companion object after the update will look, as follows: <\/p>\n

companion object {
\n val localStack: LocalStackContainer = LocalStackContainer(
\n DockerImageName.parse(«localstack\/localstack:3.7.2»)
\n ).apply {
\n start()
\n }
\n}<\/p>\n

Basically, we use the Kotlin scope function (you can learn more about it in my Kotlin course<\/a>) to invoke the start() function on the localStack instance. And as the name suggests, this function will start the container (and pull the image, if necessary).<\/p>\n

And basically, that is all we need to do here.<\/strong> With the above code, the container will be started when the base class is loaded and shared across all inheriting test classes. <\/p>\n

Of course, there is also the stop() function that we can invoke to kill and remove the container. <\/p>\n

Nevertheless, we do not have to do it<\/strong>. Why? Let\u2019s figure out. <\/p>\n

Ryuk<\/h3>\n

Ryuk is a kind of \u201cgarbage collector\u201d in Testcontainers. <\/p>\n

Whenever we run integration tests, Testcontianers core starts one more container<\/strong>: <\/p>\n

Long story short, this container is responsible for removing containers\/networks\/volumes created by our test cases. So, even if we do not clean the environment ourselves- for example with the stop() function- the Ryuk container will take care of that. <\/p>\n

Test Properties With DynamicPropertySource<\/h3>\n

With that done, we need to update our environment configuration. <\/p>\n

If we try to run our Spring Boot application at this point, our logic responsible for communication with Amazon S3 will try to reach the actual AWS instance<\/strong>. It will use the defaults, or make use of the things we configured in the application.yaml. <\/p>\n

And this is not what we want, right? Instead, we would like to connect to the Testcontainer LocalStack instance. <\/p>\n

In some examples, you might have seen the usage of application properties files. Nevertheless, if we want to be more flexible and make use of containers started on random ports, then the @DynamicPropertySource<\/strong> is our best friend here: <\/p>\n

companion object {
\n val localStack: LocalStackContainer = LocalStackContainer(
\n DockerImageName.parse(«localstack\/localstack:3.7.2»)
\n ).apply {
\n start()
\n }<\/p>\n

@JvmStatic
\n @DynamicPropertySource
\n fun overrideProperties(registry: DynamicPropertyRegistry) {
\n registry.add(«spring.cloud.aws.region.static») { localStack.region }
\n registry.add(«spring.cloud.aws.credentials.access-key») { localStack.accessKey }
\n registry.add(«spring.cloud.aws.credentials.secret-key») { localStack.secretKey }
\n registry.add(«spring.cloud.aws.s3.endpoint») { localStack.getEndpointOverride(S3).toString() }
\n }<\/p>\n

}<\/p>\n

Thanks to that annotation, we can dynamically provide values to our test environment based on the LocalStack instance<\/strong>.<\/p>\n

Of course, we must remember that methods annotated with @DynamicPropertySource<\/em> must be static<\/strong>! And that\u2019s why we use it with the @JvmStatic annotation.<\/p>\n

Utilize LocalStack AWS CLI<\/h2>\n

At this point, we have our base class ready, but before we head to the tests, I would like to show you the LocalStack AWS CLI<\/strong> and why and how to use it. <\/p>\n

As the first step, let\u2019s create the util package and add the LocalStackUtil.kt file:<\/p>\n

import org.testcontainers.containers.localstack.LocalStackContainer<\/p>\n

fun LocalStackContainer.createBucket(bucketName: String) {
\n this.execInContainer(«awslocal», «s3api», «create-bucket», «—bucket», bucketName)
\n}<\/p>\n

fun LocalStackContainer.deleteBucket(bucketName: String) {
\n this.execInContainer(«awslocal», «s3api», «delete-bucket», «—bucket», bucketName)
\n}<\/p>\n

fun LocalStackContainer.deleteObject(bucketName: String, objectName: String) {
\n this.execInContainer(«awslocal», «s3api», «delete-object», «—bucket», bucketName, «—key», objectName)
\n}<\/p>\n

As we can see, we introduced 3 helper extension functions that we will later use to create and delete buckets and objects. This way, we can easily clean up<\/strong> between the tests (we use the shared approach, right?). Moreover, it will simplify the setup process<\/strong> for each test case.<\/p>\n

The above code combines the execInContainer \u2013 which will run the passed command in our running LocalStack container, just like with the docker exec, and the awslocal\u2013 a LocalStack wrapper around the AWS CLI. So, if you\u2019ve ever been working with the AWS command line interface, then you will see that this is 1:1. <\/p>\n

Unfortunately, we must provide the command as a separate String value, because otherwise, we will end up with: <\/p>\n

OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: \u201cawslocal s3api create-bucket \u2013bucket bucket-1\u201d: executable file not found in $PATH: unknown<\/p>\n

Write Integration Test Cases<\/h2>\n

With all of that LocalStack preparation done (I know, quite a bunch of things to learn, but once you learn this, it will be a simple copy-paste), we can finally write some integration tests for our Spring Boot S3 integration.<\/p>\n

Firstly, let\u2019s create the controller package and put the BucketControllerIntegrationTest class:<\/p>\n

class BucketControllerIntegrationTest(
\n @Autowired private val webTestClient: WebTestClient
\n) : LocalStackIntegrationTest() { }<\/p>\n

As we can see, no annotations are required. We simply extend the LocalStackIntegrationTest class and inject the WebTestClient.<\/p>\n

Test No Buckets Exist<\/h3>\n

Nextly, let\u2019s introduce our first test case. If we do not do anything, we expect that no buckets exist in our S3 instance: <\/p>\n

@Test
\nfun `Given no existing buckets When getting list of buckets Then return an empty array`() {
\n val buckets = webTestClient
\n .get().uri(«\/buckets»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody(object : ParameterizedTypeReference<List<String>>() {})
\n .returnResult()
\n .responseBody<\/p>\n

assertNotNull(buckets)
\n assertTrue(buckets.isEmpty())
\n}<\/p>\n

As mentioned before, we use the WebTestClient<\/em> to make a GET HTTP request to the \/buckets endpoint. Then, we use a small hack with ParameterizedTypeReference\u2013 because the endpoint returns a list of Strings and we use Kotlin- and we obtain the response body. <\/p>\n

Lastly, we have plain assertions. We verify that the response body is not null and that our S3 bucket list is empty.<\/p>\n

Verify S3 Bucket Exists In LocalStack<\/h3>\n

Following, let\u2019s see our helper functions in action:<\/p>\n

@Test
\nfun `Given one existing bucket When getting list of buckets Then return an array with expected bucket name`() {
\n val bucketName = «bucket-1»
\n localStack.createBucket(bucketName)<\/p>\n

val expectedJson = «»»
\n [ «Bucket #1: $bucketName» ]
\n «»»<\/p>\n

webTestClient
\n .get().uri(«\/buckets»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody()
\n .json(expectedJson)<\/p>\n

localStack.deleteBucket(bucketName)
\n}<\/p>\n

This time, we utilize the createBucket and make sure that the \/buckets endpoint returns the expected JSON. Please note that this is another way to assert the response body. <\/p>\n

After all, we delete the existing bucket, so it won\u2019t affect other test cases.<\/p>\n

Assert Bucket Created Successfully<\/h3>\n

As the next step, let\u2019s take a look at how we can check if our endpoint responsible for creating new S3 buckets works. And I see two paths we can go here.<\/p>\n

The first one, using the execInContainer:<\/p>\n

@Test
\nfun `Given no existing buckets When creating bucket Then create bucket successfully`() {
\n val bucketName = «bucket-2»<\/p>\n

webTestClient
\n .post().uri(«\/buckets»)
\n .bodyValue(BucketRequest(bucketName = bucketName))
\n .exchange()
\n .expectStatus().isOk()<\/p>\n

val execResult = localStack.execInContainer(«awslocal», «s3api», «list-buckets»).stdout<\/p>\n

assertTrue(execResult.contains(bucketName))
\n localStack.deleteBucket(bucketName)
\n}<\/p>\n

The important thing to mention here is that the execInContainer returns the ExecResult. And thanks to that, we can read additional info, like stdout, stderr, or exitCode.<\/p>\n

And thanks to the stdout, we can get this JSON to verify it contains particular bucket name (or even we could parse that to an object): <\/p>\n

{
\n «Buckets»: [
\n {
\n «Name»: «bucket-2»,
\n «CreationDate»: «2024-09-19T05:28:42.000Z»
\n }
\n ],
\n «Owner»: {
\n «DisplayName»: «webfile»,
\n «ID»: «75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a»
\n }
\n}<\/p>\n

Alternatively, we can use the \/buckets endpoint once again, too:<\/p>\n

@Test
\nfun `Given no existing buckets When creating bucket Then create bucket successfully`() {
\n val bucketName = «bucket-2»<\/p>\n

webTestClient
\n .post().uri(«\/buckets»)
\n .bodyValue(BucketRequest(bucketName = bucketName))
\n .exchange()
\n .expectStatus().isOk()<\/p>\n

val expectedJson = «»»
\n [ «Bucket #1: $bucketName» ]
\n «»»
\n webTestClient
\n .get().uri(«\/buckets»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody()
\n .json(expectedJson)<\/p>\n

localStack.deleteBucket(bucketName)
\n}<\/p>\n

Test Remaining Cases<\/h3>\n

The remaining cases of our integration test use a more or less similar approach, so I will simply copy-paste them here so that you can analyze them. <\/p>\n

At this point, I am pretty sure you understand the general idea behind what I understand by testing of Spring Boot S3 integration with LocalStack, so I don\u2019t see the need for explaining them one- by one:<\/p>\n

@Test
\nfun `Given no objects existing in the bucket When getting objects of a bucket Then return an empty array`() {
\n val bucketName = «bucket-3»
\n localStack.createBucket(bucketName)<\/p>\n

val objects = webTestClient
\n .get().uri(«\/buckets\/$bucketName\/objects»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody(object : ParameterizedTypeReference<List<String>>() {})
\n .returnResult()
\n .responseBody<\/p>\n

assertNotNull(objects)
\n assertTrue(objects.isEmpty())<\/p>\n

localStack.deleteBucket(bucketName)
\n}<\/p>\n

@Test
\nfun `Given no objects When creating example object Then return created object`() {
\n val bucketName = «bucket-4»
\n val objectName = «example.json»
\n localStack.createBucket(bucketName)<\/p>\n

val expectedJson = «»»
\n {
\n «id»: «123»,
\n «name»: «Some name»
\n }
\n «»»<\/p>\n

webTestClient
\n .post().uri(«\/buckets\/$bucketName\/objects»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody()
\n .json(expectedJson)<\/p>\n

localStack.deleteObject(bucketName, objectName)
\n localStack.deleteBucket(bucketName)
\n}<\/p>\n

@Test
\nfun `Given created object When getting list of objects Then return array with one object`() {
\n val bucketName = «bucket-5»
\n val objectName = «example.json»
\n localStack.createBucket(bucketName)<\/p>\n

val expectedJson = «»»
\n [ $objectName ]
\n «»»<\/p>\n

webTestClient
\n .post().uri(«\/buckets\/$bucketName\/objects»)
\n .exchange()
\n .expectStatus().isOk()<\/p>\n

webTestClient
\n .get().uri(«\/buckets\/$bucketName\/objects»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody()
\n .json(expectedJson)<\/p>\n

localStack.deleteObject(bucketName, objectName)
\n localStack.deleteBucket(bucketName)
\n}<\/p>\n

@Test
\nfun `Given existing object When getting object by key Then return object content`() {
\n val bucketName = «bucket-6»
\n val objectName = «example.json»
\n localStack.createBucket(bucketName)<\/p>\n

val expected = «»»
\n {
\n «id»: «123»,
\n «name»: «Some name»
\n }
\n «»»<\/p>\n

webTestClient
\n .post().uri(«\/buckets\/$bucketName\/objects»)
\n .exchange()<\/p>\n

webTestClient
\n .get().uri(«\/buckets\/$bucketName\/objects\/$objectName»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody()
\n .json(expected)<\/p>\n

localStack.deleteObject(bucketName, objectName)
\n localStack.deleteBucket(bucketName)
\n}<\/p>\n

@Test
\nfun `Given existing bucket with object When deleting bucket Then bucket is removed`() {
\n val bucketName = «bucket-7»
\n localStack.createBucket(bucketName)<\/p>\n

webTestClient
\n .post().uri(«\/buckets\/$bucketName\/objects»)
\n .exchange()
\n .expectStatus().isOk()<\/p>\n

webTestClient
\n .delete().uri(«\/buckets\/$bucketName»)
\n .exchange()
\n .expectStatus().isOk()<\/p>\n

val buckets = webTestClient
\n .get().uri(«\/buckets»)
\n .exchange()
\n .expectStatus().isOk()
\n .expectBody(object : ParameterizedTypeReference<List<String>>() {})
\n .returnResult()
\n .responseBody<\/p>\n

assertNotNull(buckets)
\n assertTrue(buckets.isEmpty())
\n}<\/p>\n

Summary<\/h2>\n

And that is all for this tutorial, in which we learned how to implement integration tests <\/strong>for Spring Boot AWS S3 <\/strong>integration with LocalStack <\/strong>and Testcontainers.<\/strong><\/p>\n

I hope you enjoyed it and for the source code, please visit this GitHub repository<\/a>. <\/p>\n

Have a great day and see you in the next articles! <\/p>\n

The post Test Spring Boot AWS S3 with Localstack and Testcontainers<\/a> appeared first on Codersee | Kotlin, Ktor, Spring<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"

Welcome to the last article in a series dedicated to integrating a Spring Boot Kotlin app with AWS S3 Object Storage, in which we will focus on integration testing with LocalStack and Testcontainers. And although we will focus on Object Storage, the approach we will use can be easily replicated … <\/p>\n

Read More<\/a><\/div>\n","protected":false},"author":0,"featured_media":137,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-136","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"_links":{"self":[{"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=\/wp\/v2\/posts\/136"}],"collection":[{"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=136"}],"version-history":[{"count":0,"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=\/wp\/v2\/posts\/136\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=\/wp\/v2\/media\/137"}],"wp:attachment":[{"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/imcodinggenius.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}