Software Integration Series: Spring Boot and Amazon S3 Integration

A detailed guide on integrating Spring Boot and Amazon S3 with a sample application and Infrastructure-as-Code templates.

Welcome to my first article in the Software Integration Series. In this series, I will deep dive into various topics using Java tech stack, AWS cloud and many more. Without further ado let’s get started.

Amazon Simple Storage Service (Amazon S3) is a scalable, secure, and performant object storage service. It provides management features so that we can optimize, organize, and configure access to our data to meet specific business and compliance requirements.

This article walkthrough is a step-by-step guide about how to integrate Amazon S3 with a Spring Boot application.

Creating an Amazon S3 bucket using IaC tools

We will use Infrastructure-as-Code (IaC) template to create a S3 bucket. You can find two sample templates using both CloudFormation and Terraform tools in my GitHub repository. Use the tool of your choice and create a S3 bucket using the template.

The CloudFormation template creates IAM user, access key and outputs the values of access key and secret in the console. As a security best practice, it is not recommended to use IAM access keys when making API calls from applications. It is recommended to use IAM roles which use AWS Security Token Service (STS) to create temporary, limited-privilege credentials. It is also not recommended to output values of access key and secret in CloudFormation stacks especially when you are working in a team.

If you are using Terraform, create IAM access key via AWS console. The access key, secret and bucket name are passed as variables when applying the template.

Creating an Amazon S3 bucket and IAM access key using console

Login to your AWS account and search for S3 using search bar. Once you are in S3 console, click on “Create Bucket” button.

Provide a bucket name of your choice. The bucket name must be unique across AWS, not just your account.

Leave the rest of the settings with default values and click on “Create Bucket” button at the end of the screen.

Using the search bar again, search for IAM. Once you are in IAM console, click on “Add users” under Users section. Provide a user name and click on “Next”.

We can attach policies here, so let’s attach “AmazonS3FullAccess” policy. Note that we are attaching full access to S3 only for testing. As a best practice, attach only the relevant policies depending on use cases.

Always use the principle of least privilege to grant access to resources.

Click on the user that you just created and go to Security credentials tab.

Create an access key and make sure you grab the generated secret as you won’t be able to retrieve it later. This secret will be used later in our Spring Boot application to make API calls to S3 bucket.

Initializing a Spring Boot Application

Create a sample Spring Boot application and add the following AWS SDK for S3 dependency in pom.xml. This dependency provides client classes that are used for communicating with Amazon Simple Storage Service (S3).

<dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-java-sdk-s3</artifactId>
   <version>1.12.439</version>
</dependency>

If you are integrating multiple AWS services in your project, then you can include the SDK dependency with artifactId “aws-java-sdk”.

Provide access key, secret and bucket name in application.properties file. Be aware that it is not a best practice to check in credentials to source code repository. The credentials are usually provided via environment variables or other secure ways depending on the environment where applications are deployed.

Let’s configure S3 client with credentials and region.

@Configuration
public class S3Config {

    @Value("${aws.region}")
    private String awsRegion;

    @Value("${aws.s3.bucketName}")
    private String bucket;

    @Value("${aws.credentials.access}")
    private String awsAccessKeyId;

    @Value("${aws.credentials.secret}")
    private String awsKeySecret;

    public AWSCredentials credentials() {
        return new BasicAWSCredentials(
                awsAccessKeyId,
                awsKeySecret
        );
    }

    @Bean
    public AmazonS3 amazonS3() {
        return AmazonS3ClientBuilder
                .standard()
                .withCredentials(new AWSStaticCredentialsProvider(credentials()))
                .withRegion(awsRegion)
                .build();
    }

    @Bean(name = "bucket")
    public String getBucket() {
        return bucket;
    }
}

Let’s look at some of the API calls using S3 client.

Listing objects in a S3 bucket:

public List<String> listFiles(){
    ObjectListing objectListing = amazonS3Client.listObjects(bucket);
    return objectListing.getObjectSummaries().stream().map(S3ObjectSummary::getKey).collect(toList());
}

Upload an object to S3 bucket:

public void uploadFile(FileDTO fileDTO) {
    try {
        File file = createFile(fileDTO);
        amazonS3Client.putObject(bucket, fileDTO.getName(), file);
    } catch (Exception e){
        throw new RuntimeException(e.getMessage());
    }
}

Download an object from S3:

public void downloadObject(String objectName){
    S3Object s3object = amazonS3Client.getObject(bucket, objectName);
    try {
        FileUtils.copyInputStreamToFile(s3object.getObjectContent(), new File(objectName));
    } catch (IOException e) {
        throw new RuntimeException(e.getMessage());
    }
}

Delete an object from S3:

public void deleteFile(String fileName){
    amazonS3Client.deleteObject(bucket, fileName);
}

Finally, we will add some sample REST API methods to invoke these API calls to S3.

@RestController
@RequestMapping(value = "/files/")
@RequiredArgsConstructor
public class FileController {

    private final FileService fileService;

    @GetMapping(value = "/list")
    public List<String> listFiles(){
        return fileService.listFiles();
    }

    @PostMapping(value = "/upload")
    public void uploadFile(@RequestBody FileDTO fileDTO) {
        fileService.uploadFile(fileDTO);
    }

    @GetMapping(value = "/download/{name}")
    public File downloadFile(@PathVariable String name) {
        fileService.downloadObject(name);
        return new File("./" + name);
    }

    @DeleteMapping(value = "/{fileName}")
    public void deleteFile(@PathVariable String fileName) {
        fileService.deleteFile(fileName);
    }
}

We can test our integration by invoking these REST API endpoints via cURL or Postman tools.

Public vs. Private access to S3 buckets

Amazon S3 buckets are created as private by default. Only the object owner has permission to access these objects in a private bucket. Optionally, private buckets can be accessed by users with right IAM policies or permissions.

Private buckets can be granted with cross-account access using bucket policies, signed URLs, Access Control Lists (ACLs), etc. Public buckets are open to the public (anyone). It is generally a best practice to leave S3 buckets private and grant access using IAM roles and policies.

Conclusion

AWS SDK for S3 offers many more features such as managing buckets, ACLs, policies and so on. Check out the official documentation for more details.

The complete code for this article can be found on my GitHub repository.