Building a File Transfer Service with Go and gRPC (Part 2)

Wrapping up our data transfer service

In Part 1 of building our gRPC Service, we uncovered the design and initial implementation of a file transfer service, written in Golang (Go) using gRPC. This design is mostly centered around a 3-part API where client code can start an upload, upload the raw bytes of the data (file), and then close the upload. To summarize:

1. We break the API down into 3 parts to allow Server & Client a flexible arrangement: Server can deny an upload if validation checks don’t pass, and Client can choose to wait to upload or close until a later point in time.
2. We left implementation loose on the examples, but client and server implementations can be done in a way to make the APIs fully idempotent. This allows the client to fail and repeat as often as it wishes with no harmful side effects.
3. By making file transfers into discrete steps, debugging and servicing the code in production should become easier. Each of these three APIs can easily be instrumented with Prometheus or another metrics framework.

At the end of this Part 2 article, you should have more than enough to get a basic service headed in the right direction, using gRPC and Golang, with AWS or another cloud provider as your backend.

Server Implementation

FinishUpload API

In this section, we’ll implement the FinishUpload API on the server side. This API handles finalizing the upload process and responding to the client.

Create a file named finish_upload.go in the server directory and add the following code:

package main

import (
 "context"
 "log"
 "net"

 pb "path_to_your_protos/file_transfer"
 "google.golang.org/grpc"
)

// Reminder!  You may want to introduce an interface here and allow
// server to implement that behavior, for easier testing & dependency
// injection.
type server struct {
 pb.UnimplementedFileTransferServiceServer
}

func (s *server) FinishUpload(ctx context.Context, req *pb.FinishUploadRequest) (*pb.FinishUploadResponse, error) {
 // Validate the upload completion status (example)
 isSuccessful := validateUploadStatus(req.UploadId)

 if isSuccessful {
  return &pb.FinishUploadResponse{
   Status: pb.UploadStatus_SUCCESS,
  }, nil
 }

 return &pb.FinishUploadResponse{
  Status: pb.UploadStatus_FAILURE,
 }, nil
}

func validateUploadStatus(uploadID string) bool {
 // Implement logic to validate upload status
 // Example: Query the database for the upload status
 return true
}

In this implementation, the FinishUpload function receives the FinishUploadRequest from the client and checks the upload status. It returns a FinishUploadResponse with either a success or failure status, based on your implementation logic.

The validateUploadStatus function is a placeholder for validating the upload status. You can query the database or perform any necessary checks to determine if the upload was successful.

With this implementation, the server now has the capability to handle the entire file upload process. The StartUpload, Upload, and FinishUpload APIs are fully implemented.

‼️ Protip: If you decide to add on additional behavior on the server beyond the defined APIs from gRPC, introduce a dedicated interface for that! Your server can implement multiple interfaces, and it makes testing much easier. It’s already implementing the gRPC API interface, which allows you to mock and test those aspects elsewhere, but you can easily add in additional behavior into server (maybe some code that validates clients, or validates files) and define that as an interface type.

Client Implementation

gRPC Setup

Before we dive into the individual APIs, let’s set up the gRPC client. Create a file named main.go in the client directory and add the following code:

package main

import (
 "context"
 "fmt"
 "log"

 pb "path_to_your_protos/file_transfer"
 "google.golang.org/grpc"
)

func main() {
 // Set up a connection to the server
 conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
 if err != nil {
  log.Fatalf("Failed to connect: %v", err)
 }
 defer conn.Close()

 // Initialize the gRPC client
 client := pb.NewFileTransferServiceClient(conn)

 // Call the client APIs (we'll implement them in the following sections)
}

In this code, we establish a connection to the server using grpc.Dial and create a client using the generated NewFileTransferServiceClient function.

Now let’s proceed with the implementation of the individual client APIs.

StartUpload Client

Create a file named start_upload_client.go in the client directory and add the following code:

package main

import (
 "context"
 "fmt"
 "log"

 pb "path_to_your_protos/file_transfer"
)

func main() {
 / Set up a connection to the server
 conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure())
 if err != nil {
  log.Fatalf("Failed to connect: %v", err)
 }
 defer conn.Close()

 // Initialize the gRPC client
 client := pb.NewFileTransferServiceClient(conn)


 // Call the StartUpload API
 startUploadResponse, err := client.StartUpload(context.Background(), &pb.StartUploadRequest{
  FileName: "example.txt",  // Tip: this is often called the "stem", and we could omit 'txt.
  FileType: "txt",
  FileSize: 1024,
 })
 if err != nil {
  log.Fatalf("Error calling StartUpload: %v", err)
 }

 fmt.Printf("Upload ID: %s\n", startUploadResponse.UploadId)
}

In this code, we call the StartUpload API with a sample file metadata. The response includes the unique upload ID generated by the server.

Now, let’s implement the Upload and FinishUpload clients.

Upload Client

Create a file named upload_client.go in the client directory and add the following code:

package main

import (
 "context"
 "fmt"
 "io"
 "log"
 "os"

 pb "path_to_your_protos/file_transfer"
)

func main() {
 // Rest of the code remains the same...

 // Open the file for reading
 file, err := os.Open("path_to_your_file.txt")
 if err != nil {
  log.Fatalf("Error opening file: %v", err)
 }
 defer file.Close()

 // Create a stream to send the metadata and file data
 stream, err := client.Upload(context.Background())
 if err != nil {
  log.Fatalf("Error creating stream: %v", err)
 }

 // Send metadata to the server
 err = stream.Send(&pb.UploadRequest{
  UploadId: startUploadResponse.UploadId,
  FileName: "example.txt",
  FileType: "txt",
  FileSize: 1024,
 })
 if err != nil {
  log.Fatalf("Error sending metadata: %v", err)
 }

 // Send file data in chunks
 buf := make([]byte, 1024)
 for {
  n, err := file.Read(buf)
  if err == io.EOF {
   break
  }
  if err != nil {
   log.Fatalf("Error reading file: %v", err)
  }
  if err := stream.Send(&pb.UploadRequest{
   Chunk: buf[:n],
  }); err != nil {
   log.Fatalf("Error sending chunk: %v", err)
  }
 }

 // Close the stream and receive the response
 uploadResponse, err := stream.CloseAndRecv()
 if err != nil {
  log.Fatalf("Error receiving response: %v", err)
 }

 fmt.Println(uploadResponse.Message)
}

In this code, we open the file to be uploaded, create a stream to send the metadata and file data, and then send the data in chunks to the server. After sending all the data, we receive the response from the server.

FinishUpload Client

Create a file named finish_upload_client.go in the client directory and add the following code:

package main

import (
 "context"
 "fmt"
 "log"

 pb "path_to_your_protos/file_transfer"
)

func main() {
 // Rest of the code remains the same...

 // Call the FinishUpload API
 finishUploadResponse, err := client.FinishUpload(context.Background(), &pb.FinishUploadRequest{
  UploadId: "sample_unique_id", // Replace with actual upload ID
 })
 if err != nil {
  log.Fatalf("Error calling FinishUpload: %v", err)
 }

 if finishUploadResponse.Status == pb.UploadStatus_SUCCESS {
  fmt.Println("Upload completed successfully!")
 } else {
  fmt.Println("Upload failed.")
 }
}

In this code, we call the FinishUpload API with the upload ID obtained from the StartUpload response. We then process the response to determine whether the upload was successful or not.

Conclusion

Congratulations! You’ve successfully implemented a file transfer service using Go and gRPC. This tutorial covered setting up the project structure, creating gRPC API definitions, implementing the server-side APIs, and building the client-side applications. You’ve learned how to interact with AWS RDS for metadata storage and AWS S3 for file storage.

Feel free to customize and extend this implementation for your own use cases. Remember that this is a skeleton and general approach to writing a data transfer service, you’ll need an enormous amount of more work to productionize this (logging, error handling, instrumentation, completeness of the example API implementations, leveraging gRPC status package, and much more). This will get you going in the right direction, with an overall architecture design that is quite capable.

• Enjoyed the Article? Leave me a 👏
• Like the format and content? Join me here and give me a follow 🙌
• Thank You! 🙏

Thank you for reading until the end. Please consider following the writer and this publication. Visit Stackademic to find out more about how we are democratizing free programming education around the world.