Think and Develop Serverless Applications as ‘Set-Pieces’
Serverless story-telling
In my previous articles on serverless, I used themes such as cooking, growing, fishing, and hiring. In this, I talk about football, movies, and music. “What do they have to do with serverless?”, you might ask. Well, there isn’t anything directly common between these themes and serverless. However, with some imagination and story-telling, we can easily build the connections ourselves.
I often make analogies to convey the serverless message. Serverless adoption happens at different levels, not just at the core technical level. Therefore, why not add a bit of fun, mix it up with some variety, throw in a few fresh ideas, provoke your thoughts, and above all, relieve you from the serverless overdose!
This is a two-part write-up on how to approach your serverless development. In this first part, I talk about the set-piece concept and discuss how it might help to tune your serverless mindset. In the next part, I will walk you through a real use case and explain how to apply the set-piece thinking in serverless development.
Ok, let’s start…
Mountain to molehills
It was the time when I got to know and fell in love with serverless. Around the same time, the daunting task of migrating a monolith eCommerce application to microservices also began.
Monolith to microservices migration is probably one of the most popular conference talks of recent years. Reading anything on that gives you more thrills than a crime novel. But the moment you sit down and start to think about moving your monolith mountain into molehills of microservices, that’s when the reality strikes and the panic sets in. If you then decide to also carve those molehills out of serverless, the picture of a bull in a china shop is likely to flash through your mind!
The ‘goal’ that changed my thinking
One day, with my mind filled with serverless and the migration task ahead, I sat down to watch a football match (English football, aka soccer). Eyes were on the TV, but my mind was in the clouds.
I got distracted when the host team scored a goal. The goal was scored from a free-kick. Commentators were bragging about the goal and how good the team was in set-piece play.
Though I knew the term set-piece before, I never thought through it beyond a goal-scoring context. Here is the football definition of the set-piece.
The term ‘set-piece’ refers to a situation when the ball is returned to open play following a stoppage. Many goals result from set-piece play. Thus, set-piece play is an important skill, and players spend much time practicing. Set pieces are one area where tactics and routines can be worked out in training in advance of matches.
An idea around the set-piece play sparked in my mind, but I needed more use cases than just football to develop my thinking. Further exploration revealed that the set-piece terminology is also common in film making, theater, and music.
Set-piece in film making
Here is an abstract set-piece definition from the film industry.
‘Set-piece’ is a scene or sequence of scenes whose execution requires complex logistical planning. Often, screenplays are written around a list of such set pieces. Set pieces are planned using storyboards and rehearsals. … different groups of people will work on different set pieces individually.
Set-piece in music composing
Many composers follow the set-piece concept while composing music. The process starts off with individual music parts, where the notes are written first, rehearsed, recorded, and then edited together as one.
I am sure there are so many experiences in life that resemble the above, the so-called set-piece experience!
The more I studied, the more refined my thoughts became. This helped me to identify 5 distinctive characteristics of a set-piece from the above contexts.
Set-piece characteristics
1. Focus is on the part
2. Planning and practice make things perfect
3. Rehearsing at the practice ground takes it closer to the main stage
4. Different sets of people work on different set-pieces
5. All the pieces together make the whole
1. Focus is on the part
Even though the vision is to win the match or complete the entire movie, they focus on a part or a scene to get that piece done to perfection. This is how I developed the concept of the “Visualize big but focus small” mantra that I often talk about.
2. Planning and practice make things perfect
Set-piece play on a football pitch requires clear planning. This includes the players, their position, the angle, the target, and the execution. When it comes to movie making, each set-piece scene requires careful and often complex planning. What this teaches us is that, even though we focus on a subset, the preparation of that piece plays an important role towards its perfection.
3. Rehearsing at the practice ground takes it closer to the main stage
It is common in many sports that the teams go through their practice routines up to the main event. This helps them adjust their tactics, tune their movements, and effortlessly carry those into the big match.
This is also the case in film making. Artists spend several days at the rehearsal studio before the final shoot.
4. Different sets of people work on different set-pieces
In football, certain play styles suit certain players. In films, certain scenes are ideal for certain artists. Identifying the needs of each part and associating the right personnel is important for its success.
5. All the pieces together make the whole
Though I’ve been focusing on the parts, all the pieces must come together for a match or a movie. The glue that makes this happen differs. In football, it is the play system — the faults, the stoppage, and the rules. For a movie, it is the editing, the background score, and the voice recording that stitches the pieces together.
Let me take a brief pause here before translating the above set-piece characteristics into serverless realization.
Why am I focusing on such simple things for so long? How do they fit within the frontiers of serverless?
Well, inspirations are all around us. Even the simplest of things in life can inspire us. When inspiration and the imagination magically come together, the result can be awesomely pleasing!
Pleasing is one thing; becoming purposeful is a completely different matter. Part of my ambition here is to make it purposeful.
I am here to convey my thoughts and the practices from my experience. You don’t need to be a football fan, movie-goer, or a music lover. I will keep it simple with no software jargon, principles, or theorems.
Here are the few observations that pushed me towards this analogy to connect it with serverless.
Different paths. One destination
The journey to serverless is different for each one of us. For some, it could be an easy transition from past experience. For others, it requires further learning. Not everyone carries the same technical background or the skills to become instantly successful. Hence, simple inspirations may help to find some purpose in our thinking as we get closer to serverless.
One mind. Two thoughts
You must have heard many times that serverless requires a different mindset. It becomes quite evident when you look at the technologies associated with serverless.
Along with the initial mindset, serverless requires a change of thinking on how we develop applications. This is different from the traditional developmental approaches you may be familiar with. I often refer to this as the mind-shift.
Think big. Practice small
This has been said many times in many ways. Engineers who do not have the exposure to microservices or the incremental developmental practices may find it difficult to grasp certain aspects of serverless development. Hence, it is important to start them on their journey with small projects to help grow in confidence and remind them that any complex problem can be tackled bit by bit.
Scary names. Simple technology
I once had a conversation with an engineer. I suggested writing a lambda function to solve a particular problem. Even though they’d been programming for a while, they panicked. “Ooh, I haven’t done anything of that sort, so I’m not sure!”, was their reply. My response was to think of it just like a function!
SQS is just a queue. SNS is just a topic. EventBridge is just an event bus. S3 is just a data store. I can go on and on. These are the strange names people hear when they start with serverless. The moment you describe what these services are using common terminology, the conversation becomes much easier!
Early success. Endless possibilities!
When I first moved to Java after developing many years in C and C++, I felt the change. The entire ecosystem around Java just expanded and brought everyone more opportunities. The myth of distributed computing was busted and brought it closer to our doorsteps.
Similar to the above but in a different way, I see the entire serverless spectrum bringing us opportunities at cloud-scale. The initial success you experience with serverless is crucial. It acts as the springboard for further victories. Hence, my advice to keep things simple in order to make a winning start.
It is easier to climb a molehill than to conquer a mountain!
Transporting the set-piece thinking into serverless
1. Focus is on the part
A monolith application is often viewed as a scary monster. Though it may look impossible to break a monolith into parts, there are always ways to view it as a combination of logical parts.
If you are familiar with the terminologies such as domains, components, modules, subsystems, and packages, then these can help with your thinking. Here are few initial ideas to assist you in identifying the parts.
- Look for logical grouping of functionality. For example, in the following diagram, the User admin is for validating the user identity, checking the user permissions, and establishing a user session.
- Identify features that are not related. A background task such as Reports generation may not have anything in common with, say, the User admin.
- Identify dependent functionalities that do not belong in the same group. Both Fraud check and Payment authorization are required to complete a payment request, but they don’t belong in the same group. Payment authorization uses the service offered by the Fraud check.
- Separate the features that act on-demand. For example, a successful Payment authorization and other internal state changes may trigger the Settlement to initiate a fund capturing process.
The above example depicts some of the parts of a Payment System. If we view it from a level above, the Payment System itself could be a part of a bigger enterprise application.
Once you have identified some logical groupings, treat each of them as a monolith and try to break them into further pieces. Don’t get constrained by what a microservices text book says or the definition of a domain is. At this stage, what you are interested in is finding the pieces. Big or small, simply does not matter.
I often talk about the cosmos analogy. The cosmos as a whole is difficult to comprehend. But, if you focus on a distant orange dot, you will find a galaxy behind it. You explore the galaxy to see the stars. You observe a star to discover its planets and their moons. You keep going until you find you and me somewhere!
2. Planning and practice make things perfect
Once you have identified the parts, select one logical piece to make a start. Be humble, and select the simplest of the pieces, to begin with.
The planning process can be challenging, especially if this is your first serverless development. There can be many technical and non-technical elements to look into and plan for.
When you start fresh, don’t get distracted with most of the things in the above diagram. Identify the immediate needs, and the rest will follow suit as the solution evolves.
Develop an architectural vision for the piece that you are starting with. Think of the serverless services that are fit for purpose to the best of your knowledge. Come up with a design that looks to provide a minimum viable product (MVP). You can always improve as you gain more experience.
If necessary, practice the design with a minimalist implementation, also known as, the Proof of Concept (PoC). Make sure the time you invest in doing the PoC is not wasted. Never start with a throw-away PoC mindset. Think of the PoC as your first iteration towards your design. With cloud and serverless, there is no reason why you should waste your effort.
As you go through the design and the implementation of one piece, you slowly start to think of other related parts. Your thoughts expand, ideas develop and the mountainous task on hand sees a glimmer of hope!
3. Rehearsing at the practice ground takes it closer to the main stage
The ease with which a solution can be tested in a production-like environment has never been this convenient. So, make full use of the power of serverless.
Create separate cloud accounts for your test and production. Whether you call it a test account or a production account, for the cloud provider, these are simply a logical separation of their same platform. It’s us who make that differentiation by configuring different access controls and account privileges to mark the separation.
Two of the most common patterns are-
1. One developer account to be shared by all the engineers. One QA account. One Acceptance or Pre-Production account and the Production account.
2. One developer account per engineer. One QA account. One Acceptance or Pre-Production account and the Production account.
Having an individual developer account has the benefit of hassle-free developer experience. At the same time, this requires more administrative overhead to set up the accounts, access permissions, and keep track of the monthly bills.
Having a shared developer account can become a deployment nightmare with one engineer overwriting another’s deployment of the same service. But, there are ways to overcome this.
If a large team shares one account, then it may encounter the account level service and resource limits set by the cloud provider. Though most are soft-limits, you may encounter hard-limit situations as well.
Where possible, keep the service configurations the same across all the environments. Configurations such as lambda timeout and memory, SQS polling interval, Firehose buffer size, and other services’ properties can be the same in production and non-production environments. However, there are situations where this is not always going to be the case. For example, due to legal obligations, the data retention policy can differ between production and non-production environments.
4. Different sets of people work on different set-pieces
A single purpose function does one thing and one thing well. When you work on small independent services, you channel your energy and focus on that. The same concept, when practiced across the team, increases the overall motivation and productivity. Among other things, the two important factors that help to achieve this are,
- The architectural design
- The development environment
As mentioned in the section below, designing each piece as an independent service plays an important part. The event-driven and decoupled nature of the services allows the team to work on different pieces in parallel. Together, they contribute towards a shared goal.
To aid service isolation and prevent dependencies between services forming, the development tools and the environment must be supportive. In the code repository, think of individual packages and dedicated commit, test and deployment pipelines for each service.
5. All the pieces together make the whole
No matter how we break down an application, all its parts must come together and function as one.
In a movie, the scenes are edited together with dialogues, background score, and sound effects. In serverless, we have events, messages, and APIs, as the medium to knit all the set-pieces together. They are the nerves behind the serverless applications that bring life to the set-piece development.
Above is an intentionally blurred architecture of an email notification application. A traditional text book style microservices application would have implemented this as one (macro)service to run in a container. In the next article, we will see how this monolith microservice thinking can be broken down and developed as several serverless set-piece services.
The beginning of a new way of thinking!
Visualize your serverless applications as a collection of set-pieces. Serverless gives us functional and granular level service features. By utilizing these unique characteristics, we can easily shift our thinking from a coarse-grained microservice to fine-grained serverless services.
When you start your serverless development, free your mind to go beyond the tradition to bring fresh ideas. New technology requires new thinking. If serverless gives you wings, then make full use of them and accelerate your cloud journey to enter new territories.
Go Build Serverless!
