Go Development Patterns
Golang Concurrency Patterns : Fan in, Fan out
One of the most prominent reasons of loving Golang personally, is how easy we can build a highly available concurrent and non blocking program.
In this series of posts, I will try to reminisce the patterns available in Golang. I will take every pattern and discuss in detail where they fit and how to use them effectively.
What is Fan in and Fan out. It’s a way to converge and diverge data into a single data stream from multiple streams or from one stream to multiple streams or pipelines.
A generator function
For this pattern to discuss, we need a data source first. Here is a generator of data which can be used as a data source.
The above function as it is evident returns a receive only channel.
The consumer of this function can only receive data from the channel.
Note, the function when defines the channel is as simple as making a channel. But by denoting <- in front of the response type, we are making it a receive only channel.
Fan in
Now that we have a data source, let’s create the fan in pattern. Let’s see this function:
Analysis
- In line 2, 3 we are making 2 data generators c1 and c2.
- In line 5 we are making the fanin channel which will be the converging channel that will take data from c1 and c2.
- In line 9, 10 based on the data availability from channel c1 and c2, the appropriate case will be selected and that same data will be pushed in to channel fanin.
Scenario where useful
Think of a scenario where we have to consolidate all the events
Fan out
For the fan out function we need a set of acceptors where our generator function will keep on sending messages or jobs to be processed.
For this scenario, let change the generator function to have some delay.
let’s discuss the pattern piece by piece.
Analysis
- In line 21 and 26 we are declaring a Processor and a Worker struct.
The Processor has a list of workers, which will be used as background processes to process data coming form the generator function( The data source)
2. Line 40 defines a function to create an instance of the Processor and start its processing in line 50.
3. We interact with the processor instance with the postJob method in line 73 which is happening in line 85. We are sending 11 messages to the processor instance to be processed.
4. In line 74, we get the message from the generator and channel it to the jobChannel channel in the processor instance.
5. Inside the startProcess method we have 2 selects. In line 62, we get the messages send by the generator in line 74 inside the postJob method, only when there is a worker(line 59).
6. We select the worker in line 61(which is always the top worker of the worker slice inside the processor instance).
In real scenario we should build a priority queue based worker pool so that the work is evenly distributed and the Processor is not blocked.
Also this setup is not backpressure aware. Line 62 can block if there is no jobs. In those cases make sure to add backpressure handling.
7. In line 62 we give the data to the selected worker in line 61 and also send the done channel of the processor instance.
8. The worker does the processing in a separate goroutine in line 32 and notifies the processor instance via the done channel.
9. The signal from the worker is caught in line 64 and the worker is appended to the worker list again.
If we run the code we will see,
This concludes our Fan In and Fan Out pattern. I will post another design pattern in coming posts.
Happy learning and sharing. 👐🏻
If you like this post, please share or comment on the post. Any sort of interaction is highly appreciated. If you would like to support me and get access to all great things in medium, please join and become a member. Here is my referral link:
My other posts on Golang:
