Dog Fight — Python VS Golang VS Rust for JSON Processing

Real-world use case comparison

I have been started my writing journey for almost 3 years now. Your support is the most important motivation to keep me moving forward and write more interesting learning and sharing.

Disclaimer: The articles I wrote in the medium only relative to my learning and opinions. Will NOT represent or relative any of my previous or current position and company.

2 years ago, I started my weekend code and learned on medium. This journal is keeping moving forward. Enjoy every moment on it.

I don't want to join into comparison tooling, solution, programming language, etc. Reason being:

There is not prefect solution/tooling/programming language, the only prefect one is based on the context and the timestamp.

Based on the belief above, I always encourage my engineering team to find the best fit solution based on the use-case, scalability, and cost-saving in mind.

I have a friend recently working on some stock research program, so he wants to get some historical stock data for the symbol identified.

We found an available dataset in Kaggle:

Let’s get a high-level summary for these requirements:

1. Go over the dataset to find all the files relative to the symbol code provided.
2. Speed and performance are critical.
3. Easy to deploy to the cloud.

Analysis phase — dataset

In this available dataset, we have:

1. Data partitioned by the year quarter for almost ten years.
2. The data format is JSON.
3. The total data size is around 2.2 GB.
4. There are around 241,310 files in total.

Solution design should consider:

1. Massive I/O read and file open; we should consider the worker queue based on the VM CPU/Cores.
2. The application can run on the smallest VM for most cloud providers. (Small binary or run time)
3. Based on the SLA, the code can easy to scale up for fast execution run time.

Comparison factors:

1. Programming experiences.
2. Scalability.
3. Performance. (Included runtime CPU consumption)
4. Deployment.

All right, lets code

Business Logic

1. Walk a directory/Recursively.
2. Parallel executing based on the CPU Cores.
3. Put each file to the worker pool for allocating execution.
4. Filter the symbol key mache the search target.
5. Output the file_path and name.

Walk a directory / recursively

CPU & worker pool allocation implementation

For Python, we are using the multiprocessing packages

For Go, we are using the built-in concurrency green thread, worker implementation.

For Rust, we will leverage the rayon worker pool implementation, and meanwhile, we also follow the Golang channel communication best practices.

“Do not communicate by sharing memory; instead, share memory by communicating.”

We are using Flume for communication. Compare with other libs, the Flume is much faster than the Tokio channel and mspc channel.

Worker implementation

Filter the symbol key mache the search target.

For Python, it's easy to understand and implemented.

For Go, we are using the fastjson package for JSON parser; compare with the original encoding/JSON standard libs for Go, this package has a great benchmark so far; it seems our application is tracking the performance. So every ns counts.

For the Rust, there are a couple of taking away here; if we are not attending on the implementation, the performance will have some challenges:

1. Open file using the serde_json from_reader function is usually slower than reading a file completely into memory, which is from_str. And with the 2 G files, the difference is huge, from 20 mins -> 10 seconds.
2. For handling JSON in rust, we need some benchmark testing to ensure we have the best performance. Please kindly see some benchmark below.

Final output

This is running based on the MacBook Pro: 2.6 GHz 6-Core Intel Core i7

Containerization for further analysis.

As we also need to consider the deployment phase, so we need to deploy to the local docker to simulate the production run.

The Rust runtime CPU is the most consistent and low footprint compare with the Python and Golang. I also surprise the Go have a higher CPU usage rate than Python, which I may need to spend for pprof. (out of scope here)

Closing thoughts

Python (version 3.8.2) is most easy to write for this use-cases compare with Go and Rust; the multiprocessing libs take care of most of the concurrency implementation; it only takes 42 lines of code. I did not spend a lot of time to tune the performance, but it really does the job with the expected performance.

Golang (1.15.2), with its green thread, goroutine implementation, is easy to understand and can scale up (green thread VS thread) with more complicated uses cases, enjoyable to write. The performance is better than Python, also for the deployment part, the min-container size is only 2.46 MB, with the multi-stage build.

Rust (1.47.0) with its small CPU footprint, performance out of the box, and concurrent 100% safe. When you compare with Python, you may sometimes need to tuned and optimize the code to ensure outstanding performance.

And for this use-case, if the non-functional requirement (NFR) is not crazy about seconds/sub-seconds performance. Personally speaking, Python should be the better choice here; the main reason, it's for the maintainability and easy to write; however, if you are tracing the performance, than Go or Rust is something on the table for your need.

That's all for this long weekend. See you in the next article.

See my other articles below for more weekend code and learn, which you may be interested in.