Using Rust with Python — Python Gets Even Greater

Using Rust with Python — Python gets even greater.

Supplementing the advantages of Rust and Python. Using Python for prototyping and shifting performance bottlenecks to Rust.

Python and Rust are very different languages, but they actually complement each other quite well.

But before discussing how to combine Python and Rust, I’d like to introduce Rust itself. You may have heard of this language, but you might not be familiar with its background.

What is Rust? Rust, like C and C++, is a low-level language, meaning that programmers are dealing with something closer to how a computer “actually” works.

For example, integer types are defined in terms of size in bytes, corresponding to the types supported by the CPU. Although we’d like to say Rust corresponds to a single machine instruction a + b, it's not quite that simple!

The Rust compiler chain is quite complex.

**It’s useful as a rough approximation to view statements like these as “somewhat” correct for the first go:

Rust’s goal is zero-cost abstractions, meaning that many abstractions available at the language level are compiled away at runtime.

For example, unless explicitly requested, objects are allocated on the stack. So, creating native objects in Rust has no runtime cost (though they might need initialization).

Finally, Rust is a memory-safe language. There are other memory-safe languages and languages that support zero-cost abstractions. But usually, these are two different categories of languages.

Memory safety doesn’t mean that memory violations are impossible in Rust. It does mean that there are only two ways memory conflicts can occur:

Compiler error: Code explicitly declared as unsafe.

Rust’s standard library has a lot of code marked as unsafe, though less than many people might expect. This doesn't mean that the declaration is meaningless.

With the exception of cases where you have to write unsafe code yourself (very few), memory conflicts are usually caused by the infrastructure.

Why Rust? Why did people create Rust? Are there problems that no existing programming language can solve?

Rust was designed to be both efficient and memory-safe. In the modern interconnected world, safety is an increasingly important concern.

A typical use case for Rust is low-level protocol parsing. The data to be parsed often comes from untrusted sources and needs to be parsed efficiently.

If this sounds like what a web browser does, that’s no coincidence. Rust initially originated at the Mozilla Foundation with the goal of improving the Firefox web browser.

Today, it’s not just web browsers that need both security and speed. Even common microservices architectures must be able to parse untrusted data quickly while remaining secure.

Example: Counting Characters To understand “Wrapping Rust” examples, we need to solve a problem that meets the following criteria:

Should be easy to solve
Should allow writing high-performance loops for optimization.
Should have some practical relevance.

One example of this problem is determining if a character appears more than X times in a string. This problem is not easily solvable with efficient regular expressions.

Even using specialized numpy code might not be fast enough, as it typically doesn’t need to scan the entire string. You can imagine combining some Python libraries and tricks to solve this problem. However, implementing a direct algorithm in a low-level language like Rust would be much faster and more readable.

To make the problem a bit more interesting and showcase some interesting parts of Rust, we added some variations. The algorithm supports resetting the count at newlines, meaning whether a character appears more than X times in a line, or at spaces, meaning whether a character appears more than X times in a word.

Enum Support Rust supports enums. You can do a lot of interesting things with enums.

Currently, only a simple three-choice enum is used, with no other variations. This enum encodes which character resets the count.

#[derive(Copy)]
 enum Reset {
     NewlinesReset,
     SpacesReset,
     NoReset,
 }

Struct Support The next component in Rust is structures, similar to Python’s dataclasses to some extent. You can do more complex things with structures.

#[pyclass]
 struct Counter {
     what: char,
     min_number: u64,
     reset: Reset,
 }

Implementation Module In Rust, an implementation module called an “impl” is used to add methods to structures. The specifics of this are beyond the scope of this article. In this example, the method calls an external function.

This is mainly for code organization. More complex use cases would instruct the Rust compiler to inline functions for readability without incurring any runtime cost.

#[pymethods]
 impl Counter {
     #[new]
     fn new(what: char, min_number: u64, reset: Reset) -> Self {
         Counter {
             what: what,
             min_number: min_number,
             reset: reset,
         }
     }
 
     fn has_count(&self, data: &str) -> bool {
         has_count(self, data.chars())
     }
 }

Function By default, Rust variables are constants. As the current count must be modified, it is declared as a mutable variable.

This loop iterates over characters and calls the function. This is also for code organization, and it clearly indicates which functions can modify values.

fn has_count(cntr: &Counter, chars: std::str::Chars) -> bool {
     let mut current_count: u64 = 0;
     for c in chars {
         if got_count(cntr, c, &mut current_count) {
             return true;
         }
     }
     false
 }

Count Function The function resets the counter, increments the counter, and checks the counter. Rust uses pattern matching for this.

In Rust, a complete description of matching requires a semester-level course and is not suitable for unrelated lectures. This example matches one of the two options of a tuple.

fn got_count(cntr: &Counter, c: char, current_count: &mut u64) -> bool {
     maybe_reset(cntr, c, current_count);
     maybe_incr(cntr, c, current_count);
     *current_count >= cntr.min_number
 }

Reset Logic The reset logic demonstrates another useful feature in Rust: pattern matching.

In Rust, a full description of matching requires a semester-level course and is not suitable for unrelated lectures. This example matches one of the two options of a tuple.

fn maybe_reset(cntr: &Counter, c: char, current_count: &mut u64) -> () {
     match (c, cntr.reset) {
         ('\n', Reset::NewlinesReset) | (' ', Reset::SpacesReset) => {
             *current_count = 0;
         }
         _ => {}
     };
 }

Increment Logic The increment logic compares characters to the desired character and increments the counter if there’s a match.

fn maybe_incr(cntr: &Counter, c: char, current_count: &mut u64) -> () {
     if c == cntr.what {
         *current_count += 1;
     };
 }

For demonstration purposes, the code in this article has been optimized. It is not necessarily an example of best practices for Rust code or for designing a good API.

Wrapping Rust in Python To wrap Rust code in Python, you can use the PyO3 library, which is a Rust crate that enables you to easily wrap Rust code for Python, making it more accessible for integration.

Use PyO3 Basic Library First, you need to include the PyO3 basic library in your Rust project.

use pyo3::prelude::*;

Wrap Enumerations Enums also need to be wrapped. The derive clause is necessary for wrapping enums in PyO3, as it allows classes to be copied and cloned, making them easier to use in Python.

#[pyclass]
 #[derive(Clone)]
 #[derive(Copy)]
 enum Reset {
     /* ... */
 }

Wrap Structures Structures in Rust need to be wrapped as well. In Rust, these are called “structs,” and they generate the required interface bits.

#[pyclass]
 struct Counter {
     /* ... */
 }

Wrap Implementation Wrapping the implementation is more interesting. You need to add another macro called pymethods. This method is marked to let PyO3 know how to expose constructors for built-in objects.

#[pymethods]
 impl Counter {
     #[new]
     fn new(what: char, min_number: u64, reset: Reset) -> Self {
         Counter {
             what: what,
             min_number: min_number,
             reset: reset,
         }
     }
     /* ... */
 }

Define Module Finally, define a module initialization function. This function has a specific signature and must have the same name as the module and be decorated with pymodule. The ? indicates that this function can fail, for example, if the classes are not configured correctly.

#[pymodule]
 fn counter(_py: Python, m: &PyModule) -> PyResult<()> {
     m.add_class::<Counter>()?;
     m.add_class::<Reset>()?;
     Ok(())
 }

PyResult is converted to Python exceptions when imported. To check quickly, you can use the library maturin to build and install it in your current virtual environment.

$ maturin develop

Building and Importing Using Python’s library is the nice part. There’s no indication that this differs from writing Python code. One useful aspect is that if you’re optimizing an existing library with unit tests, you can use Python unit tests for the Rust library.

You can import it after building:

import counter

Building Constructor functions are precisely defined in Rust, so you can construct objects from Python. This isn’t always the case. Sometimes, objects are returned from more complex functions.

cntr = counter.Counter(
     'c',
     3,
     counter.Reset.NewlinesReset,
 )

Finally, the moment of truth:

Check if this string contains at least three ‘c’ characters:

>>> cntr.has_count("hello-c-c-c-goodbye")
 True

Adding newlines causes the rest to start over, and without three ‘c’ characters in the middle, it returns:

>>> cntr.has_count("hello-c-c-\nc-goodbye")
 False

Rust and Python can be easily combined with some “glue” code. They have complementary strengths and weaknesses.

Rust is excellent for high-performance, secure code.

Rust has a steep learning curve and can be a bit unwieldy for rapid prototyping solutions.

Python is easy to get started with and supports very concise iterative loops. Python does have a “speed limit,” and it can be challenging to get better performance from Python beyond a certain point.

Combining them is perfect.

Prototyping in Python and shifting performance bottlenecks to Rust using PyO3 makes your development and deployment process easier. Develop, build, and enjoy this combination!

Stackademic

Thank you for reading until the end. Before you go:

Please consider clapping and following the writer! 👏
Follow us on Twitter(X), LinkedIn, and YouTube.
Visit Stackademic.com to find out more about how we are democratizing free programming education around the world.

Summarize

Using Rust with Python — Python Gets Even Greater

Stackademic