Should We Just Stop Using For Loops In Julia?

Julia’s syntax is pretty dynamic, can we use this to avoid traditional iterative ‘for’ loops?

Introduction

One of the most obvious, yet difficult problems to solve in computing is how to translate human logic into that of a computational central processing units. Humans do not read bits and bytes very often in real life unless they are robo-cop or have a ridiculous amount of spare time. Instead, thanks to the legends in the Computer-Science field before us, we use modern interpreters and compilers that bring our data down to the processor without requiring any work with bits, bytes, registries, or even a faint glance at memory. Modern high-level languages have done an absolutely incredible job of driving the high-level, declarative landscape of today’s most popular programming languages, and of this group there is one rather new language that stands out, Julia.

For the uninitiated, Julia is a high-performance programming language released out of MIT around a decade ago. While the language is still young, it has already made some rather impressive feats — such as micro-benchmarks that often trail or C and Rust performance in many applications. Julia is also a member of the “ petaflop” club, which you can read more about in this overview on Reddit:

In a lot of ways, Julia has been effective of separating the software from the hardware. This has been done by remaining an easy-to-use, declarative, dynamically-typed language, but matching speeds of more traditional and statically-typed languages like C. The speed is only a very small portion of the reason I am such a big fan of the language, but speed is an important topic.

With any language, there are shortcomings and with every quantity of speed there becomes a new upper limit to what the software is capable. Thus, even in a language like Julia, where rarely you need to worry about performance unless you are actually software engineering or trying to optimize something.

For context on that last sentence, I am from the Scientific domain; where environments are often ‘ interactive.’

In the past, I have done a lot of comparisons between various aspects of this language and others, and one thing that I really enjoy is taking a simple idea that could potentially benefit or hinder Julia performance and investigating and sharing how exactly the performance is going to differ. A great example of this format of post is my view of the effects of Julia’s annotations on performance:

Looping and Comprehensions

Today the performance focus is on looping and comprehensions. The question I wanted to pose today is if it could be possible to use comprehensions in place of most conventional for loops in Julia. Furthermore, if this is possible to do, then is it something we should do? Additionally, what kind of effect will this have on the performance?

First, we need to find out when we can and cannot replace a for loop with a comprehension. In most other languages that include this feature, a comprehension is not as capable as a for loop. As a result, comprehensions are generally reserved to do a lot less as far as actually calculating things. Comprehensions also always provide a return, so that is certainly a substantial difference between the two as well. If you wanted to get a “ return” from a for loop, you would really create a new vector outside of the loop and then add values to it on each iteration. Needless to say, that is far less efficient when compared to a comprehension.

As discussed prior, comprehensions are typically used for simple statements. Consider the following example:

x = Vector{Int64}()
for y in 1:10

    push!(x, y)
end

This is certainly one that I would resign to a much more simple — perhaps more readable version using a comprehension.

x = [y for y in 1:10]

Needless to say, this can save a lot of time. Turning up the range on these loops, it also becomes quite obvious that the performance is also substantially better when using comprehensions:

x = Vector{Int64}()
@benchmark for y in 1:100000
    push!(x, y)
end
@benchmark x = [y for y in 1:100000]

Considering all of the above, it certainly seems nice to be able to use comprehensions everywhere. The limitations of the typical programming language’s syntax in a comprehension is usually the largest downfall of the comprehension, but in Julia; things are a bit different. There is a lot more that you can do inside of a comprehension like this, additionally, the code can be incredibly readable. In most cases, this sort of code in something like Python would become an absolute mess.

x = [if y < 5 10 end for y in 1:10]

I am not going to go incredibly in-depth on this concept here, so if you would like to read more on the topic, I do have an entire article on them that might be helpful:

Absurdity

notebook

Okay, so we have witnessed some basic examples of comprehensions being flexible. Now it is time to take things to a new level by creating a crazy loop, then comparing the same loop in a comprehension. The loop I am going to create for this demonstration is going to be pretty simple, the loop will simply randomly jostle an array around, switching elements and providing a return of each number’s switch. The number of switches will be changed using an argument. To start the function, we initialize our return and then begin our loop.

function jostle(x::Vector{Int64}, count::Integer = 2000000)
    jostles = Vector{Pair{Int64, Vector{Int64}}}()
    for shift in 1:count

Inside of the loop, we simply grab two random indexes and then flip them before pushing this data into jostles.

for shift in 1:count
        switch1, switch2 = rand(1:length(x)), rand(1:length(x))
        if switch1 == switch2
            switch2 = rand(1:length(x))
        end
        n1, n2 = x[switch1], x[switch2]
        x[switch1] = n2
        x[switch2] = n1
        push!(jostles, shift => [n1, n2])
    end

Finally, we return jostles , completing our method:

function jostle(x::Vector{Int64}, count::Integer = 2000000)
    jostles = Vector{Pair{Int64, Vector{Int64}}}()
    for shift in 1:count
        switch1, switch2 = rand(1:length(x)), rand(1:length(x))
        if switch1 == switch2
            switch2 = rand(1:length(x))
        end
        n1, n2 = x[switch1], x[switch2]
        x[switch1] = n2
        x[switch2] = n1
        push!(jostles, shift => [n1, n2])
    end
    jostles
end

Writing this into a comprehension is incredibly straightforward.

function compjostle(x::Vector{Int64}, count::Integer = 2000000)
    [begin
        switch1, switch2 = rand(1:length(x)), rand(1:length(x))
        if switch1 == switch2
            switch2 = rand(1:length(x))
        end
        n1, n2 = x[switch1], x[switch2]
        x[switch1] = n2
        x[switch2] = n1
        shift => [n1, n2]
            end for shift in 1:count]
end

Not only is that incredibly short, it is also really readable and could be interpreted as subjectively better in terms of syntax by some. Here is the comparison!

x = [5, 1, 5, 6, 7, 1, 4, 5]
@benchmark y = jostle(x)

@benchmark y = compjostle(x)

To Conclude

Comprehensions provide a very big and very bold boost to performance in a multitude of entirely different programming languages. Needless to say, this is probably a great one to use in your toolbox. That being said, this could be an easy one to accidentally overuse. Then the question becomes what constitutes overuse? The conventional explanation for why one should not use comprehensions all of the time comes in the form of keeping code looking pretty. I think this can be a great choice, because it is better to have something that performs slightly worse that people can actually work on.

However, with Julia the comprehension syntax is a lot more dynamic, capable, and of course; readable. Thanks to the begin/end block, we can wrap entire expressions into a comprehension! This is something that is pretty awesome and, at least in this manor, unique to Julia. There is a great boost to performance, but I am hesitant to say that comprehensions should always be used because of the inherit difference between a comprehension and a loop. A comprehension applies a function across an iterable, whereas iteration literally allocates each value into memory and approaches things more tangibly. This means that awesome things like break or continue will not be available in your comprehension, and generally, just writing things in a comprehension could be more difficult. I have also found that while CPU process times decrease when using comprehensions, memory usage typically increases, albeit slightly.

Thank you so much for reading this article! I hope this cool little overview on comprehensions was fun to read. Do you think all Julians should just replace for loops with comprehensions at all times, given that Julia does allow it and it provides speed and syntactical benefits?