Understand Python’s Iterators and Iterables and Create Custom Iterators

Iteration is one of the most important concepts in Python. Two terms that pertain to iteration are iterators and iterables. Learn what they are in this article.

One essential principle of software development is that Don’t Repeat Yourself, which is elaborated in The Pragmatic Programmer Book as below:

“Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.” — Andy Hunt and Dave Thomas

One specific application of this principle in modern programming is the use of iteration that involves going over a list of items, on which a defined operation is performed. One of the most basic forms of iteration is a for a loop. Although many other languages such as Swift and JavaScript use three-expression for-loop, Python uses a more concise syntax.

# Swift, JavaScript, etc
for (i = 0; i < 10; i++) {
    expression
}

# Python
for i in (0, 1, 2, 3):
    expression

Iterables & Iterators

Iterables

Iterables are objects that can be iterated in iterations. In Python, many basic data structures like strings and lists are iterables, such that we can use them in a for a loop as shown below.

# str as an iterable
>>> for i in 'abc':
...     print(i)
... 
a
b
c

# list as an iterable
>>> for i in [1, 2, 3]:
...     print(str(i*2))
... 
2
4
6

Iterators

Iterators are objects that produce a data value at a time using the __next__() method. Don’t worry if these concepts are confusing you for now. Follow along with this article, and we’ll sort them out soon. To understand what iterators are, and let’s see a simplified example below.

>>> number_iterator = iter([1, 2, 3])
>>> type(number_iterator)
<class 'list_iterator'>
>>> next(number_iterator)
1
>>> next(number_iterator)
2
>>> next(number_iterator)
3
>>> next(number_iterator)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Specifically, in the above code, we first created an iterator called number_iterator bypassing a list of numbers to the iter() method. When we checked its type, we found out that it was indeed an iterator, or more precisely a list_iterator. Every time we called the next() method on the number_iterator, the iterator produced an integer for us until the StopIteration exception is raised.

Relationships between iterators and iterables

One thing to note is that iterators are iterables, and thus using the iterator in a for loop will still work.

>>> number_iterator = iter([1, 2, 3])
>>> for i in number_iterator:
...     print(i)
... 
1
2
3

However, the opposite is not always true. For example, iterables like strings and lists are not iterators such that they don’t have the next() method.

>>> 'abc'.__next__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute '__next__'

>>> [1, 2, 3].__next__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute '__next__'

Given these differences between iterables and iterators, they’re used differently in for loops. As shown in the example below, we use the same iterable (i.e., a list) in two for loops without any errors. By contrast, the iterator can be used just once, as completing the first for loop has already made the iterator iterate all elements such that no more elements to be iterated.

# use iterables in for-loops for multiple times
>>> number_iterable = [1, 2, 3]
>>> for i in number_iterable:
...     print(i)
... 
1
2
3
>>> for i in number_iterable:
...     print(i)
... 
1
2
3

# use iterators in for-loops for multiple times
>>> number_iterator = iter([1, 2, 3])
>>> for i in number_iterator:
...     print(i)
... 
1
2
3
>>> for i in number_iterator:
...     print(i)
...
# nothing is printed

Create Custom Iterators

As briefly mentioned above, we can create an iterator by using the iter() method, in which we pass in an object. In the example above, we use a list to generate an iterable by calling iter([1, 2, 3]), which is equivalent to [1, 2, 3].__iter__(). This usage is similar to the built-in len() method, which is also known as __len__() method.

With this in mind, it may be wondered that what makes an object qualify for an iterator. Let’s use a common introspection function — hasattr() (see my previous article on Python’s introspection) to run a quick experiment below.

>>> hasattr(3, '__iter__')
False
>>> iter(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable

>>> hasattr([1, 2], '__iter__')
True
>>> iter([1, 2])
<list_iterator object at 0x10efcb110>

From the code above, we can see that if an object has the __iter__() method, it’s iterable, and thus it can be used to create an iterator.

Another important building requirement for an iterator — as mentioned above — is the implementation of the next() method. Thus, we can create a custom iterator class by implementing both __iter__() and __next__() methods. An example is given below.

We can certainly use this class in a for loop or a list comprehension (see my previous article on list comprehensions if interested).

>>> for i in PerfectSquare(4):
...     print(i)
... 
1
4
9
>>> [x for x in PerfectSquare(10)]
[1, 4, 9, 16, 25, 36, 49, 64, 81]

Before You Go

Extra “Bonus” Notes

For the built-in iter() method, the official documentation shows its complete form: iter(object[, sentinel]). The argument sentinel is optional, the presence of which will impact the behaviour of the generated iterator. A couple of extra notes are listed below.

• If the sentinel is not present, the object has to be iterable by implementing the iteration protocol (i.e., __iter__() method), which has been discussed intensively in this article. Alternatively, the object can be a sequence type by implementing the __getitem__() method, which is less commonly used.
• If the sentinel is present, the object must be callable. The iterator created will call object with no arguments for each call to its next() method. If the value returned is equal to the sentinel, StopIteration will be raised. An example is given below on this.

>>> import random
>>> def roll_dice():
...     return random.choice([1, 2, 3, 4, 5, 6])
... 
>>> def records_before_six():
...     return [x for x in iter(roll_dice, 6)]
... 
>>> records_before_six()
[1, 4, 5, 4, 4, 1, 4, 1, 3, 1, 1, 2]
>>> records_before_six()
[1]

Conclusions

This article reviewed the basic concepts on iterators and iterables and showed you an example of how we can implement a custom iterator class.

Apparently, Python provides a convenient way for us to create custom iterators. You can take advantage of this feature in your own coding.