Python Iterators, Generators And Decorators Made Easy
A Quick Implementation Guide
Recently I received an email from one of my readers asking me to write about Python’s complex topics such as Iterators, Generators, and Decorators. In this post, I’m going to cover the basics, implementation, and how to use them in your code.
Projects Videos —
All the projects, data structures, SQL, algorithms, system design, Data Science and ML , Data Analytics, Data Engineering, , Implemented Data Science and ML projects, Implemented Data Engineering Projects, Implemented Deep Learning Projects, Implemented Machine Learning Ops Projects, Implemented Time Series Analysis and Forecasting Projects, Implemented Applied Machine Learning Projects, Implemented Tensorflow and Keras Projects, Implemented PyTorch Projects, Implemented Scikit Learn Projects, Implemented Big Data Projects, Implemented Cloud Machine Learning Projects, Implemented Neural Networks Projects, Implemented OpenCV Projects,Complete ML Research Papers Summarized, Implemented Data Analytics projects, Implemented Data Visualization Projects, Implemented Data Mining Projects, Implemented Natural Leaning Processing Projects, MLOps and Deep Learning, Applied Machine Learning with Projects Series, PyTorch with Projects Series, Tensorflow and Keras with Projects Series, Scikit Learn Series with Projects, Time Series Analysis and Forecasting with Projects Series, ML System Design Case Studies Series videos will be published on our youtube channel ( just launched).
Subscribe today!
Let’s dive in!
- Iterators: In Python, an iterator is an object that implements the
__iter__method and the__next__method, allowing it to be used in a for loop or with thenextfunction. The__iter__method returns the iterator object itself, and the__next__method returns the next value in the iteration. When there are no more values to return, the__next__method raises aStopIterationexception. - Generators: A generator is a special type of iterator that uses the
yieldstatement to return values one at a time, rather than returning all values at once. Generators are created using a generator function, which is a function that contains at least oneyieldstatement. When the generator function is called, it returns a generator object that can be used in a for loop or with thenextfunction. - Decorators: In Python, a decorator is a special type of function that is used to modify the behavior of another function. A decorator is applied to a function using the
@symbol followed by the name of the decorator function. Decorators can be used to add or modify the behavior of a function, such as adding logging, timing, or authentication.
In summary, iterators and generators are used to implement custom iterable objects in Python, and decorators are used to modify the behavior of functions. These concepts are important building blocks for creating flexible and reusable code in Python.
Iterators
An iterator is an object that can be iterated upon which means that you can traverse through all the values. List, tuples, dictionaries, and sets are all iterable objects.
To create an object as an iterator you have to implement the methods __iter__() and __next__() to your object where —
__iter__() returns the iterator object itself. This is used in for and in statements.
__next__() method returns the next value in the sequence. In order to avoid the iteration to go on forever, raise the StopIteration exception.
For the implementation, I wrote a simple code:
class example_range:
def __init__(self, n):
self.i = 4
self.n = ndef __iter__(self):
return selfdef __next__(self):
if self.i < self.n:
i = self.i
self.i += 1
return i
else:
raise StopIteration()Output:
n= example_range(10)
list(n)Output--[4, 5, 6, 7, 8, 9]You can use this in your code. For example like:
n= example_range(15)
for i in n :
print (i, end=',')Writing Efficient Python Code — Part 2
Use these hacks and techniques…
medium.datadriveninvestor.com
Output:
4,5,6,7,8,9,10,11,12,13,14As compared to for loop above:
iterator = iter(n)
while True:
try:
x = iterator.__next__()
print(x, end=',')
except StopIteration as e:
breakOutput:
4,5,6,7,8,9,10,11,12,13,14Some of the other best Series —
How to solve any System Design Question ( approach that you can take)?
30 days of Data Structures and Algorithms and System Design Simplified
Data Science and Machine Learning Research ( papers) Simplified **
100 days : Your Data Science and Machine Learning Degree Series with projects
Complete Data Visualization and Pre-processing Series with projects
Exceptional Github Repos — Part 1
Exceptional Github Repos — Part 2
Tech Newsletter —
If you are interested, you can join my newsletter through which I send tech interview tips, techniques, patterns, hacks — Software Development, ML, Data Science, Startups and Technology projects to more than 30K readers. You can subscribe to Tech Brew :
Why use iterators?
Iterators allow us to create and work with lazy iterable which means you can use an iterator for the lazy evaluation. This allows you to get the next element in the list without re-calculating all of the previous elements. Iterators can save us a lot of memory and CPU time.
Python has many built-in classes that are iterators, e.g — enumerate, map ,filer , zipand reversed etc. objects are iterators.
Generators
Generator functions act just like regular functions with just one difference that they use the Python yieldkeyword instead of return . A generator function is a function that returns an iterator. A generator expression is an expression that returns an iterator. Generator objects are used either by calling the next method on the generator object or using the generator object in a “for in” loop.
Code:
def test_sequence():
num = 0
while num<10:
yield num
num += 1
for i in test_sequence():
print(i, end=",")Output:
0,1,2,3,4,5,6,7,8,9A return statement terminates a function entirely but a yield statement pauses the function saving all its states and later continues from there on successive calls.
Python Generators with a Loop
#Reverse a string
def reverse_str(test_str):
length = len(test_str)
for i in range(length - 1, -1, -1):
yield test_str[i]for char in reverse_str("Trojan"):
print(char,end =" ")Output:
n a j o r TGenerator Expression
Generator expressions can be used as the function arguments. Just like list comprehensions, generator expressions allow you to quickly create a generator object within minutes with just a few lines of code.
Code:
# Initialize the list
test_list = [1, 3, 6, 10]# list comprehension
list_comprehension = [x**3 for x in test_list]# generator expression
test_generator = (x**3 for x in test_list)print(list_)
print(type(test_generator))
print(tuple(test_generator))Output:
[1, 9, 36, 100]
<class 'generator'>
(1, 27, 216, 1000)The major difference between a list comprehension and a generator expression is that a list comprehension produces the entire list while the generator expression produces one item at a time as lazy evaluation. For this reason, compared to a list comprehension, a generator expression is much more memory efficient that can be understood from profiling code below —
import sys
cubed_list = [i ** 3 for i in range(10000)]
print("List comprehension size(bytes):", sys.getsizeof(cubed_list))cubed_generator = (i ** 3 for i in range(10000))
print("Generator Expression object(bytes):", sys.getsizeof(cubed_generator))Output:
List comprehension size(bytes): 87624
Generator Expression object(bytes): 120When we run the profiling:
import cProfile as profiling
#list comprehension profiling
profiling.run('sum([i ** 3 for i in range(10000)])')
#Generator Expression profiling
profiling.run('sum((i ** 3 for i in range(10000)))')Output:
5 function calls in 0.004 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function)
1 0.004 0.004 0.004 0.004 <string>:1(<listcomp>)
1 0.000 0.000 0.004 0.004 <string>:1(<module>)
1 0.000 0.000 0.004 0.004 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {built-in method builtins.sum}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
10005 function calls in 0.014 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function)
10001 0.012 0.000 0.012 0.000 <string>:1(<genexpr>)
1 0.000 0.000 0.014 0.014 <string>:1(<module>)
1 0.000 0.000 0.014 0.014 {built-in method builtins.exec}
1 0.002 0.002 0.014 0.014 {built-in method builtins.sum}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}Decorator
A decorator in Python is any callable Python object that is used to modify a function or a class. It takes in a function, adds some functionality, and returns it. Decorators are a very powerful and useful tool in Python since it allows programmers to modify/control the behavior of function or class. Decorators are usually called before the definition of a function you want to decorate. There are two different kinds of decorators in Python:
- Function decorators
- Class decorators
Code:
def test_decorator(func):
def function_wrapper(x):
print("Before calling " + func.__name__)
res = func(x)
print(res)
print("After calling " + func.__name__)
return function_wrapper@test_decorator
def sqr(n):
return n ** 2sqr(54)Output:
Before calling sqr
2916
After calling sqrMultiple Decorators to a Single Function
When using Multiple Decorators to a single function, the decorators will be applied in the order they’ve been called.
Code:
def lowercase_decorator(function):
def wrapper():
func = function()
make_lowercase = func.lower()
return make_lowercasereturn wrapperdef split_string(function):
def wrapper():
func = function()
split_string = func.split()
return split_stringreturn wrapper
@split_string
@lowercase_decorator
def test_func():
return 'MOTHER OF DRAGONS'
test_func()Output:
['mother', 'of', 'dragons']You can also pass the arguments to the wrapper function.
Here are some tricks for using iterators, generators, and decorators in Python:
- Iterators:
- Use the
enumeratefunction to iterate over a sequence and keep track of the index. - Use the
zipfunction to iterate over multiple sequences in parallel. - Use the
itertoolsmodule for advanced iteration techniques, such as chain and cycle.
2.Generators:
- Use generators to implement lazy evaluation and avoid loading large amounts of data into memory.
- Use the
yieldstatement to implement a generator function that returns values one at a time. - Use generator expressions to create generators with a compact syntax.
3.Decorators:
- Use decorators to add logging, timing, or error handling to functions.
- Use the
functools.wrapfunction to preserve the original function's metadata, such as its name and docstring. - Use the
functools.partialfunction to partially apply arguments to a function and create a new function with fewer arguments.
In summary, these tricks can help you write more concise and efficient code using iterators, generators, and decorators in Python.
All the Complete System Design Series Parts —
6. Networking, How Browsers work, Content Network Delivery ( CDN)
Github —
Thanks for reading. Keep Learning :)
Advanced SQL Series
Day 2 : SQL Basics, Query Structure, Built In functions Conditions
Day 4 : Set Theory Operations, Stored Procedures and CASE statements in SQL
Day 6 : Subqueries, Group by, order by and Having clauses in SQL and Analytical Functions
Day 7 : Window Functions, Grouping Sets and Constraints in SQL
Day 8 : BigQuery Basics, SELECT, FROM, WHERE and Date and Extract in BigQuery
Day 9 : Common Expression Table, UNNEST Clause, SQL vs NoSQL Databases
Day 10 : Triggers, Pivot and Cursors in SQL
Day 14 : MySQL in Depth
Day 15 : PostgreSQL inDepth
Anyways, For Day 15 of 15 days of Advanced SQL, we will cover —
PostgreSQL inDepth
Github for Advanced SQL that you can follow —
All the projects, data structures, algorithms, system design, Data Science and ML, Data Engineering, MLOps and Deep Learning videos will be published on our youtube channel ( just launched).
Subscribe today!
System Design Case Studies — In Depth
Complete Data Structures and Algorithm Series
Github —
