If you are interested to learn about the Python decorator
What is Python Generator?
Python Generators are the functions that return the traversal object and used to create iterators. It traverses the entire items at once. Python provides a generator to create your own iterator function. A generator is a special type of function which does not return a single value, instead, it returns an iterator object with a sequence of values. In a generator function, a yield statement is used rather than a return statement. The generator can also be an expression in which syntax is similar to the list comprehension in Python. There is a lot of complexity in creating iteration in Python; we need to implement __iter__() and __next__() method to keep track of internal states. It is a lengthy process to create iterators. That’s why the generator plays an essential role in simplifying this process. If there is no value found in iteration, it raises StopIteration exception.
Why do we Need Python Generators?
There are several reasons in order to make/Create a Python Generator. They are Easy Implementation, Memory Efficiency, Infinite Stream representation, and Generators Pipelining.
- Easy Implementation: Python Generators are implemented in an easy and clear, concise way as compared to the iterator class’s counterpart.
- Memory Efficiency: This will be a sequence that is memory friendly because it will only iterate/produce only one item at a time, whereas a normal function with the return will return the sequence, which creates the whole sequence in the memory before providing the result. So the Python Generator Function is widely preferred compared to the normal function creation because the normal function’s sequences memory is so large than many Function/Functions.
- Infinite Stream Representation: Representing the infinite stream of vast data need an excellent and great medium, and that medium is nothing but the Functions.
- Generators Pipelining: Python Generator Function/Functions are used in creating the connection to the series of operations like a pipeline, so it is called Pipelining Generators/Generators Pipelining. It’s like connecting the series of the sequences one by one, just like the iteration.
Syntax:
def function(): List of statements Yield statement Variable= sum(function)
Explanation: The above syntax creates a function with some list of statements and the yield statement whenever required. Using the yield term inside of the normal function makes the function as Python Generator Function/Python Generator. Then a variable is created and assigned with the sum(function) value. One can also use the print in order to know what is the value of the sum() function’s value which is stored in the variable.
How to Create Python Generators?
Creating Python Generator/Generator Function is very simple with the help of the “yield” statement and the normal function. The yield statement is used instead of the “return” statement.
If the “yield” statement is used in a normal function, then the function is called a generator function. Yield is similar to the return function. These “yield” and “return” will provide mostly the same value from the function. The “yield” statement usually pauses the normal function by saving all of its states, and later the functions will be continued from the remaining successive calls. Generator Function may contain many yield statements as required/needed.
How to Create Generator function in Python?
It is quite simple to create a generator in Python. It is similar to the normal function defined by the def keyword and uses a yield keyword instead of return. Or we can say that if the body of any function contains a yield statement, it automatically becomes a generator function. Consider the following example:
def simple():
for i in range(10):
if(i%2==0):
yield i
#Successive Function call using for loop
for i in simple():
print(i) Output:
0
2
4
6
8
yield vs. return
The yield statement is responsible for controlling the flow of the generator function. It pauses the function execution by saving all states and yielded to the caller. Later it resumes execution when a successive function is called. We can use the multiple yield statement in the generator function. The return statement returns a value and terminates the whole function and only one return statement can be used in the function.
Using multiple yield Statement
We can use the multiple yield statement in the generator function. Consider the following example.
def multiple_yield():
str1 = "First String"
yield str1
str2 = "Second string"
yield str2
str3 = "Third String"
yield str3
obj = multiple_yield()
print(next(obj))
print(next(obj))
print(next(obj)) Output:
First String
Second string
Third StringDifference between Generator function and Normal function
- Normal function contains only one Lreturn statement whereas generator function can contain one or more yield statement.
- When the generator functions are called, the normal function is paused immediately and control transferred to the caller.
- Local variable and their states are remembered between successive calls.
- StopIteration exception is raised automatically when the function terminates.
Generator Expression
We can easily create a generator expression without using user-defined function. It is the same as the lambda function which creates an anonymous function; the generator’s expressions create an anonymous generator function. The representation of generator expression is similar to the Python list comprehension. The only difference is that square bracket is replaced by round parentheses. The list comprehension calculates the entire list, whereas the generator expression calculates one item at a time. Consider the following example:
list = [1,2,3,4,5,6,7] # List Comprehension z = [x**3 for x in list] # Generator expression a = (x**3 for x in list) print(a) print(z)
Output:
<generator object <genexpr> at 0x01BA3CD8> [1, 8, 27, 64, 125, 216, 343]
In the above program, list comprehension has returned the list of cube of elements whereas generator expression has returned the reference of calculated value. Instead of applying a for loop, we can also call next() on the generator object. Let’s consider another example:
list = [1,2,3,4,5,6] z = (x**3 for x in list) print(next(z)) print(next(z)) print(next(z)) print(next(z))
Output:
1
8
27
64Note:- When we call the next(), Python calls __next__() on the function in which we have passed it as a parameter.
In the above program, we have used the next() function, which returned the next item of the list.
Example: Write a program to print the table of the given number using the generator.
def table(n):
for i in range(1,11):
yield n*i
i = i+1
for i in table(15):
print(i) Output:
15
30
45
60
75
90
105
120
135
150In the above example, a generator function is iterating using for loop.
Advantages of Generators
There are various advantages of Generators. Few of them are given below:
1. Easy to implement
Generators are easy to implement as compared to the iterator. In iterator, we have to implement __iter__() and __next__() function.
2. Memory efficient
Generators are memory efficient for a large number of sequences. The normal function returns a sequence of the list which creates an entire sequence in memory before returning the result, but the generator function calculates the value and pause their execution. It resumes for successive call. An infinite sequence generator is a great example of memory optimization. Let’s discuss it in the below example by using sys.getsizeof() function.
import sys
# List comprehension
nums_squared_list = [i * 2 for i in range(1000)]
print(sys.getsizeof("Memory in Bytes:"nums_squared_list))
# Generator Expression
nums_squared_gc = (i ** 2 for i in range(1000))
print(sys.getsizeof("Memory in Bytes:", nums_squared_gc)) Output:
Memory in Bytes: 4508
Memory in Bytes: 56
We can observe from the above output that list comprehension is using 4508 bytes of memory, whereas generator expression is using 56 bytes of memory. It means that generator objects are much efficient than the list compression.
3. Pipelining with Generators
Data Pipeline provides the facility to process large datasets or stream of data without using extra computer memory. Suppose we have a log file from a famous restaurant. The log file has a column (4th column) that keeps track of the number of burgers sold every hour and we want to sum it to find the total number of burgers sold in 4 years. In that scenario, the generator can generate a pipeline with a series of operations. Below is the code for it:
with open('sells.log') as file:
burger_col = (line[3] for line in file) per_hour = (int(x) for x in burger_col if x != 'N/A')
print("Total burgers sold = ",sum(per_hour)) 4. Generate Infinite Sequence
The generator can produce infinite items. Infinite sequences cannot be contained within the memory and since generators produce only one item at a time, consider the following example:
def infinite_sequence():
num = 0
while True:
yield num
num += 1
for i in infinite_sequence():
print(i) Output:
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
.........
..........
315
316
317
Traceback (most recent call last):
File "C:\Users\DEVANSH SHARMA\Desktop\generator.py", line 33, in <module>
print(i)
KeyboardInterrupt
Applications : Suppose we to create a stream of Fibonacci numbers, adopting the generator approach makes it trivial; we just have to call next(x) to get the next Fibonacci number without bothering about where or when the stream of numbers ends.
A more practical type of stream processing is handling large data files such as log files. Generators provide a space efficient method for such data processing as only parts of the file are handled at one given point in time. We can also use Iterators for these purposes, but Generator provides a quick way (We don’t need to write __next__ and __iter__ methods here).
