Table of Contents
Generators are a special type of iterable in Python that allow you to iterate over a sequence of values without creating a full data structure in memory. This makes them a powerful tool for handling large datasets and creating efficient pipelines. In this post, we will explore what generators are, how to create and use them, and the benefits they offer.

1. Understanding Generators
Generators are functions that return an iterator, which we can iterate over (one value at a time). They allow you to declare a function that behaves like an iterator.
Basic Generator Example:
python
def simple_generator():
yield 1
yield 2
yield 3
gen = simple_generator()
print(next(gen)) # Output: 1
print(next(gen)) # Output: 2
print(next(gen)) # Output: 3
2. Creating Generators with yield
The yield keyword is used to return values from a generator function. When the generator is called, it returns a generator object without executing the function. When next() is called on the generator object, the function executes until it encounters yield.
Example:
python
def count_up_to(max):
count = 1
while count <= max:
yield count
count += 1
counter = count_up_to(5)
for num in counter:
print(num)
Output:
1
2
3
4
5
3. Generator Expressions
Generator expressions provide a concise way to create generators. They are similar to list comprehensions but with parentheses instead of square brackets.
Example:
python
gen_exp = (x*x for x in range(5))
for num in gen_exp:
print(num)
Output:
0
1
4
9
16
4. Benefits of Using Generators
- Memory Efficiency: Generators do not store the entire sequence in memory. Instead, they generate each value on-the-fly, which is ideal for large datasets.
- Lazy Evaluation: Generators evaluate each value only when it is needed, reducing overhead.
- Composability: Generators can be easily composed with other generators, allowing for efficient data pipelines.
5. Using yield in a Loop
You can use yield within loops to produce a sequence of values.
Example:
python
def fibonacci(n):
a, b = 0, 1
while n > 0:
yield a
a, b = b, a + b
n -= 1
for num in fibonacci(10):
print(num)
Output:
0
1
1
2
3
5
8
13
21
34
6. Sending Values to a Generator
Generators can also receive values using the send() method. This allows two-way communication with the generator.
Example:
python
def running_total():
total = 0
while True:
num = yield total
if num is None:
break
total += num
gen = running_total()
print(next(gen)) # Output: 0
print(gen.send(10)) # Output: 10
print(gen.send(20)) # Output: 30
print(gen.send(30)) # Output: 60
gen.close()
7. Handling Exceptions in Generators
You can handle exceptions within generators using the try and except blocks. You can also raise exceptions using the throw() method.
Example:
python
def controlled_generator():
try:
yield 1
yield 2
yield 3
except GeneratorExit:
print("Generator closed")
except ValueError as e:
print(f"Error: {e}")
gen = controlled_generator()
print(next(gen)) # Output: 1
print(next(gen)) # Output: 2
gen.throw(ValueError, "Custom error") # Output: Error: Custom error
8. Infinite Generators
Generators can be used to create infinite sequences. These are particularly useful with lazy evaluation where you only consume as much as needed.
Example:
python
def infinite_sequence():
num = 0
while True:
yield num
num += 1
gen = infinite_sequence()
print(next(gen)) # Output: 0
print(next(gen)) # Output: 1
print(next(gen)) # Output: 2
9. Practical Use Cases for Generators
- Reading Large Files: You can use generators to read large files line by line without loading the entire file into memory.
- Streaming Data: Generators are perfect for streaming data from an API or a real-time data source.
- Pipelines: Generators can be composed to create efficient data processing pipelines.
Example: Reading Large Files
python
def read_large_file(file_path):
with open(file_path, 'r') as file:
for line in file:
yield line.strip()
for line in read_large_file('large_file.txt'):
print(line)
Conclusion
In this post, we covered the basics of generators in Python, including how to create and use them, the benefits they offer, and practical use cases. Generators are a powerful tool for writing memory-efficient and performant code. In the next post, we will explore context managers in Python, which allow you to manage resources effectively using the with statement. Stay tuned!
Check out our Trending Courses Demo Playlist
Data Analytics with Power Bi and Fabric |
Could Data Engineer |
Data Analytics With Power Bi Fabic |
AWS Data Engineering with Snowflake |
Azure Data Engineering |
Azure & Fabric for Power bi |
Full Stack Power Bi |
Kick Start Your Career With Our Data Job
Social Media channels
► KSR Datavizon Website :- https://www.datavizon.com
► KSR Datavizon LinkedIn :- https://www.linkedin.com/company/datavizon/
► KSR Datavizon You tube :- https://www.youtube.com/c/KSRDatavizon
► KSR Datavizon Twitter :- https://twitter.com/ksrdatavizon
► KSR Datavizon Instagram :- https://www.instagram.com/ksr_datavision
► KSR Datavizon Face book :- https://www.facebook.com/KSRConsultingServices
► KSR Datavizon Playstore :- https://play.google.com/store/apps/details?id=com.datavizon.courses&hl=en-IN
► KSR Datavizon Appstore :- https://apps.apple.com/in/app/ksr-datavizon/id1611034268
Most Commented