Forge Learn/Functional Python & Iterators

Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Iterators & the Iterator Protocol

4 min read

You'll learn to

-Explain what makes an object iterable versus an iterator
-Use iter() and next() directly, and understand StopIteration
-Implement __iter__ and __next__ to make a custom class iterable
-Explain why an iterator is single-use, and how to make a class that can be looped over again

You have written dozens of for item in some_list: loops without needing to know what happens underneath. This chapter opens up that machinery. It is a small amount of new vocabulary that immediately explains generators (next chapter) and lets you make your own classes work seamlessly with for loops.

An Analogy: A Cookbook and a Finger on the Page

A cookbook is something you can read as many times as you like. But reading it aloud takes a reader: someone with a finger on the current page who moves on one recipe at a time. Hand the cookbook to two readers and each has their own finger, in their own place. When a reader reaches the end, their finger is off the last page, and to read again you need a new reader, not the same one.

-The cookbook is the iterable: it can produce a fresh reader whenever you ask (iter()).
-The reader with a finger on the page is the iterator: it remembers its position and hands over the next item (next()).
-Reaching the end of the book is StopIteration.

Iterable vs. Iterator

An iterable is anything you can call iter() on to get an iterator: lists, tuples, strings, dicts, sets, files. An iterator is the object that actually produces values one at a time, via next(), and raises StopIteration when there is nothing left. A for loop is just convenient syntax that calls iter() once, then next() repeatedly, catching StopIteration for you automatically.

What a for loop does under the hood

numbers = [10, 20, 30]

# What "for n in numbers:" actually does, spelled out:
iterator = iter(numbers)
while True:
    try:
        n = next(iterator)
    except StopIteration:
        break
    print(n)
# 10
# 20
# 30

# The actual for loop - identical result:
for n in numbers:
    print(n)

A list is iterable, but it is not itself an iterator. iter([1, 2, 3]) returns a separate list_iterator object. That is why you can have two independent for loops over the same list running at different positions at once.

An Iterator Is Used Up

The finger only moves forward. Once an iterator has been exhausted, it stays exhausted, and asking it for more gives nothing. A list, being the cookbook, can be iterated as often as you like:

An iterator is single-use; the list it came from is not

it = iter([1, 2, 3])
print(list(it))    # [1, 2, 3]
print(list(it))    # []   <- the iterator is used up

numbers = [1, 2, 3]
print(list(numbers), list(numbers))    # [1, 2, 3] [1, 2, 3]   <- a list can be iterated again

Implementing iter and next on a Custom Class

To make your own class work in a for loop, implement __iter__ (returning an iterator, often self) and __next__ (returning the next value, or raising StopIteration when exhausted).

A custom class implementing the iterator protocol

class CountUpTo:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        self.current += 1
        return self.current

for n in CountUpTo(5):
    print(n)
# 1
# 2
# 3
# 4
# 5

The Trap in Returning self

Look again at CountUpTo. Its __iter__ returns self, so the object is its own iterator, and the position (self.current) lives inside it. That makes it single-use: after one for loop, self.current has reached the limit, and a second loop over the same object finds it already finished. The fix is to make __iter__ hand out a fresh iterator each time:

Returning self makes a class single-use; returning a fresh iterator lets you loop again

class CountUpTo:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self                      # the object is its own iterator

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        self.current += 1
        return self.current

c = CountUpTo(3)
print(list(c), list(c))                  # [1, 2, 3] []   <- second loop sees nothing

class Countdown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):                  # a fresh iterator on every call
        return iter(range(self.start, 0, -1))

d = Countdown(3)
print(list(d), list(d))                  # [3, 2, 1] [3, 2, 1]

One more reason this design matters: an iterator produces values one at a time, so nothing has to be stored. sum(range(10_000_000)) adds ten million numbers without ever building a ten-million-item list, and the generators in the next chapter rely on the same idea.

-iter(obj) calls obj.__iter__(). It produces an iterator.
-next(it) calls it.__next__(). It produces the next value or raises StopIteration.
-A for loop is sugar for calling iter() once and next() repeatedly until StopIteration.
-Any class implementing both __iter__ and __next__ works directly with for, list(...), sum(...), and every other function that expects an iterable.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Virtual Environments & pip, Briefly Generators & yield

Discussion0

Join the Discussion

No comments yet. Be the first to start the thread!