Generators: Functions that look like Iterators Python includes a special kind of function called a generator (also known classically in Computer Science as a coroutine). With generators we can easily (almost trivially) write iterators. In this lecture we will first study generators by themselves, to understand their semantics, and then use them to write a variety of iterators easily. I believe that the generator concept is one of the most intuitively easy to understand features in Python (and powerful to boot), so I think you will enjoy reading this lecture. Generators are defined almost exactly like functions; the one difference is that inside a generator there are yield statements (one or more), not return statements. A yield statement has the same form as a return statment: each keyword is followed by an expression that is evaluated, terminating the function/generator and returning that value (but generators can be restarted, where they left off). When we call a function, its parameters receive values from its arguments, its body executes, and when it executes a return statement the specified value is returned (or None is automatically returned if the last statement of a function is executed without ever executing a return). When a function returns this value, it is done and forgets the state of its parameter and local variables. When called again it starts at the beginning. When we call a generator, likewise its parameters receive values from its arguments, but its body does not execute. The generator "suspends". To execute its body we use the generator as an argument to the next(...) function. So calling a generator is similar to calling iter(...) on an object: the result in both cases is something on which next(...) can be called. Each time next is called the generator resumes from where it was suspended, further executing its body. When it executes a yield statement the specified value is returned (or StopIteration is automatically raised if the last statement of a generator is executed; the generator can also raise this exception explicitly and calling return raises StopIteration as well). When a function yields a value it suspends (remembering the state of its parameters, local variables, and execution point). When we call next(...) on it again, it resumes where it left off. Now it is time for a concrete example that we will study, going back to apply the rules stated above in particular cases. In the generator below, the while loop is infinite (if max == None) or finite (p will get no bigger than max if it != None). Each iteration checks whether the current p is a prime and yields p's value if so; otherwise it checks if a p one bigger is prime (unless p has surpassed max). To primes is a generate that produces only prime numbers. from predicate import is_prime # Could use any predicate def primes(max = None): p = 2 while max == None or p <= max: if is_prime(p): yield p p += 1 Quick note: thie while's test uses the short-circuit property of or: if max == None is True, then the entire expression is True, so Python does not have to evaluate the right operand (and if it did, it would raise an exception because Python cannot compare p (an int value) to None; instead it would raise the exception: TypeError: unorderable types: int() <= NoneType(). Before looking at an example in detail that uses this generator, it is easy to determine what it does: it yield only prime numbers, either without limit or whose values are limited to be <= max. The code is straightford does the obvious thing: to most programmers its mechanism, suspending and resuming execution, is easy to understand and use to write generators. Let us determine what happens if we execute the following Python code, which stores the result of calling the primes generator in pg (prime generator) and then continues to print the values returned by calling next(pg). Note that calling a generator like primes(10) is equivalent to calling iter(primes(10)): both prepare the generator for use as a parameter to next(...) pg = primes(10) print(pg) print(next(pg)) print(next(pg)) print(next(pg)) print(next(pg)) print(next(pg)) When Python executes this script, it produces the following results. An explanation of these results follows. 2 3 5 7 Traceback (most recent call last): File "C:\Users\Pattis\workspace\zexperiment\experiment.py", line 12, in print(next(pg)) StopIteration When we call primes(10), this generator binds its parameter max to 10 and suspends before executing its body. Generators print much like functions, indicating they are a generator, named primes, and stored at a memory location (a value we are unconcerned with knowoing). The first call of next(pg) resumes the suspended execution of the generator, which binds p to 2 and then starts the while loop. Since max != None it will continue to loop so long as p <= 10. It executes the body of the loop and the if statement checks whether p (now 2) is prime; it is, so it yields the value 2 (returning it from the generator and suspending the generator). Python prints 2. The second call of next(pg) resumes the suspended execution of the generator after the yield, which increments p to 3 and then re-executes the while loop's body (p is <= 10). The if statment checks whether p (now 3) is prime; it is, so it yields the value 3 (returning it from the generator and suspending the generator). Python prints 3. The third call of next(pg) resumes the suspended execution of the generator after the yield, which increments p to 4 and then re-executes the while loop's body (p is <= 10). The if statment checks whether p (now 4) is prime; it isn't, so Python increments p to 5 and re-executes the while loop's body (p <= 10). The if statment checks whether p (now 5) is prime; it is, so, it yields the value 5 (returning it from the generator and suspending the generator). Python prints 5. The fourth call of next(pg) resumes the suspended execution of the generator after the yield, which increments p to 6 and then re-executes the while loop's body (p is <= 10). The if statment checks whether p (now 6) is prime; it isn't, so Python increments p to 7 and re-executes the while loop's body (p <= 10). The if statment checks whether p (now 7) is prime; it is, so, it yields the value 7 (returning it from the generator and suspending the generator). Python prints 7. The fifth call of next(pg) resumes the suspended execution of the generator, after the yield, which increments p to 8 and then re-executes the while loop's body (p is <= 10). The if statment checks whether p (now 8) is prime; it isn't, so Python increments p to 9 and re-executes the while loop's body (p <= 10). The if statment checks whether p (now 9) is prime; it isn't so Python increments p to 10 and re-executes the while loop's body (p <= 10). The if statment checks whether p (now 10) is prime; it isn't, so Python increments p to 11 but terminates the loop (does not re-execute its body), because p > 10. Because there are no more statements to execute in the generator, it raises the StopIteration exception (which is not handled, so it propagates to Python which terminates execution of the script. Note that if we reset pg to a new call of primes(10) print(next(pg)) print(next(pg)) print(next(pg)) pg = primes(10) print(next(pg)) print(next(pg)) Python would print 2 3 5 2 3 because calling primes(10) starts the generator over. If we wrote the script p1 = primes(10) p2 = primes(10) print(next(p1)) print(next(p1)) print(next(p1)) print(next(p1)) print(next(p1)) print(next(p1)) Python would print 2 2 3 3 5 5 because we now have two different (not shared) generators, and each works independently of the other. Finally, although we have been slogging around with explicit calls to next(...), which we can do with generators, they are most simple to use it the typical contexts in which we use iterators. The script for i in primes(50): print(i,end=' ') would print all the primes <= 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47 And the script for i in primes(): print(i) binds the parameter max to None (its default value) so the while loop would be infinite, and it would keep printing primes forever: so the primes function can either be a definite (bounded) or indefinite (unbounded) generator of primes (because there are an infinite number of primes). We could alter the meaning of primes to bound not the VALUE of the prime produced, but to bound the NUMBER of primes produced. from predicate import is_prime def primes(max_to_return = None): p = 2 while max_to_return == None or max_to_return > 0: if is_prime(p): if max_to_return != None: max_to_return -= 1 yield p p += 1 for i in primes(20): print(i,end=' ') This script produces not primes up to 20, but 20 primes. When run it prints 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47 53 59 61 67 71 That finishes our discussion of generators unto themselves. The next sections discuss how to use generators to implement the __iter__/__next__ protocal of iterators for classes, and how to write iterator decorators. ------------------------------------------------------------------------------ Generators for implementing __iter__/__next__ in classes In the previous lecture, I wrote the __iter__ method as follows. I could have just returned iter(list(self.histogram)) using the standard list iterator, but I wanted to show how to write an iterator that did its own indexing of the list (as the list iterator actually, does, to showyou how to do it). # standard __iter__: defines a class with __init__/__next__ and returns # an object from that class def __iter__(self): class PH_iter: def __init__(self,histogram): # copy so iteration not affected by mutation self.histogram = list(histogram) self.next = 0 def __next__(self): if self.next == 10: raise StopIteration answer = self.histogram[self.next] self.next += 1 return answer return PH_iter(self.histogram) Using generators, we can rewrite this much more simply, using a class function (named _gen, so it is not meant to be used by methods outside the class). It still using indexes to iterate over the list; it does not use the list iterator. def _gen(bins): for i in range(10): yield bins[i] def __iter__(self): # copy so iteration not affected by mutation (e.g., clear/tally method) return Percent_Histogram._gen(list(self.histogram)) Similarly to how we defined a class in __iter__ we can define the _gen generator there as well, hiding this helper even deeper. def __iter__(self): def _gen(bins): for i in range(10): yield bins[i] # copy so iteration not affected by mutation (e.g., clear/tally method) return _gen(list(self.histogram)) ------------------------------------------------------------------------------ Iterable Decorators: Generators (iterable) that use Iterable Arguments Also in the previous lecture I wrote a few classes that implemented iterables, each of which took an iterable argument (and possibly some other arguments). These classes were not large (about a dozen lines: __init__, __iter__, and an embedded class with __next__: that is a lot of infrastructure), but each can be written almost trivially as generators (just a few lines, and much simpler to understand code: not split across __init__, __iter__, and __next__ ). Note the term decorator means that the thing created is the same type of thing that is its argument (but decorated: with a change). So all these generators take iterable as an argument, and because they are generators they are iterable themselves. Therefore we can compose iterable on top of iterable on top of iterable (see the last example in this section). Here are the rewrite of classes as generators. 1) repeatedly produce values from an iterable (over and over again) def repeat(iterable): while True: for i in iterable: yield i which I generalized to def repeat(iterable,max_times=None): while self.max_times == None or self.max_times > 0: for i in iterable: yield max_times -= 1 2) produce unique values (never the same twice) def unique(iterable,max_times=1): iterated = set() for i in iterable: if i not in iterated: iterated.add(i) yield i which I generalized to from collections import defaultdict def unique(iterable,max_times=1): times = defaultdict(int) for i in iterable: if times[i] < max_times: times[i] += 1 yield i 3) Filter an iterable, producing only values for which predicate p returns True def pfilter(iterable,p): for i in iterable: if p(i): yield i 4) Produce values in a sorted sequence, according to key/reverse def psorted(iterable,key=None,reverse=False): l = list(iterable) l.sort(key=key,reverse=reverse) for i in l: yield i We can easily test generators on strings, which are iterable (returning the individual characters). E.g., the following example print only the vowels, in sorted order, uniquely, that are in my name: for i in pfilter(psorted(unique('richrdepattis')), lambda x : x in 'aeiou'): print(i,end='') Let's now return to the original primes generator at the start of this lecture. We now have some tools that we can use to simplify (actually avoid) this generator. The general component below generates a sequence of integers, either bound or unbounded (unlike range, which is always bounded). def ints(start,max = None): i = start while max == None or i <= max: yield i i += 1 Now, instead of calling primes we would call pfilter(ints(2),is_prime) to represent the same iterator. And if we defined any other predicates, we could supply them to pfilter to generate only values that satisfied the predicate. Python has a module called itertools (see the library documentation) which define many iterator decorators, whose composition allows for powerful iteration in Python. ------------------------------------------------------------------------------ Space Efficiency Note that generators embody a small amount of code and often don't store any large data structures (unique is unique among the generators above in that is uses a set/dict). Generators store/produce one value at a time, unlike, say, a comprehension that produces an entire tupe/list/set/dict of values that must all be stored at the same time. So when writing generators, we should always try to avoid storing a large number in a data structure. It might make writing the generator more difficult, but few generators need to store large amounts of data to work correctly. But, for generators like reversed and sorted, Python must look at all the iterated values to decide what to return first, so they typically work by first storing all the values iterated over in some data structure (a tuple or list: see psorted above and preversed below and then process values in the tupe/list (which we hope is not huge) def preversed(iterable) l = list(iterable) for i in irange(len(l)-1,0,-1): yield l[i] But there actually is a way to generate reversed values without every storing the iterable in a list, but the time is takes (in its doubly nested loop) can be very big; so for choosing the implementation we want, we have to make a choice whether time or space is more important: maybe we should two generators: reversed_save_time and reversed_save_space. def reversed_save_space(iterable): length = 0; for i in iterable: length += 1 for c in range(length): # find the value at index (length-c) by skipping length-c-1 and then # yeilding the next value temp = iter(iterable) for i in range(length-c-1): dummy = next(temp) yield next(temp) While this seems very fast for small lists it can run slowly for large lists, because it is repeatedly skipping large numbers of values in the inner loop for each yield in the outer loop. for i in reversed_save_space([i for i in range(1000000)]): print(i)