Simply put, a Python generator is a function that potentially can return many values, and it is also able to maintain its own state between the returns. This means that you can call the function multiple times and it will return a new result every time. Each time you call it subsequently, it knows its last location and will continue from that point.
The following example function generates Fibonacci numbers:
When this function is called for the first time, it will initiate x and y and enter the infinite loop. The first statement in the loop is to return the value of y (note that in generators you must use the yield statement). The next time you call this function it'll start from the point where it stopped execution and returned a value—the yield statement. The next statement is to reassign x and y with new values, where x becomes the old y and the new y is a sum of old y and x. It is important to note that calling the generator function does not return the values the function is meant to calculate—it returns the actual generator object. You can then either iterate through it as you would normally do with a list or call next () method, which will get you the next value:
13 21 34
As you can see, generators are actually functions and not lists, but they can be used as lists. Sometimes, as in the Fibonacci example, the virtual lists can be infinite. When the generator has a limited set of results, such as lines in a file or rows in a database query, it must raise a Stoplteration exception, which will signal the caller that there are no more results available.
You can use generators to go through all lines in the file. This will effectively return the next line whenever you call the next () function without actually loading the whole file into memory. Once it is defined as a generator, you can just iterate through it.
In my code I have a get_suspect () function, which is effectively a generator that returns excerpts of text from the log file that potentially might be an exception stack trace. This function accepts a generator as its argument and iterates through it, therefore retrieving all the lines. First of all I create a generator that returns all lines in the file:
And then I use this generator to retrieve the lines in my function:
def get_suspect(g): line = g.next() next_line = g.next() while 1:
<do something with line and next yield result try:
line, next_line = next_line, except:
I enclose the call to next() in the try: ... except: clause because when the last line of the file has been reached, the generator will raise an exception. Therefore, when the file cannot be read anymore, I simply raise the StopIteration exception, which acts as a signal to the iterator that the generator has exhausted all its values.
Was this article helpful?