In the Spider class, the process_page() method is separate from run() because run() is easier to understand when you don't have to look at all the code in process_page() at the same time. This choice exemplifies two reasons behind Python's modular design philosophy.
It's a good idea to separate out chunks of code that do different things:
• Separating tasks into chunks makes it possible to use each chunk on its own.
• Reading a chunk of code that does only one thing is easier than reading a chunk of code that does multiple things.
Was this article helpful?