Basic Usage Patterns and Examples

As mentioned earlier, the multiprocessing library API is very similar to the threading library. The listings and code snippets in this section provide several examples of how to create multiple processes and exchange data between them.

You can define the code you want to run in a separate process either as a function or a class that inherits from the multiprocessing.Process class. There are no hard rules about which approach to use and when; it largely depends on the task size and complexity of the code. I prefer to use classes instead of functions because it allows me to extend the code base more easily; also, the new classes can be extended, so the application code can be used as a base library for the new applications that extend functionality.

Listing 9-5 demonstrates creating processes with the multiprocessing library.

Listing 9-5. Creating processes with the multiprocessing library import multiprocessing import time def sleeper(timeout):

print "function: I am a sleeper function and going to sleep for %s seconds" % timeout time.sleep(timeout) print "function: I'm done!"

class SleeperClass(multiprocessing.Process): def __init__(self, timeout): self.timeout = timeout print "Class: I am a class and can do initialisation tasks before starting" super(SleeperClass, self).__init__()

def run(self):

print "Class: I have been told to run now"

print "Class: So I'm going to sleep for %s seconds" % self.timeout time.sleep(self.timeout) print "Class: I'm done."

pi = multiprocessing.Process(target=sleeper, args=(5,))

p2 = SleeperClass(i0)

pi.start()

p2.start()

As you can see, if you're using classes you have the advantage of running some initialization tasks before the process is started. Running the example code will produce the following results:

Class: I am a class and can do initialisation tasks before starting function: I am a sleeper function and going to sleep for 5 seconds Class: I have been told to run now

Class: So I'm going to sleep for l0 seconds function: I'm done! Class: I'm done.

When you develop applications that spawn multiple processes, and especially if they are going to be long-running processes, such as services, you have to handle interrupts, so that all processes are terminated gracefully. Now let's do a quick experiment and see what happens if you hit Ctrl-C when the program is running:

Class: I am a class and can do initialisation tasks before starting function: I am a sleeper function and going to sleep for 5 seconds

Class: I have been told to run now

Class: So I'm going to sleep for l0 seconds

ACTraceback (most recent call last):

File "./example_processes.py", line 26, in <module> pl.join()

File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/^ multiprocessing/process.py", line 119, in join Process Process-1: Traceback (most recent call last): Process SleeperClass-2:

File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/^ multiprocessing/process.py", line 231, in _bootstrap Traceback (most recent call last):

File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/^ multiprocessing/process.py", line 231, in _bootstrap res = self._popen.wait(timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/^ multiprocessing/forking.py", line 117, in wait self.run()

File "./example_processes.py", line 19, in run self.run()

File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/^ multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "./example_processes.py", line 7, in sleeper time.sleep(timeout) return self.poll(o) time.sleep(self.timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/^ multiprocessing/forking.py", line 106, in poll pid, sts = os.waitpid(self.pid, flag) KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt

As you can see, this is pretty poor behavior—both of the processes have received the KeyboardInterrupt exception and terminated abnormally. Also, if you try this experiment multiple times, you may get different results each time. The actual result depends on where the processes were in the CPU execution queue at the time they received the keyboard interrupt signal.

To resolve this issue, I need to catch and handle the interrupts in each of my processes, so that when the interrupt arrives, the process finishes what it was doing and exits gracefully. I am going to wrap both functions into try: ... except KeyboardInterrupt: ... clauses, which allows me to catch all interrupts received by the processes. It is also important to know that the main process also receives the interrupt signal and therefore needs to handle it as well. But what is the main process doing while the child processes are running? It is just waiting for them to finish, so basically it is "stuck" at the p1.join() statement. If there is nothing else for the main process to do, it is best to make it check for the number of running child processes and join them back once all have finished their work. You can see this in Listing 9-6.

Listing 9-6. Multiple processes handling interrupts import multiprocessing import time def sleeper(timeout): try:

print "function: I am a sleeper seconds" % timeout time.sleep(timeout) print "function: I'm done!" except KeyboardInterrupt:

print "function: I have receive function and going to sleep for a signal to stop, exiting... "

class SleeperClass(multiprocessing.Process): def __init__(self, timeout): self.timeout = timeout print "Class: I am a class and can do initialisation tasks before starting" super(SleeperClass, self).__init__()

print "Class: I have been told to run now"

print "Class: So I'm going to sleep for %s seconds" % self.timeout time.sleep(self.timeout) print "Class: I'm done." except KeyboardInterrupt:

print "Class: I must stop now, exiting... "

p1 = multiprocessing.Process(target=sleeper, args=(5,)) p2 = SleeperClass(10) p1.start() p2.start()

try:

while len(multiprocessing.active_children()) != 0: time.sleep(i) except KeyboardInterrupt: pi.terminate() p2.terminate() pi.join() p2.join()

In this example I am calling the multithreading. active_children()function, which returns a list of active processes running. If the list is not empty, the main process just sleeps for one second before checking the list again. When the keyboard interrupt is received, the main process will attempt to terminate the child processes. When you press Ctrl-C, all processes are going to receive this interrupt and will therefore stop their execution. However if you send a SIGINT signal to the main process, it will terminate because the SIGINT will actually raise the KeyboardInterrupt, but unlike with the Ctrl-C combination this signal is not cascaded to the child processes. Therefore, you must either send a signal to the child processes or simply terminate them.

Was this article helpful?

0 0

Post a comment