Unlike threads, processes do not support shared objects. Although you can create shared values and arrays as shown in the previous section, this doesn't work for more advanced Python objects such as dictionaries, lists, or instances of user-defined classes.The multiprocessing module does, however, provide a way to work with shared objects if they run under the control of a so-called manager.A manager is a separate subprocess where the real objects exist and which operates as a server. Other processes access the shared objects through the use of proxies that operate as clients of the manager server.
The most straightforward way to work with simple managed objects is to use the Manager() function.
Creates a running manager server in a separate process. Returns an instance of type SyncManager which is defined in the multiprocessing.managers submodule.
An instance m of SyncManager as returned by Manager() has a series of methods for creating shared objects and returning a proxy which can be used to access them. Normally, you would create a manager and use these methods to create shared objects before launching any new processes. The following methods are defined:
Creates a shared Array instance on the server and returns a proxy to it. See the "Shared Data and Synchronization" section for a description of the arguments.
Creates a shared threading.BoundedSemaphore instance on the server and returns a proxy to it.
Creates a shared threading.Condition instance on the server and returns a proxy to it. lock is a proxy instance created by m.Lock() or m.Rlock().
Creates a shared dict instance on the server and returns a proxy to it.The arguments to this method are the same as for the built-in dict() function.
Creates a shared threading.Event instance on the server and returns a proxy to it. m.list([seguence])
Creates a shared list instance on the server and returns a proxy to it.The arguments to this method are the same as for the built-in list() function.
Creates a shared threading.Lock instance on the server and returns a proxy to it. m.Namespace()
Creates a shared namespace object on the server and returns a proxy to it. A namespace is an object that is somewhat similar to a Python module. For example, if n is a namespace proxy, you can assign and read attributes using (.) such as n.name = value or value = n. name. However, the choice of name is significant. If name starts with a letter, then that value is part of the shared object held by the manager and is accessible in all other processes. If name starts with an underscore, it is only part of the proxy object and is not shared.
Creates a shared Queue.Queue object on the server and returns a proxy to it. m.RLock()
Creates a shared threading.Rlock object on the server and returns a proxy to it. m.Semaphore([value])
Creates a shared threading.Semaphore object on the server and returns a proxy to it. m.Value(typecode, value)
Creates a shared Value object on the server and returns a proxy to it. See the "Shared Data and Synchronization" section for a description of the arguments.
The following example shows how you would use a manager in order to create a dictionary shared between processes.
import multiprocessing import time
# print out d whenever the passed event gets set def watch(d, evt): while True:
m = multiprocessing.Manager() d = m.dict() # Create a shared dict evt = m.Event() # Create a shared Event
# Launch a process that watches the dictionary p = multiprocessing.Process(target=watch,args=(d,evt))
# Update the dictionary and notify the watcher d['foo'] = 42
# Update the dictionary and notify the watcher d['bar'] = 37
# Terminate the process and manager p.terminate()
If you run this example, the watch() function prints out the value of d every time the passed event gets set. In the main program, a shared dictionary and event are created and manipulated in the main process. When you run this, you will see the child process printing data.
If you want to have shared objects of other types such as instances of user-defined classes, you have to create your custom manager object. To do this, you create a class that inherits from BaseManager, which is defined in the multiprocessing.managers submodule.
managers.BaseManager([address [, authkey]])
Base class used to create custom manager servers for user-defined objects. address is an optional tuple (hostname, port) that specifies a network address for the server. If omitted, the operating system will simply assign an address corresponding to some free port number. authkey is a string that is used to authenticate clients connecting to the server. If omitted, the value of current_process().authkey is used.
If mgrclass is a class that inherits from BaseManager, the following class method is used to create methods for returning proxies to shared objects.
mgrclass.register(typeid [, callable [, proxytype [, exposed [, method_to_typeid [, create_method]]]]])
Registers a new data type with the manager class. typeid is a string that is used to name a particular kind of shared object. This string should be a valid Python identifier. callable is a callable object that creates or returns the instance to be shared. proxytype is a class that provides the implementation of the proxy objects to be used in clients. Normally, these classes are generated by default so this is normally set to None. exposed is a sequence of method names on the shared object that will be exposed to proxy objects. If omitted, the value of proxytype ._exposed_ is used and if that is undefined, then all public methods (all callable methods that don't start with an underscore (_) are used). method_to_typeid is a mapping from method names to type IDS that is used to specify which methods should return their results using proxy objects. If a method is not found in this mapping, the return value is copied and returned. If method_to_typeid is None, the value of proxytype ._method_to_typeid_ is used if it is defined. create_method is a Boolean flag that specifies whether a method with the name typeid should be created in mgrclass. By default, this is True.
An instance m of a manager derived from BaseManager must be manually started to operate. The following attributes and methods are related to this:
A tuple (hostname, port) that has the address being used by the manager server. m.connect()
Connects to a remote manager object, the address of which was given to the BaseManager constructor.
Runs the manager server in the current process. m.shutdown()
Shuts down a manager server launched by the m.start() method. m.start()
Starts a separate subprocess and starts the manager server in that process.
The following example shows how to create a manager for a user-defined class:
import multiprocessing from multiprocessing.managers import BaseManager class A(object):
return self.x def setX(self,value): self.x = value def__iadd__(self,value):
self.x += value return self class MyManager(BaseManager): pass MyManager.register("A",A)
In this example, the last statement creates an instance of A that lives on the manager server. The variable a in the previous code is only a proxy for this instance. The behavior of this proxy is similar to (but not completely identical to) referent, the object on the server. First, you will find that data attributes and properties cannot be accessed. Instead, you have to use access functions:
Traceback (most recent call last):
File "<stdin>", line 1, in <module> AttributeError: 'AutoProxy[A]' object has no attribute 'x' >>> a.getX()
With proxies, the repr() function returns a string representing the proxy, whereas str() returns the output of__repr__() on the referent. For example:
<AutoProxy[A] object, typeid 'A' at 0xcef230> >>> print(a)
Special methods and any method starting with an underscore (_) are not accessible on proxies. For example, if you tried to invoke a.__iadd__(), it doesn't work:
Traceback (most recent call last):
File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for +=: 'AutoProxy[A]' and 'int' >>> a._ _iadd_ _(37) Traceback (most recent call last): File "<stdin>", line 1, in <module>
AttributeError: 'AutoProxy[A]' object has no attribute '__iadd__'
In more advanced applications, it is possible to customize proxies to more carefully control access.This is done by defining a class that inherits from BaseProxy,which is defined in multiprocessing.managers.The following code shows how you could make a custom proxy to the A class in the previous example that properly exposes the __iadd__() method and which uses a property to expose the x attribute:
from multiprocessing.managers import BaseProxy class AProxy(BaseProxy):
# A list of all methods exposed on the referent _exposed_ = ['__iadd__','getX','setX']
# Implement the public interface of the proxy def__iadd__(self,value):
return self @property def x(self):
return self._callmethod('getX',()) @x.setter def x(self,value):
class MyManager(BaseManager): pass MyManager.register("A", A, proxytype=AProxy)
An instance proxy of a class derived from BaseProxy has the following methods:
Calls the method name on the proxy's referent object. name is a string with the method name, args is a tuple containing positional arguments, and kwargs is a dictionary of keyword arguments. The method name must be explicitly exposed. Normally this is done by including the name in the _exposed_ class attribute of the proxy class.
Returns a copy of the referent in the caller. If this call is made in a different process, the referent object is pickled, sent to the caller, and is unpickled. An exception is raised if the referent can't be pickled.
Was this article helpful?