Memorymappedfile arrays

Memory-mapped files are useful for reading and/or modifying small segments of a large file with regular layout, without reading the entire file into memory. A simple subclass of the ndarray uses a memory-mapped file for the data buffer of the array. For small files, the over-head of reading the entire file into memory is typically not significant, however for large files using memory mapping can save considerable resources.

NOTE

Memory-mapped arrays use the the Python memory-map object which (prior to Python 2.5) does not allow files to be larger than a certain size depending on the platform. This size is always < 2GB even on 64-bit systems.

The class is called memmap and is available in the NumPy namespace. The __new__ method of the class has been re-written to have the following syntax:

__new., (els, filename, dtype=uint8, mode='r+', offset=0, shape=None, order=0)

filename The file name to be used as the array data buffer dtype A data-type object used to interpret the file contents (including byteorder).

mode The mode to open the file in. Valid modes are 'readonly' or 'r', 'copy-onwrite' or 'c', 'readwrite' or 'r+', and 'write' or 'w+'. This mode determines the WRITEABLE flag of the returned array.

offset An offset into the file to start the array data.

shape The desired shape of the array. If this is None, then the returned array will be 1-d with the number of elements determined by the file size and data type.

order Either 'C' or 'Fortran' to indicate the order that an N-D array should be interpreted. This only has an effect if the shape is greater than 2-D.

Memory-mapped-file arrays have one additional method (besides those they inherit from the ndarray): self.sync () which must be called manually by the user to ensure that any changes to the array actually get written to disk.

Example:

>>> a = memmap('newfile.dat',

dtype=float, mode='w+'

shaj

>>> a [10] =10.0

>>> a[30] = 30.0

>>> del a

>>> b = fromfileCnewfile.dat

, dtype=float)

>>> print b[10] , b[30]

10.0 30.0

>>> a = memmap('newfile.dat',

dtype=float)

>>> print a[10], a[30]

10.0 30.0

e=1000)

0 0

Post a comment