python���������format������,pickle — Python object serialization — Python 3.10.0 documentation

关注
pickle — Python object serialization¶

Source code: Lib/pickle.py

The pickle module implements binary protocols for serializing andde-serializing a Python object structure.“Pickling” is the processwhereby a Python object hierarchy is converted into a byte stream, and“unpickling” is the inverse operation, whereby a byte stream(from a binary file or bytes-like object) is convertedback into an object hierarchy.Pickling (and unpickling) is alternativelyknown as “serialization”, “marshalling,” 1 or “flattening”; however, toavoid confusion, the terms used here are “pickling” and “unpickling”.

Warning

The pickle module is not secure. Only unpickle data you trust.

It is possible to construct malicious pickle data which will executearbitrary code during unpickling. Never unpickle data that could have comefrom an untrusted source, or that could have been tampered with.

Consider signing data with hmac if you need to ensure that it has notbeen tampered with.

Safer serialization formats such as json may be more appropriate ifyou are processing untrusted data. See Comparison with json.

Relationship to other Python modules¶Comparison with marshal¶

Python has a more primitive serialization module called marshal, but ingeneral pickle should always be the preferred way to serialize Pythonobjects.marshal exists primarily to support Python’s .pycfiles.

The pickle module differs from marshal in several significant ways:

The pickle module keeps track of the objects it has already serialized,so that later references to the same object won’t be serialized again.marshal doesn’t do this.

This has implications both for recursive objects and object sharing.Recursiveobjects are objects that contain references to themselves.These are nothandled by marshal, and in fact, attempting to marshal recursive objects willcrash your Python interpreter.Object sharing happens when there are multiplereferences to the same object in different places in the object hierarchy beingserialized.pickle stores such objects only once, and ensures that allother references point to the master copy.Shared objects remain shared, whichcan be very important for mutable objects.

marshal cannot be used to serialize user-defined classes and theirinstances.pickle can save and restore class instances transparently,however the class definition must be importable and live in the same module aswhen the object was stored.

The marshal serialization format is not guaranteed to be portableacross Python versions.Because its primary job in life is to support.pyc files, the Python implementers reserve the right to change theserialization format in non-backwards compatible ways should the need arise.The pickle serialization format is guaranteed to be backwards compatibleacross Python releases provided a compatible pickle protocol is chosen andpickling and unpickling code deals with Python 2 to Python 3 type differencesif your data is crossing that unique breaking change language boundary.

Comparison with json¶

There are fundamental differences between the pickle protocols andJSON (JavaScript Object Notation):

JSON is a text serialization format (it outputs unicode text, althoughmost of the time it is then encoded to utf-8), while pickle isa binary serialization format;

JSON is human-readable, while pickle is not;

JSON is interoperable and widely used outside of the Python ecosystem,while pickle is Python-specific;

JSON, by default, can only represent a subset of the Python built-intypes, and no custom classes; pickle can represent an extremely largenumber of Python types (many of them automatically, by clever usageof Python’s introspection facilities; complex cases can be tackled byimplementing specific object APIs);

Unlike pickle, deserializing untrusted JSON does not in itself create anarbitrary code execution vulnerability.

See also

The json module: a standard library module allowing JSONserialization and deserialization.

Data stream format¶

The data format used by pickle is Python-specific.This has theadvantage that there are no restrictions imposed by external standards such asJSON or XDR (which can’t represent pointer sharing); however it means thatnon-Python programs may not be able to reconstruct pickled Python objects.

By default, the pickle data format uses a relatively compact binaryrepresentation.If you need optimal size characteristics, you can efficientlycompress pickled data.

The module pickletools contains tools for analyzing data streamsgenerated by pickle.pickletools source code has extensivecomments about opcodes used by pickle protocols.

There are currently 6 different protocols which can be used for pickling.The higher the protocol used, the more recent the version of Python neededto read the pickle produced.

Protocol version 0 is the original “human-readable” protocol and isbackwards compatible with earlier versions of Python.

Protocol version 1 is an old binary format which is also compatible withearlier versions of Python.

Protocol version 2 was introduced in Python 2.3.It provides much moreefficient pickling of new-style classes.Refer to PEP 307 forinformation about improvements brought by protocol 2.

Protocol version 3 was added in Python 3.0.It has explicit support forbytes objects and cannot be unpickled by Python 2.x.This wasthe default protocol in Python 3.0–3.7.

Protocol version 4 was added in Python 3.4.It adds support for very largeobjects, pickling more kinds of objects, and some data formatoptimizations.It is the default protocol starting with Python 3.8.Refer to PEP 3154 for information about improvements brought byprotocol 4.

Protocol version 5 was added in Python 3.8.It adds support for out-of-banddata and speedup for in-band data.Refer to PEP 574 for information aboutimprovements brought by protocol 5.

Note

Serialization is a more primitive notion than persistence; althoughpickle reads and writes file objects, it does not handle the issue ofnaming persistent objects, nor the (even more complicated) issue of concurrentaccess to persistent objects.The pickle module can transform a complexobject into a byte stream and it can transform the byte stream into an objectwith the same internal structure.Perhaps the most obvious thing to do withthese byte streams is to write them onto a file, but it is also conceivable tosend them across a network or store them in a database.The shelvemodule provides a simple interface to pickle and unpickle objects onDBM-style database files.

Module Interface¶

To serialize an object hierarchy, you simply call the dumps() function.Similarly, to de-serialize a data stream, you call the loads() function.However, if you want more control over serialization and de-serialization,you can create a Pickler or an Unpickler object, respectively.

The pickle module provides the following constants:

pickle.HIGHEST_PROTOCOL¶

An integer, the highest protocol versionavailable.This value can be passed as a protocol value to functionsdump() and dumps() as well as the Picklerconstructor.

pickle.DEFAULT_PROTOCOL¶

An integer, the default protocol version usedfor pickling.May be less than HIGHEST_PROTOCOL.Currently thedefault protocol is 4, first introduced in Python 3.4 and incompatiblewith previous versions.

Changed in version 3.0: The default protocol is 3.

Changed in version 3.8: The default protocol is 4.

The pickle module provides the following functions to make the picklingprocess more convenient:

pickle.dump(obj, file, protocol=None, *, fix_imports=True, buffer_callback=None)¶

Write the pickled representation of the object obj to the openfile object file.This is equivalent toPickler(file, protocol).dump(obj).

Arguments file, protocol, fix_imports and buffer_callback havethe same meaning as in the Pickler constructor.

Changed in version 3.8: The buffer_callback argument was added.

pickle.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)¶

Return the pickled representation of the object obj as a bytes object,instead of writing it to a file.

Arguments protocol, fix_imports and buffer_callback have the samemeaning as in the Pickler constructor.

Changed in version 3.8: The buffer_callback argument was added.

pickle.load(file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)¶

Read the pickled representation of an object from the open file objectfile and return the reconstituted object hierarchy specified therein.This is equivalent to Unpickler(file).load().

The protocol version of the pickle is detected automatically, so noprotocol argument is needed.Bytes past the pickled representationof the object are ignored.

Arguments file, fix_imports, encoding, errors, strict and buffershave the same meaning as in the Unpickler constructor.

Changed in version 3.8: The buffers argument was added.

pickle.loads(data, /, *, fix_imports=True, encoding="ASCII", errors="strict", buffers=None)¶

Return the reconstituted object hierarchy of the pickled representationdata of an object. data must be a bytes-like object.

The protocol version of the pickle is detected automatically, so noprotocol argument is needed.Bytes past the pickled representationof the object are ignored.

Arguments file, fix_imports, encoding, errors, strict and buffershave the same meaning as in the Unpickler constructor.

Changed in version 3.8: The buffers argument was added.

The pickle module defines three exceptions:

exception pickle.PickleError¶

Common base class for the other pickling exceptions.It inheritsException.

exception pickle.PicklingError¶

Error raised when an unpicklable object is encountered by Pickler.It inherits PickleError.

Refer to What can be pickled and unpickled? to learn what kinds of objects can bepickled.

exception pickle.UnpicklingError¶

Error raised when there is a problem unpickling an object, such as a datacorruption or a security violation.It inherits PickleError.

Note that other exceptions may also be raised during unpickling, including(but not necessarily limited to) AttributeError, EOFError, ImportError, andIndexError.

The pickle module exports three classes, Pickler,Unpickler and PickleBuffer:

class pickle.Pickler(file, protocol=None, *, fix_imports=True, buffer_callback=None)¶

This takes a binary file for writing a pickle data stream.

The optional protocol argument, an integer, tells the pickler to usethe given protocol; supported protocols are 0 to HIGHEST_PROTOCOL.If not specified, the default is DEFAULT_PROTOCOL.If a negativenumber is specified, HIGHEST_PROTOCOL is selected.

The file argument must have a write() method that accepts a single bytesargument.It can thus be an on-disk file opened for binary writing, anio.BytesIO instance, or any other custom object that meets thisinterface.

If fix_imports is true and protocol is less than 3, pickle will try tomap the new Python 3 names to the old module names used in Python 2, sothat the pickle data stream is readable with Python 2.

If buffer_callback is None (the default), buffer views areserialized into file as part of the pickle stream.

If buffer_callback is not None, then it can be called any numberof times with a buffer view.If the callback returns a false value(such as None), the given buffer is out-of-band;otherwise the buffer is serialized in-band, i.e. inside the pickle stream.

It is an error if buffer_callback is not None and protocol isNone or smaller than 5.

Changed in version 3.8: The buffer_callback argument was added.

dump(obj)¶

Write the pickled representation of obj to the open file object given inthe constructor.

persistent_id(obj)¶

Do nothing by default.This exists so a subclass can override it.

If persistent_id() returns None, obj is pickled as usual.Anyother value causes Pickler to emit the returned value as apersistent ID for obj.The meaning of this persistent ID should bedefined by Unpickler.persistent_load().Note that the valuereturned by persistent_id() cannot itself have a persistent ID.

See Persistence of External Objects for details and examples of uses.

dispatch_table¶

A pickler object’s dispatch table is a registry of reductionfunctions of the kind which can be declared usingcopyreg.pickle().It is a mapping whose keys are classesand whose values are reduction functions.A reduction functiontakes a single argument of the associated class and shouldconform to the same interface as a __reduce__()method.

By default, a pickler object will not have adispatch_table attribute, and it will instead use theglobal dispatch table managed by the copyreg module.However, to customize the pickling for a specific pickler objectone can set the dispatch_table attribute to a dict-likeobject.Alternatively, if a subclass of Pickler has adispatch_table attribute then this will be used as thedefault dispatch table for instances of that class.

See Dispatch Tables for usage examples.

New in version 3.3.

reducer_override(self, obj)¶

Special reducer that can be defined in Pickler subclasses. Thismethod has priority over any reducer in the dispatch_table.Itshould conform to the same interface as a __reduce__() method, andcan optionally return NotImplemented to fallback ondispatch_table-registered reducers to pickle obj.

For a detailed example, see Custom Reduction for Types, Functions, and Other Objects.

New in version 3.8.

fast¶

Deprecated. Enable fast mode if set to a true value.The fast modedisables the usage of memo, therefore speeding the pickling process by notgenerating superfluous PUT opcodes.It should not be used withself-referential objects, doing otherwise will cause Pickler torecurse infinitely.

Use pickletools.optimize() if you need more compact pickles.

class pickle.Unpickler(file, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)¶

This takes a binary file for reading a pickle data stream.

The protocol version of the pickle is detected automatically, so noprotocol argument is needed.

The argument file must have three methods, a read() method that takes aninteger argument, a readinto() method that takes a buffer argumentand a readline() method that requires no arguments, as in theio.BufferedIOBase interface.Thus file can be an on-disk fileopened for binary reading, an io.BytesIO object, or any othercustom object that meets this interface.

The optional arguments fix_imports, encoding and errors are usedto control compatibility support for pickle stream generated by Python 2.If fix_imports is true, pickle will try to map the old Python 2 namesto the new names used in Python 3.The encoding and errors tellpickle how to decode 8-bit string instances pickled by Python 2;these default to ‘ASCII’ and ‘strict’, respectively.The encoding canbe ‘bytes’ to read these 8-bit string instances as bytes objects.Using encoding='latin1' is required for unpickling NumPy arrays andinstances of datetime, date andtime pickled by Python 2.

If buffers is None (the default), then all data necessary fordeserialization must be contained in the pickle stream.This meansthat the buffer_callback argument was None when a Picklerwas instantiated (or when dump() or dumps() was called).

If buffers is not None, it should be an iterable of buffer-enabledobjects that is consumed each time the pickle stream referencesan out-of-band buffer view.Such buffers have beengiven in order to the buffer_callback of a Pickler object.

Changed in version 3.8: The buffers argument was added.

load()¶

Read the pickled representation of an object from the open file objectgiven in the constructor, and return the reconstituted object hierarchyspecified therein.Bytes past the pickled representation of the objectare ignored.

persistent_load(pid)¶

Raise an UnpicklingError by default.

If defined, persistent_load() should return the object specified bythe persistent ID pid.If an invalid persistent ID is encountered, anUnpicklingError should be raised.

See Persistence of External Objects for details and examples of uses.

find_class(module, name)¶

Import module if necessary and return the object called name from it,where the module and name arguments are str objects.Note,unlike its name suggests, find_class() is also used for findingfunctions.

Subclasses may override this to gain control over what type of objects andhow they can be loaded, potentially reducing security risks. Refer toRestricting Globals for details.

Raises an auditing event pickle.find_class with arguments module, name.

class pickle.PickleBuffer(buffer)¶

A wrapper for a buffer representing picklable data.buffer must be abuffer-providing object, such as abytes-like object or a N-dimensional array.

PickleBuffer is itself a buffer provider, therefore it ispossible to pass it to other APIs expecting a buffer-providing object,such as memoryview.

PickleBuffer objects can only be serialized using pickleprotocol 5 or higher.They are eligible forout-of-band serialization.

New in version 3.8.

raw()¶

Return a memoryview of the memory area underlying this buffer.The returned object is a one-dimensional, C-contiguous memoryviewwith format B (unsigned bytes).BufferError is raised ifthe buffer is neither C- nor Fortran-contiguous.

release()¶

Release the underlying buffer exposed by the PickleBuffer object.

What can be pickled and unpickled?¶

The following types can be pickled:

None, True, and False

integers, floating point numbers, complex numbers

strings, bytes, bytearrays

tuples, lists, sets, and dictionaries containing only picklable objects

functions defined at the top level of a module (using def, notlambda)

built-in functions defined at the top level of a module

classes that are defined at the top level of a module

instances of such classes whose __dict__ or the result ofcalling __getstate__() is picklable(see section Pickling Class Instances fordetails).

Attempts to pickle unpicklable objects will raise the PicklingErrorexception; when this happens, an unspecified number of bytes may have alreadybeen written to the underlying file.Trying to pickle a highly recursive datastructure may exceed the maximum recursion depth, a RecursionError will beraised in this case.You can carefully raise this limit withsys.setrecursionlimit().

Note that functions (built-in and user-defined) are pickled by “fully qualified”name reference, not by value. 2This means that only the function name ispickled, along with the name of the module the function is defined in.Neitherthe function’s code, nor any of its function attributes are pickled.Thus thedefining module must be importable in the unpickling environment, and the modulemust contain the named object, otherwise an exception will be raised. 3

Similarly, classes are pickled by named reference, so the same restrictions inthe unpickling environment apply.Note that none of the class’s code or data ispickled, so in the following example the class attribute attr is notrestored in the unpickling environment:

class Foo:attr = 'A class attribute'picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined inthe top level of a module.

Similarly, when class instances are pickled, their class’s code and data are notpickled along with them.Only the instance data are pickled.This is done onpurpose, so you can fix bugs in a class or add methods to the class and stillload objects that were created with an earlier version of the class.If youplan to have long-lived objects that will see many versions of a class, it maybe worthwhile to put a version number in the objects so that suitableconversions can be made by the class’s __setstate__() method.

Pickling Class Instances¶

In this section, we describe the general mechanisms available to you to define,customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable.Bydefault, pickle will retrieve the class and the attributes of an instance viaintrospection. When a class instance is unpickled, its __init__() methodis usually not invoked.The default behaviour first creates an uninitializedinstance and then restores the saved attributes.The following code shows animplementation of this behaviour:

def save(obj):return (obj.__class__, obj.__dict__)def load(cls, attributes):obj = cls.__new__(cls)obj.__dict__.update(attributes)return obj

Classes can alter the default behaviour by providing one or several specialmethods:

object.__getnewargs_ex__()¶

In protocols 2 and newer, classes that implements the__getnewargs_ex__() method can dictate the values passed to the__new__() method upon unpickling.The method must return a pair(args, kwargs) where args is a tuple of positional argumentsand kwargs a dictionary of named arguments for constructing theobject.Those will be passed to the __new__() method uponunpickling.

You should implement this method if the __new__() method of yourclass requires keyword-only arguments.Otherwise, it is recommended forcompatibility to implement __getnewargs__().

Changed in version 3.6: __getnewargs_ex__() is now used in protocols 2 and 3.

object.__getnewargs__()¶

This method serves a similar purpose as __getnewargs_ex__(), butsupports only positional arguments.It must return a tuple of argumentsargs which will be passed to the __new__() method upon unpickling.

__getnewargs__() will not be called if __getnewargs_ex__() isdefined.

Changed in version 3.6: Before Python 3.6, __getnewargs__() was called instead of__getnewargs_ex__() in protocols 2 and 3.

object.__getstate__()¶

Classes can further influence how their instances are pickled; if the classdefines the method __getstate__(), it is called and the returned objectis pickled as the contents for the instance, instead of the contents of theinstance’s dictionary.If the __getstate__() method is absent, theinstance’s __dict__ is pickled as usual.

object.__setstate__(state)¶

Upon unpickling, if the class defines __setstate__(), it is called withthe unpickled state.In that case, there is no requirement for the stateobject to be a dictionary.Otherwise, the pickled state must be a dictionaryand its items are assigned to the new instance’s dictionary.

Note

If __getstate__() returns a false value, the __setstate__()method will not be called upon unpickling.

Refer to the section Handling Stateful Objects for more information about how to usethe methods __getstate__() and __setstate__().

Note

At unpickling time, some methods like __getattr__(),__getattribute__(), or __setattr__() may be called upon theinstance.In case those methods rely on some internal invariant beingtrue, the type should implement __new__() to establish such aninvariant, as __init__() is not called when unpickling aninstance.

As we shall see, pickle does not use directly the methods described above.Infact, these methods are part of the copy protocol which implements the__reduce__() special method.The copy protocol provides a unifiedinterface for retrieving the data necessary for pickling and copyingobjects. 4

Although powerful, implementing __reduce__() directly in your classes iserror prone.For this reason, class designers should use the high-levelinterface (i.e., __getnewargs_ex__(), __getstate__() and__setstate__()) whenever possible.We will show, however, cases whereusing __reduce__() is the only option or leads to more efficient picklingor both.

object.__reduce__()¶

The interface is currently defined as follows.The __reduce__() methodtakes no argument and shall return either a string or preferably a tuple (thereturned object is often referred to as the “reduce value”).

If a string is returned, the string should be interpreted as the name of aglobal variable.It should be the object’s local name relative to itsmodule; the pickle module searches the module namespace to determine theobject’s module.This behaviour is typically useful for singletons.

When a tuple is returned, it must be between two and six items long.Optional items can either be omitted, or None can be provided as theirvalue.The semantics of each item are in order:

A callable object that will be called to create the initial version of theobject.

A tuple of arguments for the callable object.An empty tuple must be givenif the callable does not accept any argument.

Optionally, the object’s state, which will be passed to the object’s__setstate__() method as previously described.If the object has nosuch method then, the value must be a dictionary and it will be added tothe object’s __dict__ attribute.

Optionally, an iterator (and not a sequence) yielding successive items.These items will be appended to the object either usingobj.append(item) or, in batch, using obj.extend(list_of_items).This is primarily used for list subclasses, but may be used by otherclasses as long as they have append() and extend() methods withthe appropriate signature.(Whether append() or extend() isused depends on which pickle protocol version is used as well as the numberof items to append, so both must be supported.)

Optionally, an iterator (not a sequence) yielding successive key-valuepairs.These items will be stored to the object using obj[key] =value.This is primarily used for dictionary subclasses, but may be usedby other classes as long as they implement __setitem__().

Optionally, a callable with a (obj, state) signature. Thiscallable allows the user to programmatically control the state-updatingbehavior of a specific object, instead of using obj’s static__setstate__() method. If not None, this callable will havepriority over obj’s __setstate__().

New in version 3.8: The optional sixth tuple item, (obj, state), was added.

object.__reduce_ex__(protocol)¶

Alternatively, a __reduce_ex__() method may be defined.The onlydifference is this method should take a single integer argument, the protocolversion.When defined, pickle will prefer it over the __reduce__()method.In addition, __reduce__() automatically becomes a synonym forthe extended version.The main use for this method is to providebackwards-compatible reduce values for older Python releases.

Persistence of External Objects¶

For the benefit of object persistence, the pickle module supports thenotion of a reference to an object outside the pickled data stream.Suchobjects are referenced by a persistent ID, which should be either a string ofalphanumeric characters (for protocol 0) 5 or just an arbitrary object (forany newer protocol).

The resolution of such persistent IDs is not defined by the picklemodule; it will delegate this resolution to the user-defined methods on thepickler and unpickler, persistent_id() andpersistent_load() respectively.

To pickle objects that have an external persistent ID, the pickler must have acustom persistent_id() method that takes an object as anargument and returns either None or the persistent ID for that object.When None is returned, the pickler simply pickles the object as normal.When a persistent ID string is returned, the pickler will pickle that object,along with a marker so that the unpickler will recognize it as a persistent ID.

To unpickle external objects, the unpickler must have a custompersistent_load() method that takes a persistent ID object andreturns the referenced object.

Here is a comprehensive example presenting how persistent ID can be used topickle external objects by reference.

# Simple example presenting how persistent ID can be used to pickle# external objects by reference.import pickleimport sqlite3from collections import namedtuple# Simple class representing a record in our database.MemoRecord = namedtuple("MemoRecord", "key, task")class DBPickler(pickle.Pickler):def persistent_id(self, obj):# Instead of pickling MemoRecord as a regular class instance, we emit a# persistent ID.if isinstance(obj, MemoRecord):# Here, our persistent ID is simply a tuple, containing a tag and a# key, which refers to a specific record in the database.return ("MemoRecord", obj.key)else:# If obj does not have a persistent ID, return None. This means obj# needs to be pickled as usual.return Noneclass DBUnpickler(pickle.Unpickler):def __init__(self, file, connection):super().__init__(file)self.connection = connectiondef persistent_load(self, pid):# This method is invoked whenever a persistent ID is encountered.# Here, pid is the tuple returned by DBPickler.cursor = self.connection.cursor()type_tag, key_id = pidif type_tag == "MemoRecord":# Fetch the referenced record from the database and return it.cursor.execute("SELECT * FROM memos WHERE key=?", (str(key_id),))key, task = cursor.fetchone()return MemoRecord(key, task)else:# Always raises an error if you cannot return the correct object.# Otherwise, the unpickler will think None is the object referenced# by the persistent ID.raise pickle.UnpicklingError("unsupported persistent object")def main():import ioimport pprint# Initialize and populate our database.conn = sqlite3.connect(":memory:")cursor = conn.cursor()cursor.execute("CREATE TABLE memos(key INTEGER PRIMARY KEY, task TEXT)")tasks = ('give food to fish','prepare group meeting','fight with a zebra',)for task in tasks:cursor.execute("INSERT INTO memos VALUES(NULL, ?)", (task,))# Fetch the records to be pickled.cursor.execute("SELECT * FROM memos")memos = [MemoRecord(key, task) for key, task in cursor]# Save the records using our custom DBPickler.file = io.BytesIO()DBPickler(file).dump(memos)print("Pickled records:")pprint.pprint(memos)# Update a record, just for good measure.cursor.execute("UPDATE memos SET task='learn italian' WHERE key=1")# Load the records from the pickle data stream.file.seek(0)memos = DBUnpickler(file, conn).load()print("Unpickled records:")pprint.pprint(memos)if __name__ == '__main__':main()Dispatch Tables¶

If one wants to customize pickling of some classes without disturbingany other code which depends on pickling, then one can create apickler with a private dispatch table.

The global dispatch table managed by the copyreg module isavailable as copyreg.dispatch_table.Therefore, one maychoose to use a modified copy of copyreg.dispatch_table as aprivate dispatch table.

For example

f = io.BytesIO()p = pickle.Pickler(f)p.dispatch_table = copyreg.dispatch_table.copy()p.dispatch_table[SomeClass] = reduce_SomeClass

creates an instance of pickle.Pickler with a private dispatchtable which handles the SomeClass class specially.Alternatively,the code

class MyPickler(pickle.Pickler):dispatch_table = copyreg.dispatch_table.copy()dispatch_table[SomeClass] = reduce_SomeClassf = io.BytesIO()p = MyPickler(f)

does the same, but all instances of MyPickler will by defaultshare the same dispatch table.The equivalent code using thecopyreg module is

copyreg.pickle(SomeClass, reduce_SomeClass)f = io.BytesIO()p = pickle.Pickler(f)Handling Stateful Objects¶

Here’s an example that shows how to modify pickling behavior for a class.The TextReader class opens a text file, and returns the line number andline contents each time its readline() method is called. If aTextReader instance is pickled, all attributes except the file objectmember are saved. When the instance is unpickled, the file is reopened, andreading resumes from the last location. The __setstate__() and__getstate__() methods are used to implement this behavior.

class TextReader:"""Print and number lines in a text file."""def __init__(self, filename):self.filename = filenameself.file = open(filename)self.lineno = 0def readline(self):self.lineno += 1line = self.file.readline()if not line:return Noneif line.endswith('\n'):line = line[:-1]return "%i: %s" % (self.lineno, line)def __getstate__(self):# Copy the object's state from self.__dict__ which contains# all our instance attributes. Always use the dict.copy()# method to avoid modifying the original state.state = self.__dict__.copy()# Remove the unpicklable entries.del state['file']return statedef __setstate__(self, state):# Restore instance attributes (i.e., filename and lineno).self.__dict__.update(state)# Restore the previously opened file's state. To do so, we need to# reopen it and read from it until the line count is restored.file = open(self.filename)for _ in range(self.lineno):file.readline()# Finally, save the file.self.file = file

A sample usage might be something like this:

>>> reader = TextReader("hello.txt")>>> reader.readline()'1: Hello world!'>>> reader.readline()'2: I am line number two.'>>> new_reader = pickle.loads(pickle.dumps(reader))>>> new_reader.readline()'3: Goodbye!'Custom Reduction for Types, Functions, and Other Objects¶

New in version 3.8.

Sometimes, dispatch_table may not be flexible enough.In particular we may want to customize pickling based on another criterionthan the object’s type, or we may want to customize the pickling offunctions and classes.

For those cases, it is possible to subclass from the Pickler class andimplement a reducer_override() method. This method can return anarbitrary reduction tuple (see __reduce__()). It can alternatively returnNotImplemented to fallback to the traditional behavior.

If both the dispatch_table andreducer_override() are defined, thenreducer_override() method takes priority.

Note

For performance reasons, reducer_override() may not becalled for the following objects: None, True, False, andexact instances of int, float, bytes,str, dict, set, frozenset, listand tuple.

Here is a simple example where we allow pickling and reconstructinga given class:

import ioimport pickleclass MyClass:my_attribute = 1class MyPickler(pickle.Pickler):def reducer_override(self, obj):"""Custom reducer for MyClass."""if getattr(obj, "__name__", None) == "MyClass":return type, (obj.__name__, obj.__bases__,{'my_attribute': obj.my_attribute})else:# For any other object, fallback to usual reductionreturn NotImplementedf = io.BytesIO()p = MyPickler(f)p.dump(MyClass)del MyClassunpickled_class = pickle.loads(f.getvalue())assert isinstance(unpickled_class, type)assert unpickled_class.__name__ == "MyClass"assert unpickled_class.my_attribute == 1Out-of-band Buffers¶

New in version 3.8.

In some contexts, the pickle module is used to transfer massive amountsof data.Therefore, it can be important to minimize the number of memorycopies, to preserve performance and resource consumption.However, normaloperation of the pickle module, as it transforms a graph-like structureof objects into a sequential stream of bytes, intrinsically involves copyingdata to and from the pickle stream.

This constraint can be eschewed if both the provider (the implementationof the object types to be transferred) and the consumer (the implementationof the communications system) support the out-of-band transfer facilitiesprovided by pickle protocol 5 and higher.

Provider API¶

The large data objects to be pickled must implement a __reduce_ex__()method specialized for protocol 5 and higher, which returns aPickleBuffer instance (instead of e.g. a bytes object)for any large data.

A PickleBuffer object signals that the underlying buffer iseligible for out-of-band data transfer.Those objects remain compatiblewith normal usage of the pickle module.However, consumers can alsoopt-in to tell pickle that they will handle those buffers bythemselves.

Consumer API¶

A communications system can enable custom handling of the PickleBufferobjects generated when serializing an object graph.

On the sending side, it needs to pass a buffer_callback argument toPickler (or to the dump() or dumps() function), whichwill be called with each PickleBuffer generated while picklingthe object graph.Buffers accumulated by the buffer_callback will notsee their data copied into the pickle stream, only a cheap marker will beinserted.

On the receiving side, it needs to pass a buffers argument toUnpickler (or to the load() or loads() function),which is an iterable of the buffers which were passed to buffer_callback.That iterable should produce buffers in the same order as they were passedto buffer_callback.Those buffers will provide the data expected by thereconstructors of the objects whose pickling produced the originalPickleBuffer objects.

Between the sending side and the receiving side, the communications systemis free to implement its own transfer mechanism for out-of-band buffers.Potential optimizations include the use of shared memory or datatype-dependentcompression.

Example¶

Here is a trivial example where we implement a bytearray subclassable to participate in out-of-band buffer pickling:

class ZeroCopyByteArray(bytearray):def __reduce_ex__(self, protocol):if protocol >= 5:return type(self)._reconstruct, (PickleBuffer(self),), Noneelse:# PickleBuffer is forbidden with pickle protocols >> import pickle>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")hello world0

In this example, the unpickler imports the os.system() function and thenapply the string argument “echo hello world”.Although this example isinoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizingUnpickler.find_class().Unlike its name suggests,Unpickler.find_class() is called whenever a global (i.e., a class ora function) is requested.Thus it is possible to either completely forbidglobals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from thebuiltins module to be loaded:

import builtinsimport ioimport picklesafe_builtins = {'range','complex','set','frozenset','slice',}class RestrictedUnpickler(pickle.Unpickler):def find_class(self, module, name):# Only allow safe classes from builtins.if module == "builtins" and name in safe_builtins:return getattr(builtins, name)# Forbid everything else.raise pickle.UnpicklingError("global '%s.%s' is forbidden" % (module, name))def restricted_loads(s):"""Helper function analogous to pickle.loads()."""return RestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working has intended:

>>> restricted_loads(pickle.dumps([1, 2, range(15)]))[1, 2, range(0, 15)]>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")Traceback (most recent call last):...pickle.UnpicklingError: global 'os.system' is forbidden>>> restricted_loads(b'cbuiltins\neval\n'...b'(S\'getattr(__import__("os"), "system")'...b'("echo hello world")\'\ntR.')Traceback (most recent call last):...pickle.UnpicklingError: global 'builtins.eval' is forbidden

As our examples shows, you have to be careful with what you allow to beunpickled.Therefore if security is a concern, you may want to consideralternatives such as the marshalling API in xmlrpc.client orthird-party solutions.

Performance¶

Recent versions of the pickle protocol (from protocol 2 and upwards) featureefficient binary encodings for several common features and built-in types.Also, the pickle module has a transparent optimizer written in C.

Examples¶

For the simplest code, use the dump() and load() functions.

import pickle# An arbitrary collection of objects supported by pickle.data = {'a': [1, 2.0, 3, 4+6j],'b': ("character string", b"byte string"),'c': {None, True, False}}with open('data.pickle', 'wb') as f:# Pickle the 'data' dictionary using the highest protocol available.pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)

The following example reads the resulting pickled data.

import picklewith open('data.pickle', 'rb') as f:# The protocol version used is detected automatically, so we do not# have to specify it.data = pickle.load(f)

See also

Module copyreg

Pickle interface constructor registration for extension types.

Module pickletools

Tools for working with and analyzing pickled data.

Module shelve

Indexed databases of objects; uses pickle.

Module copy

Shallow and deep object copying.

Module marshal

High-performance serialization of built-in types.

Footnotes

1

Don’t confuse this with the marshal module

2

This is why lambda functions cannot be pickled:alllambda functions share the same name:.

3

The exception raised will likely be an ImportError or anAttributeError but it could be something else.

4

The copy module uses this protocol for shallow and deep copyingoperations.

5

The limitation on alphanumeric characters is due to the factthe persistent IDs, in protocol 0, are delimited by the newlinecharacter.Therefore if any kind of newline characters occurs inpersistent IDs, the resulting pickle will become unreadable.