cloudpickle
Extended Python pickling for distributed computing — serializes Python objects that standard pickle cannot, including lambdas, closures, locally defined functions, and class definitions. cloudpickle features: cloudpickle.dumps()/loads() API compatible with pickle, serializes lambda functions (not picklable with standard pickle), closures capturing local variables, inner functions, dynamically defined classes, generator functions, functools.partial with lambda args, and is used internally by Ray, Dask, PySpark, and joblib. Essential for sending Python callables across process boundaries.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
CRITICAL SECURITY WARNING: cloudpickle deserialization executes arbitrary Python code — equivalent to remote code execution if data source is untrusted. Never deserialize cloudpickle/pickle data from external sources, user input, or any untrusted channel. Use HMAC signing if cloudpickle data must traverse network boundaries. Consider alternatives (JSON + function registry, protobuf) for security-sensitive scenarios. Serialized closures may capture and transmit sensitive data (API keys, passwords) captured in outer scope.
⚡ Reliability
Best When
Sending Python callables (lambdas, closures, local functions) to worker processes in distributed computing frameworks — cloudpickle enables Ray, Dask, and Spark to serialize Python functions that standard pickle cannot handle.
Avoid When
You need persistent storage (use JSON/protobuf), cross-language communication, or are serializing data from untrusted sources (pickle is a security risk).
Use Cases
- • Agent parallel lambda execution — import cloudpickle; import concurrent.futures; fn = lambda x: x * factor; pickled = cloudpickle.dumps(fn); executor.submit(cloudpickle.loads(pickled), value) — serialize lambda for worker process; agent parallel processing sends lambda transform to workers; standard pickle raises AttributeError on lambda
- • Agent Ray remote functions — import ray; @ray.remote; def agent_task(data): return process(data) — Ray uses cloudpickle internally; agent distributes closures capturing local state; Ray actors serialize entire class definition with cloudpickle; no special serialization code needed
- • Agent Dask computation — import dask; result = dask.delayed(lambda df: df.groupby('agent').mean())(large_df) — Dask uses cloudpickle; agent distributed DataFrame processing sends lambda transformations to workers; dask.compute() executes lambda in worker process
- • Agent closure serialization — def make_agent(config): threshold = config['threshold']; def agent_fn(data): return data > threshold; return agent_fn; fn = make_agent({'threshold': 0.9}); bytes = cloudpickle.dumps(fn) — closure captures threshold from outer scope; cloudpickle serializes closure with captured variables; standard pickle fails with PicklingError
- • Agent cross-process class transfer — import cloudpickle; class DynamicAgent: pass; DynamicAgent.score = compute_score; serialized = cloudpickle.dumps(DynamicAgent); sent_class = cloudpickle.loads(serialized) — dynamically modified class transferred between processes; agent metaclass-generated classes work with cloudpickle; class definition included in serialized bytes
Not For
- • Long-term storage — cloudpickle serialization is Python-version dependent; do not use for persistent storage; use JSON, msgpack, or protocol buffers
- • Security boundaries — pickle/cloudpickle deserialization can execute arbitrary code; never deserialize cloudpickle data from untrusted sources
- • Cross-language serialization — cloudpickle is Python-only; for cross-language use protobuf, JSON, or Apache Arrow
Interface
Authentication
No auth — local serialization library. CRITICAL: pickle/cloudpickle deserialization can execute arbitrary code — only deserialize from trusted sources.
Pricing
cloudpickle is BSD licensed. Free for all use.
Agent Metadata
Known Gotchas
- ⚠ CRITICAL: Never deserialize cloudpickle data from untrusted sources — cloudpickle.loads() executes arbitrary Python code; attacker-controlled pickled bytes can execute os.system('rm -rf /'); agent systems receiving serialized functions from external sources must use code signing or alternative serialization; only deserialize from trusted internal sources
- ⚠ Serialized bytes are not stable across Python versions — cloudpickle.dumps(fn) on Python 3.10 may not deserialize on Python 3.11; agent distributed systems must use same Python version on all workers; do not store cloudpickle bytes to disk for later use across upgrades; for stable storage use JSON or protobuf
- ⚠ C extensions and file handles cannot be serialized — cloudpickle.dumps(open('/tmp/file')) raises PicklingError; cloudpickle cannot serialize objects holding OS resources (file descriptors, sockets, locks); agent functions capturing file handles in closures must close before serialization or use path strings instead of file objects
- ⚠ Module availability required on target — cloudpickle.loads() on worker requires all imported modules available; agent code: lambda x: numpy.mean(x) serialized and sent to worker without numpy installed raises ImportError on load; worker environment must match sender environment for imports; use requirements.txt or Docker to ensure consistency
- ⚠ Class instances serialized as class definition + state — cloudpickle serializes class definition with instance; if class changes between serialize and deserialize, deserialization may fail or produce incorrect objects; agent rolling deployments where workers have old class definitions and sender has new class fail to deserialize; coordinate deployment
- ⚠ Captured variables in closures increase serialized size — def make_fn(large_df): return lambda x: large_df.loc[x]; fn = make_fn(huge_dataframe); cloudpickle.dumps(fn) includes entire huge_dataframe; agent lambda capturing large objects serializes entire object to worker; use IDs and remote data access instead of capturing large objects in closure
Alternatives
Full Evaluation Report
Detailed scoring breakdown, competitive positioning, security analysis, and improvement recommendations for cloudpickle.
Scores are editorial opinions as of 2026-03-06.