delta-rs
Native Rust implementation of the Delta Lake protocol with Python bindings (deltalake package). Enables reading and writing Delta Lake tables without Spark — works in Python, Rust, and through Arrow/Pandas integration. Used to interact with Delta Lake from Python data pipelines, ML workflows, and agent backends. The Python deltalake package wraps the Rust implementation for high performance.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
Cloud credential handling follows standard practices. Data encrypted at rest by object store. Delta log provides audit trail for all writes.
⚡ Reliability
Best When
You're building Python data pipelines, ML workflows, or agent services that need to read/write Delta Lake tables without the complexity of running Spark.
Avoid When
You need streaming ingestion, high-throughput ETL, or complex Delta Lake schema evolution — use Spark Delta Lake.
Use Cases
- • Read and write Delta Lake tables from Python without a Spark cluster using delta-rs' efficient Rust implementation
- • Use ACID transactions to write reliable data from Python agent pipelines to shared Delta Lake tables
- • Query Delta Lake table history (time travel) for data validation, auditing, and debugging in Python workflows
- • Build Python ML pipelines that read from and write to Delta Lake tables in S3, Azure ADLS, or GCS
- • Integrate Delta Lake data with Pandas, Polars, or PyArrow DataFrames via delta-rs' Arrow-based read interface
Not For
- • High-throughput production ETL — Spark Delta Lake is faster for large-scale writes; use delta-rs for smaller workloads
- • Delta Lake schema enforcement at write time — delta-rs enforces schema but complex schema evolution may need Spark
- • Real-time streaming into Delta Lake — use Spark Structured Streaming or Apache Kafka + Spark for streaming ingestion
Interface
Authentication
Authentication via cloud storage credentials (AWS IAM/boto3, Azure SPN/ADLS, GCS service account) configured via storage_options dict.
Pricing
Apache 2.0 license. Open source Delta Lake specification.
Agent Metadata
Known Gotchas
- ⚠ Concurrent writes to the same Delta table require transaction retry logic — ConcurrentWriteError is expected in multi-writer scenarios; implement retry with DeltaTable refresh
- ⚠ Write mode ('append', 'overwrite', 'error', 'ignore') must be chosen carefully — 'overwrite' replaces the entire table; use 'append' for incremental writes
- ⚠ storage_options must include the appropriate cloud credentials format (AWS, Azure, GCS differ) — missing credentials cause opaque permission errors
- ⚠ Reading Delta tables returns an Arrow RecordBatch — convert to pandas with .to_pandas() or use polars with .to_polars() for DataFrame workflows
- ⚠ Table optimization (delta_table.optimize()) and vacuum (delta_table.vacuum()) must be called separately from writes — maintenance is not automatic
- ⚠ Delta table version history retained by default (no automatic cleanup) — vacuum() with retention_hours must be called to free object storage space
Alternatives
Full Evaluation Report
Detailed scoring breakdown, competitive positioning, security analysis, and improvement recommendations for delta-rs.
Scores are editorial opinions as of 2026-03-06.