SLAF vs TileDB Performance Benchmarks¶

This document provides a comprehensive performance comparison between SLAF and TileDB SOMA across bioinformatics and machine learning workflows, focusing on modern columnar storage formats.

Overview¶

Both SLAF and TileDB represent modern approaches to single-cell data storage, using Arrow-interoperable columnar formats. However, SLAF demonstrates consistent performance advantages across metadata filtering and dataloader throughput, highlighting the benefits of its optimized streaming architecture and efficient data access patterns.

Key Performance Summary¶

Conversion Performance¶

Conversion from h5ad to modern formats demonstrates SLAF's efficiency in data ingestion workflows.

Input Dataset: synthetic_50k_processed (49,955 cells × 25,000 genes, ~722MB h5ad file)

Key Advantages:

• 9.9x faster conversion from h5ad to SLAF format
• 38% smaller output size compared to TileDB SOMA
• 5.3x more memory efficient - SLAF uses only 827MB vs TileDB's 4.4GB peak memory
• Optimized chunked processing with efficient memory management

TileDB to SLAF Conversion¶

Converting from TileDB SOMA to SLAF format is straightforward and efficient:

TileDB to SLAF Performance: synthetic_50k_processed (49,955 cells × 25,000 genes)

Migration Benefits:

• Fast conversion: 50k cell dataset converts in just 2 seconds
• Linear scaling: Performance scales linearly with the number of cells
• Simple command: slaf convert data.tiledb output.slaf

Bioinformatics Benchmarks¶

Cell Filtering Performance¶

Cell filtering operations demonstrate the efficiency of modern columnar storage for metadata operations.

Average Performance: 9.4x faster

Gene Filtering Performance¶

Gene filtering operations show consistent performance advantages for SLAF's optimized Polars operations.

Average Performance: 8.6x faster

Expression Queries Performance¶

Expression queries demonstrate the efficiency of optimized sparse matrix operations, where TileDB often wins.

Average Performance:

• SLAF wins cell-centric (1.2x-4.0x),
• TileDB wins gene-centric (3.3x-10x) and submatrix (2.5x-5.0x)

Machine Learning Benchmarks¶

Raw Data Loading Performance¶

Raw data loading performance demonstrates the advantages of SLAF's optimized streaming architecture.

Performance Comparison: 47.5x faster

GPU-Ready Output Performance¶

SLAF provides pre-tokenized sequences ready for GPU training, while TileDB DataLoader only provides raw data.

Technical Implementation Comparison¶

Key Technical Advantages¶

Migration Benefits¶

Performance Improvements¶

• 9.4x faster bioinformatics operations
• 47.5x faster machine learning data loading
• Faster conversion from h5ad to SLAF format, enabling rapid migration of existing datasets
• Easy TileDB migration: 2-second conversion for 50k cell datasets with linear scaling

Developer Experience¶

• Simplified API with familiar Polars operations
• Scanpy-native lazy workflows with drop-in replacement of AnnData objects, enabling efficient lazy computation graphs for preprocessing and filtering pipelines
• Enhanced ML integration with native PyTorch support
• Built-in tokenization for foundation model training

Use Case Recommendations¶

Choose SLAF for:¶

• High-throughput machine learning workflows requiring fast data loading
• Foundation model training requiring GPU-ready tokenized sequences
• Streaming applications where continuous data flow is critical
• Memory-constrained environments where efficiency is important

Choose TileDB for:¶

• Existing TileDB infrastructure where migration costs are high
• Multi-modal data where TileDB's broader ecosystem is beneficial

Conclusion¶

SLAF provides strong performance advantages over TileDB across most benchmark categories:

• Metadata filtering: 9.4x-8.6x faster (cell and gene filtering)
• Machine learning data loading: 47.5x faster
• Expression queries: Mixed performance (SLAF wins cell-centric, TileDB wins gene-centric)

While both systems represent modern approaches to single-cell data storage, SLAF's optimized streaming architecture, enhanced ML integration, and simplified API provide significant advantages for most use cases. The benchmarks demonstrate that SLAF's performance optimizations and developer-friendly design make it the preferred choice for high-throughput bioinformatics and machine learning workflows.

For detailed migration guidance including TileDB to SLAF conversion, see Migrating to SLAF. For comprehensive benchmark results, see Bioinformatics Benchmarks and ML Benchmarks.