Curriculum & Daily Articles
- Day 0: Motivations, Technical Inspirations & Non-Goals Why build a columnar analytical database engine in C11 from scratch? Architectural influences from ClickHouse, APL/K, DuckDB, and Apache Arrow.
- Day 1: 64-Byte SIMD Alignment & Dynamic Column Buffers How modern analytical engines achieve 40-80 GB/s scans. Why 64-byte alignment prevents cache-line splits and how to construct zero-overhead C11 column buffers.
- Day 2: Zero-Copy Typed Slices & C11 _Generic Polymorphism How physical query operators read column data with 0 bytes copied. Implementing typed slices and C11 compile-time generic dispatch.
- Day 3: Zero-Copy Memory-Mapped Files (mmap) & OS Page Cache Reading multi-gigabyte column files with zero byte copying. POSIX mmap, madvise prefetching, and mmap_file_t architecture.
- Day 4: Build System, Compiler Optimizations & POSIX Micro-Benchmarking How optimization flags (-O3 -march=native) and high-resolution POSIX timing (clock_gettime) impact hardware memory bandwidth scanning throughput.
- Day 5: Column Vectors, Selection Vectors and Branchless Filtering How ClickHouse and DuckDB execute query filters without CPU branch mispredictions. Late materialization, selection vectors (sel_t), and zero-copy gather kernels.
- Day 6: High Cardinality String Dictionaries Efficient dictionary encoding for strings to reduce memory footprint and speed up equality comparisons.
- Day 7: Bitmaps & Dynamic Schema Registry Handling missing data in C without ruining SIMD pipelines or memory bandwidth. Building a dynamic schema registry for JSON ingestion.
- Day 8: SIMD APL Vector Primitives (Compress & Reduction) Write SIMD filter & compress C kernel executing sub-millisecond selection over 4 million elements.
- Day 9: Streaming Uncompressed Storage Refactoring binary serializers to create immutable column parts with pure .bin files, avoiding OS file descriptor exhaustion.
- Day 10: The Sparse Primary Index (2-Tier Query Pruning) Building a ClickHouse-style sparse index using a 2-Tier Pruner and Granule Bitmaps for extreme memory efficiency.
- Day 11: The Column Reader (Zero-Copy Data Loading) Building the Column Reader with zero-copy mmap and vectorized Granule pulling.
- Day 12: Free-Text Inverted Block Indexing & BM25 Scoring Building a native Inverted Block Index combined with BM25 Scoring in C11 to support Full-Text Search within our columnar architecture.
- Day 13: Roaring Bitset Deletion Vectors Implementing roaring bitset deletion vectors in C11 to efficiently handle DELETE operations in our immutable columnar storage engine.
- Day 14: The Table Abstraction (Ingestion Layer) Building the coordinating ingestion layer to multiplex rows into columnar files and update indexes simultaneously.
- Day 14 Bis: Table Flush & Immutable Part Serialization Implementing streaming disk flush from RAM columnar buffers to immutable .bin files and managing part metadata.
- Day 15: External Merge Sort (Disk-Spilling ORDER BY) Implementing out-of-core external merge sort with fixed memory budgets and K-way streaming min-heap merge.
- Day 16: Partitioned Grace Hash Aggregation (Disk Spilling GROUP BY) Implementing partitioned grace hash aggregation for high-cardinality GROUP BY with bounded memory.
- Day 17: The Multi-Part Table Scanner (Lazy Data Loading & Granule Pruning) Building a Volcano-model Table Scanner to stream data from disk parts and memory buffers.
- Day 18: APL / K Vectorized Physical Execution Pipeline Assembling execution pipeline executing scan, filter, gather, aggregate.
- Day 19: The MergeTree Part Compactor Merging multiple immutable on-disk parts into a single consolidated, deduplicated part.
- Day 20: Minimal SQL & Query Parser (Lexer, Recursive Descent & AST) Building an analytical SQL Lexer, recursive descent parser, and Abstract Syntax Tree (AST) in C11.
- Day 21: The Interactive REPL CLI & Query Engine Building an interactive analytical database REPL, dynamic schema inference, streaming NDJSON ingestion with yyjson, and vectorized query execution.
- Day 22: The REPL Read Path & Query Planner (Binding SQL SELECT to the Vectorized Pipeline) Building a Query Planner and Binder that translates SQL SELECT ASTs into physical operator trees for vectorized execution.
- Day 23: Can 5,000 Lines of C11 Outrun ClickHouse, DuckDB, and MonetDB? The Benchmark Showdown Benchmarking our custom C11 columnar engine against ClickHouse, DuckDB, and MonetDB across 10 million rows. How zero-abstraction SIMD matches 10 parallel CPU cores.
- Analytics & CI/CD: My Astro Blog Infrastructure How I built a custom Temporal Python CI/CD pipeline and serverless analytics with Tinybird.
- How This Blog Is Written (LLM Socratic Method) An overview of how this blog and database engine are co-authored using an AI agent as a documentation and code-review assistant.
- Fast Vim / Neovim Editing Guide for Zed Editor (C11 Edition) A comprehensive guide to editing C11 system code in Zed Editor using modern Neovim/Vim modal editing techniques, text objects, search/replace patterns, LSP shortcuts, and debugging.