3 posts
An experiment in whether plain JSONL on S3 can be queried efficiently without Parquet. Range binary search, an index binary search, a B-tree and an LSM tree, built directly on a real bucket, with round trips and time measured. Querying works — but the verdict is still Parquet, for maintainability rather than speed.
From why columns are stored together, through row groups, chunks, pages and the table of contents at the tail, a walk through the inside of a Parquet file. Seven animations show min/max and Bloom filters pruning row groups and HTTP Range fetching just the needed slices.
Delivery-history JSON sitting in S3 is hard to query as it stands. Four options weighed on always-on infrastructure, the shape of the query cost, and freshness, and why a nightly Parquet conversion won.