5 posts
How do you get files that must not be public into the browser? A server proxy, presigned URLs, CloudFront signed URLs, and OAC with WAF, compared along one axis: who checks access. The belief that a proxy reads the whole file was checked with a 200MB file.
An experiment in whether plain JSONL on S3 can be queried efficiently without Parquet. Range binary search, an index binary search, a B-tree and an LSM tree, built directly on a real bucket, with round trips and time measured. Querying works — but the verdict is still Parquet, for maintainability rather than speed.
From why columns are stored together, through row groups, chunks, pages and the table of contents at the tail, a walk through the inside of a Parquet file. Seven animations show min/max and Bloom filters pruning row groups and HTTP Range fetching just the needed slices.
Delivery-history JSON sitting in S3 is hard to query as it stands. Four options weighed on always-on infrastructure, the shape of the query cost, and freshness, and why a nightly Parquet conversion won.
Where should delivery history live? The trade-offs weighed across a dedicated relational DB, a serial S3 pipeline, SNS fan-out, and a Firehose route.