LiquidCache
VLDB 2026

LiquidCache

Efficient Pushdown Caching for Cloud-Native Data Analytics
Xiangpeng Hao  ·  Andrew Lamb (InfluxData)  ·  Yibo Wu  ·  Andrea Arpaci-Dusseau  ·  Remzi Arpaci-Dusseau
University of Wisconsin–Madison
Supported by InfluxData Bauplan Spiral
Takeaway
Cache a better format
than the storage format.
Object store CSV JSON Parquet MCAP Cache Liquid Liquid Liquid Liquid Compute DataFusion any engine transcode once serve every query CSV Liquid JSON Liquid Parquet Liquid MCAP Liquid

Decoupled

The archive keeps its stable format; the cache serves what hardware wants.

Essentially free

Transcoding uses idle cache CPUs and hides behind the cache-miss I/O.

2 / 16
Background · 1 / 2

Cloud-native analytics already reads remote data through a cache

Object store Parquet Cache tier Parquet the same bytes, closer Compute BI SQL ML 100–200 ms $ per GET ~1 ms Standard practice: Alluxio, Databricks disk cache, and in-house caching tiers.
3 / 16
Background · 2 / 2

LiquidCache answers in a better format
than it fetches

Object store Parquet unchanged LiquidCache Parquet Liquid transcodes on first access, keeps only the better format, evaluates filters in place Compute DataFusion any engine unchanged on miss · 100–200 ms matching rows · ~1 ms
4 / 16
Cache a better format than the storage format —
1

Why cache a different format?

2

Why is it essentially free?

5 / 16
Part 1 · Why a different format · 1 / 4

Writers pick formats for the archive

The archive needs
  • Stability over decades
  • Universal compatibility
  • Compression
  • Open governance
Today’s hardware wants
  • Cheap decoding
  • SIMD-friendly layouts
  • Random access
  • Freedom to evolve
What lands in the lake
CSV JSON Nimble in-house
6 / 16
Part 1 · Why a different format · 2 / 4

File formats are not created equal: each new generation is faster to query

CSV · JSON row-oriented text Parquet · ORC columnar Vortex · Liquid decoding-first design + column pruning + compression, statistics + SIMD encodings + random access faster to query 10× lower decoding CPU, same compression as Parquet
7 / 16
Part 1 · Why a different format · 3 / 4

Switching formats takes a decade

Option A

Re-encode the whole lake

  • petabyte-scale ETL, paid up front
  • two copies of everything
  • converts data no query reads
Option B

Ask every writer to switch

  • thousands of writers to convince
  • breaks the oldest consumer
  • churn vs. archival stability
IN PRACTICE New formats keep arriving; production data stays in years-old Parquet — held back by governance, not performance.
8 / 16
Part 1 · Why a different format · 4 / 4

Keep the archive as is —
cache the better format instead

Object store CSV JSON Parquet MCAP all unchanged Cache server Liquid our format ephemeral Compute DataFusion any engine nothing to adopt transcode serve fast Whatever the writers chose goes in; one format we control comes out. Liquid filters strings without fully decoding them; it owes no compatibility.
9 / 16
Cache a better format than the storage format —
1

Why cache a different format?

2

Why is it essentially free?

10 / 16
Part 2 · Why it’s essentially free · 1 / 5

Disaggregation already made the cache mandatory

Object store Cache Compute on miss 100–200 ms ~1 ms The server, its DRAM, and its network are already paid for — whatever format the cache happens to store.
11 / 16
Part 2 · Why it’s essentially free · 2 / 5

Cache servers bundle CPUs that caching barely uses

A cache server network serving reads DRAM · SSD holding data CPU idle spend the idle cycles on transcoding The cores come with the box — transcoding is what they are good for.
12 / 16
Part 2 · Why it’s essentially free · 3 / 5

Only the data that queries touch is ever transcoded

DATA LAKE — PARQUET written once, read rarely on miss LIQUIDCACHE — LIQUID hot subset, transcoded No eager ETL — the cold 90% is never converted at all.
13 / 16
Part 2 · Why it’s essentially free · 4 / 5

Transcoding hides behind the miss I/O

Parquet LiquidCache (blocking) LiquidCache transcoding CPU · 2.2 s 0 10 20 cold-run time (s) Warm runs 2× faster; cold runs match Parquet everywhere but memory.
14 / 16
Part 2 · Why it’s essentially free · 5 / 5

Free to run — and it pays

10×
lower cache CPU per query
not end-to-end latency
4×
smaller than Arrow in memory
vs. Arrow, still query-ready
100×
lower network traffic
filters run in the cache
ClickBench and TPC-H on Apache DataFusion, against non-pushdown caches.
15 / 16
Conclusion
Cache a better format
than the storage format.

Decoupled

The archive stays Parquet forever; the cache format evolves with the hardware.

Essentially free

Idle CPUs, only touched data, hidden behind miss I/O — 10× lower CPU, 100× lower network.

github.com/XiangpengHao/liquid-cache
Techniques upstreamed to Apache Arrow, Parquet, and DataFusion.
InfluxData Bauplan Spiral
16 / 16