Avoid using TaskTypes as keys in storage instead use hashes #88904

lukesandberg · 2026-01-22T17:30:25Z

Save space in the persisent store by only recording TaskType one time instead of twice.

What?

Instead of using an encoded TaskType struct as a key in persistent storage just use a 64 bit hash. Now when looking up we actually need to do a cascading read

read task ids that match the hash
restore all the tasks until we find one that matches our CachedTaskType

Full cache misses perform the same and so do cache hits since we always end up reading the TaskData anyway (in the connect_child operation that immediately follows). This just slightly changes when that second read happens, as such we shouldn't really expect it to slow down.

In the case of a hash collisions we do end up doing strictly more work (reading and restoring TaskData entries for the wrong task), but this work is cached and this should be extremely rare assuming a good hash function..

Why?

Currently we encode 2 copies of ever CachedTaskType in the database.

as the key of the TaskType->TaskID map (aka TaskCache keyspace)
as a part of the TaskStorage struct stored in the TaskData keyspace

this redundancy is wasteful. Instead we can make the TaskCache map much smaller and add a bit of complexity to lookups.

Future Work

Right now to compute the hashes we are just running encode and then hashing the bytes. This is not optimal, but we do not have a hash function that is suitable for this usecase. So we should create a new PersistentHash trait that TaskInputs implement in order to support this without encoding.

lukesandberg · 2026-01-22T17:31:07Z

Avoid using TaskTypes as keys in storage instead use hashes #88904 👈 (View in Graphite)
Turbopack: refactor data storage to avoid reverse task cache #88492 : 1 other dependent PR (#88889 )
Turbopack: add caching to chunking #88561
Turbopack: use selective reads for merged modules #88560
Turbopack: selective read for module_chunk_groups in ChunkGroupInfo #88559
Turbopack: MappedReadRef should be Send and Sync #88558
canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs

turbopack/crates/turbo-tasks-backend/src/backend/operation/mod.rs

nextjs-bot · 2026-01-22T17:44:53Z

Failing test suites

Commit: 131bc3f | About building and testing Next.js

pnpm test-start test/e2e/app-dir/segment-cache/prefetch-runtime/prefetch-runtime.test.ts (job)

runtime prefetching > passed to a public cache > can completely prefetch a page that uses cookies and no uncached IO (DD)

Expand output

● runtime prefetching › passed to a public cache › can completely prefetch a page that uses cookies and no uncached IO

apiRequestContext.fetch: read ECONNRESET
Call log:
  - → GET http://localhost:41059/passed-to-public-cache/cookies-only?_rsc=gtqjj
  -   user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/130.0.6723.31 Safari/537.36
  -   accept: */*
  -   accept-encoding: gzip,deflate,br
  -   cookie: testCookie=initialValue
  -   next-test-fetch-priority: low
  -   referer: http://localhost:41059/
  -   next-router-prefetch: 1
  -   next-router-segment-prefetch: /!KGRlZmF1bHQp/passed-to-public-cache
  -   next-url: /
  -   rsc: 1
  -   sec-ch-ua: "Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"
  -   sec-ch-ua-mobile: ?0
  -   sec-ch-ua-platform: "Linux"

  225 |             // server; we pass the request to the server the immediately.
  226 |             result: (async () => {
> 227 |               const originalResponse = await page.request.fetch(request, {
      |                                                           ^
  228 |                 maxRedirects: 0,
  229 |               })
  230 |

  at fetch (lib/router-act.ts:227:59)
  at lib/router-act.ts:245:13
  at routeHandler (lib/router-act.ts:257:7)

● runtime prefetching › passed to a public cache › can completely prefetch a page that uses cookies and no uncached IO

apiRequestContext.fetch: read ECONNRESET
Call log:
  - → GET http://localhost:41059/passed-to-public-cache/cookies-only?_rsc=gtqjj
  -   user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/130.0.6723.31 Safari/537.36
  -   accept: */*
  -   accept-encoding: gzip,deflate,br
  -   cookie: testCookie=initialValue
  -   next-test-fetch-priority: low
  -   referer: http://localhost:41059/
  -   next-router-prefetch: 1
  -   next-router-segment-prefetch: /!KGRlZmF1bHQp/passed-to-public-cache
  -   next-url: /
  -   rsc: 1
  -   sec-ch-ua: "Chromium";v="130", "HeadlessChrome";v="130", "Not?A_Brand";v="99"
  -   sec-ch-ua-mobile: ?0
  -   sec-ch-ua-platform: "Linux"

  225 |             // server; we pass the request to the server the immediately.
  226 |             result: (async () => {
> 227 |               const originalResponse = await page.request.fetch(request, {
      |                                                           ^
  228 |                 maxRedirects: 0,
  229 |               })
  230 |

  at fetch (lib/router-act.ts:227:59)
  at lib/router-act.ts:245:13
  at routeHandler (lib/router-act.ts:257:7)

codspeed-hq · 2026-01-22T17:51:42Z

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing direct_reverse_mapping (131bc3f) with canary (f51e3ac)}

Summary

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks¹

3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

nextjs-bot · 2026-01-22T17:52:37Z

Stats from current PR

✅ No significant changes detected

📊 All Metrics

📖 Metrics Glossary

Dev Server Metrics:

Listen = TCP port starts accepting connections
First Request = HTTP server returns successful response
Cold = Fresh build (no cache)
Warm = With cached build artifacts

Build Metrics:

Fresh = Clean build (no .next directory)
Cached = With existing .next directory

Change Thresholds:

Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
Size: Changes < 1KB AND < 1% are insignificant
All other changes are flagged to catch regressions

⚡ Dev Server

Metric	Canary	PR	Change	Trend
Cold (Listen)	456ms	455ms	✓	█▁▁▁▁
Cold (Ready in log)	442ms	440ms	✓	█▁▂▁▂
Cold (First Request)	1.138s	1.158s	✓	█▁▃▁▃
Warm (Listen)	457ms	456ms	✓	▆▁▁▁▁
Warm (Ready in log)	443ms	443ms	✓	▇▁▁▁▁
Warm (First Request)	341ms	345ms	✓	▇▁▁▁▁

📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric	Canary	PR	Change	Trend
Cold (Listen)	456ms	455ms	✓	▁▁▁▁▁
Cold (Ready in log)	438ms	437ms	✓	▂▇▄▇▂
Cold (First Request)	1.822s	1.821s	✓	▂▆▂▆▁
Warm (Listen)	456ms	456ms	✓	▁▁▁▁▁
Warm (Ready in log)	437ms	437ms	✓	▁▅▁▅▁
Warm (First Request)	1.829s	1.821s	✓	▂▅▁▅▁

⚡ Production Builds

Metric	Canary	PR	Change	Trend
Fresh Build	4.347s	4.333s	✓	█▁▁▁▁
Cached Build	4.376s	4.370s	✓	█▁▁▁▁

📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric	Canary	PR	Change	Trend
Fresh Build	13.969s	13.981s	✓	▃▃▁▂▁
Cached Build	14.061s	14.035s	✓	▃▂▁▃▁
node_modules Size	460 MB	460 MB	✓	▁▁▁██

📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles: **432 kB** → **432 kB** ✅ -50 B

82 files with content-based hashes (individual files not comparable between builds)

Server

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	764 B	759 B	✓
Total	764 B	759 B	✅ -5 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	450 B	451 B	✓
Total	450 B	451 B	⚠️ +1 B

📦 Webpack

Client

Main Bundles

	Canary	PR	Change
2086.HASH.js gzip	169 B	N/A	-
2161-HASH.js gzip	5.47 kB	N/A	-
2747-HASH.js gzip	4.53 kB	N/A	-
4322-HASH.js gzip	52.7 kB	N/A	-
ec793fe8-HASH.js gzip	62.3 kB	N/A	-
framework-HASH.js gzip	59.8 kB	59.8 kB	✓
main-app-HASH.js gzip	251 B	253 B	✓
main-HASH.js gzip	38.7 kB	39.1 kB	✓
webpack-HASH.js gzip	1.68 kB	1.68 kB	✓
1596.HASH.js gzip	N/A	169 B	-
2658-HASH.js gzip	N/A	52.4 kB	-
6349-HASH.js gzip	N/A	4.52 kB	-
7019-HASH.js gzip	N/A	5.49 kB	-
b17a3386-HASH.js gzip	N/A	62.3 kB	-
Total	226 kB	226 kB	⚠️ +55 B

Polyfills

	Canary	PR	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Total	39.4 kB	39.4 kB	✓

Pages

	Canary	PR	Change
_app-HASH.js gzip	194 B	193 B	✓
_error-HASH.js gzip	182 B	182 B	✓
css-HASH.js gzip	336 B	335 B	✓
dynamic-HASH.js gzip	1.8 kB	1.8 kB	✓
edge-ssr-HASH.js gzip	256 B	256 B	✓
head-HASH.js gzip	352 B	349 B	✓
hooks-HASH.js gzip	385 B	384 B	✓
image-HASH.js gzip	580 B	580 B	✓
index-HASH.js gzip	259 B	258 B	✓
link-HASH.js gzip	2.5 kB	2.51 kB	✓
routerDirect..HASH.js gzip	319 B	317 B	✓
script-HASH.js gzip	385 B	387 B	✓
withRouter-HASH.js gzip	316 B	315 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Total	7.97 kB	7.96 kB	✅ -8 B

Server

Edge SSR

	Canary	PR	Change
edge-ssr.js gzip	126 kB	126 kB	✓
page.js gzip	244 kB	240 kB	🟢 4.71 kB (-2%)
Total	370 kB	366 kB	✅ -4.58 kB

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	617 B	618 B	✓
middleware-r..fest.js gzip	155 B	156 B	✓
middleware.js gzip	33 kB	33.3 kB	✓
edge-runtime..pack.js gzip	842 B	842 B	✓
Total	34.7 kB	35 kB	⚠️ +297 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	736 B	738 B	✓
Total	736 B	738 B	⚠️ +2 B

Build Cache

	Canary	PR	Change
0.pack gzip	3.71 MB	3.72 MB	🔴 +4.95 kB (+0%)
index.pack gzip	101 kB	101 kB	✓
index.pack.old gzip	101 kB	102 kB	✓
Total	3.92 MB	3.92 MB	⚠️ +5.32 kB

🔄 Shared (bundler-independent)

Runtimes

	Canary	PR	Change
app-page-exp...dev.js gzip	306 kB	306 kB	✓
app-page-exp..prod.js gzip	163 kB	163 kB	✓
app-page-tur...dev.js gzip	306 kB	306 kB	✓
app-page-tur..prod.js gzip	163 kB	163 kB	✓
app-page-tur...dev.js gzip	302 kB	302 kB	✓
app-page-tur..prod.js gzip	161 kB	161 kB	✓
app-page.run...dev.js gzip	303 kB	303 kB	✓
app-page.run..prod.js gzip	161 kB	161 kB	✓
app-route-ex...dev.js gzip	69.4 kB	69.4 kB	✓
app-route-ex..prod.js gzip	48.2 kB	48.2 kB	✓
app-route-tu...dev.js gzip	69.4 kB	69.4 kB	✓
app-route-tu..prod.js gzip	48.2 kB	48.2 kB	✓
app-route-tu...dev.js gzip	69 kB	69 kB	✓
app-route-tu..prod.js gzip	47.9 kB	47.9 kB	✓
app-route.ru...dev.js gzip	68.9 kB	68.9 kB	✓
app-route.ru..prod.js gzip	47.9 kB	47.9 kB	✓
dist_client_...dev.js gzip	324 B	324 B	✓
dist_client_...dev.js gzip	326 B	326 B	✓
dist_client_...dev.js gzip	318 B	318 B	✓
dist_client_...dev.js gzip	317 B	317 B	✓
pages-api-tu...dev.js gzip	42.4 kB	42.4 kB	✓
pages-api-tu..prod.js gzip	32.2 kB	32.2 kB	✓
pages-api.ru...dev.js gzip	42.4 kB	42.4 kB	✓
pages-api.ru..prod.js gzip	32.2 kB	32.2 kB	✓
pages-turbo....dev.js gzip	51.7 kB	51.7 kB	✓
pages-turbo...prod.js gzip	38.8 kB	38.8 kB	✓
pages.runtim...dev.js gzip	51.7 kB	51.7 kB	✓
pages.runtim..prod.js gzip	38.8 kB	38.8 kB	✓
server.runti..prod.js gzip	62.4 kB	62.4 kB	✓
Total	2.73 MB	2.73 MB	⚠️ +3 B

turbopack/crates/turbo-persistence/src/db.rs

turbopack/crates/turbo-tasks-backend/src/database/by_key_space.rs

instead store hashes, this makes lookups slightly more complex but greatly reduces storage requirements.

…he db lookup methods

nextjs-bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Jan 22, 2026

sokra mentioned this pull request Jan 22, 2026

Turbopack: refactor data storage to avoid reverse task cache #88492

Merged

lukesandberg mentioned this pull request Jan 22, 2026

Turbopack: change invalidator and immutable to data category #88889

Merged

vercel bot reviewed Jan 22, 2026

View reviewed changes

turbopack/crates/turbo-tasks-backend/src/backend/storage_schema.rs Outdated Show resolved Hide resolved

turbopack/crates/turbo-tasks-backend/src/backend/operation/mod.rs Outdated Show resolved Hide resolved

sokra changed the base branch from sokra/remove-reverse-task-cache to graphite-base/88904 January 22, 2026 20:50

sokra force-pushed the graphite-base/88904 branch from 0c1d348 to 9e97759 Compare January 23, 2026 08:05

sokra force-pushed the direct_reverse_mapping branch from aebd28d to 9562e78 Compare January 23, 2026 08:05

graphite-app bot changed the base branch from graphite-base/88904 to canary January 23, 2026 08:05

sokra force-pushed the direct_reverse_mapping branch from 9562e78 to e929b37 Compare January 23, 2026 08:05

sokra reviewed Jan 23, 2026

View reviewed changes

turbopack/crates/turbo-persistence/src/db.rs Outdated Show resolved Hide resolved

sokra reviewed Jan 23, 2026

View reviewed changes

turbopack/crates/turbo-tasks-backend/src/database/by_key_space.rs Outdated Show resolved Hide resolved

lukesandberg changed the title ~~Add a new keyspace for the reverse task mappiong to avoid storing task-types multiple times~~ Avoid using TaskTypes as keys in storage instead use hashes Jan 25, 2026

lukesandberg added 2 commits January 24, 2026 19:22

Stop using TaskTypes as keys!

3203bc6

instead store hashes, this makes lookups slightly more complex but greatly reduces storage requirements.

clean up the old lookup methods

a33253f

lukesandberg force-pushed the direct_reverse_mapping branch from e929b37 to a33253f Compare January 25, 2026 06:11

Use smallvecs, add tracing and share more implementation details in t…

131bc3f

…he db lookup methods

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid using TaskTypes as keys in storage instead use hashes #88904

Avoid using TaskTypes as keys in storage instead use hashes #88904

lukesandberg commented Jan 22, 2026 •

edited

Loading

Uh oh!

lukesandberg commented Jan 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

nextjs-bot commented Jan 22, 2026 •

edited

Loading

Uh oh!

codspeed-hq bot commented Jan 22, 2026 •

edited

Loading

Uh oh!

nextjs-bot commented Jan 22, 2026 •

edited

Loading

⚡ Dev Server

📦 Dev Server (Webpack)

⚡ Production Builds

📦 Production Builds (Webpack)

Bundle Sizes

⚡ Turbopack

📦 Webpack

🔄 Shared (bundler-independent)

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Avoid using TaskTypes as keys in storage instead use hashes #88904

Are you sure you want to change the base?

Avoid using TaskTypes as keys in storage instead use hashes #88904

Conversation

lukesandberg commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

Future Work

Uh oh!

lukesandberg commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nextjs-bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failing test suites

Uh oh!

codspeed-hq bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging this PR will not alter performance

Summary

Footnotes

Uh oh!

nextjs-bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stats from current PR

✅ No significant changes detected

⚡ Dev Server

📦 Dev Server (Webpack)

⚡ Production Builds

📦 Production Builds (Webpack)

Bundle Sizes

⚡ Turbopack

📦 Webpack

🔄 Shared (bundler-independent)

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lukesandberg commented Jan 22, 2026 •

edited

Loading

lukesandberg commented Jan 22, 2026 •

edited

Loading

nextjs-bot commented Jan 22, 2026 •

edited

Loading

codspeed-hq bot commented Jan 22, 2026 •

edited

Loading

nextjs-bot commented Jan 22, 2026 •

edited

Loading