63 Bytes Over the Line: How Unused Fonts Took Down Production After a Patch Upgrade

This post assumes you're comfortable with Next.js SSR, HTTP response headers, and a reverse proxy like NGINX.

How Claude was used in writing this post

Short version: the engineering is mine, and so is the first draft. Claude helps me get it published.

• Engineering and source material. All work described here was done by me. The background and technical detail come from reviewing my own PRs, their summaries, and our post-mortems.
• Research. Claude helped gather outside documentation and sources. I read each one myself to judge relevance and applicability before it informed the post.
• Drafting. With the material gathered, I sat down and wrote the rough draft manually in Neovim.
• Editing. Claude then assisted with edits, organization, and structure.

Like most of us I don't have much spare time, and these write-ups happen on weekends. Claude is what lets me actually finish them instead of leaving them half-written. They exist primarily for my own learning and growth as an engineer.

---

1. Summary

One morning, minutes after a scheduled production deploy, our monitoring lit up with 502 Bad Gateway errors. The symptom was specific and bizarre: users could log in successfully, and then the very next page load would 502. Not intermittently. Every full page load, for every user, deterministically.

The application is a data-heavy internal analytics platform built on Next.js (App Router) running on an EC2 instance behind an NGINX reverse proxy, fronted by a load balancer. Nothing about the outage touched application logic, the database, or the API. The entire failure lived in the size of the HTTP response headers our Next.js server was emitting, and a buffer limit in NGINX that we had never configured and never thought about.

The rest of this post is the chain of small, individually reasonable decisions that stacked into a house of cards, the diagnostic path we took to find it, and the fix.

---

2. Timeline

Times are relative to the first alert (T+0) to keep this generic. The whole incident, from first report to production restored, was under an hour.

The key call here was ordering. We rolled back before we understood the bug. A rollback to the previous release tag is a known-safe operation; a root-cause fix under incident pressure is not. Restoring service was our first priority, and it let us focus our debugging without the added pressure of a live production outage.

---

3. The symptoms, and why they were the whole clue

Before the mechanism, review the symptom table.

That last row is the tell. The identical page worked through client-side navigation and failed through a hard navigation.

There was even an accidental bypass that deepened the confusion at first: the post-login redirect used a client-side router.push(), so the redirect target loaded fine. It was only when someone refreshed, or opened a page in a new tab, that the 502 appeared. "Works right after login, breaks on refresh" sounds like a session bug. It wasn't. It was the type of navigation all along.

Concept box: why full loads and client navigations differ

On a full document load, Next.js server-renders the whole HTML page. As part of that response it advertises the fonts the page will need, using HTTP Link: <url>; rel=preload header entries so the browser can start fetching them early. One header entry per registered font face.

On a client-side <Link> navigation, there is no new HTML document. React fetches a React Server Components (RSC) payload - essentially JSON describing what changed - and the browser reuses the fonts it already has. No document, no preload headers.

So the two navigation styles produce structurally different response headers for the same page. That's the crack the bug fell through.

---

4. Root cause analysis

The outage was the collision of two things that were each fine on their own: too many preloaded fonts, and a framework upgrade that made every asset URL slightly longer.

4a. The latent problem: 31 fonts, 22 of them dead weight

Early in the project we bought a commercial admin template to skip weeks of boilerplate - login screen, nav shell, a themed component library, and fonts, all pre-wired. We stripped it down to a baseline and built our real product on top.

Here's the part I want to be precise about, because it's easy to pin on the template and that would be wrong: the 31 font files were not something we inherited. We introduced them ourselves.

What the template actually shipped with, font-wise, was small: two <link> tags pulling two font families from a hosted font CDN (think Google Fonts) in the document <head>. Two network calls. Nothing that would ever trouble an HTTP header buffer.

Later, as the app matured, we decided to move font loading in-house using the framework's built-in local-font system and drop the CDN entirely. The rationale was sound, and it's the generally recommended direction: serving fonts from your own origin removes a third-party DNS/TLS round-trip, closes a privacy leak (every page view otherwise pings the CDN with the user's IP and user-agent), and removes a class of font-swap layout shift.

The catch was in how I did the migration. At the time, the local-font API didn't let me point at a family and pick up every weight and style automatically - each face had to be declared explicitly. So to preserve the exact set of weights, italics, and bolds the CDN had been serving, I did the quick-and-dirty thing: I opened up the two families, enumerated every weight and style in use, downloaded each file, and registered them one by one. That is how two <link> tags turned into 31 registered local font files - 18 faces of the primary family (nine weights, normal and italic), 12 of a secondary family we barely used, and a logo asset. It rendered correctly and threw no errors or warnings we noticed at the time, so we moved on.

Concept box: what "font preloading" actually costs

A preload tells the browser: fetch this now, at high priority, before you've even parsed the CSS that references it.

<link rel="preload" href="/font.woff2" as="font" type="font/woff2" crossorigin>

Preloading one or two above-the-fold faces is a clear win - it collapses a serial chain of "download HTML → download CSS → parse CSS → discover font → fetch font" into a head start.

But each preload is a high-priority fetch that competes with genuinely critical resources (JS, CSS, the LCP image). Preload twenty faces the page may never use and you're not optimizing - you're forcing the browser to prioritize bytes nobody needs, and you're bloating the response headers that advertise them. Preloading is a scalpel, not a default-on firehose.

When you register a local font this way, the framework emits a preload Link header entry for it on every full page render. So the real trade I'd made wasn't "two network calls for 31 tidy local files." It was "two network calls for 31 preload header entries stamped onto every server-rendered response."

I'll own this one. I was a junior engineer at the time, single-handedly building out the frontend, and I didn't have the experience to recognize the code smell. My mental model was shallow - 31 local imports, resolved server-side, fonts render, done - and I never took the step of understanding how the local-font API actually surfaced those files to the browser. The connection I missed is the entire bug: each of those registrations doesn't just load a file, it adds a line to the response headers. The fonts rendered correctly, there were more urgent features in the pipeline, and "good enough" won. It should have been a step-back moment. It wasn't.

Here is what the header block looked like, by the numbers:

4b. The NGINX buffer nobody configured

Our NGINX config never set proxy_buffer_size. That means it used the platform default, which on 64-bit Linux is one memory page: 4,096 bytes.

proxy_buffer_size controls the buffer NGINX uses to read the first part of the upstream response - specifically the status line and the response headers. If the upstream's headers don't fit in that single buffer, NGINX doesn't stream them anyway. It gives up on the response and returns a 502, with this exact log line:

upstream sent too big header while reading response header from upstream

At 4,159 bytes of headers against a 4,096-byte buffer, we were 63 bytes over. Which raises the obvious question: if we were already over the limit, why did production work at all right up until the deploy?

We weren't over the limit before. We were just under it.

4c. The trigger: a hash-encoding change in a patch upgrade

The deploy that morning included a batch of routine dependency updates. Buried in it was a minor-version framework bump whose changelog said nothing dramatic. But that release included an upstream change to the bundler (vercel/next.js#91137): asset content hashes switched from hex to base40 encoding.

The net effect on filenames was concrete and measurable: the content-hash segment in asset filenames grew from 8 characters to 13 characters.

Every hashed asset filename got 5 characters longer. Fonts are hashed assets. Each of our 31 font URLs grew by 5 characters:

31 fonts × 5 characters ≈ 155 bytes
4,159 - 155 = 4,004 bytes   (before the upgrade: under the 4,096 limit)
4,159 bytes                 (after the upgrade: 63 bytes over, permanently)

That's the whole bug. A patch-level upgrade that touched filename encoding silently pushed a header block that had been living 90-odd bytes under a buffer limit to 63 bytes over it. No application code changed. No font changed. Just the length of some strings in a header.

The full chain

Template shipped 2 CDN font links
        │
        ▼
We migrated fonts in-house         →  registered 31 local faces = 31 preload Link header entries per full render
        │
        ▼
Header block sits at ~4,004 bytes   →  quietly under NGINX's 4,096-byte default buffer
        │
        ▼
Framework minor upgrade             →  asset hashes grow 8 → 13 chars (+5 per URL)
        │
        ▼
+155 bytes                          →  header block hits 4,159 bytes (63 over)
        │
        ▼
NGINX: "upstream sent too big header"  →  502 on every full page load

---

5. Diagnosis: how we actually found it

We didn't get to that clean chain on the first try. The morning-of investigation was hypothesis elimination on staging, and it's worth showing because the process is more reusable than the answer.

The suspect deploy bundled several changes. We had three candidates and worked them like a bisect:

1. Hypothesis: a circular-dependency cleanup changed module init order. We reverted it on staging. The 502 persisted. Eliminated.
2. Hypothesis: an ESLint plugin migration changed build output. We reverted that too, on top of the first revert. The 502 persisted. Eliminated.
3. Remaining suspect: the batch of dependency updates (which contained the framework bump), versus a purely feature-level change with no infra footprint.

At this point we stopped guessing at the application layer and read the NGINX error log. The upstream sent too big header line reframed the entire problem. This was never an application bug. It was a response-header-size bug. From there, connecting "header too big" to "we emit a preload header per font" to "we register 31 fonts" took one grep of the font config.

The lesson embedded in the diagnosis: when navigation type determines success, read the proxy logs before you read your own code. We spent our initial efforts reverting application-layer changes because that's the code we own and understand. The answer was one line in a log from the layer we'd never configured.

---

6. The fix

The root-cause fix was scoped to a single file: the local-font configuration. No component changed, because the fonts we removed were never actually used in rendering - they were registered, preloaded, and ignored.

We reduced font registrations from 31 to 9:

• Kept the primary family at the weights the app genuinely uses.
• Dropped the ultra-light weights (100/200/300) nobody rendered.
• Dropped every italic except the single one in real use.
• Dropped the entire secondary family except one weight, kept only because a theme token still referenced it.

The result:

That 2,619 bytes of headroom matters. It absorbs future framework upgrades, deployment-id suffixes on asset URLs, and variable cookie sizes, without flirting with the limit again.

The fork in the road: fix the config, or fix the cause?

There's a second, tempting fix: bump NGINX's buffer.

# The defensive one-liner
proxy_buffer_size 8k;

We documented this as a defensive option and deliberately did not rely on it as the fix. Here's the reasoning, because it's the most useful decision in the whole incident:

• Raising proxy_buffer_size to 8k would have stopped the 502 instantly. But it would have left 22 unused fonts preloading on every page, still bloating headers, still competing for browser fetch priority, still growing with every future upgrade. We'd have paid for the buffer with a slower page and a hidden liability that would resurface at 8k someday.
• Removing the dead fonts fixed the actual problem - the header block was oversized because it was advertising resources nobody used. Once that was gone, the buffer default was fine.

The config change treats the symptom. The font change treats the disease. We treated the disease, and noted the buffer bump as a belt-and-suspenders option for anyone who wants defense in depth. (If you run Next.js behind NGINX and can't audit your fonts today, raising proxy_buffer_size is a perfectly good stopgap - just don't mistake it for a fix.)

---

7. Prevention: the dev/prod parity gap

The deepest lesson isn't about fonts. It's about why this was invisible locally.

In local development, your browser talks to the Next.js dev server directly. There is no reverse proxy in the loop, so there is no proxy_buffer_size, so there is no buffer to overflow. The exact same oversized header block that 502s in production sails through in development, because the component that rejects it isn't running on your laptop.

This is a dev/prod parity gap, and it hides an entire class of failures. Anything that lives in the proxy layer - header limits, buffer sizes, timeout behavior, body-size caps - is simply absent from the default local stack. You can't catch a bug with a tool you don't run.

We evaluated two prevention options:

Option A: a full-page-load header-size E2E test. A browser test that navigates via hard loads (not just <Link> clicks), captures the response headers, and asserts the total stays under a threshold with margin. The catch: to be meaningful it needs the proxy in the loop, so it's a manual gate against a full local stack rather than a cheap CI check.

Option B: run NGINX locally via Docker Compose, mirroring the production defaults. This closes the parity gap directly - a developer who oversizes the headers sees the identical 502 on their machine, before it ever reaches production. The tradeoff is adding Docker as a development dependency and keeping the local proxy config in sync with production.

Both are reasonable. The framing that matters: a bug that can only manifest behind a proxy can only be caught behind a proxy. If production runs a component, your test environment should too.

---

8. Lessons learned

1. Unused dependencies accumulate risk silently. Those 22 dead fonts sat in the config for well over a year, harmless, until an unrelated upgrade turned them into an outage. Dead code isn't neutral. It's a loaded liability waiting for a trigger.
2. Patch and minor version bumps can carry outsized side effects. The changelog entry was about hash encoding. The impact was a production outage. "It's just a minor bump" is exactly the assumption that lets a five-character string change take down a login flow.
3. Client-side and server-side navigation produce fundamentally different HTTP responses. Internalizing this turns a baffling "works via link, fails via refresh" symptom into an immediate "check response header size" instinct. It's a useful mental model for debugging infrastructure behind a proxy.
4. Infrastructure defaults were written for a different era. NGINX's 4,096-byte header buffer - one memory page - is a sensible default for a world of lean, hand-written responses. Modern frameworks emit far richer headers (preloads, early hints, deployment metadata). When a decades-old default meets a framework that treats headers as a resource-optimization channel, the default can quietly become a ceiling.
5. Restore first, understand second. Rolling back to a known-good tag before diagnosing turned a P1 into a controlled investigation. Debugging under "the site is down" pressure is how a one-line fix becomes a three-line mistake.

The whole thing came down to 63 bytes. Not a logic error, not a race condition, not a bad query - just a header block that grew five characters at a time until it crossed a line nobody knew was there.

---

Further Reading

• NGINX proxy_buffer_size / proxy_buffers - the buffer at the center of this incident
• Next.js font optimization (next/font) - how local fonts and preloading work
• next/font/local API reference
• MDN: the Link HTTP header
• MDN: rel=preload - the mechanism, and its costs
• React Server Components - why client navigation ships JSON, not HTML
• The upstream change that triggered it (vercel/next.js#91137) - hex to base40 asset hashes

---

Written after a real P1. Framework and infrastructure versions and behaviors described are as of early 2026; buffer defaults and framework internals change, so verify against your own stack.