◀ BACK TO BLOG
· 11 min read

The night all the chinese pages died

webhttpurlscachingseo

THE 3AM VERSION

I was crawling a big multilingual site. English fine. French fine. Then the crawler hit /zh/ and every single Chinese page came back the same way:

301 → 301 → 301 → 301 → ... Exceeded 30 redirects.

Every one. Hundreds of URLs, all dead, all looping. Meanwhile I could open the exact same pages in Chrome and they loaded instantly. curl fetched them fine too — which, as I’d learn, depended entirely on which curl. Only my crawler was drowning.

The difference between “loads instantly” and “loops forever” turned out to be a single character of case: %E9 versus %e9.

THE ONE HEX DIGIT

Chinese characters in a URL get percent-encoded into bytes like %e9%81%97. And here’s the thing every spec agrees on: the case of those hex digits doesn’t matter. RFC 3986 says it flat out — if two URLs differ only in the case of percent-encoded hex, they are the same URL. Uppercase is a gentle “should,” not a “must.” %E9 and %e9 are identical twins.

So why was my crawler in a fistfight with the server over it?

THE FISTFIGHT

Two reasonable programs, each doing the reasonable thing, in opposite directions:

  • My crawler (Python requests) normalizes escapes to uppercase on every hop — that’s RFC 3986’s recommendation, and urllib3 does it on purpose with no off switch. So it requests …%E9….
  • The server (a multilingual WordPress plugin) had decided its canonical URLs were lowercase, and it 301-redirected anything else back down: …%E9… → …%e9….

Watch them go:

  1. Crawler requests %E9 (just uppercased it).
  2. Server: “301, the real URL is %e9.”
  3. Crawler reads that, re-uppercases it back to %E9, requests again.
  4. Server: “301, it’s %e9.”
  5. → forever.
CLIENT Python requests / urllib3 %E9 ALWAYS UPPERCASE SERVER i18n plugin canonical=lower %e9 301 → LOWERCASE GET …%E9… 301 Location: …%e9… VS REDIRECTS ×30 → GAME OVER
Fig 1 — The client always uppercases; the server always answers lowercase. Neither blinks, so it never ends.

Browsers never join this fight. The URL in your address bar follows the WHATWG spec, not RFC 3986, and the WHATWG parser preserves escape case — it never rewrites %e9 to %E9, so it never hands the server anything to redirect. Node and Go behave the same way.

So the real dividing line isn’t who’s compliant — it’s whether your client rewrites %e9 into %E9 at all. Python’s requests does, on every hop, which is exactly why it loops. And curl? It used to leave the case alone and stay out of the fight — until 8.20 started uppercasing escapes too, quietly moving it across the line. Whether that yields a single redirect or the same infinite loop depends on internals I wouldn’t bet on either way; the point is curl is no longer the innocent bystander it was in 8.19.

PRESERVES CASE never triggers the redirect · no loop Browsers (WHATWG URL) Node.js · Go net/url · Java URI → 200 OK REWRITES → %E9 fights the lowercasing server · triggers 301 Python requests / urllib3 ∞ LOOP — confirmed curl 8.20+ (new arrival) loops? unconfirmed — but poisons caches curl ≤8.19 lived here… 8.20 (Apr 2026) moved curl across the line
Fig 2 — The only question that decides the fight: does your client rewrite %e9 → %E9? curl crossed the line at 8.20.

Who’s actually wrong? The server. Not because it prefers lowercase — that’s legal — but because it redirects at all over a difference the spec says isn’t a difference. The right answer to both %E9… and %e9… is the same page, 200, no redirect. (For the WordPress folks: core isn’t the culprit — a case-sensitive string comparison in a translation plugin is. Same class of bug as Yoast #22212, which deindexed a Korean site’s /ko/ URLs the same way.)

So far, so annoying. I could fix my crawler in five lines — allow_redirects=False, then compare the Location to my request after unquote-ing both, and treat a case-only difference as “same URL, stop.” Done. But then I looked closer at why the browser and curl were fine and I wasn’t, and found the part that actually scared me.

THE PART THAT ACTUALLY SCARED ME

That site sat behind a cache. And the cache computed its key case-insensitively over the escapes — it folded %E9 and %e9 to the same hash.

Play it out. Some client — my crawler, a security scanner, anything that uppercases — requests %E9. The origin says 301 → %e9. The cache stores that 301 under a case-folded key. Now the next visitor asks for the perfectly-fine lowercase %e9… hashes to the same key… and gets handed the cached 301, pointing at a URL that hashes right back to the same poisoned entry.

Loop. For everybody. Including the browsers and other case-preserving clients that were previously immune, because now the poison lives in the shared cache, not in any one client’s quirks. One wrong-case request breaks the URL for all of them.

CACHE key folds case → %E9 = %e9 stores: 301 ☠ BROWSER asks for %e9 CURL asks for %E9 SCANNER mixed case 301 301 301 everyone loops now
Fig 3 — Fold %E9 and %e9 into one cache key and the 301 is stored once, then served to everyone — browsers included.

And that’s the ugly part: the trigger is free. Uppercase and mixed-case escapes are exactly what vulnerability scanners and WAF-bypass fuzzers spray by the thousand. So the cache can get poisoned by accident, on any random Tuesday, by a bot that isn’t even aiming at you. Or on purpose — script uppercase requests at the homepage, re-poison every time the cache entry expires, and you’ve got a no-auth, no-payload denial-of-service. It has a name: CPDoS, cache-poisoned denial-of-service.

The blast radius is quietly growing, too: libcurl started uppercasing path escapes in 8.20.0 and widened it in 8.21.0. libcurl is under PHP, Git, package managers, half the internet’s tooling — so the pool of clients that can trip a case-folded cache key is expanding as that version rolls into base images.

THE FIX (WHERE IT ACTUALLY BELONGS)

Fixing my crawler stopped my pain, but it does nothing for the next visitor. The real fixes are two, and either one alone breaks the loop:

  1. Make the cache key case-sensitive on escapes — or canonicalize the case before hashing, so %E9 and %e9 don’t collide into one poisonable entry.
  2. Stop the origin redirecting on hex case. Serve 200 for both forms, exactly as the spec’s equivalence rule already requires.

If you run a cache, one quick thing worth checking: does an inbound Cache-Control: no-cache actually bypass it, or does your config strip that header? If it’s stripped, a poisoned client can’t even rescue itself — a request for a clean copy still gets the cached 301. That’s the difference between “annoying” and “outage,” and it’s one curl -I to find out.

THE MORAL

Nobody shipped a bug here. The spec said “should.” The browser vendors made a defensible call. The Python library followed the RFC. The cache did a normal optimization. Every single piece passes its own conformance test.

The outage lives in the seams — where two systems each did the reasonable thing, in opposite directions, and a cache turned the disagreement into a shared, sticky failure instead of letting it cancel out.

So when you’re staring at something that “cannot possibly be happening” — the same page loads in your browser and loops in your script — stop looking for the broken component. Look at the seams between the correct ones.

%E9 and %e9 are the same. Right up until an entire language section of a website goes dark.