The night all the chinese pages died

THE 3AM VERSION

I was crawling a big multilingual site. English fine. French fine. Then the crawler hit /zh/ and every single Chinese page came back the same way:

301 → 301 → 301 → 301 → ... Exceeded 30 redirects.

Every one. Hundreds of URLs, all dead, all looping. Meanwhile I could open the exact same pages in Chrome and they loaded instantly. curl fetched them fine too — which, as I’d learn, depended entirely on which curl. Only my crawler was drowning.

The difference between “loads instantly” and “loops forever” turned out to be a single character of case: %E9 versus %e9.

THE ONE HEX DIGIT

Chinese characters in a URL get percent-encoded into bytes like %e9%81%97. And here’s the thing every spec agrees on: the case of those hex digits doesn’t matter. RFC 3986 says it flat out — if two URLs differ only in the case of percent-encoded hex, they are the same URL. Uppercase is a gentle “should,” not a “must.” %E9 and %e9 are identical twins.

So why was my crawler in a fistfight with the server over it?

THE FISTFIGHT

Two reasonable programs, each doing the reasonable thing, in opposite directions:

• My crawler (Python requests) normalizes escapes to uppercase on every hop — that’s RFC 3986’s recommendation, and urllib3 does it on purpose with no off switch. So it requests …%E9….
• The server (a multilingual WordPress plugin) had decided its canonical URLs were lowercase, and it 301-redirected anything else back down: …%E9… → …%e9….

Watch them go:

1. Crawler requests %E9 (just uppercased it).
2. Server: “301, the real URL is %e9.”
3. Crawler reads that, re-uppercases it back to %E9, requests again.
4. Server: “301, it’s %e9.”
5. → forever.

Browsers never join this fight. The URL in your address bar follows the WHATWG spec, not RFC 3986, and the WHATWG parser preserves escape case — it never rewrites %e9 to %E9, so it never hands the server anything to redirect. Node and Go behave the same way.

So the real dividing line isn’t who’s compliant — it’s whether your client rewrites %e9 into %E9 at all. Python’s requests does, on every hop, which is exactly why it loops. And curl? It used to leave the case alone and stay out of the fight — until 8.20 started uppercasing escapes too. I didn’t want to guess, so I built curl 8.21.0 and pointed it at a lowercasing server: it loops exactly like Python. 8.20+ re-uppercases the redirect Location on every hop — I confirmed it in the source too (the redirect path forces curl’s URL-encoder on, and that pass rewrites %e9 → %E9 unconditionally). Only pre-8.20 curl stays out of the fight; anything 8.20 or newer is now a full participant.

Same dividing line, with the version numbers spelled out:

Client / stack	Version	Rewrites %e9 → %E9?	Verdict vs a lowercasing server
Browsers (Chrome / Firefox / Safari)	all	No — preserves case	Safe — 200, never redirected
Node.js (WHATWG URL)	all	No	Safe
Go net/url	all	No	Safe
Java java.net.URI	all	No	Safe
Python requests / urllib3	2.x	Yes, every hop	Loops ∞ — confirmed †
curl / libcurl	≤ 8.19	No	Settles (0–1 hop) †
curl / libcurl	8.20+	Yes, every hop	Loops ∞ — confirmed †

† Reproduced locally — built curl 8.21.0 and 8.5.0 against a lowercasing test server, plus Python requests 2.33.1 / urllib3 2.6.3. The browser / Node / Go / Java rows follow from each stack’s documented URL handling.

Who’s actually wrong? The server. Not because it prefers lowercase — that’s legal — but because it redirects at all over a difference the spec says isn’t a difference. The right answer to both %E9… and %e9… is the same page, 200, no redirect. (For the WordPress folks: core isn’t the culprit — a case-sensitive string comparison in a translation plugin is. Same class of bug as Yoast #22212, which deindexed a Korean site’s /ko/ URLs the same way.)

So far, so annoying. I could fix my crawler in five lines — allow_redirects=False, then compare the Location to my request after unquote-ing both, and treat a case-only difference as “same URL, stop.” Done. But then I looked closer at why the browser and curl were fine and I wasn’t, and found the part that actually scared me.

THE PART THAT ACTUALLY SCARED ME

That site sat behind a cache. And the cache computed its key case-insensitively over the escapes — it folded %E9 and %e9 to the same hash.

Play it out. Some client — my crawler, a security scanner, anything that uppercases — requests %E9. The origin says 301 → %e9. The cache stores that 301 under a case-folded key. Now the next visitor asks for the perfectly-fine lowercase %e9… hashes to the same key… and gets handed the cached 301, pointing at a URL that hashes right back to the same poisoned entry.

Loop. For everybody. Including the browsers and other case-preserving clients that were previously immune, because now the poison lives in the shared cache, not in any one client’s quirks. One wrong-case request breaks the URL for all of them.

And that’s the ugly part: the trigger is free. Uppercase and mixed-case escapes are exactly what vulnerability scanners and WAF-bypass fuzzers spray by the thousand. So the cache can get poisoned by accident, on any random Tuesday, by a bot that isn’t even aiming at you. Or on purpose — script uppercase requests at the homepage, re-poison every time the cache entry expires, and you’ve got a no-auth, no-payload denial-of-service. It has a name: CPDoS, cache-poisoned denial-of-service.

The blast radius is quietly growing, too: libcurl started uppercasing path escapes in 8.20.0 and widened it in 8.21.0 — and as we just saw, that’s enough to make it loop against a lowercasing server and poison a case-folded cache. libcurl is under PHP, Git, package managers, half the internet’s tooling, so the pool of clients that both trip and spread this is expanding as 8.20+ rolls into base images.

THE FIX (WHERE IT ACTUALLY BELONGS)

Fixing my crawler stopped my pain, but it does nothing for the next visitor. The real fixes are two, and either one alone breaks the loop:

1. Make the cache key case-sensitive on escapes — or canonicalize the case before hashing, so %E9 and %e9 don’t collide into one poisonable entry.
2. Stop the origin redirecting on hex case. Serve 200 for both forms, exactly as the spec’s equivalence rule already requires.

If you run a cache, one quick thing worth checking: does an inbound Cache-Control: no-cache actually bypass it, or does your config strip that header? If it’s stripped, a poisoned client can’t even rescue itself — a request for a clean copy still gets the cached 301. That’s the difference between “annoying” and “outage,” and it’s one curl -I to find out.

THE MORAL

Nobody shipped a bug here. The spec said “should.” The browser vendors made a defensible call. The Python library followed the RFC. The cache did a normal optimization. Every single piece passes its own conformance test.

The outage lives in the seams — where two systems each did the reasonable thing, in opposite directions, and a cache turned the disagreement into a shared, sticky failure instead of letting it cancel out.

So when you’re staring at something that “cannot possibly be happening” — the same page loads in your browser and loops in your script — stop looking for the broken component. Look at the seams between the correct ones.

%E9 and %e9 are the same. Right up until an entire language section of a website goes dark.