THE INSPIRATION
Earlier this year, John Mueller did something delightfully absurd. He turned his robots.txt into an audio file. Not a link to an audio file. The actual robots.txt file was a WAV file that you could play and hear him read the directives aloud.
How? The robots.txt specification ignores anything it doesn’t understand. Comments, random text, ASCII art, binary data - the parser just skips past it looking for valid directives. Mueller exploited WAV’s chunked container format to embed valid robots.txt content inside a playable audio file.
Crawlers saw directives. Browsers heard his voice. Same file.
A DIFFERENT FORMAT, SAME TRICK
WAV files require audio software to appreciate. But what if the “hidden layer” could render directly in the browser? What if opening robots.txt transformed into an experience?
Enter XML and XSLT.
The robots.txt on this site is technically an XML document. When a browser requests it, it receives:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="/robots.xsl"?>
<robots-txt xmlns="https://wunsch.dk/robots-txt">
User-agent: *
Allow: /
User-agent: Bytespider
Disallow: /
...
</robots-txt>That xml-stylesheet processing instruction tells the browser to transform the document using an XSL stylesheet. The stylesheet renders a full HTML page - with CSS, animations, and JavaScript.
WHY IT WORKS FOR CRAWLERS
Here’s the thing: robots.txt parsers are remarkably forgiving. The specification states that any line not matching the expected format should be ignored. Our XML declaration? Ignored. The processing instruction? Ignored. The XML tags wrapping the content? Ignored.
The parser finds User-agent: *, finds Allow: /, finds our other directives, and carries on. It’s valid robots.txt as far as crawlers are concerned.
Google’s Gary Illyes confirmed this behavior when Mueller’s audio file made the rounds. Googlebot parsed the WAV file and extracted the robots.txt directives from within. Our XML file gets the same treatment.
MYSTERY SCIENCE THEATER
Since we’re in the business of transforming boring files into experiences, we went full MST3K.
Visit /robots.txt in a browser and watch as:
- The plain text appears (briefly looking like a normal robots.txt)
- The lights dim - the background goes dark
- Red velvet curtains slide in from the sides
- The text becomes a movie screen
- Silhouettes rise from the bottom - Crow T. Robot, Joel/Mike, and Tom Servo
- They proceed to heckle the robots.txt
The banter includes gems like:
“Bytespider? A spider made of bytes? Sounds like a diet plan.”
“Wait… HAL9000?! THE HAL9000?!”
“I’m sorry Dave, I’m afraid you can’t crawl that.”
Yes, we added HAL9000 to our robots.txt just for the joke.
THE TECHNICAL BITS
The XSLT stylesheet does the heavy lifting. It’s a complete web application embedded in an XML transformation:
- CSS theater staging - Curtains, screen positioning, silhouette animations
- Pure CSS character silhouettes - Crow’s spiky head, Tom’s gumball dome, the human’s generic outline
- JavaScript timing - Subtitle display, character “talking” animations, replay functionality
- Responsive design - Works on mobile with scaled-down theater
The robots.txt content is extracted using <xsl:value-of select="r:robots-txt"/> and displayed as preformatted text on the “screen.” Everything else is the theater experience wrapper.
THE CONSTRAINT
Unlike Mueller’s WAV approach, we can’t serve arbitrary binary content. We’re limited to what can be expressed as valid XML. But for text-based files like robots.txt, this isn’t a limitation - it’s perfect.
The XML wrapper is minimal. The content remains readable in view-source. The transformation only happens when the browser processes the stylesheet.
WHAT ABOUT SITEMAP.XML?
We did the same thing with our sitemap - except that one becomes a playable Space Invaders game. Apparently we have a thing for transforming infrastructure files into entertainment.
CREDIT WHERE DUE
Full credit to John Mueller for the original inspiration. His WAV file trick demonstrated that robots.txt’s permissive parsing could enable creative dual-format files. We just swapped audio for visual theater.
Sometimes the most interesting creative constraints come from the most boring files on your website.
IN THE NOT TOO DISTANT FUTURE, NEXT SUNDAY A.D., THERE WAS A GUY NAMED JOEL, NOT TOO DIFFERENT FROM YOU OR ME…