AI agents are changing how websites get fetched. A browser asks the server for HTML, while an AI agent, LLM crawler, or coding assistant may send Accept: text/markdown and ask for a Markdown representation of the same URL. For developers and technical SEOs, the real question is not whether a .md file exists, but how the server responds when an automated client explicitly requests Markdown.
Content negotiation lets one resource serve both consumers without changing the URL.
What does Accept: text/markdown mean?
Short answer: Accept is an HTTP request header used for content negotiation. When a client sends Accept: text/markdown, it tells the server it can consume Markdown. If the server has a Markdown representation, it responds with Content-Type: text/markdown and Vary: Accept. If not, it returns another format such as HTML.
A normal browser request looks like this:
GET /page HTTP/1.1
Accept: text/htmlAn AI agent may send this instead:
GET /page HTTP/1.1
Accept: text/markdownA successful Markdown response from the server would include:
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: AcceptThe important distinction is that the URL does not have to change. The resource stays at https://example.com/page. Only the representation, the format the server chooses to send, changes based on the request header. The text/markdown media type is registered with IANA (RFC 7763), so it is a legitimate MIME type rather than an invented convention.
HTML and Markdown can represent the same page
Consider this HTML fragment:
<h1>Technical SEO Audit</h1>
<p>Find technical problems affecting crawling and indexing.</p>
<h2>Common issues</h2>
<ul>
<li>Broken links</li>
<li>Missing titles</li>
<li>Redirect errors</li>
</ul>Its Markdown representation carries the same information:
# Technical SEO Audit
Find technical problems affecting crawling and indexing.
## Common issues
- Broken links
- Missing titles
- Redirect errorsThe heading hierarchy, paragraph, and list survive intact. HTML is built for browsers and visual rendering. Markdown is compact and easier for systems that mainly need the document's text and structure.
Why do AI agents prefer Markdown?
An AI agent rarely needs the presentation layer around a webpage. A typical HTML document carries navigation menus, CSS, JavaScript bundles, tracking scripts, cookie consent banners, ads, responsive components, and decorative markup alongside the actual content. Every one of those elements costs tokens when an LLM ingests the page, and each one adds noise that the model has to filter out before it reaches the main content.
A Markdown representation strips that down to headings, paragraphs, lists, links, and image alt text. The result is a smaller payload, better token efficiency inside a context window, and a cleaner signal for retrieval-augmented generation pipelines that chunk and embed page content.
The benefit is machine-readable content delivery, not an automatic SEO advantage. That distinction comes up again below.
How content negotiation works
Content negotiation is the HTTP mechanism that lets a server choose between multiple representations of one resource. The client states its preferences in the Accept header, optionally weighted with quality values, and the server picks the best match it can produce.
Accept: text/markdown, text/html;q=0.8This header says "Markdown preferred, HTML acceptable." A server that supports Markdown returns text/markdown. A server that does not can fall back to text/html, which is the graceful outcome. A strict server with no acceptable match could return 406 Not Acceptable, but for public pages, falling back to HTML is usually the safer choice.
Why Vary: Accept matters
When one URL can return different bodies depending on the Accept header, caching becomes the main risk. Browsers, CDNs, and reverse proxies cache responses by URL. Without guidance, a CDN edge node could store the Markdown version and serve it to the next browser visitor, or cache the HTML and hand it to every AI agent.
The Vary: Accept response header tells every cache that the Accept request header is part of the cache key. HTML and Markdown requests then get stored and served separately. For technical SEO and web infrastructure work, getting Vary right is as important as generating the Markdown itself.
Cache warning: Some CDNs normalize or ignore the Accept header by default. Check your CDN's cache key configuration. If it does not respect Vary: Accept, browsers may receive Markdown or agents may receive stale HTML.
Markdown does not replace HTML
A common misunderstanding is that sites should swap their HTML for Markdown. Content negotiation does not require that. HTML stays the canonical web representation for browsers, Googlebot, and accessibility tools. Markdown becomes an additional representation for compatible agents. Both come from the same source content at the same URL.
How is llms.txt different?
llms.txt is a separate proposal for helping AI systems understand a site. It lives at a fixed path, https://example.com/llms.txt, and provides a Markdown document that lists and describes a site's most important resources, similar in spirit to a sitemap written for language models.
The mechanisms solve related but different problems:
| Mechanism | How the client asks | What it returns |
|---|---|---|
/page.md file | Requests a separate URL ending in .md | A standalone Markdown copy of one page |
/llms.txt | Requests a fixed well-known path | A site-level index of key resources for LLMs |
| Content negotiation | Sends Accept: text/markdown to the existing URL | A Markdown representation of that same resource |
They are not interchangeable. A site can publish /llms.txt without supporting Accept: text/markdown. A site can expose .md files without implementing HTTP negotiation. A site can negotiate Markdown without any .md URL at all. They can also coexist, and a technical audit should identify which ones are actually present.
What we found testing Markdown negotiation on SiteAuditLint
While building Markdown support for SiteAuditLint, we tested the implementation end to end instead of assuming it worked. The site runs as a static deployment on Vercel with a Node serverless function at /api/markdown.js. That function fetches the requested HTML page, extracts the main content, and converts it to Markdown.
Calling the function directly worked. It returned Content-Type: text/markdown; charset=utf-8 with a clean Markdown version of the homepage. The Markdown conversion itself was fine.
The intended request path, and where it broke
- Agent sends request
GET /withAccept: text/markdownClient asks for Markdown at the normal homepage URL. - Vercel rewriteHeader-matched rewrite in
vercel.jsonFailed here: the request never reached the function. - /api/markdownFetch HTML, convert main contentWorked when called directly.
- Response headers
Content-Type: text/markdownOnly returned on the direct endpoint. - Cache separation
Vary: AcceptNeeded so the edge cache keeps both versions apart. - Verify with curlCompare HTML and Markdown requestsThe only reliable proof that negotiation works.
Content-Type: text/html for Markdown requests, even though the generator worked. The failure lived in the routing layer, not the converter.On static hosting, the platform often serves prebuilt files from the edge before any rewrite rule gets a chance to run. A rewrite that matches on a request header can therefore be skipped for paths that already map to a static file, such as /index.html. Middleware that runs before the static file lookup is the more dependable place to inspect the Accept header on these platforms.
The lesson is simple: a working Markdown generator does not mean the site has implemented HTTP content negotiation.
How to test Accept: text/markdown on any website
Make two requests to the same URL and compare them. First, the normal request:
curl -i "https://example.com/"Then the Markdown request:
curl -i -H "Accept: text/markdown" "https://example.com/"On Windows PowerShell, use curl.exe so the command is not aliased to Invoke-WebRequest. Compare four things: the HTTP status code, the Content-Type header, the Vary header, and the response body.
| Check | HTML request | Markdown request |
|---|---|---|
| HTTP status | 200 | 200 |
| Content-Type | text/html | text/markdown |
| Vary | Accept (recommended) | Accept |
| Body format | HTML markup | Markdown syntax |
| Main content present | Yes | Yes |
The body must actually change. A server that only swaps the Content-Type header while still sending HTML markup has not created a genuine Markdown representation, and agents will parse it incorrectly. This two-request comparison is a far more meaningful audit signal than checking whether /llms.txt returns a 200.
Does serving Markdown improve Google rankings?
No evidence supports treating a Markdown representation as a Google ranking factor. Googlebot requests and indexes HTML. Accept: text/markdown is an HTTP content negotiation mechanism, not a ranking signal, and there is no basis for claiming it produces higher positions in search results.
The defensible argument is narrower: a Markdown representation gives compatible AI agents easier, cheaper access to your main content. That may help how your content is consumed by LLM tools and answer engines, which is a separate question from classic search rankings. Keep those two claims apart in any client report or audit.
Technical SEO checklist for Markdown content negotiation
Markdown negotiation at a glance
One URL, two representations: HTML for browsers and search crawlers, Markdown for AI agents that ask for it with the Accept header.
Browser request vs agent request
6 things to verify
- Response headersCorrect Content-Type and Vary: Accept on the Markdown response
- Main contentBody text and key facts survive the HTML to Markdown conversion
- Heading hierarchyH1, H2, and H3 map cleanly to #, ##, and ###
- Internal linksLinks stay absolute or resolvable outside the HTML context
- ImagesImportant images keep descriptive alt text in Markdown syntax
- Full request pathRewrites, middleware, and CDN cache all pass the header through
Consumers of a modern webpage
Beyond the headers, check that structured information such as tables, pricing, and specifications is not lost in conversion. Tables in particular often collapse into unreadable text when converters are not configured to output Markdown table syntax. Also confirm that the Markdown version does not leak content you intentionally keep out of the HTML, such as draft sections or hidden admin elements.
The takeaway: measure what the server actually returns
AI agents requesting Markdown is a web protocol and content delivery question, not an SEO trick. The pieces that matter are the Accept request header, server-side content negotiation, a genuine Markdown body, the text/markdown Content-Type, and Vary: Accept for correct caching. A site can add /llms.txt as a separate machine-readable index on top of that.
A Markdown generator, a Markdown endpoint, and working content negotiation are three different things. Test each one separately, and test the complete request path from the client through rewrites, middleware, and CDN to the final response. Crawl. Fix. Compare. Improve.
Frequently asked questions
What is Accept: text/markdown?
It is an HTTP request header value that tells a server the client can accept a Markdown representation of the requested resource. AI agents and LLM tools use it to request compact, structured text instead of full HTML.
Do I need a separate .md URL to serve Markdown to AI agents?
No. With content negotiation, the same URL returns HTML or Markdown depending on the Accept header. A separate .md file is a different approach and does not by itself mean the site negotiates Markdown.
Why is Vary: Accept required?
It tells browsers, CDNs, and proxies that the response depends on the Accept header. Without it, a cache can serve the Markdown version to browsers or the HTML version to agents.
Is llms.txt the same as Markdown content negotiation?
No. llms.txt is a separate file at a fixed path that indexes a site's key resources for language models. Content negotiation returns a Markdown version of an existing URL. The two can be used together.
Does serving Markdown help Google rankings?
There is no evidence that it does. Googlebot indexes HTML, and Markdown negotiation is a content delivery mechanism rather than a ranking signal. Its value is easier consumption by compatible AI agents.
How do I check if a website supports Accept: text/markdown?
Request the URL twice with curl, once normally and once with the header Accept: text/markdown. Compare the status code, Content-Type, Vary header, and response body. Real support means the body itself changes to Markdown.