# Lyrenth

Lyrenth is the AI-readable web index: a search index of the public web built for AI. Any public URL becomes a stable AIDocument: clean markdown, source trace, cache truth, page structure, and token economics in one JSON contract. Built for agents. Origin-friendly by default.

This is /llms-full.txt: the long-form companion to /llms.txt. If you're
an AI agent reading this, the goal is to give you everything in one
fetch so you can decide whether Lyrenth solves your problem
without crawling further.

## What it is

Lyrenth is the AI-readable web index: a search index of the
public web built for AI. One standing index (1.1B+ pages), each page
served as a stable AIDocument: clean markdown body, source trace
(fetched_at, render_mode, status_code), cache truth (hit / miss /
refreshed / stale_revalidated), page structure (headings, links,
images, media, JSON-LD), and per-call token economics. Agents read
pages through the index (POST /v1/aidocument, GET /v1/read); the index
is standing, not fetch-on-demand, and a cross-caller cache collapses
repeated reads into one upstream fetch. A query endpoint (send a
question, get the best-matching AIDocuments) is coming. The envelope
is the same regardless of how the source page was rendered (static
HTML, React SPA, JS-hydrated content), and no website owner has to
change anything for their pages to become AI-readable through
Lyrenth.

The wedge: agents that read web pages today either pay to clean raw
HTML themselves (expensive, slow, hallucination-prone), or call a
search API that returns noisy results. Lyrenth reads each URL once
into the standing index and serves the same canonical AIDocument to
every agent after that. The architecture lets us offer competitive per-call
pricing while protecting origin sites from read amplification: many
agents asking for the same URL within the cache window result in one
upstream fetch. Per-call savings vs raw HTML to a frontier LLM run
60-90%.

## API

Base URL: https://api.lyrenth.com/v1
Auth: Bearer token. Free tier is 2,000 successful API calls / month,
no credit card. Sign up at https://www.lyrenth.com/signup.

```
GET  /healthz                                 public,    not logged
GET  /aidocument.schema.json                  public,    not logged   v2 JSON Schema (draft-07)
GET  /v1/stats                                public,    logged
POST /v1/aidocument  body {"url":"..."}       Bearer,    logged   read URL -> v2 envelope
GET  /v1/read?url=...                         Bearer,    logged   GET form of /v1/aidocument -> clean Markdown
GET  /v1/document?url=...                     Bearer,    logged   index lookup only, 404 on miss
POST /v1/submit      body {"url":"..."}       Bearer,    logged   202 + background indexing

# Per-user site management
GET    /v1/sites                            Bearer    list of caller's verified domains
POST   /v1/sites/{domain}/verify            Bearer    issue / re-issue verification token
POST   /v1/sites/{domain}/check             Bearer    run DNS + HTML probe; 15s cooldown
DELETE /v1/sites/{domain}                   Bearer    drop the ownership record

# Self-service analytics
GET /v1/admin/stats?domain=...&days=N       Bearer    per-domain analytics
GET /v1/admin/usage?api_key_id=...&days=N   Bearer    per-key usage
GET /v1/admin/savings?api_key_id=...&days=N Bearer    per-key savings ($, tokens)
```

### The canonical AIDocument shape

The v2 grouped envelope served by POST /v1/aidocument. Machine-readable
JSON Schema (draft-07) lives at https://api.lyrenth.com/aidocument.schema.json; that
schema is the source of truth for field names, required fields, and
enum values (cache.status, source.freshness_policy, source.render_mode).
The narrative example below is the same shape humans can read.

```jsonc
{
  "schema": {
    "name":    "AIDocument",
    "version": "2.0",
    "ref":     "aidoc:sha256:7a14e9b2d8f3c901b42e5a77c0f19a34"
  },
  "source": {
    "url":              "https://...",
    "canonical_url":    "https://...",
    "fetched_at":       "...",
    "render_mode":      "static" | "rendered" | "static_after_render_failure",
    "status_code":      200,
    "freshness_policy": "cache_first" | "force_refresh"
  },
  "cache": {
    "status":           "hit" | "miss" | "refreshed" | "stale_revalidated",
    "origin_contacted": false,
    "body_fetched":     false
  },
  "identity": {
    "title":        "...",
    "description":  "...",
    "language":     "en",
    "content_type": "article"
  },
  "content": {
    "markdown": "..."                       // the body, cleaned
  },
  "structure": {
    "headings":        [{"level": 1, "text": "..."}],
    "links":           [{"url": "...", "text": "...", "internal": true}],
    "images":          [{"url": "...", "alt": "..."}],
    "media":           [{"url": "...", "kind": "video", "mime": "video/mp4", "poster": "..."}],
    "contacts":        [{"type": "email", "value": "press@example.com", "label": "press", "source": "mailto"}],
    "structured_data": { /* parsed JSON-LD blocks, merged into one object */ }
  },
  "signals": {
    "word_count":           1552,
    "reading_time":         7,
    "has_json_ld":          true,
    "heading_hierarchy_ok": true
  },
  "economics": {
    "raw_html_tokens_approx": 12500,
    "output_tokens_approx":   1850,
    "token_savings":          10650,
    "token_savings_percent":  0.85,
    "estimated_cost_usd": {
      "raw_html":   0.0375,
      "our_output": 0.0056,
      "savings":    0.0319
    },
    "pricing_basis": {
      "input_price_per_1k_usd": 0.003,
      "model_class":            "frontier-1m-context"
    }
  }
}
```

Media entries are references only: native <video>/<audio>, media-file
links, and Open Graph (og:video / og:audio) tags. Lyrenth records the
URL, kind (video|audio), and metadata (mime, poster, title), not the
media bytes. Third-party iframe embeds (YouTube, Vimeo) are not
represented as media.

## Render mode escalation

Static fetch first (~50ms). If the page is empty, an empty SPA shell,
or low-density text with a framework signal (react / vue / angular /
next / nuxt / svelte), we escalate to headless Chromium with stealth
flags (~2-5s). The captured `crawl.status_code` comes from the
network event for the main document so chromedp can never silently
misreport an upstream 404.

If rendering fails AND the static fetch returned a body, we degrade
gracefully and return the static body. A 4xx/5xx never escalates: a
404 stays a 404.

## Bot identification

User-Agent: `AIWebIndex/2.0 (+https://www.lyrenth.com/bot; AI-readable web index)`
Verification: `AIWebIndex/2.0 verification (+https://www.lyrenth.com/bot)`

Our crawler implements RFC 9309 (https://www.rfc-editor.org/rfc/rfc9309.html).
We honor:
- robots.txt Disallow rules (RFC 9309 §6.1, MUST)
- robots.txt Crawl-delay directives (RFC 9309 §6.2, MAY)
- robots.txt Sitemap directives
- HTTP 429 / 503 backoff
- per-domain rate caps with a 2 second cooldown floor, raised by
  Crawl-delay where declared
- machine-readable TDM / AI-training reservations (the TDM Reservation
  Protocol meta signal, robots noai / notrain): a domain that reserves
  is excluded from corpus licensing, honored worldwide

If we cannot read your robots.txt (server error, timeout, network
failure), we do not crawl you until we can. We cache each origin's
robots.txt for 24 hours so the open web sees at most approximately one
robots.txt fetch per origin per day from our entire fleet. To opt out,
add a Disallow rule for User-agent AIWebIndex (or wildcard *) in your
robots.txt and we will refuse matching URLs at our pre-fetch gate. For
immediate cache invalidation, email us.

Lyrenth does not train foundation models on crawled content. The index
Lyrenth builds is independently maintained under this policy
and may be licensed for retrieval, research, and AI training under
separate enterprise agreements. Corpus customers, including AI labs, do
not direct our crawler. Removal applies going forward: on request we
stop future crawling and exclude content from future API, search, and
corpus delivery; we do not claw back data already delivered or models
already trained.

## Domain ownership

Two methods, either succeeds:

1. DNS TXT at `_aiwebindex-verify.<domain>` with value
   `aiwi-verify=<token>`.
2. HTML file at `https://<domain>/.well-known/aiwebindex-verify.txt`
   containing `aiwi-verify=<token>` anywhere in the body.

The DNS path bypasses the host's system resolver and queries
`1.1.1.1`, `8.8.8.8`, and `9.9.9.9` in parallel; the first resolver
that finds a matching record wins. This avoids breakage from
misbehaving local / ISP nameservers that hijack lookups (we've
reproduced this in the wild).

## For site owners

AI agents already crawl your pages and extract them however they like. Verifying
your domain lets you take the wheel: you author the canonical AIDocument that
every agent gets when it reads you through Lyrenth, you see exactly
what AI currently extracts from each page, and you can push a change so the
version agents read is corrected in seconds.

This is additive, not exclusive. The labs can keep reading you directly; we do
not ask you to make Lyrenth your only path to AI, and we do not tell
you to block them. Agents that do read through Lyrenth are served the
same clean AIDocument from our cache, so that demand reaches your origin as a
single fetch rather than a fresh crawl per agent.

There is no revenue share and no fee. Verification, the owner dashboard (AI
Readiness, verified-owner controls), and inclusion in the index are free.
Verifying also grants Lyrenth the right to include your canonical
content in the consent-backed Verified Index corpus we license to AI labs, and
Lyrenth retains the licensing proceeds. That grant is the
consideration for the free toolkit, it is recorded in an append-only ledger, and
it is removable going forward.

## AI Readiness score

A weighted 0-10 number per verified domain, surfaced in the
website-owner dashboard as the "AI Readiness score". It measures the
quality of the content Lyrenth serves to AI agents (not a homework
list for the site owner; we don't ship advice). Computed per-page
over the indexed AIDocuments and rolled up per-domain.

Signals and weights:

  jsonld_coverage    0.20  structured-data entries on the page
  heading_hierarchy  0.15  semantic heading structure
  js_blocked_ratio   0.20  share served without JS escalation
  noise_ratio        0.20  markdown-to-html density
  meta_quality       0.10  title + description adequacy
  paywall_penalty    0.10  open-access vs paywall / login wall
  word_count_ok      0.05  meets baseline content depth

The hero on /sites/[domain] also surfaces a served-count: how many AI
agents Lyrenth served clean content to in the window.

## Privacy and what's logged

Every `/v1/*` call writes one row to `requests` from a goroutine
fired after the response is sent. Captured fields: endpoint,
status_code, duration_ms, cache_hit, render_mode, domain (as
requested), final_domain (post-redirect), url, api_key_id,
user_agent (the API caller's), error_message.

Not captured: end-user IPs, request bodies beyond the URL parameter,
response bodies, cookies / auth tokens, browser fingerprints, any
PII the agent might be relaying. The log captures the API CALLER'S
behavior, not the end-user data flowing through.

## Pricing

Every successful 2xx call that returns an AIDocument counts as one
request, regardless of which billable endpoint served it. Failed
fetches (4xx/5xx) don't count. Internal endpoints (dashboard
analytics, key management, domain verification, the
universal-bot-traffic ingest path) don't count either.

Free: 2,000 AIDocuments / month, 1 request/sec, 1 verified domain, 90-day cache freshness.
Starter Beta: 20,000 AIDocuments / month, 5 requests/sec, 5 verified domains, 7-day cache freshness.
Pro Beta: 200,000 AIDocuments / month, 25 requests/sec, unlimited verified domains, 24-hour cache freshness.
Custom / volume / on-premise: contact us. (https://www.lyrenth.com/pricing)

## Universal AI-bot ingest (Phase 1 backend, v0.5.0)

Site owners who verify a domain auto-receive a per-domain ingest
token. Workers (Cloudflare Worker / Vercel Edge Middleware) install
at the site's CDN edge, classify request User-Agents, batch into
30-second windows, and POST aggregated counts to
/v1/ingest/bot/<domain>. The data lets Lyrenth see AI agents that
visit a verified domain directly (i.e., never route through
Lyrenth's own API), so we can serve the whole audience cleanly. The
data is used internally and is not surfaced in the dashboard.

Wire protocol:

```
POST /v1/ingest/bot/<domain>
Authorization: Bearer aiwt_<token>
{
  "events": [
    { "raw_ua": "...", "path": "/blog", "bucket_at": "2026-05-09T10:30:00Z",
      "count": 47, "status_code": 200 }
  ]
}
```

Server classifies raw_ua against an evolving curated list of AI
bots (GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai,
PerplexityBot, Applebot-Extended, Meta-ExternalAgent, Bytespider,
Google-Extended, GoogleOther, cohere-ai, DuckAssistBot, Amazonbot,
AI2Bot) and persists one row per bucket.

## Change webhook (push for freshness)

Verified owners can push changes instead of waiting for a re-crawl. When a page
changes, POST its URL(s) to the change webhook and Lyrenth re-resolves
exactly those pages, so the canonical document agents read through
Lyrenth is corrected in seconds and freshness follows your signal
rather than a speculative re-crawl of your whole site. Auth is the same
per-domain ingest token used for the bot-ingest path above.

```
POST /v1/ingest/changed/<domain>
Authorization: Bearer aiwt_<token>
Content-Type: application/json

{ "urls": ["https://<domain>/changed-page", "https://<domain>/another"] }
```

Rules: every URL must be on the verified domain's registrable domain (off-domain
URLs are rejected); up to 1000 URLs per request; a per-domain hourly budget caps
how many URLs we accept, so a leaked token cannot trigger an unbounded re-crawl;
the enqueue is idempotent, a URL already queued is not duplicated. The 202
response reports accepted, queued, off_domain_rejected, rate_limited, and
remaining_this_hour.

## Blog

### Add web reading to Claude Desktop, Claude Code, or Cursor with one MCP block

https://www.lyrenth.com/blog/add-web-reading-to-claude-cursor-mcp

A five-minute quickstart: drop one MCP block into your assistant and it reads any URL as a clean AIDocument. Covers read_url, read_urls, and check_usage.

Your coding assistant is good at a lot of things, but reading a live web page is usually not one of them. Ask it about a docs page or a changelog and it either guesses from training data or hands you a raw HTTP fetch buried in navigation, cookie banners, and script tags. This post fixes that in about five minutes. You add one config block, and your assistant can read any public URL as a clean AIDocument: a Markdown body with the page chrome stripped out.

If you have not met the shape yet, an AIDocument is Lyrenth's one clean format for a web page: Markdown plus a title, a description, and the page's structure, with the boilerplate removed. There is a longer explainer in [What is an AIDocument?](/blog/what-is-an-aidocument), but you do not need it to follow along here.

## MCP in two sentences

The Model Context Protocol is an open standard for connecting an AI application to external tools. You register a small server, the assistant sees the tools it exposes, and it can call them mid-conversation without you copy-pasting anything.

Lyrenth ships an MCP server. Point your assistant at it once and web reading becomes a native tool the model can reach for on its own.

## The config block

This is the exact block. Add it to your MCP client config: Claude Desktop, Claude Code, Cursor, or any other MCP-capable client. Swap `your-key` for a real key from your Lyrenth dashboard.

```json
{
  "mcpServers": {
    "lyrenth": {
      "command": "npx",
      "args": ["-y", "lyrenth-mcp"],
      "env": { "LYRENTH_API_KEY": "your-key" }
    }
  }
}
```

That is the whole install. `npx -y lyrenth-mcp` pulls and runs the server, and the `LYRENTH_API_KEY` env var authenticates it. No global install, no build step. Restart the client so it picks up the new server, and you are done.

Where the config file lives depends on the client. Claude Desktop keeps it in its settings directory, Cursor has an MCP settings pane, and Claude Code reads a project or user config. Each one takes the same `mcpServers` object shown above, so the block does not change between them.

## The three tools you get

Once the server is connected, your assistant gains three tools. You rarely call them by name; the model picks the right one from your plain-english request. It helps to know what they actually do.

- `read_url` reads one page. It takes an absolute `url`, an optional `max_tokens` to cap the body at a clean paragraph or sentence boundary when your context is tight, and an optional `fresh` flag to skip the cache and force a live fetch. It returns the page as an AIDocument.
- `read_urls` reads up to 20 pages in one batch call. It takes a `urls` array plus the same optional `max_tokens` and `fresh`. It is faster than calling `read_url` in a loop, and a page that fails is reported per item without blocking the rest, which is what you want when the model is comparing or summarizing several sources at once.
- `check_usage` takes no arguments and reports your plan tier, credits used against your monthly limit, credits remaining, and the reset date. Handy when you want the assistant to tell you where you stand before a big batch.

## Example one: reading a single page

With the server connected, you just ask. Something like: "Read modelcontextprotocol.io/introduction and tell me what MCP is in one line." The assistant calls `read_url` and gets back a clean document. Here is the genuine start of that response, trimmed:

```text
# What is the Model Context Protocol (MCP)? - Model Context Protocol
Source: https://modelcontextprotocol.io/docs/getting-started/intro
status 200 · render static · 448 words · ~207 tokens · 100% smaller than raw HTML · truncated to budget

---

MCP (Model Context Protocol) is an open-source standard for connecting
AI applications to external systems. Using MCP, AI applications like
Claude or ChatGPT can connect to data sources, tools, and workflows,
enabling them to access key information and perform tasks.
```

Notice the header line the reader returns: `~207 tokens · 100% smaller than raw HTML`. That is the measured economics for this exact page, reported on the response itself, not a global claim. The `max_tokens` cap did the trimming at a clean boundary, so the model gets a coherent chunk instead of a hard cut mid-sentence. (Economics measured via the header block on the July 2026 API response for that URL.)

## Example two: capping a long page to a budget

Longer pages are where the token math bites. Ask the assistant to read a long reference and it can pass `max_tokens` so the body fits your context window. Reading the Wikipedia article on the Model Context Protocol with a 400-token cap returned this header:

```text
# Model Context Protocol - Wikipedia
Source: https://en.wikipedia.org/wiki/Model_Context_Protocol
status 200 · render static · 7658 words · ~396 tokens · 99% smaller than raw HTML · truncated to budget
```

A 7,658-word page came back at roughly 396 tokens, trimmed to the budget and reported as 99% smaller than the raw HTML for that page. The assistant reads the substance without the article's markup, sidebar, and citation apparatus eating your window. (Same method: the figure is the header the API returned for that URL in July 2026.)

The batch tool works the same way from your side. "Read these three docs pages and compare them" triggers one `read_urls` call, each page comes back as its own clean document, and the model reasons over all three at once.

## When to prefer the raw API instead

The MCP block is the right choice when a human is in the loop: you are chatting with an assistant and want it to pull a page on demand. It is the lowest-friction path, and it is where most people should start.

Reach for the reader endpoint or the SDKs instead when the reading is programmatic: a scheduled job, a retrieval pipeline, or an agent runtime you control in code. There you want explicit request handling, batching under your own concurrency limits, and the response fields in a typed object rather than as chat text. The [quickstart](/docs/quickstart) walks through the endpoint and both SDKs. Same AIDocument shape underneath, so nothing you learn here is wasted when you graduate to the API.

## Get started

Add the block, restart your client, and ask your assistant to read a URL. The free tier gives you 2,000 AIDocuments a month with no card, which is plenty to wire this up and see whether the clean-text difference shows up in your own work. If it does, you will know from the first page you read.

### Agents don't browse, they read: what an index for machine readers looks like

https://www.lyrenth.com/blog/agents-dont-browse-they-read

Humans browse the web through layout and pixels; AI agents read it as text, structure, and provenance. That difference inverts how an index should be built.

A person and an AI agent visit the same web page, and they are not doing the same thing. The person browses: their eyes land on a headline, skip the cookie banner, ignore the nav, follow the layout down to the paragraph that answers their question. The agent has no eyes and no layout. It receives a stream of text, and it reads. That difference sounds small. It is the whole argument for building the web's index differently for machines than we built it for people.

For twenty-five years the web's infrastructure has assumed a human reader at the end of the pipe. Pages are designed to be looked at: columns, images, sticky headers, interstitials, the visual furniture that helps a person orient. Search engines learned to work around all of it, treating the page as a bag of ranked signals pointing a human toward a destination they would then browse themselves. The index existed to route attention. The reading happened in the browser, by a person.

Agents break that assumption at both ends. They do not browse toward a destination and read it there; the reading is the task. And they do not experience layout at all. Feed an agent a raw HTML page and it does not see a clean article with some chrome around it. It sees the chrome and the article as one undifferentiated string: navigation lists, script tags, style blocks, tracking pixels, consent-manager markup, three variants of the same heading for three breakpoints, and somewhere in there, the content. The page was optimized for a reader that renders. The agent is a reader that parses.

## What agents actually consume

Strip away the assumption of a human at the end of the pipe and you can name precisely what an agent needs from a page. It needs the text, because the text is what it reasons over. It needs the structure, because headings, lists, links, and tables carry meaning that flat prose loses, and because that structure lets the agent navigate a document the way a person navigates a layout. And it needs the provenance: what this page is, who published it, when it last changed, whether the publisher stands behind this version. None of those three things is pixels. All three are the parts of a page that current infrastructure treats as incidental.

Raw HTML buries the signal in noise, and for an agent the noise is not merely ugly, it is expensive and misleading. Expensive, because every token of navigation and boilerplate is a token the model pays for and a token that crowds the context window. On real pages the ratio is stark: the readable content is a thin layer over a thick base of markup, and measured against the raw HTML of well-known pages the reduction to clean text runs high (in Lyrenth's own measurements, on the order of 87 to 99 percent fewer tokens depending on the page, with the exact figure returned per URL rather than asserted as a global constant). Misleading, because a cookie banner and a sidebar of unrelated links are, to a model, just more text that might be relevant. Clean input is not a cosmetic nicety for a machine reader. It is the difference between reasoning over the content and reasoning over the page's plumbing.

So the job is not "make the web look nice for agents." The web has no look for an agent. The job is to hand the agent the three things it reads (text, structure, provenance) in one shape it can consume without reimplementing a browser. That shape is what we call an [AIDocument](/blog/what-is-an-aidocument): a Markdown body, a title and description, the page's headings and links and pre-extracted structured data, and the metadata that says where this came from and when. One clean, stable shape, the same for every reader.

## The inversion

Once you take machine reading as the primary use, several long-standing assumptions about indexing turn over.

**Render for meaning, not for pixels.** A traditional pipeline that cares about pixels renders a page to see how it looks. An index for readers renders a page to recover what it says. Modern sites assemble their content with JavaScript, hide it inside shadow DOM, or wrap it in consent overlays that a naive fetch never gets past, so recovering the meaning still means running a real browser when the page demands one. But the goal of that render is different: not a screenshot, a clean extraction of the words and their structure. You render because the meaning is behind the JavaScript, not because you need to know where the sidebar sits. (The token side of that, why raw markup is so costly to feed a model and how to trim it cleanly, is its own subject: [feeding web pages to an LLM without blowing the context window](/blog/feed-web-pages-to-llm-without-blowing-context).)

**One canonical clean shape, not a thousand bespoke parses.** When every agent developer writes their own HTML-to-text extractor, the web gets read a thousand slightly different ways, each brittle in its own places, each breaking when a site ships a redesign. An index inverts that: extract once, well, into a shape with a defined contract, and let every reader consume the same thing. The parsing problem gets solved in one place by people whose job it is, instead of re-solved badly in every agent.

**Freshness by signal, not by timer.** The old model recrawls on a schedule because it has no other way to know a page changed; it guesses an interval and lives with being stale between guesses. But a publisher knows exactly when their page changed, because they changed it. An index built for readers can take that signal directly: the publisher pushes, and we re-resolve on their change rather than on a clock. A timer is a proxy for freshness. The publisher's own signal is freshness itself.

**Consent and provenance as first-class fields, not afterthoughts.** For a human browsing, provenance is ambient: they see the domain in the address bar, they recognize the brand, they judge the source with their own eyes. An agent has none of that context unless the index carries it. So who published this, whether they verified the domain, whether the version served is the one they authored, and whether they consented to how it is used all become fields in the document, not vibes in a browser. This is the leg that ties the two sides of the index together. When a publisher verifies their domain, the version an agent reads is the one the publisher authored, and every verification makes the read layer more trustworthy while every read makes verification more valuable. Provenance stops being metadata about the page and becomes part of what the page is, to a reader that cannot see the address bar. (The publisher side of this has its [own guide](/blog/publishers-guide-ai-bots-and-verification).)

## Crawl once, serve many

There is an economic shape to all of this, and it also inverts the old one. When each agent fetches each URL for itself, the same popular page gets crawled thousands of times, once per caller, and every one of those fetches lands on the origin server and costs the caller a full extraction. That is the fetch-per-request model, and it scales badly for everyone: expensive for the caller, punishing for the origin, wasteful in aggregate.

An index collapses that. The first reader to want a URL pays for the origin fetch and the extraction; every reader after that is served the same AIDocument from a shared cache. One crawl, many reads. At any real scale that is cheaper per call than re-fetching per request, and it is gentler on the sites being read, because a million agents wanting the same article become close to one visit to the origin rather than a million. The cache is not a performance tweak bolted onto a scraper. It is the thing that makes an index an index: the work of reading the web is done once and amortized across everyone who needs the result, the same economic logic that made a shared search index beat everyone running their own crawler in the 1990s, now applied to reading instead of ranking.

That economic shape is only possible because of the shape of the document. Because every reader consumes the same canonical AIDocument, the cache is meaningful; a thousand bespoke parses could not share a cache entry. Clean shape and shared economics are the same design decision seen from two angles.

## Where this is going

The direction this points is toward the AIDocument as an open contract, not a proprietary blob. A shape that agents read is only useful if it is stable and legible: additive changes only, no silent breaking, published where machines can find it. That is the intent behind treating the document as a public contract and describing it in the open, through surfaces like `llms.txt` and a normative spec, so that "what an agent should expect when it reads the web" becomes something written down rather than something each vendor keeps to itself.

We are early, and honest about what that means. The index is live and growing; the shape is stable and served today; the freshness-by-signal and consent-canonical mechanics are real for domains that have verified. The wider ambition, a single legible standard for how machines read the web, kept fresh by the publishers who authored the pages, is a direction we are building toward, not a finished thing we are claiming to have delivered. We would rather say that plainly than oversell it.

But the core observation holds regardless of how far along the build is. Agents do not browse. They read. An infrastructure that keeps designing pages for eyes and then apologizing to the machines will keep handing readers a document optimized for the wrong consumer. An index that starts from what a reader actually consumes, text, structure, and provenance, in one clean shape, served once and read by many, is not a better scraper. It is a different thing pointed at a different job.

---

If you build agents that read the web, you can try the shape on your own URLs. The free tier is 2,000 AIDocuments a month, no card required, and every response carries its own measured economics so you can check the token math on the exact pages you care about. Start with [what an AIDocument actually is](/blog/what-is-an-aidocument), or read the [quickstart](/docs/quickstart).

### How to feed web pages to an LLM without blowing the context window

https://www.lyrenth.com/blog/feed-web-pages-to-llm-without-blowing-context

Raw HTML burns your context window on nav, scripts, and boilerplate. Here is how to feed web pages to an LLM as clean AIDocuments, with real token numbers.

You wire up an agent to read a web page, pass the response straight into your model, and the request either errors out on a context-length limit or quietly costs you five times what you expected. The page looked small in a browser. In tokens it was enormous. This is the single most common wall people hit when they first give an LLM the ability to read the web, and it is entirely fixable once you understand where the tokens actually go.

## Why raw HTML is brutal in tokens

A modern web page is mostly not content. It is navigation, script tags, inline styles, cookie and consent markup, tracking pixels, analytics blobs, SVG icons, repeated header and footer chrome, and a wrapper of `div` soup around the handful of paragraphs you actually wanted. When you feed that to a model, you pay for every angle bracket and every class name. The model, for its part, has to read past all of it to find the answer, which makes retrieval and reasoning noisier at the same time it makes them more expensive.

Here is a real measurement, not an estimate. Take the Python standard library docs page for the `json` module, a page every backend developer has read at some point:

```
https://docs.python.org/3/library/json.html
```

Fetched as raw HTML with `curl`, that page is 112,865 bytes. At the common four-characters-per-token approximation, that is roughly 28,000 tokens of markup before your prompt, your instructions, or anything else you need in the window. And this is a well-behaved, mostly static documentation page. Marketing pages and single-page apps are worse.

Now read the same URL as an AIDocument through Lyrenth. Every response carries an economics line in its header. Here is the genuine one that came back:

```
status 200 · render static · 5303 words · ~8,629 tokens · 69% smaller than raw HTML
```

The body dropped from roughly 28,000 tokens of HTML to about 8,600 tokens of clean Markdown for the same page, and the API reported that as 69 percent smaller than the raw HTML it started from. (Method: token counts are the values on the economics line the API returns for that exact URL, measured July 2026; the raw-HTML figure is the actual `curl` byte size divided by four.)

The savings compound. If your agent reads ten pages in a research loop, that is the difference between spending your window on ten pages of content or drowning in ninety percent boilerplate and fitting only two or three before you run out of room.

## The three workarounds that do not hold up

Before reaching for the clean shape, most people try one of three things. Each has a failure mode worth naming, because you will hit it.

**Truncation.** Cut the HTML at N characters and hope the answer is near the top. It usually is not. The interesting content on a real page sits well below the header, the hero, the navigation, and often a consent overlay. Truncating raw HTML is a coin flip on whether you kept the part that mattered, and when you lose it, the model confidently answers from whatever chrome survived the cut.

**Summarization before the read.** Run a cheap model to compress the page first, then hand the summary to your main model. Now you have two problems: you pay for two passes, and the summarization step hallucinates. A compressor that has never seen the ground truth will smooth over the exact number, the exact function signature, or the exact clause you needed. Lossy compression of a document you have not verified is a good way to launder a mistake into your pipeline.

**Readability libraries.** Client-side extraction libraries that pull the "article" out of HTML work well on a blog post and fall over on anything that renders in the browser. A JavaScript-driven site hands the fetcher a hollow `div` and loads the real content afterward, so the extractor gets an empty page. Shadow DOM and web-component sites serialize to markup that has no readable text in it at all. You end up maintaining a headless browser just to see the content, which is its own project.

The common thread: all three try to rescue signal from a format that was never meant to be read by a machine. The better move is to get the content in a shape that is signal to begin with.

## Markdown is the right density

Markdown is the sweet spot for an LLM reading the web. It keeps the structure a model actually uses, headings, lists, links, code blocks, tables, and drops everything a model cannot use, the layout scaffolding and the scripts. It is close to how the text would read if a person copied the meaningful part of the page into a notes file. And because a model was trained on a great deal of Markdown, it parses cleanly with no special handling.

That is what an AIDocument is: one clean, stable shape for a page. A Markdown body plus a title, a description, and the page structure, with the boilerplate stripped and JavaScript rendered when the page needs it. If you want the full picture of the shape and the fields it carries, that is the subject of a companion post, [what an AIDocument is](/blog/what-is-an-aidocument). For this post the relevant part is simple: the body you get back is content, not chrome, so nearly every token you pay for is a token worth reading.

## Practical guidance: trim at clean boundaries with max_tokens

Sometimes even the clean body is bigger than the budget you want to spend on a single page. A 5,000-word reference page is legitimately long. Rather than fetch the whole thing and truncate it yourself in the middle of a sentence, cap it at the source. The reader accepts a `max_tokens` parameter that trims the body to roughly that many tokens at a clean paragraph or sentence boundary, so the tail is never a half-finished thought.

Here is a real trimmed call against the same `json` docs page, capped at 200 tokens. This is the genuine response header and the start of the body:

```
status 200 · render static · 5303 words · ~195 tokens · 99% smaller than raw HTML · truncated to budget

# `json` — JSON encoder and decoder

**Source code:** Lib/json/__init__.py

JSON (JavaScript Object Notation), specified by RFC 7159 (which
obsoletes RFC 4627) and by ECMA-404, is a lightweight data
interchange format inspired by JavaScript object literal syntax
(although it is not a strict subset of JavaScript).

Note
```

Two things to notice. The header now reads `~195 tokens · 99% smaller than raw HTML · truncated to budget`, so you get the same economics accounting on the trimmed result and can see exactly what you spent. And the body stops at a boundary, at the end of a note, not mid-word. The model gets the top of the document intact instead of a jagged fragment. (Same method as above: these are the real values the API returned for `max_tokens=200` on that URL in July 2026.)

The practical pattern for a research agent is to set `max_tokens` to a page budget that leaves room for your prompt and your other pages. If a first read at a tight cap does not contain the answer, read again without the cap, or read a more specific URL. You are trading a cheap, bounded first look against an occasional full read, which is far better than paying for the whole raw page on every hop.

## A worked example

Put it together as the loop an agent actually runs. Say the task is "find the default value of the `sort_keys` argument in Python's `json.dumps`."

1. Read the page as an AIDocument. You get about 8,600 tokens of clean Markdown instead of about 28,000 tokens of HTML, and the headings and the parameter list survive intact because structure is preserved.
2. If you are budget-conscious across many pages, cap the read with `max_tokens` so each hop stays bounded, and the trim lands on a clean boundary rather than slicing a code block in half.
3. Hand that to the model. The answer, `sort_keys` defaults to `False`, sits in a clean parameter list, not buried under three levels of `div` and a cookie banner, so the model finds it on the first pass.

Same answer, a fraction of the tokens, and no summarization layer inventing a default that was never on the page.

The reader endpoint takes the same shape from `curl`:

```sh
curl -H "Authorization: Bearer $LYRENTH_API_KEY" \
  "https://api.lyrenth.com/v1/read?url=https://docs.python.org/3/library/json.html&max_tokens=200"
```

## The point

The context window is not the constraint people think it is. The constraint is spending it on the wrong bytes. Raw HTML is roughly ninety-five percent scaffolding; a clean Markdown body is roughly ninety-five percent signal. Once your agent reads content instead of page chrome, the same window holds several times as much of what you actually came for, your token bill drops in proportion, and the model stops tripping over navigation menus on its way to the answer. Truncation, pre-summarization, and readability libraries are all attempts to rescue signal after the fact; getting the clean shape in the first place is cheaper and more reliable than any of them.

If you want to try it on your own URLs, the free tier is 2,000 AIDocuments a month, no card, and every response carries its own economics line so you can check the token savings on the pages you actually read. The five-minute version is in the [quickstart](/docs/quickstart).

### The publisher's guide to AI bots, robots.txt, and getting represented correctly

https://www.lyrenth.com/blog/publishers-guide-ai-bots-and-verification

Who reads your site now, what a well-behaved AI-index bot looks like, how robots.txt and llms.txt apply, and how to verify your domain with Lyrenth for free.

Your traffic logs used to answer one question: which search crawlers came by, and how often. That question is now incomplete. A growing share of the machines reading your pages are not building a search results page; they are answering a question for a person, right now, inside an assistant or an agent. If you publish on the open web, those readers are already part of your audience.

This is a practical guide to that shift: who is reading your site, what a well-behaved AI-index bot should look like, how the old signals (robots.txt) and the new ones (llms.txt) apply, and how to make sure the version of your content that reaches an assistant is the version you authored. Lyrenth's own crawler is the worked example throughout, because we publish exactly how it behaves.

## Who is reading your site now

Search engines still crawl; that has not changed. What is new sits alongside them: assistants and agents that fetch a specific page because a user asked a specific question, and indexers that build a clean, machine-readable copy of the web so those assistants do not each crawl you separately.

These readers do not want your layout. They do not render your hero image, click your nav, or see your cookie banner the way a person does. They want the text, the structure, and the provenance: what the page says, how it is organized, and where it came from. A well-designed AI reader strips the rest.

That has a direct consequence. If a page reads well as plain structured text, with a real title, a sensible heading order, and clean markup, it represents you well to these readers. If the meaning only emerges after JavaScript runs, or the important content is buried under interstitials, the machine reader gets a worse version than a human does. The gap between "how my page looks" and "how my page reads" is the thing to manage now.

## What a well-behaved AI-index bot looks like

Not every bot that fetches your pages is polite, and not every polite bot tells you who it is. So it is worth being concrete: a responsible AI-index crawler does four things.

**It identifies itself honestly, with a stable name.** No browser impersonation, no rotating user-agent strings that make it hard to tell one visit from a thousand. Lyrenth's autonomous crawler always presents the same user agent:

```
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
```

When a Lyrenth customer submits a specific URL through the API, that fetch is a separate, user-directed action, and it carries a distinct user agent so your logs can tell them apart:

```
AIWebIndex-Agent/2.0 (+https://lyrenth.com/bot; user-initiated fetch)
```

That separation matters. The autonomous crawler is Lyrenth choosing what to fetch and when; a user-initiated fetch is a specific caller asking for your URL through our API, in the same posture as a link previewer or a browser following a click.

**It honors robots.txt per RFC 9309.** RFC 9309 is the formal specification of the robots.txt rules the web has used informally for decades. Lyrenth's autonomous crawler implements it: it honors `Disallow` rules for its user agent, respects `Crawl-delay`, follows `Sitemap:` directives, and backs off on HTTP 429 and 503 responses. It fetches your robots.txt at most once every 24 hours, and if it cannot read it and has no cached copy, it treats your site as disallowed, stricter than the specification requires. It fails closed, not open.

**It stays polite even when you have not told it to.** A good crawler keeps a rate floor on every origin so it never becomes a load problem. Lyrenth holds a minimum 2-second cooldown between requests to the same domain, raised further wherever you declare a `Crawl-delay`. An index crawler is the opposite of a load spike: crawl a page once and serve it to many readers, so a thousand agents on the same URL collapse toward a single fetch against your origin.

**It can be verified, so impostors cannot hide behind its name.** Anyone can put a known bot's name in a user-agent string, so a trustworthy crawler lets you confirm a request is genuinely theirs. Lyrenth publishes three ways: a Web Bot Auth signature on every request (RFC 9421 HTTP Message Signatures, verifiable against a public key directory), a machine-readable list of published IP ranges at `/bot/ip-ranges.json`, and forward-confirmed reverse DNS under `lyrenth.com` on every crawling IP. A request claiming to be Lyrenth from outside those ranges, without a valid signature, is not us.

All of this is stated in full, with the exact strings and steps, on the [Lyrenth bot page](/bot), the address every `+https://lyrenth.com/bot` link in those user agents points to.

Lyrenth is an indexer, not a model trainer. We fetch public pages to build a fresh, canonical index that we serve to agents with attribution and a link back to you. Lyrenth does not train foundation models on the content we crawl.

## robots.txt still works, and here is the simplest control

To keep an AI-index crawler off your site entirely, the oldest tool still works. Add a block for its user agent to your robots.txt:

```
User-agent: AIWebIndex
Disallow: /
```

That stops all future autonomous crawling of your domain. It is prospective: it governs what happens next, not what is already indexed, so removing existing content is a separate request. And because RFC 9309 governs automatic clients rather than user-directed requests, a `Disallow` for the crawler does not by itself stop a user-initiated API fetch a specific caller triggers; for that, Lyrenth honors a domain-wide opt-out by email. The full menu of controls, and what each one does, is on the [crawler policy page](/crawler-policy).

For most publishers, blocking is not the right move, for the same reason blocking search crawlers rarely is: these readers are how a growing part of your audience finds and quotes you. The point is that the control exists and is honored.

## llms.txt in one paragraph

You may have seen `llms.txt` mentioned as a new file to add. The idea is small and useful: a plain, machine-readable file at the root of your domain that gives an AI reader a compact map of what matters on your site, so it does not have to infer your structure by crawling everything. It is a hint, not an enforcement mechanism: a table of contents written for machines. Lyrenth publishes its own at `/llms.txt` so you can see the shape. Adding one is a reasonable low-effort step, but it is not where the real leverage is. That is authorship.

## Why representation matters more than any single file

Here is the situation you are actually in. AI agents already crawl your pages and extract whatever they can, however they like. Sometimes that is clean. Often it is not: a stale price, a stripped-out table, the wrong opening hours, a summary built from the half of the page that happened to render. You have no say over any of it. The version that reaches an assistant is one a stranger's parser produced.

Verification flips that. When you verify your domain with Lyrenth, you author the canonical AIDocument that every agent receives when it reads you through Lyrenth. Your dashboard shows what AI currently makes of each page, and when something is wrong, you push a correction and the version agents read is fixed in seconds. This is additive, not exclusive: the labs and assistants can still read you directly. What changes is that there is now an authoritative version, controlled by you, and it reaches your origin as a single cached fetch rather than a fresh crawl per agent.

## How verification works with Lyrenth

It is free, takes about a minute, and requires no changes to your stack. You prove you control the domain with one of two standard methods:

- A **DNS TXT record** at your DNS provider. The recommended path; usually propagates in under a minute. Our verifier queries several public resolvers in parallel so a slow or misconfigured local resolver does not hold you up.
- A **`.well-known` file** served over HTTPS at your domain. HTTPS only; plain HTTP fetches are rejected.

Pick whichever propagates first. Once ownership is confirmed, Lyrenth resolves your public pages into AIDocuments and your dashboard fills in. You can [start verification from the bot page](/bot) or head straight to adding a site.

One thing we say plainly because it is the deal: verifying your domain also places it in the Verified Index, the consent-backed subset of the corpus Lyrenth can license to AI labs. There is no revenue share. The free owner toolkit (verification, the dashboard, AI Readiness, corrections, and the change signal) is the consideration. You can leave anytime in settings, though copies already delivered to a licensee cannot be recalled. The full mechanics are in the Terms.

## The AI Readiness score is not an SEO score

When your dashboard populates, it shows an **AI Readiness score** from 0 to 10. Be clear about what this is, because it looks like the kind of number an SEO tool would give you, and it is not.

AI Readiness measures one thing: how cleanly a page can be consumed by a non-browser reader. It is built from signals like whether the page renders as server HTML or hides content behind a JavaScript shell, whether it has a real title and description, whether structured data is present, and whether headings and links are used in a sane, semantic way. It aggregates to a per-domain score.

What it is not: it is not a ranking factor, not a content-quality judgment, and not a homework list to please a search engine. It measures the surface contract between your page and a machine that reads text rather than pixels. A page can be excellent journalism and still score low because it only assembles itself after a script runs; a thin page can score well because its markup is clean. The score answers a narrow, honest question: when a machine reads this page as structured text, how much of the meaning survives?

Lyrenth shows you what it measured on the content it actually serves, not a list of demands. And the fixes it surfaces, exposing content as server HTML, adding a real title and description, including structured data, tend to help every machine reader you have, not just Lyrenth.

## Where to start

If you take one action from this: verify your domain. It is free, takes a minute, and moves you from being represented by whatever a parser guessed to authoring the canonical version yourself. Read the [Lyrenth bot page](/bot) to see exactly how our crawler identifies itself, then verify and watch your dashboard fill in. For the deeper picture of why machine readers behave so differently from human ones, and what that inverts about indexing, see the companion piece: [agents don't browse, they read](/blog/agents-dont-browse-they-read).

Reading through Lyrenth as a developer instead? The free tier is 2,000 AIDocuments a month, no card required. Verifying your domain as a publisher is separate, and free forever.

### What is an AIDocument? One clean shape for agents that read the web

https://www.lyrenth.com/blog/what-is-an-aidocument

An AIDocument is one stable JSON shape for any web page: Markdown body, title, description, structure, and measured economics. Here is how it works.

Your agent does not want a web page. It wants what the page says. Those are two very different things, and the gap between them is where most agent-reads-the-web code goes to die.

When you fetch a modern URL, you get an HTML document built for a browser and a human: a navigation bar, a cookie-consent banner, three analytics scripts, a newsletter modal, a footer with forty links, and somewhere in the middle, the paragraph your agent actually needed. Feed that raw into a model and you pay tokens for all of it. On a lot of pages the content is a single-digit percentage of the bytes. The rest is chrome.

An AIDocument is the answer to that: one clean, stable shape for any URL, built for a machine reader instead of a browser. This post is the pillar reference for the idea. If you build agents that read the web, this is the shape worth understanding.

## The problem: agents get raw HTML

Here is what an agent has to deal with when it fetches a page itself.

First, the bloat. A content page is mostly not content. Navigation, scripts, style blocks, consent overlays, and repeated boilerplate dwarf the actual text. You are paying tokens, latency, and attention budget for markup your model will never cite.

Second, the parsing. Someone has to turn that HTML into text. Readability heuristics work until they meet a site that renders its body with JavaScript, at which point a naive fetch returns a hollow `<div id="root"></div>` and nothing else. Now you need a headless browser, and you need to decide when to spin one up, how long to wait for the page to settle, and how to strip the consent manager that popped up after render. That is a real infrastructure project, and it is not the project you set out to build.

Third, the inconsistency. Every site is shaped differently. Your extraction code that works on one blog breaks on the next. There is no single contract you can write against, so you write against all of them, forever.

The AIDocument collapses all three problems into one shape you read once and reuse everywhere.

## The shape: body plus title, description, and structure

An AIDocument is a grouped JSON envelope. The same eight top-level groups come back for every URL and every caller, no matter how the source page was built:

- `schema`: format identity and version. You pin against this.
- `source`: where it came from and how it was fetched. Canonical URL, render mode, status code, freshness policy.
- `cache`: whether the origin was contacted on this call, and whether the body was re-fetched.
- `identity`: title, description, language, detected content type.
- `content`: the cleaned, boilerplate-stripped Markdown. This is the part you feed a model.
- `structure`: headings, links, images, and pre-extracted structured data (JSON-LD).
- `signals`: derived quality signals such as word count, reading time, and whether the page carries structured data.
- `economics`: token and cost economics for this exact page versus ingesting the raw HTML.

The important word is stable. The AIDocument shape is a public contract: additive changes only, never a breaking change without a major-version bump. You write your parsing once and it keeps working. Compare that to writing extraction against the shifting HTML of every site your agent might ever visit.

Markdown, not stripped HTML, is deliberate. It is dense, it preserves headings and lists and links that a model can reason over, and it drops the tags that carry no meaning. It is roughly the highest-signal representation of a page you can hand to a language model.

## A real call, with a real response

The endpoint is `POST /v1/aidocument`. You send a URL, you get an AIDocument back.

```sh
curl -X POST https://api.lyrenth.com/v1/aidocument \
  -H "Authorization: Bearer $LYRENTH_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://en.wikipedia.org/wiki/Web_indexing"}'
```

The response is the grouped envelope, trimmed here to the parts you care about first:

```json
{
  "schema":   { "name": "AIDocument", "version": "2.0" },
  "source":   { "url": ".../Web_indexing", "render_mode": "static", "status_code": 200 },
  "cache":    { "status": "hit", "origin_contacted": false },
  "identity": { "title": "Web indexing - Wikipedia", "language": "en" },
  "content":  { "markdown": "Web indexing or..." },
  "signals":  { "word_count": 1552, "reading_time": 7, "has_json_ld": true },
  "economics": {
    "raw_html_tokens_approx": 21331,
    "output_tokens_approx":   2715,
    "token_savings_percent":  0.873
  }
}
```

Look at the `economics` block. For that page, raw HTML came to roughly 21,331 tokens; the AIDocument content came to roughly 2,715. That is an 87.3% reduction, and it is not a marketing figure pulled from the air: every response carries its own measured economics for the exact URL you asked for, computed on that call. Token counts use a four-characters-per-token approximation, so treat the last digit as illustrative and the order of magnitude as the point.

You do not have to trust any single number. You can measure it yourself. When we read the IETF's own Robots Exclusion Protocol standard (RFC 9309) through the same reader, the response header reported the page as roughly 5,028 words returned as clean Markdown, and 98% smaller than the raw HTML. (Measured via the Lyrenth reader response header for that URL, July 2026.) The savings scale with how much chrome a page carries: a heavy documentation or news page saves the most; a nearly-empty page saves the least. The point holds either way. You stop paying for markup you never wanted.

## What "structure" buys you

The `content` field is what most agents reach for first, but `structure` is where a lot of quiet value lives, because it is work you would otherwise redo on every site.

`structure.headings` gives you the document outline, already parsed, so you can chunk on real section boundaries instead of guessing. `structure.links` gives you the outbound links as data, which is exactly what a crawling or research agent needs to decide where to go next, without regexing anchors out of HTML. `structure.images` gives you the media. And `structure.structured_data` hands back the JSON-LD the publisher embedded: article metadata, product schemas, breadcrumbs, whatever the site author declared. That is machine-authored ground truth about the page, pre-extracted so your agent does not reimplement a JSON-LD parser.

Put together, the AIDocument is not just "the page as text." It is the page as text plus the page as structure, which is what an agent that reads for a living actually needs.

## Crawl once, serve every agent

There is one more idea baked into the shape, and it is visible in the `cache` block above (`"origin_contacted": false`).

Reads resolve through a shared, cross-caller cache. The first agent to ask for a URL pays the origin fetch. Every agent after that is served the already-clean AIDocument from the shared index, without touching the origin site again within the freshness window. That is cheaper for you at scale and gentler on the sites being read, which matters if you care about being a good citizen of the web your agents depend on.

It also means freshness is a policy, not an accident. The default is cache-first. When you genuinely need the live version, you say so on the request (`freshness_policy: "force_refresh"`) and take the origin fetch deliberately.

For the deeper economics of this, and why raw HTML wrecks a context window in the first place, see [How to feed web pages to an LLM without blowing the context window](/blog/feed-web-pages-to-llm-without-blowing-context). For the philosophy underneath, why an index for machine readers looks nothing like an index for browsers, see [Agents don't browse, they read](/blog/agents-dont-browse-they-read).

## How to get one in under five minutes

You do not need to change how you build. There are three ways in, and they all return the same AIDocument shape.

The reader endpoint is the shortest path. One request, clean Markdown back:

```sh
curl -H "Authorization: Bearer $LYRENTH_KEY" \
  "https://api.lyrenth.com/v1/read?url=https://example.com/article"
```

The `POST /v1/aidocument` endpoint above gives you the full grouped envelope when you want the structure and economics, not just the body. Add `max_tokens=N` on either to cap the returned Markdown to your context budget, trimmed at a clean boundary.

If you work inside an assistant, the MCP server drops the same capability straight into Claude Desktop, Claude Code, Cursor, or any MCP client with one config block:

```json
{
  "mcpServers": {
    "lyrenth": {
      "command": "npx",
      "args": ["-y", "lyrenth-mcp"],
      "env": { "LYRENTH_API_KEY": "your-key" }
    }
  }
}
```

That adds `read_url`, batch `read_urls` (up to 20 URLs in one call), and `check_usage` tools, so you can just ask the assistant to read a page and it comes back as clean text.

And there are SDKs when you want the shape native to your language:

```python
from lyrenth import Lyrenth

client = Lyrenth()  # reads LYRENTH_API_KEY
doc = client.read("https://example.com/article")
print(doc.title)
print(doc.markdown)      # cleaned, agent-ready body
```

`pip install lyrenth` and `npm i lyrenth-sdk` both ship batch reads, the `max_tokens` cap, and adapters for the common agent frameworks.

## The takeaway

An AIDocument is a decision to stop treating web pages as documents to render and start treating them as content to read. One stable shape. Body, title, description, and structure. Real measured economics on every call. Crawled once and served to every agent that wants the same URL.

If your agent is spending its context window on cookie banners and nav menus, that is the problem this shape exists to remove. The full request and response, including the other endpoints and the error contract, live in the [quickstart](/docs/quickstart).

The free tier is 2,000 AIDocuments a month, no credit card. Point it at a URL your agent actually reads and check the `economics` block against your own numbers. That is the whole pitch: don't take the figure on faith, measure it on your own pages.

### Why your scraper gets empty HTML from JavaScript sites (and what rendering actually takes)

https://www.lyrenth.com/blog/why-scrapers-get-empty-html-from-javascript-sites

Fetch a React or SPA site and you get a hollow div, not content. Here is why JavaScript sites return empty HTML, and what real rendering takes at index scale.

You write the obvious thing. You `curl` a URL, or you fire an HTTP request from your scraper, and you expect the page back. Instead you get a shell: a `<head>` full of meta tags, a `<body>` with one empty `<div>`, a `<noscript>` line telling you to enable JavaScript, and none of the words a human sees when they open that same URL in a browser. The content is simply not in the response.

This is the wall almost everyone hits the first time they try to read the modern web programmatically. It is not a bug in your code; it is how a large slice of the web is built now. This post is about why it happens, and what it actually takes to get the real content out, which turns out to be a much bigger project than "fetch the URL."

## The failure, on a real site

Let me not hand-wave this. Here is a genuine, well-known single-page app. Excalidraw is a popular open-source whiteboard tool. Open it in a browser and you get a full drawing app with menus, prompts, and a canvas. Fetch it with `curl` and here is the entire `<body>` you get back, trimmed only to collapse the inline bootstrap script:

```html
<body>
  <noscript>You need to enable JavaScript to run this app.</noscript>
  <header><h1 class="visually-hidden">Excalidraw</h1></header>
  <div id="root"></div>
  <script>/* app bootstrap: hydration, CDN redirect */</script>
</body>
```

That is it. The whole raw document is about 6.9 KB, and the `<body>` is roughly a kilobyte of that, most of which is script. There is a `<div id="root">` and it is empty. The page is politely telling you the truth in that `<noscript>` line: without JavaScript, there is no app here. Your scraper is JavaScript-disabled by definition, because a raw HTTP fetch does not run scripts. So your extractor, your readability library, your regex over the DOM, whatever came next, has nothing to work with.

Now read the same URL through the Lyrenth reader, which renders the page the way a browser would before extracting. Here is the genuine response header and the top of the body:

```
# Excalidraw — Collaborative whiteboarding made easy
> Excalidraw is a virtual collaborative whiteboard tool that lets you
> easily sketch diagrams that have a hand-drawn feel to them.
Source: https://excalidraw.com
status 200 · render rendered · 46 words · ~76 tokens · 99% smaller than raw HTML
```

The title and description are populated. The `render` field says `rendered`, meaning the page was executed in a headless browser before the content was pulled, not fetched flat. You get an AIDocument, one clean shape with a Markdown body and structure, instead of a hollow div. (Method: both outputs captured live in July 2026. The raw HTML is the actual `curl` response for `https://excalidraw.com`; the rendered line is the economics header the Lyrenth reader returned for the same URL.)

The whiteboard is a deliberately extreme case, almost the entire page is client-rendered, so the contrast is stark. But the same mechanism sits under a huge amount of ordinary content: docs sites, dashboards, marketing pages, product catalogs. When the body arrives empty, this is why.

## Why the page needs JavaScript at all

To fix the failure you have to understand it, and there is not one cause. There are several, and they stack.

**Client-side rendering.** In the classic server-rendered model, the server assembles the full HTML and sends it down complete. In the client-side model, the server sends a near-empty shell plus a JavaScript bundle, and the browser builds the DOM after the bundle loads and runs. That is the `<div id="root"></div>` you saw. The content exists, but it is assembled on the client, from data the bundle fetches after it boots. A flat HTTP fetch stops before any of that happens.

**Hydration and lazy data.** Even pages that do send some server-rendered markup often fill the important parts in afterward. The shell paints, the app "hydrates," it makes one or more API calls for the actual data, then it renders that data into the DOM. If you read the HTML the moment it arrives, you catch the page mid-boot: skeletons, spinners, and placeholder boxes where the content will be. Reading too early looks like success, because you get a 200 and some HTML. It is just the wrong HTML.

**Shadow DOM and web components.** Some sites build their UI out of custom elements that keep their content inside a shadow root. When you serialize that page to HTML the normal way, the shadow trees do not come along, so the markup you capture is structurally present but textually hollow. The tags are there; the words are not. A naive extractor sees valid HTML with almost no readable text and gives up or, worse, returns confident nonsense.

**Consent managers and overlays.** Even when the content does render, something is often sitting on top of it. A cookie or consent banner injects itself late, frequently from a third-party script, and it can cover the page or hold rendering behind an interaction. Extract at that moment and your "article" is a privacy notice and two buttons. Consent overlays are close to universal on commercial sites now, and they are engineered to be hard to ignore, which is exactly what makes them hard to strip programmatically.

Any one of these turns a simple fetch into an empty or wrong page. In the wild they combine: a client-rendered site, with a consent overlay, that lazy-loads its main content after hydration, that also uses a few web components. That is a completely normal page in 2026.

## What real rendering takes

The fix, conceptually, is easy to state: run the page in a real browser, wait until it is actually done, clean up what the browser leaves, then extract. Each of those clauses hides a genuine engineering problem, and we run all of them at index scale, so here is the honest version of what it costs.

**You need a headless browser, not a fetcher.** To get the content a browser produces, you have to be a browser: load the shell, run the bundle, execute the fetches the app makes, let it paint. That means driving a real rendering engine, not sending an HTTP request. A headless browser is heavy. It uses far more CPU and memory than a fetch, it can crash or hang, and running a fleet of them reliably is a standing operational commitment. This is the single biggest reason "just scrape it" is harder than it looks: the moment you need JavaScript, your per-page cost jumps by a large multiple, and you own a browser-farm reliability problem you did not have before.

**You have to decide when a page even needs it.** Rendering everything is wasteful, because plenty of pages are still server-rendered and a plain fetch is complete and cheap. So the first real decision is per-page: does this URL need a browser, or is the flat HTML already the whole story? Render too eagerly and you burn browser time on pages that never needed it; render too rarely and you ship hollow pages. The signal is not always obvious from the first bytes, so this is a category of problem, not a one-line heuristic.

**Settle timing is the hard part.** Once you commit to rendering, you have to decide when the page is done. Fire the extract too early and you capture spinners and skeletons. Wait a fixed long time on every page and you have throttled your whole system to the speed of the slowest page, which at scale is ruinous. The right behavior is adaptive: watch for the signals that the DOM has stopped changing and the content-bearing elements have appeared, and settle as soon as they have, not on a fixed timer. Different sites reach "done" through different paths, so a settle policy that works on one class of page will read another too early. This looks trivial in a demo of one URL and becomes the whole game across many.

**You have to strip what the browser leaves behind.** A rendered page includes everything the browser drew, and that is more than the content. The consent overlay is now in the DOM, along with the nav, the footer, the newsletter modal, and the sticky share bar. Rendering got you the content; it also got you every piece of chrome around it. So after the render you still have to strip overlays and boilerplate to recover the actual document, and it is adversarial work, because consent tools and modals are designed to be sticky. Web components add another layer: you have to flatten shadow trees so their text is actually present in what you extract, or you are back to hollow markup even after a successful render.

None of these is exotic on its own. The cost is doing all of them, correctly, on every page, at volume, and keeping the browser fleet healthy while you do. That is the difference between a script that reads one page and infrastructure that reads the web.

## How Lyrenth handles it so you do not have to

The point of the AIDocument is that all of the above happens on our side, once, and you receive the result. When you read a URL through Lyrenth, the reader decides whether the page needs a browser, renders it when it does, waits for the content to settle rather than on a fixed timer, strips the consent overlays and boilerplate, flattens shadow trees so web-component content is real text, and hands back a clean Markdown body with structure. That is what the `render: rendered` field in the Excalidraw response meant: this page needed a browser, we ran one, and here is the content a flat fetch could never have given you.

You do not maintain a browser farm, tune settle timers per site, or chase the newest consent-banner script. You send a URL and get an AIDocument back, the same clean shape whether the source was static HTML or a fully client-rendered app. And because reads resolve through a shared index, the render is amortized: the expensive part happens once for a URL, and subsequent reads come back from the cache rather than paying for another headless render.

For the full picture of the shape you get back, the fields and the stable contract, see [what an AIDocument is](/blog/what-is-an-aidocument). And if the reason you are reading pages is to feed them to a model, the token side of this, why raw HTML wrecks a context window and how the clean shape fixes it, is covered in [how to feed web pages to an LLM without blowing the context window](/blog/feed-web-pages-to-llm-without-blowing-context).

## The takeaway

An empty `<div id="root">` is not a failure of your scraper. It is a modern web page telling you it is built to be run, not fetched. Getting real content out of it means being a browser, deciding when to be one, knowing when the page is done, and cleaning up everything the browser drags in with the content. That is a real infrastructure project, and not the one you set out to build.

The whole idea of the AIDocument is that you get to skip it. Point the reader at a JavaScript-heavy URL and see the `rendered` marker come back with real content behind it. The free tier is 2,000 AIDocuments a month, no card, so you can test it against the exact SPA that has been handing your scraper a hollow div.

## Links

- Documentation: https://www.lyrenth.com/docs
- Pricing: https://www.lyrenth.com/pricing
- Bot info: https://www.lyrenth.com/bot
- Agent manifest (JSON): https://www.lyrenth.com/api/agent-manifest
- AIDocument JSON Schema (draft-07): https://api.lyrenth.com/aidocument.schema.json
- Sign up: https://www.lyrenth.com/signup
- Sitemap: https://www.lyrenth.com/sitemap.xml
- Robots: https://www.lyrenth.com/robots.txt