The machine view, part 2: what a crawler extracts and how retrieval chunks your page

Where part one left off

This is the second half of a two part series on the machine view at By Robots For Robots, a small laboratory site I built about how machines read web pages. Part one covered the Outline and Structured Data panels: how a machine reads your headings as a table of contents, and your JSON-LD as a statement of what the page is.

The Extraction and Chunks panels are where it gets more interesting, because they show the page after the machine has finished taking it apart. Everything below comes from the live machine view on the By Robots For Robots home page at the time of writing.

What does the Extraction view show?

Open the machine view (Shift + M), click Extraction, and you get this:

Field	Value on the home page
Title	By Robots For Robots | A laboratory for the machine-readable web
Meta description	An open laboratory for the machine-readable web: working instruments and field notes on how crawlers, retrievers and answer engines actually read a page.
Canonical	https://www.byrobotsforrobots.ai/
Lang	en
Robots	index, follow, max-snippet:-1, max-image-preview:large
H1 count	1
Word count	636
Est. tokens	~939
Links (internal)	8
Links (external)	2
Images	0
Landmarks	7

Under the table the panel says: "This is roughly the payload a crawler keeps after stripping presentation." That is the right way to think about it. Twelve facts and a block of text. Everything you designed is gone.

What does each extraction field tell you?

• Title and meta description are the page's own summary of itself. They often become the label a search result or an AI citation shows, so they should say what the page is in plain words.
• Canonical tells the machine which URL is the real one. If it points somewhere unexpected, every other signal gets credited to the wrong page.
• Lang sounds trivial and is often missing. It tells a parser which language the text is in, which affects how it is tokenized, matched, and shown to searchers.
• Robots is the permission slip. max-snippet:-1 means "quote as much as you like," and max-image-preview:large allows large image previews. Google's documentation says its snippet controls, such as nosnippet and max-snippet, also apply to how content can be shown in its AI features. If you restrict snippets, you restrict what can be quoted.
• H1 count is a quick sanity check that the page has one clear topic.
• Word count and estimated tokens are the size of the text payload. Tokens are the unit language models count in. Here, 636 words come out to about 939 tokens, roughly one and a half tokens per word. That ratio matters later, because chunk sizes are measured in tokens, not words.
• Links are how a crawler finds the next page and how it learns which pages you consider important. 8 internal and 2 external is a small, deliberate set.
• Images: 0. The site has no images on purpose. On most sites the number is higher, and the useful question is how much of the meaning lives inside images, where the text extraction cannot see it.
• Landmarks: 7. Header, navigation, main, sections, and footer elements. They let a machine separate the content from the chrome around it.

The point of the panel is that every one of these is a decision you made, or failed to make, in your HTML. None of them depends on how the page looks.

What does the Chunks view show?

Click Chunks and the same text comes back cut into pieces:

Chunk	Est. tokens	Label (from the section heading)
01	~236	By Robots For Robots
02	~205	Three things it is trying to be
03	~160	See the page the way the machine does
04	~209	Experiments
05	~50	Notes
06	~79	Permanently unfinished

The footer of the panel explains the method: "6 retrievable chunks, ~939 tokens total. Splitting is done on section boundaries, the same seam most retrieval pipelines look for." And then the line I think every content writer should tape to their monitor:

"A chunk that carries its own heading survives the trip into an index with its context intact; an unlabelled one arrives as orphaned prose."

Each chunk preview starts with the heading text and runs into the first lines of the section. Chunk 02 is a good one to look at closely. It begins "What this is Three things it is trying to be 01 · Premise Machines read first..." The small eyebrow label above the heading ("What this is") and the decorative numbering ("01 · Premise") are now part of the chunk text. A person sees them as design. The machine sees them as words.

Why do chunks matter for AI search?

When an AI assistant answers a question using web sources, it usually does not read your whole page. A retrieval system has already split pages into passages, turned each passage into a numeric representation of its meaning, and stored it. When a question comes in, the system finds the passages whose meaning is closest to the question and hands those to the model to write the answer.

So the unit of competition is the chunk, not the page. Your page can be excellent overall and still lose if the specific passage that answers the question is vague, mislabeled, or split in the wrong place. I covered the broader mechanism in LLM SEO: how language models decide what to cite. The Chunks panel makes it visible.

A caveat that matters: real pipelines differ. Some split by headings, some by a fixed number of tokens with overlap, some by sentences, and most do not publish their settings. The machine view uses section boundaries because that is the most common natural seam. Do not tune your pages to one tool's chunk size. Tune them so they split well under any reasonable method.

How should you write content that survives chunking?

From a content perspective, chunking rewards a handful of habits.

1. Make every section stand on its own. Imagine the section is the only thing a reader will ever see. Does it make sense without the one above it? Avoid openings like "As mentioned above" or "This is why."
2. Write headings that would work as a search result. The heading becomes the chunk's label. "Notes" is a weak label. "What we learned from the first build" is a strong one. This is the same habit as part one, with higher stakes.
3. Lead each section with the answer. Retrieval matches a question to a passage. Put the direct answer in the first sentence or two, then support it.
4. Repeat the subject instead of leaning on pronouns. "It" at the start of a chunk refers to nothing. Name the thing: the product, the service, the city.
5. Keep sections a sensible size. On the By Robots For Robots home page, the Notes chunk is about 50 tokens: one heading and a teaser. That is a thin chunk with little to retrieve. A 1,500 token section, on the other hand, will probably get split somewhere you did not choose. A few hundred tokens per section, built around one idea, is a practical target.
6. Put the facts in the text. Prices, locations, dates, names, and numbers that live only in images, charts, or tooltips do not reach the chunk.

What should you build so chunks come out clean?

From a technical perspective, the goal is to make the boundaries real and the text present.

• Server-render the text. If the content arrives late through JavaScript, some machines will extract an empty page. Check the raw HTML, not the rendered view.
• Wrap sections in real elements. A <section> or <article> with its heading inside gives any splitter an unambiguous seam. A page built from one giant <div> forces the machine to guess.
• Keep decoration out of the text stream. Eyebrow labels, step numbers, and "Read more" fragments end up inside chunks, as chunk 02 shows. If an element is purely visual, consider whether it belongs in the text at all, or whether CSS can generate it.
• Use landmarks so the chrome is separable. Header, nav, main, and footer let a parser drop the menu and the footer links instead of folding them into your first and last chunks.
• Check your robots and snippet directives. A stray nosnippet or a tight max-snippet limits how much of a chunk can be quoted.
• Make tables readable as text. Use real <table> markup with header cells. A table drawn with styled divs becomes a jumble of numbers once extracted.

What happens when you chunk a real blog post?

I ran a rough version of the same split on my own September progress report, using its H2 sections and the same ratio of about 1.5 tokens per word:

Section	Words	Approx. tokens
Summary box and intro	168	~249
What do the numbers actually say?	491	~727
Why did impressions grow 587 percent while clicks fell?	254	~376
Why is the Authority Score stuck at 2?	183	~271
The disavow file I built and added anyway	403	~596
So what should you actually do about spam links?	189	~280
The opportunity I found while looking for the actual fix	291	~431
Why LLM SEO and AI visibility tracking are next	362	~536
What does GA4 say about the people who do arrive?	159	~235
What happens over the next 90 days	206	~305
FAQ (six questions)	429	~635

What I learned from that:

• The question headings work. "Why is the Authority Score stuck at 2?" is a label that would match a real question, and the section answers it in its first lines.
• The numbers section is the weak chunk. At about 727 tokens it is the biggest, and most of it is two tables. Split in the middle, the second half would be a table with no explanation. It should be two sections, each with its own heading.
• Two headings are statements, not questions. "The disavow file I built and added anyway" is a good story heading and a weak retrieval label. A clearer version would be "Should you upload a disavow file without a manual action?"
• The FAQ is already chunk-shaped. Each question and answer is a self-contained passage, which is a good reason FAQs keep showing up in AI answers.

A short checklist

1. Open your page's raw HTML and confirm the main text is there without JavaScript.
2. Check the title, meta description, canonical, lang, and robots directives in one place, as the Extraction panel does.
3. Read each section as if it were the only thing anyone will see. Fix the ones that depend on the section above.
4. Rewrite vague headings into specific labels, ideally questions.
5. Put the direct answer in the first two sentences of each section.
6. Split oversized sections and merge thin ones.
7. Move facts out of images and into text.
8. Wrap sections in real elements and keep decoration out of the text stream.

Or skip the manual work: flip the machine view on byrobotsforrobots.ai to see the method, and paste your own HTML into the Readability Probe to check yours. If you want a person to do it across a whole site, that is part of a Double Atari GEO and AEO engagement.

Extraction and chunks FAQ

What is content chunking in AI search?

Content chunking is the step where a retrieval system splits a web page into smaller passages before indexing it. When someone asks a question, the system finds the passages closest in meaning to the question and gives those to the language model, so each chunk competes on its own rather than the page as a whole.

How big should a content chunk be?

There is no single answer, because retrieval pipelines use different methods and rarely publish their settings. A practical target is sections of a few hundred tokens built around one idea, each with a descriptive heading. Very short sections give little to retrieve, and very long ones get split in places you did not choose.

What does a crawler extract from a web page?

Roughly: the title, meta description, canonical URL, language, robots directives, headings, the main text, links, images, and the page landmarks. The By Robots For Robots machine view shows this as an Extraction panel, along with word count and an estimated token count.

Why do headings matter for chunking?

Many retrieval systems split pages at section headings, and the heading becomes the label for that passage. A chunk with a descriptive heading keeps its context when indexed. A chunk with a vague heading, or none, arrives as orphaned text that is harder to match to a question.

Does JavaScript-rendered content get chunked?

Only if the system that reads the page runs the JavaScript. Many crawlers and AI fetchers read the raw HTML, so content that only appears after scripts run may be missing from the extraction and from every chunk built from it. Server-rendering the main text avoids the problem.