robots.txt, sitemaps and markup: how a site talks to machines

A person looks at the page. A machine reads the document: its structure, its markup, and three files at the root a person never opens. They decide what reaches search results, how a link unfurls in a messenger, and what an assistant says when asked about your company.

robots.txt

The smallest file and the most dangerous. It says where a crawler may go, and one extra line removes the site from search entirely — usually along with a release, when the staging rule was never taken out.

What it does not do: hide a page. Disallowing a crawl is not disallowing a listing; a blocked page can stay in results as a line with no description. If a page must stay out of search, that is said with a `noindex` meta on the page itself, not in `robots.txt`. Here, only the service endpoints are closed.

The sitemap

A list of the addresses you consider pages. It does not move rankings, it moves speed: without it search finds your pages by following links; with it, it gets the list at once.

Two mistakes. The first is a map edited by hand: it is always behind by the latest publication. The second is a map holding redirected addresses and pages marked noindex — a conversation in which you hand the machine data you know to be wrong.

Page markup

Headings by level, not by size. `JSON-LD` for the organisation and for the article — the thing a rich result is built from. Open Graph for messengers: without it a link unfurls as a blank rectangle, and it is the one format where the mistake is instantly visible to everyone.

A social card is better kept as an image on disk than drawn on the fly: on-the-fly generation fails with the runtime, and a file does not.

llms.txt

A new file, optional, and still argued about. The idea is simple: a short description for language models — what the product is, who it is for, where things are. The rule is the same as for meta: **do not put figures in it that you will forget to update.** A stale price in a file an assistant quotes is worse than no price.

What the template does about it

`robots.txt`, the sitemap and `llms.txt` ship with the template and are tied to the site config rather than written separately. The map walks the routes and the articles. `JSON-LD` and Open Graph are assembled from the same meta the social cards are drawn from — one image per route per language, so a Russian page does not unfurl with an English title.