The 7 Files Every Website Should Have (And Most Don't)
There are seven files that sit at the root of a well-built website. They don’t appear in your analytics, your CMS, or your sitemap. But search engines, ad platforms, security researchers, and AI systems check for them before doing much else.
Most sites have one or two. A few have none. Here’s what each file does, and what missing it actually costs.
The files that tell machines how to navigate your site
robots.txt
robots.txt tells crawlers which parts of a site they’re allowed to index. It’s been a web standard since 1994, formalised as RFC 9309 in 2022. Google’s robots.txt documentation covers practical implementation and common pitfalls.
A missing robots.txt means crawlers will index whatever they can reach, including admin paths, staging content, and pages you didn’t intend to surface. A misconfigured one can accidentally block the pages you do need indexed, it’s one of the most common causes of organic traffic drops after a site migration.
Even a minimal file that says “crawl everything” is worth having. A missing robots.txt generates a 404 on every crawler visit, which adds noise to your server logs and signals the file was overlooked rather than omitted deliberately.
manifest.json
A web app manifest (usually named manifest.json or site.webmanifest) tells browsers how to display the site when someone adds it to their home screen or installs it as a progressive web app. Unlike the other files here, it doesn’t need to live at the root, it just needs to be linked from the HTML . It sets the name, icons, and theme colour.
Without one, browsers won’t prompt users to install the site, and link previews in mobile browsers and some search results will pull from whatever they can find rather than a defined source. The MDN reference for Web App Manifests covers implementation. It’s a short file with a meaningful effect on how a site is perceived on mobile.
The files that establish trust and accountability
security.txt
security.txt lives at /.well-known/security.txt and gives security researchers a standardised path to report a vulnerability they’ve found. It contains a contact address, an optional PGP key for encrypted reports, and an expiry date.
The problem it solves: most security researchers who find an issue want to report it responsibly. Without a security.txt, there’s no obvious way to do that. The result is either silence (the issue stays unfixed), a support form that goes nowhere, or a public disclosure. The file was standardised as RFC 9116 in 2022.
One detail that trips people up: the expiry field is required, and an expired security.txt signals the file isn’t maintained, which is arguably worse than none.
ads.txt
ads.txt (Authorised Digital Sellers) lists the companies authorised to sell advertising on a site. If a site runs display advertising, this file isn’t optional.
Without it, fraudulent networks can misrepresent your ad inventory to buyers, running ads through unauthorised resellers and claiming the revenue. Ad platforms like Google use it to verify legitimate sellers, so a missing file can affect both fill rates and revenue. The IAB Tech Lab’s ads.txt specification is the standard.
If a site doesn’t run advertising, skip this one. If it does, missing it is leaving control on the table.
humans.txt
humans.txt is an informal convention for crediting the people who built a site: developers, designers, writers, anyone who contributed. It lives at /humans.txt with no formal specification, maintained loosely by humanstxt.org.
No crawler gives it special weight. It won’t move a ranking. But it’s a sensible place to put a credit without cluttering the footer, and it’s the kind of detail that signals care in how a site is built. It takes five minutes and no client will object to it.
The files that are becoming expected
llms.txt
llms.txt gives AI systems structured context about a site: what it is, who it’s for, and what content is available. As AI tools increasingly summarise, recommend, and navigate the web, this file shapes how they represent a site to users.
It’s the AI-era equivalent of robots.txt. Where robots.txt addresses traditional search crawlers, llms.txt addresses AI consumers. The convention is maintained at llmstxt.org, and while it’s still being adopted, early implementation positions a site well ahead of how AI-driven search is developing.
llms-full.txt
llms-full.txt is the extended companion to llms.txt. Where llms.txt is a concise summary, llms-full.txt contains the complete content: documentation, FAQs, product copy, anything that would give an AI system an accurate picture of the site’s full scope.
Not every site needs both. A brochure site might only need llms.txt. A product with documentation or a knowledge base benefits from having the full version available so AI tools don’t have to infer from partial content.
A website files checklist
For each domain you manage, check that these files exist and return a 200 response:
/robots.txt– does it exist, and does it reflect what should and shouldn’t be crawled?/manifest.jsonor/site.webmanifest– is it linked in the HTML ?/.well-known/security.txt– does it have a current contact address and a future expiry date?/ads.txt– if the site runs advertising, is it present and current?/humans.txt– optional, but worth the five minutes/llms.txt– structured context for AI systems/llms-full.txt– full content version, if the site has substantial documentation
The harder part isn’t adding these files at launch. It’s knowing when they change or disappear after a migration, a platform update, or a hosting move.
If you manage multiple sites, kant.au tracks site files across every domain you look after and flags anything missing or changed. Start a free 14-day trial and see where yours stand.