Skip to content

Plain notes on robots.txt for small websites

Robots.txt, explained for small websites

Robots.txt is a small text file that tells search crawlers where they can and can't go on your site. These notes cover what it does, how to write the rules, how to test them, and the slips that hide pages from search. Each page weighs the options side by side.

Read this first