robots.txt

robots.txt is a root file instructing crawlers which paths they may or may not fetch.

Use SearchDock's free robots.txt tester for Googlebot and AI bots.