Question 1

What does robots.txt do?

Accepted Answer

The robots.txt file sits at the root of your domain and instructs search engine crawlers which pages or paths they are allowed or not allowed to request. It is a courtesy protocol — well-behaved crawlers respect it.

Question 2

What is the difference between Disallow and Noindex?

Accepted Answer

Disallow in robots.txt prevents a crawler from fetching the page. Noindex in a meta tag allows fetching but tells the search engine not to include the page in its index. A disallowed page can still appear in search results if other sites link to it.

Question 3

Can robots.txt prevent my page from being indexed?

Accepted Answer

Not reliably. Blocking crawling stops Google from reading the page content, but the URL can still appear in search results based on links pointing to it. To fully de-index a page, use a noindex meta tag on a crawlable page.

Question 4

What are common robots.txt mistakes?

Accepted Answer

The most damaging mistake is accidentally disallowing / which blocks the entire site. Others include blocking CSS and JS files (which prevents rendering), syntax errors, and conflicting rules where a broader allow overrides a specific disallow.

Question 5

How do I test if my robots.txt is working?

Accepted Answer

Enter your site URL in our robots tester to fetch and parse your robots.txt live. Then enter a path to see whether Googlebot or any other crawler would be allowed or blocked from crawling it.

Gratis Robots.txt Tester

Hvad er en Robots.txt fil?

Bedste Praksis for Robots.txt

Robots.txt og AI-crawlere