Technical SEO: crawling, indexing, and redirects

Technical SEO is the part of SEO that decides whether your pages get found, crawled, and indexed at all. Content quality only matters after those three succeed.

Most technical SEO problems are not subtle. They are a noindex left on after launch, a canonical pointing at the wrong URL, or a redirect chain that loses the signal. This page covers the controls and what each one actually does.

Robots.txt

robots.txt sits at the root of a domain and tells crawlers which paths they may request. It is a crawling directive, not an indexing one — and that distinction causes more damage than any other misunderstanding in SEO.

A URL blocked in robots.txt can still be indexed if other pages link to it. Google simply indexes it without being able to see the content, which produces those "No information is available for this page" results. Worse: if a page is blocked by robots.txt, Google cannot read a noindex tag on it, so blocking is actively counterproductive when your goal is de-indexing.

• Use robots.txt to save crawl budget on genuinely worthless paths — faceted search, internal endpoints, infinite calendars.
• Use noindex to keep something out of the index. Never both on the same URL.
• Reference your sitemap from it.
• Remember it is public. Do not list secret paths there.

XML sitemaps

A sitemap lists the URLs you want crawled, optionally with lastmod. It is a hint, not a command — it helps discovery, especially on large sites or pages with few internal links, but it does not force indexing.

• Include only canonical, indexable, 200-returning URLs. A sitemap full of redirects, 404s, or noindex pages actively wastes crawl budget and erodes trust in the file.
• Keep lastmod honest. Bulk-updating every date on every deploy trains Google to ignore it.
• Split into a sitemap index above 50,000 URLs or 50 MB.
• Submit in Search Console and check the coverage report against what you expected.

Canonical tags

<link rel="canonical"> tells search engines which URL is the preferred version when the same or very similar content is reachable at several addresses — tracking parameters, session IDs, sort orders, http vs https, trailing slash variants.

Without it, ranking signals split across duplicates and none of them rank as well as one consolidated URL would.

• Make it self-referential on every page. A page whose canonical points at itself is the normal, correct case.
• Use absolute URLs.
• The canonical must return 200. Pointing at a redirect or a 404 makes the whole hint unreliable and Google will ignore it.
• Keep canonical and sitemap in agreement. If your sitemap lists a URL that canonicalises elsewhere, you are sending contradictory signals.
• It is a hint, not a directive. Google can and does override it when other signals disagree.

Noindex

noindex — as a meta robots tag or an X-Robots-Tag header — tells search engines not to include a page in the index. Unlike robots.txt, it is a genuine indexing directive.

Use it for thin, duplicate, or private-but-not-secret pages: internal search results, filtered listings, thank-you pages, staging content.

Two things to get right:

• The page must remain crawlable for noindex to be seen. Blocking it in robots.txt at the same time means the directive is never read.
• Use noindex, follow when the page should still pass link equity through to the pages it links to.

The most expensive SEO bug in existence is a site-wide noindex that survives a launch. Check it first when traffic vanishes overnight.

Redirects and HTTP status codes

A redirect sends a client from one URL to another. Which code you use determines what happens to your ranking signals.

• Use 301 for permanent moves. Using 302 for a permanent move tells Google to keep the old URL — a very common and very costly mistake during site migrations.
• Redirect to the final destination directly. Chains dilute signals and each hop adds latency; more than a couple and crawlers may stop following.
• Never mass-redirect retired pages to the homepage. Google treats that as a soft 404 and it helps nobody. Redirect to the closest genuine equivalent, or return 410.
• Watch for redirect loops — they take the URL out of the index entirely.

Crawl budget

Crawl budget is how many URLs a search engine will fetch from your site in a given period. It is a function of how fast your server responds and how much the crawler thinks your content is worth.

For most sites under a few thousand pages, this is not your problem and worrying about it is a distraction. It matters on large sites, e-commerce with faceted navigation, or anywhere URL parameters generate combinatorial explosions.

• Fix slow responses first. A faster server earns more crawling directly — this is where TTFB meets SEO.
• Eliminate infinite URL spaces: faceted filters, sort orders, session IDs, calendars.
• Cut redirect chains and soft 404s; they burn budget for nothing.
• Keep internal linking shallow so important pages are reachable in few clicks.

Hreflang

hreflang tells search engines which language and regional version of a page to serve. It matters for any site serving multiple languages, including this one.

• Annotations must be reciprocal: if A points to B, B must point back to A. Non-reciprocal hreflang is ignored.
• Include a self-referencing entry on every page.
• Add x-default for the fallback when no language matches.
• Use correct codes — ISO 639-1 for language, ISO 3166-1 Alpha 2 for region (da, da-DK, en-US).
• Every URL referenced must be indexable and canonical to itself. hreflang pointing at a noindex page or a non-canonical URL is silently discarded.

URL structure

Readable URLs help users more than algorithms, but structure still matters for consistency and deduplication.

• Keep them short, lowercase, hyphen-separated, and descriptive.
• Pick one convention for trailing slashes and www and redirect everything else to it. Serving the same content at four spellings is a self-inflicted duplicate content problem.
• Avoid unnecessary parameters; when unavoidable, canonicalise.
• Do not change URLs without a good reason. If you must, 301 every old URL to its specific replacement.

Mobile-first indexing

Google indexes the mobile version of your site. Not the desktop version — the mobile one. This has been the default for all sites since 2023.

The practical consequence is that content hidden, trimmed, or lazily loaded only on mobile effectively does not exist for ranking purposes.

• Serve the same content, headings, and structured data on mobile as desktop.
• Ensure mobile pages carry the same internal links; a stripped-down mobile nav can orphan pages.
• Verify with the URL Inspection tool, which shows the rendered mobile HTML Google actually sees.

llms.txt

llms.txt is a proposed convention — a Markdown file at the root of a domain offering a curated, machine-readable map of a site's most useful content for large language models, in the way robots.txt addresses crawlers.

Be honest about its status: it is not an established standard, and no major AI provider has committed to honouring it. It costs almost nothing to publish and may help, but it is not a substitute for the things that demonstrably drive AI citation — clear headings, factual accuracy, structured data, and content that is actually worth quoting.

You can validate one with Sitecheck's llms.txt tester.

See also

• On-page SEO: content, meta tags, and structured data
• Core Web Vitals and page speed metrics
• Web performance: delivery, caching, and uptime