crawlie

Technical SEO and GEO crawler for the terminal

TLDR

Crawl an entire site and print a report

$ crawlie crawl [https://example.com] --format pretty

Audit a single page

$ crawlie audit [https://example.com/pricing]

Save an HTML report to a file

$ crawlie crawl [https://example.com] --format html -o [report.html]

Limit the number of pages fetched

$ crawlie crawl [https://example.com] --max-pages [100]

Fail a CI build when errors are found

$ crawlie crawl [https://example.com] --fail-on error

Explain why a finding matters

$ crawlie explain [geo-not-answerable]

SYNOPSIS

crawlie subcommand [url] [options]

crawlie is a fast, open-source crawler that audits a website for technical SEO and GEO (Generative Engine Optimization) issues. It seeds from sitemap.xml, respects robots.txt, follows internal links, and applies a set of rules covering broken links, missing or duplicate metadata, performance hints, mobile and international optimization, and AI-search readiness.Results can be printed to the terminal or exported as JSON, CSV, or a shareable HTML report. The same engine is exposed as an MCP server (crawlie-mcp) for use inside agent workflows.

PARAMETERS

--format fmt

Output format: pretty (terminal), json, csv, or html (default: json).

-o, --output file

Write the report to a file instead of standard output.

--max-pages n

Maximum number of pages to fetch (default: 500).

--max-depth n

Maximum click depth from the seed URL.

--concurrency n

Number of parallel requests (default: 16).

--include glob

Only crawl URLs matching the pattern.

--exclude glob

Skip URLs matching the pattern.

--no-robots

Do not parse or honor robots.txt.

--no-sitemap

Do not seed the crawl from sitemap.xml.

--no-external

Skip checking external links.

--severity level

Filter findings by severity: error, warning, or notice.

--save

Store the result in the local report history.

--fail-on level

Exit with a non-zero status when findings of error or warning severity are present (useful in CI).

SUBCOMMANDS

crawl url

Crawl an entire site, seeding from the sitemap and following internal links.

audit url

Audit a single page or a specific set of pages.

explain finding

Describe why a finding matters and how to fix it.

reports

List the local report history.

report id

Retrieve a specific saved report by its ID.

HISTORY

crawlie is written in Rust (with a TypeScript desktop companion) and released under the MIT license. It was published in 2026 as a free, agent-friendly alternative to commercial SEO crawlers, adding GEO checks aimed at how AI search engines read a site.

RESOURCES

Source code