jevcrawl

jevcrawl [-v] [command] [options]

jevcrawl is a Python 3.11+ CLI and MCP server that crawls a website for a specific question. TypeSafe's Jev decision model scores links and passages; it never writes text. Everything returned is copied from the page, with URL, heading path, and character offsets so a quote can be checked later.A crawl stays on the start host and its subdomains, fetches up to four pages at a time, and can offer sitemap URLs that share words with the goal. Blocks scoring at or above 0.35 are kept; scores in [0.20, 0.35) are uncertain. The crawl stops when "do the kept blocks answer the goal?" reaches 0.85, or when `stoppedbecause` is `frontierempty`, `maxpages`, `maxseconds`, `budget`, or `auth_error`. After three failed Jev requests the crawl continues breadth-first, returns every block unscored in `uncertain`, and sets `stats.degraded`.The console script is `jevcrawl.cli:main` (`pip install jevcrawl` or `uv tool install jevcrawl`). Version 0.1.0. Human-readable output lists kept and uncertain blocks, then a summary line with pages fetched, stop reason, token counts, Jev call count, cost, and elapsed time. Pages that need JavaScript are listed in `stats.needsjsurls` and are not rendered.

(no command)

Start the MCP server on stdio. MCP clients typically launch this with `uvx jevcrawl` and `TYPESAFEAPIKEY` in the environment.

Fetch from url toward question. Default --max-pages 30, --max-seconds 120, --return-tokens 4000, --max-usd 0.05 (hard cap on Jev spend; a request that would pass it is never sent). Stops when the kept passages are judged to answer the goal, the frontier is empty, or a budget is hit.

Fetch one page. With --goal, return only the relevant passages. Without it, return the page content and make no Jev call.

Rank URLs found from url for question. Default --limit 50, --max-pages 10. Prints `priority url (anchor)` lines, or the JSON list with --json.

Print version, endpoint, model, whether an API key is set (length only, never the key), and cache path, then list models from Jev. Exit 0 if Jev accepted the key; exit 1 if the key is missing, rejected, or the endpoint cannot be reached.

Print the full result object instead of the human-readable blocks. On a crawl error, print `{"error": "...", "message": "..."}`.

Repeatable. Once logs INFO to stderr; twice or more logs DEBUG. Default is WARNING.

Print `jevcrawl` and the package version.

TYPESAFE_API_KEY

Jev API key. Required for crawl, map, scrape-with-goal, doctor, and the MCP server. Create a key at https://console.typesafe.ai/. Never written to disk or logs.

Jev-compatible HTTP endpoint. Default `https://api.typesafe.ai`.

Model id. Default `jev-1.13.0`. Thresholds are calibrated per version; changing the model without retuning them changes keep/stop behaviour.

Local cache of Jev answers. Default `~/.jevcrawl/cache`. Repeating the same goal on the same pages does not spend again.

Requires a TypeSafe Jev API key and network access. Page text (blocks, link labels, and URLs) is sent to TypeSafe to be scored. Jev charges for input tokens only; `jev-1.13.0` is billed at $0.042 per million input tokens.Does not read `robots.txt` and does not rate-limit beyond four concurrent fetches. It never logs in, does not try to pass paywalls or CAPTCHAs, and does not run JavaScript. You are responsible for the sites you crawl and their terms.A missing or rejected key stops a crawl with `auth_error`. Install from PyPI (`jevcrawl` 0.1.0); Linux distribution packages may not exist.

jevcrawl was published in October 2026 by Rayen Mlayeh (PyPI 0.1.0 on 2026-10-13) as an Apache-2.0 MCP server and CLI on top of TypeSafe Jev. It is measured against Crawl4AI's goal-directed crawl and against an LLM that picks each link, on WebWalkerQA tasks whose answers sit several clicks into a site.

wget(1), curl(1), scrapy(1), httpx(1), jevchat(1), jevals(1), uv(1)

Source code