scrapy

Create new project

$ scrapy startproject [project_name]

$ scrapy genspider [spider_name] [domain.com]

$ scrapy crawl [spider_name]

$ scrapy crawl [spider_name] -o [output.json]

$ scrapy shell "[https://example.com]"

$ scrapy check [spider_name]

$ scrapy list

$ scrapy fetch [https://example.com]

scrapy command [-o output] [-s setting=value] [options] [arguments]

Scrapy is a Python framework for web scraping and crawling. It handles requests, parsing, and data extraction with built-in support for following links, handling cookies, and respecting robots.txt.
Projects contain spiders - classes that define how to scrape sites. Spiders specify start URLs, parse response with CSS/XPath selectors, and yield items or further requests.
The shell command provides interactive testing. You can experiment with selectors on live pages before writing spider code. Response object methods match spider context.
Items define the data structure being scraped. Item pipelines process scraped data: validation, cleaning, storage to databases or files. Multiple output formats are supported.
Middleware handles request/response processing: user agents, proxies, retries, cookies. Settings control behavior: concurrency, delays, download timeouts, and more.
Extensions add functionality: stats collection, throttling, autothrottle, and custom callbacks.

startproject NAME

Create new Scrapy project.

Generate spider from template.

Run a spider.

Interactive shell for testing.

List available spiders.

Run contract checks.

Fetch URL and print.

Open URL in browser.

Parse URL with spider.

Run spider from file.

Output file (json, csv, xml).

Override setting.

Spider argument.

Output format.

Disable logging.

Log level: DEBUG, INFO, WARNING.

JavaScript-rendered content requires Splash or Selenium integration. Some sites block scrapers via rate limiting or CAPTCHAs. Aggressive scraping may violate ToS. Robots.txt should be respected. Debug shell doesn't persist state.

Scrapy was created by Pablo Hoffman and Shane Evans at Insophia around 2008. It grew from internal tools to a general-purpose framework. The project became one of the most popular Python scraping tools, with commercial company Scrapinghub (now Zyte) providing support and services.

curl(1), wget(1), beautifulsoup(1), playwright(1)