scrapy
scrapy
TLDR
Create a project
$ scrapy startproject [project_name]
Create a spider (in project directory)
$ scrapy genspider [spider_name] [website_domain]
Edit spider (in project directory)
$ scrapy edit [spider_name]
Run spider (in project directory)
$ scrapy crawl [spider_name]
Fetch a webpage as Scrapy sees it and print the source to stdout
$ scrapy fetch [url]
Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity)
$ scrapy view [url]
Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available)
$ scrapy shell [url]
