Crawl a single domain outputting the website's sitemap as a list of pages with each page's static assets and links to other pages.
- Install the dependencies from
requirements.txt:
$ pip install -r requirements.txt- And then run
crawl.py:
$ python crawl.py