- Don't respect robots.txt file by default, PR#41
- Add
WaybackArchiver::respect_robots_txt=configuration option, to control whether to respect robots.txt file or not - Update
spidrgem, resolves issue#25 - Set default concurrency to
1due to harsher rate limiting on Wayback Machine - Support for crawling multiple hosts, for example www.example.com, example.com and app.example.com PR#27
- Archive every page found, not only HTML pages - #24 thanks @chlorophyll-zz.
- Track what urls have been visited in sitemapper and don't visit them twice
- Protect sitemap index duplicates
Is history...