URNs do not have any generally defined hierarchical components

[kareljuricka]

Problem:

Problem is in file lib/crawler.js.

On same pages URI.js throws this exceptions where its method absoluteTo is called.

Solution:

I think there should by Try/Catch block around that lines 653+ like it is around that in other cases (line 774 or 850).

var redirectTarget = uri(response.headers.location)
.absoluteTo(robotsTxtUrl)
.normalize();

[fredrikekelund]

Hi! This was addressed in 9f4e4ec. More details in #363 as well. A patch release with these changes is due soon! Thanks for reporting 👍