Resty in my server logs

[modernjess]

The following user agent showed up in my server logs:

go-resty/2.5.0-dev (https://github.com/go-resty/resty)

My problem with this is that the user agent presents as a crawler, and the address is simply this project's main page, which contains no explanation for why this crawler is showing up in my logs. If you're going to present a user agent that looks like a crawler, the least you could do is to explain why or what it does in the pointed-to page. This project does neither, and so can be considered to be operating outside of well-established etiquette for crawlers.

And that, coupled with the fact that I had to bother posting this, earns resty (which I still don't know what it is) a spot on my sites blacklist.

Cheers,
Jess

[moorereason]

So, your request is to remove the comment/URL after the version string?

[modernjess]

No, my request is to point to a page that actually explains what resty is and why I, as a webmaster, would want to allow it. This is standard operating procedure for every well-behaved crawler. Lack of this very basic requirement is generally viewed as a sign of a hostile crawler.

[moorereason]

my request is to point to a page that actually explains what resty is

The Github project page describes resty as a "Simple HTTP and REST client library for Go." I'm not sure how that could be more clear, but I'll give it a go:

Resty is not a crawler. It's an HTTP client library for the Go programming language. People use resty to handle HTTP client requests in their own applications. Whether they decide to create a crawler app and use resty to handle the individual HTTP client requests or not is up to them.

Resty ships with a default User-Agent header. It's up to the user to override the default header.

Resty is not a crawler.

[modernjess]

I take it you've never actually seen a crawler info page. Here's an example of one: https://bot.seekport.com

The Github project page describes resty as a "Simple HTTP and REST client library for Go."

That's about as generic and meaningless as it gets. From a webmaster perspective, that statement means absolutely nothing. What we want to know is:

• Whose bot is this?
• What is the purpose of this bot?
• Why should the webmaster allow this bot to crawl their site (aka "what's in it for me")

Resty is not a crawler. It's an HTTP client library for the Go programming language.

That's the kind of information that should be front and center. See? Not so hard.

Note that webmasters have a very limited amount of patience for hunting down info for the thousands of entities that crawl a typical website. If this information isn't top-dead-center on a dedicated landing page (and not buried in a GitHub readme) then likely as not they will just ban any client presenting that user agent string.

Resty ships with a default User-Agent header. It's up to the user to override the default header.

Taking off my webmaster hat and putting on my programmer hat, I think that's a poor design decision. It will likely get resty blamed for the sins of others. Your choice, though.

I think the fundamental problem here is that resty looks like a crawler. It has an address buried in the user agent string, and that generally can be inferred to mean that it's a crawler. To put an address in the user agent string and then not provide any of the customary crawler information is just poor form. Either provide an address and a suitably concise explanation of resty, or don't provide any address.

Again, your choice.

[moorereason]

Opened #609