'Host' header contains :80 or :443

[paambaati]

Most clients (including curl) and browsers leave out the :80 & :443 port suffixes in the Host header, and include it only when it is neither of the two. Dropping the port prefix for known services is still RFC-compliant; see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23

urllib, however, always includes the port and this can cause incompatibilities with other server implementations.

Examples of other projects stripping the port suffix —

1. Ruby - https://github.com/ruby/ruby/blob/d46336e71731aa64d71d4573b2741b7de43ec340/lib/net/http/generic_request.rb#L18
2. Request module - Port 80 is included in the 'Host' request field (instead of defaulted) when it isn't with other popular HTTP clients request/request#515 (comment)
3. Omit port in Host header for default ports waterlink/rack-reverse-proxy#3
4. Use request.hostname, not request.headers.host twilio/twilio-node#151
5. https://stackoverflow.com/a/19169664

[fengmk2]

urllib won't auto set 'Host' header by default.
Is there any way to repeat your problem?

[paambaati]

I'll close this now. I tried replicating this with a minimal setup, and it works and I see no issue. This must've been injected elsewhere in our stack.

Thanks for responding, and sorry for the trouble @fengmk2!

For anyone else interested, here's a simple echo server that logs request headers —

// Server that listens on 'localhost:80'.
var http = require('http');
 
http.createServer(function(request,response){
	console.log('HEADERS: ', request.headers);
	response.writeHead(200);
	request.on('data',function(message){
		response.write(message);
	});
	request.on('end',function(){
		response.end();
	});
}).listen(80);

And a simple request to the echo server —

// Send a request to 'localhost:80' using urllib
var urllib = require('urllib');

urllib.request('http://localhost:80', function (err, data, res) {
  if (err) {
    throw err; // you need to handle error
  }
  console.log(res.statusCode);
  console.log(res.headers);
  // data is Buffer instance
  console.log(data.toString());
});