Expand HTTP request response monitoring

Expand HTTP request/response monitoring

Background information: Firefox supports remote developer tools - ie. communicating with an arbitrary server that implements a protocol for exposing information about web content. Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by exposing further information about HTTP requests and responses that web content initiates.

Initial steps:

• compile Servo and ensure that it runs on tests/html/about-mozilla.html
• email the mozilla.dev.servo mailing list introducing your group and your progress
• find an interesting site that doesn't crash Servo and attach the developer tools; look for network requests that are logged when navigating, for example.
• address the TODOs in http_loader.rs related to only sending messages if there is a PipelineId present (ie. LoadData::pipeline_id), and add a test for the new behaviour to tests/unit/net/http_loader.rs
• refactor duplicate calls to send_request_to_devtools to avoid repetition
• pass values to send_request_to_devtools derived from the actual request (eg. req.headers_mut()), instead of the original LoadData, and add a test for the new behaviour to tests/unit/net/http_loader.rs
• add an Option<PipelineId> argument to load_whole_resource and pass it to the LoadData constructor

Subsequent steps:

• implement the getRequestHeaders/getRequestPostData message handlers in components/devtools/actors/network_event.rs based on the values in the HttpReqest struct
• add support for more flavours of the networkEventUpdate message than just responseStart, by building the NetworkEventUpdateMsg` from a freeform JSON object instead of a serialized struct declaration
• send real values for the fields of ResponseStartMsg (requires obtaining those values in http_loader.rs and sending them in send_response_to_devtools)
• implement the getRequestCookies and getResponseCookies message handlers in components/devtools/actors/network_event.rs based on the Cookie/Set-Cookie headers in the HttpRequest/HttpResponse structs
• implement the getResponseContent/getResponseHeaders message handlers based on the data obtained in add_http_response
• implement remaining flavours of networkEventUpdate message
• address the TODO for event_actor in network_event.rs by providing the missing information as part of the original message transmitted by send_request_to_devtools

Combined, all of these changes should improve the accuracy of the information displayed by the Firefox web console when looking at logged information about network requests.

Other useful information Live packet dumps captured from a Firefox session. This is ideally what the messages that Servo's server sends to Firefox should resemble:

{
  "from": "server1.conn0.consoleActor2",
  "type": "networkEvent",
  "eventActor": {
    "actor": "server1.conn0.netEvent45",
    "startedDateTime": "2015-04-22T20:47:08.545Z",
    "url": "https://aus4.mozilla.org/update/3/Firefox/40.0a1/20150421152828/Darwin_x86_64-gcc3/en-US/default/Darwin%2013.4.0/default/default/update.xml?force=1",
    "method": "GET",
    "isXHR": true,
    "private": false
  }
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "requestCookies",
  "cookies": 4
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "responseStart",
  "response": {
    "httpVersion": "HTTP/1.1",
    "remoteAddress": "63.245.217.43",
    "remotePort": 443,
    "status": "200",
    "statusText": "OK",
    "headersSize": 337,
    "discardResponseBody": true
  }
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "securityInfo",
  "state": "secure"
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "responseHeaders",
  "headers": 9,
  "headersSize": 337
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "responseCookies",
  "cookies": 0
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "eventTimings",
  "totalTime": 798
}

{
  "from": "server1.conn0.netEvent45",
  "type": "networkEventUpdate",
  "updateType": "responseContent",
  "mimeType": "text/xml; charset=utf-8",
  "contentSize": 0,
  "transferredSize": 42,
  "discardResponseBody": true
}