HTTP/2 Support

[dshafik]

This issue is intended to document the current and ongoing status of HTTP/2 support in Guzzle.

Current Status

Using Curl

The only handler that can support HTTP/2 at present is Curl (see below for streams handler info).

For Curl to support HTTP/2 needs to be built with --with-nghttp2 and be >= 7.33.0. For multiplex support, you need >= 7.43.0.

To detect support for HTTP/2 in libcurl from PHP, you must use:

if (curl_version()["features"] & CURL_VERSION_HTTP2 !== 0) {
    // HTTP/2 support
}

The CURL_VERSION_HTTP2 constant was added in PHP 5.5.24, by using the constant value, 65536, or adding something like this to cover all HTTP/2 related constants:

defined('CURL_VERSION_HTTP2') || define('CURL_VERSION_HTTP2', 65536);
defined('CURL_HTTP_VERSION_2_0') || define('CURL_HTTP_VERSION_2_0', 3);
defined('CURL_HTTP_VERSION_2') || define('CURL_HTTP_VERSION_2', CURL_HTTP_VERSION_2_0);
defined('CURLPIPE_NOTHING') || define(CURLPIPE_NOTHING, 0);
defined('CURLPIPE_HTTP1') || define('CURLPIPE_HTTP1', 1);
defined('CURLPIPE_MULTIPLEX') || define('CURLPIPE_MULTIPLEX', 2);

At present, with the merge of #1246, it is now possible to use HTTP/2 for synchronous requests:

use GuzzleHttp\Client;

$client = new Client();
$client->get('https://http2.akamai.com/demo/tile-0.png', [
    'curl' => [
        CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_2_0,
    ],
    'debug' => true,
]);

This will yield debug output including the following, indicating it's using HTTP/2 successfully:

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8084840800)
> GET /demo/tile-0.png HTTP/1.1
Host: http2.akamai.com
User-Agent: GuzzleHttp/6.0.2 curl/7.44.0 PHP/7.0.0-dev

* http2_recv: 16384 bytes buffer at 0x7f8084841170 (stream 1)
* http2_recv: 16384 bytes buffer at 0x7f8084841170 (stream 1)
* http2_recv: 16384 bytes buffer at 0x7f8084841170 (stream 1)
* http2_recv: returns 1314 for stream 1
< HTTP/2.0 200

Unfortunately, if you attempt to use it with libcurl compiled without http2 support, then the following happens:

First the response from the request is output to STDOUT (because CURLOPT_RETURNTRANSFER is set to false and it seems to fail to use the streams stuff), then the following exception is thrown:

GuzzleHttp\Exception\RequestException: cURL error 0: The cURL request was retried 3 times and did not succeed. The most likely reason for the failure is that cURL was unable to rewind the body of the request and subsequent retries resulted in the same error. Turn on the debug option to see what went wrong. See https://bugs.php.net/bug.php?id=47204 for more information. (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in /Users/dshafik/src/php-curl-http2-multiplex/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 170

Call Stack:
    0.0075     230648   1. {main}() src/php-curl-http2-multiplex/http2-guzzle-sync.php:0
    0.0900    1089168   2. GuzzleHttp\Client->get() src/php-curl-http2-multiplex/http2-guzzle-sync.php:12
    0.0900    1089688   3. GuzzleHttp\Client->__call() src/php-curl-http2-multiplex/http2-guzzle-sync.php:12
    0.0900    1089760   4. GuzzleHttp\Client->request() src/php-curl-http2-multiplex/vendor/guzzlehttp/guzzle/src/Client.php:88
    0.1536    1527176   5. GuzzleHttp\Promise\Promise->wait() src/php-curl-http2-multiplex/vendor/guzzlehttp/guzzle/src/Client.php:130

Backwards Compatibility

We should transparently downgrade to HTTP/1.1, and CURLPIPE_HTTP1, if libcurl does not support HTTP/2, and/or multiplexing respectively.

Using Streams

To implement HTTP/2 for the streams adapter would require either:

1. HTTP/2 support in the http/s streams
2. A new http2 stream, either in core, or as an extension
3. A user land implementation and custom stream. It is possible to override an existing stream (unregister, and register custom one), so it could be transparent, or just use h2://or http2:// or some such (or both).
4. A new http2 extension, which can be used to implement an http2 supported stream

Testing HTTP/2

To properly test HTTP/2 support, we need to do a few different test approaches:

HTTP/2 Client -> HTTP/2 Server

The current server.js only supports HTTP/1.1, which means that we cannot test actual HTTP/2 requests against it. However, there is a fairly drop-in replacement for node's http module, which is http2 found here: https://github.com/molnarg/node-http2. I currently have update GuzzleHttp\Tests\Server to allow you to specify which version of HTTP being used, and spin up server.js or a new server2.js — however these scripts are nearly identical and it would be trivial to handle that as an argument sent to server.js. The entire diff looks like this:

--- tests/server.js 2015-07-13 08:00:16.000000000 +1000
+++ tests/server2.js    2015-09-06 14:30:09.000000000 +1000
@@ -1,7 +1,9 @@
 /**
- * Guzzle node.js test server to return queued responses to HTTP requests and
+ * Guzzle node.js test HTTP/2 server to return queued responses to HTTP requests and
  * expose a RESTful API for enqueueing responses and retrieving the requests
  * that have been received.
+ * 
+ * Run `npm install` to install dependencies
  *
  * - Delete all requests that have been received:
  *      > DELETE /guzzle-server/requests
@@ -38,8 +40,9 @@
  * @license See the LICENSE file that was distributed with this source code.
  */

-var http = require('http');
+var http = require('http2');
 var url = require('url');
+var fs = require('fs');

 /**
  * Guzzle node.js server
@@ -199,8 +202,12 @@
   };

   this.start = function() {
-
-    that.server = http.createServer(function(req, res) {
+    var options = {
+      key: fs.readFileSync(__dirname + '/node_modules/http2/example/localhost.key'),
+      cert: fs.readFileSync(__dirname + '/node_modules/http2/example/localhost.crt')
+    };
+    
+    that.server = http.createServer(options, function(req, res) {

       var parts = url.parse(req.url, false);
       var request = {

HTTP/2 Client -> HTTP/1.1 Server

Currently libcurl always does an HTTP/1.1 request and attempts to upgrade the connection to HTTP/2, this method is available so that you can always make HTTP/2 requests regardless of server support, and it will automatically fall back to HTTP/1.1. Support for prior-knowledge requests are planned for libcurl, so it might be possible to do native HTTP/2 rather than HTTP/1.1+Upgrade in the future.

This means we should also run HTTP/2 intended requests against the existing HTTP/1.1 server.

It would be great if it were possible to detect failure to upgrade and to switch to CURLPIPE_HTTP1 transparently if they were attempting to use CURLPIPE_MULTIPLEX.

Desired API

Enabling HTTP/2 should be as simple as setting the request option version => 2 (or 2.0). Due to the HTTP/2 upgrade negotiation, it's entirely BC to actually set this as the default (see performance related section below).

This means that it will automatically use HTTP/1.1 if HTTP/2 is not available, we do not have to handle this ourselves.

Once the version is set, concurrent requests should attempt to use HTTP/2 multiplexing. Failing to do so should switch to HTTP pipelining transparently.

While the overhead of CURLPIPE_HTTP1 is greater, it drastically faster than not using it, giving closer to the performance expected with CURLPIPE_MULTIPLEX.

Server Push

Server push is the ability for the HTTP server to inject responses into the clients cache in response to a different request that it expects will also be wanted. An example of this would be requesting an HTML page, and having the server also push you the CSS, JS, and images, rather than waiting for the client to have to parse the HTML and make additional requests for those resources.

libcurl supports this feature in >= 7.44.0, and it works by registering a callback to accept or reject the request. The callback accepts (among other things) an easy handle representing the request. Rejecting the request requires returning CURL_PUSH_DENY, otherwise if you return CURL_PUSH_OK it will add the easy handle to the multihandler and you can handle it as you would normally from that point.

If the callback is not defined then all server pushes are rejected automatically.

PHP does not currently support this feature.

Also, while this feature is callback based, it might be the most difficult to implement consistently among handlers. I would recommend waiting till at least one other handler supports it before implementing.

HTTP/2 Performance & When To Use It

Currently, sending a single HTTP request using HTTP/2 is slower than using HTTP/1.1. I believe this might be to do with the HTTP/1.1+Upgrade that is done. This means if you are doing just 1:1 request:host then you should use HTTP/1.1 at present.

However you are doing multiple HTTPS requests to the same host, and using multiplexing, then you get a significant performance increase due to only needing a single SSL negotiation for all requests on the same multiplexed connection.

[TheBabaYaga]

👍

[mtdowling]

Thanks for putting this together, @dshafik!

Note that the preferred way of specifying the HTTP version is through the version request option:

use GuzzleHttp\Client;

$client = new Client();
$client->get('https://http2.akamai.com/demo/tile-0.png', [
    'versionl' => 2.0,
    'debug' => true,
]);

First the response from the request is output to STDOUT (because CURLOPT_RETURNTRANSFER is set to false and it seems to fail to use the streams stuff), then the following exception is thrown:

Because we are using callbacks to receive data, this seems like a bug in either PHP or cURL. Setting CURLOPT_RETURNTRANSFER, from what I understand, would unnecessarily buffer the response in memory.

[dshafik]

Can confirm that 'version' => 2 works just fine, and in fact is what I put in my slides ;)

[jeroenjoosen]

👍

[stijnsymons]

👍

[rubenvdb]

👍

[jdecoster]

👍

[belens]

👍

[dshafik]

@mtdowling check out master...dshafik:http2-tests — I'm really not happy about this, so if you have a better suggestions, that'd be great.

Because the current server.js only handles HTTP/1.1, I updated server.js to use http2 when appropriate. It now allows you to specify the version as the first argument, and starts the correct version of the HTTP server. Note: it now uses npm to install the http2 dependency (npm install in the project root, uses package.json).

So once I had that I the ability to specify a HTTP version for Server which will send that on to server.js as the first argument. Because of this, you can no longer start the server (Server::start()) in the bootstrap, I created a simple trait, ServerTrait and Http2\ServerTrait which will add setupBeforeClass() and tearDownAfterClass() methods that set the version and start/stop the server.

It will start the HTTP/1.1 server on port 8126, and HTTP/2 on 8127 (with SSL).

I then simply extend the original test, and use Http2\ServerTrait, to create an HTTP/2 variant of the test.

Goals:

• Run all existing tests against an HTTP/2 server — by extending the HTTP/1.1 test we avoid duplication

Examples:

• ServerTrait
• Http2\Handler\CurlFactoryTest

Issues:

• For some reason, some HTTP/2 tests are still hitting the HTTP/1 server, not sure why, probably due to statics. I think removing the static stuff will fix this, but I didn't want to make this drastic a change without input
• The library should transparently fall back to HTTP/1.1 if ext/curl is not compiled with nghttp2, which would mean that the test suite would fall back. This means that it's difficult to test both behaviors (HTTP/2 with and without support). Ideally we'd have a "HTTP/2" test suite, that would skip the tests if HTTP/2 support isn't available, and an "HTTP/2 without support" test suite, that would run on both systems with and without HTTP/2 support.

[mtdowling]

Just have a few minutes for now to look at this. I'll try to find more time later to provide more feedback.

I don't like changing the existing static public properties of the test class to return arrays and use functions. I think it would be better to somehow add new matching properties for HTTP/2 variables (e.g., $urlHttp2).

The library should transparently fall back to HTTP/1.1

I don't think Guzzle, PHP's HTTP/2 support, or HTTP/2 in general are ready to be the default way that Guzzle sends requests. I think HTTP/2 should be an opt-in at this point.

[dshafik]

Guzzle HTTP/2 comment

@mtdowling I agree about not having HTTP/2 as default, but I meant it should fall back to HTTP/1.1 when attempting to use HTTP/2 on a system where libcurl is not compiled with HTTP/2 support.

Ideally the only difference between the two APIs would be that you set the version to 2, HTTP/1.1 would use pipelining for async, where 2 would use multiplexing. Non-async requests would just use the requested protocol if possible.

As I said, I'm not really happy with the test suite, so any more concrete suggestions would be great!

[rafaga]

Nice! 👍

[dshafik]

Some minor updates:

• PHP 7.1 will support curl's Server Push support
• It is possible to have curl default to HTTP/2 for TLS requests, and stick to HTTP/1.1 for non-encrypted requests by setting CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_2TLS (added in curl 7.47)
• Curl now supports prior knowledge connections by setting CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE (added in curl 7.49)

[dshafik]

@mtdowling I'd very much like to move this forward, where do you stand on it?

[philsturgeon]

Hey @dshafik @mtdowling I'm very curious about this myself. HTTP/2 has been around for a while now, and we can't keep ignoring it forever. I'm starting to use it over in Ruby land more and more, but our company has a bunch of PHP and I want them to get in on the party.

My experiments are using a ruby implementation https://github.com/ostinelli/net-http2, which is a little different to your curl-based approach.

Anyway, necrobump.

[theavuthnhel]

Hello anyone please help me?

I have two project separated: (1.) Laravel for user Key-in Data (2.) Laravel for API
(1.) Laravel for user Key-in Data work fine in local machine send Data to (2.) but on server not send body's data.

(1.) Key-in Data Project

$client = new Client(['headers' => ['X-Client-Code' => env('KEY_CODE')]]);
$request_param = [
            'client_id'    => $request->client_id,
            'code'         => $request->code,
            'phone_number' => $request->phone_number,
            'sms_type'     => 'card'
        ];
$request_data = json_encode($request_param);
$res = $client->request(
            'POST',
            url(env('API_URL').'api/v0/user/activation'),
            [
                'headers' => [
                    'Accept'     => 'application/json',
                    'Authorization' => file_get_contents(storage_path('credential').'/.token')

                ],
                'body'   => $request_data
            ]
        );
return $res->getBody()->getContents();

(2.) Laravel API

$ehealth_code = $request->headers->get('x-client-code');
$data = json_decode($request->getContent(),true); // return empty

Anyone know why on Ubuntu server 18.04 the Guzzle not sending body's data but in local iMac it's working fine.

[vv12131415]

this issue is from 2015.

Can I make PR for this feature?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 2 weeks if no further activity occurs. Thank you for your contributions.

[philsturgeon]

Hey folks! I know we're all super busy and I wouldn't normally hassle or bump, but this bot is throwing some shade into a very important issue which has so far been progressing in a rather underwhelming fashion.

Is anything going to happen here? Can we be whelmed?! HTTP/3 is on the way so now more than ever it is time to consider moving beyond the restrictions of ye-olde HTTP/1.

[sagikazarmark]

Yeah, it would nice to have this finally in Guzzle! Sadly, I don't think we have any resources for this right now, so any help would be much appreciated (it's going to be Hacktoberfest soon after all).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 2 weeks if no further activity occurs. Thank you for your contributions.

[Nyholm]

I dont follow the specifications discussions too carefully, but isn't H2 push considered as a misstake? Ie, it is cool and all, but it is not as useful and we first thought.