HTTP/2 multiplexing

[Daniel Stenberg]

Hey friends,

I've started out this week working on HTTP/2 multiplexing for libcurl. My idea
is that when using the multi interface and adding easy handles, all handles
that identify the same "origin" (host + port + protocol combination really)
will use the same single connection and do "real" HTTP/2 multiplexing over it.

The current libcurl version (7.42.0) does not support this but it will simply
do one connection for each HTTP/2 transfer.

There are some outstanding questions, like if we should do this by default
without any options to ask for it and if we really should forcibly limit the
number of connections to the same host to one. And I'm sure there will pop up
a few more questions going forwarwd.

Right now I use CURLMOPT_PIPELINING as a signal to activate multiplexing. I
think this is fine for now, but I also imagine that there are users who'd like
HTTP/2 multiplexing but not HTTP/1.1 pipelining so they'd have to use
different bits if so...

Right now I need CURLMOPT_MAX_HOST_CONNECTIONS set to 1 to really stay on one
connection only, and again, this may be fine for HTTP/2-only users but it may
not be what users want for all protocols at once. Not sure how to handle this,
but I'm not entirely happy with this and neither would I be to add a totally
option like CURLMOPT_MAX_HOST_HTTP2_CONNECTIONS for this specific protocol
version.

Both these choices need considerations before we let this out in a release -
not that we're there yet.

Git branch! I'm about to create an 'http2-multiplex' branch in git and push my
ongoing http2 multiplexing work to allow interested people to join in,
participate or just follow along and test the bleeding edge. It is meant to be
experimental and

Status!

I run nghttpd locally as a test server. HTTPS only to start with.

With only a few changes I've managed to get libcurl to send off two requests
multiplexed to the server. libcurl cannot yet handle the multiplexed responses
since we really have no proper concept internally to split up such a response
into the individual streams. Oh, and btw, each easy handle added to the multi
handle becomes a separate stream which feels completely sane and natural from
an API user's point of view.

Stay tuned for more details as things develop. Feel free to join in and help
or ask or suggest or whatever you like!

[Lucas Pardue]

Hello Daniel,

It is good to see work starting on this feature.

Post by Daniel Stenberg
Right now I need CURLMOPT_MAX_HOST_CONNECTIONS set to 1 to really
stay on one connection only, and again, this may be fine for HTTP/2-only
users but it may not be what users want for all protocols at once. Not sure
how to handle this, but I'm not entirely happy with this and neither would I
be to add a totally option like
CURLMOPT_MAX_HOST_HTTP2_CONNECTIONS for this specific protocol
version.

The connection management issues are interesting, after rereading section 9.1 of the HTTP/2 spec it seems like having to set a value for either CURLMOPT_MAX_HOST_CONNECTIONS or CURLMOPT_MAX_HOST_HTTP2_CONNECTIONS would be a bit awkward. The spec states "Clients SHOULD NOT open more than one HTTP/2 connection to a given host and port pair" and mentions a few cases where additional connections may be created as replacements; stream ID exhaustion, refresh of TLS keying material or replacement for connection encountering errors. To me this area feels like something better handled implicitly in libcurl, with the standard behaviour being a single connection.

Post by Daniel Stenberg
I run nghttpd locally as a test server. HTTPS only to start with.

I would gently ask to test non-HTTPS (h2c) soon too. In the past we have found issues with the h2c connection upgrade and related behaviour. The issues have been fixed along the way but it would be unfortunate to regress. For example, with the new changes if we can send multiple requests in parallel, which one performs the connection upgrade and does it stall the other requests until completion?

Regards
Lucas

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http:/

[Daniel Stenberg]

Post by Lucas Pardue
The connection management issues are interesting, after rereading section
9.1 of the HTTP/2 spec it seems like having to set a value for either
CURLMOPT_MAX_HOST_CONNECTIONS or CURLMOPT_MAX_HOST_HTTP2_CONNECTIONS would
be a bit awkward. The spec states "Clients SHOULD NOT open more than one
HTTP/2 connection to a given host and port pair" and mentions a few cases
where additional connections may be created as replacements; stream ID
exhaustion, refresh of TLS keying material or replacement for connection
encountering errors. To me this area feels like something better handled
implicitly in libcurl, with the standard behaviour being a single
connection.

Oh yes, we're up for some fun juggling with ideas and concepts on how we want
to allow applications to control this!

One additional little detail to the HTTP/2 spec language is that we won't know
if we can do HTTP/2 or not for a specific transfer until after it has been
negotiated. If we for example add N transfers at once to libcurl, it'll do
them in parallell (ie it will favour getting stuff done now versus waiting and
seeing if it can do something else later).

That's also why I (still for now in my http2 multiplexing test application)
limit the number of connections to the same host to 1 as then the second
connection will wait for the first to negotiate h2 and then it can go on
multiplexed...

So yeah, there are plenty of outstanding questions and decisions to make. I'll
still move on with the simple approach I have now since I'm focusing on
getting the multiplexing stuff to work first before I dig deeper into how to
set the limits or handle connections.

Post by Lucas Pardue
I would gently ask to test non-HTTPS (h2c) soon too. In the past we have
found issues with the h2c connection upgrade and related behaviour.

Of course it will be tested. I'm just focusing on HTTPS for now since it
(ironically) is easier.

I also want to add proper http2 tests to the curl test suite by

1. short term use nghttpd as a test server

and then

2. build a custom http2 server with the nghttp2 API and use _that_ so that we
can instruct the server from the test scripts to excercise curl in various fun
and weird ways that it should deal with

Post by Lucas Pardue
The issues have been fixed along the way but it would be unfortunate to
regress. For example, with the new changes if we can send multiple requests
in parallel, which one performs the connection upgrade and does it stall the
other requests until completion?

Back to the connection limiter. If we limit the number of connections to the
same host to 1, the first easy handle that use a specific host will do the
connect and once it gets past the "connect state" it'll allow other easy
handles to use the same connection... and then we get the situation that the
following streams can use an already setup HTTP/2 connection in the plain but
without Upgrade.

Hm, btw, I think I'll also push my http2 test setup to github in case someone
wants to play along. I'll do that in a separate test repo and I'll mention it
here when it is up.

[Daniel Stenberg]

Post by Daniel Stenberg
Hm, btw, I think I'll also push my http2 test setup to github in case
someone wants to play along. I'll do that in a separate test repo and I'll
mention it here when it is up.

My playground area for testing my multiplexing work is now on github:

https://github.com/bagder/curl-http2-dev

[Yehezkel Horowitz]

Hi Daniel

I've started out this week working on HTTP/2 multiplexing for libcurl. My idea is that when using the multi interface and adding easy handles, all handles that identify the same "origin" (host + port + protocol combination really) will use the same single connection and do "real" HTTP/2 multiplexing over it.

Please take connection security into account (like the well-known issues we already have with NTLM in HTTP/1.1)

For the current stage of dev. it is fine to ignore it, but it will wait to be handled before the release...

Thanks

Yehezkel

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.ht

[Daniel Stenberg]

Post by Yehezkel Horowitz

Post by Daniel Stenberg
I've started out this week working on HTTP/2 multiplexing for libcurl. My
idea is that when using the multi interface and adding easy handles, all
handles that identify the same "origin" (host + port + protocol combination
really) will use the same single connection and do "real" HTTP/2
multiplexing over it.

Please take connection security into account (like the well-known issues we
already have with NTLM in HTTP/1.1)

Ah yes, thanks. I'm sure that I'll get reasons to double-check the logic for
this a few times. Adding things like NTLM to the HTTP/2 mix is certainly going
to give me some interesting moments!

Post by Yehezkel Horowitz
For the current stage of dev. it is fine to ignore it, but it will wait to
be handled before the release...

Roger that. And thanks for keeping me on my toes!

[Nico Williams]

Post by Daniel Stenberg
There are some outstanding questions, like if we should do this by
default without any options to ask for it and if we really should
forcibly limit the number of connections to the same host to one.
And I'm sure there will pop up a few more questions going forwarwd.

Limiting the number of connections to one per-multi handle has a nice
property that it's predictable and gets libcurl out of the business of
having to pick a connection optimally. The downside of that is that it
gets the application into that business, when it could just have been
libcurl.

Post by Daniel Stenberg
Right now I use CURLMOPT_PIPELINING as a signal to activate
multiplexing. I think this is fine for now, but I also imagine that
there are users who'd like HTTP/2 multiplexing but not HTTP/1.1
pipelining so they'd have to use different bits if so...

Yes, probably!

Just add CURLMOPT_MULTIPLEXING, no?

Post by Daniel Stenberg
Right now I need CURLMOPT_MAX_HOST_CONNECTIONS set to 1 to really
stay on one connection only, and again, this may be fine for
HTTP/2-only users but it may not be what users want for all
protocols at once. Not sure how to handle this, but I'm not entirely
happy with this and neither would I be to add a totally option like
CURLMOPT_MAX_HOST_HTTP2_CONNECTIONS for this specific protocol
version.

Multiple connections -> load balancing between them. Which is fine, but
probably hard to get right. If the application knows how to get it
right, let it multiple across multiple multi handles. Otherwise let
libcurl load balance as it wishes.

Using multiple multi handles with one event loop has to be easy though.
(I've never tried it. But looking at the API it looks easy enough.)

Nico

[Alcides Viamontes Esquivel]

Post by Daniel Stenberg

Post by Daniel Stenberg
Hm, btw, I think I'll also push my http2 test setup to github in case
someone wants to play along. I'll do that in a separate test repo and

I'll

Post by Daniel Stenberg

Post by Daniel Stenberg
mention it here when it is up.

https://github.com/bagder/curl-http2-dev

This is cool. I was playing with your code and noticed that so far it does
one request after another... or so it seems to me. Just in case you have
any use for it, I'm putting below curl's log.

The server code that I used to play with this can be found at
https://github.com/alcidesv/lock_step_transfer ... it is a server that
refuses to finish one stream while the other has not been opened.

== 0 Info: STATE: INIT => CONNECT handle 0x24e75d8; line 1034 (connection
#-5000)
== 0 Info: Added connection 0. The cache now contains 1 members
== 0 Info: Trying 127.0.0.1...
== 0 Info: STATE: CONNECT => WAITCONNECT handle 0x24e75d8; line 1087
(connection #0)
== 0 Info: Connected to localhost (127.0.0.1) port 8443 (#0)
== 0 Info: Marked for [keep alive]: HTTP default
== 0 Info: ALPN, offering h2-14, http/1.1
== 0 Info: successfully set certificate verify locations:
== 0 Info: CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
== 0 Info: TLSv1.2, TLS Unknown, Unknown (22):
0 => Send SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS handshake, Client hello (1):
0 => Send SSL data, 512 bytes (0x200)
0000: ......C...{....p:..k....D...x.."a...>0....0.,.(.$.............k.
0040: j.i.h.9.8.7.6.........2...*.&.......=.5.../.+.'.#.............g.
0080: @.?.>.3.2.1.0.........E.D.C.B.1.-.).%.......<./...A.............
00c0: ............3.........localhost...........:.8...................
0100: ........................................ .......................
0140: ..............3t.........h2-14.http/1.1.........................
0180: ................................................................
01c0: ................................................................
== 0 Info: STATE: WAITCONNECT => PROTOCONNECT handle 0x24e75d8; line 1223
(connection #0)
== 1 Info: STATE: INIT => CONNECT handle 0x24f04f8; line 1034 (connection
#-5000)
== 1 Info: Found bundle for host localhost: 0x24fe2b8
== 1 Info: Server doesn't support pipelining
== 1 Info: No connections available.
== 1 Info: STATE: CONNECT => CONNECT_PEND handle 0x24f04f8; line 1053
(connection #-5000)
== 0 Info: SSLv2, Unknown (22):
0 <= Recv SSL data, 5 bytes (0x5)
0000: ....n
== 0 Info: TLSv1.2, TLS handshake, Server hello (2):
0 <= Recv SSL data, 110 bytes (0x6e)
0000: ...j..(j.`.T.Y..R\..O.......l....>.... P...9K.}.&.>..uN..,@o....
0040: c..L..../..".............................h2-14
== 0 Info: SSLv2, Unknown (22):
0 <= Recv SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS handshake, CERT (11):
0 <= Recv SSL data, 1025 bytes (0x401)
0000: ..........0...0.............6...s.0...*.H........0..1.0...U....S
0040: E1.0...U....Weirdo1.0...U....Internet1.0...U....curl hackers gal
0080: ore1.0...U....Moo1.0...U....debugit1.0...****@example.
00c0: com0...150425181751Z..180424181751Z0..1.0...U....SE1.0...U....We
0100: irdo1.0...U....Internet1.0...U....curl hackers galore1.0...U....
0140: Moo1.0...U....debugit1.0...****@example.com0.."0...*.H
0180: .............0.........9k.-faY..`J......V.........a..?....i.9...
01c0: 0..)....B..e...:....uMF......;..Am..{g......2...C.!.f&.1z<.}....
0200: ....[+***@..fo....;(.t.T_.ne......?uO.j
0240: .nnG..zxYFv8Q...+...3.P;'.h.....JK......Z....rd..bR...3V?.qKl...
0280: ....xn..].cS....Vp.*q.......P0N0...U........X.`_.w9..:..Q...BK0.
02c0: ..U.#..0.....X.`_.w9..:..Q...BK0...U....0....0...*.H............
0300: ..I.......ExBA-..{X=.|..[.......k.E....{...q.'...E..#....[.K/O.v
0340: {g._...06..+.V7.....%n#Y:+&..%.......6.....G...$[H...Nk.....D<..
0380: ...sIx.~../..O..._a.......I#.....O.P{....M....V".<..6t...< .M'Z.
03c0: .g.......m.,"...+..Kv.M...f...x ...V3.g.'k..Q..B....Y.A.~n....&e
0400: w
== 0 Info: SSLv2, Unknown (22):
0 <= Recv SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS handshake, Server key exchange (12):
0 <= Recv SSL data, 413 bytes (0x19d)
0000: ............4.......4...(Cx.K}....fa.az5}..8#.....-.....A.....q5
0040: X..p........S.,....J.n.)***@.1....b
0080: @.C2.........:..`}E..MC......b...R...l...a...1Se....`r|...C...).
00c0: ...?***@..#..E...n"*n.+.ql...w.vi+..m...!-.V+.).$.(.mw..b...r.\cH.
0100: L...1.....9......[..^.^t.. Q...+../.?....%.5.!.....v}....f_44...
0140: .Y6...\.>O..1~.=L...<.C...+.....i......f6h..-.gN..p.(.b.i3..UH
0180: ....h...C....D..zd:Q..D......
== 0 Info: SSLv2, Unknown (22):
0 <= Recv SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS handshake, Server finished (14):
0 <= Recv SSL data, 4 bytes (0x4)
0000: ....
== 0 Info: SSLv2, Unknown (22):
0 => Send SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS handshake, Client key exchange (16):
0 => Send SSL data, 150 bytes (0x96)
0000: ........i.........'y,A..r....W......% ..).:.9$.Lh.;...}.8..w.-l
0040: X...U.#......I.Kcv^\V........iX*.Z.h.....)..H.........N.{pJ..?[.
0080: .l....C.......Qc.c..Q.
== 0 Info: SSLv2, Unknown (20):
0 => Send SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS change cipher, Client hello (1):
0 => Send SSL data, 1 bytes (0x1)
0000: .
== 0 Info: SSLv2, Unknown (22):
0 => Send SSL data, 5 bytes (0x5)
0000: ....(
== 0 Info: TLSv1.2, TLS handshake, Finished (20):
0 => Send SSL data, 16 bytes (0x10)
0000: .....>.d........
== 0 Info: SSLv2, Unknown (20):
0 <= Recv SSL data, 5 bytes (0x5)
0000: .....
== 0 Info: TLSv1.2, TLS change cipher, Client hello (1):
0 <= Recv SSL data, 1 bytes (0x1)
0000: .
== 0 Info: SSLv2, Unknown (22):
0 <= Recv SSL data, 5 bytes (0x5)
0000: ....(
== 0 Info: TLSv1.2, TLS handshake, Finished (20):
0 <= Recv SSL data, 16 bytes (0x10)
0000: ..........!{hBb.
== 0 Info: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
== 0 Info: ALPN, server accepted to use h2-14
== 0 Info: Server certificate:
== 0 Info: subject: C=SE; ST=Weirdo; L=Internet; O=curl hackers galore;
OU=Moo; CN=debugit; emailAddress=***@example.com
== 0 Info: start date: 2015-04-25 18:17:51 GMT
== 0 Info: expire date: 2018-04-24 18:17:51 GMT
== 0 Info: issuer: C=SE; ST=Weirdo; L=Internet; O=curl hackers galore;
OU=Moo; CN=debugit; emailAddress=***@example.com
== 0 Info: SSL certificate verify result: self signed certificate (18),
continuing anyway.
== 0 Info: STATE: PROTOCONNECT => WAITDO handle 0x24e75d8; line 1241
(connection #0)
== 0 Info: WAITDO: Conn 0 send pipe 1 inuse FALSE athead TRUE
== 0 Info: STATE: WAITDO => DO handle 0x24e75d8; line 1267 (connection #0)
== 0 Info: Using HTTP2
== 0 Info: SSLv2, Unknown (23):
0 => Send SSL data, 5 bytes (0x5)
0000: ....0
== 0 Info: http2_send len=63
== 0 Info: before_frame_send() was called
== 0 Info: SSLv2, Unknown (23):
0 => Send SSL data, 5 bytes (0x5)
0000: ....!
== 0 Info: on_frame_send() was called
== 0 Info: before_frame_send() was called
== 0 Info: SSLv2, Unknown (23):
0 => Send SSL data, 5 bytes (0x5)
0000: ....5
== 0 Info: on_frame_send() was called
0 => Send header, 63 bytes (0x3f)
0000: GET /index.html HTTP/1.1
001a: Host: localhost:8443
0030: Accept: */*
003d:
== 0 Info: STATE: DO => DO_DONE handle 0x24e75d8; line 1314 (connection #0)
== 1 Info: STATE: CONNECT_PEND => CONNECT handle 0x24f04f8; line 2780
(connection #-5000)
== 0 Info: STATE: DO_DONE => WAITPERFORM handle 0x24e75d8; line 1441
(connection #0)
== 0 Info: STATE: WAITPERFORM => PERFORM handle 0x24e75d8; line 1454
(connection #0)
== 0 Info: http2_recv: 16384 bytes buffer
== 0 Info: SSLv2, Unknown (23):
0 <= Recv SSL data, 5 bytes (0x5)
0000: ....!
== 0 Info: nread=9
== 0 Info: on_frame_recv() was called with header 4
== 0 Info: nghttp2_session_mem_recv() returns 9
== 0 Info: before_frame_send() was called
== 0 Info: SSLv2, Unknown (23):
0 => Send SSL data, 5 bytes (0x5)
0000: ....!
== 0 Info: on_frame_send() was called
== 1 Info: Found bundle for host localhost: 0x24fe2b8
== 1 Info: Server doesn't support pipelining
== 1 Info: No connections available.
== 1 Info: STATE: CONNECT => CONNECT_PEND handle 0x24f04f8; line 1053
(connection #-5000)
== 0 Info: http2_recv: 16384 bytes buffer
== 0 Info: SSLv2, Unknown (23):
0 <= Recv SSL data, 5 bytes (0x5)
0000: ....!
== 0 Info: nread=9
== 0 Info: on_frame_recv() was called with header 4
== 0 Info: nghttp2_session_mem_recv() returns 9
== 0 Info: http2_recv: 16384 bytes buffer
== 0 Info: SSLv2, Unknown (23):
0 <= Recv SSL data, 5 bytes (0x5)
0000: ....1
== 0 Info: nread=25
== 0 Info: on_begin_headers() was called
== 0 Info: got http2 header: server: lock_step_transfer
== 0 Info: on_frame_recv() was called with header 1
== 0 Info: nghttp2_session_mem_recv() returns 25
== 0 Info: HTTP 2 or upgraded connection do not support pipelining for now
0 <= Recv header, 14 bytes (0xe)
0000: HTTP/2.0 200
0 <= Recv header, 27 bytes (0x1b)
0000: server:lock_step_transfer
0 <= Recv header, 2 bytes (0x2)
0000:
== 0 Info: http2_recv: 16384 bytes buffer
== 0 Info: SSLv2, Unknown (23):
0 <= Recv SSL data, 5 bytes (0x5)
0000: ....K
== 0 Info: nread=51
== 0 Info: on_data_chunk_recv() len = 42, stream = 1
== 0 Info: 42 data written
== 0 Info: on_frame_recv() was called with header 0
== 0 Info: nghttp2_session_mem_recv() returns 51
0 <= Recv data, 42 bytes (0x2a)
0000: .long line and block with ordinal number 0

[Daniel Stenberg]

Post by Alcides Viamontes Esquivel
This is cool. I was playing with your code and noticed that so far it does
one request after another... or so it seems to me. Just in case you have any
use for it, I'm putting below curl's log.

It doesn't for me. I've done three parallel transfers and they mostly work - I
still have not really figured out how to deal with the buffers for the
different streams but I have some more ideas I'll work on next.

Post by Alcides Viamontes Esquivel
The server code that I used to play with this can be found at
https://github.com/alcidesv/lock_step_transfer ... it is a server that
refuses to finish one stream while the other has not been opened.

I run nghttpd for my tests so it may explain some differences. See the log

Post by Alcides Viamontes Esquivel
== 1 Info: Server doesn't support pipelining

It could be worth figuring out why curl says so, but it certainly explains why
it waits with the second request... as we're re-using the pipeline concept
internally even for multiplexing.

[Alcides Viamontes Esquivel]

Post by Daniel Stenberg
It could be worth figuring out why curl says so, but it certainly explains
why it waits with the second request... as we're re-using the pipeline
concept internally even for multiplexing.

Thanks for the tip, the pipelining message is no longer in curl sources: an
older version was being used at runtime in the test program, my blunder!

I can confirm that it works concurrently!

[Daniel Stenberg]

Post by Alcides Viamontes Esquivel
Thanks for the tip, the pipelining message is no longer in curl sources: an
older version was being used at runtime in the test program, my blunder!
I can confirm that it works concurrently!

Cool, yes I needed to make some adjustments to make multiplexing work slightly
better than pipelining as we don't have to "wait in line" now. I'll revisit
the "maximum" concurrency question once I've gotten the transfers to run
stable. I'm sure there are more to do about it.