curl_easy_setopt for SSL proxy

[nirdroid droid]

I have been looking at the no_proxy conversation for some time now, and I
wonder why nobody talks about ssl_proxy settings...
the only option for setting an ssl proxy is with the environment setting
it would be nice to have an curl_easy_setopt for that

[Daniel Stenberg]

Post by nirdroid droid
I have been looking at the no_proxy conversation for some time now, and I
wonder why nobody talks about ssl_proxy settings... the only option for
setting an ssl proxy is with the environment setting

Not at all. With libcurl you set the proxy to use with the URL you specify. So
if you give it a HTTPS URL, it will use the specified proxy for that. I figure
that is what a "ssl_proxy" is... Or did I misunderstand?

Post by nirdroid droid
it would be nice to have an curl_easy_setopt for that

In what way would it differ from CURLOPT_PROXY and why?

[A. Craig West]

Post by Daniel Stenberg
Not at all. With libcurl you set the proxy to use with the URL you specify.
So if you give it a HTTPS URL, it will use the specified proxy for that. I
figure that is what a "ssl_proxy" is... Or did I misunderstand?

...

Post by Daniel Stenberg
In what way would it differ from CURLOPT_PROXY and why?

I believe what he is looking for is a way to specify a different proxy
for each protocol separately, which is something I will need to
implement eventually as well. The difficult thing would be dealing
with possible future or optional protocols gracefully. I have thought
of one possible implementation, if we make CURLOPT_PROXY a comma
separated list of proxies, and if the proxy includes a protocol, use
the proxy for that protocol only. An example would be:
curl_easy_setopt(CURLOPT_PROXY,
"https://sslproxy.com:8443,ftp://ftpproxy.com,defaultproxy.com");
Which would set up 3 proxies, one for https: urls, one for ftp: urls,
and one for everything else... Authentication could complicate things,
but you could also include user:password@ in the definition to deal
with that. This has the advantage of not requiring new options for
every protocol, but seems somewhat kludgy.
-Craig

[Daniel Stenberg]

I believe what he is looking for is a way to specify a different proxy for
each protocol separately, which is something I will need to implement
eventually as well. The difficult thing would be dealing with possible
future or optional protocols gracefully. I have thought of one possible
implementation, if we make CURLOPT_PROXY a comma separated list of proxies,
and if the proxy includes a protocol, use the proxy for that protocol only.
An example would be: curl_easy_setopt(CURLOPT_PROXY,
"https://sslproxy.com:8443,ftp://ftpproxy.com,defaultproxy.com"); Which
would set up 3 proxies, one for https: urls, one for ftp: urls, and one for
everything else... Authentication could complicate things, but you could
the advantage of not requiring new options for every protocol, but seems
somewhat kludgy.

I thought about this some more and it struck me that this solution of
providing multiple proxies would still not be sufficient to cover all
imaginable scenarios.

First out, NO_PROXY could probably be different on different proxies. Then, we
have the chances of the proxies being different kinds! We want HTTP over a
HTTP proxy, but FTP we want over a socks4 while SCP always goes over
socks5-hostname... Then we even have different auth methods for the cases
where two protocols go over HTTP proxies but one of them is to use NTLM while
another is fine only with Digest...

And then we go back to the *actual* use-case I've seen in real-life many times
that certainly won't be covered either: PAC setups that send the broswer to
different proxies depending on the specific host name used in the URL...

I'm not sure if anything of this is worth bothering about, I just thought I
should mention this when we start to even consider a wider support for
"multiple" proxies.

[A. Craig West]

Post by Daniel Stenberg
I thought about this some more and it struck me that this solution of
providing multiple proxies would still not be sufficient to cover all
imaginable scenarios.
First out, NO_PROXY could probably be different on different proxies. Then,
we have the chances of the proxies being different kinds! We want HTTP over
a HTTP proxy, but FTP we want over a socks4 while SCP always goes over
socks5-hostname... Then we even have different auth methods for the cases
where two protocols go over HTTP proxies but one of them is to use NTLM
while another is fine only with Digest...
And then we go back to the *actual* use-case I've seen in real-life many
times that certainly won't be covered either: PAC setups that send the
broswer to different proxies depending on the specific host name used in the
URL...
I'm not sure if anything of this is worth bothering about, I just thought I
should mention this when we start to even consider a wider support for
"multiple" proxies.

The best way to configure something like this would probably be some
type of configuration file. It then occurred to me that there are
already implementations of PAC file parsers out there, and that would
do the trick quite nicely. I am now thinking that if I can find one
with a suitable license, it would be worth including in libcurl...
-Craig

[Daniel Stenberg]

Post by Daniel Stenberg
I'm not sure if anything of this is worth bothering about, I just thought I
should mention this when we start to even consider a wider support for
"multiple" proxies.

The best way to configure something like this would probably be some type of
configuration file. It then occurred to me that there are already
implementations of PAC file parsers out there, and that would do the trick
quite nicely. I am now thinking that if I can find one with a suitable
license, it would be worth including in libcurl...

PAC is javascript and we've been down this road before. Getting such an
interpreter into libcurl is not an option, that should be left for the app
that uses libcurl.

[A. Craig West]

Post by Daniel Stenberg
PAC is javascript and we've been down this road before. Getting such an
interpreter into libcurl is not an option, that should be left for the app
that uses libcurl.

I was thinking something along the lines of adding in optional support
for the pacparser library ( http://code.google.com/p/pacparser/ ), as
I believe it does just about everything we need. That way, we don't
need to deal with the javascript parsing ourselves. If the person
building the library has pacparser installed and enabled, we just use
it, like how we use other third party libraries like the openssl
stuff. We could certainly leave it out, but it would greatly simplify
some of the proxy configuration issues we are dealing with now.
Redirects are one example where having the user do the pac parsing
separately would be cumbersome, as they would be forced to manually
deal with the redirect if the redirected site needed a different proxy
than the original site.
Even given something like pacparser, we will probably need some way to
have authentication information stored on a per-proxy basis. Do we
currently have a way of storing authentication per-site, in some type
of external file? It seems like a useful thing to do...
-Craig

[Daniel Stenberg]

Post by Daniel Stenberg
PAC is javascript and we've been down this road before. Getting such an
interpreter into libcurl is not an option, that should be left for the app
that uses libcurl.

I was thinking something along the lines of adding in optional support for
the pacparser library ( http://code.google.com/p/pacparser/ ), as I believe
it does just about everything we need. That way, we don't need to deal with
the javascript parsing ourselves. If the person building the library has
pacparser installed and enabled, we just use it, like how we use other third
party libraries like the openssl stuff.

I'm far from convinced about the goodness of something like that. Sure, adding
a dependency on another lib can be good but no matter what it adds to the code
and maintainability and people will expect it to remain working etc.

I can't but to remain in the position that figuring out the proxy is a job for
the outside. libcurl uses what the outside tells it.

We could certainly leave it out, but it would greatly simplify some of the
proxy configuration issues we are dealing with now.

Would it really? I can't see how it simplifies anything. It'd add a lot of
code for the glue layer.

Redirects are one example where having the user do the pac parsing
separately would be cumbersome, as they would be forced to manually deal
with the redirect if the redirected site needed a different proxy than the
original site.

Yes, you'd do the redirect logic yourself but with modern libcurl that is
about three to five lines of additional code to get the new url and set it,
and then you can do whatever magic you think is necessary before the next
request.

I think as a general rule that whenever it starts to feel like something is
getting clumsy or complicated when we think of things to do between requests
when libcurl follows locations, then we should consider just leaving it for
the app to deal with completely.

Even given something like pacparser, we will probably need some way to have
authentication information stored on a per-proxy basis. Do we currently have
a way of storing authentication per-site, in some type of external file? It
seems like a useful thing to do...

Nope, neither concept exist today the way I think you mean.

[nirdroid droid]

my app uses proxy - both http and https
the proxy has http port and https port
so the code, as you suggest should be:

1. curl_easy_setopt(curl, CURLOPT_PROXY, "1.2.3.4");
2. curl_easy_setopt(curl, CURLOPT_PROXYPORT, 80);
3. curl_easy_setopt(curl, CURLOPT_PROXYPORT, 8443);

doesn't #3 runs over #2?

[Daniel Stenberg]

my app uses proxy - both http and https the proxy has http port and https
1. curl_easy_setopt(curl, CURLOPT_PROXY, "1.2.3.4");
2. curl_easy_setopt(curl, CURLOPT_PROXYPORT, 80);
3. curl_easy_setopt(curl, CURLOPT_PROXYPORT, 8443);
doesn't #3 runs over #2?

No, I don't suggest that and yes #3 will run over #2.

[nir droid]

Post by nirdroid droid
my app uses proxy - both http and https
the proxy has http port and https port
1. curl_easy_setopt(curl, CURLOPT_PROXY, "1.2.3.4");
2. curl_easy_setopt(curl, CURLOPT_PROXYPORT, 80);
3. curl_easy_setopt(curl, CURLOPT_PROXYPORT, 8443);
doesn't #3 runs over #2?

or if you prefer:
curl_easy_setopt(CURLOPT_PROXY, "http://1.2.3.4:80")<https://sslproxy.com:8443/>
curl_easy_setopt(CURLOPT_PROXY,
"https://1.2.3.4:8443")<https://sslproxy.com:8443/>

doesn't #2 runs over #1?

there are many http urls that redirect to htts page (usually login page), so
if you use proxy, you need both http and https proxy

[Daniel Stenberg]

Post by nir droid
curl_easy_setopt(CURLOPT_PROXY, "http://1.2.3.4:80")<https://sslproxy.com:8443/>
curl_easy_setopt(CURLOPT_PROXY,
"https://1.2.3.4:8443")<https://sslproxy.com:8443/>
doesn't #2 runs over #1?
there are many http urls that redirect to htts page (usually login page), so
if you use proxy, you need both http and https proxy

No I don't "prefer" that way either since it doesn't work.

You clearly didn't read my explanation of how this works, or Craig West's
explanation of what he'd like to do and I would agree that it seems like a
good thing to provide.

In current and past libcurl versions however you specify the proxy to use and
then the URL that will go over that specified proxy - basically whatever the
protocol is. (There's an additional complexity added by the fact that libcurl
also supports the "standard" environment variables but I'll ignore that for
now).

If you need to do more advanced stuff for redirects that what libcurl
supports, the fix is generally to not use libcurl's automatic following but
instead do it yourself. By using CURLINFO_REDIRECT_URL, that's not even hard
or a lot of work.

[nir droid]

Post by nir droid
curl_easy_setopt(CURLOPT_PROXY, "http://1.2.3.4:80")<

Post by nir droid
https://sslproxy.com:8443/>
curl_easy_setopt(CURLOPT_PROXY,
"https://1.2.3.4:8443")<https://sslproxy.com:8443/>
doesn't #2 runs over #1?
there are many http urls that redirect to htts page (usually login page),
so if you use proxy, you need both http and https proxy

No I don't "prefer" that way either since it doesn't work.
You clearly didn't read my explanation of how this works, or Craig West's
explanation of what he'd like to do and I would agree that it seems like a
good thing to provide.

Sorry for the bad impression; I did read but did not fully understand

Post by nir droid
In current and past libcurl versions however you specify the proxy to use
and then the URL that will go over that specified proxy - basically whatever
the protocol is. (There's an additional complexity added by the fact that
libcurl also supports the "standard" environment variables but I'll ignore
that for now).
If you need to do more advanced stuff for redirects that what libcurl
supports, the fix is generally to not use libcurl's automatic following but
instead do it yourself. By using CURLINFO_REDIRECT_URL, that's not even hard
or a lot of work.
--
/ daniel.haxx.se

Now I understand, thanks.
Just to clear things, I use the environment variables to set the proxies and
it works fine enough; As a newbie I didn't find any example that refers to
this (quite common) scenario

[Daniel Stenberg]

Post by nir droid
Just to clear things, I use the environment variables to set the proxies and
it works fine enough; As a newbie I didn't find any example that refers to
this (quite common) scenario

Feel free to provide an example and I'll happily include it!