HTTPS CURL get slow when calling at first time

[sagar hagawne]

Hi,

I am using a CURL version 7.27.0 and passing the url using HTTPS
but the first call to the curl_easy_perform(curl) is very very slow, but
all subsequent calls are fast
the problem seems to be anything to be initialization of CURL handle or
seems need to connect to socket and wait to open an interface.

I have already posted some threads on curl_users list of that
issue, but not find a perfect solution and i am facing a same problem
please go through the following threads,

http://curl.haxx.se/mail/archive-2012-11/0025.html
http://curl.haxx.se/mail/archive-2012-11/0026.html
http://curl.haxx.se/mail/archive-2012-11/0027.html
http://curl.haxx.se/mail/archive-2012-11/0029.html
http://curl.haxx.se/mail/archive-2012-11/0030.html
http://curl.haxx.se/mail/archive-2012-11/0031.html
http://curl.haxx.se/mail/archive-2012-11/0032.html
http://curl.haxx.se/mail/archive-2012-11/0033.html


Any Solution?

Thanks,
Sagar

[Lijo Antony]

Post by sagar hagawne
Hi,
I am using a CURL version 7.27.0 and passing the url using HTTPS
but the first call to the curl_easy_perform(curl) is very very slow, but
all subsequent calls are fast
the problem seems to be anything to be initialization of CURL handle or
seems need to connect to socket and wait to open an interface.

For the first easy_perform, libcurl creates a new connection which
involves many steps including SSL handshake. But for the subsequent
easy_performs, libcurl reuses cached information and even the connection
itself. That is why they seem faster. I don't think this is a bug.

-lijo

Post by sagar hagawne
I have already posted some threads on curl_users list of that
issue, but not find a perfect solution and i am facing a same problem
please go through the following threads,
http://curl.haxx.se/mail/archive-2012-11/0025.html
http://curl.haxx.se/mail/archive-2012-11/0026.html
http://curl.haxx.se/mail/archive-2012-11/0027.html
http://curl.haxx.se/mail/archive-2012-11/0029.html
http://curl.haxx.se/mail/archive-2012-11/0030.html
http://curl.haxx.se/mail/archive-2012-11/0031.html
http://curl.haxx.se/mail/archive-2012-11/0032.html
http://curl.haxx.se/mail/archive-2012-11/0033.html
Any Solution?
Thanks,
Sagar
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

[Daniel Stenberg]

I am using a CURL version 7.27.0 and passing the url using HTTPS but
the first call to the curl_easy_perform(curl) is very very slow, but all
subsequent calls are fast the problem seems to be anything to be
initialization of CURL handle or seems need to connect to socket and wait to
open an interface.

If I were to guess, I'd say that the problem is most likely a weird and slow
network setup, firewall or NAT in your network environment.

If you want more than blind guesses, you need to provide more data and logs.

[Gokhan Sengun]

Post by sagar hagawne
I am using a CURL version 7.27.0 and passing the url using HTTPS

Post by sagar hagawne
but the first call to the curl_easy_perform(curl) is very very slow, but
all subsequent calls are fast the problem seems to be anything to be
initialization of CURL handle or seems need to connect to socket and wait
to open an interface.

If I were to guess, I'd say that the problem is most likely a weird and
slow network setup, firewall or NAT in your network environment.
If you want more than blind guesses, you need to provide more data and logs.

+1.

This interesting issue deserves better data for libcurl community to
diagnose. First of all, what is "very very slow"? Is it 1 msec or 1 sec? Do
you have any wireshark capture (possibly with annotations)? Do you have
timestamped strace output? Etc. Please elaborate.

Before delving into the wild details, as per Daniel's guess, you may want
to check misbehaving network element or remote party. If you have a chance
to carry a controlled experiment on the outside world, please consider it
too.

[sagar hagawne]

Post by sagar hagawne

Post by sagar hagawne
I am using a CURL version 7.27.0 and passing the url using HTTPS

but

Post by sagar hagawne
the first call to the curl_easy_perform(curl) is very very slow, but all
subsequent calls are fast the problem seems to be anything to be
initialization of CURL handle or seems need to connect to socket and

wait to

Post by sagar hagawne
open an interface.

here is some logs Attached in mail.

I cleaned them up for you.

Post by sagar hagawne
our_https_log_1.txt: the first communication to our https server
our_https_log_2.txt: the second communication to our https server
gmail_https_log_1.txt: the first communication to gmail server
gmail_https_log_2.txt: the second communication to gmail server
In our first file, there is a gap of time from 15:29:00 and 15:29:23 (23
seconds).
In gmail first file, there is a gap of time from 16:07:13 and 16:07:37
(24 seconds).
***By the way, 74.125.225.118 is an IP for gmail. I used it versus a
hostname because the IP kept changing for gmail.
But in the second file, at the exact same spot in the logs, it's less
than a second difference.

Thanks,
Sagar

[Gokhan Sengun]

Post by sagar hagawne
In our first file, there is a gap of time from 15:29:00 and 15:29:23 (23

Post by sagar hagawne
seconds).
In gmail first file, there is a gap of time from 16:07:13 and 16:07:37
(24 seconds).

Thanks, I can now clearly see the problem. Your program (maybe libcurl) is
not initiating the SSL handshake for a while.

1. Have you tried our good old curl binary from command line to possibly
repeat this problem? This might isolate a network element problem.
2. Is there a small piece of code that we can repeat this problem?

Cheers.

[sagar hagawne]

Thanks Gokhan Sengun,

my code snippet is as follows please try to find if anyone knows better.

curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();

struct data config;
config.trace_ascii = 1; /* enable ascii tracing */
struct curl_slist *chunk = NULL;
curl_slist_append( chunk, "Content-Type: application/json");

curl = curl_easy_init();

struct AppMemoryStruct chunk1;
chunk1.memory = (char *)malloc(1); /* will be grown as needed by
the realloc above */
chunk1.size = 0; /* no data at this point */

if(curl) {

curl_easy_setopt(curl, CURLOPT_URL,
url.GetPlatformString().c_str());
curl_easy_setopt(curl,CURLOPT_SSLCERTTYPE,"PEM");
curl_easy_setopt(curl, CURLOPT_SSLCERT,
"C:\\test\\omg.aps.net.pem");

curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2);
curl_easy_setopt(curl,CURLOPT_KEYPASSWD,"");

char error[1024];
curl_easy_setopt ( curl, CURLOPT_ERRORBUFFER, error );

/* send all data to this function */
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
&AppWriteMemoryCallback);

/* we pass our 'chunk' struct to the callback function */
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void
*)&chunk1);

//heres the performance get down its around 40 seconds to call the
server

res = curl_easy_perform(curl);
//till here

if(CURLE_OK != res)
{

errorCode = kFalse;
if(chunk1.memory)
{

free(chunk1.memory);
}
curl_easy_cleanup(curl);
return errorCode;
}
errorCode = kTrue;

if(chunk1.memory)
{

returndata.SetCString(chunk1.memory);
free(chunk1.memory);

}

/* always cleanup */
curl_easy_cleanup(curl);curl_global_init(CURL_GLOBAL_ALL);

}
please try to find a key issue why the performance is very bad using https
curl.


Thanks,
Sagar

[Gokhan Sengun]

Do you use HTML for mail? If so please switch to text. I actually asked for
a compilable whole program not a snippet. Comments inside.

First, what is this code inside? A main function or a while body? Please
show us the code as a whole.

my code snippet is as follows please try to find if anyone knows better.

Post by sagar hagawne
curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();
struct data config;
config.trace_ascii = 1; /* enable ascii tracing */
struct curl_slist *chunk = NULL;
curl_slist_append( chunk, "Content-Type: application/json");
curl = curl_easy_init();

Any reason to initialize easy handle second time here? It should not affect
anything but just wondered.

Post by sagar hagawne
struct AppMemoryStruct chunk1;
chunk1.memory = (char *)malloc(1); /* will be grown as needed by
the realloc above */
chunk1.size = 0; /* no data at this point */

Realloc above is -- I guess in the write callback -- suspicious. Again I
would like to see the whole code :-)

Post by sagar hagawne
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL,
url.GetPlatformString().c_str());
curl_easy_setopt(curl,CURLOPT_SSLCERTTYPE,"PEM");
curl_easy_setopt(curl, CURLOPT_SSLCERT,
"C:\\test\\omg.aps.net.pem");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2);
curl_easy_setopt(curl,CURLOPT_KEYPASSWD,"");
char error[1024];
curl_easy_setopt ( curl, CURLOPT_ERRORBUFFER, error );

the error buffer should be off size CURL_ERROR_SIZE. I suggest you define
it in outer blocks to avoid possible stack corruption.

Post by sagar hagawne
/* send all data to this function */
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
&AppWriteMemoryCallback);
/* we pass our 'chunk' struct to the callback function */
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void
*)&chunk1);

Again seeing the whole code and repeating problem, we will better help you
here.

[Oscar Koeroo]

Post by sagar hagawne
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL,
url.GetPlatformString().c_str());
curl_easy_setopt(curl,CURLOPT_SSLCERTTYPE,"PEM");
curl_easy_setopt(curl, CURLOPT_SSLCERT,
"C:\\test\\omg.aps.net.pem");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2);
curl_easy_setopt(curl,CURLOPT_KEYPASSWD,"");

Hello Sagar,

Have you Wiresharked an empty SSL handshake? It is interesting to see how
fast your TCP/IP is handhaking and the latency from SSL/TLS Client Hello and
Server Hello to a complete session. IMHO guessing is inefficient, measuring
is the start of science.

Do you need to push a client certificate? Mutual authentication is known to
be a pretty slow process. Both client and server need to verify the
certificate stack and signatures.

Also, the CURLOPT_SSL_VERIFYPEER is set to false, so this should increase
the potential handshake as your libcurl-based client is skipping the peer
verification steps (read: it disables security on your client end).


Oscar


House-hold request:
Could you reply on the previous email to not break the mail thread with each
reply? It's hard to follow the conversation as a conversation without it.
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

[Dan Fandrich]

Post by Oscar Koeroo
Have you Wiresharked an empty SSL handshake? It is interesting to see how
fast your TCP/IP is handhaking and the latency from SSL/TLS Client Hello and
Server Hello to a complete session. IMHO guessing is inefficient, measuring
is the start of science.

Another big potential source of latency on startup is the DNS lookup. A
few bad DNS servers in the lookup path, or a misconfigured name service
switch config could potentially add up to 40 seconds of delay.
Subsequent connects wouldn't see that delay because the results would be
cached for a while.

Post by Oscar Koeroo

Dan

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

[g***@gmail.com]

Post by Dan Fandrich
Another big potential source of latency on startup is the DNS lookup. A
few bad DNS servers in the lookup path, or a misconfigured name service
switch config could potentially add up to 40 seconds of delay.
Subsequent connects wouldn't see that delay because the results would be
cached for a while.

According to the logs, client resolves the ip and connects to 443 port w/o delay. it is the start of ssl handshake failing to kick of for a long time.


-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

[plot.lost]

Post by g***@gmail.com

Post by Dan Fandrich
Another big potential source of latency on startup is the DNS lookup. A
few bad DNS servers in the lookup path, or a misconfigured name service
switch config could potentially add up to 40 seconds of delay.
Subsequent connects wouldn't see that delay because the results would be
cached for a while.

According to the logs, client resolves the ip and connects to 443 port w/o delay. it is the start of ssl handshake failing to kick of for a long time.

The DNS delay could be on the server side - is the server maybe doing
some reverse lookups or similar? Do you have access to the server logs
and/or configuration?

What happens if you repeat the test immediately - i.e. start the test
process where the first connection has the delay and let that move onto
the next connections which work ok, stop the process and then start it
again right away. Does the first connection still have the delay?


-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

[Dan Fandrich]

Post by g***@gmail.com
According to the logs, client resolves the ip and connects to 443 port w/o delay. it is the start of ssl handshake failing to kick of for a long time.

Is this a Windows platform? What SSL back-end is libcurl compiled to
use? Have you tried another?

Post by g***@gmail.com

Dan

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html

[sagar hagawne]

Thanks Gokhan Sengun,

Currently we are using InDesign Software and we have a InDesign specific
code. But I think U Wont have that.
So I am creating a simple c++ empty project so you can compile that code on
your side.
It will take some time.
As soon as it done I will post it.

Thank you,
Sagar.

[sagar hagawne]

Hi,

Actually after done a simple empty project with c++ using Curl, it seems
instantaneous to me
without delay at all calls,

( note :: the issue of delay at first time using cURL
I am using The InDesign framework, and i need to get it done using cURL,
but on InDesign it seems wired at first call and takes so much time it is
only for https cURL
)


I'm happy to have a part of this forum. In the short time you have been
replied me,
I really appreciate your willingness to help out wherever needed.
It's that kind of flexibility that will help a lot,

Thank you for all your help.
Sagar.