HTTPS request with curl library gives "out of memory" error

[Ambarish Mitra]

Hi all,

When sending a HTTPS traffic with CURL library (<http://curl.haxx.se/>) in a
C program, the curl API function "curl_easy_perform" throws an error code
CURLE_OUT_OF_MEMORY, which the 'curl_easy_strerror' translates to "out of
memory".

It is working in non-SSL mode, making me believe that there is some issue
with the SSL setup/config.

Moreover, it is working when we use the "curl" command in the command line:
[***@ps2844 authzplugin]$ curl
https://secure-server:2443/home.html --cacert <file> // This works, the html
output is not shown.

I am on curl 7.12.1, and SSL support is enabled. Also, openssl version is:
OpenSSL 0.9.7a Feb 19 2003.

Any help/pointers will be very helpful.

Thanks, Ambarish Mitra.


Code snippet:

CURL* hCurl;
CURLcode res;
curl_slist* requestHeaders = NULL;

hCurl = curl_easy_init(); //Null checking done, not shown

// plethora of easy_setopt done, not shown here. Only the SSL options are
shown.
// if you think sending out all the options will be helpful, let me know,
and I will do that.

curl_easy_setopt(hCurl, CURLOPT_SSLVERSION, 3);

curl_easy_setopt(hCurl,CURLOPT_SSLCERTTYPE,"PEM");
curl_easy_setopt(hCurl,CURLOPT_SSLCERT,"/home/ambarish/project/client.crt");

curl_easy_setopt(hCurl,CURLOPT_SSLKEY,"/home/ambarish/project/client.key");
curl_easy_setopt(hCurl,CURLOPT_SSLKEYTYPE, "PEM");
curl_easy_setopt(hCurl,CURLOPT_SSLKEYPASSWD,"password");


curl_easy_setopt(hCurl, CURLOPT_SSL_VERIFYPEER, 0); // Verify Peer = 0, so
CA cert is not needed.



// CURL options to set/unset the request headers
requestHeaders = curl_slist_append(requestHeaders , "Pragma:" ) ;
//remove header
requestHeaders = curl_slist_append(requestHeaders , "Accept:" ) ;
//remove header
requestHeaders = curl_slist_append(requestHeaders ,
"Content-Type:text/xml");

curl_easy_setopt( hCurl , CURLOPT_HTTPHEADER , requestHeaders);
res = curl_easy_setopt(hCurl, CURLOPT_URL,
https://secure-server:2443/program.cgi); // the endpoint

res = curl_easy_perform(hCurl);
if (res == CURLE_OK)
{
// OK. Set the data. Not shown here.
}
else
{
fprintf(stderr,"CURL perform error:%s",curl_easy_strerror(res)); // This
gives "out of memory"
curl_slist_free_all(requestHeaders);
curl_easy_cleanup(hCurl); // cleanup

}

curl_slist_free_all(requestHeaders);
curl_easy_cleanup(hCurl); // cleanup



DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

[Daniel Stenberg]

Post by Ambarish Mitra
OpenSSL 0.9.7a Feb 19 2003.

Both these versions are very old and contain numerous bugs and security flaws.
You should start by trying more recent versions of both libcurl and openssl.

If that still cause problems, there's no a lot much else to do than to fire up
your debugger and follow the code flow to find the function that fails. Or
possibly run with strace first.

[Ambarish Mitra]

Post by Ambarish Mitra
OpenSSL 0.9.7a Feb 19 2003.

Both these versions are very old and contain numerous bugs and security
flaws.
You should start by trying more recent versions of both libcurl and openssl.


--Daniel, Thanks for your response. I have now upgraded the versions of
both. I am now on curl 7.16.0 and OpenSSL 0.9.8g 19 Oct 2007

[***@ps0887 authzplugin]$ curl --version
curl 7.16.0 (i686-pc-linux-gnu) libcurl/7.16.0 OpenSSL/0.9.7a zlib/1.2.1.2
libidn/0.5.6
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: IDN IPv6 Largefile NTLM SSL libz


However, the problem still persists. The behaviour is exactly the same. In
SSL mode, the curl_easy_perform gives "out of memory" error, whereas in
non-SSL mode, it is working fine.

However, the good part is: in the trace mode, this version of curl gives a
helpful message, which was not coming in the older version. In the verbose
mode, the following is printed:

== Info: SSL: couldn't create a context!

I think this is the root cause, but no idea why a SSL context could not be
created. Any help/pointers would be helpful.

Thanks, Ambarish.


DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

[Michal Marek]

Post by Ambarish Mitra
--Daniel, Thanks for your response. I have now upgraded the versions of
both. I am now on curl 7.16.0 and OpenSSL 0.9.8g 19 Oct 2007

That was close, the latest released curl version is 7.17.1 :)

Anyway...

Post by Ambarish Mitra
However, the good part is: in the trace mode, this version of curl gives a
helpful message, which was not coming in the older version. In the verbose
== Info: SSL: couldn't create a context!
I think this is the root cause, but no idea why a SSL context could not be
created. Any help/pointers would be helpful.

$ grep -n -C3 "couldn't create a context\!" lib/*.c
lib/ssluse.c-1295- connssl->ctx = SSL_CTX_new(req_method);
lib/ssluse.c-1296-
lib/ssluse.c-1297- if(!connssl->ctx) {
lib/ssluse.c:1298: failf(data, "SSL: couldn't create a context!");
lib/ssluse.c-1299- return CURLE_OUT_OF_MEMORY;
lib/ssluse.c-1300- }
lib/ssluse.c-1301-

IOW openSSL's SSL_CTX_new() failed, libcurl interprets this as an out of
memory condition, which may or may not be true (I've never programmed
with openSSL, so I don't know and can't provide a patch right now).

But the problem you're seeing is caused by openSSL. Download the latest
libcurl source, improve the error message in libcurl's ssluse.c to see
what's openSSL really complaining about, fix the issue with openSSL and
post the ssluse.c patch here if you find out that SSL_CTX_new() can
really fail due to other than memory reasons ;)

hth,
Michal

[Ambarish Mitra]

That was close, the latest released curl version is 7.17.1 :)

--- I already have a 7.16.0 version in the same project application, and I
cannot convince them to upgrade now- we cannot have 2 versions of the same
library in the same application. :-)


IOW openSSL's SSL_CTX_new() failed, libcurl interprets this as an out of
memory condition, which may or may not be true (I've never programmed
with openSSL, so I don't know and can't provide a patch right now).

--- I have figured out the cause. The application was linking with some
other library, which again linked with the openssl library. Therefore, the
SSL_CTX_new symbol resolved to the other call. A couple of things were done
to diagnose this error. 1) Right after the SSL call failure, I added the
following code:

unsigned long ErrCode = ERR_get_error();
char ErrBuf[120];
ERR_error_string(ErrCode, ErrBuf);
fprintf(stderr, "[%s]\n", ErrBuf);

and it gave me the output: [error:1A09400E:SSLCERT routines:STORE_NEW:not
supported]

This gave a hint that the symbol searching was not proper. 2) Then running a
strace with LD_DEBUG=bindings showed how the call resolved to the other
library.

But the problem you're seeing is caused by openSSL. Download the latest
libcurl source, improve the error message in libcurl's ssluse.c to see
what's openSSL really complaining about, fix the issue with openSSL and
post the ssluse.c patch here if you find out that SSL_CTX_new() can
really fail due to other than memory reasons ;)


--- So, there is nothing to fix because nothing is broken.

Thanks for all those who have taken their time on this.


// Pls ignore the disclaimer.


DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.