Discussion:
How to POST binary data with libcurl?
Tillmann Steinbrecher
2006-07-26 18:03:07 UTC
Permalink
Hi,

I have a very simple, basic problem; however despite reading of
documentation, mailing list archives, etc, I just can't find a solution.

What I'd like to do is simply POST binary data using libcurl, as a
normal multipart/formdata HTTP POST.

The problem that I have is that the data posted is truncated as soon as
the first zero byte occurs - as if strlen was being used; but DESPITE
the fact that I used CURLFORM_CONTENTSLENGTH in curl_formadd(). In fact,
the behaviour is very strange: It seems that libcurl either uses the
value provided after CURLFORM_CONTENTSLENGTH, OR the string length,
whichever is lower.

Example:

curl_formadd(&post, &last,
CURLFORM_COPYNAME, "mydata",
CURLFORM_PTRCONTENTS, "test\0test",
CURLFORM_CONTENTSLENGTH, 9,
CURLFORM_END);

-> 4 bytes are posted, but not 9

curl_formadd(&post, &last,
CURLFORM_COPYNAME, "mydata",
CURLFORM_PTRCONTENTS, "test\0test",
CURLFORM_CONTENTSLENGTH, 2,
CURLFORM_END);

-> 2 bytes are posted

Using CURLOPT_POSTFIELDSIZE doesn't seem to help either; as apparently
that option doesn't work with curl_formadd for creating rfc1867-style
(multipart) POSTs.

Any help would be greatly appreciated.

bye,
Till
--
Dipl.-Ing. Tillmann Steinbrecher http://www.igd.fhg.de/~tsteinbr/
Cognitive Computing & Medical Imaging Phone: +49 6151 155-485
Fraunhofer IGD, Fraunhoferstr. 5, D-64283 Darmstadt, Germany
Daniel Stenberg
2006-07-26 21:27:17 UTC
Permalink
Post by Tillmann Steinbrecher
curl_formadd(&post, &last,
CURLFORM_COPYNAME, "mydata",
CURLFORM_PTRCONTENTS, "test\0test",
CURLFORM_CONTENTSLENGTH, 9,
CURLFORM_END);
-> 4 bytes are posted, but not 9
I just built a test application using exactly this snippet. Worked exactly
like documented, and sent the 9 bytes fine:

=> Send data, 150 bytes (0x96)
0000: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
0010: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 64 35 --------------d5
0020: 66 34 30 63 31 30 63 39 32 66 0d 0a 43 6f 6e 74 f40c10c92f..Cont
0030: 65 6e 74 2d 44 69 73 70 6f 73 69 74 69 6f 6e 3a ent-Disposition:
0040: 20 66 6f 72 6d 2d 64 61 74 61 3b 20 6e 61 6d 65 form-data; name
0050: 3d 22 6d 79 64 61 74 61 22 0d 0a 0d 0a 74 65 73 ="mydata"....tes
0060: 74 00 74 65 73 74 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d t.test..--------
0070: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
0080: 2d 2d 2d 2d 2d 2d 64 35 66 34 30 63 31 30 63 39 ------d5f40c10c9
0090: 32 66 2d 2d 0d 0a 2f--..
--
Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
Tillmann Steinbrecher
2006-07-27 12:01:30 UTC
Permalink
Hi,
Post by Daniel Stenberg
Post by Tillmann Steinbrecher
curl_formadd(&post, &last,
CURLFORM_COPYNAME, "mydata",
CURLFORM_PTRCONTENTS, "test\0test",
CURLFORM_CONTENTSLENGTH, 9,
CURLFORM_END);
-> 4 bytes are posted, but not 9
I just built a test application using exactly this snippet. Worked exactly
=> Send data, 150 bytes (0x96)
0000: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d ----------------
Thanks for your time. It seems like I'm hitting a bug in php on the
server side then, and there's nothing wrong with curl. What did you use
to created this dump? It would help me in debugging.

bye,
Till
--
Dipl.-Ing. Tillmann Steinbrecher http://www.igd.fhg.de/~tsteinbr/
Cognitive Computing & Medical Imaging Phone: +49 6151 155-485
Fraunhofer IGD, Fraunhoferstr. 5, D-64283 Darmstadt, Germany
Daniel Stenberg
2006-07-27 21:22:01 UTC
Permalink
What did you use to created this dump? It would help me in debugging.
CURLOPT_DEBUGFUNCTION, and a callback pretty much exactly like the one in the
debug.c example:

http://curl.haxx.se/lxr/source/docs/examples/debug.c
--
Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
Tillmann Steinbrecher
2006-07-27 13:32:13 UTC
Permalink
Hi,
Post by Tillmann Steinbrecher
It seems like I'm hitting a bug in php on the
server side then, and there's nothing wrong with curl.
I can now confirm that; I was running into a bug in PHP which was already reported 5 years ago (http://bugs.php.net/bug.php?id=14321) - Daniel, you were also apparently involved back then :); but the bug STILL isn't fixed in the very latest PHP release.

Maybe this post will prevent others from wasting time this, like I did.

bye,
Tillmann
Daniel Stenberg
2006-07-27 17:54:09 UTC
Permalink
Post by Tillmann Steinbrecher
I can now confirm that; I was running into a bug in PHP which was already
reported 5 years ago (http://bugs.php.net/bug.php?id=14321) - Daniel, you
were also apparently involved back then :); but the bug STILL isn't fixed in
the very latest PHP release.
I've said it before and I can say it again: the PHP multipart formpost
receiver is stupid. It was back then and it obviously still is. The current
libcurl code in fact still inhibits the sending of a
Content-Transfer-Encoding: header just because of its stupidness.
--
Commercial curl and libcurl Technical Support: http://haxx.se/curl.html
Tillmann Steinbrecher
2006-07-27 18:11:57 UTC
Permalink
Hi,
Post by Daniel Stenberg
Post by Tillmann Steinbrecher
I can now confirm that; I was running into a bug in PHP which was
already
reported 5 years ago (http://bugs.php.net/bug.php?id=14321) - Daniel,
you
were also apparently involved back then :); but the bug STILL isn't
fixed in
the very latest PHP release.
I've said it before and I can say it again: the PHP multipart formpost
receiver is stupid. It was back then and it obviously still is. The
current
libcurl code in fact still inhibits the sending of a
Content-Transfer-Encoding: header just because of its stupidness.
But there's hope. I filed a new PHP bug report today, and the bug is fixed in CVS. I guess we'll have to wait for PHP6 until a fixed released version is available, but that's better than nothing.

As a workaround, sending as file (CURLFORM_BUFFER/CURLFORM_BUFFERPTR CURLFORM_BUFFERLENGTH) works; though it causes the data to be unnecessarily stored in a temporary file on the server.

Thanks again for your help!

bye,
Till
Michael Wallner
2006-07-27 19:53:58 UTC
Permalink
Post by Tillmann Steinbrecher
But there's hope. I filed a new PHP bug report today, and the bug is
fixed in CVS. I guess we'll have to wait for PHP6 until a fixed
released version is available, but that's better than nothing.
Nope, the fix will be in PHP-5.2, thanks for your bug report.

Regards,
--
Michael
Loading...