CURLE_OPERATION_TIMEDOUT when FTP upload with large files

[A-ZYSTEMS]

Hello,

I am a beginner user of libcurl under windows with wxWidget.

I have writed a function for uploading files to a ftp-Server.

When I upload a file less as 40 MB, i have no problems. When I load a bigger
file (for example 56 MB) "curl_easy_perform" return a
CURLE_OPERATION_TIMEDOUT. But the uploaded file is complete save on the
ftp-Server.

I have no idea what is wrong in my code.


// Dateigröße ermitteln für Progressdialog
char* p_filename = "...\File.zip"
struct stat sb;
if( stat( p_filename, &sb ) < 0 ) {
return false;
}


//-----------------------------------------------------------
CURL *curl;
FILE * hd_src ;
CURLcode res;


// Filie size for CURLOPT_INFILESIZE_LARGE
struct stat file_info;
/* get the file size of the local file */
int hd = _open(p_filename, O_RDONLY) ;
fstat(hd, &file_info);
_close(hd) ;


/* get a FILE * of the same file, could also be made with
fdopen() from the previous descriptor, but hey this is just
an example! */
hd_src = fopen(p_filename, "rb");

/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);

/* get a curl handle */
curl = curl_easy_init();
if(curl)
{
struct curl_slist *pHeaders = NULL;

/* we want to use our own read function */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);

wxString sDestinationServer =
"ftp://test.fastwebserver.de/printer_1/incoming"
m_strOrder = wxT("12345");

wxFileName fn(m_strSendFile);
char* sMkDir = CPXTConvert::wxStringToCharPtr(wxT("MKDIR "
+ m_strOrder),ISO8859_1);
char* sAccess =
CPXTConvert::wxStringToCharPtr(wxT("pixit:pixit.921"));
char* sftp =
CPXTConvert::wxStringToCharPtr(sDestinationServer,ISO8859_1);
char* url =
CPXTConvert::wxStringToCharPtr(sDestinationServer + wxT("/") + m_strOrder +
wxT("/") + fn.GetFullName(),ISO8859_1);


pHeaders = curl_slist_append( pHeaders , sMkDir) ;


/* FTP PUT please */
curl_easy_setopt(curl, CURLOPT_URL, sftp);

// FTP Password
curl_easy_setopt(curl, CURLOPT_USERPWD, sAccess);

/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);

/* clean up the FTP commands list */
curl_slist_free_all (pHeaders);

/* FTP PUT please */
curl_easy_setopt(curl, CURLOPT_URL, url);

// FTP Password
curl_easy_setopt(curl, CURLOPT_USERPWD, sAccess);


/* enable uploading */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1) ;
curl_easy_setopt(curl, CURLOPT_FTP_CREATE_MISSING_DIRS, 1) ;

/* now specify which file to upload */
curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);

/* provide the size of the upload, we specicially typecast the value
to curl_off_t since we must be sure to use the correct data size */
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE,
(curl_off_t)file_info.st_size);

/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);


/* always cleanup */
curl_easy_cleanup(curl);

delete sMkDir;
delete url;
delete sAccess;

}
fclose(hd_src); /* close the local file */
curl_global_cleanup();
delete p_filename;







Thanks for helping


Alexander

[Daniel Stenberg]

This post might be inappropriate. Click to display it.

[A-ZYSTEMS]

Post by A-ZYSTEMS
When I upload a file less as 40 MB, i have no problems. When I load a
bigger file (for example 56 MB) "curl_easy_perform" return a
CURLE_OPERATION_TIMEDOUT. But the uploaded file is complete save on
the ftp-Server.
It sounds like the typical case of the control connection having timed-out

(through a firewall or NAT) by the time the data

Post by A-ZYSTEMS
connection is complete. The only sensible thing you can do to fix that

problem is to extend the timeouts.


When I extend the timeouts with CURLOPT_TIMEOUT
Or CURLOPT_FTP_RESPONSE_TIMEOUT to 3 Hours for example, curl_easy_perform
hang for 3 ours.

My Program is a photo book application. After creating a photo book, the
photo book should uploaded to a backend Server.
A photo book can have 1 MB to 300 MB and more. Files with to 20 minutes
uploadtime, i have no problem. But a file with a upload time over 30
minutes, "curl_easy_perform" need over one minutes to finish the upload
porccess with CURLE_OPERATION_TIMEDOUT.

Alexander

[Daniel Stenberg]

Post by Daniel Stenberg
It sounds like the typical case of the control connection having timed-out
(through a firewall or NAT) by the time the data
connection is complete.

When I extend the timeouts with CURLOPT_TIMEOUT Or
CURLOPT_FTP_RESPONSE_TIMEOUT to 3 Hours for example, curl_easy_perform hang
for 3 ours.

Uh, the timeout I was talking about is in the firewall or NAT, not libcurl.

My Program is a photo book application.

It really doesn't matter what your application is. If the reason for the
problem is the timeout I'm guessing at, then there's not a lot your app or
libcurl can do. You'd need to make some kind of work around in your app to
make sure the transfers are always shorter than the timeout period.

You can of course easily test if this is what's happening and then you'd know
for sure. If it isn't this timeout, then I'm very puzzled to what it would be
and we'd need to start digging for real to find the cure.

[Michael Wood]

Post by Daniel Stenberg

Post by Daniel Stenberg
It sounds like the typical case of the control connection having timed-out
(through a firewall or NAT) by the time the data
connection is complete.

When I extend the timeouts with CURLOPT_TIMEOUT Or
CURLOPT_FTP_RESPONSE_TIMEOUT to 3 Hours for example, curl_easy_perform hang
for 3 ours.

Uh, the timeout I was talking about is in the firewall or NAT, not libcurl.

My Program is a photo book application.

It really doesn't matter what your application is. If the reason for the
problem is the timeout I'm guessing at, then there's not a lot your app or
libcurl can do. You'd need to make some kind of work around in your app to
make sure the transfers are always shorter than the timeout period.

What about using CURLOPT_SOCKOPTFUNCTION to set SO_KEEPALIVE on the
connection? Or is this already done by default?

Alexander, searching here http://curl.haxx.se/search.html for
"keepalive" also turns up some old discussions that might be relevant.
I only looked at a couple of the messages, though.

[Daniel Stenberg]

Post by Michael Wood

then there's not a lot your app or libcurl can do. You'd need to make some
kind of work around in your app to make sure the transfers are always
shorter than the timeout period.

What about using CURLOPT_SOCKOPTFUNCTION to set SO_KEEPALIVE on the
connection? Or is this already done by default?

Ah, stupid me. That's of course one of the first things to do, indeed.

Even though the exact timeout that that'll provide will differ between
platforms and some (at least oldish) systems will have that inconveniently
long by default.

[A-ZYSTEMS]

Sorry I am a beginner with libcurl

I must only insert curl_easy_setopt(curl, CURLOPT_SOCKOPTFUNCTION,
SO_KEEPALIVE)?

My Programm are going to use on a lot of different pc with LAN, WLAN etc. I
have try 3 different FTP-Server by different Webhoster.
By all I have the same problem, when I upload a big file (more as 40 MB)
curl_easy_perform return a CURLE_OPERATION_TIMEDOUT after the upload.

Actually I ignore CURLE_OPERATION_TIMEDOUT return from "curl_easy_perform",
because my uploaded files are all ok. But that is not a nice solution,
because the user must wait one minute after the upload until
curl_easy_perform return with CURLE_OPERATION_TIMEDOUT;

Post by A-ZYSTEMS
When I upload a file less as 40 MB, i have no problems. When I load a
bigger file (for example 56 MB) "curl_easy_perform" return a
CURLE_OPERATION_TIMEDOUT. But the uploaded file is complete save on
the ftp-Server.
It sounds like the typical case of the control connection having timed-out

(through a firewall or NAT) by the time the data

Post by A-ZYSTEMS
connection is complete. The only sensible thing you can do to fix that

problem is to extend the timeouts.


When I extend the timeouts with CURLOPT_TIMEOUT
Or CURLOPT_FTP_RESPONSE_TIMEOUT to 3 Hours for example, curl_easy_perform
hang for 3 ours.

My Program is a photo book application. After creating a photo book, the
photo book should uploaded to a backend Server.
A photo book can have 1 MB to 300 MB and more. Files with to 20 minutes
uploadtime, i have no problem. But a file with a upload time over 30
minutes, "curl_easy_perform" need over one minutes to finish the upload
porccess with CURLE_OPERATION_TIMEDOUT.

Alexander

[Daniel Stenberg]

Post by A-ZYSTEMS
Sorry I am a beginner with libcurl

And you're top-posting. Please don't:
http://curl.haxx.se/mail/etiquette.html#top-post

Post by A-ZYSTEMS
I must only insert curl_easy_setopt(curl, CURLOPT_SOCKOPTFUNCTION,
SO_KEEPALIVE)?

No. Please read the docs for the CURLOPT_SOCKOPTFUNCTION option.

Post by A-ZYSTEMS
My Programm are going to use on a lot of different pc with LAN, WLAN etc. I
have try 3 different FTP-Server by different Webhoster. By all I have the
same problem, when I upload a big file (more as 40 MB) curl_easy_perform
return a CURLE_OPERATION_TIMEDOUT after the upload.

You probably go through the same NAT/firewall every time so naturally you get
the same timeout every time.

Post by A-ZYSTEMS
When I extend the timeouts with CURLOPT_TIMEOUT Or
CURLOPT_FTP_RESPONSE_TIMEOUT to 3 Hours for example, curl_easy_perform hang
for 3 ours.

I've said this several times already but I'll say it again: you won't fix this
problem by changing timeouts in libcurl. libcurl is not your problem.

[A-ZYSTEMS]

Post by Daniel Stenberg
http://curl.haxx.se/mail/etiquette.html#top-post

Sorry, that is my firt time in using a mailing list

Post by Daniel Stenberg
No. Please read the docs for the CURLOPT_SOCKOPTFUNCTION option.

I have read the tutorial and found the reason for SO_KEEPALIVE option
(denial-of-service attack)

I must create a sockopt_Callback with

curl_easy_setopt(curl, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);

And here the callback function:

size_t CThreadUpload::sockopt_callback(void *clientp, curl_socket_t curlfd,
curlsocktype purpose)
{
return 0;
}

But how can I set the SO_KEEPALIVE option in the callback.

I found no example or description in the docs and google.

Thx


Alexander

[Daniel Stenberg]

Post by A-ZYSTEMS
I have read the tutorial and found the reason for SO_KEEPALIVE option
(denial-of-service attack)

Surely SO_KEEPALIVE has very little to do with DOS attacks?

Post by A-ZYSTEMS
curl_easy_setopt(curl, CURLOPT_SOCKOPTFUNCTION, sockopt_callback);
size_t CThreadUpload::sockopt_callback(void *clientp, curl_socket_t curlfd,
curlsocktype purpose)
{
return 0;
}

You can't make the callback to be a non-static C++ class member since the
callback from libcurl is plain C.

Post by A-ZYSTEMS
But how can I set the SO_KEEPALIVE option in the callback.
I found no example or description in the docs and google.

setsockopt and SO_KEEPALIVE should be two fine keywords to search for.

And curl uses this to do exactly this, so the src/main.c file is a good place
to check out too.