Henrik Holst via curl-library
2021-04-01 23:32:47 UTC
Dear libcurl devs,
in the docs the curl_multi_timeout() function has this note:
"An application that uses the multi_socket API SHOULD NOT use this
function, but SHOULD instead use curl_multi_setopt and its
CURLMOPT_TIMERFUNCTION option for proper and desired behavior."
However as I see there are instances where curl_multi_timeout() have an
role to play even in the muli_socket API (unless of course there are some
internal class if one uses this function, I have not gone through the
sources that deep) when the event handler is used for not only libcurl
events.
If the event handler is only handling libcurl events then all is just fine
with the timer callback, but if other sockets or file descriptors are
handled by the same event handler and one ends up in a situation where
those other sockets/descriptors are very busy then it's quite possible that
the event handler will not return back with a timeout for quite a while,
and since libcurl can only update it's internal state and call the timer
callback if one calls curl_multi_socket_action() then one either have to
call curl_multi_socket_action() after each event handler return (and I
assume here without checking that curl_multi_socket_action() is a much
heavier function to call than curl_multi_timeout(), but I could of course
be wrong), or one have to track the time taken during the event handler
call (which is hard on e.g Linux since select/poll/epoll doesn't indicate
how long it really waited).
What I mean is that the naive implementation of the multi socket API that
works 100% of the time if curl is the only one generating events and 99% of
the time if you have other external events but not much traffic on them:
for (;;)
int ret = poll (fds, nfds, timeout);
if (ret == 0) {/* timeout */
curl_multi_socket_action (curlm, CURL_SOCKET_TIMEOUT, 0,
&running_handles);
} else if (ret != -1) { /* events */
if (fds[0].revents != 0)
curl_multi_socket_action (curlm, fds[0].fd, fds[0].revents,
&running_handles);
else if (fds[1]).revents != 0)
...
}
}
Yes I know that we cannot put fds[0].revents directly into curl but this
was just a small example of the typical event structure from someone using
poll().
Now if curl is the only one generating events here the timer function
callback will be called by curl and the "timeout" variable will be set
appropriate. However if we add some other socket here on say fds[1] above
that is extremely high in volume then it's quite possible that poll() will
very seldom return 0 even when the timeout from curl is very low leading to
a situation where curl will "hang", typically I have seen this directly
after a call to curl_multi_add_handle() where curl tends to first call the
timer function callback with a timeout of 0 and then after the first call
to curl_multi_socket_action() the timer function callback is called again
with a timeout of 1 but that 1 can be enough for a very highly stressed
other socket prohibiting poll from timing out even for 1ms.
Therefore I have noticed that the following logic and where we don't set
any timer function callback at all, works much better:
for (;;) {
long timeout;
curl_multi_timeout (curlm, &timeout);
if (timeout == 0)
curl_multi_socket_action (curlm, CURL_SOCKET_TIMEOUT, 0,
&running_handles);
int ret = poll (fds, nfds, timeout);
if (ret > 0) { /* events */
if (fds[0].revents != 0)
curl_multi_socket_action (curlm, fds[0].fd, fds[0].revents,
&running_handles);
else if (fds[1]).revents != 0)
...
}
}
So unless there is some potential lethal overhead in curl_multi_timeout() I
actually find this to work much better than the timer function callback.
Regards, Henrik Holst
in the docs the curl_multi_timeout() function has this note:
"An application that uses the multi_socket API SHOULD NOT use this
function, but SHOULD instead use curl_multi_setopt and its
CURLMOPT_TIMERFUNCTION option for proper and desired behavior."
However as I see there are instances where curl_multi_timeout() have an
role to play even in the muli_socket API (unless of course there are some
internal class if one uses this function, I have not gone through the
sources that deep) when the event handler is used for not only libcurl
events.
If the event handler is only handling libcurl events then all is just fine
with the timer callback, but if other sockets or file descriptors are
handled by the same event handler and one ends up in a situation where
those other sockets/descriptors are very busy then it's quite possible that
the event handler will not return back with a timeout for quite a while,
and since libcurl can only update it's internal state and call the timer
callback if one calls curl_multi_socket_action() then one either have to
call curl_multi_socket_action() after each event handler return (and I
assume here without checking that curl_multi_socket_action() is a much
heavier function to call than curl_multi_timeout(), but I could of course
be wrong), or one have to track the time taken during the event handler
call (which is hard on e.g Linux since select/poll/epoll doesn't indicate
how long it really waited).
What I mean is that the naive implementation of the multi socket API that
works 100% of the time if curl is the only one generating events and 99% of
the time if you have other external events but not much traffic on them:
for (;;)
int ret = poll (fds, nfds, timeout);
if (ret == 0) {/* timeout */
curl_multi_socket_action (curlm, CURL_SOCKET_TIMEOUT, 0,
&running_handles);
} else if (ret != -1) { /* events */
if (fds[0].revents != 0)
curl_multi_socket_action (curlm, fds[0].fd, fds[0].revents,
&running_handles);
else if (fds[1]).revents != 0)
...
}
}
Yes I know that we cannot put fds[0].revents directly into curl but this
was just a small example of the typical event structure from someone using
poll().
Now if curl is the only one generating events here the timer function
callback will be called by curl and the "timeout" variable will be set
appropriate. However if we add some other socket here on say fds[1] above
that is extremely high in volume then it's quite possible that poll() will
very seldom return 0 even when the timeout from curl is very low leading to
a situation where curl will "hang", typically I have seen this directly
after a call to curl_multi_add_handle() where curl tends to first call the
timer function callback with a timeout of 0 and then after the first call
to curl_multi_socket_action() the timer function callback is called again
with a timeout of 1 but that 1 can be enough for a very highly stressed
other socket prohibiting poll from timing out even for 1ms.
Therefore I have noticed that the following logic and where we don't set
any timer function callback at all, works much better:
for (;;) {
long timeout;
curl_multi_timeout (curlm, &timeout);
if (timeout == 0)
curl_multi_socket_action (curlm, CURL_SOCKET_TIMEOUT, 0,
&running_handles);
int ret = poll (fds, nfds, timeout);
if (ret > 0) { /* events */
if (fds[0].revents != 0)
curl_multi_socket_action (curlm, fds[0].fd, fds[0].revents,
&running_handles);
else if (fds[1]).revents != 0)
...
}
}
So unless there is some potential lethal overhead in curl_multi_timeout() I
actually find this to work much better than the timer function callback.
Regards, Henrik Holst