lkml.org 
[lkml]   [2012]   [Aug]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: regression with poll(2)?
On Wed, 15 Aug 2012, Atchley, Scott wrote:
> On Aug 15, 2012, at 3:46 PM, Sage Weil wrote:
>
> > I'm experiencing a stall with Ceph daemons communicating over TCP that
> > occurs reliably with 3.6-rc1 (and linus/master) but not 3.5. The basic
> > situation is:
> >
> > - the socket is two processes communicating over TCP on the same host, e.g.
> >
> > tcp 0 2164849 10.214.132.38:6801 10.214.132.38:51729 ESTABLISHED
> >
> > - one end writes a bunch of data in
> > - the other end consumes data, but at some point stalls.
> > - reads are nonblocking, e.g.
> >
> > int got = ::recv( sd, buf, len, MSG_DONTWAIT );
> >
> > and between those calls we wait with
> >
> > struct pollfd pfd;
> > short evmask;
> > pfd.fd = sd;
> > pfd.events = POLLIN;
> > #if defined(__linux__)
> > pfd.events |= POLLRDHUP;
> > #endif
> >
> > if (poll(&pfd, 1, msgr->timeout) <= 0)
> > return -1;
> >
> > - in my case the timeout is ~15 minutes. at that point it errors out,
> > and the daemons reconnect and continue for a while until hitting this
> > again.
> >
> > - at the time of the stall, the reading process is blocked on that
> > poll(2) call. There are a bunch of threads stuck on poll(2), some of them
> > stuck and some not, but they all have stacks like
> >
> > [<ffffffff8118f6f9>] poll_schedule_timeout+0x49/0x70
> > [<ffffffff81190baf>] do_sys_poll+0x35f/0x4c0
> > [<ffffffff81190deb>] sys_poll+0x6b/0x100
> > [<ffffffff8163d369>] system_call_fastpath+0x16/0x1b
> >
> > - you'll note that the netstat output shows data queued:
> >
> > tcp 0 1163264 10.214.132.36:6807 10.214.132.36:41738 ESTABLISHED
> > tcp 0 1622016 10.214.132.36:41738 10.214.132.36:6807 ESTABLISHED
> >
> > etc.
> >
> > Is this a known regression? Or might I be misusing the API? What
> > information would help track it down?
> >
> > Thanks!
> > sage
>
>
> Sage,
>
> Do you see the same behavior when using two hosts (i.e. not loopback)? If different, how much data is in the pipe in the localhost case?

I have only seen it in the loopback case, and have independently diagnosed
it a half dozen or so times now.

:/
sage


>
> Scott
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>


\
 
 \ /
  Last update: 2012-08-15 23:43    [W:1.321 / U:0.024 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site