Messages in this thread | | | Date | Wed, 28 May 2008 18:22:20 -0300 | From | "Fausto Richetti Blanco" <> | Subject | Re: Pipe buffers' limit of 16 * 4K |
| |
I think you did not receive it so I'm sending again....
On Mon, May 26, 2008 at 2:43 PM, Fausto Richetti Blanco <fausto.blanco@gmail.com> wrote: > Hi Rik, > > I'll try to explain my problem... > > Well, we have a cgi which uses a lib. This lib is used by a lot of > other systems and we don't want to change it nor create a specific > version of the lib to solve our problem. The webserver passes the post > content to the cgi via stdin which, in this case, is a pipe. > > Our problem is that our application must look at the post content > (ie.: by reading stdin and, consequently, removing its content from > the pipe buffer), take some decisions and then call the lib. The lib > itself will also look at the stdin and here is the problem: since our > application had already consumed the input content, we must restore it > in order to the lib to be able to read it again. > > We can't write to stdin because we only have access to the 'read > side' of the pipe. The way we find to circumvent this is by creating > another pipe. So, our application reads the stdin and saves its > content in a buffer. To restore the stdin, we are doing this: > > void restore_input(void){ > int filedes[2]; > pipe(filedes); > write(filedes[1], newstdin, stdbuffsz); // *1 > close(filedes[1]); > close(STDIN_FILENO); > dup2(filedes[0], STDIN_FILENO); > close(filedes[0]); > } > // *1 newstdin is the buffer we have saved the input > > So, we can: > read_input_and_save_it() > take_our_decisions() > restore_input() > call_the_lib() > > It works very well, except when the input has more than 4K (or 16 > * 4K in more recent kernels) because the restore_input() blocks at > this limit. > > I kwow there are other solutions to my problem (e.g: using a > thread, moving our decisions to the lib, etc...) by I'm wondering if > making the pipe buffers' limit adjustable is not a good idea. Maybe it > should be helpful for another things too (like Jan Engelhardt said in > his email). > > In fact, I didn't find any way of restoring the input (with the > input being the 'read side' of a pipe) other than using pipes. That's > because I've decided to ask this in the linux-kernel list. Is there a > reason for this limit not to be an adjustable parameter ? > > Thanks in advance, > > Fausto Richetti Blanco > > On Fri, May 23, 2008 at 9:56 PM, Rik van Riel <riel@redhat.com> wrote: >> On Fri, 23 May 2008 21:19:13 -0300 >> "Fausto Richetti Blanco" <fausto.blanco@gmail.com> wrote: >> >>> Hello guys, >>> >>> I'm working with a 2.6.9 kernel (ok, I know it's quite old) and >>> faced a problem with the 4K (one page) buffer limit for the pipes. >>> I've found that in the 2.6.11 the pipes' buffers were changed to a >>> circular list of pages which increased this limit to 16 * 4K. This >>> limit is hardcoded here /usr/src/linux/include/linux/pipe_fs_i.h:6 >>> #define PIPE_BUFFERS (16) >>> >>> Is there a reason for this not to be an adjustable parameter >>> (eg.: by an ulimit in the userspace) ? >> >> What is the problem you found? >> >> Why do you need to change the limit from 16? >> >> Did it bring you any performance enhancements? >> >> If so, how much? >> >> -- >> All rights reversed. >> >
| |