lkml.org 
[lkml]   [2012]   [Oct]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [sqlite] light weight write barriers
On Thu, 25 Oct 2012, Theodore Ts'o wrote:

> Or does rsyslog *really* need to issue an fsync after each log
> message? Or could it batch updates so that every N seconds, it
> flushes writes to the disk?

In part this depends on how paranoid the admin is. By default rsyslog
doesn't do fsyncs, but admins can configure it to do so and can configure
the batch size.

However, what I'm talking about here is not normal message traffic, it's
the case where the admin has decided that they don't want to use the
normal inmemory queues, they want to have the queues be on disk so that if
the system crashes the queued data will still be there to be processed
after the crash (In addition, this can get used to cover cases where you
want queue sizes larger than your available RAM)

In this case, the extreme, and only at the explicit direction of the
admin, is to fsync after every message.

The norm is that it's acceptable to loose the last few messages, but
loosing a chunk out of the middle of the queue file can cause a whole lot
more to be lost, passing the threshold of acceptable.

> Sometimes, the answer is not to try to create exotic database like
> functionality in the file system --- the answer is to be more
> intelligent at the application leyer. Not only will the application
> be more portable, it will also in the end be more efficient, since
> even with the most exotic database technologies, the most efficient
> transactional commit is the unneeded commit that you optimize away at
> the application layer.

I agree, this is why I'm trying to figure out the recommended way to do
this without needing to do full commits.

Since in most cases it's acceptable to loose the last few chunks written,
if we had some way of specifying ordering, without having to specify
"write this NOW", the solution would be pretty obvious.

David Lang


\
 
 \ /
  Last update: 2012-10-25 20:41    [W:0.075 / U:0.384 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site