lkml.org 
[lkml]   [1999]   [Mar]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subject[pre-announce] snapfs, a snapshot-based filesystem
I'm working on a new filesystem I call snapfs.  This message is a
preliminary announcement: I hope the code will be ready for a first
public release in a month or two. I think it will be pretty cool, but
can't make any promises, and I'm writing snapfs as much for my own
education and entertainment as anything else.

I'm mindful of the etiquette about announcing before the code is
ready, and so I'm posting this just to get in touch with anybody else
working in a similar area. Please mail me if you're interested.

--
Martin Pool




README for snapfs
Copyright (C) 1998, 1999 Martin Pool

$Id: README,v 11.3 1999-03-15 20:24:54+10 mbp Exp $

Please send comments/bug reports to <mbp@humbug.org.au>

* Introduction

snapfs is a new filesystem type for Linux. It offers two cool new
features: access to previous versions of edited, replaced, or deleted
files; and recovery from crashes with no need to fsck. snapfs has
some features in common with the NetApp WAFL filesystem and
log-structured filesystems, but is an original design and
implementation.

* Caveats

Reliablity is of course a very strong requirement for snapfs, but it's
still fairly young code. Please make (and test!) backups of any
information you keep on a snapfs partitions.

* Freedom

snapfs is Free Software(tm): you can use and redistribute it freely,
but only under the terms of the GNU General Public License. In
particular, you may not distribute snapfs with an operating system
unless that operating system is licensed under the GPL.

* Features and benefits

** Roll back to previous versions

This is like undelete, but much _much_ better: you can revisit
previous versions of a file, no matter whether it was deleted,
replaced or edited. Snapshots are taken over the whole filesystem, so
deleted files are accessible even if the directory hierarchy that
contained them has been changed or deleted. Snapshots behave like
read-only copies of the whole filesystem, except that because they're
made using a copy-on-write mechanism the copies are made nearly
instantaneously, and they occupy much less space. Snapshots are
accessed using a special filename syntax.

Files within snapshots are never overwritten and lost or damaged --
they're retained read-only on disk until the snapshot is discarded.

** Configurable snapshots

Snapshots can be taken and released at any time. Minor snapshots are
made and released every minute and act as regular consistency points
in the case of a crash. Major snapshots may be retained for days or
weeks at the option of the administration.

** Self-consistent backups

Snapshots are perfect for making backups: because they're an image of
the filesystem at one point in time, they can be safely used to make a
consistent backup without needing to halt operations on the machine.
Databases, relational or otherwise, need only be brought momentarily
to a quiet point to take the snapshot -- they need not stay down for
the duration of the backup.

** Zero fscking delays

If the machine crashes, there's no need to check filesystem
consistency -- snapfs simply rolls back to the last consistent
snapshot, which is usually less than a minute old. This should
greatly reduce the time for a machine to boot after a crash.
Consistency includes both the filesystem structure and the contents of
the files.

If a program/user has just written some really important information
to disk, they might want to do a sync(2) to make sure it really is
written to disk. snapfs interprets this as a request to synchronously
take a snapshot: when it returns, the data will have been reliably
written to disk and will be used to recover until a new snapshot is
written.

Any promises about recoverability are naturally void in the presence
of hard disk errors.

There is a cksnapfs(8) program that acts as a self-test against the
snapfs code, but there should be no need to run it against stable
versions. cksnapfs(8) may be even run against snapshots on a mounted
partition to check that everything is running correctly.

** Standard filesystem interface

No changes to applications or the kernel are required. snapfs loads
as a module into Linux 2.1 kernels, and registers a new filesystem
type. The package also includes tools to create, check, and debug
snapfs filesystems.

Old versions of the filesystem simply appear in hidden directories,
and they can be operated upon like normal (read-only) files: simply
diff the file against the new version to find out what has changed,
and copy it out to retrieve it.

** Expandable filesystems

If the size of a snapfs partition increases, whether by concatenating
additional RAID disks or by rearranging disk partitions, snapfs can
take advantage of the additional space without building a new
filesystem. inodes and other metadata are allocated on the fly to
fill up the partition.

inodes are stored in a metafile that expands on demand, so there's no
need to guess how many inodes will be required at the time of
filesystem creation.

* Limits

File sizes are stored on disk as 64-bit numbers, and inodes and block
numbers are stored as 32-bit numbers. This allows filesystems on the
order of 1TB: larger filesystems are out of the scope of the project
for the time being. (Note that current (as of 2.1.132) Linux kernels
limit file sizes to 2GB on 32-bit processors.)

File names may be over 256 bytes, to accomodate future kernel support
for Unicode filenames.

Blocks are currently fixed at 4kB. Future versions may allow block
sizes to be set when the filesystem is created.

snapfs doesn't currently support fragments or other means of sub-block
allocation: a one-byte file occupies one inode and one data block.
This may be addressed in the future by storing small files within the
inode and/or sub-block allocations.

* Compatibility

** Operating systems

The current implementation runs on Linux 2.2 (and late model 2.1)
kernels. There is no reason why the code could not be ported to other
systems.

** Architectures

The current version has only been tested on x86. I'd be delighted to
work on making it run on other architectures. Filesystems may be
shared between different architectures without endianess or size
problems.

** Filesystems

The on-disk format is unique to snapfs, but snapfs filesystems may of
course be mounted simultaneously with other filesystems.

** Support for RAID

snapfs is designed to work well with software and hardware RAID
systems, including Linux's md, though there is no explicit support.

Snapshots seem to present an useful way to restore filesystems between
disks when a broken mirrored disk fails and is replaced.

* Nonfeatures

** Metadata and forks

With the exception of snapshots mapped to special directories, snapfs
presents completely standard UNIX filesystem semantics to the
operating system and applications.

In particular, snapfs files do not include resource forks, metadata,
attributes, multiple streams or similar concepts. A file is a file.

** Compression

Compression within the filesystem is another controversial topic, and
probably not appropriate.

** Encryption

Filesystems may be encrypted at the block-level through the loopback
device driver.

** Quotas

Quotas are handled at the VFS layer and should work fine on snapfs.

** Access control

snapfs uses standard POSIX file permissions. POSIX ACLs may be added
later when they're supported by the kernel.

* Requirements

** Partition

Like other filesystem types, snapfs stores its data on a block
device. This is usually a hard disk partition, such as /dev/hda2 or
/dev/sda2.

snapfs can run on a loopback device, which maps a file into a block
device. This is a good option for trying out snapfs without
repartitioning your machine. However, if snapfs crashes you may not
be able to cleanly unmount the host partition.

snapfs can also run on a ramdisk, which is an even cleaner way of
testing it. (Of course, the information will not persist over reboots
unless you backup the system.) To do this, just make sure the ramdisk
block device is configured into your kernel either builtin or as a
module, and use a device like /dev/ram0.

* FAQ

** I get a mysterious error when mounting the filesystem

mount(8) doesn't display many useful messages -- check the console or
syslog messages instead. Often, these messages are redirected by
klogd and syslogd into /var/log/messages or /var/log/kern. On 2.1,
kernel messages may be available in /proc/kmsg.

** Is snapfs a log-structured / journalling filesystem?

I don't think it's a LFS, but it depends on how you define it.

snapfs and LFSs offer similar features: quick recovery from crashes
and being able to use stable snapshots to make backups is the same in
some ways.

In some ways, a snapshot-based filesystem is better: it takes a small
constant amount of time to recover from a crash (rather than time
proportional to activity), and it can more easily support a number of
snapshots at different points in history. It seems like a snapshot
filesystem can be implemented more simply and efficiently. (Famous
last words...)

LFSs usually require some kind of garbage-collection process, which
takes a certain amount of process time and leaks some space, whereas
snapfs can tell immediately and 'deterministically' which blocks are
free and in use.
[unhandled content-type:application/pgp-signature]
\
 
 \ /
  Last update: 2005-03-22 13:50    [W:0.066 / U:2.024 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site