lkml.org 
[lkml]   [2000]   [Aug]   [12]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: NTFS-like streams?
From

This discussion is looking like an year-old discussion we had in lkml before.

I think that a streams-like structure can be created in userspace, and made a
specification for a "libalbod" to manage it. But I buried the project after I
noticed it was useless.

So, here's the design for libalbod I made long ago. I'm not going to implement
it, so feel free to make a sourceforge project out of it.

--
Cesar Eduardo Barros
cesarb@nitnet.com.br
cesarb@dcc.ufrj.br
Here is my proposal for a working implementation of albods.

The current requeriments is:

1. It must be backwards compatible without the need for LD_PRELOAD hacks, but
should be able to use them

2. For some file formats (like mp3), the data part is the only important one;
for some others (e.g. a new albod-only fileformat) the albod should be traeted
as a single unit for non-albod-aware programs

3. The albod must be represented as a flat directory; however, we have to think
hard about albods-within-albods and compatibility issues. Also, it must have a
'packed' representation and should be able to convert between them on the fly.
The packed representation should be readable by standard unix tools. Also, to
confuse a bit more, it can have subdirs which can pack/unpack on the fly like
itself (so it can still be treated like a flat directory).

[FIXME: any other ideas?]

So I planned this structure:
- Albod is filename.albod, e.g., my_song.mp3.albod
- Data part, if using compatibility mode, is my_song.mp3
- Dot files within the albod are reserved
- Applications should agree to what should be the name of their subparts

The structure is this:
my_song.mp3.albod/
.options
data
icon.xpm
my_song.mp3 -> hardlink to my_song.mp3.albod/data

Hard links generate maximum compatibility, but causes some randomness if they
are deleted. My current idea is to ignore it to avoid confusion and creating an
utility to fix the albod manually.

Here are the explanations for the files:
my_song.mp3.albod/ - Treated as a file called my_song.mp3 by albod
aware apps
.options - Contains (without quotes, one per line):
MUST BE THE FIRST ELEMENT WHEN PACKED!
FIRST LINE IS ALWAYS "ALBOD=6518F161"
(file magic)!

"compat=data" if the compatibility link should
be a hard link to the 'data' part
It is used for already existing formats like
.mp3

"compat=dir" if the compatibility link should
be a symlink to the whole albod dir (we
can't hardlink directories)
Used when you want to treat the albod as a dir
when using old apps

"compat=albod" if the compatibility link should
be a file which is the whole albod in
packed form (treated as compat=none if
there is no LD_PRELOAD hack running)
Used when you want to be able to 'cp' a whole
albod. This is the best idea for new apps.

"compat=none" means there is no compatibility
link and that non-albod apps should not
be doing anything with this albod
Used when you're picky and thinks legacy apps
should not be used in your shiny new file. Also
used when the LD_PRELOAD hack is not available.

DEFAULT: compat=none

"pack=always" means it should be unpacked on
the fly by the lib and should not be
stored as a normal dir
This uses less disk space and is slower. This
also makes compat=albod work without LD_PRELOAD
hacks.

"pack=auto" means it can be stored as a dir
and will be autounpacked to a dir
(replacing the packed file)
This uses less disk space but requires a lot of
it while unpacking. This is used so that you
can copy an albod as a file by some random app
and it will be unpacked to its normal form when
some albod-aware lib touches it.
The autounpack option controls the way it is
done.

"pack=never" means it _should_ be stored as a
dir. A program needs override flags to
force packing anyways. It can read it
as a file if it wants, but when created
from file will fully unpack.
With pack=auto, a cron job/disk packing daemon/
user requested pack-all-my-dirs-because-I-need-
more-space can repack the albod. This option
is not intended to be used except in some very
special cases.

DEFAULT: pack=auto

"autounpack=full" means that when albod is
opened it is fully unpacked if it was
packed. The original file is replaced
by the dir.
This avoids the loss of disk space caused by
constant unpacking and is faster.

"autounpack=auto" means that only the
acessed files will be unpacked. After
all are acessed (and unpacked), the
file will be replaced by the dir.
Faster startup than autounpack=full, but uses
a lot of disk space before it gets fully
unpacked.

"autounpack=partial"
Like autounpack=auto, but files will be
repacked after being closed. The dir
will never apear unless explicitly
requested.
Faster than pack=always, and uses less space
than autounpack=auto. Opens the whole albod
faster than autounpack=full.

"autounpack=never"
Like autounpack=partial, but do not
allow the dir to appear even if
requested.
Used when you want the faster speed of
pack=auto but do not want to allow it to appear
as a dir.

DEFAULT: autounpack=full

"version=0"
If version is _not_ zero, the program
must report incompatible file formats.
Used for forward compatibility.

DEFAULT: version=0

"mimetype=application/octet-stream"
MIME type for this albod's 'data' part.
If it is "application/x-albod" it means
this albod's data is understood only
by albod-aware apps.

DEFAULT: if compat=data
mimetype=application/octet-stream
else
mimetype=application/x-albod

Note that compat=data won't make sense with
pack=always or autounpack=never. Also
autounpack makes no sense with pack != auto.
.lock - Subdir with one item per item in the root of
the albod. If it is present here, it has the
FQDN:PID of the program using this item
(part/dotfile).
This subdir won't appear in packed albods, and
is the only item that can't be locked (for
obvious reasons).
data - The main data part. Required if compat=data or
mimetype is not albod.
Optional if compat != data and
mimetype==application/x-albod or x-albod-subdir.
data.* - Application-specific parts start with data. .
icon.xpm - An example of a non-application-specific part.
A icon. Completely optional.
.my_song.mp3.albod - A packed albod. It is a normal file, not a dir.
.my_song.mp3.albod.tmp/ - The partially-unpacked albod dir
.my_song.mp3.albod.lock - This locks the whole albod. Used when messing
with the directory/packed file structure.
It contains FQDN:PID.
my_song.mp3 - This is the compatibility link to:
my_song.mp3.albod{,.tmp}/data (hardlink); or
my_song.mp3.aldob{,.tmp}/ (symlink); or
my_song.mp3.albod (symlink to file); or
nothing (it isn't there).

If any of the content parts (data, data.* and the non-app-specific parts) is a
subdir, it is treated like an albod (the .options file is inherited with the
following changes: compat=none, and mimetype for compat=none is
application/x-albod-sudir. These can be overriden).

When an albod-aware wants to open an albod, it calls the albod lib with the
filename without the extension. This means:
1. If my_song.mp3.albod exists, it does its part (magic unpacks/whatever) and
returns the whole albod;
2. If it doesn't but my_song.mp3 exists, it creates a 'virtual albod' in memory
with only the 'data' part (the default values for the options file are right
in this case, since compat=data and mimetype=application/octet-stream). If
the albod is changed (not the locks or the data), a real albod is created
(holding the main albod lock, creating the dir and hardlinking the data
part, creating the changed file, getting the lock on it and releasing the
albod global lock)

The packed albod is just an ar file. If the albod has other albods within it,
they will be packed first. If needed, a dir called .my_song.mp3.albod.pack and
its lockfile (.my_song.mp3.albod.pack.lock) will be created to hold temporary
files.

Albods _must_ have .options files. If it doesn't or its magic is invalid it is
treated as a normal dir/file.

Files within an albod are treated like files outside an albod for albod-aware
apps, that is:
- If my_song.mp3.albod exists, treat it as my_song.mp3 and ignore the real
my_song.mp3 if it exists
- If my_song.mp3 exists but my_song.mp3.albod doesn't, treat it like a default
albod (like when opening it)
- dotfiles are ignored (this is to be consistent with what happens inside an
albod).
- Normal dirs are treated like subdir albods (compat=none, mimetype
x-albod-subdir).

[FIXME: s/exists/exists and is valid/ above]

[FIXME: explain this mess better]
\
 
 \ /
  Last update: 2005-03-22 13:58    [W:0.076 / U:0.116 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site