lkml.org 
[lkml]   [2023]   [Feb]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem
    [+overlayfs list]

    On Mon, Feb 6, 2023 at 7:16 PM Amir Goldstein <amir73il@gmail.com> wrote:
    >
    > On Mon, Feb 6, 2023 at 6:34 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
    > >
    > > On Mon, 6 Feb 2023 at 14:31, Amir Goldstein <amir73il@gmail.com> wrote:
    > > >
    > > > > > > My little request again, could you help benchmark on your real workload
    > > > > > > rather than "ls -lR" stuff? If your hard KPI is really what as you
    > > > > > > said, why not just benchmark the real workload now and write a detailed
    > > > > > > analysis to everyone to explain it's a _must_ that we should upstream
    > > > > > > a new stacked fs for this?
    > > > > > >
    > > > > >
    > > > > > I agree that benchmarking the actual KPI (boot time) will have
    > > > > > a much stronger impact and help to build a much stronger case
    > > > > > for composefs if you can prove that the boot time difference really matters.
    > > > > >
    > > > > > In order to test boot time on fair grounds, I prepared for you a POC
    > > > > > branch with overlayfs lazy lookup:
    > > > > > https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata
    > > > >
    > > > > Sorry about being late to the party...
    > > > >
    > > > > Can you give a little detail about what exactly this does?
    > > > >
    > > >
    > > > Consider a container image distribution system, with base images
    > > > and derived images and instruction on how to compose these images
    > > > using overlayfs or other methods.
    > > >
    > > > Consider a derived image L3 that depends on images L2, L1.
    > > >
    > > > With the composefs methodology, the image distribution server splits
    > > > each image is split into metadata only (metacopy) images M3, M2, M1
    > > > and their underlying data images containing content addressable blobs
    > > > D3, D2, D1.
    > > >
    > > > The image distribution server goes on to merge the metadata layers
    > > > on the server, so U3 = M3 + M2 + M1.
    > > >
    > > > In order to start image L3, the container client will unpack the data layers
    > > > D3, D2, D1 to local fs normally, but the server merged U3 metadata image
    > > > will be distributed as a read-only fsverity signed image that can be mounted
    > > > by mount -t composefs U3.img (much like mount -t erofs -o loop U3.img).
    > > >
    > > > The composefs image format contains "redirect" instruction to the data blob
    > > > path and an fsverity signature that can be used to verify the redirected data
    > > > content.
    > > >
    > > > When composefs authors proposed to merge composefs, Gao and me
    > > > pointed out that the same functionality can be achieved with minimal changes
    > > > using erofs+overlayfs.
    > > >
    > > > Composefs authors have presented ls -lR time and memory usage benchmarks
    > > > that demonstrate how composefs performs better that erofs+overlayfs in
    > > > this workload and explained that the lookup of the data blobs is what takes
    > > > the extra time and memory in the erofs+overlayfs ls -lR test.
    > > >
    > > > The lazyfollow POC optimizes-out the lowerdata lookup for the ls -lR
    > > > benchmark, so that composefs could be compared to erofs+overlayfs.
    > >
    > > Got it, thanks.
    > >
    > > >
    > > > To answer Alexander's question:
    > > >
    > > > > Cool. I'll play around with this. Does this need to be an opt-in
    > > > > option in the final version? It feels like this could be useful to
    > > > > improve performance in general for overlayfs, for example when
    > > > > metacopy is used in container layers.
    > > >
    > > > I think lazyfollow could be enabled by default after we hashed out
    > > > all the bugs and corner cases and most importantly remove the
    > > > POC limitation of lower-only overlay.
    > > >
    > > > The feedback that composefs authors are asking from you
    > > > is whether you will agree to consider adding the "lazyfollow
    > > > lower data" optimization and "fsverity signature for metacopy"
    > > > feature to overlayfs?
    > > >
    > > > If you do agree, the I think they should invest their resources
    > > > in making those improvements to overlayfs and perhaps
    > > > other improvements to erofs, rather than proposing a new
    > > > specialized filesystem.
    > >
    > > Lazy follow seems to make sense. Why does it need to be optional?
    >
    > It doesn't.
    >
    > > Does it have any advantage to *not* do lazy follow?
    > >
    >
    > Not that I can think of.
    >
    > > Not sure I follow the fsverity requirement. For overlay+erofs case
    > > itsn't it enough to verify the erofs image?
    > >
    >
    > it's not overlay{erofs+erofs}
    > it's overlay{erofs+ext4} (or another fs-verity [1] supporting fs)
    > the lower layer is a mutable fs with /objects/ dir containing
    > the blobs.
    >
    > The way to ensure the integrity of erofs is to setup dm-verity at
    > erofs mount time.
    >
    > The way to ensure the integrity of the blobs is to store an fs-verity
    > signature of each blob file in trusted.overlay.verify xattr on the
    > metacopy and for overlayfs to enable fsverity on the blob file before
    > allowing access to the lowerdata.
    >

    Perhaps I should have mentioned that the lower /objects dir, despite
    being mutable (mostly append-only) is shared among several overlays.

    This technically breaks the law of no modification to lower layer,
    but the /objects dir itself is a whiteout in the metadata layer, so
    the blobs are only accessible via absolute path redirect and there
    is no /objects overlay dir, so there is no readdir cache to invalidate.
    Naturally, the content addressable blobs are not expected to be
    renamed/unlinked while an overlayfs that references them is mounted.

    Thanks,
    Amir.

    \
     
     \ /
      Last update: 2023-03-27 00:11    [W:4.370 / U:0.120 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site