lkml.org 
[lkml]   [2020]   [Mar]   [25]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Subject[PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs
    Date
    From: Nick Terrell <terrelln@fb.com>

    Hi all,

    I attempted to send this patch on March 19, but it got rejected by the
    mailing lists for some reason, so I'm resending it.

    This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
    initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
    are supported on all architectures. The ZSTD-compressed kernel is only
    hooked up to x86 in this patch set.

    Zstandard requires slightly more memory during the kernel decompression
    on x86 (192 KB vs 64 KB), and the memory usage is independent of the
    window size.

    Zstandard requires memory proprortional to the window size used during
    compression for decompressing the ramdisk image, since streaming mode is
    used. Newer versions of zstd (1.3.2+) list the window size of a file
    with `zstd -lv <file>'. The absolute maximum amount of memory required
    is just over 8 MB, but it can be controlled at compression time.

    This patch set has been boot tested with buildroot and QEMU based off
    of linux-5.6-rc6.

    On i386 and x86_64 I have tested the following configurations:
    * zstd compressed kernel and a separate zstd compressed initramfs
    * zstd compressed kernel and a built-in zstd compressed initramfs
    * gzip compressed kernel and a separate gzip compressed initramfs
    * gzip compressed kernel and a built-in gzip compressed initramfs

    On arm and aarch64 I tested the same configurations, except that the kernel is
    always gzip compressed.

    Facebook has been using v1 of these patches on x86_64 devices for more than 6
    months. When we switched from a xz compressed initramfs to a zstd compressed
    initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
    switched from a xz compressed kernel to a zstd compressed kernel we saved 2
    seconds of boot time.

    Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
    When we switched from an lzma compressed initramfs to a zstd compressed initramfs
    decompression time shrunk from 27 seconds to 8 seconds.

    The zstd compressed kernel is smaller than the gzip compressed kernel but larger
    than the xz or lzma compressed kernels, and it decompresses faster than
    everything except lz4. See the table below for the measurement of an x86_64
    kernel ordered by compressed size:

    algo size
    xz 6,509,792
    lzma 6,856,576
    zstd 7,399,157
    gzip 8,522,527
    bzip 8,629,603
    lzo 9,808,035
    lz4 10,705,570
    none 32,565,672

    v1 -> v2:
    - Rebase
    - usr/Makefile and init/Kconfig were changed so the patches were updated
    - No functional changes except to rebase
    - Split the patches up into smaller chunks

    v2 -> v3:
    - Add *.zst to the .gitignore in patch 8
    - Style nits in patch 3
    - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
    1 through 3

    Best,
    Nick Terrell

    Adam Borowski (1):
    .gitignore: add ZSTD-compressed files

    Nick Terrell (7):
    lib: prepare zstd for preboot environment
    lib: prepare xxhash for preboot environment
    lib: add zstd support to decompress
    init: add support for zstd compressed kernel
    usr: add support for zstd compressed initramfs
    x86: bump ZO_z_extra_bytes margin for zstd
    x86: Add support for ZSTD compressed kernel

    .gitignore | 1 +
    Documentation/x86/boot.rst | 6 +-
    arch/x86/Kconfig | 1 +
    arch/x86/boot/compressed/Makefile | 5 +-
    arch/x86/boot/compressed/misc.c | 4 +
    arch/x86/boot/header.S | 8 +-
    arch/x86/include/asm/boot.h | 6 +-
    include/linux/decompress/unzstd.h | 11 +
    init/Kconfig | 15 +-
    lib/Kconfig | 4 +
    lib/Makefile | 1 +
    lib/decompress.c | 5 +
    lib/decompress_unzstd.c | 338 ++++++++++++++++++++++++++++++
    lib/xxhash.c | 21 +-
    lib/zstd/decompress.c | 2 +
    lib/zstd/fse_decompress.c | 9 +-
    lib/zstd/zstd_internal.h | 14 +-
    scripts/Makefile.lib | 15 ++
    usr/Kconfig | 20 ++
    usr/Makefile | 1 +
    20 files changed, 460 insertions(+), 27 deletions(-)
    create mode 100644 include/linux/decompress/unzstd.h
    create mode 100644 lib/decompress_unzstd.c

    --
    2.25.1

    \
     
     \ /
      Last update: 2020-03-25 20:58    [W:4.634 / U:0.572 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site