lkml.org 
[lkml]   [2019]   [Feb]   [18]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [PATCH v2 1/2] Provide in-kernel headers for making it easy to extend the kernel
    On Tue, Feb 19, 2019 at 01:14:11PM +0900, Masahiro Yamada wrote:
    > On Fri, Feb 15, 2019 at 11:48 PM Alexei Starovoitov
    > <alexei.starovoitov@gmail.com> wrote:
    > >
    > > On Mon, Feb 11, 2019 at 09:35:59AM -0500, Joel Fernandes (Google) wrote:
    > > > Introduce in-kernel headers and other artifacts which are made available
    > > > as an archive through proc (/proc/kheaders.txz file).
    >
    >
    > The extension '.txz' is not used in kernel code.
    >
    >
    > '.tar.xz' is used for 'tarxz-pkg', 'perf-tarxz-src-pkg' etc.
    >
    >
    > $ git grep '\.txz'
    > $ git grep '\.tar\.xz'
    > Documentation/admin-guide/README.rst: xz -cd linux-4.X.tar.xz | tar xvf -
    > arch/x86/crypto/camellia-aesni-avx-asm_64.S: *
    > http://koti.mbnet.fi/axh/crypto/camellia-BSD-1.2.0-aesni1.tar.xz
    > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
    > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
    > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
    > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
    > crypto/testmgr.h: * https://bench.cr.yp.to/supercop/supercop-20170228.tar.xz
    > scripts/package/Makefile: @echo ' perf-tarxz-src-pkg - Build
    > $(perf-tar).tar.xz source tarball'
    > tools/testing/selftests/gen_kselftest_tar.sh:
    > ext=".tar.xz"
    >
    >
    >
    > I prefer '.tar.xz' for consistency.

    Ok, I will change it to that.

    > BTW, have you ever looked at scripts/extract-ikconfig?
    >
    > You added IKHD_ST and IKHD_ED
    > just to mimic kernel/configs.c
    >
    > It is currently pointless without the extracting tool,
    > but you might think it is useful to extract headers
    > from vmlinux or the module without mounting procfs.

    I am planing add to extraction support in the future, so I prefer to leave the
    markers as it is for now. Hope that's Ok with you.


    > > > This archive makes
    > > > it possible to build kernel modules, run eBPF programs, and other
    > > > tracing programs that need to extend the kernel for tracing purposes
    > > > without any dependency on the file system having headers and build
    > > > artifacts.
    > > >
    > > > On Android and embedded systems, it is common to switch kernels but not
    > > > have kernel headers available on the file system. Raw kernel headers
    > > > also cannot be copied into the filesystem like they can be on other
    > > > distros, due to licensing and other issues. There's no linux-headers
    > > > package on Android. Further once a different kernel is booted, any
    > > > headers stored on the file system will no longer be useful. By storing
    > > > the headers as a compressed archive within the kernel, we can avoid these
    > > > issues that have been a hindrance for a long time.
    > >
    > > The set looks good to me and since the main use case is building bpf progs
    > > I can route it via bpf-next tree if there are no objections.
    > > Masahiro, could you please ack it?
    >
    >
    > Honestly, I was not tracking this thread
    > since I did not know I was responsible for this.
    >
    >
    > I just started to take a closer look, then immediately got scared.
    >
    > This version is not mature enough for the merge.
    > First of all, this patch cannot be compiled out-of-tree (O= option).

    Oh, ok. This is not how I build the kernel. So I am sorry for missing that. I
    will try this out-of-tree build option and fix it.


    > I do not know why 0-day bot did not catch this apparent breakage.
    >
    >
    > $ make -j8 O=hoge
    > make[1]: Entering directory '/home/masahiro/workspace/bsp/linux/hoge'
    > GEN Makefile
    > Using .. as source for kernel
    > DESCEND objtool
    > CALL ../scripts/checksyscalls.sh
    > CHK include/generated/compile.h
    > make[2]: *** No rule to make target 'Module.symvers', needed by
    > 'kernel/kheaders_data.txz'. Stop.
    > make[2]: *** Waiting for unfinished jobs....
    > /home/masahiro/workspace/bsp/linux/Makefile:1043: recipe for target
    > 'kernel' failed
    > make[1]: *** [kernel] Error 2
    > make[1]: *** Waiting for unfinished jobs....
    > make[1]: Leaving directory '/home/masahiro/workspace/bsp/linux/hoge'
    > Makefile:152: recipe for target 'sub-make' failed
    > make: *** [sub-make] Error 2
    >
    >
    >
    >
    >
    > I was able to compile it in-tree
    > but it makes the incremental build extremely slow.
    >
    > (Here, the incremental build means
    > "make" without changing any code after the full build.)
    >
    >
    >
    >
    > Before this patch, "make -j8" took 11 sec on my machine.
    >
    > real 0m11.777s
    > user 0m16.608s
    > sys 0m5.164s
    >
    >
    >
    > After this patch, x86_64_defconfig + CONFIG_IKHEADERS_PROC=y
    > takes 53 sec for me since kernel/kheaders_data.txz is regenerated
    > every time even when you did not touch any source file.

    Hmm, true. Let me know look into that as well..


    > $ time make -j8
    > DESCEND objtool
    > CALL scripts/checksyscalls.sh
    > CHK include/generated/compile.h
    > GEN kernel/kheaders_data.txz
    > UPD kernel/kheaders_data.h
    > CC kernel/kheaders.o
    > AR kernel/built-in.a
    > GEN .version
    > CHK include/generated/compile.h
    > UPD include/generated/compile.h
    > CC init/version.o
    > AR init/built-in.a
    > AR built-in.a
    > LD vmlinux.o
    > MODPOST vmlinux.o
    > KSYM .tmp_kallsyms1.o
    > KSYM .tmp_kallsyms2.o
    > LD vmlinux
    > SORTEX vmlinux
    > SYSMAP System.map
    > Building modules, stage 2.
    > CC arch/x86/boot/version.o
    > MODPOST 17 modules
    > VOFFSET arch/x86/boot/compressed/../voffset.h
    > OBJCOPY arch/x86/boot/compressed/vmlinux.bin
    > RELOCS arch/x86/boot/compressed/vmlinux.relocs
    > CC arch/x86/boot/compressed/kaslr.o
    > GZIP arch/x86/boot/compressed/vmlinux.bin.gz
    > CC arch/x86/boot/compressed/misc.o
    > MKPIGGY arch/x86/boot/compressed/piggy.S
    > AS arch/x86/boot/compressed/piggy.o
    > LD arch/x86/boot/compressed/vmlinux
    > ZOFFSET arch/x86/boot/zoffset.h
    > OBJCOPY arch/x86/boot/vmlinux.bin
    > AS arch/x86/boot/header.o
    > LD arch/x86/boot/setup.elf
    > OBJCOPY arch/x86/boot/setup.bin
    > BUILD arch/x86/boot/bzImage
    > Setup is 15612 bytes (padded to 15872 bytes).
    > System is 12673 kB
    > CRC 697aaf88
    > Kernel: arch/x86/boot/bzImage is ready (#6)
    >
    > real 0m53.024s
    > user 0m32.076s
    > sys 0m9.296s
    >
    >
    >
    >
    > Also, I notice $(ARCH) must be fixed to $(SRCARCH),
    > but that is one of minor issues.

    Ok.

    > We should take time for careful review and test.

    Yes, I agree.

    > Please give me more time for thorough review.

    Sure. Thanks a lot for the feedback provided so far. I will send another
    revision soon with all the suggestions addressed.

    thanks,

    - Joel

    \
     
     \ /
      Last update: 2019-02-19 05:35    [W:3.526 / U:0.872 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site