lkml.org 
[lkml]   [2023]   [Feb]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    From
    Date
    SubjectRe: [PATCH v6 01/12] kbuild: add a tool to list files ignored by git
    On Wed, Feb 15, 2023 at 10:21 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
    >
    > In short, the motivation of this commit is to build a source package
    > without cleaning the source tree.
    >
    > The deb-pkg and (src)rpm-pkg targets first run 'make clean' before
    > creating a source tarball. Otherwise build artifacts such as *.o,
    > *.a, etc. would be included in the tarball. Yet, the tarball ends up
    > containing several garbage files since 'make clean' does not clean
    > everything.
    >
    > Cleaning the tree every time is annoying since it makes the incremental
    > build impossible. It is desirable to create a source tarball without
    > cleaning the tree.
    >
    > In fact, there are some ways to achieve this.
    >
    > The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses
    > it, but I do not like it because it works only when the source tree is
    > managed by git, and all files you want in the tarball must be committed
    > in advance.
    >
    > I want to make it work without relying on git. We can do this.
    >
    > Files that are ignored by git are generated files, so should be excluded
    > from the source tarball. We can list them out by parsing the .gitignore
    > files. Of course, .gitignore does not cover all the cases, but it works
    > well enough.
    >
    > tar(1) claims to support it:
    >
    > --exclude-vcs-ignores
    >
    > Exclude files that match patterns read from VCS-specific ignore files.
    > Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.
    >
    > The best scenario would be to use 'tar --exclude-vcs-ignores', but this
    > option does not work. --exclude-vcs-ignore does not understand any of
    > the negation (!), preceding slash, following slash, etc.. So, this option
    > is just useless.
    >
    > Hence, I wrote this gitignore parser. The previous version [1], written
    > in Python, was so slow. This version is implemented in C, so it works
    > much faster.
    >
    > I imported the code from git (commit: 23c56f7bd5f1), so we get the same
    > result.
    >
    > This tool traverses the source tree, parsing all .gitignore files, and
    > prints file paths that are ignored by git.
    >
    > The output is similar to 'git ls-files --ignored --directory --others
    > --exclude-per-directory=.gitignore', except
    >
    > [1] Not sorted
    > [2] No trailing slash for directories
    >
    > [2] is intentional because tar's --exclude-from option cannot handle
    > trailing slashes.
    >
    > [How to test this tool]
    >
    > $ git clean -dfx
    > $ make -s -j$(nproc) defconfig all # or allmodconifg or whatever
    > $ git archive -o ../linux1.tar --prefix=./ HEAD
    > $ tar tf ../linux1.tar | LANG=C sort > ../file-list1 # files emitted by 'git archive'
    > $ make scripts_package
    > HOSTCC scripts/list-gitignored
    > $ scripts/list-gitignored --prefix=./ -o ../exclude-list
    > $ tar cf ../linux2.tar --exclude-from=../exclude-list .
    > $ tar tf ../linux2.tar | LANG=C sort > ../file-list2 # files emitted by 'tar'
    > $ diff ../file-list1 ../file-list2 | grep -E '^(<|>)'
    > < ./Documentation/devicetree/bindings/.yamllint
    > < ./drivers/clk/.kunitconfig
    > < ./drivers/gpu/drm/tests/.kunitconfig
    > < ./drivers/hid/.kunitconfig
    > < ./fs/ext4/.kunitconfig
    > < ./fs/fat/.kunitconfig
    > < ./kernel/kcsan/.kunitconfig
    > < ./lib/kunit/.kunitconfig
    > < ./mm/kfence/.kunitconfig
    > < ./tools/testing/selftests/arm64/tags/
    > < ./tools/testing/selftests/arm64/tags/.gitignore
    > < ./tools/testing/selftests/arm64/tags/Makefile
    > < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
    > < ./tools/testing/selftests/arm64/tags/tags_test.c
    > < ./tools/testing/selftests/kvm/.gitignore
    > < ./tools/testing/selftests/kvm/Makefile
    > < ./tools/testing/selftests/kvm/config
    > < ./tools/testing/selftests/kvm/settings
    >
    > The source tarball contains most of files that are tracked by git. You
    > see some diffs, but it is just because some .gitignore files are wrong.
    >
    > $ git ls-files -i -c --exclude-per-directory=.gitignore
    > Documentation/devicetree/bindings/.yamllint
    > drivers/clk/.kunitconfig
    > drivers/gpu/drm/tests/.kunitconfig
    > drivers/hid/.kunitconfig
    > fs/ext4/.kunitconfig
    > fs/fat/.kunitconfig
    > kernel/kcsan/.kunitconfig
    > lib/kunit/.kunitconfig
    > mm/kfence/.kunitconfig
    > tools/testing/selftests/arm64/tags/.gitignore
    > tools/testing/selftests/arm64/tags/Makefile
    > tools/testing/selftests/arm64/tags/run_tags_test.sh
    > tools/testing/selftests/arm64/tags/tags_test.c
    > tools/testing/selftests/kvm/.gitignore
    > tools/testing/selftests/kvm/Makefile
    > tools/testing/selftests/kvm/config
    > tools/testing/selftests/kvm/settings
    >
    > [1]: https://lore.kernel.org/all/20230128173843.765212-1-masahiroy@kernel.org/
    >
    > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
    > ---


    01 - 11 applied to linux-kbuild.


    I will update 12/12 (postponed to v6.4)



    --
    Best Regards
    Masahiro Yamada

    \
     
     \ /
      Last update: 2023-03-27 00:35    [W:4.566 / U:0.544 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site