This note aims to demonstrate automatic git bisect using the Linux kernel as an example. With the subsequent search for the official version starting with which everything broke and the last good version.

Toolkit Description

The toolkit [^ 7] is a simple project for assembling the kernel and modules with a minimum config sufficient to run in qemu, minimal busybox, configuration files and a small number of scripts.

• gcc;)
• binutils
• make
• qemu
• expect (only if you were going to run bisect from the example)

The kernel, along with the initramsfs image, is launched using qemu :

$ qemu-system-x86_64 -cpu host \ -kernel build-linux/arch/x86/boot/bzImage \ -initrd initramfs.cpio.xz \ -nographic -append "nokaslr console=ttyS0 root=/dev/ram" \ -enable-kvm -serial mon:stdio
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

There is no password and we are immediately thrown into the console.

With this project a simple bisect [^ 1] can be carried out.

Attention! If you plan to repeat the manipulations presented below, or will be used as the basis for your project, keep in mind:

1. the kernel is heavily dependent on the version of gcc and binutils , some kernels
   
   can only get together with a specific version of gcc, can get together with
   
   small edits or suppression of errors or not to collect at all
2. with the new version of binutils (starting from version 2.31) there may be problems with
   
   loading modules on kernel versions prior to v4.16-rc3

The versions of gcc 7.3.0 and binutils 2.30 let me build and

Run kernel versions v4.14 through v5.3-rc2.

Formulation of the problem

Actually, an additional problem is presented here, solved within the framework of the main one, which it was decided to take as a basis for an example.

At some point, the path in debugfs changed its value from that originally implemented in version v4.14 :

  # ls /sys/kernel/debug/ gpio-mockup-event # ls /sys/kernel/debug/gpio-mockup-event gpio-mockup-A
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

On (seen on v5.3-rc2 ):

  # ls /sys/kernel/debug/ gpio-mockup # ls /sys/kernel/debug/gpio-mockup gpiochip1
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

What broke my tests for my program, and the task is to find when, where and who, including determining the “good” and “bad” official versions of the kernel.

Solution with git bisect run

If you repeat this experiment, do not forget to do:

 $ git submodule update --init
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

run bisect.sh through git bisect run , the script itself is very simple and consists of three actions:

1. clear all

   
   
   
   
   
   
   

2. collect everything

   
   
   
   
   
   
   

3. run tests / bisect.expect (script for expect)

   
   
   
   
   
   
   

The test starts qemu, waits for promt, loads the gpio-mockup module and checks for the presence of directories in / sys / debug / kernel .

We start the process (it is the same as for manual bisect with the exception of the last step):

  $ git -C linux bisect start $ git -C linux bisect good v4.14 #     $ git -C linux bisect bad v5.3-rc2 #     Bisecting: 73727 revisions left to test after this (roughly 16 steps) [798bba01b44b0ddf8cd6e542635b37cc9a9b739c] RDMA/core: Fail early if unsupported QP is provided
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Run git bisect run :

  $ time git -C linux bisect run ../bisect.sh #    
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

We are waiting ... We are waiting ... We are waiting ... In general, despite the attempt to save assembly time, I have one assembly:

  $ time ../bisec.sh real 2m1.695s user 11m7.409s sys 2m0.751s
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

According to preliminary estimates, git bisect should handle 16 steps.

And finally, the result:

  d51ee07a8de7d6d3f7738a5e74861133fd2d46a0 is the first bad commit commit d51ee07a8de7d6d3f7738a5e74861133fd2d46a0 Author: Bartosz Golaszewski <bgolaszewski@baylibre.com> Date: Thu Jan 17 15:04:23 2019 +0100 gpio: mockup: don't create the debugfs link named after the label User-space tests no longer use it and we're breaking the interface anyway. Signed-off-by: Bartosz Golaszewski <bgolaszewski@baylibre.com> :040000 040000 c1a1873f4cfcecace123b72fb036c3861151c9b9 61917a273f4f1f078639463a29acb8a103d50b41 M drivers bisect run success real 42m6.873s user 192m39.291s sys 33m55.932s
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Well, the result gave us the number of the commit in which everything was gone:

d51ee07a8de7d6d3f7738a5e74861133fd2d46a0 .

Now you can look at the list of all the steps done by git bisect :

  $ git bisect log git bisect start # good: [bebc6082da0a9f5d47a1ea2edc099bf671058bd4] Linux 4.14 git bisect good bebc6082da0a9f5d47a1ea2edc099bf671058bd4 # bad: [609488bc979f99f805f34e9a32c1e3b71179d10b] Linux 5.3-rc2 git bisect bad 609488bc979f99f805f34e9a32c1e3b71179d10b # good: [798bba01b44b0ddf8cd6e542635b37cc9a9b739c] RDMA/core: Fail early if unsupported QP is provided git bisect good 798bba01b44b0ddf8cd6e542635b37cc9a9b739c # good: [e266ca36da7de45b64b05698e98e04b578a88888] Merge tag 'staging-5.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging git bisect good e266ca36da7de45b64b05698e98e04b578a88888 # bad: [318222a35bfb0ae9b5ff3e359a583463e6cfcd94] Merge branch 'akpm' (patches from Andrew) git bisect bad 318222a35bfb0ae9b5ff3e359a583463e6cfcd94 # bad: [962d5ecca101e65175a8cdb1b91da8e1b8434d96] Merge tag 'regmap-v5.2' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap git bisect bad 962d5ecca101e65175a8cdb1b91da8e1b8434d96 # bad: [f47d633134f7033e3d0c667419d9f8afd69e308d] Merge tag 'tag-chrome-platform-for-v5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux git bisect bad f47d633134f7033e3d0c667419d9f8afd69e308d # good: [6c3f98faddc7f07981c5365ba2f45905ad75fcaa] Merge branch 'i2c/for-5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux git bisect good 6c3f98faddc7f07981c5365ba2f45905ad75fcaa # bad: [2901752c14b8e1b7dd898d2e5245c93e531aa624] Merge tag 'pci-v5.1-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci git bisect bad 2901752c14b8e1b7dd898d2e5245c93e531aa624 # bad: [1a29e857507046e413ca7a4a7c9cd32fed9ea255] Merge tag 'docs-5.1' of git://git.lwn.net/linux git bisect bad 1a29e857507046e413ca7a4a7c9cd32fed9ea255 # bad: [3601fe43e8164f67a8de3de8e988bfcb3a94af46] Merge tag 'gpio-v5.1-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-gpio git bisect bad 3601fe43e8164f67a8de3de8e988bfcb3a94af46 # good: [cf2e8c544cd3b33e9e403b7b72404c221bf888d1] Merge tag 'mfd-next-5.1' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/mfd git bisect good cf2e8c544cd3b33e9e403b7b72404c221bf888d1 # good: [8fab3d713ca36bf4ad4dadec0bf38f5e70b8999d] Merge tag 'gpio-v5.1-updates-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux into devel git bisect good 8fab3d713ca36bf4ad4dadec0bf38f5e70b8999d # bad: [9aac1e336c3ab3824f646224f4b2309b63c51668] Documentation: gpio: legacy: Don't use POLLERR for poll(2) git bisect bad 9aac1e336c3ab3824f646224f4b2309b63c51668 # good: [0248baca03b8f188eccbb991bda2caec4c330975] Merge tag 'intel-gpio-v5.1-1' of git://git.kernel.org/pub/scm/linux/kernel/git/andy/linux-gpio-intel into devel git bisect good 0248baca03b8f188eccbb991bda2caec4c330975 # bad: [e09313ce7ea1706d1642c7d5af103915e69fc6d0] gpio: mockup: change the signature of unlocked get/set helpers git bisect bad e09313ce7ea1706d1642c7d5af103915e69fc6d0 # good: [cbf1e092f2d86e6d7cdb7f9ff8a333f52c826232] gpio: mockup: implement get_multiple() git bisect good cbf1e092f2d86e6d7cdb7f9ff8a333f52c826232 # bad: [83336668b94eb44ecd78a0b7840e43f0859e05cb] gpio: mockup: change the type of 'offset' to unsigned int git bisect bad 83336668b94eb44ecd78a0b7840e43f0859e05cb # bad: [d51ee07a8de7d6d3f7738a5e74861133fd2d46a0] gpio: mockup: don't create the debugfs link named after the label git bisect bad d51ee07a8de7d6d3f7738a5e74861133fd2d46a0 # first bad commit: [d51ee07a8de7d6d3f7738a5e74861133fd2d46a0] gpio: mockup: don't create the debugfs link named after the label
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In fact, this is the failure of the first test (yes - there are two of them in the test):

  send "ls /sys/kernel/debug/\r" expect { "gpio-mockup-event" {} timeout { puts "gpio-mockup-event not found"; exit 1 } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Another check breaks with the commit 2a9e27408e12de455b9fcf66b5d0166f2129579e (of course it would be to separate them but lazy, so I just looked at the commits << next >>):

  send "ls /sys/kernel/debug/gpio-mockup-event/\r" expect { "gpio-mockup-A" { puts "gpio-mockup-A found" } timeout { puts "gpio-mockup-A not found"; exit 1 } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Well, let's find out when the d51ee07a8de7 commit got into the main branch to Linus and with what official kernel version it is present [^ 6].

Let's see all commits up to d51ee07a8de7 filtering only merge commits [^ 3] (merge commits) and having a direct path [^ 4] (ancestry chain):

  $ git log --pretty=oneline d51ee07a8de7d6d3f7738a5e74861133fd2d46a0..master --ancestry-path --merges
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

This gives us all merge commits between d51ee07a8de7 and master. Let's look at the end of the list (only the last three entries are shown):

  3601fe43e8164f67a8de3de8e988bfcb3a94af46 Merge tag 'gpio-v5.1-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-gpio 3dda927fdbaac926c50b550ccb51ed18c184468b Merge branch 'ib-qcom-ssbi' into devel 2f7db3c70fdfb22480a1b0aa734664fc256532f2 Merge tag 'gpio-v5.1-updates-for-linus-part-2' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux into devel
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

As we see the last commit, this is the absorption of a branch from git: //git.kernel.org/pub/scm/linux/kernel/git/brgl/linux in git: //git.kernel.org/pub/scm/linux/kernel / git / linusw / linux-gpio devel, and the first one on the list is the takeover of the branch of Comrade Linus Walleij (<< chief >> of the GPIO subsystem) by Comrade Linus Torvalds in the branch of master.

There is a very good script [^ 2], which immediately leads to the result without manual work:

  #       3601fe43e816    $ git-find-merge d51ee07a8de7d6d3f7738a5e74861133fd2d46a0 master
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Find the first version after commit 3601fe43e816 :

  $ git name-rev --name-only 3601fe43e816 tags/v5.1-rc1~102
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The number 102 here is the distance from 3601fe43e816 to v5.1-rc1 , let's check it using the first-parent [^ 5] option:

  $ git -P log --pretty --oneline --first-parent \ --graph 3601fe43e816..v5.1-rc1 | wc -l 102
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Everything seems to be in order. I can say that the first official version of the kernel in which everything << broke >> v5.1-rc1 , and in version v5.0 everything was fine:

  $ git describe 3601fe43e816 v5.0-8748-g3601fe43e816
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Conclusion

It's no secret that doing such things automatically is much more fun. It is very difficult to do such a number of steps manually, given the long assembly time, you can simply forget what you were doing.

Actually, with the requirements formulated, you can perform any manipulations, make a special program and add to initramfs , or check something of your own and have bisect on hand.

Tests can be in any convenient form while they fulfill the requirements of git bisect run . Moreover, with the help of application programs (for example, the same expect), you can limit the firmware of the board with the architecture you need and perform checks directly on it.

[^ 1]: Christian Couder. Fully automated bisecting with "git bisect run" .

[^ 2]: rmandvikar. Git hooks system (global, local hooks), utility shell scripts, configuration for HOME dir .

[^ 3]: Scott Chacon and Ben Straub. Pro Git book .

[^ 4]: void.pointer. How does ancestry path work with git log? .

[^ 5]: Marc G Gauthier. Git Log's –first-parent Option .

[^ 6]: Guillaume Morin. Find merge commit which include a specific commit .

[^ 7]: Linux kernel git bisect template

Original image used in the header: https://xkcd.com/1597/ .