# `run_bisect.py`

`run_bisect.py` is a wrapper around the general purpose
`binary_search_state.py`. It provides a user friendly interface for
bisecting various compilation errors.  The 2 currently provided
methods of bisecting are ChromeOS package and object bisection. Each
method defines a default set of options to pass to
`binary_search_state.py` and allow the user to override these defaults
(see the "Overriding" section).

Please note that all commands, examples, scripts, etc. are to be run from your
chroot unless stated otherwise.

## Bisection Methods

### ChromeOS Package

This method will bisect across all packages in a ChromeOS repository and find
the offending packages (according to your test script). This method takes the
following arguments:

* board: The board to bisect on. For example: daisy, falco, etc.
* remote: The IP address of the physical machine you're using to test with.

By default the ChromeOS package method will do a simple interactive test that
pings the machine and prompts the user if the machine is good.

1.  Setup: The ChromeOS package method requires that you have three build trees:

    ```
    /build/${board}.bad  - The build tree for your "bad" build
    /build/${board}.good - The build tree for your "good" build
    /build/${board}.work - A full copy of /build/${board}.bad
    ```

1.  Cleanup: run_bisect.py does most cleanup for you, the only thing required by
    the user is to cleanup all built images and the three build trees made in
    `/build/`

1.  Default Arguments:

    ```
    --get_initial_items='cros_pkg/get_initial_items.sh'
    --switch_to_good='cros_pkg/switch_to_good.sh'
    --switch_to_bad='cros_pkg/switch_to_bad.sh'
    --test_setup_script='cros_pkg/test_setup.sh'
    --test_script='cros_pkg/interactive_test.sh'
    --incremental
    --prune
    --file_args
    ```

1.  Additional Documentation: See `./cros_pkg/README.cros_pkg_triage` for full
    documentation of ChromeOS package bisection.

1.  Examples:

    1.  Basic interactive test package bisection, on daisy board:

        ```
        ./run_bisect.py package daisy 172.17.211.184
        ```

    2.  Basic boot test package bisection, on daisy board:

        ```
        ./run_bisect.py package daisy 172.17.211.184 -t cros_pkg/boot_test.sh
        ```

### ChromeOS Object

This method will bisect across all objects in a ChromeOS package and find
the offending objects (according to your test script). This method takes the
following arguments:

* board: The board to bisect on. For example: daisy, falco, etc.
* remote: The IP address of the physical machine you're using to test with.
* package: The package to bisect with. For example: chromeos-chrome
* dir: (Optional) the directory for your good/bad build trees. Defaults to
       $BISECT_DIR or /tmp/sysroot_bisect. This value will set $BISECT_DIR
       for all bisecting scripts.

By default the ChromeOS object method will do a simple interactive test that
pings the machine and prompts the user if the machine is good.

1.  Setup: The ChromeOS package method requires that you populate your good and
    bad set of objects. `sysroot_wrapper` will automatically detect the
    `BISECT_STAGE` variable and use this to populate emerged objects. Here is an
    example:

    ```
    # Defaults to /tmp/sysroot_bisect
    export BISECT_DIR="/path/to/where/you/want/to/store/builds/"

    export BISECT_STAGE="POPULATE_GOOD"
    ./switch_to_good_compiler.sh
    emerge-${board} -C ${package_to_bisect}
    emerge-${board} ${package_to_bisect}

    export BISECT_STAGE="POPULATE_BAD"
    ./switch_to_bad_compiler.sh
    emerge-${board} -C {package_to_bisect}
    emerge-${board} ${package_to_bisect}
    ```

1.  Cleanup: The user must clean up all built images and the populated object
    files.

1.  Default Arguments:

    ```
    --get_initial_items='sysroot_wrapper/get_initial_items.sh'
    --switch_to_good='sysroot_wrapper/switch_to_good.sh'
    --switch_to_bad='sysroot_wrapper/switch_to_bad.sh'
    --test_setup_script='sysroot_wrapper/test_setup.sh'
    --test_script='sysroot_wrapper/interactive_test.sh'
    --noincremental
    --prune
    --file_args
    ```

1.  Additional Documentation: See `./sysroot_wrapper/README` for full
    documentation of ChromeOS object file bisecting.

1.  Examples:

    1.  Basic interactive test object bisection, on daisy board for cryptohome
        package: `./run_bisect.py object daisy 172.17.211.184 cryptohome`

    2.  Basic boot test package bisection, on daisy board for cryptohome
        package: `./run_bisect.py object daisy 172.17.211.184 cryptohome
        --test_script=sysroot_wrapper/boot_test.sh`

### Android object

NOTE: Because this isn't a ChromeOS bisection tool, the concept of a
      chroot doesn't exist. Just run this tool from a normal shell.

This method will bisect across all objects in the Android source tree and
find the offending objects (according to your test script). This method takes
the following arguments:

*   `android_src`: The location of your android source tree

*   `num_jobs`: (Optional) The number of jobs to pass to make. This is dependent
    on how many cores your machine has. A good number is probably somewhere
    around 5 to 10.

*   `device_id`: (Optional) The serial code for the device you are testing on.
    This is used to determine which device should be used in case multiple
    devices are plugged into your computer. You can get serial code for your
    device by running "adb devices".

*   `dir`: (Optional) the directory for your good/bad build trees. Defaults to
    `$BISECT_DIR` or `~/ANDROID_BISECT/`. This value will set `$BISECT_DIR` for
    all bisecting scripts.

  By default the Android object method will do a simple interactive test that
  pings the machine and prompts the user if the machine is good.

1.  Setup: The Android object method requires that you populate your good and
    bad set of objects. The Android compiler wrapper will automatically detect
    the `BISECT_STAGE` variable and use this to populate emerged objects. Here
    is an example:

    ```
    # Defaults to ~/ANDROID_BISECT/
    export BISECT_DIR="/path/to/where/you/want/to/store/builds/"

    export BISECT_STAGE="POPULATE_GOOD"
    # Install the "good" compiler
    ./switch_to_good_compiler.sh
    make clean
    make -j <your_preferred_number_of_jobs>

    export BISECT_STAGE="POPULATE_BAD"
    # Install the "bad" compiler
    ./switch_to_bad_compiler.sh
    make clean
    make -j <your_preferred_number_of_jobs>
    ```

1.  Cleanup: The user must clean up all built images and the populated object
    files.

1.  Default Arguments:

    ```
    --get_initial_items='android/get_initial_items.sh'
    --switch_to_good='android/switch_to_good.sh'
    --switch_to_bad='android/switch_to_bad.sh'
    --test_setup_script='android/test_setup.sh'
    --test_script='android/interactive_test.sh'
    --incremental
    --prune
    --file_args
    ```

1.  Additional Documentation: See `./android/README.android` for full
    documentation of Android object file bisecting.

1.  Examples:

    1.  Basic interactive test android bisection, where the android source is at
        ~/android_src: `./run_bisect.py android ~/android_src`

    2. Basic boot test android bisection, where the android source is at
       `~/android_src`, and 10 jobs will be used to build android:
       `./run_bisect.py
       android ~/android_src --num_jobs=10
       --test_script=sysroot_wrapper/boot_test.sh`

### Resuming

`run_bisect.py` and `binary_search_state.py` offer the
ability to resume a bisection in case it was interrupted by a
SIGINT, power failure, etc. Every time the tool completes a
bisection iteration its state is saved to disk (usually to the file
`./bisect_driver.py.state`). If passed the --resume option, the tool
it will automatically detect the state file and resume from the last
completed iteration.

### Overriding

You can run `./run_bisect.py --help` or `./binary_search_state.py
--help` for a full list of arguments that can be overriden. Here are
a couple of examples:

Example 1 (do boot test instead of interactive test):

```
./run_bisect.py package daisy 172.17.211.182 --test_script=cros_pkg/boot_test.sh
```

Example 2 (do package bisector system test instead of interactive test, this
           is used to test the bisecting tool itself -- see comments in
           hash_test.sh for more details):

```
./run_bisect.py package daisy 172.17.211.182 \
    --test_script=common/hash_test.sh --test_setup_script=""
```

Example 3 (enable verbose mode, disable pruning, and disable verification):

```
./run_bisect.py package daisy 172.17.211.182
      --verbose --prune=False --verify=False
```

# Pass bisection

This document describes a feature for the bisection tool, which provides
pass and transformation level bisection for a bad object file.

Before reading this document, please refer to README.bisect for general usage
of the bisection tool.

The benefit of using pass level bisection is:
When building a bad object file, it can tell you which pass and transformation
in the compiler caused the error.

*Notice:* This tool will only work for LLVM/clang, since it is using options
`-opt-bisect-limit` and `print-debug-counter` that only exist in LLVM.

## Arguments

All the required arguments in object-file-level bisection tool are still
to be provided. In addition, you will need to add the following arguments:

1. `--pass_bisect`: enables pass level bisection
2. `--ir_diff`: enables output of IR differences

Please refer to `--help` or the examples below for details about how to use
them.

## HOW TO USE: ChromeOS

*TODO* - Future work: Currently this only works for Android.

## HOW TO USE: Android

1.  Prerequisites: A general setup is still needed for Android, which means that
    you need to populate good and bad set of objects with two versions of
    compilers.

    See the documentation in `README.bisect.md` for more detailed instructions.

1.  Pass/Transformation Bisection: If you do not wish to override the other
    arguments, this command should be sufficient to do pass/transformation level
    bisection:

    ```
    ./run_bisect.py android PATH_TO_ANDROID_HOME_DIR
                --pass_bisect=’android/generate_cmd.sh’
                --prune=False
                --ir_diff
                --verbose
    ```

    Where:

    ```
    --pass_bisect:
        Enables pass/transformation level bisection and with default
        script to generate the command as ‘android/generate_cmd.sh’.
    --prune:
        For now, prune must be set to False to return only the first
        bad item.
    --ir_diff:
        Optional argument to print out IR differences.
    --verbose:
        To show IR diff, verbose needs to be on.
    ```

    Other default arguments:

    ```
    --get_initial_items='android/get_initial_items.sh'
    --switch_to_good='android/switch_to_good.sh'
    --switch_to_bad='android/switch_to_bad.sh'
    --test_setup_script='android/test_setup.sh'
    --test_script='android/interactive_test.sh'
    --incremental
    --prune
    --file_args
    ```

    You can always override them if needed. See README.bisect for more
    details.

1.  Other features: Features such as resuming, number of jobs, and device id
    remain the same as before. See README.bisect for more details.

# Testing the binary search tool

This file explains how to set up and run the various kinds of bisection tests.

The bisection tool comes with several sets of tests which you should
run after updating any of the bisection tool scripts OR after updating
the Android compiler wrapper (to make sure the wrapper will still work
correctly with bisection).

## Before you start.

Before you can run the tests, your PYTHONPATH environment variable
must be correct.  This means that it must include both the
toolchain-utils directory and the `binary_search_tool` directory.  The
easiest way to set it is:

```
$ cd toolchain-utils
$ export PYTHONPATH=`pwd`:${PYTHONPATH}
$ cd binary_search_tool
$ export PYTHONPATH=`pwd`:${PYTHONPATH}
```


## Running the unittests.

To run the basic unit tests:

```
$ cd toolchain-utils/binary_search_tool/test
$ ./binary_search_tool_test.py
```

# Running the bisection tests, testing the compiler wrapper.

If you want to run the bisection tests, and test the compiler wrapper
(to make sure the `POPULATE_GOOD` and `POPULATE_BAD` stages are still
working properly) you can do the following.

If you are testing with the ANDROID COMPILER WRAPPER, you need to to some
preliminary setup:

Set up the compiler wrapper to replace GCC:

```
$ cd <android-root/prebuilts/clang/host/linux-x86/clang-368880/bin
$ cp clang gcc
$ whereis gcc
gcc: /usr/bin/gcc /usr/lib/gcc /usr/bin/X11/gcc /usr/share/man/man1/gcc.1.gz
$ cd /usr/bin
$ ls -l gcc
lrwxrwxrwx 1 root root 7 Feb  3 17:00 gcc -> gcc-4.8*
$ sudo mv gcc gcc.real
$ sudo ln -s \
    <android-root>/prebuilts/clang/host/linux-x86/clang-3688880/bin/gcc gcc
```

Move to the correct directory, then run the test script:

```
$ cd toolchain-utils/binary_search_tool
$ ./run_bisect_tests.py
```

If you are testing with the CHROMEOS COMPILER WRAPPER, you MUST run the
tests from INSIDE your CHROOT (but you don't need to do any special setup):

```
$ cd <path-to-chromeos-root>
$ cros_sdk
$ cd ~/trunk/src/third_party/toolchain-utils
```

Set up your `PYTHONPATH`:

```
$ export PYTHONPATH=`pwd`:${PYTHONPATH}
$ cd binary_search_tool
$ export PYTHONPATH=`pwd`:${PYTHONPATH}
```

Run the test script:

```
$ ./run_bisect_tests.py
```


## Running the bisection tests, without testing the compiler wrapper.

```
$ cd toolchain-utils/binary_search_tool
$ ./full_bisect_test/run-test-nowrapper.sh
```