1# Android Build System Concepts 2 3This document provides high level explanations and mapping of the internal 4build system components and concepts of the Android build system and Bazel, 5and how components communicate with each other. 6 7## High level components 8 9This table provides a high level overview of the components in the current 10Android Platform build system, and how each component maps to a concept in 11Bazel. 12 13|Android build system component|Description|Mapping to Bazel concepts| 14|---|---|---| 15|Kati|Make-compatible front-end. Encodes build logic in `.mk` scripts. Declares buildable units in `Android.mk`. Generates Ninja file directly.|Loading and analysis phase. Conceptually similar to `bazel build --nobuild`.| 16|Blueprint|Build definition syntax. Build syntax parser. Internal data structures like Modules/Variations/Context/Scope. Ninja file generator.|Starlark.| 17|Soong|Bazel-like front-end. Encodes build logic in Go. Declares build units in `Android.bp`, parsed by Blueprint. Uses Blueprint to generate Ninja file. Generates a `.mk` file with prebuilt module stubs to Kati.|Loading and analysis phase. Conceptually similar to `bazel build --nobuild command`.| 18|Ninja|Serialized command line action graph executor. Executes Ninja graph generated from Kati and Soong.|Bazel's execution phase.| 19|atest|Test executor and orchestrator.|Conceptually similar to `bazel test` command.| 20|Blueprint + Kati + Soong + Ninja + atest|The entire build pipeline for Android.|Conceptually similar to `bazel build` or `bazel test` commands.| 21|`<script>.sh`|Running arbitrary scripts in AOSP.|Conceptually similar to `bazel run` command.| 22|Make (replaced in-place by Kati)|No longer in use. Entire build system, replaced by the tools above.|Loading, analysis, execution phases. Conceptually similar to `bazel build` command.| 23|`Android.bp`|Build definition file for Soong.|`BUILD.bazel` or `BUILD`.| 24|`Android.mk`|Build definition file for Kati.|`BUILD.bazel` or `BUILD`.| 25 26## Communication between components 27 28* Kati product configuration component to generate config variables (config.mk, AndroidProducts.mk) 29 * **“Kati-config” for the purpose of this document** 30* Kati component to generate build actions in Ninja files (main.mk, Android.mk files) 31 * **“Kati-build” for the purpose of this document** 32* Kati component to generate packaging actions in Ninja files (packaging.mk file) 33 * **“Kati-package” for the purpose of this document** 34* Kati component to generate cleaning actions in Ninja files (cleanbuild.mk, CleanSpec.mk files) 35 * **"Kati-cleanspec" for the purpose of this document** 36* **soong\_build** (and **Blueprint**) component to generate build actions (Android.bp, Blueprints files) 37* **Ninja** component to execute actions from Kati-build, Kati-package and soong\_build 38* **Bazel** as the next generation of the entire build system, starting as a Ninja executor drop-in replacement 39* **soong\_ui** as the tool to orchestrate all of the above, and with auxiliary tools like finder, path\_interposer, soong\_env, minibp and bpglob. 40 41The current build system architecture primarily uses **files** as the medium 42for inter-process communication (IPC), with one known case of unix socket 43communication (e.g. 44[path\_interposer](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/paths/logs.go;l=112-133;drc=184901135cda8bdcc51cab4f16c401a28a510593)), 45and a fifo between Ninja and soong_ui for the Protobuf stream for build 46status reporting. 47 48## Component order 49 50The build system components run in the following order, orchestrated by soong\_ui: 51 521. soong\_ui bootstraps itself with microfactory (`go build` replacement) and launches. 531. soong\_ui runs auxiliary tools to aggregate files into filelists, for 54Android.mk, Android.bp, AndroidProducts.mk and several others. 551. soong\_ui runs Kati-config with 56 [the config.mk entry point](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/dumpvars.go;l=89;drc=9f43597ff7349c4facd9e338e5b4b277e625e518). 571. soong\_ui orchestrates 3 Blueprint/Soong phases to generate the main out/soong/build.ninja file: 58 minibootstrap, bootstrap, and primary. 59 1. Minibootstrap phase uses Blueprint/Microfactory to build itself 60 (minibp) so that Android.bp and Blueprint files can be used to define 61 Soong. 62 1. Bootstrap phase runs Ninja on a build.ninja file that runs minibp to 63 read all Android.bp files across the source tree that describes Soong and 64 plugins, and builds soong\_build. 65 1. Primary phase runs Ninja on a build.ninja file that runs soong_build 66 to generate the final out/soong/build.ninja file. 67 1. soong\_build also runs its own tests alongside generating 68 out/soong/build.ninja, which can be skipped with the `--skip-soong-tests` 69 argument. 701. soong\_ui runs Kati-cleanspec with 71 [the cleanbuild.mk entry point](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/kati.go;l=362;drc=b1d30d63c5d1b818ea38e77cd155da2016fe8b6c). 721. soong\_ui runs Kati-build to generate a Ninja file, with 73 [the main.mk entry point.](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/kati.go;l=202;drc=b1d30d63c5d1b818ea38e77cd155da2016fe8b6c) 741. soong\_ui runs Kati-package to generate a Ninja file, with 75 [the packaging/main.mk](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/kati.go;l=314;drc=b1d30d63c5d1b818ea38e77cd155da2016fe8b6c) 76 entry point. 771. soong\_ui generates a Ninja file to combine above Ninja files. 781. soong\_ui runs either Ninja or Bazel to execute the build, with the 79combined Ninja file as entry point. 80 81soong\_ui has a --skip-make flag that will skip Kati-config, Kati cleanspec, 82Kati-build and Kati-package, used for Soong-only builds in NDK and some 83Mainline projects. 84 85### soong\_ui 86 87soong\_ui is primarily responsible for orchestrating the build, cleaning the 88build environment, and running auxiliary tools. These tools (minibp, 89microfactory) can bootstrap other tools (soong\_build), aggregate file lists 90(finder.go), improve hermeticity (path\_interposer, nsjail) or perform checks 91against the environment (soong\_env). 92 93soong\_ui uses finder.go to generate <filename>.list files for other 94tools. For example, it generates Android.mk.list for Kati-build, 95AndroidProducts.mk.list for Kati-config, and Android.bp.list for 96soong\_build. 97 98soong\_ui uses path\_interposer to prepare an hermetic $PATH with runtime 99checks against allowlisted system tools. The $PATH contains these system 100tools with checked-in prebuilts, and uses path\_interposer to intercept calls 101and error out whenever non-allowlisted tools are used (see out/.path for 102directory of intercepted tool symlinks). 103 104soong\_ui generates a Kati suffix to ensure that Kati-generated files are 105regenerated if inputs to Kati have changed between builds. 106 107soong\_ui calls Soong and Kati to generate Ninja files, and eventually 108creates another Ninja file (out/combined-<product>.ninja) to combine the 109others, and executes either Ninja or Bazel to complete the build. 110 111soong\_ui sets up the sandbox and environment for the Ninja/Bazel process. 112 113## Kati-config 114 115As a product configuration tool, soong\_ui runs Kati-config in 116**[--dumpvars-mode](https://cs.android.com/android/platform/superproject/+/master:build/soong/cmd/soong_ui/main.go;l=298-305;drc=master)** 117to dump the values of specified Make variables at the end of an evaluation, 118with build/make/core/config.mk as the entry point. During this phase, 119Kati-config eventually evaluates[ 120soong\_config.mk](https://cs.android.com/android/platform/superproject/+/master:build/make/core/soong_config.mk;l=2?q=soong.variables) 121to generate the **[soong.variables JSON 122file](https://cs.android.com/android/platform/superproject/+/master:build/make/core/soong_config.mk;l=16-222;drc=341928ecc5da205401bcfd86f098662b0cee7857)**. 123This way, Kati-config can communicate product configuration to soong\_build, 124as soong\_build parses the dumped variables from the JSON on startup, and 125stores them into an in-memory Config object. 126 127To communicate 128[dexpreopt](https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt.go;l=115;drc=8cbc5d269b20b5743679bfb8684ed174dcf58a30) 129variables to soong\_build, [dexpreopt.config is also 130generated](https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_config.mk;l=71-142;drc=f26015449f0747b9fdeceb5ce70e30ecd76e20e8) 131as a $(shell) action and [read by 132soong\_build](https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/config.go;l=175-196;drc=1af783fae74715bcf1a94733bd75b2e6cc688e8c) 133in a similar way as Kati-config evaluates dex\_preopt\_config.mk included in 134soong\_config.mk. 135 136soong\_ui sets up a **KatiReader** to monitor Kati-config’s stdout/err for UI 137reporting and error handling purposes. 138 139## soong\_build 140 141soong\_build’s primary role is to evaluate all Android.bp files, run a series 142of mutators, and generate out/soong/build.ninja file. 143 144soong\_build communicates with Kati-build by generating Make Vars and running 145the AndroidMk singleton to generate .mk files in the output directory 146(out/soong/{Android, late, make\_vars}-<product>.mk). 147 148* Android-<product>.mk contains Soong modules as Make modules so Make 149 modules can depend on Soong modules. 150* make\_vars-<product>.mk contains Make variables for Kati-build, exported 151 from Soong modules. There are also checks built into this .mk file to ensure that 152 if a duplicate Make variable of the same name comes from another source, the Soong 153 and Make variable values are identical. 154* late-<product>.mk contains Make variables that are not read while Kati-build 155 parses the Android.mk file. (Late variables) 156 * soong\_ui invokes Kati to parse make\_vars .mk file earlier than the Android.mk 157 files,and late.mk after parsing the Android.mk files. 158 * late.mk is used to define phony rules to take advantage of Make’s ability to 159 add extra dependencies to an existing rule. late.mk is not strictly necessary to 160 make this happen at this moment, since late.mk rules don’t currently depend on any 161 variables defined during Android.mk processing (e.g. ALL\_MODULES$(module).INSTALLED). 162 163## Kati-build / Kati-package 164 165Kati-build’s primary role is to evaluate all Android.mk files with 166build/make/core/main.mk as entry point, and generate 167out/build-<product>.ninja. It also generates cleanspec.ninja for the 168product, containing statements on how to remove stale output files. 169 170Kati-build’s primary role is to evaluate all packaging .mk files with 171build/make/packaging/main.mk as entry point, including 172build/make/packaging/distdir.mk for dist-for-goals calls, and generate 173out/package-<product>.ninja. 174 175Kati-build/Kati-package’s stdout/stderr is monitored by soong\_ui’s 176KatiReader to UI and error handling. 177 178As Kati-build/Kati-package generates Ninja files, they also generate 179out/ninja-<product>.sh and out/env-<product>.sh. These scripts are 180wrappers for soong\_ui to execute Ninja with the correct Ninja files, in a 181controlled environment. 182 183## Ninja 184 185As Ninja executes files from Kati-build, Kati-package, soong\_build and other 186bootstrapping tools like Blueprint, it writes to a fifo in a proto front end 187that soong\_ui monitors with NinjaReader. NinjaReader ensures that the user 188interface for Ninja progress is consistent with the rest of the build. 189 190## Bazel 191 192As more Soong modules are converted to BUILD files, soong\_build serializes 193information about converted modules to BUILD/bzl files on disk. soong\_build 194then consumes information about these targets from Bazel by directly calling 195the Bazel client to issue `cquery` calls about these targets. 196