1# Regenerating project files 2 3Prerequisites: `python`, `pip install mako`, `go` 4 5``` 6# Regenerate the projects files using templates 7tools/buildgen/generate_projects.sh 8``` 9 10# Quick justification 11 12We've approached the problem of the build system from a lot of different 13angles. The main issue was that there isn't a single build system that 14was going to single handedly cover all of our usage cases. 15 16So instead we decided to work the following way: 17 18* A `build.yaml` file at the root is the source of truth for listing all the 19targets and files needed to build grpc and its tests, as well as a basic system 20for dependency description. 21 22* Each project file (Makefile, Visual Studio project files, Bazel's BUILD) is 23a [YAML](http://yaml.org) file used by the `build.yaml` file to generate the 24final output file. 25 26This way we can maintain as many project system as we see fit, without having 27to manually maintain them when we add or remove new code to the repository. 28Only the structure of the project file is relevant to the template. The actual 29list of source code and targets isn't. 30 31We currently have template files for GNU Make, Visual Studio 2013, 32[Bazel](http://bazel.io) and [gyp](https://gyp.gsrc.io/) (albeit only for 33Node.js). In the future, we 34would like to expand to also generate [cmake](https://cmake.org) 35project files, XCode project files, and an Android.mk file allowing to compile 36gRPC using Android's NDK. 37 38We'll gladly accept contribution that'd create additional project files 39using that system. 40 41# Structure of `build.yaml` 42 43The `build.yaml` file has the following structure: 44 45``` 46settings: # global settings, such as version number 47 ... 48filegroups: # groups of files that are automatically expanded 49 ... 50libs: # list of libraries to build 51 ... 52target: # list of targets to build 53 ... 54``` 55 56The `filegroups` are helpful to re-use a subset of files in multiple targets. 57One `filegroups` entry has the following structure: 58 59``` 60- name: "arbitrary string", # the name of the filegroup 61 public_headers: # list of public headers defined in that filegroup 62 - ... 63 headers: # list of headers defined in that filegroup 64 - ... 65 src: # list of source files defined in that filegroup 66 - ... 67``` 68 69The `libs` collection contains the list of all the libraries we describe. Some may be 70helper libraries for the tests. Some may be installable libraries. Some may be 71helper libraries for installable binaries. 72 73The `targets` array contains the list of all the binary targets we describe. Some may 74be installable binaries. 75 76One `libs` or `targets` entry has the following structure (see below for 77details): 78 79``` 80name: "arbitrary string", # the name of the library 81build: "build type", # in which situation we want that library to be 82 # built and potentially installed (see below). 83language: "...", # the language tag; "c" or "c++" 84public_headers: # list of public headers to install 85headers: # list of headers used by that target 86src: # list of files to compile 87secure: boolean, # see below 88baselib: boolean, # this is a low level library that has system 89 # dependencies 90vs_project_guid: '{...}', # Visual Studio's unique guid for that project 91filegroups: # list of filegroups to merge to that project 92 # note that this will be expanded automatically 93deps: # list of libraries this target depends on 94deps_linkage: "..." # "static" or "dynamic". Used by the Makefile only to 95 # determine the way dependencies are linkned. Defaults 96 # to "dynamic". 97dll: "..." # see below. 98dll_def: "..." # Visual Studio's dll definition file. 99vs_props: # List of property sheets to attach to that project. 100vs_config_type: "..." # DynamicLibrary/StaticLibrary. Used only when 101 # creating a library. Specifies if we're building a 102 # static library or a dll. Use in conjunction with `dll_def`. 103vs_packages: # List of nuget packages this project depends on. 104``` 105 106## The `"build"` tag 107 108Currently, the "`build`" tag have these meanings: 109 110* `"all"`: library to build on `"make all"`, and install on the system. 111* `"protoc"`: a protoc plugin to build on `"make all"` and install on the system. 112* `"private"`: a library to only build for tests. 113* `"test"`: a test binary to run on `"make test"`. 114* `"tool"`: a binary to be built upon `"make tools"`. 115 116All of the targets should always be present in the generated project file, if 117possible and applicable. But the build tag is what should group the targets 118together in a single build command. 119 120 121## The `"secure"` tag 122 123This means this target requires OpenSSL one way or another. The values can be 124`"yes"`, `"no"` and `"check"`. The default value is `"check"`. It means that 125the target requires OpenSSL, but that since the target depends on another one 126that is supposed to also import OpenSSL, the import should then be implicitely 127transitive. `"check"` should then only disable that target if OpenSSL hasn't 128been found or is unavailable. 129 130## The `"baselib"` boolean 131 132This means this is a library that will provide most of the features for gRPC. 133In particular, if we're locally building OpenSSL, protobuf or zlib, then we 134should merge OpenSSL, protobuf or zlib inside that library. That effect depends 135on the `"language"` tag. OpenSSL and zlib are for `"c"` libraries, while 136protobuf is for `"c++"` ones. 137 138## The `"dll"` tag 139 140Used only by Visual Studio's project files. "true" means the project will be 141built with both static and dynamic runtimes. "false" means it'll only be built 142with static runtime. "only" means it'll only be built with the dll runtime. 143 144## The `"dll_def"` tag 145 146Specifies the visual studio's dll definition file. When creating a DLL, you 147sometimes (not always) need a def file (see grpc.def). 148 149 150# The template system 151 152We're currently using the [mako templates](http://www.makotemplates.org/) 153renderer. That choice enables us to simply render text files without dragging 154with us a lot of other features. Feel free to explore the current templates 155in that directory. The simplest one is probably [BUILD.template](BUILD.template) 156which is used to create the [Bazel](http://bazel.io/) project file. 157 158## The renderer engine 159 160As mentioned, the renderer is using [mako templates](http://www.makotemplates.org/), 161but some glue is needed to process all of that. See the [buildgen folder](../tools/buildgen) 162for more details. We're mainly loading the build.json file, and massaging it, 163in order to get the list of properties we need, into a Python dictionary, that 164is then passed to the template while rending it. 165 166## The plugins 167 168The file build.json itself isn't passed straight to the template files. It is 169first processed and modified by a few plugins. For example, the `filegroups` 170expander is [a plugin](../tools/buildgen/plugins/expand_filegroups.py). 171 172The structure of a plugin is simple. The plugin must defined the function 173`mako_plugin` that takes a Python dictionary. That dictionary represents the 174current state of the build.json contents. The plugin can alter it to whatever 175feature it needs to add. 176