1.. _module-pw_env_setup:
2
3------------
4pw_env_setup
5------------
6A classic problem in the embedded space is reducing the time from git clone
7to having a binary executing on a device. The issue is that an entire suite
8of tools is needed for non-trivial production embedded projects. For example:
9
10 - A C++ compiler for your target device, and also for your host
11 - A build system or three; for example, GN, Ninja, CMake, Bazel
12 - A code formatting program like clang-format
13 - A debugger like OpenOCD to flash and debug your embedded device
14 - A known Python version with known modules installed for scripting
15 - A Go compiler for the Go-based command line tools
16
17...and so on
18
19In the server space, container solutions like Docker or Podman solve this;
20however, in our experience container solutions are a mixed bag for embedded
21systems development where one frequently needs access to native system
22resources like USB devices, or must operate on Windows.
23
24``pw_env_setup`` is our compromise solution for this problem that works on Mac,
25Windows, and Linux. It leverages the Chrome packaging system `CIPD`_ to
26bootstrap a Python installation, which in turn inflates a virtual
27environment. The tooling is installed into your workspace, and makes no
28changes to your system. This tooling is designed to be reused by any
29project.
30
31.. _CIPD: https://github.com/luci/luci-go/tree/master/cipd
32
33Users interact with  ``pw_env_setup`` with two commands: ``. bootstrap.sh`` and
34``. activate.sh``. The bootstrap command always pulls down the current versions
35of CIPD packages and sets up the Python virtual environment. The activate
36command reinitializes a previously configured environment, and if none is found,
37runs bootstrap.
38
39.. note::
40  On Windows the scripts used to set up the environment are ``bootstrap.bat``
41  and ``activate.bat``. For simplicity they will be referred to with the ``.sh``
42  endings unless the distinction is relevant.
43
44.. warning::
45  At this time ``pw_env_setup`` works for us, but isn’t well tested. We don’t
46  suggest relying on it just yet. However, we are interested in experience
47  reports; if you give it a try, please `send us a note`_ about your
48  experience.
49
50.. _send us a note: pigweed@googlegroups.com
51
52==================================
53Using pw_env_setup in your project
54==================================
55
56Downstream Projects Using Pigweed's Packages
57********************************************
58
59Projects using Pigweed can leverage ``pw_env_setup`` to install Pigweed's
60dependencies or their own dependencies. Projects that only want to use Pigweed's
61dependencies without modifying them can just source Pigweed's ``bootstrap.sh``
62and ``activate.sh`` scripts.
63
64An example of what your project's `bootstrap.sh` could look like is below. This
65assumes `bootstrap.sh` is at the top level of your repository.
66
67.. code-block:: bash
68
69  # Do not include a "#!" line, this must be sourced and not executed.
70
71  # This assumes the user is sourcing this file from it's parent directory. See
72  # below for a more flexible way to handle this.
73  PROJ_SETUP_SCRIPT_PATH="$(pwd)/bootstrap.sh"
74
75  export PW_PROJECT_ROOT="$(_python_abspath "$(dirname "$PROJ_SETUP_SCRIPT_PATH")")"
76
77  # You may wish to check if the user is attempting to execute this script
78  # instead of sourcing it. See below for an example of how to handle that
79  # situation.
80
81  # Source Pigweed's bootstrap utility script.
82  # Using '.' instead of 'source' for POSIX compatibility. Since users don't use
83  # dash directly, using 'source' in most documentation so users don't get
84  # confused and try to `./bootstrap.sh`.
85  . "$PW_PROJECT_ROOT/third_party/pigweed/pw_env_setup/util.sh"
86
87  pw_check_root "$PW_ROOT"
88  _PW_ACTUAL_ENVIRONMENT_ROOT="$(pw_get_env_root)"
89  export _PW_ACTUAL_ENVIRONMENT_ROOT
90  SETUP_SH="$_PW_ACTUAL_ENVIRONMENT_ROOT/activate.sh"
91  pw_bootstrap --args...  # See below for details about args.
92  pw_finalize bootstrap "$SETUP_SH"
93
94User-Friendliness
95-----------------
96
97You may wish to allow sourcing `bootstrap.sh` from a different directory. In
98that case you'll need the following at the top of `bootstrap.sh`.
99
100.. code-block:: bash
101
102  _python_abspath () {
103    python -c "import os.path; print(os.path.abspath('$@'))"
104  }
105
106  # Use this code from Pigweed's bootstrap to find the path to this script when
107  # sourced. This should work with common shells. PW_CHECKOUT_ROOT is only used in
108  # presubmit tests with strange setups, and can be omitted if you're not using
109  # Pigweed's automated testing infrastructure.
110  if test -n "$PW_CHECKOUT_ROOT"; then
111    PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$PW_CHECKOUT_ROOT/bootstrap.sh")"
112    unset PW_CHECKOUT_ROOT
113  # Shell: bash.
114  elif test -n "$BASH"; then
115    PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$BASH_SOURCE")"
116  # Shell: zsh.
117  elif test -n "$ZSH_NAME"; then
118    PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "${(%):-%N}")"
119  # Shell: dash.
120  elif test ${0##*/} = dash; then
121    PROJ_SETUP_SCRIPT_PATH="$(_python_abspath \
122      "$(lsof -p $$ -Fn0 | tail -1 | sed 's#^[^/]*##;')")"
123  # If everything else fails, try $0. It could work.
124  else
125    PROJ_SETUP_SCRIPT_PATH="$(_python_abspath "$0")"
126  fi
127
128You may also wish to check if the user is attempting to execute `bootstrap.sh`
129instead of sourcing it. Executing `bootstrap.sh` would download everything
130required for the environment, but cannot modify the environment of the parent
131process. To check for this add the following.
132
133.. code-block:: bash
134
135  # Check if this file is being executed or sourced.
136  _pw_sourced=0
137  # If not running in Pigweed's automated testing infrastructure the
138  # SWARMING_BOT_ID check is unnecessary.
139  if [ -n "$SWARMING_BOT_ID" ]; then
140    # If set we're running on swarming and don't need this check.
141    _pw_sourced=1
142  elif [ -n "$ZSH_EVAL_CONTEXT" ]; then
143    case $ZSH_EVAL_CONTEXT in *:file) _pw_sourced=1;; esac
144  elif [ -n "$KSH_VERSION" ]; then
145    [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != \
146      "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] \
147      && _pw_sourced=1
148  elif [ -n "$BASH_VERSION" ]; then
149    (return 0 2>/dev/null) && _pw_sourced=1
150  else  # All other shells: examine $0 for known shell binary filenames
151    # Detects `sh` and `dash`; add additional shell filenames as needed.
152    case ${0##*/} in sh|dash) _pw_sourced=1;; esac
153  fi
154
155  _pw_eval_sourced "$_pw_sourced"
156
157Downstream Projects Using Different Packages
158********************************************
159
160Projects depending on Pigweed but using additional or different packages should
161copy the Pigweed `sample project`'s ``bootstrap.sh`` and ``config.json`` and
162update the call to ``pw_bootstrap``. Search for "downstream" for other places
163that may require changes, like setting the ``PW_ROOT`` and ``PW_PROJECT_ROOT``
164environment variables. Explanations of parts of ``config.json`` are described
165here.
166
167.. _sample project: https://pigweed.googlesource.com/pigweed/sample_project/+/master
168
169``cipd_package_files``
170  CIPD package file. JSON file consisting of a list of dictionaries with "path"
171  and "tags" keys, where "tags" is a list of strings.
172
173``virtualenv.gn_targets``
174  Target for installing Python packages. Downstream projects will need to
175  create targets to install their packages or only use Pigweed Python packages.
176
177``virtualenv.gn_root``
178  The root directory of your GN build tree, relative to ``PW_PROJECT_ROOT``.
179  This is the directory your project's ``.gn`` file is located in. If you're
180  only installing Pigweed Python packages, use the location of the Pigweed
181  submodule.
182
183An example of a config file is below.
184
185.. code-block:: json
186
187  {
188    "cipd_package_files": [
189      "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json",
190      "pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json"
191      "tools/myprojectname.json"
192    ],
193    "virtualenv": {
194      "gn_root": ".",
195      "gn_targets": [
196        ":python.install",
197      ]
198    }
199  }
200
201In case the CIPD packages need to be referenced from other scripts, variables
202like ``PW_${BASENAME}_CIPD_INSTALL_DIR`` point to the CIPD install directories,
203where ``${BASENAME}`` is "PIGWEED" for
204"pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/pigweed.json" and "LUCI" for
205"pigweed/pw_env_setup/py/pw_env_setup/cipd_setup/luci.json". This example would
206set the following environment variables.
207
208 - ``PW_LUCI_CIPD_INSTALL_DIR``
209 - ``PW_MYPROJECTNAME_CIPD_INSTALL_DIR``
210 - ``PW_PIGWEED_CIPD_INSTALL_DIR``
211
212Environment Variables
213*********************
214The following environment variables affect env setup behavior. Most users will
215never need to set these.
216
217``CIPD_CACHE_DIR``
218  Location of CIPD cache dir. Defaults to ``$HOME/.cipd-cache-dir``.
219
220``PW_ACTIVATE_SKIP_CHECKS``
221  If set, skip running ``pw doctor`` at end of bootstrap/activate. Intended to
222  be used by automated tools but not interactively.
223
224``PW_BOOTSTRAP_PYTHON``
225  Python executable to be used, for example "python2" or "python3". Defaults to
226  "python".
227
228``PW_ENVIRONMENT_ROOT``
229  Location to which packages are installed. Defaults to ``.environment`` folder
230  within the checkout root.
231
232``PW_ENVSETUP_DISABLE_SPINNER``
233  Disable the spinner during env setup. Intended to be used when the output is
234  being redirected to a log.
235
236``PW_ENVSETUP_QUIET``
237  Disables all non-error output.
238
239Implementation
240**************
241
242The environment is set up by installing CIPD and Python packages in
243``PW_ENVIRONMENT_ROOT`` or ``<checkout>/.environment``, and saving modifications
244to environment variables in setup scripts in those directories. To support
245multiple operating systems this is done in an operating system-agnostic manner
246and then written into operating system-specific files to be sourced now and in
247the future when running ``activate.sh`` instead of ``bootstrap.sh``. In the
248future these could be extended to C shell and PowerShell. A logical mapping of
249high-level commands to system-specific initialization files is shown below.
250
251.. image:: doc_resources/pw_env_setup_output.png
252   :alt: Mapping of high-level commands to system-specific commands.
253   :align: left
254