1.. _module-pw_presubmit:
2
3============
4pw_presubmit
5============
6The presubmit module provides Python tools for running presubmit checks and
7checking and fixing code format. It also includes the presubmit check script for
8the Pigweed repository, ``pigweed_presubmit.py``.
9
10Presubmit checks are essential tools, but they take work to set up, and
11projects don’t always get around to it. The ``pw_presubmit`` module provides
12tools for setting up high quality presubmit checks for any project. We use this
13framework to run Pigweed’s presubmit on our workstations and in our automated
14building tools.
15
16The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a
17unified interface for automatically formatting code in a variety of languages.
18With ``pw format``, you can format C, C++, Python, GN, and Go code according to
19configurations defined by your project. ``pw format`` leverages existing tools
20like ``clang-format``, and it’s simple to add support for new languages.
21
22.. image:: docs/pw_presubmit_demo.gif
23   :alt: ``pw format`` demo
24   :align: left
25
26The ``pw_presubmit`` package includes presubmit checks that can be used with any
27project. These checks include:
28
29* Check code format of several languages including C, C++, and Python
30* Initialize a Python environment
31* Run all Python tests
32* Run pylint
33* Run mypy
34* Ensure source files are included in the GN and Bazel builds
35* Build and run all tests with GN
36* Build and run all tests with Bazel
37* Ensure all header files contain ``#pragma once``
38
39-------------
40Compatibility
41-------------
42Python 3
43
44-------------------------------------------
45Creating a presubmit check for your project
46-------------------------------------------
47Creating a presubmit check for a project using ``pw_presubmit`` is simple, but
48requires some customization. Projects must define their own presubmit check
49Python script that uses the ``pw_presubmit`` package.
50
51A project's presubmit script can be registered as a
52:ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw
53presubmit``.
54
55Setting up the command-line interface
56-------------------------------------
57The ``pw_presubmit.cli`` module sets up the command-line interface for a
58presubmit script. This defines a standard set of arguments for invoking
59presubmit checks. Its use is optional, but recommended.
60
61pw_presubmit.cli
62~~~~~~~~~~~~~~~~
63.. automodule:: pw_presubmit.cli
64   :members: add_arguments, run
65
66Presubmit checks
67----------------
68A presubmit check is defined as a function or other callable. The function must
69accept one argument: a ``PresubmitContext``, which provides the paths on which
70to run. Presubmit checks communicate failure by raising an exception.
71
72Presubmit checks may use the ``filter_paths`` decorator to automatically filter
73the paths list for file types they care about.
74
75Either of these functions could be used as presubmit checks:
76
77.. code-block:: python
78
79  @pw_presubmit.filter_paths(endswith='.py')
80  def file_contains_ni(ctx: PresubmitContext):
81      for path in ctx.paths:
82          with open(path) as file:
83              contents = file.read()
84              if 'ni' not in contents and 'nee' not in contents:
85                  raise PresumitFailure('Files must say "ni"!', path=path)
86
87  def run_the_build(_):
88      subprocess.run(['make', 'release'], check=True)
89
90Presubmit checks functions are grouped into "programs" -- a named series of
91checks. Projects may find it helpful to have programs for different purposes,
92such as a quick program for local use and a full program for automated use. The
93:ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define
94``quick`` and ``full`` programs.
95
96pw_presubmit
97~~~~~~~~~~~~
98.. automodule:: pw_presubmit
99   :members: filter_paths, call, PresubmitFailure, Programs
100
101.. _example-script:
102
103Example
104-------
105A simple example presubmit check script follows. This can be copied-and-pasted
106to serve as a starting point for a project's presubmit check script.
107
108See ``pigweed_presubmit.py`` for a more complex presubmit check script example.
109
110.. code-block:: python
111
112  """Example presubmit check script."""
113
114  import argparse
115  import logging
116  import os
117  from pathlib import Path
118  import re
119  import sys
120  from typing import List, Pattern
121
122  try:
123      import pw_cli.log
124  except ImportError:
125      print('ERROR: Activate the environment before running presubmits!',
126            file=sys.stderr)
127      sys.exit(2)
128
129  import pw_presubmit
130  from pw_presubmit import build, cli, environment, format_code, git_repo
131  from pw_presubmit import python_checks, filter_paths, PresubmitContext
132  from pw_presubmit.install_hook import install_hook
133
134  # Set up variables for key project paths.
135  PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT'])
136  PIGWEED_ROOT = PROJECT_ROOT / 'pigweed'
137
138  #
139  # Initialization
140  #
141  def init_cipd(ctx: PresubmitContext):
142      environment.init_cipd(PIGWEED_ROOT, ctx.output_dir)
143
144
145  def init_virtualenv(ctx: PresubmitContext):
146      environment.init_virtualenv(PIGWEED_ROOT,
147                                  ctx.output_dir,
148                                  setup_py_roots=[PROJECT_ROOT])
149
150
151  # Rerun the build if files with these extensions change.
152  _BUILD_EXTENSIONS = frozenset(
153      ['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions])
154
155
156  #
157  # Presubmit checks
158  #
159  def release_build(ctx: PresubmitContext):
160      build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release')
161      build.ninja(ctx.output_dir)
162
163
164  def host_tests(ctx: PresubmitContext):
165      build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true')
166      build.ninja(ctx.output_dir)
167
168
169  # Avoid running some checks on certain paths.
170  PATH_EXCLUSIONS = (
171      re.compile(r'^external/'),
172      re.compile(r'^vendor/'),
173  )
174
175
176  # Use the upstream pragma_once check, but apply a different set of path
177  # filters with @filter_paths.
178  @filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS)
179  def pragma_once(ctx: PresubmitContext):
180      pw_presubmit.pragma_once(ctx)
181
182
183  #
184  # Presubmit check programs
185  #
186  QUICK = (
187      # Initialize an environment for running presubmit checks.
188      init_cipd,
189      init_virtualenv,
190      # List some presubmit checks to run
191      pragma_once,
192      host_tests,
193      # Use the upstream formatting checks, with custom path filters applied.
194      format_code.presubmit_checks(exclude=PATH_EXCLUSIONS),
195  )
196
197  FULL = (
198      QUICK,  # Add all checks from the 'quick' program
199      release_build,
200      # Use the upstream Python checks, with custom path filters applied.
201      python_checks.all_checks(exclude=PATH_EXCLUSIONS),
202  )
203
204  PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL)
205
206
207  def run(install: bool, **presubmit_args) -> int:
208      """Process the --install argument then invoke pw_presubmit."""
209
210      # Install the presubmit Git pre-push hook, if requested.
211      if install:
212          install_hook(__file__, 'pre-push', ['--base', 'HEAD~'],
213                       git_repo.root())
214          return 0
215
216      return cli.run(root=PROJECT_ROOT, **presubmit_args)
217
218
219  def main() -> int:
220      """Run the presubmit checks for this repository."""
221      parser = argparse.ArgumentParser(description=__doc__)
222      cli.add_arguments(parser, PROGRAMS, 'quick')
223
224      # Define an option for installing a Git pre-push hook for this script.
225      parser.add_argument(
226          '--install',
227          action='store_true',
228          help='Install the presubmit as a Git pre-push hook and exit.')
229
230      return run(**vars(parser.parse_args()))
231
232  if __name__ == '__main__':
233      pw_cli.log.install(logging.INFO)
234      sys.exit(main())
235
236---------------------
237Code formatting tools
238---------------------
239The ``pw_presubmit.format_code`` module formats supported source files using
240external code format tools. The file ``format_code.py`` can be invoked directly
241from the command line or from ``pw`` as ``pw format``.
242