1.. _module-pw_presubmit: 2 3============ 4pw_presubmit 5============ 6The presubmit module provides Python tools for running presubmit checks and 7checking and fixing code format. It also includes the presubmit check script for 8the Pigweed repository, ``pigweed_presubmit.py``. 9 10Presubmit checks are essential tools, but they take work to set up, and 11projects don’t always get around to it. The ``pw_presubmit`` module provides 12tools for setting up high quality presubmit checks for any project. We use this 13framework to run Pigweed’s presubmit on our workstations and in our automated 14building tools. 15 16The ``pw_presubmit`` module also includes ``pw format``, a tool that provides a 17unified interface for automatically formatting code in a variety of languages. 18With ``pw format``, you can format C, C++, Python, GN, and Go code according to 19configurations defined by your project. ``pw format`` leverages existing tools 20like ``clang-format``, and it’s simple to add support for new languages. 21 22.. image:: docs/pw_presubmit_demo.gif 23 :alt: ``pw format`` demo 24 :align: left 25 26The ``pw_presubmit`` package includes presubmit checks that can be used with any 27project. These checks include: 28 29* Check code format of several languages including C, C++, and Python 30* Initialize a Python environment 31* Run all Python tests 32* Run pylint 33* Run mypy 34* Ensure source files are included in the GN and Bazel builds 35* Build and run all tests with GN 36* Build and run all tests with Bazel 37* Ensure all header files contain ``#pragma once`` 38 39------------- 40Compatibility 41------------- 42Python 3 43 44------------------------------------------- 45Creating a presubmit check for your project 46------------------------------------------- 47Creating a presubmit check for a project using ``pw_presubmit`` is simple, but 48requires some customization. Projects must define their own presubmit check 49Python script that uses the ``pw_presubmit`` package. 50 51A project's presubmit script can be registered as a 52:ref:`pw_cli <module-pw_cli>` plugin, so that it can be run as ``pw 53presubmit``. 54 55Setting up the command-line interface 56------------------------------------- 57The ``pw_presubmit.cli`` module sets up the command-line interface for a 58presubmit script. This defines a standard set of arguments for invoking 59presubmit checks. Its use is optional, but recommended. 60 61pw_presubmit.cli 62~~~~~~~~~~~~~~~~ 63.. automodule:: pw_presubmit.cli 64 :members: add_arguments, run 65 66Presubmit checks 67---------------- 68A presubmit check is defined as a function or other callable. The function must 69accept one argument: a ``PresubmitContext``, which provides the paths on which 70to run. Presubmit checks communicate failure by raising an exception. 71 72Presubmit checks may use the ``filter_paths`` decorator to automatically filter 73the paths list for file types they care about. 74 75Either of these functions could be used as presubmit checks: 76 77.. code-block:: python 78 79 @pw_presubmit.filter_paths(endswith='.py') 80 def file_contains_ni(ctx: PresubmitContext): 81 for path in ctx.paths: 82 with open(path) as file: 83 contents = file.read() 84 if 'ni' not in contents and 'nee' not in contents: 85 raise PresumitFailure('Files must say "ni"!', path=path) 86 87 def run_the_build(_): 88 subprocess.run(['make', 'release'], check=True) 89 90Presubmit checks functions are grouped into "programs" -- a named series of 91checks. Projects may find it helpful to have programs for different purposes, 92such as a quick program for local use and a full program for automated use. The 93:ref:`example script <example-script>` uses ``pw_presubmit.Programs`` to define 94``quick`` and ``full`` programs. 95 96pw_presubmit 97~~~~~~~~~~~~ 98.. automodule:: pw_presubmit 99 :members: filter_paths, call, PresubmitFailure, Programs 100 101.. _example-script: 102 103Example 104------- 105A simple example presubmit check script follows. This can be copied-and-pasted 106to serve as a starting point for a project's presubmit check script. 107 108See ``pigweed_presubmit.py`` for a more complex presubmit check script example. 109 110.. code-block:: python 111 112 """Example presubmit check script.""" 113 114 import argparse 115 import logging 116 import os 117 from pathlib import Path 118 import re 119 import sys 120 from typing import List, Pattern 121 122 try: 123 import pw_cli.log 124 except ImportError: 125 print('ERROR: Activate the environment before running presubmits!', 126 file=sys.stderr) 127 sys.exit(2) 128 129 import pw_presubmit 130 from pw_presubmit import build, cli, environment, format_code, git_repo 131 from pw_presubmit import python_checks, filter_paths, PresubmitContext 132 from pw_presubmit.install_hook import install_hook 133 134 # Set up variables for key project paths. 135 PROJECT_ROOT = Path(os.environ['MY_PROJECT_ROOT']) 136 PIGWEED_ROOT = PROJECT_ROOT / 'pigweed' 137 138 # 139 # Initialization 140 # 141 def init_cipd(ctx: PresubmitContext): 142 environment.init_cipd(PIGWEED_ROOT, ctx.output_dir) 143 144 145 def init_virtualenv(ctx: PresubmitContext): 146 environment.init_virtualenv(PIGWEED_ROOT, 147 ctx.output_dir, 148 setup_py_roots=[PROJECT_ROOT]) 149 150 151 # Rerun the build if files with these extensions change. 152 _BUILD_EXTENSIONS = frozenset( 153 ['.rst', '.gn', '.gni', *format_code.C_FORMAT.extensions]) 154 155 156 # 157 # Presubmit checks 158 # 159 def release_build(ctx: PresubmitContext): 160 build.gn_gen(PROJECT_ROOT, ctx.output_dir, build_type='release') 161 build.ninja(ctx.output_dir) 162 163 164 def host_tests(ctx: PresubmitContext): 165 build.gn_gen(PROJECT_ROOT, ctx.output_dir, run_host_tests='true') 166 build.ninja(ctx.output_dir) 167 168 169 # Avoid running some checks on certain paths. 170 PATH_EXCLUSIONS = ( 171 re.compile(r'^external/'), 172 re.compile(r'^vendor/'), 173 ) 174 175 176 # Use the upstream pragma_once check, but apply a different set of path 177 # filters with @filter_paths. 178 @filter_paths(endswith='.h', exclude=PATH_EXCLUSIONS) 179 def pragma_once(ctx: PresubmitContext): 180 pw_presubmit.pragma_once(ctx) 181 182 183 # 184 # Presubmit check programs 185 # 186 QUICK = ( 187 # Initialize an environment for running presubmit checks. 188 init_cipd, 189 init_virtualenv, 190 # List some presubmit checks to run 191 pragma_once, 192 host_tests, 193 # Use the upstream formatting checks, with custom path filters applied. 194 format_code.presubmit_checks(exclude=PATH_EXCLUSIONS), 195 ) 196 197 FULL = ( 198 QUICK, # Add all checks from the 'quick' program 199 release_build, 200 # Use the upstream Python checks, with custom path filters applied. 201 python_checks.all_checks(exclude=PATH_EXCLUSIONS), 202 ) 203 204 PROGRAMS = pw_presubmit.Programs(quick=QUICK, full=FULL) 205 206 207 def run(install: bool, **presubmit_args) -> int: 208 """Process the --install argument then invoke pw_presubmit.""" 209 210 # Install the presubmit Git pre-push hook, if requested. 211 if install: 212 install_hook(__file__, 'pre-push', ['--base', 'HEAD~'], 213 git_repo.root()) 214 return 0 215 216 return cli.run(root=PROJECT_ROOT, **presubmit_args) 217 218 219 def main() -> int: 220 """Run the presubmit checks for this repository.""" 221 parser = argparse.ArgumentParser(description=__doc__) 222 cli.add_arguments(parser, PROGRAMS, 'quick') 223 224 # Define an option for installing a Git pre-push hook for this script. 225 parser.add_argument( 226 '--install', 227 action='store_true', 228 help='Install the presubmit as a Git pre-push hook and exit.') 229 230 return run(**vars(parser.parse_args())) 231 232 if __name__ == '__main__': 233 pw_cli.log.install(logging.INFO) 234 sys.exit(main()) 235 236--------------------- 237Code formatting tools 238--------------------- 239The ``pw_presubmit.format_code`` module formats supported source files using 240external code format tools. The file ``format_code.py`` can be invoked directly 241from the command line or from ``pw`` as ``pw format``. 242