1.. currentmodule:: asyncio
2
3.. _asyncio-subprocess:
4
5============
6Subprocesses
7============
8
9This section describes high-level async/await asyncio APIs to
10create and manage subprocesses.
11
12.. _asyncio_example_subprocess_shell:
13
14Here's an example of how asyncio can run a shell command and
15obtain its result::
16
17    import asyncio
18
19    async def run(cmd):
20        proc = await asyncio.create_subprocess_shell(
21            cmd,
22            stdout=asyncio.subprocess.PIPE,
23            stderr=asyncio.subprocess.PIPE)
24
25        stdout, stderr = await proc.communicate()
26
27        print(f'[{cmd!r} exited with {proc.returncode}]')
28        if stdout:
29            print(f'[stdout]\n{stdout.decode()}')
30        if stderr:
31            print(f'[stderr]\n{stderr.decode()}')
32
33    asyncio.run(run('ls /zzz'))
34
35will print::
36
37    ['ls /zzz' exited with 1]
38    [stderr]
39    ls: /zzz: No such file or directory
40
41Because all asyncio subprocess functions are asynchronous and asyncio
42provides many tools to work with such functions, it is easy to execute
43and monitor multiple subprocesses in parallel.  It is indeed trivial
44to modify the above example to run several commands simultaneously::
45
46    async def main():
47        await asyncio.gather(
48            run('ls /zzz'),
49            run('sleep 1; echo "hello"'))
50
51    asyncio.run(main())
52
53See also the `Examples`_ subsection.
54
55
56Creating Subprocesses
57=====================
58
59.. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, \
60                          stdout=None, stderr=None, loop=None, \
61                          limit=None, \*\*kwds)
62
63   Create a subprocess.
64
65   The *limit* argument sets the buffer limit for :class:`StreamReader`
66   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
67   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
68
69   Return a :class:`~asyncio.subprocess.Process` instance.
70
71   See the documentation of :meth:`loop.subprocess_exec` for other
72   parameters.
73
74.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \
75                          stdout=None, stderr=None, loop=None, \
76                          limit=None, \*\*kwds)
77
78   Run the *cmd* shell command.
79
80   The *limit* argument sets the buffer limit for :class:`StreamReader`
81   wrappers for :attr:`Process.stdout` and :attr:`Process.stderr`
82   (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments).
83
84   Return a :class:`~asyncio.subprocess.Process` instance.
85
86   See the documentation of :meth:`loop.subprocess_shell` for other
87   parameters.
88
89.. important::
90
91   It is the application's responsibility to ensure that all whitespace and
92   special characters are quoted appropriately to avoid `shell injection
93   <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
94   vulnerabilities. The :func:`shlex.quote` function can be used to properly
95   escape whitespace and special shell characters in strings that are going
96   to be used to construct shell commands.
97
98.. note::
99
100   The default asyncio event loop implementation on **Windows** does not
101   support subprocesses. Subprocesses are available for Windows if a
102   :class:`ProactorEventLoop` is used.
103   See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>`
104   for details.
105
106.. seealso::
107
108   asyncio also has the following *low-level* APIs to work with subprocesses:
109   :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`,
110   :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`,
111   as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>`
112   and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`.
113
114
115Constants
116=========
117
118.. data:: asyncio.subprocess.PIPE
119
120   Can be passed to the *stdin*, *stdout* or *stderr* parameters.
121
122   If *PIPE* is passed to *stdin* argument, the
123   :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute
124   will point to a :class:`StreamWriter` instance.
125
126   If *PIPE* is passed to *stdout* or *stderr* arguments, the
127   :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and
128   :attr:`Process.stderr <asyncio.subprocess.Process.stderr>`
129   attributes will point to :class:`StreamReader` instances.
130
131.. data:: asyncio.subprocess.STDOUT
132
133   Special value that can be used as the *stderr* argument and indicates
134   that standard error should be redirected into standard output.
135
136.. data:: asyncio.subprocess.DEVNULL
137
138   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
139   to process creation functions.  It indicates that the special file
140   :data:`os.devnull` will be used for the corresponding subprocess stream.
141
142
143Interacting with Subprocesses
144=============================
145
146Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
147functions return instances of the *Process* class.  *Process* is a high-level
148wrapper that allows communicating with subprocesses and watching for
149their completion.
150
151.. class:: asyncio.subprocess.Process
152
153   An object that wraps OS processes created by the
154   :func:`create_subprocess_exec` and :func:`create_subprocess_shell`
155   functions.
156
157   This class is designed to have a similar API to the
158   :class:`subprocess.Popen` class, but there are some
159   notable differences:
160
161   * unlike Popen, Process instances do not have an equivalent to
162     the :meth:`~subprocess.Popen.poll` method;
163
164   * the :meth:`~asyncio.subprocess.Process.communicate` and
165     :meth:`~asyncio.subprocess.Process.wait` methods don't have a
166     *timeout* parameter: use the :func:`wait_for` function;
167
168   * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method
169     is asynchronous, whereas :meth:`subprocess.Popen.wait` method
170     is implemented as a blocking busy loop;
171
172   * the *universal_newlines* parameter is not supported.
173
174   This class is :ref:`not thread safe <asyncio-multithreading>`.
175
176   See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
177   section.
178
179   .. coroutinemethod:: wait()
180
181      Wait for the child process to terminate.
182
183      Set and return the :attr:`returncode` attribute.
184
185      .. note::
186
187         This method can deadlock when using ``stdout=PIPE`` or
188         ``stderr=PIPE`` and the child process generates so much output
189         that it blocks waiting for the OS pipe buffer to accept
190         more data. Use the :meth:`communicate` method when using pipes
191         to avoid this condition.
192
193   .. coroutinemethod:: communicate(input=None)
194
195      Interact with process:
196
197      1. send data to *stdin* (if *input* is not ``None``);
198      2. read data from *stdout* and *stderr*, until EOF is reached;
199      3. wait for process to terminate.
200
201      The optional *input* argument is the data (:class:`bytes` object)
202      that will be sent to the child process.
203
204      Return a tuple ``(stdout_data, stderr_data)``.
205
206      If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError`
207      exception is raised when writing *input* into *stdin*, the
208      exception is ignored.  This condition occurs when the process
209      exits before all data are written into *stdin*.
210
211      If it is desired to send data to the process' *stdin*,
212      the process needs to be created with ``stdin=PIPE``.  Similarly,
213      to get anything other than ``None`` in the result tuple, the
214      process has to be created with ``stdout=PIPE`` and/or
215      ``stderr=PIPE`` arguments.
216
217      Note, that the data read is buffered in memory, so do not use
218      this method if the data size is large or unlimited.
219
220   .. method:: send_signal(signal)
221
222      Sends the signal *signal* to the child process.
223
224      .. note::
225
226         On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
227         ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
228         started with a *creationflags* parameter which includes
229         ``CREATE_NEW_PROCESS_GROUP``.
230
231   .. method:: terminate()
232
233      Stop the child process.
234
235      On POSIX systems this method sends :py:data:`signal.SIGTERM` to the
236      child process.
237
238      On Windows the Win32 API function :c:func:`TerminateProcess` is
239      called to stop the child process.
240
241   .. method:: kill()
242
243      Kill the child.
244
245      On POSIX systems this method sends :py:data:`SIGKILL` to the child
246      process.
247
248      On Windows this method is an alias for :meth:`terminate`.
249
250   .. attribute:: stdin
251
252      Standard input stream (:class:`StreamWriter`) or ``None``
253      if the process was created with ``stdin=None``.
254
255   .. attribute:: stdout
256
257      Standard output stream (:class:`StreamReader`) or ``None``
258      if the process was created with ``stdout=None``.
259
260   .. attribute:: stderr
261
262      Standard error stream (:class:`StreamReader`) or ``None``
263      if the process was created with ``stderr=None``.
264
265   .. warning::
266
267      Use the :meth:`communicate` method rather than
268      :attr:`process.stdin.write() <stdin>`,
269      :attr:`await process.stdout.read() <stdout>` or
270      :attr:`await process.stderr.read <stderr>`.
271      This avoids deadlocks due to streams pausing reading or writing
272      and blocking the child process.
273
274   .. attribute:: pid
275
276      Process identification number (PID).
277
278      Note that for processes created by the :func:`create_subprocess_shell`
279      function, this attribute is the PID of the spawned shell.
280
281   .. attribute:: returncode
282
283      Return code of the process when it exits.
284
285      A ``None`` value indicates that the process has not terminated yet.
286
287      A negative value ``-N`` indicates that the child was terminated
288      by signal ``N`` (POSIX only).
289
290
291.. _asyncio-subprocess-threads:
292
293Subprocess and Threads
294----------------------
295
296Standard asyncio event loop supports running subprocesses from
297different threads, but there are limitations:
298
299* An event loop must run in the main thread.
300
301* The child watcher must be instantiated in the main thread
302  before executing subprocesses from other threads. Call the
303  :func:`get_child_watcher` function in the main thread to instantiate
304  the child watcher.
305
306Note that alternative event loop implementations might not share
307the above limitations; please refer to their documentation.
308
309.. seealso::
310
311   The :ref:`Concurrency and multithreading in asyncio
312   <asyncio-multithreading>` section.
313
314
315Examples
316--------
317
318An example using the :class:`~asyncio.subprocess.Process` class to
319control a subprocess and the :class:`StreamReader` class to read from
320its standard output.
321
322.. _asyncio_example_create_subprocess_exec:
323
324The subprocess is created by the :func:`create_subprocess_exec`
325function::
326
327    import asyncio
328    import sys
329
330    async def get_date():
331        code = 'import datetime; print(datetime.datetime.now())'
332
333        # Create the subprocess; redirect the standard output
334        # into a pipe.
335        proc = await asyncio.create_subprocess_exec(
336            sys.executable, '-c', code,
337            stdout=asyncio.subprocess.PIPE)
338
339        # Read one line of output.
340        data = await proc.stdout.readline()
341        line = data.decode('ascii').rstrip()
342
343        # Wait for the subprocess exit.
344        await proc.wait()
345        return line
346
347    if sys.platform == "win32":
348        asyncio.set_event_loop_policy(
349            asyncio.WindowsProactorEventLoopPolicy())
350
351    date = asyncio.run(get_date())
352    print(f"Current date: {date}")
353
354
355See also the :ref:`same example <asyncio_example_subprocess_proto>`
356written using low-level APIs.
357