1:mod:`venv` --- Creation of virtual environments
2================================================
3
4.. module:: venv
5   :synopsis: Creation of virtual environments.
6
7.. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
8.. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
9
10.. versionadded:: 3.3
11
12**Source code:** :source:`Lib/venv/`
13
14.. index:: pair: Environments; virtual
15
16--------------
17
18The :mod:`venv` module provides support for creating lightweight "virtual
19environments" with their own site directories, optionally isolated from system
20site directories.  Each virtual environment has its own Python binary (which
21matches the version of the binary that was used to create this environment) and
22can have its own independent set of installed Python packages in its site
23directories.
24
25See :pep:`405` for more information about Python virtual environments.
26
27.. seealso::
28
29   `Python Packaging User Guide: Creating and using virtual environments
30   <https://packaging.python.org/installing/#creating-virtual-environments>`__
31
32.. note::
33   The ``pyvenv`` script has been deprecated as of Python 3.6 in favor of using
34   ``python3 -m venv`` to help prevent any potential confusion as to which
35   Python interpreter a virtual environment will be based on.
36
37
38Creating virtual environments
39-----------------------------
40
41.. include:: /using/venv-create.inc
42
43
44.. _venv-def:
45
46.. note:: A virtual environment is a Python environment such that the Python
47   interpreter, libraries and scripts installed into it are isolated from those
48   installed in other virtual environments, and (by default) any libraries
49   installed in a "system" Python, i.e., one which is installed as part of your
50   operating system.
51
52   A virtual environment is a directory tree which contains Python executable
53   files and other files which indicate that it is a virtual environment.
54
55   Common installation tools such as ``Setuptools`` and ``pip`` work as
56   expected with virtual environments. In other words, when a virtual
57   environment is active, they install Python packages into the virtual
58   environment without needing to be told to do so explicitly.
59
60   When a virtual environment is active (i.e., the virtual environment's Python
61   interpreter is running), the attributes :attr:`sys.prefix` and
62   :attr:`sys.exec_prefix` point to the base directory of the virtual
63   environment, whereas :attr:`sys.base_prefix` and
64   :attr:`sys.base_exec_prefix` point to the non-virtual environment Python
65   installation which was used to create the virtual environment. If a virtual
66   environment is not active, then :attr:`sys.prefix` is the same as
67   :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as
68   :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment
69   Python installation).
70
71   When a virtual environment is active, any options that change the
72   installation path will be ignored from all distutils configuration files to
73   prevent projects being inadvertently installed outside of the virtual
74   environment.
75
76   When working in a command shell, users can make a virtual environment active
77   by running an ``activate`` script in the virtual environment's executables
78   directory (the precise filename is shell-dependent), which prepends the
79   virtual environment's directory for executables to the ``PATH`` environment
80   variable for the running shell. There should be no need in other
81   circumstances to activate a virtual environment—scripts installed into
82   virtual environments have a "shebang" line which points to the virtual
83   environment's Python interpreter. This means that the script will run with
84   that interpreter regardless of the value of ``PATH``. On Windows, "shebang"
85   line processing is supported if you have the Python Launcher for Windows
86   installed (this was added to Python in 3.3 - see :pep:`397` for more
87   details). Thus, double-clicking an installed script in a Windows Explorer
88   window should run the script with the correct interpreter without there
89   needing to be any reference to its virtual environment in ``PATH``.
90
91
92.. _venv-api:
93
94API
95---
96
97.. highlight:: python
98
99The high-level method described above makes use of a simple API which provides
100mechanisms for third-party virtual environment creators to customize environment
101creation according to their needs, the :class:`EnvBuilder` class.
102
103.. class:: EnvBuilder(system_site_packages=False, clear=False, \
104                      symlinks=False, upgrade=False, with_pip=False, \
105                      prompt=None)
106
107    The :class:`EnvBuilder` class accepts the following keyword arguments on
108    instantiation:
109
110    * ``system_site_packages`` -- a Boolean value indicating that the system Python
111      site-packages should be available to the environment (defaults to ``False``).
112
113    * ``clear`` -- a Boolean value which, if true, will delete the contents of
114      any existing target directory, before creating the environment.
115
116    * ``symlinks`` -- a Boolean value indicating whether to attempt to symlink the
117      Python binary rather than copying.
118
119    * ``upgrade`` -- a Boolean value which, if true, will upgrade an existing
120      environment with the running Python - for use when that Python has been
121      upgraded in-place (defaults to ``False``).
122
123    * ``with_pip`` -- a Boolean value which, if true, ensures pip is
124      installed in the virtual environment. This uses :mod:`ensurepip` with
125      the ``--default-pip`` option.
126
127    * ``prompt`` -- a String to be used after virtual environment is activated
128      (defaults to ``None`` which means directory name of the environment would
129      be used).
130
131    .. versionchanged:: 3.4
132       Added the ``with_pip`` parameter
133
134    .. versionadded:: 3.6
135       Added the ``prompt`` parameter
136
137    Creators of third-party virtual environment tools will be free to use the
138    provided ``EnvBuilder`` class as a base class.
139
140    The returned env-builder is an object which has a method, ``create``:
141
142    .. method:: create(env_dir)
143
144        This method takes as required argument the path (absolute or relative to
145        the current directory) of the target directory which is to contain the
146        virtual environment.  The ``create`` method will either create the
147        environment in the specified directory, or raise an appropriate
148        exception.
149
150        The ``create`` method of the ``EnvBuilder`` class illustrates the hooks
151        available for subclass customization::
152
153            def create(self, env_dir):
154                """
155                Create a virtualized Python environment in a directory.
156                env_dir is the target directory to create an environment in.
157                """
158                env_dir = os.path.abspath(env_dir)
159                context = self.ensure_directories(env_dir)
160                self.create_configuration(context)
161                self.setup_python(context)
162                self.setup_scripts(context)
163                self.post_setup(context)
164
165        Each of the methods :meth:`ensure_directories`,
166        :meth:`create_configuration`, :meth:`setup_python`,
167        :meth:`setup_scripts` and :meth:`post_setup` can be overridden.
168
169    .. method:: ensure_directories(env_dir)
170
171        Creates the environment directory and all necessary directories, and
172        returns a context object.  This is just a holder for attributes (such as
173        paths), for use by the other methods. The directories are allowed to
174        exist already, as long as either ``clear`` or ``upgrade`` were
175        specified to allow operating on an existing environment directory.
176
177    .. method:: create_configuration(context)
178
179        Creates the ``pyvenv.cfg`` configuration file in the environment.
180
181    .. method:: setup_python(context)
182
183        Creates a copy or symlink to the Python executable in the environment.
184        On POSIX systems, if a specific executable ``python3.x`` was used,
185        symlinks to ``python`` and ``python3`` will be created pointing to that
186        executable, unless files with those names already exist.
187
188    .. method:: setup_scripts(context)
189
190        Installs activation scripts appropriate to the platform into the virtual
191        environment.
192
193    .. method:: post_setup(context)
194
195        A placeholder method which can be overridden in third party
196        implementations to pre-install packages in the virtual environment or
197        perform other post-creation steps.
198
199    .. versionchanged:: 3.7.2
200       Windows now uses redirector scripts for ``python[w].exe`` instead of
201       copying the actual binaries. In 3.7.2 only :meth:`setup_python` does
202       nothing unless running from a build in the source tree.
203
204    .. versionchanged:: 3.7.3
205       Windows copies the redirector scripts as part of :meth:`setup_python`
206       instead of :meth:`setup_scripts`. This was not the case in 3.7.2.
207       When using symlinks, the original executables will be linked.
208
209    In addition, :class:`EnvBuilder` provides this utility method that can be
210    called from :meth:`setup_scripts` or :meth:`post_setup` in subclasses to
211    assist in installing custom scripts into the virtual environment.
212
213    .. method:: install_scripts(context, path)
214
215        *path* is the path to a directory that should contain subdirectories
216        "common", "posix", "nt", each containing scripts destined for the bin
217        directory in the environment.  The contents of "common" and the
218        directory corresponding to :data:`os.name` are copied after some text
219        replacement of placeholders:
220
221        * ``__VENV_DIR__`` is replaced with the absolute path of the environment
222          directory.
223
224        * ``__VENV_NAME__`` is replaced with the environment name (final path
225          segment of environment directory).
226
227        * ``__VENV_PROMPT__`` is replaced with the prompt (the environment
228          name surrounded by parentheses and with a following space)
229
230        * ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
231          (either ``bin`` or ``Scripts``).
232
233        * ``__VENV_PYTHON__`` is replaced with the absolute path of the
234          environment's executable.
235
236        The directories are allowed to exist (for when an existing environment
237        is being upgraded).
238
239There is also a module-level convenience function:
240
241.. function:: create(env_dir, system_site_packages=False, clear=False, \
242                     symlinks=False, with_pip=False)
243
244    Create an :class:`EnvBuilder` with the given keyword arguments, and call its
245    :meth:`~EnvBuilder.create` method with the *env_dir* argument.
246
247    .. versionchanged:: 3.4
248       Added the ``with_pip`` parameter
249
250An example of extending ``EnvBuilder``
251--------------------------------------
252
253The following script shows how to extend :class:`EnvBuilder` by implementing a
254subclass which installs setuptools and pip into a created virtual environment::
255
256    import os
257    import os.path
258    from subprocess import Popen, PIPE
259    import sys
260    from threading import Thread
261    from urllib.parse import urlparse
262    from urllib.request import urlretrieve
263    import venv
264
265    class ExtendedEnvBuilder(venv.EnvBuilder):
266        """
267        This builder installs setuptools and pip so that you can pip or
268        easy_install other packages into the created virtual environment.
269
270        :param nodist: If True, setuptools and pip are not installed into the
271                       created virtual environment.
272        :param nopip: If True, pip is not installed into the created
273                      virtual environment.
274        :param progress: If setuptools or pip are installed, the progress of the
275                         installation can be monitored by passing a progress
276                         callable. If specified, it is called with two
277                         arguments: a string indicating some progress, and a
278                         context indicating where the string is coming from.
279                         The context argument can have one of three values:
280                         'main', indicating that it is called from virtualize()
281                         itself, and 'stdout' and 'stderr', which are obtained
282                         by reading lines from the output streams of a subprocess
283                         which is used to install the app.
284
285                         If a callable is not specified, default progress
286                         information is output to sys.stderr.
287        """
288
289        def __init__(self, *args, **kwargs):
290            self.nodist = kwargs.pop('nodist', False)
291            self.nopip = kwargs.pop('nopip', False)
292            self.progress = kwargs.pop('progress', None)
293            self.verbose = kwargs.pop('verbose', False)
294            super().__init__(*args, **kwargs)
295
296        def post_setup(self, context):
297            """
298            Set up any packages which need to be pre-installed into the
299            virtual environment being created.
300
301            :param context: The information for the virtual environment
302                            creation request being processed.
303            """
304            os.environ['VIRTUAL_ENV'] = context.env_dir
305            if not self.nodist:
306                self.install_setuptools(context)
307            # Can't install pip without setuptools
308            if not self.nopip and not self.nodist:
309                self.install_pip(context)
310
311        def reader(self, stream, context):
312            """
313            Read lines from a subprocess' output stream and either pass to a progress
314            callable (if specified) or write progress information to sys.stderr.
315            """
316            progress = self.progress
317            while True:
318                s = stream.readline()
319                if not s:
320                    break
321                if progress is not None:
322                    progress(s, context)
323                else:
324                    if not self.verbose:
325                        sys.stderr.write('.')
326                    else:
327                        sys.stderr.write(s.decode('utf-8'))
328                    sys.stderr.flush()
329            stream.close()
330
331        def install_script(self, context, name, url):
332            _, _, path, _, _, _ = urlparse(url)
333            fn = os.path.split(path)[-1]
334            binpath = context.bin_path
335            distpath = os.path.join(binpath, fn)
336            # Download script into the virtual environment's binaries folder
337            urlretrieve(url, distpath)
338            progress = self.progress
339            if self.verbose:
340                term = '\n'
341            else:
342                term = ''
343            if progress is not None:
344                progress('Installing %s ...%s' % (name, term), 'main')
345            else:
346                sys.stderr.write('Installing %s ...%s' % (name, term))
347                sys.stderr.flush()
348            # Install in the virtual environment
349            args = [context.env_exe, fn]
350            p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath)
351            t1 = Thread(target=self.reader, args=(p.stdout, 'stdout'))
352            t1.start()
353            t2 = Thread(target=self.reader, args=(p.stderr, 'stderr'))
354            t2.start()
355            p.wait()
356            t1.join()
357            t2.join()
358            if progress is not None:
359                progress('done.', 'main')
360            else:
361                sys.stderr.write('done.\n')
362            # Clean up - no longer needed
363            os.unlink(distpath)
364
365        def install_setuptools(self, context):
366            """
367            Install setuptools in the virtual environment.
368
369            :param context: The information for the virtual environment
370                            creation request being processed.
371            """
372            url = 'https://bitbucket.org/pypa/setuptools/downloads/ez_setup.py'
373            self.install_script(context, 'setuptools', url)
374            # clear up the setuptools archive which gets downloaded
375            pred = lambda o: o.startswith('setuptools-') and o.endswith('.tar.gz')
376            files = filter(pred, os.listdir(context.bin_path))
377            for f in files:
378                f = os.path.join(context.bin_path, f)
379                os.unlink(f)
380
381        def install_pip(self, context):
382            """
383            Install pip in the virtual environment.
384
385            :param context: The information for the virtual environment
386                            creation request being processed.
387            """
388            url = 'https://raw.github.com/pypa/pip/master/contrib/get-pip.py'
389            self.install_script(context, 'pip', url)
390
391    def main(args=None):
392        compatible = True
393        if sys.version_info < (3, 3):
394            compatible = False
395        elif not hasattr(sys, 'base_prefix'):
396            compatible = False
397        if not compatible:
398            raise ValueError('This script is only for use with '
399                             'Python 3.3 or later')
400        else:
401            import argparse
402
403            parser = argparse.ArgumentParser(prog=__name__,
404                                             description='Creates virtual Python '
405                                                         'environments in one or '
406                                                         'more target '
407                                                         'directories.')
408            parser.add_argument('dirs', metavar='ENV_DIR', nargs='+',
409                                help='A directory in which to create the
410                                     'virtual environment.')
411            parser.add_argument('--no-setuptools', default=False,
412                                action='store_true', dest='nodist',
413                                help="Don't install setuptools or pip in the "
414                                     "virtual environment.")
415            parser.add_argument('--no-pip', default=False,
416                                action='store_true', dest='nopip',
417                                help="Don't install pip in the virtual "
418                                     "environment.")
419            parser.add_argument('--system-site-packages', default=False,
420                                action='store_true', dest='system_site',
421                                help='Give the virtual environment access to the '
422                                     'system site-packages dir.')
423            if os.name == 'nt':
424                use_symlinks = False
425            else:
426                use_symlinks = True
427            parser.add_argument('--symlinks', default=use_symlinks,
428                                action='store_true', dest='symlinks',
429                                help='Try to use symlinks rather than copies, '
430                                     'when symlinks are not the default for '
431                                     'the platform.')
432            parser.add_argument('--clear', default=False, action='store_true',
433                                dest='clear', help='Delete the contents of the '
434                                                   'virtual environment '
435                                                   'directory if it already '
436                                                   'exists, before virtual '
437                                                   'environment creation.')
438            parser.add_argument('--upgrade', default=False, action='store_true',
439                                dest='upgrade', help='Upgrade the virtual '
440                                                     'environment directory to '
441                                                     'use this version of '
442                                                     'Python, assuming Python '
443                                                     'has been upgraded '
444                                                     'in-place.')
445            parser.add_argument('--verbose', default=False, action='store_true',
446                                dest='verbose', help='Display the output '
447                                                   'from the scripts which '
448                                                   'install setuptools and pip.')
449            options = parser.parse_args(args)
450            if options.upgrade and options.clear:
451                raise ValueError('you cannot supply --upgrade and --clear together.')
452            builder = ExtendedEnvBuilder(system_site_packages=options.system_site,
453                                           clear=options.clear,
454                                           symlinks=options.symlinks,
455                                           upgrade=options.upgrade,
456                                           nodist=options.nodist,
457                                           nopip=options.nopip,
458                                           verbose=options.verbose)
459            for d in options.dirs:
460                builder.create(d)
461
462    if __name__ == '__main__':
463        rc = 1
464        try:
465            main()
466            rc = 0
467        except Exception as e:
468            print('Error: %s' % e, file=sys.stderr)
469        sys.exit(rc)
470
471
472This script is also available for download `online
473<https://gist.github.com/vsajip/4673395>`_.
474