1**************************** 2 What's New In Python 3.6 3**************************** 4 5:Editors: Elvis Pranskevichus <elvis@magic.io>, Yury Selivanov <yury@magic.io> 6 7.. Rules for maintenance: 8 9 * Anyone can add text to this document. Do not spend very much time 10 on the wording of your changes, because your text will probably 11 get rewritten to some degree. 12 13 * The maintainer will go through Misc/NEWS periodically and add 14 changes; it's therefore more important to add your changes to 15 Misc/NEWS than to this file. 16 17 * This is not a complete list of every single change; completeness 18 is the purpose of Misc/NEWS. Some changes I consider too small 19 or esoteric to include. If such a change is added to the text, 20 I'll just remove it. (This is another reason you shouldn't spend 21 too much time on writing your addition.) 22 23 * If you want to draw your new text to the attention of the 24 maintainer, add 'XXX' to the beginning of the paragraph or 25 section. 26 27 * It's OK to just add a fragmentary note about a change. For 28 example: "XXX Describe the transmogrify() function added to the 29 socket module." The maintainer will research the change and 30 write the necessary text. 31 32 * You can comment out your additions if you like, but it's not 33 necessary (especially when a final release is some months away). 34 35 * Credit the author of a patch or bugfix. Just the name is 36 sufficient; the e-mail address isn't necessary. 37 38 * It's helpful to add the bug/patch number as a comment: 39 40 XXX Describe the transmogrify() function added to the socket 41 module. 42 (Contributed by P.Y. Developer in :issue:`12345`.) 43 44 This saves the maintainer the effort of going through the Mercurial log 45 when researching a change. 46 47This article explains the new features in Python 3.6, compared to 3.5. 48Python 3.6 was released on December 23, 2016. See the 49`changelog <https://docs.python.org/3.6/whatsnew/changelog.html>`_ for a full 50list of changes. 51 52.. seealso:: 53 54 :pep:`494` - Python 3.6 Release Schedule 55 56 57Summary -- Release highlights 58============================= 59 60New syntax features: 61 62* :ref:`PEP 498 <whatsnew36-pep498>`, formatted string literals. 63 64* :ref:`PEP 515 <whatsnew36-pep515>`, underscores in numeric literals. 65 66* :ref:`PEP 526 <whatsnew36-pep526>`, syntax for variable annotations. 67 68* :ref:`PEP 525 <whatsnew36-pep525>`, asynchronous generators. 69 70* :ref:`PEP 530 <whatsnew36-pep530>`: asynchronous comprehensions. 71 72 73New library modules: 74 75* :mod:`secrets`: :ref:`PEP 506 -- Adding A Secrets Module To The Standard Library <whatsnew36-pep506>`. 76 77 78CPython implementation improvements: 79 80* The :ref:`dict <typesmapping>` type has been reimplemented to use 81 a :ref:`more compact representation <whatsnew36-compactdict>` 82 based on `a proposal by Raymond Hettinger 83 <https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_ 84 and similar to the `PyPy dict implementation`_. This resulted in dictionaries 85 using 20% to 25% less memory when compared to Python 3.5. 86 87* Customization of class creation has been simplified with the 88 :ref:`new protocol <whatsnew36-pep487>`. 89 90* The class attribute definition order is 91 :ref:`now preserved <whatsnew36-pep520>`. 92 93* The order of elements in ``**kwargs`` now 94 :ref:`corresponds to the order <whatsnew36-pep468>` in which keyword 95 arguments were passed to the function. 96 97* DTrace and SystemTap :ref:`probing support <whatsnew36-tracing>` has 98 been added. 99 100* The new :ref:`PYTHONMALLOC <whatsnew36-pythonmalloc>` environment variable 101 can now be used to debug the interpreter memory allocation and access 102 errors. 103 104 105Significant improvements in the standard library: 106 107* The :mod:`asyncio` module has received new features, significant 108 usability and performance improvements, and a fair amount of bug fixes. 109 Starting with Python 3.6 the ``asyncio`` module is no longer provisional 110 and its API is considered stable. 111 112* A new :ref:`file system path protocol <whatsnew36-pep519>` has been 113 implemented to support :term:`path-like objects <path-like object>`. 114 All standard library functions operating on paths have been updated to 115 work with the new protocol. 116 117* The :mod:`datetime` module has gained support for 118 :ref:`Local Time Disambiguation <whatsnew36-pep495>`. 119 120* The :mod:`typing` module received a number of 121 :ref:`improvements <whatsnew36-typing>`. 122 123* The :mod:`tracemalloc` module has been significantly reworked 124 and is now used to provide better output for :exc:`ResourceWarning` 125 as well as provide better diagnostics for memory allocation errors. 126 See the :ref:`PYTHONMALLOC section <whatsnew36-pythonmalloc>` for more 127 information. 128 129 130Security improvements: 131 132* The new :mod:`secrets` module has been added to simplify the generation of 133 cryptographically strong pseudo-random numbers suitable for 134 managing secrets such as account authentication, tokens, and similar. 135 136* On Linux, :func:`os.urandom` now blocks until the system urandom entropy 137 pool is initialized to increase the security. See the :pep:`524` for the 138 rationale. 139 140* The :mod:`hashlib` and :mod:`ssl` modules now support OpenSSL 1.1.0. 141 142* The default settings and feature set of the :mod:`ssl` module have been 143 improved. 144 145* The :mod:`hashlib` module received support for the BLAKE2, SHA-3 and SHAKE 146 hash algorithms and the :func:`~hashlib.scrypt` key derivation function. 147 148 149Windows improvements: 150 151* :ref:`PEP 528 <whatsnew36-pep529>` and :ref:`PEP 529 <whatsnew36-pep529>`, 152 Windows filesystem and console encoding changed to UTF-8. 153 154* The ``py.exe`` launcher, when used interactively, no longer prefers 155 Python 2 over Python 3 when the user doesn't specify a version (via 156 command line arguments or a config file). Handling of shebang lines 157 remains unchanged - "python" refers to Python 2 in that case. 158 159* ``python.exe`` and ``pythonw.exe`` have been marked as long-path aware, 160 which means that the 260 character path limit may no longer apply. 161 See :ref:`removing the MAX_PATH limitation <max-path>` for details. 162 163* A ``._pth`` file can be added to force isolated mode and fully specify 164 all search paths to avoid registry and environment lookup. See 165 :ref:`the documentation <finding_modules>` for more information. 166 167* A ``python36.zip`` file now works as a landmark to infer 168 :envvar:`PYTHONHOME`. See :ref:`the documentation <finding_modules>` for 169 more information. 170 171 172.. _PyPy dict implementation: https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html 173 174 175New Features 176============ 177 178.. _whatsnew36-pep498: 179 180PEP 498: Formatted string literals 181---------------------------------- 182 183:pep:`498` introduces a new kind of string literals: *f-strings*, or 184:ref:`formatted string literals <f-strings>`. 185 186Formatted string literals are prefixed with ``'f'`` and are similar to 187the format strings accepted by :meth:`str.format`. They contain replacement 188fields surrounded by curly braces. The replacement fields are expressions, 189which are evaluated at run time, and then formatted using the 190:func:`format` protocol:: 191 192 >>> name = "Fred" 193 >>> f"He said his name is {name}." 194 'He said his name is Fred.' 195 >>> width = 10 196 >>> precision = 4 197 >>> value = decimal.Decimal("12.34567") 198 >>> f"result: {value:{width}.{precision}}" # nested fields 199 'result: 12.35' 200 201.. seealso:: 202 203 :pep:`498` -- Literal String Interpolation. 204 PEP written and implemented by Eric V. Smith. 205 206 :ref:`Feature documentation <f-strings>`. 207 208 209.. _whatsnew36-pep526: 210 211PEP 526: Syntax for variable annotations 212---------------------------------------- 213 214:pep:`484` introduced the standard for type annotations of function 215parameters, a.k.a. type hints. This PEP adds syntax to Python for annotating 216the types of variables including class variables and instance variables:: 217 218 primes: List[int] = [] 219 220 captain: str # Note: no initial value! 221 222 class Starship: 223 stats: Dict[str, int] = {} 224 225Just as for function annotations, the Python interpreter does not attach any 226particular meaning to variable annotations and only stores them in the 227``__annotations__`` attribute of a class or module. 228 229In contrast to variable declarations in statically typed languages, 230the goal of annotation syntax is to provide an easy way to specify structured 231type metadata for third party tools and libraries via the abstract syntax tree 232and the ``__annotations__`` attribute. 233 234.. seealso:: 235 236 :pep:`526` -- Syntax for variable annotations. 237 PEP written by Ryan Gonzalez, Philip House, Ivan Levkivskyi, Lisa Roach, 238 and Guido van Rossum. Implemented by Ivan Levkivskyi. 239 240 Tools that use or will use the new syntax: 241 `mypy <http://github.com/python/mypy>`_, 242 `pytype <http://github.com/google/pytype>`_, PyCharm, etc. 243 244 245.. _whatsnew36-pep515: 246 247PEP 515: Underscores in Numeric Literals 248---------------------------------------- 249 250:pep:`515` adds the ability to use underscores in numeric literals for 251improved readability. For example:: 252 253 >>> 1_000_000_000_000_000 254 1000000000000000 255 >>> 0x_FF_FF_FF_FF 256 4294967295 257 258Single underscores are allowed between digits and after any base 259specifier. Leading, trailing, or multiple underscores in a row are not 260allowed. 261 262The :ref:`string formatting <formatspec>` language also now has support 263for the ``'_'`` option to signal the use of an underscore for a thousands 264separator for floating point presentation types and for integer 265presentation type ``'d'``. For integer presentation types ``'b'``, 266``'o'``, ``'x'``, and ``'X'``, underscores will be inserted every 4 267digits:: 268 269 >>> '{:_}'.format(1000000) 270 '1_000_000' 271 >>> '{:_x}'.format(0xFFFFFFFF) 272 'ffff_ffff' 273 274.. seealso:: 275 276 :pep:`515` -- Underscores in Numeric Literals 277 PEP written by Georg Brandl and Serhiy Storchaka. 278 279 280.. _whatsnew36-pep525: 281 282PEP 525: Asynchronous Generators 283-------------------------------- 284 285:pep:`492` introduced support for native coroutines and ``async`` / ``await`` 286syntax to Python 3.5. A notable limitation of the Python 3.5 implementation 287is that it was not possible to use ``await`` and ``yield`` in the same 288function body. In Python 3.6 this restriction has been lifted, making it 289possible to define *asynchronous generators*:: 290 291 async def ticker(delay, to): 292 """Yield numbers from 0 to *to* every *delay* seconds.""" 293 for i in range(to): 294 yield i 295 await asyncio.sleep(delay) 296 297The new syntax allows for faster and more concise code. 298 299.. seealso:: 300 301 :pep:`525` -- Asynchronous Generators 302 PEP written and implemented by Yury Selivanov. 303 304 305.. _whatsnew36-pep530: 306 307PEP 530: Asynchronous Comprehensions 308------------------------------------ 309 310:pep:`530` adds support for using ``async for`` in list, set, dict 311comprehensions and generator expressions:: 312 313 result = [i async for i in aiter() if i % 2] 314 315Additionally, ``await`` expressions are supported in all kinds 316of comprehensions:: 317 318 result = [await fun() for fun in funcs if await condition()] 319 320.. seealso:: 321 322 :pep:`530` -- Asynchronous Comprehensions 323 PEP written and implemented by Yury Selivanov. 324 325 326.. _whatsnew36-pep487: 327 328PEP 487: Simpler customization of class creation 329------------------------------------------------ 330 331It is now possible to customize subclass creation without using a metaclass. 332The new ``__init_subclass__`` classmethod will be called on the base class 333whenever a new subclass is created:: 334 335 class PluginBase: 336 subclasses = [] 337 338 def __init_subclass__(cls, **kwargs): 339 super().__init_subclass__(**kwargs) 340 cls.subclasses.append(cls) 341 342 class Plugin1(PluginBase): 343 pass 344 345 class Plugin2(PluginBase): 346 pass 347 348In order to allow zero-argument :func:`super` calls to work correctly from 349:meth:`~object.__init_subclass__` implementations, custom metaclasses must 350ensure that the new ``__classcell__`` namespace entry is propagated to 351``type.__new__`` (as described in :ref:`class-object-creation`). 352 353.. seealso:: 354 355 :pep:`487` -- Simpler customization of class creation 356 PEP written and implemented by Martin Teichmann. 357 358 :ref:`Feature documentation <class-customization>` 359 360 361.. _whatsnew36-pep487-descriptors: 362 363PEP 487: Descriptor Protocol Enhancements 364----------------------------------------- 365 366:pep:`487` extends the descriptor protocol to include the new optional 367:meth:`~object.__set_name__` method. Whenever a new class is defined, the new 368method will be called on all descriptors included in the definition, providing 369them with a reference to the class being defined and the name given to the 370descriptor within the class namespace. In other words, instances of 371descriptors can now know the attribute name of the descriptor in the 372owner class:: 373 374 class IntField: 375 def __get__(self, instance, owner): 376 return instance.__dict__[self.name] 377 378 def __set__(self, instance, value): 379 if not isinstance(value, int): 380 raise ValueError(f'expecting integer in {self.name}') 381 instance.__dict__[self.name] = value 382 383 # this is the new initializer: 384 def __set_name__(self, owner, name): 385 self.name = name 386 387 class Model: 388 int_field = IntField() 389 390 391.. seealso:: 392 393 :pep:`487` -- Simpler customization of class creation 394 PEP written and implemented by Martin Teichmann. 395 396 :ref:`Feature documentation <descriptors>` 397 398 399.. _whatsnew36-pep519: 400 401PEP 519: Adding a file system path protocol 402------------------------------------------- 403 404File system paths have historically been represented as :class:`str` 405or :class:`bytes` objects. This has led to people who write code which 406operate on file system paths to assume that such objects are only one 407of those two types (an :class:`int` representing a file descriptor 408does not count as that is not a file path). Unfortunately that 409assumption prevents alternative object representations of file system 410paths like :mod:`pathlib` from working with pre-existing code, 411including Python's standard library. 412 413To fix this situation, a new interface represented by 414:class:`os.PathLike` has been defined. By implementing the 415:meth:`~os.PathLike.__fspath__` method, an object signals that it 416represents a path. An object can then provide a low-level 417representation of a file system path as a :class:`str` or 418:class:`bytes` object. This means an object is considered 419:term:`path-like <path-like object>` if it implements 420:class:`os.PathLike` or is a :class:`str` or :class:`bytes` object 421which represents a file system path. Code can use :func:`os.fspath`, 422:func:`os.fsdecode`, or :func:`os.fsencode` to explicitly get a 423:class:`str` and/or :class:`bytes` representation of a path-like 424object. 425 426The built-in :func:`open` function has been updated to accept 427:class:`os.PathLike` objects, as have all relevant functions in the 428:mod:`os` and :mod:`os.path` modules, and most other functions and 429classes in the standard library. The :class:`os.DirEntry` class 430and relevant classes in :mod:`pathlib` have also been updated to 431implement :class:`os.PathLike`. 432 433The hope is that updating the fundamental functions for operating 434on file system paths will lead to third-party code to implicitly 435support all :term:`path-like objects <path-like object>` without any 436code changes, or at least very minimal ones (e.g. calling 437:func:`os.fspath` at the beginning of code before operating on a 438path-like object). 439 440Here are some examples of how the new interface allows for 441:class:`pathlib.Path` to be used more easily and transparently with 442pre-existing code:: 443 444 >>> import pathlib 445 >>> with open(pathlib.Path("README")) as f: 446 ... contents = f.read() 447 ... 448 >>> import os.path 449 >>> os.path.splitext(pathlib.Path("some_file.txt")) 450 ('some_file', '.txt') 451 >>> os.path.join("/a/b", pathlib.Path("c")) 452 '/a/b/c' 453 >>> import os 454 >>> os.fspath(pathlib.Path("some_file.txt")) 455 'some_file.txt' 456 457(Implemented by Brett Cannon, Ethan Furman, Dusty Phillips, and Jelle Zijlstra.) 458 459.. seealso:: 460 461 :pep:`519` -- Adding a file system path protocol 462 PEP written by Brett Cannon and Koos Zevenhoven. 463 464 465.. _whatsnew36-pep495: 466 467PEP 495: Local Time Disambiguation 468---------------------------------- 469 470In most world locations, there have been and will be times when local clocks 471are moved back. In those times, intervals are introduced in which local 472clocks show the same time twice in the same day. In these situations, the 473information displayed on a local clock (or stored in a Python datetime 474instance) is insufficient to identify a particular moment in time. 475 476:pep:`495` adds the new *fold* attribute to instances of 477:class:`datetime.datetime` and :class:`datetime.time` classes to differentiate 478between two moments in time for which local times are the same:: 479 480 >>> u0 = datetime(2016, 11, 6, 4, tzinfo=timezone.utc) 481 >>> for i in range(4): 482 ... u = u0 + i*HOUR 483 ... t = u.astimezone(Eastern) 484 ... print(u.time(), 'UTC =', t.time(), t.tzname(), t.fold) 485 ... 486 04:00:00 UTC = 00:00:00 EDT 0 487 05:00:00 UTC = 01:00:00 EDT 0 488 06:00:00 UTC = 01:00:00 EST 1 489 07:00:00 UTC = 02:00:00 EST 0 490 491The values of the :attr:`fold <datetime.datetime.fold>` attribute have the 492value ``0`` for all instances except those that represent the second 493(chronologically) moment in time in an ambiguous case. 494 495.. seealso:: 496 497 :pep:`495` -- Local Time Disambiguation 498 PEP written by Alexander Belopolsky and Tim Peters, implementation 499 by Alexander Belopolsky. 500 501 502.. _whatsnew36-pep529: 503 504PEP 529: Change Windows filesystem encoding to UTF-8 505---------------------------------------------------- 506 507Representing filesystem paths is best performed with str (Unicode) rather than 508bytes. However, there are some situations where using bytes is sufficient and 509correct. 510 511Prior to Python 3.6, data loss could result when using bytes paths on Windows. 512With this change, using bytes to represent paths is now supported on Windows, 513provided those bytes are encoded with the encoding returned by 514:func:`sys.getfilesystemencoding()`, which now defaults to ``'utf-8'``. 515 516Applications that do not use str to represent paths should use 517:func:`os.fsencode()` and :func:`os.fsdecode()` to ensure their bytes are 518correctly encoded. To revert to the previous behaviour, set 519:envvar:`PYTHONLEGACYWINDOWSFSENCODING` or call 520:func:`sys._enablelegacywindowsfsencoding`. 521 522See :pep:`529` for more information and discussion of code modifications that 523may be required. 524 525 526.. _whatsnew36-pep528: 527 528PEP 528: Change Windows console encoding to UTF-8 529------------------------------------------------- 530 531The default console on Windows will now accept all Unicode characters and 532provide correctly read str objects to Python code. ``sys.stdin``, 533``sys.stdout`` and ``sys.stderr`` now default to utf-8 encoding. 534 535This change only applies when using an interactive console, and not when 536redirecting files or pipes. To revert to the previous behaviour for interactive 537console use, set :envvar:`PYTHONLEGACYWINDOWSIOENCODING`. 538 539.. seealso:: 540 541 :pep:`528` -- Change Windows console encoding to UTF-8 542 PEP written and implemented by Steve Dower. 543 544 545.. _whatsnew36-pep520: 546 547PEP 520: Preserving Class Attribute Definition Order 548---------------------------------------------------- 549 550Attributes in a class definition body have a natural ordering: the same 551order in which the names appear in the source. This order is now 552preserved in the new class's :attr:`~object.__dict__` attribute. 553 554Also, the effective default class *execution* namespace (returned from 555:ref:`type.__prepare__() <prepare>`) is now an insertion-order-preserving 556mapping. 557 558.. seealso:: 559 560 :pep:`520` -- Preserving Class Attribute Definition Order 561 PEP written and implemented by Eric Snow. 562 563 564.. _whatsnew36-pep468: 565 566PEP 468: Preserving Keyword Argument Order 567------------------------------------------ 568 569``**kwargs`` in a function signature is now guaranteed to be an 570insertion-order-preserving mapping. 571 572.. seealso:: 573 574 :pep:`468` -- Preserving Keyword Argument Order 575 PEP written and implemented by Eric Snow. 576 577 578.. _whatsnew36-compactdict: 579 580New :ref:`dict <typesmapping>` implementation 581--------------------------------------------- 582 583The :ref:`dict <typesmapping>` type now uses a "compact" representation 584based on `a proposal by Raymond Hettinger 585<https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_ 586which was `first implemented by PyPy 587<https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html>`_. 588The memory usage of the new :func:`dict` is between 20% and 25% smaller 589compared to Python 3.5. 590 591The order-preserving aspect of this new implementation is considered an 592implementation detail and should not be relied upon (this may change in 593the future, but it is desired to have this new dict implementation in 594the language for a few releases before changing the language spec to mandate 595order-preserving semantics for all current and future Python 596implementations; this also helps preserve backwards-compatibility 597with older versions of the language where random iteration order is 598still in effect, e.g. Python 3.5). 599 600(Contributed by INADA Naoki in :issue:`27350`. Idea 601`originally suggested by Raymond Hettinger 602<https://mail.python.org/pipermail/python-dev/2012-December/123028.html>`_.) 603 604 605.. _whatsnew36-pep523: 606 607PEP 523: Adding a frame evaluation API to CPython 608------------------------------------------------- 609 610While Python provides extensive support to customize how code 611executes, one place it has not done so is in the evaluation of frame 612objects. If you wanted some way to intercept frame evaluation in 613Python there really wasn't any way without directly manipulating 614function pointers for defined functions. 615 616:pep:`523` changes this by providing an API to make frame 617evaluation pluggable at the C level. This will allow for tools such 618as debuggers and JITs to intercept frame evaluation before the 619execution of Python code begins. This enables the use of alternative 620evaluation implementations for Python code, tracking frame 621evaluation, etc. 622 623This API is not part of the limited C API and is marked as private to 624signal that usage of this API is expected to be limited and only 625applicable to very select, low-level use-cases. Semantics of the 626API will change with Python as necessary. 627 628.. seealso:: 629 630 :pep:`523` -- Adding a frame evaluation API to CPython 631 PEP written by Brett Cannon and Dino Viehland. 632 633 634.. _whatsnew36-pythonmalloc: 635 636PYTHONMALLOC environment variable 637--------------------------------- 638 639The new :envvar:`PYTHONMALLOC` environment variable allows setting the Python 640memory allocators and installing debug hooks. 641 642It is now possible to install debug hooks on Python memory allocators on Python 643compiled in release mode using ``PYTHONMALLOC=debug``. Effects of debug hooks: 644 645* Newly allocated memory is filled with the byte ``0xCB`` 646* Freed memory is filled with the byte ``0xDB`` 647* Detect violations of the Python memory allocator API. For example, 648 :c:func:`PyObject_Free` called on a memory block allocated by 649 :c:func:`PyMem_Malloc`. 650* Detect writes before the start of a buffer (buffer underflows) 651* Detect writes after the end of a buffer (buffer overflows) 652* Check that the :term:`GIL <global interpreter lock>` is held when allocator 653 functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and 654 :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called. 655 656Checking if the GIL is held is also a new feature of Python 3.6. 657 658See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python 659memory allocators. 660 661It is now also possible to force the usage of the :c:func:`malloc` allocator of 662the C library for all Python memory allocations using ``PYTHONMALLOC=malloc``. 663This is helpful when using external memory debuggers like Valgrind on 664a Python compiled in release mode. 665 666On error, the debug hooks on Python memory allocators now use the 667:mod:`tracemalloc` module to get the traceback where a memory block was 668allocated. 669 670Example of fatal error on buffer overflow using 671``python3.6 -X tracemalloc=5`` (store 5 frames in traces):: 672 673 Debug memory block at address p=0x7fbcd41666f8: API 'o' 674 4 bytes originally requested 675 The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected. 676 The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb): 677 at tail+0: 0x02 *** OUCH 678 at tail+1: 0xfb 679 at tail+2: 0xfb 680 at tail+3: 0xfb 681 at tail+4: 0xfb 682 at tail+5: 0xfb 683 at tail+6: 0xfb 684 at tail+7: 0xfb 685 The block was made by call #1233329 to debug malloc/realloc. 686 Data at p: 1a 2b 30 00 687 688 Memory block allocated at (most recent call first): 689 File "test/test_bytes.py", line 323 690 File "unittest/case.py", line 600 691 File "unittest/case.py", line 648 692 File "unittest/suite.py", line 122 693 File "unittest/suite.py", line 84 694 695 Fatal Python error: bad trailing pad byte 696 697 Current thread 0x00007fbcdbd32700 (most recent call first): 698 File "test/test_bytes.py", line 323 in test_hex 699 File "unittest/case.py", line 600 in run 700 File "unittest/case.py", line 648 in __call__ 701 File "unittest/suite.py", line 122 in run 702 File "unittest/suite.py", line 84 in __call__ 703 File "unittest/suite.py", line 122 in run 704 File "unittest/suite.py", line 84 in __call__ 705 ... 706 707(Contributed by Victor Stinner in :issue:`26516` and :issue:`26564`.) 708 709 710.. _whatsnew36-tracing: 711 712DTrace and SystemTap probing support 713------------------------------------ 714 715Python can now be built ``--with-dtrace`` which enables static markers 716for the following events in the interpreter: 717 718* function call/return 719 720* garbage collection started/finished 721 722* line of code executed. 723 724This can be used to instrument running interpreters in production, 725without the need to recompile specific debug builds or providing 726application-specific profiling/debugging code. 727 728More details in :ref:`instrumentation`. 729 730The current implementation is tested on Linux and macOS. Additional 731markers may be added in the future. 732 733(Contributed by Łukasz Langa in :issue:`21590`, based on patches by 734Jesús Cea Avión, David Malcolm, and Nikhil Benesch.) 735 736 737Other Language Changes 738====================== 739 740Some smaller changes made to the core Python language are: 741 742* A ``global`` or ``nonlocal`` statement must now textually appear 743 before the first use of the affected name in the same scope. 744 Previously this was a ``SyntaxWarning``. 745 746* It is now possible to set a :ref:`special method <specialnames>` to 747 ``None`` to indicate that the corresponding operation is not available. 748 For example, if a class sets :meth:`__iter__` to ``None``, the class 749 is not iterable. 750 (Contributed by Andrew Barnert and Ivan Levkivskyi in :issue:`25958`.) 751 752* Long sequences of repeated traceback lines are now abbreviated as 753 ``"[Previous line repeated {count} more times]"`` (see 754 :ref:`whatsnew36-traceback` for an example). 755 (Contributed by Emanuel Barry in :issue:`26823`.) 756 757* Import now raises the new exception :exc:`ModuleNotFoundError` 758 (subclass of :exc:`ImportError`) when it cannot find a module. Code 759 that currently checks for ImportError (in try-except) will still work. 760 (Contributed by Eric Snow in :issue:`15767`.) 761 762* Class methods relying on zero-argument ``super()`` will now work correctly 763 when called from metaclass methods during class creation. 764 (Contributed by Martin Teichmann in :issue:`23722`.) 765 766 767New Modules 768=========== 769 770.. _whatsnew36-pep506: 771 772secrets 773------- 774 775The main purpose of the new :mod:`secrets` module is to provide an obvious way 776to reliably generate cryptographically strong pseudo-random values suitable 777for managing secrets, such as account authentication, tokens, and similar. 778 779.. warning:: 780 781 Note that the pseudo-random generators in the :mod:`random` module 782 should *NOT* be used for security purposes. Use :mod:`secrets` 783 on Python 3.6+ and :func:`os.urandom()` on Python 3.5 and earlier. 784 785.. seealso:: 786 787 :pep:`506` -- Adding A Secrets Module To The Standard Library 788 PEP written and implemented by Steven D'Aprano. 789 790 791Improved Modules 792================ 793 794array 795----- 796 797Exhausted iterators of :class:`array.array` will now stay exhausted even 798if the iterated array is extended. This is consistent with the behavior 799of other mutable sequences. 800 801Contributed by Serhiy Storchaka in :issue:`26492`. 802 803ast 804--- 805 806The new :class:`ast.Constant` AST node has been added. It can be used 807by external AST optimizers for the purposes of constant folding. 808 809Contributed by Victor Stinner in :issue:`26146`. 810 811 812asyncio 813------- 814 815Starting with Python 3.6 the ``asyncio`` module is no longer provisional and its 816API is considered stable. 817 818Notable changes in the :mod:`asyncio` module since Python 3.5.0 819(all backported to 3.5.x due to the provisional status): 820 821* The :func:`~asyncio.get_event_loop` function has been changed to 822 always return the currently running loop when called from couroutines 823 and callbacks. 824 (Contributed by Yury Selivanov in :issue:`28613`.) 825 826* The :func:`~asyncio.ensure_future` function and all functions that 827 use it, such as :meth:`loop.run_until_complete() <asyncio.BaseEventLoop.run_until_complete>`, 828 now accept all kinds of :term:`awaitable objects <awaitable>`. 829 (Contributed by Yury Selivanov.) 830 831* New :func:`~asyncio.run_coroutine_threadsafe` function to submit 832 coroutines to event loops from other threads. 833 (Contributed by Vincent Michel.) 834 835* New :meth:`Transport.is_closing() <asyncio.BaseTransport.is_closing>` 836 method to check if the transport is closing or closed. 837 (Contributed by Yury Selivanov.) 838 839* The :meth:`loop.create_server() <asyncio.BaseEventLoop.create_server>` 840 method can now accept a list of hosts. 841 (Contributed by Yann Sionneau.) 842 843* New :meth:`loop.create_future() <asyncio.BaseEventLoop.create_future>` 844 method to create Future objects. This allows alternative event 845 loop implementations, such as 846 `uvloop <https://github.com/MagicStack/uvloop>`_, to provide a faster 847 :class:`asyncio.Future` implementation. 848 (Contributed by Yury Selivanov in :issue:`27041`.) 849 850* New :meth:`loop.get_exception_handler() <asyncio.BaseEventLoop.get_exception_handler>` 851 method to get the current exception handler. 852 (Contributed by Yury Selivanov in :issue:`27040`.) 853 854* New :meth:`StreamReader.readuntil() <asyncio.StreamReader.readuntil>` 855 method to read data from the stream until a separator bytes 856 sequence appears. 857 (Contributed by Mark Korenberg.) 858 859* The performance of :meth:`StreamReader.readexactly() <asyncio.StreamReader.readexactly>` 860 has been improved. 861 (Contributed by Mark Korenberg in :issue:`28370`.) 862 863* The :meth:`loop.getaddrinfo() <asyncio.BaseEventLoop.getaddrinfo>` 864 method is optimized to avoid calling the system ``getaddrinfo`` 865 function if the address is already resolved. 866 (Contributed by A. Jesse Jiryu Davis.) 867 868* The :meth:`loop.stop() <asyncio.BaseEventLoop.stop>` 869 method has been changed to stop the loop immediately after 870 the current iteration. Any new callbacks scheduled as a result 871 of the last iteration will be discarded. 872 (Contributed by Guido van Rossum in :issue:`25593`.) 873 874* :meth:`Future.set_exception <asyncio.futures.Future.set_exception>` 875 will now raise :exc:`TypeError` when passed an instance of 876 the :exc:`StopIteration` exception. 877 (Contributed by Chris Angelico in :issue:`26221`.) 878 879* New :meth:`loop.connect_accepted_socket() <asyncio.BaseEventLoop.connect_accepted_socket>` 880 method to be used by servers that accept connections outside of asyncio, 881 but that use asyncio to handle them. 882 (Contributed by Jim Fulton in :issue:`27392`.) 883 884* ``TCP_NODELAY`` flag is now set for all TCP transports by default. 885 (Contributed by Yury Selivanov in :issue:`27456`.) 886 887* New :meth:`loop.shutdown_asyncgens() <asyncio.AbstractEventLoop.shutdown_asyncgens>` 888 to properly close pending asynchronous generators before closing the 889 loop. 890 (Contributed by Yury Selivanov in :issue:`28003`.) 891 892* :class:`Future <asyncio.Future>` and :class:`Task <asyncio.Task>` 893 classes now have an optimized C implementation which makes asyncio 894 code up to 30% faster. 895 (Contributed by Yury Selivanov and INADA Naoki in :issue:`26081` 896 and :issue:`28544`.) 897 898 899binascii 900-------- 901 902The :func:`~binascii.b2a_base64` function now accepts an optional *newline* 903keyword argument to control whether the newline character is appended to the 904return value. 905(Contributed by Victor Stinner in :issue:`25357`.) 906 907 908cmath 909----- 910 911The new :const:`cmath.tau` (τ) constant has been added. 912(Contributed by Lisa Roach in :issue:`12345`, see :pep:`628` for details.) 913 914New constants: :const:`cmath.inf` and :const:`cmath.nan` to 915match :const:`math.inf` and :const:`math.nan`, and also :const:`cmath.infj` 916and :const:`cmath.nanj` to match the format used by complex repr. 917(Contributed by Mark Dickinson in :issue:`23229`.) 918 919 920collections 921----------- 922 923The new :class:`~collections.abc.Collection` abstract base class has been 924added to represent sized iterable container classes. 925(Contributed by Ivan Levkivskyi, docs by Neil Girdhar in :issue:`27598`.) 926 927The new :class:`~collections.abc.Reversible` abstract base class represents 928iterable classes that also provide the :meth:`__reversed__` method. 929(Contributed by Ivan Levkivskyi in :issue:`25987`.) 930 931The new :class:`~collections.abc.AsyncGenerator` abstract base class represents 932asynchronous generators. 933(Contributed by Yury Selivanov in :issue:`28720`.) 934 935The :func:`~collections.namedtuple` function now accepts an optional 936keyword argument *module*, which, when specified, is used for 937the ``__module__`` attribute of the returned named tuple class. 938(Contributed by Raymond Hettinger in :issue:`17941`.) 939 940The *verbose* and *rename* arguments for 941:func:`~collections.namedtuple` are now keyword-only. 942(Contributed by Raymond Hettinger in :issue:`25628`.) 943 944Recursive :class:`collections.deque` instances can now be pickled. 945(Contributed by Serhiy Storchaka in :issue:`26482`.) 946 947 948concurrent.futures 949------------------ 950 951The :class:`ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor>` 952class constructor now accepts an optional *thread_name_prefix* argument 953to make it possible to customize the names of the threads created by the 954pool. 955(Contributed by Gregory P. Smith in :issue:`27664`.) 956 957 958contextlib 959---------- 960 961The :class:`contextlib.AbstractContextManager` class has been added to 962provide an abstract base class for context managers. It provides a 963sensible default implementation for `__enter__()` which returns 964``self`` and leaves `__exit__()` an abstract method. A matching 965class has been added to the :mod:`typing` module as 966:class:`typing.ContextManager`. 967(Contributed by Brett Cannon in :issue:`25609`.) 968 969 970datetime 971-------- 972 973The :class:`~datetime.datetime` and :class:`~datetime.time` classes have 974the new :attr:`~time.fold` attribute used to disambiguate local time 975when necessary. Many functions in the :mod:`datetime` have been 976updated to support local time disambiguation. 977See :ref:`Local Time Disambiguation <whatsnew36-pep495>` section for more 978information. 979(Contributed by Alexander Belopolsky in :issue:`24773`.) 980 981The :meth:`datetime.strftime() <datetime.datetime.strftime>` and 982:meth:`date.strftime() <datetime.date.strftime>` methods now support 983ISO 8601 date directives ``%G``, ``%u`` and ``%V``. 984(Contributed by Ashley Anderson in :issue:`12006`.) 985 986The :func:`datetime.isoformat() <datetime.datetime.isoformat>` function 987now accepts an optional *timespec* argument that specifies the number 988of additional components of the time value to include. 989(Contributed by Alessandro Cucci and Alexander Belopolsky in :issue:`19475`.) 990 991The :meth:`datetime.combine() <datetime.datetime.combine>` now 992accepts an optional *tzinfo* argument. 993(Contributed by Alexander Belopolsky in :issue:`27661`.) 994 995 996decimal 997------- 998 999New :meth:`Decimal.as_integer_ratio() <decimal.Decimal.as_integer_ratio>` 1000method that returns a pair ``(n, d)`` of integers that represent the given 1001:class:`~decimal.Decimal` instance as a fraction, in lowest terms and 1002with a positive denominator:: 1003 1004 >>> Decimal('-3.14').as_integer_ratio() 1005 (-157, 50) 1006 1007(Contributed by Stefan Krah amd Mark Dickinson in :issue:`25928`.) 1008 1009 1010 1011distutils 1012--------- 1013 1014The ``default_format`` attribute has been removed from 1015:class:`distutils.command.sdist.sdist` and the ``formats`` 1016attribute defaults to ``['gztar']``. Although not anticipated, 1017any code relying on the presence of ``default_format`` may 1018need to be adapted. See :issue:`27819` for more details. 1019 1020 1021email 1022----- 1023 1024The new email API, enabled via the *policy* keyword to various constructors, is 1025no longer provisional. The :mod:`email` documentation has been reorganized and 1026rewritten to focus on the new API, while retaining the old documentation for 1027the legacy API. (Contributed by R. David Murray in :issue:`24277`.) 1028 1029The :mod:`email.mime` classes now all accept an optional *policy* keyword. 1030(Contributed by Berker Peksag in :issue:`27331`.) 1031 1032The :class:`~email.generator.DecodedGenerator` now supports the *policy* 1033keyword. 1034 1035There is a new :mod:`~email.policy` attribute, 1036:attr:`~email.policy.Policy.message_factory`, that controls what class is used 1037by default when the parser creates new message objects. For the 1038:attr:`email.policy.compat32` policy this is :class:`~email.message.Message`, 1039for the new policies it is :class:`~email.message.EmailMessage`. 1040(Contributed by R. David Murray in :issue:`20476`.) 1041 1042 1043encodings 1044--------- 1045 1046On Windows, added the ``'oem'`` encoding to use ``CP_OEMCP``, and the ``'ansi'`` 1047alias for the existing ``'mbcs'`` encoding, which uses the ``CP_ACP`` code page. 1048(Contributed by Steve Dower in :issue:`27959`.) 1049 1050 1051enum 1052---- 1053 1054Two new enumeration base classes have been added to the :mod:`enum` module: 1055:class:`~enum.Flag` and :class:`~enum.IntFlags`. Both are used to define 1056constants that can be combined using the bitwise operators. 1057(Contributed by Ethan Furman in :issue:`23591`.) 1058 1059Many standard library modules have been updated to use the 1060:class:`~enum.IntFlags` class for their constants. 1061 1062The new :class:`enum.auto` value can be used to assign values to enum 1063members automatically:: 1064 1065 >>> from enum import Enum, auto 1066 >>> class Color(Enum): 1067 ... red = auto() 1068 ... blue = auto() 1069 ... green = auto() 1070 ... 1071 >>> list(Color) 1072 [<Color.red: 1>, <Color.blue: 2>, <Color.green: 3>] 1073 1074 1075faulthandler 1076------------ 1077 1078On Windows, the :mod:`faulthandler` module now installs a handler for Windows 1079exceptions: see :func:`faulthandler.enable`. (Contributed by Victor Stinner in 1080:issue:`23848`.) 1081 1082 1083fileinput 1084--------- 1085 1086:func:`~fileinput.hook_encoded` now supports the *errors* argument. 1087(Contributed by Joseph Hackman in :issue:`25788`.) 1088 1089 1090hashlib 1091------- 1092 1093:mod:`hashlib` supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. 1094(Contributed by Christian Heimes in :issue:`26470`.) 1095 1096BLAKE2 hash functions were added to the module. :func:`~hashlib.blake2b` 1097and :func:`~hashlib.blake2s` are always available and support the full 1098feature set of BLAKE2. 1099(Contributed by Christian Heimes in :issue:`26798` based on code by 1100Dmitry Chestnykh and Samuel Neves. Documentation written by Dmitry Chestnykh.) 1101 1102The SHA-3 hash functions :func:`~hashlib.sha3_224`, :func:`~hashlib.sha3_256`, 1103:func:`~hashlib.sha3_384`, :func:`~hashlib.sha3_512`, and SHAKE hash functions 1104:func:`~hashlib.shake_128` and :func:`~hashlib.shake_256` were added. 1105(Contributed by Christian Heimes in :issue:`16113`. Keccak Code Package 1106by Guido Bertoni, Joan Daemen, Michaël Peeters, Gilles Van Assche, and 1107Ronny Van Keer.) 1108 1109The password-based key derivation function :func:`~hashlib.scrypt` is now 1110available with OpenSSL 1.1.0 and newer. 1111(Contributed by Christian Heimes in :issue:`27928`.) 1112 1113http.client 1114----------- 1115 1116:meth:`HTTPConnection.request() <http.client.HTTPConnection.request>` and 1117:meth:`~http.client.HTTPConnection.endheaders` both now support 1118chunked encoding request bodies. 1119(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) 1120 1121 1122idlelib and IDLE 1123---------------- 1124 1125The idlelib package is being modernized and refactored to make IDLE look and 1126work better and to make the code easier to understand, test, and improve. Part 1127of making IDLE look better, especially on Linux and Mac, is using ttk widgets, 1128mostly in the dialogs. As a result, IDLE no longer runs with tcl/tk 8.4. It 1129now requires tcl/tk 8.5 or 8.6. We recommend running the latest release of 1130either. 1131 1132'Modernizing' includes renaming and consolidation of idlelib modules. The 1133renaming of files with partial uppercase names is similar to the renaming of, 1134for instance, Tkinter and TkFont to tkinter and tkinter.font in 3.0. As a 1135result, imports of idlelib files that worked in 3.5 will usually not work in 11363.6. At least a module name change will be needed (see idlelib/README.txt), 1137sometimes more. (Name changes contributed by Al Swiegart and Terry Reedy in 1138:issue:`24225`. Most idlelib patches since have been and will be part of the 1139process.) 1140 1141In compensation, the eventual result with be that some idlelib classes will be 1142easier to use, with better APIs and docstrings explaining them. Additional 1143useful information will be added to idlelib when available. 1144 1145 1146importlib 1147--------- 1148 1149Import now raises the new exception :exc:`ModuleNotFoundError` 1150(subclass of :exc:`ImportError`) when it cannot find a module. Code 1151that current checks for ``ImportError`` (in try-except) will still work. 1152(Contributed by Eric Snow in :issue:`15767`.) 1153 1154:class:`importlib.util.LazyLoader` now calls 1155:meth:`~importlib.abc.Loader.create_module` on the wrapped loader, removing the 1156restriction that :class:`importlib.machinery.BuiltinImporter` and 1157:class:`importlib.machinery.ExtensionFileLoader` couldn't be used with 1158:class:`importlib.util.LazyLoader`. 1159 1160:func:`importlib.util.cache_from_source`, 1161:func:`importlib.util.source_from_cache`, and 1162:func:`importlib.util.spec_from_file_location` now accept a 1163:term:`path-like object`. 1164 1165 1166inspect 1167------- 1168 1169The :func:`inspect.signature() <inspect.signature>` function now reports the 1170implicit ``.0`` parameters generated by the compiler for comprehension and 1171generator expression scopes as if they were positional-only parameters called 1172``implicit0``. (Contributed by Jelle Zijlstra in :issue:`19611`.) 1173 1174To reduce code churn when upgrading from Python 2.7 and the legacy 1175:func:`inspect.getargspec` API, the previously documented deprecation of 1176:func:`inspect.getfullargspec` has been reversed. While this function is 1177convenient for single/source Python 2/3 code bases, the richer 1178:func:`inspect.signature` interface remains the recommended approach for new 1179code. (Contributed by Nick Coghlan in :issue:`27172`) 1180 1181 1182json 1183---- 1184 1185:func:`json.load` and :func:`json.loads` now support binary input. Encoded 1186JSON should be represented using either UTF-8, UTF-16, or UTF-32. 1187(Contributed by Serhiy Storchaka in :issue:`17909`.) 1188 1189 1190logging 1191------- 1192 1193The new :meth:`WatchedFileHandler.reopenIfNeeded() <logging.handlers.WatchedFileHandler.reopenIfNeeded>` 1194method has been added to add the ability to check if the log file needs to 1195be reopened. 1196(Contributed by Marian Horban in :issue:`24884`.) 1197 1198 1199math 1200---- 1201 1202The tau (τ) constant has been added to the :mod:`math` and :mod:`cmath` 1203modules. 1204(Contributed by Lisa Roach in :issue:`12345`, see :pep:`628` for details.) 1205 1206 1207multiprocessing 1208--------------- 1209 1210:ref:`Proxy Objects <multiprocessing-proxy_objects>` returned by 1211:func:`multiprocessing.Manager` can now be nested. 1212(Contributed by Davin Potts in :issue:`6766`.) 1213 1214 1215os 1216-- 1217 1218See the summary of :ref:`PEP 519 <whatsnew36-pep519>` for details on how the 1219:mod:`os` and :mod:`os.path` modules now support 1220:term:`path-like objects <path-like object>`. 1221 1222:func:`~os.scandir` now supports :class:`bytes` paths on Windows. 1223 1224A new :meth:`~os.scandir.close` method allows explicitly closing a 1225:func:`~os.scandir` iterator. The :func:`~os.scandir` iterator now 1226supports the :term:`context manager` protocol. If a :func:`scandir` 1227iterator is neither exhausted nor explicitly closed a :exc:`ResourceWarning` 1228will be emitted in its destructor. 1229(Contributed by Serhiy Storchaka in :issue:`25994`.) 1230 1231On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool 1232is initialized to increase the security. See the :pep:`524` for the rationale. 1233 1234The Linux ``getrandom()`` syscall (get random bytes) is now exposed as the new 1235:func:`os.getrandom` function. 1236(Contributed by Victor Stinner, part of the :pep:`524`) 1237 1238 1239pathlib 1240------- 1241 1242:mod:`pathlib` now supports :term:`path-like objects <path-like object>`. 1243(Contributed by Brett Cannon in :issue:`27186`.) 1244 1245See the summary of :ref:`PEP 519 <whatsnew36-pep519>` for details. 1246 1247 1248pdb 1249--- 1250 1251The :class:`~pdb.Pdb` class constructor has a new optional *readrc* argument 1252to control whether ``.pdbrc`` files should be read. 1253 1254 1255pickle 1256------ 1257 1258Objects that need ``__new__`` called with keyword arguments can now be pickled 1259using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. 1260Protocol version 4 already supports this case. (Contributed by Serhiy 1261Storchaka in :issue:`24164`.) 1262 1263 1264pickletools 1265----------- 1266 1267:func:`pickletools.dis()` now outputs the implicit memo index for the 1268``MEMOIZE`` opcode. 1269(Contributed by Serhiy Storchaka in :issue:`25382`.) 1270 1271 1272pydoc 1273----- 1274 1275The :mod:`pydoc` module has learned to respect the ``MANPAGER`` 1276environment variable. 1277(Contributed by Matthias Klose in :issue:`8637`.) 1278 1279:func:`help` and :mod:`pydoc` can now list named tuple fields in the 1280order they were defined rather than alphabetically. 1281(Contributed by Raymond Hettinger in :issue:`24879`.) 1282 1283 1284random 1285------- 1286 1287The new :func:`~random.choices` function returns a list of elements of 1288specified size from the given population with optional weights. 1289(Contributed by Raymond Hettinger in :issue:`18844`.) 1290 1291 1292re 1293-- 1294 1295Added support of modifier spans in regular expressions. Examples: 1296``'(?i:p)ython'`` matches ``'python'`` and ``'Python'``, but not ``'PYTHON'``; 1297``'(?i)g(?-i:v)r'`` matches ``'GvR'`` and ``'gvr'``, but not ``'GVR'``. 1298(Contributed by Serhiy Storchaka in :issue:`433028`.) 1299 1300Match object groups can be accessed by ``__getitem__``, which is 1301equivalent to ``group()``. So ``mo['name']`` is now equivalent to 1302``mo.group('name')``. (Contributed by Eric Smith in :issue:`24454`.) 1303 1304:class:`~re.Match` objects now support 1305:meth:`index-like objects <object.__index__>` as group 1306indices. 1307(Contributed by Jeroen Demeyer and Xiang Zhang in :issue:`27177`.) 1308 1309 1310readline 1311-------- 1312 1313Added :func:`~readline.set_auto_history` to enable or disable 1314automatic addition of input to the history list. (Contributed by 1315Tyler Crompton in :issue:`26870`.) 1316 1317 1318rlcompleter 1319----------- 1320 1321Private and special attribute names now are omitted unless the prefix starts 1322with underscores. A space or a colon is added after some completed keywords. 1323(Contributed by Serhiy Storchaka in :issue:`25011` and :issue:`25209`.) 1324 1325 1326shlex 1327----- 1328 1329The :class:`~shlex.shlex` has much 1330:ref:`improved shell compatibility <improved-shell-compatibility>` 1331through the new *punctuation_chars* argument to control which characters 1332are treated as punctuation. 1333(Contributed by Vinay Sajip in :issue:`1521950`.) 1334 1335 1336site 1337---- 1338 1339When specifying paths to add to :attr:`sys.path` in a `.pth` file, 1340you may now specify file paths on top of directories (e.g. zip files). 1341(Contributed by Wolfgang Langner in :issue:`26587`). 1342 1343 1344sqlite3 1345------- 1346 1347:attr:`sqlite3.Cursor.lastrowid` now supports the ``REPLACE`` statement. 1348(Contributed by Alex LordThorsen in :issue:`16864`.) 1349 1350 1351socket 1352------ 1353 1354The :func:`~socket.socket.ioctl` function now supports the 1355:data:`~socket.SIO_LOOPBACK_FAST_PATH` control code. 1356(Contributed by Daniel Stokes in :issue:`26536`.) 1357 1358The :meth:`~socket.socket.getsockopt` constants ``SO_DOMAIN``, 1359``SO_PROTOCOL``, ``SO_PEERSEC``, and ``SO_PASSSEC`` are now supported. 1360(Contributed by Christian Heimes in :issue:`26907`.) 1361 1362The :meth:`~socket.socket.setsockopt` now supports the 1363``setsockopt(level, optname, None, optlen: int)`` form. 1364(Contributed by Christian Heimes in :issue:`27744`.) 1365 1366The socket module now supports the address family 1367:data:`~socket.AF_ALG` to interface with Linux Kernel crypto API. ``ALG_*``, 1368``SOL_ALG`` and :meth:`~socket.socket.sendmsg_afalg` were added. 1369(Contributed by Christian Heimes in :issue:`27744` with support from 1370Victor Stinner.) 1371 1372New Linux constants ``TCP_USER_TIMEOUT`` and ``TCP_CONGESTION`` were added. 1373(Contributed by Omar Sandoval, issue:`26273`). 1374 1375 1376socketserver 1377------------ 1378 1379Servers based on the :mod:`socketserver` module, including those 1380defined in :mod:`http.server`, :mod:`xmlrpc.server` and 1381:mod:`wsgiref.simple_server`, now support the :term:`context manager` 1382protocol. 1383(Contributed by Aviv Palivoda in :issue:`26404`.) 1384 1385The :attr:`~socketserver.StreamRequestHandler.wfile` attribute of 1386:class:`~socketserver.StreamRequestHandler` classes now implements 1387the :class:`io.BufferedIOBase` writable interface. In particular, 1388calling :meth:`~io.BufferedIOBase.write` is now guaranteed to send the 1389data in full. (Contributed by Martin Panter in :issue:`26721`.) 1390 1391 1392ssl 1393--- 1394 1395:mod:`ssl` supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. 1396(Contributed by Christian Heimes in :issue:`26470`.) 1397 13983DES has been removed from the default cipher suites and ChaCha20 Poly1305 1399cipher suites have been added. 1400(Contributed by Christian Heimes in :issue:`27850` and :issue:`27766`.) 1401 1402:class:`~ssl.SSLContext` has better default configuration for options 1403and ciphers. 1404(Contributed by Christian Heimes in :issue:`28043`.) 1405 1406SSL session can be copied from one client-side connection to another 1407with the new :class:`~ssl.SSLSession` class. TLS session resumption can 1408speed up the initial handshake, reduce latency and improve performance 1409(Contributed by Christian Heimes in :issue:`19500` based on a draft by 1410Alex Warhawk.) 1411 1412The new :meth:`~ssl.SSLContext.get_ciphers` method can be used to 1413get a list of enabled ciphers in order of cipher priority. 1414 1415All constants and flags have been converted to :class:`~enum.IntEnum` and 1416:class:`~enum.IntFlags`. 1417(Contributed by Christian Heimes in :issue:`28025`.) 1418 1419Server and client-side specific TLS protocols for :class:`~ssl.SSLContext` 1420were added. 1421(Contributed by Christian Heimes in :issue:`28085`.) 1422 1423 1424statistics 1425---------- 1426 1427A new :func:`~statistics.harmonic_mean` function has been added. 1428(Contributed by Steven D'Aprano in :issue:`27181`.) 1429 1430 1431struct 1432------ 1433 1434:mod:`struct` now supports IEEE 754 half-precision floats via the ``'e'`` 1435format specifier. 1436(Contributed by Eli Stevens, Mark Dickinson in :issue:`11734`.) 1437 1438 1439subprocess 1440---------- 1441 1442:class:`subprocess.Popen` destructor now emits a :exc:`ResourceWarning` warning 1443if the child process is still running. Use the context manager protocol (``with 1444proc: ...``) or explicitly call the :meth:`~subprocess.Popen.wait` method to 1445read the exit status of the child process. (Contributed by Victor Stinner in 1446:issue:`26741`.) 1447 1448The :class:`subprocess.Popen` constructor and all functions that pass arguments 1449through to it now accept *encoding* and *errors* arguments. Specifying either 1450of these will enable text mode for the *stdin*, *stdout* and *stderr* streams. 1451(Contributed by Steve Dower in :issue:`6135`.) 1452 1453 1454sys 1455--- 1456 1457The new :func:`~sys.getfilesystemencodeerrors` function returns the name of 1458the error mode used to convert between Unicode filenames and bytes filenames. 1459(Contributed by Steve Dower in :issue:`27781`.) 1460 1461On Windows the return value of the :func:`~sys.getwindowsversion` function 1462now includes the *platform_version* field which contains the accurate major 1463version, minor version and build number of the current operating system, 1464rather than the version that is being emulated for the process 1465(Contributed by Steve Dower in :issue:`27932`.) 1466 1467 1468telnetlib 1469--------- 1470 1471:class:`~telnetlib.Telnet` is now a context manager (contributed by 1472Stéphane Wirtel in :issue:`25485`). 1473 1474 1475time 1476---- 1477 1478The :class:`~time.struct_time` attributes :attr:`tm_gmtoff` and 1479:attr:`tm_zone` are now available on all platforms. 1480 1481 1482timeit 1483------ 1484 1485The new :meth:`Timer.autorange() <timeit.Timer.autorange>` convenience 1486method has been added to call :meth:`Timer.timeit() <timeit.Timer.timeit>` 1487repeatedly so that the total run time is greater or equal to 200 milliseconds. 1488(Contributed by Steven D'Aprano in :issue:`6422`.) 1489 1490:mod:`timeit` now warns when there is substantial (4x) variance 1491between best and worst times. 1492(Contributed by Serhiy Storchaka in :issue:`23552`.) 1493 1494 1495tkinter 1496------- 1497 1498Added methods :meth:`~tkinter.Variable.trace_add`, 1499:meth:`~tkinter.Variable.trace_remove` and :meth:`~tkinter.Variable.trace_info` 1500in the :class:`tkinter.Variable` class. They replace old methods 1501:meth:`~tkinter.Variable.trace_variable`, :meth:`~tkinter.Variable.trace`, 1502:meth:`~tkinter.Variable.trace_vdelete` and 1503:meth:`~tkinter.Variable.trace_vinfo` that use obsolete Tcl commands and might 1504not work in future versions of Tcl. 1505(Contributed by Serhiy Storchaka in :issue:`22115`). 1506 1507 1508.. _whatsnew36-traceback: 1509 1510traceback 1511--------- 1512 1513Both the traceback module and the interpreter's builtin exception display now 1514abbreviate long sequences of repeated lines in tracebacks as shown in the 1515following example:: 1516 1517 >>> def f(): f() 1518 ... 1519 >>> f() 1520 Traceback (most recent call last): 1521 File "<stdin>", line 1, in <module> 1522 File "<stdin>", line 1, in f 1523 File "<stdin>", line 1, in f 1524 File "<stdin>", line 1, in f 1525 [Previous line repeated 995 more times] 1526 RecursionError: maximum recursion depth exceeded 1527 1528(Contributed by Emanuel Barry in :issue:`26823`.) 1529 1530 1531tracemalloc 1532----------- 1533 1534The :mod:`tracemalloc` module now supports tracing memory allocations in 1535multiple different address spaces. 1536 1537The new :class:`~tracemalloc.DomainFilter` filter class has been added 1538to filter block traces by their address space (domain). 1539 1540(Contributed by Victor Stinner in :issue:`26588`.) 1541 1542 1543.. _whatsnew36-typing: 1544 1545typing 1546------ 1547 1548Since the :mod:`typing` module is :term:`provisional <provisional api>`, 1549all changes introduced in Python 3.6 have also been 1550backported to Python 3.5.x. 1551 1552The :mod:`typing` module has a much improved support for generic type 1553aliases. For example ``Dict[str, Tuple[S, T]]`` is now a valid 1554type annotation. 1555(Contributed by Guido van Rossum in `Github #195 1556<https://github.com/python/typing/pull/195>`_.) 1557 1558The :class:`typing.ContextManager` class has been added for 1559representing :class:`contextlib.AbstractContextManager`. 1560(Contributed by Brett Cannon in :issue:`25609`.) 1561 1562The :class:`typing.Collection` class has been added for 1563representing :class:`collections.abc.Collection`. 1564(Contributed by Ivan Levkivskyi in :issue:`27598`.) 1565 1566The :const:`typing.ClassVar` type construct has been added to 1567mark class variables. As introduced in :pep:`526`, a variable annotation 1568wrapped in ClassVar indicates that a given attribute is intended to be used as 1569a class variable and should not be set on instances of that class. 1570(Contributed by Ivan Levkivskyi in `Github #280 1571<https://github.com/python/typing/issues/280>`_.) 1572 1573A new :const:`~typing.TYPE_CHECKING` constant that is assumed to be 1574``True`` by the static type chekers, but is ``False`` at runtime. 1575(Contributed by Guido van Rossum in `Github #230 1576<https://github.com/python/typing/issues/230>`_.) 1577 1578A new :func:`~typing.NewType` helper function has been added to create 1579lightweight distinct types for annotations:: 1580 1581 from typing import NewType 1582 1583 UserId = NewType('UserId', int) 1584 some_id = UserId(524313) 1585 1586The static type checker will treat the new type as if it were a subclass 1587of the original type. (Contributed by Ivan Levkivskyi in `Github #189 1588<https://github.com/python/typing/issues/189>`_.) 1589 1590 1591unicodedata 1592----------- 1593 1594The :mod:`unicodedata` module now uses data from `Unicode 9.0.0 1595<http://unicode.org/versions/Unicode9.0.0/>`_. 1596(Contributed by Benjamin Peterson.) 1597 1598 1599unittest.mock 1600------------- 1601 1602The :class:`~unittest.mock.Mock` class has the following improvements: 1603 1604* Two new methods, :meth:`Mock.assert_called() 1605 <unittest.mock.Mock.assert_called>` and :meth:`Mock.assert_called_once() 1606 <unittest.mock.Mock.assert_called_once>` to check if the mock object 1607 was called. 1608 (Contributed by Amit Saha in :issue:`26323`.) 1609 1610* The :meth:`Mock.reset_mock() <unittest.mock.Mock.reset_mock>` method 1611 now has two optional keyword only arguments: *return_value* and 1612 *side_effect*. 1613 (Contributed by Kushal Das in :issue:`21271`.) 1614 1615 1616urllib.request 1617-------------- 1618 1619If a HTTP request has a file or iterable body (other than a 1620bytes object) but no ``Content-Length`` header, rather than 1621throwing an error, :class:`~urllib.request.AbstractHTTPHandler` now 1622falls back to use chunked transfer encoding. 1623(Contributed by Demian Brecht and Rolf Krahl in :issue:`12319`.) 1624 1625 1626urllib.robotparser 1627------------------ 1628 1629:class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and 1630``Request-rate`` extensions. 1631(Contributed by Nikolay Bogoychev in :issue:`16099`.) 1632 1633 1634venv 1635---- 1636 1637:mod:`venv` accepts a new parameter ``--prompt``. This parameter provides an 1638alternative prefix for the virtual environment. (Proposed by Łukasz Balcerzak 1639and ported to 3.6 by Stéphane Wirtel in :issue:`22829`.) 1640 1641 1642warnings 1643-------- 1644 1645A new optional *source* parameter has been added to the 1646:func:`warnings.warn_explicit` function: the destroyed object which emitted a 1647:exc:`ResourceWarning`. A *source* attribute has also been added to 1648:class:`warnings.WarningMessage` (contributed by Victor Stinner in 1649:issue:`26568` and :issue:`26567`). 1650 1651When a :exc:`ResourceWarning` warning is logged, the :mod:`tracemalloc` module is now 1652used to try to retrieve the traceback where the destroyed object was allocated. 1653 1654Example with the script ``example.py``:: 1655 1656 import warnings 1657 1658 def func(): 1659 return open(__file__) 1660 1661 f = func() 1662 f = None 1663 1664Output of the command ``python3.6 -Wd -X tracemalloc=5 example.py``:: 1665 1666 example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'> 1667 f = None 1668 Object allocated at (most recent call first): 1669 File "example.py", lineno 4 1670 return open(__file__) 1671 File "example.py", lineno 6 1672 f = func() 1673 1674The "Object allocated at" traceback is new and is only displayed if 1675:mod:`tracemalloc` is tracing Python memory allocations and if the 1676:mod:`warnings` module was already imported. 1677 1678 1679winreg 1680------ 1681 1682Added the 64-bit integer type :data:`REG_QWORD <winreg.REG_QWORD>`. 1683(Contributed by Clement Rouault in :issue:`23026`.) 1684 1685 1686winsound 1687-------- 1688 1689Allowed keyword arguments to be passed to :func:`Beep <winsound.Beep>`, 1690:func:`MessageBeep <winsound.MessageBeep>`, and :func:`PlaySound 1691<winsound.PlaySound>` (:issue:`27982`). 1692 1693 1694xmlrpc.client 1695------------- 1696 1697The :mod:`xmlrpc.client` module now supports unmarshalling 1698additional data types used by the Apache XML-RPC implementation 1699for numerics and ``None``. 1700(Contributed by Serhiy Storchaka in :issue:`26885`.) 1701 1702 1703zipfile 1704------- 1705 1706A new :meth:`ZipInfo.from_file() <zipfile.ZipInfo.from_file>` class method 1707allows making a :class:`~zipfile.ZipInfo` instance from a filesystem file. 1708A new :meth:`ZipInfo.is_dir() <zipfile.ZipInfo.is_dir>` method can be used 1709to check if the :class:`~zipfile.ZipInfo` instance represents a directory. 1710(Contributed by Thomas Kluyver in :issue:`26039`.) 1711 1712The :meth:`ZipFile.open() <zipfile.ZipFile.open>` method can now be used to 1713write data into a ZIP file, as well as for extracting data. 1714(Contributed by Thomas Kluyver in :issue:`26039`.) 1715 1716 1717zlib 1718---- 1719 1720The :func:`~zlib.compress` and :func:`~zlib.decompress` functions now accept 1721keyword arguments. 1722(Contributed by Aviv Palivoda in :issue:`26243` and 1723Xiang Zhang in :issue:`16764` respectively.) 1724 1725 1726Optimizations 1727============= 1728 1729* The Python interpreter now uses a 16-bit wordcode instead of bytecode which 1730 made a number of opcode optimizations possible. 1731 (Contributed by Demur Rumed with input and reviews from 1732 Serhiy Storchaka and Victor Stinner in :issue:`26647` and :issue:`28050`.) 1733 1734* The :class:`asyncio.Future` class now has an optimized C implementation. 1735 (Contributed by Yury Selivanov and INADA Naoki in :issue:`26081`.) 1736 1737* The :class:`asyncio.Task` class now has an optimized 1738 C implementation. (Contributed by Yury Selivanov in :issue:`28544`.) 1739 1740* Various implementation improvements in the :mod:`typing` module 1741 (such as caching of generic types) allow up to 30 times performance 1742 improvements and reduced memory footprint. 1743 1744* The ASCII decoder is now up to 60 times as fast for error handlers 1745 ``surrogateescape``, ``ignore`` and ``replace`` (Contributed 1746 by Victor Stinner in :issue:`24870`). 1747 1748* The ASCII and the Latin1 encoders are now up to 3 times as fast for the 1749 error handler ``surrogateescape`` 1750 (Contributed by Victor Stinner in :issue:`25227`). 1751 1752* The UTF-8 encoder is now up to 75 times as fast for error handlers 1753 ``ignore``, ``replace``, ``surrogateescape``, ``surrogatepass`` (Contributed 1754 by Victor Stinner in :issue:`25267`). 1755 1756* The UTF-8 decoder is now up to 15 times as fast for error handlers 1757 ``ignore``, ``replace`` and ``surrogateescape`` (Contributed 1758 by Victor Stinner in :issue:`25301`). 1759 1760* ``bytes % args`` is now up to 2 times faster. (Contributed by Victor Stinner 1761 in :issue:`25349`). 1762 1763* ``bytearray % args`` is now between 2.5 and 5 times faster. (Contributed by 1764 Victor Stinner in :issue:`25399`). 1765 1766* Optimize :meth:`bytes.fromhex` and :meth:`bytearray.fromhex`: they are now 1767 between 2x and 3.5x faster. (Contributed by Victor Stinner in :issue:`25401`). 1768 1769* Optimize ``bytes.replace(b'', b'.')`` and ``bytearray.replace(b'', b'.')``: 1770 up to 80% faster. (Contributed by Josh Snider in :issue:`26574`). 1771 1772* Allocator functions of the :c:func:`PyMem_Malloc` domain 1773 (:c:data:`PYMEM_DOMAIN_MEM`) now use the :ref:`pymalloc memory allocator 1774 <pymalloc>` instead of :c:func:`malloc` function of the C library. The 1775 pymalloc allocator is optimized for objects smaller or equal to 512 bytes 1776 with a short lifetime, and use :c:func:`malloc` for larger memory blocks. 1777 (Contributed by Victor Stinner in :issue:`26249`). 1778 1779* :func:`pickle.load` and :func:`pickle.loads` are now up to 10% faster when 1780 deserializing many small objects (Contributed by Victor Stinner in 1781 :issue:`27056`). 1782 1783* Passing :term:`keyword arguments <keyword argument>` to a function has an 1784 overhead in comparison with passing :term:`positional arguments 1785 <positional argument>`. Now in extension functions implemented with using 1786 Argument Clinic this overhead is significantly decreased. 1787 (Contributed by Serhiy Storchaka in :issue:`27574`). 1788 1789* Optimized :func:`~glob.glob` and :func:`~glob.iglob` functions in the 1790 :mod:`glob` module; they are now about 3--6 times faster. 1791 (Contributed by Serhiy Storchaka in :issue:`25596`). 1792 1793* Optimized globbing in :mod:`pathlib` by using :func:`os.scandir`; 1794 it is now about 1.5--4 times faster. 1795 (Contributed by Serhiy Storchaka in :issue:`26032`). 1796 1797* :class:`xml.etree.ElementTree` parsing, iteration and deepcopy performance 1798 has been significantly improved. 1799 (Contributed by Serhiy Storchaka in :issue:`25638`, :issue:`25873`, 1800 and :issue:`25869`.) 1801 1802* Creation of :class:`fractions.Fraction` instances from floats and 1803 decimals is now 2 to 3 times faster. 1804 (Contributed by Serhiy Storchaka in :issue:`25971`.) 1805 1806 1807Build and C API Changes 1808======================= 1809 1810* Python now requires some C99 support in the toolchain to build. 1811 Most notably, Python now uses standard integer types and macros in 1812 place of custom macros like ``PY_LONG_LONG``. 1813 For more information, see :pep:`7` and :issue:`17884`. 1814 1815* Cross-compiling CPython with the Android NDK and the Android API level set to 1816 21 (Android 5.0 Lollilop) or greater runs successfully. While Android is not 1817 yet a supported platform, the Python test suite runs on the Android emulator 1818 with only about 16 tests failures. See the Android meta-issue :issue:`26865`. 1819 1820* The ``--enable-optimizations`` configure flag has been added. Turning it on 1821 will activate expensive optimizations like PGO. 1822 (Original patch by Alecsandru Patrascu of Intel in :issue:`26359`.) 1823 1824* The :term:`GIL <global interpreter lock>` must now be held when allocator 1825 functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and 1826 :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called. 1827 1828* New :c:func:`Py_FinalizeEx` API which indicates if flushing buffered data 1829 failed. 1830 (Contributed by Martin Panter in :issue:`5319`.) 1831 1832* :c:func:`PyArg_ParseTupleAndKeywords` now supports :ref:`positional-only 1833 parameters <positional-only_parameter>`. Positional-only parameters are 1834 defined by empty names. 1835 (Contributed by Serhiy Storchaka in :issue:`26282`). 1836 1837* ``PyTraceback_Print`` method now abbreviates long sequences of repeated lines 1838 as ``"[Previous line repeated {count} more times]"``. 1839 (Contributed by Emanuel Barry in :issue:`26823`.) 1840 1841* The new :c:func:`PyErr_SetImportErrorSubclass` function allows for 1842 specifying a subclass of :exc:`ImportError` to raise. 1843 (Contributed by Eric Snow in :issue:`15767`.) 1844 1845* The new :c:func:`PyErr_ResourceWarning` function can be used to generate 1846 a :exc:`ResourceWarning` providing the source of the resource allocation. 1847 (Contributed by Victor Stinner in :issue:`26567`.) 1848 1849* The new :c:func:`PyOS_FSPath` function returns the file system 1850 representation of a :term:`path-like object`. 1851 (Contributed by Brett Cannon in :issue:`27186`.) 1852 1853* The :c:func:`PyUnicode_FSConverter` and :c:func:`PyUnicode_FSDecoder` 1854 functions will now accept :term:`path-like objects <path-like object>`. 1855 1856 1857Other Improvements 1858================== 1859 1860* When :option:`--version` (short form: :option:`-V`) is supplied twice, 1861 Python prints :data:`sys.version` for detailed information. 1862 1863 .. code-block:: shell-session 1864 1865 $ ./python -VV 1866 Python 3.6.0b4+ (3.6:223967b49e49+, Nov 21 2016, 20:55:04) 1867 [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] 1868 1869 1870Deprecated 1871========== 1872 1873New Keywords 1874------------ 1875 1876``async`` and ``await`` are not recommended to be used as variable, class, 1877function or module names. Introduced by :pep:`492` in Python 3.5, they will 1878become proper keywords in Python 3.7. Starting in Python 3.6, the use of 1879``async`` or ``await`` as names will generate a :exc:`DeprecationWarning`. 1880 1881 1882Deprecated Python behavior 1883-------------------------- 1884 1885Raising the :exc:`StopIteration` exception inside a generator will now 1886generate a :exc:`DeprecationWarning`, and will trigger a :exc:`RuntimeError` 1887in Python 3.7. See :ref:`whatsnew-pep-479` for details. 1888 1889The :meth:`__aiter__` method is now expected to return an asynchronous 1890iterator directly instead of returning an awaitable as previously. 1891Doing the former will trigger a :exc:`DeprecationWarning`. Backward 1892compatibility will be removed in Python 3.7. 1893(Contributed by Yury Selivanov in :issue:`27243`.) 1894 1895A backslash-character pair that is not a valid escape sequence now generates 1896a :exc:`DeprecationWarning`. Although this will eventually become a 1897:exc:`SyntaxError`, that will not be for several Python releases. 1898(Contributed by Emanuel Barry in :issue:`27364`.) 1899 1900When performing a relative import, falling back on ``__name__`` and 1901``__path__`` from the calling module when ``__spec__`` or 1902``__package__`` are not defined now raises an :exc:`ImportWarning`. 1903(Contributed by Rose Ames in :issue:`25791`.) 1904 1905 1906Deprecated Python modules, functions and methods 1907------------------------------------------------ 1908 1909asynchat 1910~~~~~~~~ 1911 1912The :mod:`asynchat` has been deprecated in favor of :mod:`asyncio`. 1913(Contributed by Mariatta in :issue:`25002`.) 1914 1915 1916asyncore 1917~~~~~~~~ 1918 1919The :mod:`asyncore` has been deprecated in favor of :mod:`asyncio`. 1920(Contributed by Mariatta in :issue:`25002`.) 1921 1922 1923dbm 1924~~~ 1925 1926Unlike other :mod:`dbm` implementations, the :mod:`dbm.dumb` module 1927creates databases with the ``'rw'`` mode and allows modifying the database 1928opened with the ``'r'`` mode. This behavior is now deprecated and will 1929be removed in 3.8. 1930(Contributed by Serhiy Storchaka in :issue:`21708`.) 1931 1932 1933distutils 1934~~~~~~~~~ 1935 1936The undocumented ``extra_path`` argument to the 1937:class:`~distutils.Distribution` constructor is now considered deprecated 1938and will raise a warning if set. Support for this parameter will be 1939removed in a future Python release. See :issue:`27919` for details. 1940 1941 1942grp 1943~~~ 1944 1945The support of non-integer arguments in :func:`~grp.getgrgid` has been 1946deprecated. 1947(Contributed by Serhiy Storchaka in :issue:`26129`.) 1948 1949 1950importlib 1951~~~~~~~~~ 1952 1953The :meth:`importlib.machinery.SourceFileLoader.load_module` and 1954:meth:`importlib.machinery.SourcelessFileLoader.load_module` methods 1955are now deprecated. They were the only remaining implementations of 1956:meth:`importlib.abc.Loader.load_module` in :mod:`importlib` that had not 1957been deprecated in previous versions of Python in favour of 1958:meth:`importlib.abc.Loader.exec_module`. 1959 1960The :class:`importlib.machinery.WindowsRegistryFinder` class is now 1961deprecated. As of 3.6.0, it is still added to :attr:`sys.meta_path` by 1962default (on Windows), but this may change in future releases. 1963 1964os 1965~~ 1966 1967Undocumented support of general :term:`bytes-like objects <bytes-like object>` 1968as paths in :mod:`os` functions, :func:`compile` and similar functions is 1969now deprecated. 1970(Contributed by Serhiy Storchaka in :issue:`25791` and :issue:`26754`.) 1971 1972re 1973~~ 1974 1975Support for inline flags ``(?letters)`` in the middle of the regular 1976expression has been deprecated and will be removed in a future Python 1977version. Flags at the start of a regular expression are still allowed. 1978(Contributed by Serhiy Storchaka in :issue:`22493`.) 1979 1980ssl 1981~~~ 1982 1983OpenSSL 0.9.8, 1.0.0 and 1.0.1 are deprecated and no longer supported. 1984In the future the :mod:`ssl` module will require at least OpenSSL 1.0.2 or 19851.1.0. 1986 1987SSL-related arguments like ``certfile``, ``keyfile`` and ``check_hostname`` 1988in :mod:`ftplib`, :mod:`http.client`, :mod:`imaplib`, :mod:`poplib`, 1989and :mod:`smtplib` have been deprecated in favor of ``context``. 1990(Contributed by Christian Heimes in :issue:`28022`.) 1991 1992A couple of protocols and functions of the :mod:`ssl` module are now 1993deprecated. Some features will no longer be available in future versions 1994of OpenSSL. Other features are deprecated in favor of a different API. 1995(Contributed by Christian Heimes in :issue:`28022` and :issue:`26470`.) 1996 1997tkinter 1998~~~~~~~ 1999 2000The :mod:`tkinter.tix` module is now deprecated. :mod:`tkinter` users 2001should use :mod:`tkinter.ttk` instead. 2002 2003venv 2004~~~~ 2005 2006The ``pyvenv`` script has been deprecated in favour of ``python3 -m venv``. 2007This prevents confusion as to what Python interpreter ``pyvenv`` is 2008connected to and thus what Python interpreter will be used by the virtual 2009environment. (Contributed by Brett Cannon in :issue:`25154`.) 2010 2011 2012Deprecated functions and types of the C API 2013------------------------------------------- 2014 2015Undocumented functions :c:func:`PyUnicode_AsEncodedObject`, 2016:c:func:`PyUnicode_AsDecodedObject`, :c:func:`PyUnicode_AsEncodedUnicode` 2017and :c:func:`PyUnicode_AsDecodedUnicode` are deprecated now. 2018Use the :ref:`generic codec based API <codec-registry>` instead. 2019 2020 2021Deprecated Build Options 2022------------------------ 2023 2024The ``--with-system-ffi`` configure flag is now on by default on non-macOS 2025UNIX platforms. It may be disabled by using ``--without-system-ffi``, but 2026using the flag is deprecated and will not be accepted in Python 3.7. 2027macOS is unaffected by this change. Note that many OS distributors already 2028use the ``--with-system-ffi`` flag when building their system Python. 2029 2030 2031Removed 2032======= 2033 2034API and Feature Removals 2035------------------------ 2036 2037* Unknown escapes consisting of ``'\'`` and an ASCII letter in 2038 regular expressions will now cause an error. In replacement templates for 2039 :func:`re.sub` they are still allowed, but deprecated. 2040 The :const:`re.LOCALE` flag can now only be used with binary patterns. 2041 2042* ``inspect.getmoduleinfo()`` was removed (was deprecated since CPython 3.3). 2043 :func:`inspect.getmodulename` should be used for obtaining the module 2044 name for a given path. 2045 (Contributed by Yury Selivanov in :issue:`13248`.) 2046 2047* ``traceback.Ignore`` class and ``traceback.usage``, ``traceback.modname``, 2048 ``traceback.fullmodname``, ``traceback.find_lines_from_code``, 2049 ``traceback.find_lines``, ``traceback.find_strings``, 2050 ``traceback.find_executable_lines`` methods were removed from the 2051 :mod:`traceback` module. They were undocumented methods deprecated since 2052 Python 3.2 and equivalent functionality is available from private methods. 2053 2054* The ``tk_menuBar()`` and ``tk_bindForTraversal()`` dummy methods in 2055 :mod:`tkinter` widget classes were removed (corresponding Tk commands 2056 were obsolete since Tk 4.0). 2057 2058* The :meth:`~zipfile.ZipFile.open` method of the :class:`zipfile.ZipFile` 2059 class no longer supports the ``'U'`` mode (was deprecated since Python 3.4). 2060 Use :class:`io.TextIOWrapper` for reading compressed text files in 2061 :term:`universal newlines` mode. 2062 2063* The undocumented ``IN``, ``CDROM``, ``DLFCN``, ``TYPES``, ``CDIO``, and 2064 ``STROPTS`` modules have been removed. They had been available in the 2065 platform specific ``Lib/plat-*/`` directories, but were chronically out of 2066 date, inconsistently available across platforms, and unmaintained. The 2067 script that created these modules is still available in the source 2068 distribution at :source:`Tools/scripts/h2py.py`. 2069 2070* The deprecated ``asynchat.fifo`` class has been removed. 2071 2072 2073Porting to Python 3.6 2074===================== 2075 2076This section lists previously described changes and other bugfixes 2077that may require changes to your code. 2078 2079Changes in 'python' Command Behavior 2080------------------------------------ 2081 2082* The output of a special Python build with defined ``COUNT_ALLOCS``, 2083 ``SHOW_ALLOC_COUNT`` or ``SHOW_TRACK_COUNT`` macros is now off by 2084 default. It can be re-enabled using the ``-X showalloccount`` option. 2085 It now outputs to ``stderr`` instead of ``stdout``. 2086 (Contributed by Serhiy Storchaka in :issue:`23034`.) 2087 2088 2089Changes in the Python API 2090------------------------- 2091 2092* :func:`open() <open>` will no longer allow combining the ``'U'`` mode flag 2093 with ``'+'``. 2094 (Contributed by Jeff Balogh and John O'Connor in :issue:`2091`.) 2095 2096* :mod:`sqlite3` no longer implicitly commits an open transaction before DDL 2097 statements. 2098 2099* On Linux, :func:`os.urandom` now blocks until the system urandom entropy pool 2100 is initialized to increase the security. 2101 2102* When :meth:`importlib.abc.Loader.exec_module` is defined, 2103 :meth:`importlib.abc.Loader.create_module` must also be defined. 2104 2105* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** 2106 argument is not set. Previously only ``NULL`` was returned. 2107 2108* The format of the ``co_lnotab`` attribute of code objects changed to support 2109 a negative line number delta. By default, Python does not emit bytecode with 2110 a negative line number delta. Functions using ``frame.f_lineno``, 2111 ``PyFrame_GetLineNumber()`` or ``PyCode_Addr2Line()`` are not affected. 2112 Functions directly decoding ``co_lnotab`` should be updated to use a signed 2113 8-bit integer type for the line number delta, but this is only required to 2114 support applications using a negative line number delta. See 2115 ``Objects/lnotab_notes.txt`` for the ``co_lnotab`` format and how to decode 2116 it, and see the :pep:`511` for the rationale. 2117 2118* The functions in the :mod:`compileall` module now return booleans instead 2119 of ``1`` or ``0`` to represent success or failure, respectively. Thanks to 2120 booleans being a subclass of integers, this should only be an issue if you 2121 were doing identity checks for ``1`` or ``0``. See :issue:`25768`. 2122 2123* Reading the :attr:`~urllib.parse.SplitResult.port` attribute of 2124 :func:`urllib.parse.urlsplit` and :func:`~urllib.parse.urlparse` results 2125 now raises :exc:`ValueError` for out-of-range values, rather than 2126 returning :const:`None`. See :issue:`20059`. 2127 2128* The :mod:`imp` module now raises a :exc:`DeprecationWarning` instead of 2129 :exc:`PendingDeprecationWarning`. 2130 2131* The following modules have had missing APIs added to their :attr:`__all__` 2132 attributes to match the documented APIs: 2133 :mod:`calendar`, :mod:`cgi`, :mod:`csv`, 2134 :mod:`~xml.etree.ElementTree`, :mod:`enum`, 2135 :mod:`fileinput`, :mod:`ftplib`, :mod:`logging`, :mod:`mailbox`, 2136 :mod:`mimetypes`, :mod:`optparse`, :mod:`plistlib`, :mod:`smtpd`, 2137 :mod:`subprocess`, :mod:`tarfile`, :mod:`threading` and 2138 :mod:`wave`. This means they will export new symbols when ``import *`` 2139 is used. 2140 (Contributed by Joel Taddei and Jacek Kołodziej in :issue:`23883`.) 2141 2142* When performing a relative import, if ``__package__`` does not compare equal 2143 to ``__spec__.parent`` then :exc:`ImportWarning` is raised. 2144 (Contributed by Brett Cannon in :issue:`25791`.) 2145 2146* When a relative import is performed and no parent package is known, then 2147 :exc:`ImportError` will be raised. Previously, :exc:`SystemError` could be 2148 raised. (Contributed by Brett Cannon in :issue:`18018`.) 2149 2150* Servers based on the :mod:`socketserver` module, including those 2151 defined in :mod:`http.server`, :mod:`xmlrpc.server` and 2152 :mod:`wsgiref.simple_server`, now only catch exceptions derived 2153 from :exc:`Exception`. Therefore if a request handler raises 2154 an exception like :exc:`SystemExit` or :exc:`KeyboardInterrupt`, 2155 :meth:`~socketserver.BaseServer.handle_error` is no longer called, and 2156 the exception will stop a single-threaded server. (Contributed by 2157 Martin Panter in :issue:`23430`.) 2158 2159* :func:`spwd.getspnam` now raises a :exc:`PermissionError` instead of 2160 :exc:`KeyError` if the user doesn't have privileges. 2161 2162* The :meth:`socket.socket.close` method now raises an exception if 2163 an error (e.g. ``EBADF``) was reported by the underlying system call. 2164 (Contributed by Martin Panter in :issue:`26685`.) 2165 2166* The *decode_data* argument for the :class:`smtpd.SMTPChannel` and 2167 :class:`smtpd.SMTPServer` constructors is now ``False`` by default. 2168 This means that the argument passed to 2169 :meth:`~smtpd.SMTPServer.process_message` is now a bytes object by 2170 default, and ``process_message()`` will be passed keyword arguments. 2171 Code that has already been updated in accordance with the deprecation 2172 warning generated by 3.5 will not be affected. 2173 2174* All optional arguments of the :func:`~json.dump`, :func:`~json.dumps`, 2175 :func:`~json.load` and :func:`~json.loads` functions and 2176 :class:`~json.JSONEncoder` and :class:`~json.JSONDecoder` class 2177 constructors in the :mod:`json` module are now :ref:`keyword-only 2178 <keyword-only_parameter>`. 2179 (Contributed by Serhiy Storchaka in :issue:`18726`.) 2180 2181* Subclasses of :class:`type` which don't override ``type.__new__`` may no 2182 longer use the one-argument form to get the type of an object. 2183 2184* As part of :pep:`487`, the handling of keyword arguments passed to 2185 :class:`type` (other than the metaclass hint, ``metaclass``) is now 2186 consistently delegated to :meth:`object.__init_subclass__`. This means that 2187 :meth:`type.__new__` and :meth:`type.__init__` both now accept arbitrary 2188 keyword arguments, but :meth:`object.__init_subclass__` (which is called from 2189 :meth:`type.__new__`) will reject them by default. Custom metaclasses 2190 accepting additional keyword arguments will need to adjust their calls to 2191 :meth:`type.__new__` (whether direct or via :class:`super`) accordingly. 2192 2193* In :class:`distutils.command.sdist.sdist`, the ``default_format`` 2194 attribute has been removed and is no longer honored. Instead, the 2195 gzipped tarfile format is the default on all platforms and no 2196 platform-specific selection is made. 2197 In environments where distributions are 2198 built on Windows and zip distributions are required, configure 2199 the project with a ``setup.cfg`` file containing the following:: 2200 2201 [sdist] 2202 formats=zip 2203 2204 This behavior has also been backported to earlier Python versions 2205 by Setuptools 26.0.0. 2206 2207* In the :mod:`urllib.request` module and the 2208 :meth:`http.client.HTTPConnection.request` method, if no Content-Length 2209 header field has been specified and the request body is a file object, 2210 it is now sent with HTTP 1.1 chunked encoding. If a file object has to 2211 be sent to a HTTP 1.0 server, the Content-Length value now has to be 2212 specified by the caller. 2213 (Contributed by Demian Brecht and Rolf Krahl with tweaks from 2214 Martin Panter in :issue:`12319`.) 2215 2216* The :class:`~csv.DictReader` now returns rows of type 2217 :class:`~collections.OrderedDict`. 2218 (Contributed by Steve Holden in :issue:`27842`.) 2219 2220* The :const:`crypt.METHOD_CRYPT` will no longer be added to ``crypt.methods`` 2221 if unsupported by the platform. 2222 (Contributed by Victor Stinner in :issue:`25287`.) 2223 2224* The *verbose* and *rename* arguments for 2225 :func:`~collections.namedtuple` are now keyword-only. 2226 (Contributed by Raymond Hettinger in :issue:`25628`.) 2227 2228* On Linux, :func:`ctypes.util.find_library` now looks in 2229 ``LD_LIBRARY_PATH`` for shared libraries. 2230 (Contributed by Vinay Sajip in :issue:`9998`.) 2231 2232* The :class:`imaplib.IMAP4` class now handles flags containing the 2233 ``']'`` character in messages sent from the server to improve 2234 real-world compatibility. 2235 (Contributed by Lita Cho in :issue:`21815`.) 2236 2237* The :func:`mmap.write() <mmap.write>` function now returns the number 2238 of bytes written like other write methods. 2239 (Contributed by Jakub Stasiak in :issue:`26335`.) 2240 2241* The :func:`pkgutil.iter_modules` and :func:`pkgutil.walk_packages` 2242 functions now return :class:`~pkgutil.ModuleInfo` named tuples. 2243 (Contributed by Ramchandra Apte in :issue:`17211`.) 2244 2245* :func:`re.sub` now raises an error for invalid numerical group 2246 references in replacement templates even if the pattern is not 2247 found in the string. The error message for invalid group references 2248 now includes the group index and the position of the reference. 2249 (Contributed by SilentGhost, Serhiy Storchaka in :issue:`25953`.) 2250 2251* :class:`zipfile.ZipFile` will now raise :exc:`NotImplementedError` for 2252 unrecognized compression values. Previously a plain :exc:`RuntimeError` 2253 was raised. Additionally, calling :class:`~zipfile.ZipFile` methods 2254 on a closed ZipFile or calling the :meth:`~zipfile.ZipFile.write` method 2255 on a ZipFile created with mode ``'r'`` will raise a :exc:`ValueError`. 2256 Previously, a :exc:`RuntimeError` was raised in those scenarios. 2257 2258* when custom metaclasses are combined with zero-argument :func:`super` or 2259 direct references from methods to the implicit ``__class__`` closure 2260 variable, the implicit ``__classcell__`` namespace entry must now be passed 2261 up to ``type.__new__`` for initialisation. Failing to do so will result in 2262 a :exc:`DeprecationWarning` in 3.6 and a :exc:`RuntimeWarning` in the future. 2263 2264Changes in the C API 2265-------------------- 2266 2267* The :c:func:`PyMem_Malloc` allocator family now uses the :ref:`pymalloc allocator 2268 <pymalloc>` rather than the system :c:func:`malloc`. Applications calling 2269 :c:func:`PyMem_Malloc` without holding the GIL can now crash. Set the 2270 :envvar:`PYTHONMALLOC` environment variable to ``debug`` to validate the 2271 usage of memory allocators in your application. See :issue:`26249`. 2272 2273* :c:func:`Py_Exit` (and the main interpreter) now override the exit status 2274 with 120 if flushing buffered data failed. See :issue:`5319`. 2275 2276 2277CPython bytecode changes 2278------------------------ 2279 2280There have been several major changes to the :term:`bytecode` in Python 3.6. 2281 2282* The Python interpreter now uses a 16-bit wordcode instead of bytecode. 2283 (Contributed by Demur Rumed with input and reviews from 2284 Serhiy Storchaka and Victor Stinner in :issue:`26647` and :issue:`28050`.) 2285 2286* The new :opcode:`FORMAT_VALUE` and :opcode:`BUILD_STRING` opcodes as part 2287 of the :ref:`formatted string literal <whatsnew36-pep498>` implementation. 2288 (Contributed by Eric Smith in :issue:`25483` and 2289 Serhiy Storchaka in :issue:`27078`.) 2290 2291* The new :opcode:`BUILD_CONST_KEY_MAP` opcode to optimize the creation 2292 of dictionaries with constant keys. 2293 (Contributed by Serhiy Storchaka in :issue:`27140`.) 2294 2295* The function call opcodes have been heavily reworked for better performance 2296 and simpler implementation. 2297 The :opcode:`MAKE_FUNCTION`, :opcode:`CALL_FUNCTION`, 2298 :opcode:`CALL_FUNCTION_KW` and :opcode:`BUILD_MAP_UNPACK_WITH_CALL` opcodes 2299 have been modified, the new :opcode:`CALL_FUNCTION_EX` and 2300 :opcode:`BUILD_TUPLE_UNPACK_WITH_CALL` have been added, and 2301 ``CALL_FUNCTION_VAR``, ``CALL_FUNCTION_VAR_KW`` and ``MAKE_CLOSURE`` opcodes 2302 have been removed. 2303 (Contributed by Demur Rumed in :issue:`27095`, and Serhiy Storchaka in 2304 :issue:`27213`, :issue:`28257`.) 2305 2306* The new :opcode:`SETUP_ANNOTATIONS` and :opcode:`STORE_ANNOTATION` opcodes 2307 have been added to support the new :term:`variable annotation` syntax. 2308 (Contributed by Ivan Levkivskyi in :issue:`27985`.) 2309