1.. _tut-io:
2
3****************
4Input and Output
5****************
6
7There are several ways to present the output of a program; data can be printed
8in a human-readable form, or written to a file for future use. This chapter will
9discuss some of the possibilities.
10
11
12.. _tut-formatting:
13
14Fancier Output Formatting
15=========================
16
17So far we've encountered two ways of writing values: *expression statements* and
18the :func:`print` function.  (A third way is using the :meth:`write` method
19of file objects; the standard output file can be referenced as ``sys.stdout``.
20See the Library Reference for more information on this.)
21
22Often you'll want more control over the formatting of your output than simply
23printing space-separated values. There are several ways to format output.
24
25* To use :ref:`formatted string literals <tut-f-strings>`, begin a string
26  with ``f`` or ``F`` before the opening quotation mark or triple quotation mark.
27  Inside this string, you can write a Python expression between ``{`` and ``}``
28  characters that can refer to variables or literal values.
29
30  ::
31
32     >>> year = 2016
33     >>> event = 'Referendum'
34     >>> f'Results of the {year} {event}'
35     'Results of the 2016 Referendum'
36
37* The :meth:`str.format` method of strings requires more manual
38  effort.  You'll still use ``{`` and ``}`` to mark where a variable
39  will be substituted and can provide detailed formatting directives,
40  but you'll also need to provide the information to be formatted.
41
42  ::
43
44     >>> yes_votes = 42_572_654
45     >>> no_votes = 43_132_495
46     >>> percentage = yes_votes / (yes_votes + no_votes)
47     >>> '{:-9} YES votes  {:2.2%}'.format(yes_votes, percentage)
48     ' 42572654 YES votes  49.67%'
49
50* Finally, you can do all the string handling yourself by using string slicing and
51  concatenation operations to create any layout you can imagine.  The
52  string type has some methods that perform useful operations for padding
53  strings to a given column width.
54
55When you don't need fancy output but just want a quick display of some
56variables for debugging purposes, you can convert any value to a string with
57the :func:`repr` or :func:`str` functions.
58
59The :func:`str` function is meant to return representations of values which are
60fairly human-readable, while :func:`repr` is meant to generate representations
61which can be read by the interpreter (or will force a :exc:`SyntaxError` if
62there is no equivalent syntax).  For objects which don't have a particular
63representation for human consumption, :func:`str` will return the same value as
64:func:`repr`.  Many values, such as numbers or structures like lists and
65dictionaries, have the same representation using either function.  Strings, in
66particular, have two distinct representations.
67
68Some examples::
69
70   >>> s = 'Hello, world.'
71   >>> str(s)
72   'Hello, world.'
73   >>> repr(s)
74   "'Hello, world.'"
75   >>> str(1/7)
76   '0.14285714285714285'
77   >>> x = 10 * 3.25
78   >>> y = 200 * 200
79   >>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...'
80   >>> print(s)
81   The value of x is 32.5, and y is 40000...
82   >>> # The repr() of a string adds string quotes and backslashes:
83   ... hello = 'hello, world\n'
84   >>> hellos = repr(hello)
85   >>> print(hellos)
86   'hello, world\n'
87   >>> # The argument to repr() may be any Python object:
88   ... repr((x, y, ('spam', 'eggs')))
89   "(32.5, 40000, ('spam', 'eggs'))"
90
91The :mod:`string` module contains a :class:`~string.Template` class that offers
92yet another way to substitute values into strings, using placeholders like
93``$x`` and replacing them with values from a dictionary, but offers much less
94control of the formatting.
95
96
97.. _tut-f-strings:
98
99Formatted String Literals
100-------------------------
101
102:ref:`Formatted string literals <f-strings>` (also called f-strings for
103short) let you include the value of Python expressions inside a string by
104prefixing the string with ``f`` or ``F`` and writing expressions as
105``{expression}``.
106
107An optional format specifier can follow the expression. This allows greater
108control over how the value is formatted. The following example rounds pi to
109three places after the decimal::
110
111   >>> import math
112   >>> print(f'The value of pi is approximately {math.pi:.3f}.')
113   The value of pi is approximately 3.142.
114
115Passing an integer after the ``':'`` will cause that field to be a minimum
116number of characters wide.  This is useful for making columns line up. ::
117
118   >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678}
119   >>> for name, phone in table.items():
120   ...     print(f'{name:10} ==> {phone:10d}')
121   ...
122   Sjoerd     ==>       4127
123   Jack       ==>       4098
124   Dcab       ==>       7678
125
126Other modifiers can be used to convert the value before it is formatted.
127``'!a'`` applies :func:`ascii`, ``'!s'`` applies :func:`str`, and ``'!r'``
128applies :func:`repr`::
129
130   >>> animals = 'eels'
131   >>> print(f'My hovercraft is full of {animals}.')
132   My hovercraft is full of eels.
133   >>> print(f'My hovercraft is full of {animals!r}.')
134   My hovercraft is full of 'eels'.
135
136For a reference on these format specifications, see
137the reference guide for the :ref:`formatspec`.
138
139.. _tut-string-format:
140
141The String format() Method
142--------------------------
143
144Basic usage of the :meth:`str.format` method looks like this::
145
146   >>> print('We are the {} who say "{}!"'.format('knights', 'Ni'))
147   We are the knights who say "Ni!"
148
149The brackets and characters within them (called format fields) are replaced with
150the objects passed into the :meth:`str.format` method.  A number in the
151brackets can be used to refer to the position of the object passed into the
152:meth:`str.format` method. ::
153
154   >>> print('{0} and {1}'.format('spam', 'eggs'))
155   spam and eggs
156   >>> print('{1} and {0}'.format('spam', 'eggs'))
157   eggs and spam
158
159If keyword arguments are used in the :meth:`str.format` method, their values
160are referred to by using the name of the argument. ::
161
162   >>> print('This {food} is {adjective}.'.format(
163   ...       food='spam', adjective='absolutely horrible'))
164   This spam is absolutely horrible.
165
166Positional and keyword arguments can be arbitrarily combined::
167
168   >>> print('The story of {0}, {1}, and {other}.'.format('Bill', 'Manfred',
169                                                          other='Georg'))
170   The story of Bill, Manfred, and Georg.
171
172If you have a really long format string that you don't want to split up, it
173would be nice if you could reference the variables to be formatted by name
174instead of by position.  This can be done by simply passing the dict and using
175square brackets ``'[]'`` to access the keys ::
176
177   >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
178   >>> print('Jack: {0[Jack]:d}; Sjoerd: {0[Sjoerd]:d}; '
179   ...       'Dcab: {0[Dcab]:d}'.format(table))
180   Jack: 4098; Sjoerd: 4127; Dcab: 8637678
181
182This could also be done by passing the table as keyword arguments with the '**'
183notation. ::
184
185   >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
186   >>> print('Jack: {Jack:d}; Sjoerd: {Sjoerd:d}; Dcab: {Dcab:d}'.format(**table))
187   Jack: 4098; Sjoerd: 4127; Dcab: 8637678
188
189This is particularly useful in combination with the built-in function
190:func:`vars`, which returns a dictionary containing all local variables.
191
192As an example, the following lines produce a tidily-aligned
193set of columns giving integers and their squares and cubes::
194
195   >>> for x in range(1, 11):
196   ...     print('{0:2d} {1:3d} {2:4d}'.format(x, x*x, x*x*x))
197   ...
198    1   1    1
199    2   4    8
200    3   9   27
201    4  16   64
202    5  25  125
203    6  36  216
204    7  49  343
205    8  64  512
206    9  81  729
207   10 100 1000
208
209For a complete overview of string formatting with :meth:`str.format`, see
210:ref:`formatstrings`.
211
212
213Manual String Formatting
214------------------------
215
216Here's the same table of squares and cubes, formatted manually::
217
218   >>> for x in range(1, 11):
219   ...     print(repr(x).rjust(2), repr(x*x).rjust(3), end=' ')
220   ...     # Note use of 'end' on previous line
221   ...     print(repr(x*x*x).rjust(4))
222   ...
223    1   1    1
224    2   4    8
225    3   9   27
226    4  16   64
227    5  25  125
228    6  36  216
229    7  49  343
230    8  64  512
231    9  81  729
232   10 100 1000
233
234(Note that the one space between each column was added by the
235way :func:`print` works: it always adds spaces between its arguments.)
236
237The :meth:`str.rjust` method of string objects right-justifies a string in a
238field of a given width by padding it with spaces on the left. There are
239similar methods :meth:`str.ljust` and :meth:`str.center`. These methods do
240not write anything, they just return a new string. If the input string is too
241long, they don't truncate it, but return it unchanged; this will mess up your
242column lay-out but that's usually better than the alternative, which would be
243lying about a value. (If you really want truncation you can always add a
244slice operation, as in ``x.ljust(n)[:n]``.)
245
246There is another method, :meth:`str.zfill`, which pads a numeric string on the
247left with zeros.  It understands about plus and minus signs::
248
249   >>> '12'.zfill(5)
250   '00012'
251   >>> '-3.14'.zfill(7)
252   '-003.14'
253   >>> '3.14159265359'.zfill(5)
254   '3.14159265359'
255
256
257Old string formatting
258---------------------
259
260The ``%`` operator can also be used for string formatting. It interprets the
261left argument much like a :c:func:`sprintf`\ -style format string to be applied
262to the right argument, and returns the string resulting from this formatting
263operation. For example::
264
265   >>> import math
266   >>> print('The value of pi is approximately %5.3f.' % math.pi)
267   The value of pi is approximately 3.142.
268
269More information can be found in the :ref:`old-string-formatting` section.
270
271
272.. _tut-files:
273
274Reading and Writing Files
275=========================
276
277.. index::
278   builtin: open
279   object: file
280
281:func:`open` returns a :term:`file object`, and is most commonly used with
282two arguments: ``open(filename, mode)``.
283
284::
285
286   >>> f = open('workfile', 'w')
287
288.. XXX str(f) is <io.TextIOWrapper object at 0x82e8dc4>
289
290   >>> print(f)
291   <open file 'workfile', mode 'w' at 80a0960>
292
293The first argument is a string containing the filename.  The second argument is
294another string containing a few characters describing the way in which the file
295will be used.  *mode* can be ``'r'`` when the file will only be read, ``'w'``
296for only writing (an existing file with the same name will be erased), and
297``'a'`` opens the file for appending; any data written to the file is
298automatically added to the end.  ``'r+'`` opens the file for both reading and
299writing. The *mode* argument is optional; ``'r'`` will be assumed if it's
300omitted.
301
302Normally, files are opened in :dfn:`text mode`, that means, you read and write
303strings from and to the file, which are encoded in a specific encoding. If
304encoding is not specified, the default is platform dependent (see
305:func:`open`). ``'b'`` appended to the mode opens the file in
306:dfn:`binary mode`: now the data is read and written in the form of bytes
307objects.  This mode should be used for all files that don't contain text.
308
309In text mode, the default when reading is to convert platform-specific line
310endings (``\n`` on Unix, ``\r\n`` on Windows) to just ``\n``.  When writing in
311text mode, the default is to convert occurrences of ``\n`` back to
312platform-specific line endings.  This behind-the-scenes modification
313to file data is fine for text files, but will corrupt binary data like that in
314:file:`JPEG` or :file:`EXE` files.  Be very careful to use binary mode when
315reading and writing such files.
316
317It is good practice to use the :keyword:`with` keyword when dealing
318with file objects.  The advantage is that the file is properly closed
319after its suite finishes, even if an exception is raised at some
320point.  Using :keyword:`!with` is also much shorter than writing
321equivalent :keyword:`try`\ -\ :keyword:`finally` blocks::
322
323    >>> with open('workfile') as f:
324    ...     read_data = f.read()
325    >>> f.closed
326    True
327
328If you're not using the :keyword:`with` keyword, then you should call
329``f.close()`` to close the file and immediately free up any system
330resources used by it. If you don't explicitly close a file, Python's
331garbage collector will eventually destroy the object and close the
332open file for you, but the file may stay open for a while.  Another
333risk is that different Python implementations will do this clean-up at
334different times.
335
336After a file object is closed, either by a :keyword:`with` statement
337or by calling ``f.close()``, attempts to use the file object will
338automatically fail. ::
339
340   >>> f.close()
341   >>> f.read()
342   Traceback (most recent call last):
343     File "<stdin>", line 1, in <module>
344   ValueError: I/O operation on closed file.
345
346
347.. _tut-filemethods:
348
349Methods of File Objects
350-----------------------
351
352The rest of the examples in this section will assume that a file object called
353``f`` has already been created.
354
355To read a file's contents, call ``f.read(size)``, which reads some quantity of
356data and returns it as a string (in text mode) or bytes object (in binary mode).
357*size* is an optional numeric argument.  When *size* is omitted or negative, the
358entire contents of the file will be read and returned; it's your problem if the
359file is twice as large as your machine's memory. Otherwise, at most *size* bytes
360are read and returned.
361If the end of the file has been reached, ``f.read()`` will return an empty
362string (``''``).  ::
363
364   >>> f.read()
365   'This is the entire file.\n'
366   >>> f.read()
367   ''
368
369``f.readline()`` reads a single line from the file; a newline character (``\n``)
370is left at the end of the string, and is only omitted on the last line of the
371file if the file doesn't end in a newline.  This makes the return value
372unambiguous; if ``f.readline()`` returns an empty string, the end of the file
373has been reached, while a blank line is represented by ``'\n'``, a string
374containing only a single newline.  ::
375
376   >>> f.readline()
377   'This is the first line of the file.\n'
378   >>> f.readline()
379   'Second line of the file\n'
380   >>> f.readline()
381   ''
382
383For reading lines from a file, you can loop over the file object. This is memory
384efficient, fast, and leads to simple code::
385
386   >>> for line in f:
387   ...     print(line, end='')
388   ...
389   This is the first line of the file.
390   Second line of the file
391
392If you want to read all the lines of a file in a list you can also use
393``list(f)`` or ``f.readlines()``.
394
395``f.write(string)`` writes the contents of *string* to the file, returning
396the number of characters written. ::
397
398   >>> f.write('This is a test\n')
399   15
400
401Other types of objects need to be converted -- either to a string (in text mode)
402or a bytes object (in binary mode) -- before writing them::
403
404   >>> value = ('the answer', 42)
405   >>> s = str(value)  # convert the tuple to string
406   >>> f.write(s)
407   18
408
409``f.tell()`` returns an integer giving the file object's current position in the file
410represented as number of bytes from the beginning of the file when in binary mode and
411an opaque number when in text mode.
412
413To change the file object's position, use ``f.seek(offset, from_what)``.  The position is computed
414from adding *offset* to a reference point; the reference point is selected by
415the *from_what* argument.  A *from_what* value of 0 measures from the beginning
416of the file, 1 uses the current file position, and 2 uses the end of the file as
417the reference point.  *from_what* can be omitted and defaults to 0, using the
418beginning of the file as the reference point. ::
419
420   >>> f = open('workfile', 'rb+')
421   >>> f.write(b'0123456789abcdef')
422   16
423   >>> f.seek(5)      # Go to the 6th byte in the file
424   5
425   >>> f.read(1)
426   b'5'
427   >>> f.seek(-3, 2)  # Go to the 3rd byte before the end
428   13
429   >>> f.read(1)
430   b'd'
431
432In text files (those opened without a ``b`` in the mode string), only seeks
433relative to the beginning of the file are allowed (the exception being seeking
434to the very file end with ``seek(0, 2)``) and the only valid *offset* values are
435those returned from the ``f.tell()``, or zero. Any other *offset* value produces
436undefined behaviour.
437
438File objects have some additional methods, such as :meth:`~file.isatty` and
439:meth:`~file.truncate` which are less frequently used; consult the Library
440Reference for a complete guide to file objects.
441
442
443.. _tut-json:
444
445Saving structured data with :mod:`json`
446---------------------------------------
447
448.. index:: module: json
449
450Strings can easily be written to and read from a file.  Numbers take a bit more
451effort, since the :meth:`read` method only returns strings, which will have to
452be passed to a function like :func:`int`, which takes a string like ``'123'``
453and returns its numeric value 123.  When you want to save more complex data
454types like nested lists and dictionaries, parsing and serializing by hand
455becomes complicated.
456
457Rather than having users constantly writing and debugging code to save
458complicated data types to files, Python allows you to use the popular data
459interchange format called `JSON (JavaScript Object Notation)
460<http://json.org>`_.  The standard module called :mod:`json` can take Python
461data hierarchies, and convert them to string representations; this process is
462called :dfn:`serializing`.  Reconstructing the data from the string representation
463is called :dfn:`deserializing`.  Between serializing and deserializing, the
464string representing the object may have been stored in a file or data, or
465sent over a network connection to some distant machine.
466
467.. note::
468   The JSON format is commonly used by modern applications to allow for data
469   exchange.  Many programmers are already familiar with it, which makes
470   it a good choice for interoperability.
471
472If you have an object ``x``, you can view its JSON string representation with a
473simple line of code::
474
475   >>> import json
476   >>> json.dumps([1, 'simple', 'list'])
477   '[1, "simple", "list"]'
478
479Another variant of the :func:`~json.dumps` function, called :func:`~json.dump`,
480simply serializes the object to a :term:`text file`.  So if ``f`` is a
481:term:`text file` object opened for writing, we can do this::
482
483   json.dump(x, f)
484
485To decode the object again, if ``f`` is a :term:`text file` object which has
486been opened for reading::
487
488   x = json.load(f)
489
490This simple serialization technique can handle lists and dictionaries, but
491serializing arbitrary class instances in JSON requires a bit of extra effort.
492The reference for the :mod:`json` module contains an explanation of this.
493
494.. seealso::
495
496   :mod:`pickle` - the pickle module
497
498   Contrary to :ref:`JSON <tut-json>`, *pickle* is a protocol which allows
499   the serialization of arbitrarily complex Python objects.  As such, it is
500   specific to Python and cannot be used to communicate with applications
501   written in other languages.  It is also insecure by default:
502   deserializing pickle data coming from an untrusted source can execute
503   arbitrary code, if the data was crafted by a skilled attacker.
504