README
1Python Misc subdirectory
2========================
3
4This directory contains files that wouldn't fit in elsewhere. Some
5documents are only of historic importance.
6
7Files found here
8----------------
9
10ACKS Acknowledgements
11AIX-NOTES Notes for building Python on AIX
12BeOS-NOTES Notes for building on BeOS
13BeOS-setup.py setup.py replacement for BeOS, see BeOS-NOTES
14build.sh Script to build and test latest Python from the repository
15cheatsheet Quick summary of Python by Ken Manheimer
16developers.txt A history of who got developer permissions, and why
17gdbinit Handy stuff to put in your .gdbinit file, if you use gdb
18HISTORY News from previous releases -- oldest last
19indent.pro GNU indent profile approximating my C style
20maintainers.rst A list of maintainers for library modules
21NEWS News for this release (for some meaning of "this")
22NEWS.help How to edit NEWS
23Porting Mini-FAQ on porting to new platforms
24PURIFY.README Information for Purify users
25pymemcompat.h Memory interface compatibility file.
26python-config.in Python script template for python-config
27python.man UNIX man page for the python interpreter
28python-mode.el Emacs mode for editing Python programs
29python.pc.in Package configuration info template for pkg-config
30python-wing.wpr Wing IDE project file
31README The file you're reading now
32README.coverity Information about running Coverity's Prevent on Python
33README.klocwork Information about running Klocwork's K7 on Python
34README.OpenBSD Help for building problems on OpenBSD
35README.valgrind Information for Valgrind users, see valgrind-python.supp
36RFD Request For Discussion about a Python newsgroup
37setuid-prog.c C helper program for set-uid Python scripts
38SpecialBuilds.txt Describes extra symbols you can set for debug builds
39TextMate A TextMate bundle for Python development
40valgrind-python.supp Valgrind suppression file, see README.valgrind
41vgrindefs Python configuration for vgrind (a generic pretty printer)
42Vim Python development utilities for the Vim editor
43
README.AIX
1
2This documentation tries to help people who intend to use Python on
3AIX.
4
5There used to be many issues with Python on AIX, but the major ones
6have been corrected for version 3.2, so that Python should now work
7rather well on this platform. The remaining known issues are listed in
8this document.
9
10
11======================================================================
12 Compiling Python
13----------------------------------------------------------------------
14
15You can compile Python with gcc or the native AIX compiler. The native
16compiler used to give better performances on this system with older
17versions of Python. With Python 3.2 it may not be the case anymore,
18as this compiler does not allow compiling Python with computed gotos.
19Some benchmarks need to be done.
20
21Compiling with gcc:
22
23cd Python-3.2
24CC=gcc OPT="-O2" ./configure --enable-shared
25make
26
27There are various aliases for the native compiler. The recommended
28alias for compiling Python is 'xlc_r', which provides a better level of
29compatibility and handles thread initialization properly.
30
31It is a good idea to add the '-qmaxmem=70000' option, otherwise the
32compiler considers various files too complex to optimize.
33
34Compiling with xlc:
35
36cd Python-3.2
37CC=xlc_r OPT="-O2 -qmaxmem=70000" ./configure --without-computed-gotos --enable-shared
38make
39
40Note:
41On AIX 5.3 and earlier, you will also need to specify the
42"--disable-ipv6" flag to configure. This has been corrected in AIX
436.1.
44
45
46======================================================================
47 Memory Limitations
48----------------------------------------------------------------------
49
50Note: this section may not apply when compiling Python as a 64 bit
51application.
52
53By default on AIX each program gets one segment register for its data
54segment. As each segment register covers 256 MB, a Python program that
55would use more than 256MB will raise a MemoryError. The standard
56Python test suite is one such application.
57
58To allocate more segment registers to Python, you must use the linker
59option -bmaxdata or the ldedit tool to specify the number of bytes you
60need in the data segment.
61
62For example, if you want to allow 512MB of memory for Python (this is
63enough for the test suite to run without MemoryErrors), you should run
64the following command at the end of compilation:
65
66ldedit -b maxdata:0x20000000 ./python
67
68You can allow up to 2GB of memory for Python by using the value
690x80000000 for maxdata.
70
71It is also possible to go beyond 2GB of memory by activating Large
72Page Use. You should consult the IBM documentation if you need to use
73this option. You can also follow the discussion of this problem
74in issue 11212 at bugs.python.org.
75
76http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds3/ldedit.htm
77
78
79======================================================================
80 Known issues
81----------------------------------------------------------------------
82
83Those issues are currently affecting Python on AIX:
84
85* Python has not been fully tested on AIX when compiled as a 64 bit
86 application.
87
88* issue 3526: the memory used by a Python process will never be
89 released to the system. If you have a Python application on AIX that
90 uses a lot of memory, you should read this issue and you may
91 consider using the provided patch that implements a custom malloc
92 implementation
93
94* issue 11184: support for large files is currently broken
95
96* issue 11185: os.wait4 does not behave correctly with option WNOHANG
97
98* issue 1745108: there may be some problems with curses.panel
99
100* issue 11192: test_socket fails
101
102* issue 11190: test_locale fails
103
104* issue 11193: test_subprocess fails
105
106* issue 9920: minor arithmetic issues in cmath
107
108* issue 11215: test_fileio fails
109
110* issue 11188: test_time fails
111
112
113======================================================================
114 Implementation details for developers
115----------------------------------------------------------------------
116
117Python and python modules can now be built as shared libraries on AIX
118as usual.
119
120AIX shared libraries require that an "export" and "import" file be
121provided at compile time to list all extern symbols which may be
122shared between modules. The "export" file (named python.exp) for the
123modules and the libraries that belong to the Python core is created by
124the "makexp_aix" script before performing the link of the python
125binary. It lists all global symbols (exported during the link) of the
126modules and the libraries that make up the python executable.
127
128When shared library modules (.so files) are made, a second shell
129script is invoked. This script is named "ld_so_aix" and is also
130provided with the distribution in the Modules subdirectory. This
131script acts as an "ld" wrapper which hides the explicit management of
132"export" and "import" files; it adds the appropriate arguments (in the
133appropriate order) to the link command that creates the shared module.
134Among other things, it specifies that the "python.exp" file is an
135"import" file for the shared module.
136
137This mechanism should be transparent.
138
README.Emacs
1=============
2Emacs support
3=============
4
5If you want to edit Python code in Emacs, you should download python-mode.el
6and install it somewhere on your load-path. See the project page to download:
7
8 https://launchpad.net/python-mode
9
10While Emacs comes with a python.el file, it is not recommended.
11python-mode.el is maintained by core Python developers and is generally
12considered more Python programmer friendly. For example, python-mode.el
13includes a killer feature called `pdbtrack` which allows you to set a pdb
14breakpoint in your code, run your program in an Emacs shell buffer, and do gud
15style debugging when the breakpoint is hit.
16
17python-mode.el is compatible with both GNU Emacs from the FSF, and XEmacs.
18
19For more information and bug reporting, see the above project page. For help,
20development, or discussions, see the python-mode mailing list:
21
22 http://mail.python.org/mailman/listinfo/python-mode
23
24
25..
26 Local Variables:
27 mode: indented-text
28 indent-tabs-mode: nil
29 sentence-end-double-space: t
30 fill-column: 78
31 coding: utf-8
32 End:
33
README.OpenBSD
1
22005-01-08
3
4If you are have a problem building on OpenBSD and see output like this
5while running configure:
6
7checking curses.h presence... yes
8configure: WARNING: curses.h: present but cannot be compiled
9configure: WARNING: curses.h: check for missing prerequisite headers?
10configure: WARNING: curses.h: see the Autoconf documentation
11configure: WARNING: curses.h: section "Present But Cannot Be Compiled"
12configure: WARNING: curses.h: proceeding with the preprocessor's result
13configure: WARNING: curses.h: in the future, the compiler will take precedence
14
15there is likely a problem that will prevent building python.
16If you see the messages above and are able to completely build python,
17please tell python-dev@python.org indicating your version of OpenBSD
18and any other relevant system configuration.
19
20The build error that occurs while making may look something like this:
21
22 /usr/include/sys/event.h:53: error: syntax error before "u_int"
23 /usr/include/sys/event.h:55: error: syntax error before "u_short"
24
25To fix this problem, you will probably need update Python's configure
26script to disable certain options. Search for a line that looks like:
27
28 OpenBSD/2.* | OpenBSD/3.@<:@012345678@:>@)
29
30If your version is not in that list, e.g., 3.9, add the version
31number. In this case, you would just need to add a 9 after the 8.
32If you modify configure.ac, you will need to regenerate configure
33with autoconf.
34
35If your version is already in the list, this is not a known problem.
36Please submit a bug report here:
37
38 http://sourceforge.net/tracker/?group_id=5470&atid=105470
39
README.coverity
1
2Coverity has a static analysis tool (Prevent) which is similar to Klocwork.
3They run their tool on the Python source code (SVN head) on a daily basis.
4The results are available at:
5
6 http://scan.coverity.com/
7
8About 20 people have access to the analysis reports. Other
9people can be added by request.
10
11Prevent was first run on the Python 2.5 source code in March 2006.
12There were originally about 100 defects reported. Some of these
13were false positives. Over 70 issues were uncovered.
14
15Each warning has a unique id and comments that can be made on it.
16When checking in changes due to a warning, the unique id
17as reported by the tool was added to the SVN commit message.
18
19False positives were annotated so that the comments can
20be reviewed and reversed if the analysis was incorrect.
21
22Contact python-dev@python.org for more information.
23
README.klocwork
1
2Klocwork has a static analysis tool (K7) which is similar to Coverity.
3They will run their tool on the Python source code on demand.
4The results are available at:
5
6 https://opensource.klocwork.com/
7
8Currently, only Neal Norwitz has access to the analysis reports. Other
9people can be added by request.
10
11K7 was first run on the Python 2.5 source code in mid-July 2006.
12This is after Coverity had been making their results available.
13There were originally 175 defects reported. Most of these
14were false positives. However, there were numerous real issues
15also uncovered.
16
17Each warning has a unique id and comments that can be made on it.
18When checking in changes due to a K7 report, the unique id
19as reported by the tool was added to the SVN commit message.
20A comment was added to the K7 warning indicating the SVN revision
21in addition to any analysis.
22
23False positives were also annotated so that the comments can
24be reviewed and reversed if the analysis was incorrect.
25
26A second run was performed on 10-Aug-2006. The tool was tuned to remove
27some false positives and perform some additional checks. ~150 new
28warnings were produced, primarily related to dereferencing NULL pointers.
29
30Contact python-dev@python.org for more information.
31
README.valgrind
1This document describes some caveats about the use of Valgrind with
2Python. Valgrind is used periodically by Python developers to try
3to ensure there are no memory leaks or invalid memory reads/writes.
4
5If you don't want to read about the details of using Valgrind, there
6are still two things you must do to suppress the warnings. First,
7you must use a suppressions file. One is supplied in
8Misc/valgrind-python.supp. Second, you must do one of the following:
9
10 * Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c,
11 then rebuild Python
12 * Uncomment the lines in Misc/valgrind-python.supp that
13 suppress the warnings for PyObject_Free and PyObject_Realloc
14
15If you want to use Valgrind more effectively and catch even more
16memory leaks, you will need to configure python --without-pymalloc.
17PyMalloc allocates a few blocks in big chunks and most object
18allocations don't call malloc, they use chunks doled about by PyMalloc
19from the big blocks. This means Valgrind can't detect
20many allocations (and frees), except for those that are forwarded
21to the system malloc. Note: configuring python --without-pymalloc
22makes Python run much slower, especially when running under Valgrind.
23You may need to run the tests in batches under Valgrind to keep
24the memory usage down to allow the tests to complete. It seems to take
25about 5 times longer to run --without-pymalloc.
26
27Apr 15, 2006:
28 test_ctypes causes Valgrind 3.1.1 to fail (crash).
29 test_socket_ssl should be skipped when running valgrind.
30 The reason is that it purposely uses uninitialized memory.
31 This causes many spurious warnings, so it's easier to just skip it.
32
33
34Details:
35--------
36Python uses its own small-object allocation scheme on top of malloc,
37called PyMalloc.
38
39Valgrind may show some unexpected results when PyMalloc is used.
40Starting with Python 2.3, PyMalloc is used by default. You can disable
41PyMalloc when configuring python by adding the --without-pymalloc option.
42If you disable PyMalloc, most of the information in this document and
43the supplied suppressions file will not be useful. As discussed above,
44disabling PyMalloc can catch more problems.
45
46If you use valgrind on a default build of Python, you will see
47many errors like:
48
49 ==6399== Use of uninitialised value of size 4
50 ==6399== at 0x4A9BDE7E: PyObject_Free (obmalloc.c:711)
51 ==6399== by 0x4A9B8198: dictresize (dictobject.c:477)
52
53These are expected and not a problem. Tim Peters explains
54the situation:
55
56 PyMalloc needs to know whether an arbitrary address is one
57 that's managed by it, or is managed by the system malloc.
58 The current scheme allows this to be determined in constant
59 time, regardless of how many memory areas are under pymalloc's
60 control.
61
62 The memory pymalloc manages itself is in one or more "arenas",
63 each a large contiguous memory area obtained from malloc.
64 The base address of each arena is saved by pymalloc
65 in a vector. Each arena is carved into "pools", and a field at
66 the start of each pool contains the index of that pool's arena's
67 base address in that vector.
68
69 Given an arbitrary address, pymalloc computes the pool base
70 address corresponding to it, then looks at "the index" stored
71 near there. If the index read up is out of bounds for the
72 vector of arena base addresses pymalloc maintains, then
73 pymalloc knows for certain that this address is not under
74 pymalloc's control. Otherwise the index is in bounds, and
75 pymalloc compares
76
77 the arena base address stored at that index in the vector
78
79 to
80
81 the arbitrary address pymalloc is investigating
82
83 pymalloc controls this arbitrary address if and only if it lies
84 in the arena the address's pool's index claims it lies in.
85
86 It doesn't matter whether the memory pymalloc reads up ("the
87 index") is initialized. If it's not initialized, then
88 whatever trash gets read up will lead pymalloc to conclude
89 (correctly) that the address isn't controlled by it, either
90 because the index is out of bounds, or the index is in bounds
91 but the arena it represents doesn't contain the address.
92
93 This determination has to be made on every call to one of
94 pymalloc's free/realloc entry points, so its speed is critical
95 (Python allocates and frees dynamic memory at a ferocious rate
96 -- everything in Python, from integers to "stack frames",
97 lives in the heap).
98