Mercurial > cpython-lru_cache-weakref

annotate Lib/test/README @ 58925:eecf1bef052d 2.6

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merged revisions 78207 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk ........ r78207 | ezio.melotti | 2010-02-17 01:26:09 +0200 (Wed, 17 Feb 2010) | 1 line #7930: fix stripid ........
author Ezio Melotti <ezio.melotti@gmail.com>
date Tue, 16 Feb 2010 23:29:44 +0000
parents f2f738a437f8
children 787b969d37f0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
1 +++++++++++++++++++++++++++++++
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
2 Writing Python Regression Tests
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
3 +++++++++++++++++++++++++++++++
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
4
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
5 :Author: Skip Montanaro
44224
f2f738a437f8 correct email address
Skip Montanaro <skip@pobox.com>
parents: 41534
diff changeset
6 :Contact: skip@pobox.com
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
7
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
8 Introduction
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
9 ============
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
10
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
11 If you add a new module to Python or modify the functionality of an existing
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
12 module, you should write one or more test cases to exercise that new
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
13 functionality. There are different ways to do this within the regression
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
14 testing facility provided with Python; any particular test should use only
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
15 one of these options. Each option requires writing a test module using the
27651
59b4ebc29d53 Fix typos.
Walter Dörwald <walter@livinglogic.de>
parents: 26447
diff changeset
16 conventions of the selected option:
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
17
41511
3c1205ca442b Convert PyUnit -> unittest.
Collin Winter <collinw@gmail.com>
parents: 40217
diff changeset
18 - unittest_ based tests
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
19 - doctest_ based tests
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
20 - "traditional" Python test modules
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
21
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
22 Regardless of the mechanics of the testing approach you choose,
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
23 you will be writing unit tests (isolated tests of functions and objects
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
24 defined by the module) using white box techniques. Unlike black box
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
25 testing, where you only have the external interfaces to guide your test case
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
26 writing, in white box testing you can see the code being tested and tailor
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
27 your test cases to exercise it more completely. In particular, you will be
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
28 able to refer to the C and Python code in the CVS repository when writing
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
29 your regression test cases.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
30
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
31 .. _unittest: http://www.python.org/doc/current/lib/module-unittest.html
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
32 .. _doctest: http://www.python.org/doc/current/lib/module-doctest.html
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
33
41511
3c1205ca442b Convert PyUnit -> unittest.
Collin Winter <collinw@gmail.com>
parents: 40217
diff changeset
34 unittest-based tests
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
35 ------------------
41511
3c1205ca442b Convert PyUnit -> unittest.
Collin Winter <collinw@gmail.com>
parents: 40217
diff changeset
36 The unittest_ framework is based on the ideas of unit testing as espoused
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
37 by Kent Beck and the `Extreme Programming`_ (XP) movement. The specific
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
38 interface provided by the framework is tightly based on the JUnit_
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
39 Java implementation of Beck's original SmallTalk test framework. Please
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
40 see the documentation of the unittest_ module for detailed information on
41511
3c1205ca442b Convert PyUnit -> unittest.
Collin Winter <collinw@gmail.com>
parents: 40217
diff changeset
41 the interface and general guidelines on writing unittest-based tests.
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
42
41534
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
43 The test_support helper module provides a function for use by
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
44 unittest-based tests in the Python regression testing framework,
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
45 ``run_unittest()``. This is the primary way of running tests in the
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
46 standard library. You can pass it any number of the following:
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
47
41534
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
48 - classes derived from or instances of ``unittest.TestCase`` or
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
49 ``unittest.TestSuite``. These will be handed off to unittest for
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
50 converting into a proper TestSuite instance.
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
51
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
52 - a string; this must be a key in sys.modules. The module associated with
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
53 that string will be scanned by ``unittest.TestLoader.loadTestsFromModule``.
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
54 This is usually seen as ``test_support.run_unittest(__name__)`` in a test
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
55 module's ``test_main()`` function. This has the advantage of picking up
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
56 new tests automatically, without you having to add each new test case
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
57 manually.
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
58
23258
9a9bec10aa8c Rewrote the PyUnit description so that it now recommends to use
Barry Warsaw <barry@python.org>
parents: 20157
diff changeset
59 All test methods in the Python regression framework have names that
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
60 start with "``test_``" and use lower-case names with words separated with
23258
9a9bec10aa8c Rewrote the PyUnit description so that it now recommends to use
Barry Warsaw <barry@python.org>
parents: 20157
diff changeset
61 underscores.
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
62
24961
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
63 Test methods should *not* have docstrings! The unittest module prints
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
64 the docstring if there is one, but otherwise prints the function name
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
65 and the full class name. When there's a problem with a test, the
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
66 latter information makes it easier to find the source for the test
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
67 than the docstring.
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
68
41511
3c1205ca442b Convert PyUnit -> unittest.
Collin Winter <collinw@gmail.com>
parents: 40217
diff changeset
69 All unittest-based tests in the Python test suite use boilerplate that
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
70 looks like this (with minor variations)::
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
71
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
72 import unittest
24364
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
73 from test import test_support
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
74
23258
9a9bec10aa8c Rewrote the PyUnit description so that it now recommends to use
Barry Warsaw <barry@python.org>
parents: 20157
diff changeset
75 class MyTestCase1(unittest.TestCase):
24961
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
76
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
77 # Define setUp and tearDown only if needed
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
78
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
79 def setUp(self):
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
80 unittest.TestCase.setUp(self)
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
81 ... additional initialization...
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
82
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
83 def tearDown(self):
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
84 ... additional finalization...
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
85 unittest.TestCase.tearDown(self)
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
86
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
87 def test_feature_one(self):
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
88 # Testing feature one
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
89 ...unit test for feature one...
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
90
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
91 def test_feature_two(self):
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
92 # Testing feature two
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
93 ...unit test for feature two...
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
94
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
95 ...etc...
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
96
23258
9a9bec10aa8c Rewrote the PyUnit description so that it now recommends to use
Barry Warsaw <barry@python.org>
parents: 20157
diff changeset
97 class MyTestCase2(unittest.TestCase):
24961
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
98 ...same structure as MyTestCase1...
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
99
043eba518c68 Document that docstrings are verboten for test functions.
Guido van Rossum <guido@python.org>
parents: 24365
diff changeset
100 ...etc...
23258
9a9bec10aa8c Rewrote the PyUnit description so that it now recommends to use
Barry Warsaw <barry@python.org>
parents: 20157
diff changeset
101
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
102 def test_main():
41534
6c4d7678af5e Standardize on test.test_support.run_unittest() (as opposed to a mix of run_unittest() and run_suite()). Also, add functionality to run_unittest() that admits usage of unittest.TestLoader.loadTestsFromModule().
Collin Winter <collinw@gmail.com>
parents: 41511
diff changeset
103 test_support.run_unittest(__name__)
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
104
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
105 if __name__ == "__main__":
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
106 test_main()
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
107
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
108 This has the advantage that it allows the unittest module to be used
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
109 as a script to run individual tests as well as working well with the
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
110 regrtest framework.
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
111
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
112 .. _Extreme Programming: http://www.extremeprogramming.org/
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
113 .. _JUnit: http://www.junit.org/
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
114
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
115 doctest based tests
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
116 -------------------
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
117 Tests written to use doctest_ are actually part of the docstrings for
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
118 the module being tested. Each test is written as a display of an
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
119 interactive session, including the Python prompts, statements that would
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
120 be typed by the user, and the output of those statements (including
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
121 tracebacks, although only the exception msg needs to be retained then).
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
122 The module in the test package is simply a wrapper that causes doctest
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
123 to run over the tests in the module. The test for the difflib module
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
124 provides a convenient example::
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
125
24365
a7186bb9142c A few updates about how/where to import test_support from.
Barry Warsaw <barry@python.org>
parents: 24364
diff changeset
126 import difflib
a7186bb9142c A few updates about how/where to import test_support from.
Barry Warsaw <barry@python.org>
parents: 24364
diff changeset
127 from test import test_support
19817
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
128 test_support.run_doctest(difflib)
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
129
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
130 If the test is successful, nothing is written to stdout (so you should not
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
131 create a corresponding output/test_difflib file), but running regrtest
19817
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
132 with -v will give a detailed report, the same as if passing -v to doctest.
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
133
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
134 A second argument can be passed to run_doctest to tell doctest to search
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
135 ``sys.argv`` for -v instead of using test_support's idea of verbosity. This
19817
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
136 is useful for writing doctest-based tests that aren't simply running a
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
137 doctest'ed Lib module, but contain the doctests themselves. Then at
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
138 times you may want to run such a test directly as a doctest, independent
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
139 of the regrtest framework. The tail end of test_descrtut.py is a good
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
140 example::
19817
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
141
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
142 def test_main(verbose=None):
24365
a7186bb9142c A few updates about how/where to import test_support from.
Barry Warsaw <barry@python.org>
parents: 24364
diff changeset
143 from test import test_support, test_descrtut
a7186bb9142c A few updates about how/where to import test_support from.
Barry Warsaw <barry@python.org>
parents: 24364
diff changeset
144 test_support.run_doctest(test_descrtut, verbose)
19817
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
145
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
146 if __name__ == "__main__":
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
147 test_main(1)
b7396287cf65 Teach regrtest how to pass on doctest failure msgs. This is done via a
Tim Peters <tim.peters@gmail.com>
parents: 17950
diff changeset
148
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
149 If run via regrtest, ``test_main()`` is called (by regrtest) without
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
150 specifying verbose, and then test_support's idea of verbosity is used. But
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
151 when run directly, ``test_main(1)`` is called, and then doctest's idea of
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
152 verbosity is used.
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
153
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
154 See the documentation for the doctest module for information on
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
155 writing tests using the doctest framework.
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
156
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
157 "traditional" Python test modules
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
158 ---------------------------------
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
159 The mechanics of how the "traditional" test system operates are fairly
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
160 straightforward. When a test case is run, the output is compared with the
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
161 expected output that is stored in .../Lib/test/output. If the test runs to
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
162 completion and the actual and expected outputs match, the test succeeds, if
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
163 not, it fails. If an ``ImportError`` or ``test_support.TestSkipped`` error
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
164 is raised, the test is not run.
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
165
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
166 Executing Test Cases
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
167 ====================
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
168 If you are writing test cases for module spam, you need to create a file
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
169 in .../Lib/test named test_spam.py. In addition, if the tests are expected
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
170 to write to stdout during a successful run, you also need to create an
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
171 expected output file in .../Lib/test/output named test_spam ("..."
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
172 represents the top-level directory in the Python source tree, the directory
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
173 containing the configure script). If needed, generate the initial version
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
174 of the test output file by executing::
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
175
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
176 ./python Lib/test/regrtest.py -g test_spam.py
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
177
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
178 from the top-level directory.
17935
cfe081c9553e Update to reflect recent changes to regrtest and the new approaches to
Fred Drake <fdrake@acm.org>
parents: 15301
diff changeset
179
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
180 Any time you modify test_spam.py you need to generate a new expected
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
181 output file. Don't forget to desk check the generated output to make sure
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
182 it's really what you expected to find! All in all it's usually better
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
183 not to have an expected-out file (note that doctest- and unittest-based
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
184 tests do not).
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
185
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
186 To run a single test after modifying a module, simply run regrtest.py
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
187 without the -g flag::
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
188
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
189 ./python Lib/test/regrtest.py test_spam.py
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
190
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
191 While debugging a regression test, you can of course execute it
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
192 independently of the regression testing framework and see what it prints::
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
193
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
194 ./python Lib/test/test_spam.py
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
195
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
196 To run the entire test suite:
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
197
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
198 - [UNIX, + other platforms where "make" works] Make the "test" target at the
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
199 top level::
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
200
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
201 make test
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
202
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
203 - [WINDOWS] Run rt.bat from your PCBuild directory. Read the comments at
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
204 the top of rt.bat for the use of special -d, -O and -q options processed
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
205 by rt.bat.
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
206
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
207 - [OTHER] You can simply execute the two runs of regrtest (optimized and
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
208 non-optimized) directly::
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
209
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
210 ./python Lib/test/regrtest.py
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
211 ./python -O Lib/test/regrtest.py
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
212
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
213 But note that this way picks up whatever .pyc and .pyo files happen to be
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
214 around. The makefile and rt.bat ways run the tests twice, the first time
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
215 removing all .pyc and .pyo files from the subtree rooted at Lib/.
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
216
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
217 Test cases generate output based upon values computed by the test code.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
218 When executed, regrtest.py compares the actual output generated by executing
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
219 the test case with the expected output and reports success or failure. It
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
220 stands to reason that if the actual and expected outputs are to match, they
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
221 must not contain any machine dependencies. This means your test cases
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
222 should not print out absolute machine addresses (e.g. the return value of
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
223 the id() builtin function) or floating point numbers with large numbers of
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
224 significant digits (unless you understand what you are doing!).
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
225
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
226
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
227 Test Case Writing Tips
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
228 ======================
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
229 Writing good test cases is a skilled task and is too complex to discuss in
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
230 detail in this short document. Many books have been written on the subject.
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
231 I'll show my age by suggesting that Glenford Myers' `"The Art of Software
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
232 Testing"`_, published in 1979, is still the best introduction to the subject
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
233 available. It is short (177 pages), easy to read, and discusses the major
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
234 elements of software testing, though its publication predates the
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
235 object-oriented software revolution, so doesn't cover that subject at all.
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
236 Unfortunately, it is very expensive (about $100 new). If you can borrow it
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
237 or find it used (around $20), I strongly urge you to pick up a copy.
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
238
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
239 The most important goal when writing test cases is to break things. A test
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
240 case that doesn't uncover a bug is much less valuable than one that does.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
241 In designing test cases you should pay attention to the following:
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
242
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
243 * Your test cases should exercise all the functions and objects defined
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
244 in the module, not just the ones meant to be called by users of your
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
245 module. This may require you to write test code that uses the module
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
246 in ways you don't expect (explicitly calling internal functions, for
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
247 example - see test_atexit.py).
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
248
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
249 * You should consider any boundary values that may tickle exceptional
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
250 conditions (e.g. if you were writing regression tests for division,
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
251 you might well want to generate tests with numerators and denominators
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
252 at the limits of floating point and integer numbers on the machine
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
253 performing the tests as well as a denominator of zero).
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
254
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
255 * You should exercise as many paths through the code as possible. This
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
256 may not always be possible, but is a goal to strive for. In
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
257 particular, when considering if statements (or their equivalent), you
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
258 want to create test cases that exercise both the true and false
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
259 branches. For loops, you should create test cases that exercise the
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
260 loop zero, one and multiple times.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
261
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
262 * You should test with obviously invalid input. If you know that a
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
263 function requires an integer input, try calling it with other types of
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
264 objects to see how it responds.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
265
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
266 * You should test with obviously out-of-range input. If the domain of a
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
267 function is only defined for positive integers, try calling it with a
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
268 negative integer.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
269
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
270 * If you are going to fix a bug that wasn't uncovered by an existing
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
271 test, try to write a test case that exposes the bug (preferably before
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
272 fixing it).
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
273
15301
2a9d2765625e Added note saying to use test_support.TESTFN for a temporary filename,
Fred Drake <fdrake@acm.org>
parents: 14127
diff changeset
274 * If you need to create a temporary file, you can use the filename in
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
275 ``test_support.TESTFN`` to do so. It is important to remove the file
15301
2a9d2765625e Added note saying to use test_support.TESTFN for a temporary filename,
Fred Drake <fdrake@acm.org>
parents: 14127
diff changeset
276 when done; other tests should be able to use the name without cleaning
2a9d2765625e Added note saying to use test_support.TESTFN for a temporary filename,
Fred Drake <fdrake@acm.org>
parents: 14127
diff changeset
277 up after your test.
2a9d2765625e Added note saying to use test_support.TESTFN for a temporary filename,
Fred Drake <fdrake@acm.org>
parents: 14127
diff changeset
278
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
279 .. _"The Art of Software Testing":
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
280 http://www.amazon.com/exec/obidos/ISBN=0471043281
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
281
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
282 Regression Test Writing Rules
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
283 =============================
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
284 Each test case is different. There is no "standard" form for a Python
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
285 regression test case, though there are some general rules (note that
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
286 these mostly apply only to the "classic" tests; unittest_- and doctest_-
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
287 based tests should follow the conventions natural to those frameworks)::
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
288
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
289 * If your test case detects a failure, raise ``TestFailed`` (found in
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
290 ``test.test_support``).
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
291
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
292 * Import everything you'll need as early as possible.
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
293
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
294 * If you'll be importing objects from a module that is at least
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
295 partially platform-dependent, only import those objects you need for
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
296 the current test case to avoid spurious ``ImportError`` exceptions
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
297 that prevent the test from running to completion.
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
298
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
299 * Print all your test case results using the ``print`` statement. For
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
300 non-fatal errors, print an error message (or omit a successful
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
301 completion print) to indicate the failure, but proceed instead of
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
302 raising ``TestFailed``.
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
303
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
304 * Use ``assert`` sparingly, if at all. It's usually better to just print
14127
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
305 what you got, and rely on regrtest's got-vs-expected comparison to
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
306 catch deviations from what you expect. ``assert`` statements aren't
14127
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
307 executed at all when regrtest is run in -O mode; and, because they
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
308 cause the test to stop immediately, can lead to a long & tedious
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
309 test-fix, test-fix, test-fix, ... cycle when things are badly broken
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
310 (and note that "badly broken" often includes running the test suite
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
311 for the first time on new platforms or under new implementations of
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
312 the language).
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
313
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
314 Miscellaneous
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
315 =============
24365
a7186bb9142c A few updates about how/where to import test_support from.
Barry Warsaw <barry@python.org>
parents: 24364
diff changeset
316 There is a test_support module in the test package you can import for
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
317 your test case. Import this module using either::
24364
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
318
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
319 import test.test_support
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
320
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
321 or::
24364
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
322
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
323 from test import test_support
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
324
aa0d2d51fbde Get rid of relative imports in all unittests. Now anything that
Barry Warsaw <barry@python.org>
parents: 23258
diff changeset
325 test_support provides the following useful objects:
13184
efecbbf72765 Describe a bit about writing test cases for Python...
Skip Montanaro <skip@pobox.com>
parents:
diff changeset
326
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
327 * ``TestFailed`` - raise this exception when your regression test detects
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
328 a failure.
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
329
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
330 * ``TestSkipped`` - raise this if the test could not be run because the
14104
da69b7d1f20f Add TestSkipped as another interesting item defined by test_support.
Fred Drake <fdrake@acm.org>
parents: 13858
diff changeset
331 platform doesn't offer all the required facilities (like large
da69b7d1f20f Add TestSkipped as another interesting item defined by test_support.
Fred Drake <fdrake@acm.org>
parents: 13858
diff changeset
332 file support), even if all the required modules are available.
da69b7d1f20f Add TestSkipped as another interesting item defined by test_support.
Fred Drake <fdrake@acm.org>
parents: 13858
diff changeset
333
28901
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
334 * ``ResourceDenied`` - this is raised when a test requires a resource that
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
335 is not available. Primarily used by 'requires'.
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
336
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
337 * ``verbose`` - you can use this variable to control print output. Many
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
338 modules use it. Search for "verbose" in the test_*.py files to see
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
339 lots of examples.
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
340
28901
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
341 * ``forget(module_name)`` - attempts to cause Python to "forget" that it
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
342 loaded a module and erase any PYC files.
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
343
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
344 * ``is_resource_enabled(resource)`` - Returns a boolean based on whether
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
345 the resource is enabled or not.
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
346
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
347 * ``requires(resource [, msg])`` - if the required resource is not
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
348 available the ResourceDenied exception is raised.
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
349
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
350 * ``verify(condition, reason='test failed')``. Use this instead of::
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
351
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
352 assert condition[, reason]
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
353
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
354 ``verify()`` has two advantages over ``assert``: it works even in -O
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
355 mode, and it raises ``TestFailed`` on failure instead of
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
356 ``AssertionError``.
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
357
28901
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
358 * ``have_unicode`` - true if Unicode is available, false otherwise.
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
359
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
360 * ``is_jython`` - true if the interpreter is Jython, false otherwise.
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
361
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
362 * ``TESTFN`` - a string that should always be used as the filename when
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
363 you need to create a temp file. Also use ``try``/``finally`` to
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
364 ensure that your temp files are deleted before your test completes.
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
365 Note that you cannot unlink an open file on all operating systems, so
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
366 also be sure to close temp files before trying to unlink them.
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
367
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
368 * ``sortdict(dict)`` - acts like ``repr(dict.items())``, but sorts the
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
369 items first. This is important when printing a dict value, because
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
370 the order of items produced by ``dict.items()`` is not defined by the
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
371 language.
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
372
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
373 * ``findfile(file)`` - you can call this function to locate a file
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
374 somewhere along sys.path or in the Lib/test tree - see
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
375 test_linuxaudiodev.py for an example of its use.
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
376
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
377 * ``fcmp(x,y)`` - you can call this function to compare two floating
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
378 point numbers when you expect them to only be approximately equal
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
379 withing a fuzz factor (``test_support.FUZZ``, which defaults to 1e-6).
13648
8ee4a209a9ef restructured a bit and added some more content...
Skip Montanaro <skip@pobox.com>
parents: 13184
diff changeset
380
40217
13b8ad679def Convert test_global, test_scope and test_grammar to unittest.
Georg Brandl <georg@python.org>
parents: 28901
diff changeset
381 * ``check_syntax_error(testcase, statement)`` - make sure that the
13b8ad679def Convert test_global, test_scope and test_grammar to unittest.
Georg Brandl <georg@python.org>
parents: 28901
diff changeset
382 statement is *not* correct Python syntax.
28901
aab0dbe3c008 Flesh out test_support docs.
Brett Cannon <bcannon@gmail.com>
parents: 27651
diff changeset
383
14127
26768e21acf9 Rehabilitate autotest.py.
Tim Peters <tim.peters@gmail.com>
parents: 14104
diff changeset
384
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
385 Some Non-Obvious regrtest Features
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
386 ==================================
17936
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
387 * Automagic test detection: When you create a new test file
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
388 test_spam.py, you do not need to modify regrtest (or anything else)
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
389 to advertise its existence. regrtest searches for and runs all
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
390 modules in the test directory with names of the form test_xxx.py.
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
391
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
392 * Miranda output: If, when running test_spam.py, regrtest does not
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
393 find an expected-output file test/output/test_spam, regrtest
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
394 pretends that it did find one, containing the single line
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
395
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
396 test_spam
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
397
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
398 This allows new tests that don't expect to print anything to stdout
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
399 to not bother creating expected-output files.
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
400
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
401 * Two-stage testing: To run test_spam.py, regrtest imports test_spam
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
402 as a module. Most tests run to completion as a side-effect of
3e69d7db5f8d Remove test_doctest's expected-output file.
Tim Peters <tim.peters@gmail.com>
parents: 17935
diff changeset
403 getting imported. After importing test_spam, regrtest also executes
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
404 ``test_spam.test_main()``, if test_spam has a ``test_main`` attribute.
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
405 This is rarely required with the "traditional" Python tests, and
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
406 you shouldn't create a module global with name test_main unless
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
407 you're specifically exploiting this gimmick. This usage does
41511
3c1205ca442b Convert PyUnit -> unittest.
Collin Winter <collinw@gmail.com>
parents: 40217
diff changeset
408 prove useful with unittest-based tests as well, however; defining
26447
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
409 a ``test_main()`` which is run by regrtest and a script-stub in the
7a09b36ca379 SF #641170, reST version of Lib/test/README
Neal Norwitz <nnorwitz@gmail.com>
parents: 24961
diff changeset
410 test module ("``if __name__ == '__main__': test_main()``") allows
20157
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
411 the test to be used like any other Python test and also work
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
412 with the unittest.py-as-a-script approach, allowing a developer
8ba3287183e0 Reflect recent refinements of the regression testing framework.
Fred Drake <fdrake@acm.org>
parents: 19819
diff changeset
413 to run specific tests from the command line.