1                                       TODO for TestNG
2
3* Pass the XmlTest in @Before/@After methods
4* Allow a testng.xml file to be passed when -testjar is used
5* Add onStart to IConfigurationListener (create a new interface, actually)
6* Add timeout to @Before/@After
7* Pass parameters from ant
8* Make it possible to specify groups on command line and classes in testng.xml
9(and any combinations thereof:  command line, ant, testng.xml)
10* DataProvider index in testng.xml
11* Create a servlet for remote driving
12* Add time-outs at the testng.xml.  Also:  test and suite time-outs? http://tinyurl.com/kbwxq
13* Add working dir to the ant task
14* Add a servlet so TestNG can be invoked from a web browser
15* Make it possible to add listeners from the Eclipse plug-in
16
17Doc:
18
19* Document the fact that @Test methods with return values are ignored.
20
21===========================================================================
22Older TODO's:
23
24* Show enabled=false methods in the reports, as well as methods
25in groups that were not run
26* Multi-threading for invocationCount and maybe for <test> too
27* Annotation to specify that a method should be called concurrently by n threads
28(on second thought, we should do that for an entire group)
29* more thread ideas: http://www.theserverside.com/news/thread.tss?thread_id=38922
30* package support for command line and ant
31* Parameters for classes (to be passed as parameters to constructors)
32* testng-dist.zip should contain a top-level directory
33* For dependent methods, if the user is trying to run an incomplete graph
34(A depends on B but B is being excluded from the run), what to do?  Ignore
35the exclusion on B and run it anyway, or abort with an exception explaining
36what's happening?)
37* Make timeOut() work with milliseconds (but keep seconds for backward
38compatibility)
39* Improve the plug-in API so people can add listeners without having to
40modify TestRunner
41* Use factories for the programmatic API.
42* Add dynamic generation of tests
43* Make Javadoc comments over methods appear in the final report
44
45Documentation:
46* IHookable
47* List<IReporter>
48
49
50DONE
51
52* Retry patch
53* If a method with invocationCount fails, don't run the others
54* Allow multiple listeners in ant task
55* Add working dir to the ant task
56* Introduce "test" and "suite" parameters to @Test at the class level to
57avoid having to use testng.xml
58* Remove TestNG stack traces from the report
59* When 0 tests were run, exit with an error http://tinyurl.com/ftng6
60* Show all the extra output for all methods in a single
61dedicated page
62* report API
63* Show parameters used to invoke a specific test
64* show skipped groups/methods in HTML report
65* beforeTestGroups
66* <testng classfileset> doesn't add to classpath
67* threadPoolSize
68* Parameter logging
69* JavaDoc for org.testng.TestNG
70* org.testng.Reporter in the HTML report (screenshot ready)
71* Document @Parameters only works for @Test (should mention @Factory and @Configuration)
72* Document: <testng classfileset> doesn't add to classpath
73* Document org.testng.Reporter in the HTML report (screenshot ready)
74* Document parameter logging
75* JavaDoc for org.testng.TestNG
76* Document threadPoolSize
77* Make sure it can run several times in the same JVM
78* Implement invocationCount and successPercentage
79* Support multiple testng.xml (TestNG allows it but not the reporters
80yet)
81* The HTML reporter collapses all the suites into one report, we should
82create one HTML report per suite
83* testng-failed.xml should contain the parameters of testng.xml (if any)
84* Create a testng-failed.xml that includes dependent methods
85* Generic reported with compare(ITestResult, ITestResult) for
86easier reporter for "slowest method first" or generate testng-failed.xml
87* Iterator factories
88* configuration methods don't respect inheritance
89- build.xml should issue a clear error message if trying to build with JDK1.4
90- Implement <tasdkdef resource="testnganttasks"> so we can define
91ant tasks for TestNG and JUnitConverter automatically
92- Write documentation to declare ant task in section 3.2.8
93- Documentation for alwaysRun
94- Allow to specify packages or prefix in the <classes> tag:
95   <classes prefix="com.beust.testng"><class name="A"><class name="B"> />
96- New assert classes
97- New ways to launch
98- JUnitConverter documentation
99- new beforeSuite/afterSuite
100
101* in testng-failures.xml include the beforeSuite/afterSuite methods (very tricky)
102* Provide log.properties configuration (not using log any more)
103* Make @ExpectedExceptions fail if no exception is thrown
104* Make timeOut() work in non-parallel mode (the default mode needs to become
105parallel=true thread-count=1
106* The exception thrown when a test passes with a @ExpectedExceptions is not
107available via the TestNG API: ITestResult.getThrowable().
108* Add assert API for arrays and collections (undecided yet:  partial asserting)
109* dependsOnMethods
110Allow to specify <groups> at the <suite> level
111Make TestNG run on class files and not just on testng.xml
112Make TestNG run on a jar file that has a testng.xml file in its root or just on all
113  the classes inside that jar file.
114
115Implement parameter passing of tests:  define a property in the properties
116file and pass it to the test method:
117
118@Test(params = { "${fn}", "${ln}" }
119public void testNames(String firstName, String lastName) {
120}
121
122Run groups of groups
123List all tests that will be run, or show methods per group
124HTML generation
125Make test and class methods discoverable
126  JUnit adapter
127Multiple ExpectedException
128Inheritance
129Test listeners
130Group regexps for launching
131
132====
133
134A new comment has been posted on your blog Otaku, Cedric's weblog, on entry
135#149 (The poor shape of Unit Testing).
136http://beust.com/weblog/archives/000149.html
137
138IP Address: 68.72.49.189
139Name: Curt Cox
140Email Address: curtcox@gmail.com
141URL:
142
143Comments:
144
145For whatever its worth, here are my problems with JUnit.  I've largely developed work-arounds.
146
1471. Interface-based testing is awkward.
1482. Only fast-fail tests are supported.
1493. The design is brittle, poorly documented, and thus hard to extend.
150
151How should interface-based testing be handled in TestNG?  A nice feature would be bundled tests for interfaces in the java* namespace that are automatically applied.  In other words:
152- all classes will fail a (supressible) test if they violate the contract for equals() and hashCode()
153- classes that implement Comparable will fail a test if they fail to implement Comparable
154- and likewise for java.io.Serializable, java.util.Map, java.util.List, etc...
155
156When I say all tests are fast-fail in JUnit, what I mean is that the first failed assertion in a method causes the test to exit.  While this is usually desirable, a more conversational style of tests can sometimes be much easier to read and write.  Such a conversational test doesn't generate a simple failure, but rather a score.  The score could be either x of y passed or x percent passed.  The important part is that the first failure doesn't terminate the test.
157
158As I said, I've largely developed work-arounds for doing these in JUnit, but developing tools for conversational tests that play nice with the various JUnit runners was a real challenge.  The exact contract that Eclipse expects of JUnit tests turns out to be different than what either of the bundled runners or the Ant task expect.  Anyone who considers either the JUnit code or interfaces well-documented has a much different concept of well-documented than I do.
159
160
161