Jim and I have been working on a number of rather significant changes to the framework for Beta 2, which I'd like to list here.
Up until the public release of Beta 1, we had limited our exposure of xUnit.net to our own team, as well as a few select friends. Now that we've released Beta 1, it's received a lot of critical review, and we've had many good and spirited conversations with people about the pros and cons of our framework. Many of these conversations took place at the ALT.NET conference a few weeks ago in Austin, but many more took place in blogs, via e-mail, and even over beers as people visited Redmond. We took a lot of that feedback to heart and fed it into what will be Beta 2.
During the process of getting the engine ready to support both Resharper (which will ship in Beta 2) and a future GUI test runner (which will not), we needed to flip the execution engine inside out, so to speak. Our existing system of using delegation + the Command pattern worked well for our console runner and TestDriven.NET, but with systems that wanted to take a more active interest in what got run (and when), it fell a little short. Because the execution engine is itself part of the extensibility of xUnit.net, it also meant that we were going to have to break some of the extensibility points.
We decided to, well, break everything all at once. :) So if you're using Beta 1, here is a list of the things we've changed going forward in Beta 2.
In addition to these changes, we've also fixed several bugs. See the
Beta 2 release for a complete list of associated work items.
Changes for xUnit.net users
Renamed [Test] to [Fact]
This was something we had been considering since before we even released Beta 1, and some of the feedback we got told us it was probably the right thing to do.
The primary motivation is that it changes the expectations that what you do with this framework is capital-T Testing (aka, quality assurance). In reality, Test Driven Development is not really about testing at all. It's an example-driven design methodology which uses code to express the intentions of the class that is under design. The fact that it increases quality is a secondary benefit, and should not be considered a replacement for the work done by capital-T Testers. In Brian Marick's
four quadrants of exploration through example, TDD-style code really only represents one of those quadrants (the
technology-facing programmer support quadrant).
Additionally, as a word,
[Fact] has very good symmetry with
[Theory]. The two kinds of tests are fundamentally different; a
[Fact] is an invariant statement which is always true, and a
[Theory] is a statement which is true for all the given input values. In Marick's quadrants,
[Theory]s are the
business-facing team support.
Renamed [Property] to [Trait]
We've heard feedback that naming the attribute Property is a bit too generic, and likely to conflict with other attributes named Property. We considered being more specific (say, TestProperty) but in the end decided that Trait was an accurate word that is not in common use. Properties, nee Traits, are used to provided descriptive metadata about a test; common uses might include the original author of the code, the test case which it covers, etc. While the runner does not use these Traits, it does output them into the XML so that they can be viewed with the results.
Replaced ITestFixture with IUseFixture<T>
Jamie Cansdale (the author of
TestDriven.NET) made this suggestion after we'd mentioned our unhappiness associated with
ITestFixture. The problems we were trying to solve were related to the apparent dichotomy of implementing an interface which in turn needed to set static data, as well as need to instantiate the test class one extra time.
Now users can decorate their test classes with one or more instances of
IUseFixture<T> to indicate that they use a fixture class. In turn, they must implement a method (
void SetFixture(T fixture)). The test runner will create an instance of the fixture class (T) once before running any tests, and then just before running each test, it calls SetFixture with the fixture instance. Once all the tests have run, the fixture is disposed (if it implements
IDisposable).
This offers several benefits. In addition to getting two issues mentioned above, it also offers a clean separation of the concerns of "fixture" from the concerns of "test". It allows fixture behavior to be reused across multiple test classes. It also allows a "mix-in" style behavior, since a test class can utilize multiple fixtures at once.
Added support for IEquatable
If
Assert.Equal is handed objects which implement
IEquatable, it can utilize that.
Added support for private test methods
Although we don't personally use (nor recommend) private test methods, we heard from numerous people who have non-technical constraints that require them to use private test methods.
Changes for xUnit.net extenders
Note: We eliminated the Xunit.Runner namespace, merging its content into the Xunit.Sdk namespace. Our original purpose for separating the two namespaces had more or less disappered when the extensibility model was fully fleshed out.
Changes to ITestClassCommand
The changes required to
ITestClassCommand to support Resharper were quite extensive. Previously, a single Execute() method was required; now, there are many methods to be implemented that are about enumerating test methods and test commands. Extenders who wished to use
[RunWith] need to provide an implementation of this interface.
For test method enumeration, an abstraction layer has been added in the form of
ITypeInfo,
IMethodInfo, and
IAttributeInfo. This abstraction was necessary, since the pure reflection versions (
Type,
MethodInfo, and
AttributeInfo) aren't available to code being evaluation by Resharper. To wrap the reflection counterparts into the new interfaces, use the
Reflector.Wrap() static method.
In addition to test method enumeration, there are two execution methods (
ClassStart and
ClassFinish) which are used by the execution engine.
ClassStart is called before any test methods of a class have been called, and
ClassFinish is called after all the test methods have been called.
A property named ObjectUnderTest was added to support runners which wish to use the same object instance for all test commands. If you want the framework to automatically create a new test class instance for each test command, you may return null.
Finally, a method named
ChooseNextTest has been provided to allow the test class to say which order tests should be run in. The implementation in
TestClassCommand (used for
[Fact] tests) runs the tests in random order. Note that some runners will choose their own run order (notably Resharper), so this feature is not always used.
Changes to ITestCommand
Implementers of
ITestCommand must now implement a
Parameters property, which should provide a list of the parameters used to invoke the test. For tests without parameters, you may return null.
Additionally, the Execute() method now passes along the existing instance of the test class, created by the CreationTestCommand that the execution engine wraps around the execution chain. This was necessary to support
IUseFixture<T> and means that test commands are no longer responsible for knowing how to create the test class.
Changes to FactAttribute (used to be TestAttribute)
Previously, you needed to override public method
CreateTestCommands; now, you should override protected method
EnumerateTestCommands. Execution engine changes necessitated this change. Otherwise, the work that is expected from this method is identical to Beta 1.
Added support for test methods with return values
Although it is traditional that you attribute the method with the test code in it, there's really nothing that requires that to actually be true in xUnit.net. In order to support this non-traditional use of method attribution, we allowed test methods to return values. Note that if a traditional test method (i.e., one marked with
[Fact]) has a return value, it is ignored by the test runner.