Dashboard > MbUnit > ... > MbUnit Home > CombinatorialTestAttribute
  MbUnit Log In View a printable version of the current page.  
  CombinatorialTestAttribute
Added by Jonathan de Halleux, last edited by Graham Hay on Jun 07, 2007  (view change)
Labels: 
(None)

The TestFixture supports combinatorial tests through the use of the CombinatorialTestAttribute.

A simple example

Suppose that we have a ICountX interface that defines a method that counts the number of 'x' in a string:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
public interface ICountX
{
   int Count(string s);
}

You have two classes that implement that interface:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
class SickCountX : ICountX
{
   public int Count(string s)
   {
       return 2;
    }
}

class CountX : ICountX
{
   public int Count(string s)
   {
        int count = 0;
        foreach(char c in s)
            if (c=='x')
               count++;
       return count;
   }
}

Now you would like to test that those two interface implementations work correctly. Let's start by writing a test method inside a TestFixture. The CountIsExact method receives a string, the number of x in the string and checks that a provided ICountX implementation computes the expected x count:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
[TestFixture]
public class ICountXTest
{
   public class StringX
   {
       public StringX(string value, int xcount){Value = value; XCount = xcount;}
       public string Value;
       public int XCount;
       public override string ToString() { return String.Format("{0},{1}",this.Value,this.XCount);}
   }

   [CombinatorialTest]
   public void CountIsExact(
       ICountX counter,
       StringX s
       )
   {
       Assert.AreEqual(s.XCount, counter.Count(s.Value));
    }
}

Next, we create a few strings and xcount that can serve for creating test case. By simplicity we use the new C# 2.0 iterators to implement that:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
[Factory(typeof(StringX))]
public IEnumerable Strings()
{
    yield return new StringX("",0);
    yield return new StringX("x",1);
    yield return new StringX("xa",1);
    yield return new StringX("xax",2);
    yield return new StringX("aaa",0);
}

Note that we have tagged the method with [FactoryAttribute] so that MbUnit knows this method creates StringX instances. We also create another method that will create instance of ICountX to test:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
[Factory(typeof(ICountX))]
public IEnumerable Counters()
{
   yield return new SickCountX();
   yield return new CountX();
}

Now, we would like to cross product the output of Counters and StringX and feed the combination in the CountIsExact test method. To do so, we use the UsingFactories attribute to tag the parameters of the method:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
[CombinatorialTest]
public void CountIsExact(
   [UsingFactories("Counters")] ICountX counter,
   [UsingFactories("Strings")] StringX s)
{...}

The fixture is ready, so we can give it a run. The output of the test execution is as follows:

[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(,0))
 [mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(x,1))
 [mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(xa,1))
 [mbunit][Success] CountIsExact(Counters(SandBox.SickCountX),Strings(xax,2))
 [mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(aaa,0))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(,0))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(x,1))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xa,1))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xax,2))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(aaa,0))6 succeeded, 4 failed, 0 skipped, took 0,00 seconds.

It worked! As expected, we have 5 x 2 = 10 test case generated. Each test case name is generated out of the data source and different data values. Of course, you can "bind" data to the parameters using various other ways and the tests support more that 2 parameters as we will see in the following. We could also have added multiple Using Attributes.

Prerequisites

This new features uses the [TestFuOperationsframework] in the background.

Rationale

The test has the following workflow:

for each parameter of the test method,
    get all the domains D of the parameter from the [UsingAttributes] (there are a bunch of those).

This creates a collection of domain for each parameter which is also a domain itself, which we'll call parameter domain: PD = collection of D,

foreach tuple in the [CartesianProduct] of the parameter domains
    foreach ptuple in the [PairWizeProduct] of the domain in the tuple (each entry is a domain for the corresponding method parameters)
        Create a test case that will invoke the test method with the values in ptuple.

Note that in the example above,

PD = {
  { Counters }
  ,
  { Strings }
}

hence the Cartesian product has only one element.

{Counters}
 x
{Strings}
 = (Counters,Strings)

Built in Using Attributes

MbUnit comes with a number of built-in using attribute that should get you started almost all situations.

  • [UsingEnumAttribute]
  • [UsingLinearAttribute]
  • [UsingLiteralsAttribute]
  • [UsingFactoriesAttribute]
  • [UsingImplementationsAttribute]
  • [ExtendingUsingAttributes], how to write your own [UsingAttribute]

What about tuple filtering ?

In some situation you may want to filter out tuple. For example, we want to check that a methods throws ArgumentNullException on null arguments:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
[Factory]
public string[] Strings()
{
    return new string[] { null, "hello" };
}

[CombinatorialTest]
[ExpectedArgumentNullException]
public void TestWithValidator(
    [UsingFactories("Strings")] string s1,
    [UsingFactories("Strings")] string s2
)
{
    if (s1 == null)
        throw new ArgumentNullException();
    if (s2 == null)
        throw new ArgumentNullException();
}
note

The ExpectedExceptionAttribute and all other decorators work on the combinatorial tests, however if you mark a test with the IgnoreAttribute or ExplicitAttribute it's factories will not be run.

If we run the test now, we will have unwanted tuples such as (null, null) and ("hello", "hello") included in our tests: the first does not give much information and the second will not throw and hence, is erroneous. In this kind of situation, we can specify a "filter" method that takes the same arguments as the test method and return a bool:

Unable to find source-code formatter for language: cs. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
public bool IsValid(string s1, string s2)
{
   if (s1 == null && s2 == null)
      return false;
   if (s1 != null && s2 != null)
      return false;
   return true;
}

[CombinatorialTest(TupleValidatorMethod = "IsValid")]
[ExpectedArgumentNullException]
public void TestWithValidator(...
--- output
 [mbunit][Success] TestWithValidator(Strings(),Strings(hello))
 [mbunit][Success] TestWithValidator(Strings(hello),Strings())

As required, the unwanted tuples have been dropped from the test.

Site powered by a free Open Source Project / Non-profit License (more) of Confluence - the Enterprise wiki.
Learn more or evaluate Confluence for your organisation.
Powered by Atlassian Confluence, the Enterprise Wiki. (Version: 2.2.9 Build:#527 Sep 07, 2006) - Bug/feature request - Contact Administrators