Fakes, Stubs, Shims, Moles and how to verify them?

Background

If you have been following my posts here and here before you probably know that when I have to I use Moles from Microsoft Research. So the other day a friend of mine mentioned this new cool thing called Fakes, Stubs and Shims in Visual Studio 11 so I decided to look into that.

The first thing I actually did was to ask the following question on Twitter because I was curious to see if Moles and Fakes were similar:

Ok so Fakes in VS 11 are derived from Moles. My next question to Peter was one I’ve been thinking about alot lately:

So Microsoft are looking into adding verify support to Fakes, great!!! this means that you don’t have to use this hack going forward 🙂 Thanks Peter for that fast feedback!

For all you using Moles this hack will still apply. And for all you SharePoint developers out there; YES this is for you guys because you need to be doing more TDD!

My next step was to create a bootable vhd with Windows 8 Consumer Preview and Visual Studio 11 using my preferred method here and start hacking away.

Step 1

Get up to speed with Fakes, Stubs, Shims by reading “Using shims to isolate calls to non-virtual functions in unit test methods” found here; go on I’ll wait until you’re finished.

You’ll probably end up with a test method looking like the one shown below:

    [TestClass]
    public class Y2KCheckerTests_Step1
    {
        [TestMethod]
        [ExpectedException(typeof(ApplicationException))]
        public void Check_WhenCalledWithTheDate_2000_1_1_ThenApplicationExceptionIsThrowned()
        {
            using(ShimsContext.Create())
            {
                ShimDateTime.NowGet = () => { return new DateTime(2000, 1, 1); };

                Y2KChecker.Check();
            }
        }
    }

Well this is all fine, but what if you have a thousand tests that want to detour DateTime.Now? If you read my previous post here I introduced a Testable object that encapsulates behaviour.

Step 2

So the next step is to encapsulate this into a Testable object that I call Y2KShimDateTime like the code below shows:

[TestClass]
public class Y2KCheckerTests_Step2
{
    [TestMethod]
    [ExpectedException(typeof(ApplicationException))]
    public void Check_WhenCalledWithY2KShimDateTime_ThenApplicationExceptionIsThrowned()
    {
        using (ShimsContext.Create())
        {
            Y2KShimDateTime shim = new Y2KShimDateTime();

            Y2KChecker.Check();
        }
    }
}

public class Y2KShimDateTime
{
    public Y2KShimDateTime()
    {
        ShimDateTime.NowGet = () => { return new DateTime(2000, 1, 1); };
    }
}

Wow now we’re getting somewhere. We can reuse our Y2KShimDateTime in all our test methods. But hey what if we really just want to verify that a specific method has been called? You could just add some counter to the Y2KShimDateTime and call a property like the recommendation here.

Step 3

I use Moq to Mock my code because it’s simple and I love the way I can control return values with setups but mostly I love the way I can verify that a certain method was called with a specific value! If you haven’t looked at Moq yet I strongly encourage you to do so!

To accomplish this with Fakes, Shims or Moles you will need to trick Moq a bit (a hack I know). First of all let’s take a look at my verifiable Y2KShimDateTime class called VY2KShimDateTime.

public class VY2KShimDateTime
{
    public void InitFake()
    {
        ShimDateTime.NowGet = () =>
        {
            return VNowGet();
        };
    }

    public virtual DateTime VNowGet()
    {
        return new DateTime();
    }
}

The first part of the trick is to move the initialization of the detour from the constructor to a public method. This enables us to initialize the detours from Moq as you will see soon. The next thing is to create a public virtual method (VNowGet in this case) that will be called by Moq. There you have it! Easy enough?

Lets see how we use the VY2KShimDateTime from Moq.

using (ShimsContext.Create())
{
    Mock mock = new Mock();
    mock.Object.InitFake();
    mock.Setup(v => v.VNowGet()).Returns(new DateTime(2000, 1, 1));

    Y2KChecker.Check();
}

As you can see from the highlighted row above we’re initializing our detours after we create our mock this way we can be sure our detours will work. And now we can use the usual setup from Moq to Control what our detour will return. Brilliant don’t you say?

Let’s throw a verify into this equation and look how that looks.

using (ShimsContext.Create())
{
    Mock mock = new Mock();
    mock.Object.InitFake();
    mock.Setup(v => v.VNowGet()).Returns(new DateTime(1, 1, 1));

    Y2KChecker.Check();

    //Uncomment line below to see verify that mock.Verify throws correct message
    //Y2KChecker.Check();

    mock.Verify(v => v.VNowGet(), Times.Once());
}

In this case we’ll have to supply a date other than Y2K because otherwise we’ll get an exception but as you can see in the verify method we verify that DateTime.Now is called only once. If I would to uncomment the second Y2KChecker.Clean(); call the verify on our mock would throw an exception.

Summary

I’ve showed how to create verifiable Fakes, Stubs and Shims (works for Moles as well) using a small hack. I use this approach daily when I work with Moles because I really like the combination of Moq verify and setup and Fakes/Moles isolation and detouring.

Hopefully Microsoft will incorporate this in a near future in Fakes but until then give this a try.

All code can be downloaded from GitHub here.

Cheers,

Hugo

Build recipe for running unit tests using file masks in TFS Build 2008

Background

The background for this build script recipe can be found here and here. Instructions on how to install and configure a build server to run these build scripts can be found here and here.

The Challenge

The build recipe has to conform to these constraints/requirements:

  • No test lists
  • No maintenance of testsettings/testrunconfig files
  • Using file mask .Tests to identify which assemblies that contain unit tests
  • TFS 2008
  • Visual Studio 2010
  • BizTalk 2010
  • MSTest

The Dev.CI.testsettings file

As you might already know the testsettings file (formely know as testrunconfig) defines important configuration that tells MSTest various information; where to look for deployment items for instance.

I create one testsettings file per build that I’ve safely in source control and in that file I add this:

<Deployment>
  <DeploymentItem filename=\\bldserv\Bld\1\TestResults\DeplymentItems\/>
</Deployment>

These lines tells MSTest to look for deployment items in the specified directory/share. This works for now as we only have one build server that executes the CI-builds but when we add build servers later on we’ll have to come up with a better solution using environment variables for instance.

It would be great if TFS Build could automatically pick up the testsettings file for each solution in the build but as we only do ONE testrun per TFS Build that wouldn’t work either. The best would be if we ran MSTest / solution in the TFS Build.

The TFSBuild.proj file

Under the ItemGroup for testing I add the following line to tell MSTest to look for testcontainers with the “.Tests.dll” filemask.

<TestContainer Include="$(OutDir)\%2a.Tests.dll" />

The next step is to introduce a TFS Build override where we specify our testsettings file and do the copying to our deploymentitems folder specified by our testsettings file. There is also delete statement that I was added later because I found an assembly with the correct file mask but that wasn’t a unit test assembly, read more about that adventure here.

  <Target Name="BeforeTestConfiguration">

    <!–We need to set the path to our xml-files and schema-

    files for MSTest to include this can only be done from one

    single testsettings-file.–>

    <PropertyGroup>

      <RunConfigFile>$(SolutionRoot)Dev.CI.testsettings</RunConfigFile>

    </PropertyGroup>

   

    <!–Copy all testinstance-files.–>

    <CreateItem Include="$(SolutionRoot)\**\TestInstances\*">

      <Output TaskParameter ="Include" ItemName ="MyTestFiles"/>

    </CreateItem>

    <Copy SourceFiles="@(MyTestFiles)" 

          DestinationFolder="$(TestResultsRoot)\DeplymentItems"/>

    <!–Copy all schema-files.–>

    <CreateItem Include="$(SolutionRoot)\**\*.xsd">

      <Output TaskParameter ="Include" ItemName ="MySchemaFiles"/>

    </CreateItem>

    <Copy SourceFiles="@(MySchemaFiles)" 

          DestinationFolder="$(TestResultsRoot)\DeplymentItems"/>

    <!–Delete all non unittest-files.–>

    <CreateItem 

      Include="$(BinariesRoot)\**\[Customer].[BtsApp].OrchestrationsHelper.Tests.dll">

      <Output TaskParameter ="Include" ItemName ="MyNonUnitTestFiles"/>

    </CreateItem>

    <Delete Files="@(MyNonUnitTestFiles)"/>

  </Target>

The drawback to this approach is that your testinstance/schema files (often xml-files in BizTalk projects) need to be uniquely named or you would overwrite them in this step.

Lessons learned

  • You should start to think about automated build processes from day one in a dev project.
  • We need to find a way to not use hardcoded paths, maybe we could use environment variables.
  • Maybe we should look into other testing frameworks like XUnit, NUnit and see if they offer better support for this kind of situation

Please let me know what you think of this solution considering the constraints and requirements at hand. The examples for this post can be found here.

Cheers!

Hugo

The perils of using “.Tests” as a file mask for unit tests in TFS Build

Background

As some of you might already have read here I’m currently working with a large BizTalk implementation that has been going on for more or less a year.

So there’s already tons of unit tests in place and one of the first things I’d hade to do is setup some CI-builds to verify that each check-in keeps the integrity of the source control tree intact and run unit tests.

The Challenge

The hardest challenge when I start coaching a team with devs with no automated builds this late in the game is:

  • Talking about the value that automation brings to their day to day development work
  • Introducing automation without changing the already established routines, i.e. keeping balance between the new and established routines

Here is a team that already established a routine for creating unit test projects for instance where they name all their unit test projects [Customer].[BTSSystem].[BTSArtifact].Tests. Not going to change this routine right? I’m going to use it in my CI-builds instead right?

I create a CI-build (read more here) that looks for the file mask .Tests (we’re using TFS 2008) and off we go!

The Error

Some days later I check back in (I’m not at the customer every day, otherwise I would constantly monitor the builds)  to follow up how the builds are going and I found one very odd build that was constantly “partially succeeded”. Learning from my earlier mistakes a read my blog post here about “partially succeeded” builds and found nothing that could help me identify the error. Our build log looked like this:

Using “TestToolsTask” task from assembly “E:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll”.
Task “TestToolsTask”
Command:
E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo
/runconfig:”E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings”
/searchpathroot:”E:\Blds\73\Binaries\Debug”
/resultsfileroot:”E:\Blds\73\TestResults”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Maps.Tests.dll”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Orchestration.Tests.dll”
/publish:”http://sometfs:8080/”
/publishbuild:”vstfs:///Build/Build/6829″
/teamproject:”[Customer].
/platform:”Any CPU”
/flavor:”Debug”
The “TestToolsTask” task is using “MSTest.exe” from “E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe”.
Loading E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings…
Loading E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Maps.Tests.dll…
Loading E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Orchestration.Tests.dll…
Starting execution…
TESTTOOLSTASK : warning : The disabled test ‘SomeTest’ was removed from the test run. [E:\Blds\73\BuildType\TFSBuild.proj]

Results               Top Level Tests
——-               —————
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest1
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest2
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest3
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest4
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest5
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest6
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest7
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest8
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest9
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest10
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest11
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest12
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest13
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest14
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest15
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest16
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest17
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest18
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest19
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest20
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest21
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest22
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest23
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest24
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest25
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest26
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest27
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest28
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest29
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest30
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest1
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest2
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest3
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest4
34/34 test(s) Passed

Summary
——-
Test Run Warning.
Passed  34
———-
Total   34
Results file:  E:\Blds\73\TestResults\SOME-TFS 2011-04-01 15_29_30_Any CPU_Debug.trx
Test Settings: Local

Run has the following issue(s):
Waiting to publish…
Publishing results of test run SOME-TFS 2011-04-01 15:29:30_Any CPU_Debug to http://sometfs:8080/…
..Publish completed successfully.
Command:
C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe /nologo
/runconfig:”E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings”
/searchpathroot:”E:\Blds\73\Binaries\Debug”
/resultsfileroot:”E:\Blds\73\TestResults”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].OrchestrationsHelper.Tests.dll”
/publish:”http://sometfs:8080/”
/publishbuild:”vstfs:///Build/Build/6829″
/teamproject:”[Customer].
/platform:”Any CPU”
/flavor:”Debug”
The “TestToolsTask” task is using “MSTest.exe” from “C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe”.

Microsoft (R) Test Execution Command Line Tool Version 9.0.21022.8
Copyright (c) Microsoft Corporation. All rights reserved.

The type initializer for ‘Microsoft.VisualStudio.TestTools.Utility.LicenseHelper’ threw an exception.
For switch syntax, type “MSTest /help”
MSBUILD : warning MSB6006: “MSTest.exe” exited with code 1. [E:\Blds\73\BuildType\TFSBuild.proj]
The previous error was converted to a warning because the task was called with ContinueOnError=true.
Build continuing because “ContinueOnError” on the task “TestToolsTask” is set to “true”.
Done executing task “TestToolsTask” — FAILED.

As you see all the tests are passing but after publishing the build calls the Visual Studio 2008 version of MSTest.exe. And this got me puzzled. But the error message The type initializer for ‘Microsoft.VisualStudio.TestTools.Utility.LicenseHelper’ threw an exception. got me really thinking…someone even suggested that I was messing with MS licensing, no WAY I would ever do that!

The Solution

The solution was really simple, someone in the devteam had created an ordinary BizTalk orchestration project named [Customer].[BtsApp].OrchestrationsHelper.Tests but because of that it WAS NOT a unit test project so that’s why the build returned that error.

The best solution would have been to rename the project but after asking the developers that renaming would have costs us a lot of time and refactoring so we decided to exclude that particular assembly from our tests. (details here)

Lessons learned

  • You should start to think about automated build processes from day one in a dev project.
  • Partially succeeded builds can also be a symptom of trying to execute “regular” assemblies through MSTest.

Enjoy!

Hugo