The perils of using “.Tests” as a file mask for unit tests in TFS Build

Background

As some of you might already have read here I’m currently working with a large BizTalk implementation that has been going on for more or less a year.

So there’s already tons of unit tests in place and one of the first things I’d hade to do is setup some CI-builds to verify that each check-in keeps the integrity of the source control tree intact and run unit tests.

The Challenge

The hardest challenge when I start coaching a team with devs with no automated builds this late in the game is:

  • Talking about the value that automation brings to their day to day development work
  • Introducing automation without changing the already established routines, i.e. keeping balance between the new and established routines

Here is a team that already established a routine for creating unit test projects for instance where they name all their unit test projects [Customer].[BTSSystem].[BTSArtifact].Tests. Not going to change this routine right? I’m going to use it in my CI-builds instead right?

I create a CI-build (read more here) that looks for the file mask .Tests (we’re using TFS 2008) and off we go!

The Error

Some days later I check back in (I’m not at the customer every day, otherwise I would constantly monitor the builds)  to follow up how the builds are going and I found one very odd build that was constantly “partially succeeded”. Learning from my earlier mistakes a read my blog post here about “partially succeeded” builds and found nothing that could help me identify the error. Our build log looked like this:

Using “TestToolsTask” task from assembly “E:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll”.
Task “TestToolsTask”
Command:
E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo
/runconfig:”E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings”
/searchpathroot:”E:\Blds\73\Binaries\Debug”
/resultsfileroot:”E:\Blds\73\TestResults”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Maps.Tests.dll”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Orchestration.Tests.dll”
/publish:”http://sometfs:8080/”
/publishbuild:”vstfs:///Build/Build/6829″
/teamproject:”[Customer].
/platform:”Any CPU”
/flavor:”Debug”
The “TestToolsTask” task is using “MSTest.exe” from “E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe”.
Loading E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings…
Loading E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Maps.Tests.dll…
Loading E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Orchestration.Tests.dll…
Starting execution…
TESTTOOLSTASK : warning : The disabled test ‘SomeTest’ was removed from the test run. [E:\Blds\73\BuildType\TFSBuild.proj]

Results               Top Level Tests
——-               —————
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest1
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest2
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest3
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest4
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest5
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest6
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest7
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest8
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest9
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest10
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest11
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest12
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest13
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest14
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest15
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest16
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest17
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest18
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest19
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest20
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest21
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest22
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest23
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest24
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest25
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest26
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest27
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest28
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest29
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest30
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest1
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest2
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest3
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest4
34/34 test(s) Passed

Summary
——-
Test Run Warning.
Passed  34
———-
Total   34
Results file:  E:\Blds\73\TestResults\SOME-TFS 2011-04-01 15_29_30_Any CPU_Debug.trx
Test Settings: Local

Run has the following issue(s):
Waiting to publish…
Publishing results of test run SOME-TFS 2011-04-01 15:29:30_Any CPU_Debug to http://sometfs:8080/…
..Publish completed successfully.
Command:
C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe /nologo
/runconfig:”E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings”
/searchpathroot:”E:\Blds\73\Binaries\Debug”
/resultsfileroot:”E:\Blds\73\TestResults”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].OrchestrationsHelper.Tests.dll”
/publish:”http://sometfs:8080/”
/publishbuild:”vstfs:///Build/Build/6829″
/teamproject:”[Customer].
/platform:”Any CPU”
/flavor:”Debug”
The “TestToolsTask” task is using “MSTest.exe” from “C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe”.

Microsoft (R) Test Execution Command Line Tool Version 9.0.21022.8
Copyright (c) Microsoft Corporation. All rights reserved.

The type initializer for ‘Microsoft.VisualStudio.TestTools.Utility.LicenseHelper’ threw an exception.
For switch syntax, type “MSTest /help”
MSBUILD : warning MSB6006: “MSTest.exe” exited with code 1. [E:\Blds\73\BuildType\TFSBuild.proj]
The previous error was converted to a warning because the task was called with ContinueOnError=true.
Build continuing because “ContinueOnError” on the task “TestToolsTask” is set to “true”.
Done executing task “TestToolsTask” — FAILED.

As you see all the tests are passing but after publishing the build calls the Visual Studio 2008 version of MSTest.exe. And this got me puzzled. But the error message The type initializer for ‘Microsoft.VisualStudio.TestTools.Utility.LicenseHelper’ threw an exception. got me really thinking…someone even suggested that I was messing with MS licensing, no WAY I would ever do that!

The Solution

The solution was really simple, someone in the devteam had created an ordinary BizTalk orchestration project named [Customer].[BtsApp].OrchestrationsHelper.Tests but because of that it WAS NOT a unit test project so that’s why the build returned that error.

The best solution would have been to rename the project but after asking the developers that renaming would have costs us a lot of time and refactoring so we decided to exclude that particular assembly from our tests. (details here)

Lessons learned

  • You should start to think about automated build processes from day one in a dev project.
  • Partially succeeded builds can also be a symptom of trying to execute “regular” assemblies through MSTest.

Enjoy!

Hugo


Generating BizTalk binding files with Excel automation in a TFS Build

This is yet another of those things I really need to blog about so that I don’t ever forget this personally

Background

So the background is as follows:

  • A large BizTalk 2010 implementation at a large private Swedish Enterprise customer.
  • Visual Studio 2010 and TFS 2008
  • Developers have already been working more or less 1 year before I arrive at the scene.
  • My mission is to automate the whole build/release process.
  • The devs have written an Excel macro that takes values from the Excel WorkSheet and together with a template-file for BizTalk bindings it generates all the different binding-files for the specified environments in the Excel Worksheet. I’ve seen a lot’s of different ways to do this before and this is yet another.
  • The Excel macro is called from a vbs-file.
  • The vbs-file is called from a powershell script, yes I see a lot of improvement here.

The challenge

So I started my journey today with the hopes of just hooking up the existing scripts for generating the BizTalk binding-files with a TFS build. But for some reason no files were generated but I could see that Excel started and so forth.

After I while I decided to leave the Excel macro thingee and rewrite the complete Excel macro in C#, and I did and it worked just fine.

I hooked up my little console app with my build and got the following message:

System.Runtime.InteropServices.COMException (0x800A03EC): Microsoft Office Excel cannot access the file ‘c:\temp\test.xls’. There are several possible reasons:

• The file name or path does not exist.
• The file is being used by another program.
• The workbook you are trying to save has the same name as a currently open workbook.

Finally! some error I could search for.

The solution

I found the solution here and it stipulates that you should add the following folder structure to the server where you’re running Excel automation:

  • Windows 2008 Server x64
      Please create this folder: C:\Windows\SysWOW64\config\systemprofile\Desktop
  • Windows 2008 Server x86
      Please create this folder: C:\Windows\System32\config\systemprofile\Desktop

It almost felt silly after some 4 hours of rewriting an Excel macro to C# to just add a the “Desktop” folder under C:\Windows\SysWOW64\config\systemprofile\ and the build just worked perfectly!

Warning read this official Microsoft kb about support for this here.

Lessons learned

  • VBA code can be very messy
  • You should start to think about automated build processes from day one in a dev project.
  • I never thought adding a folder could solve this

Cheers!

Hugo

Creating a bootable VHD the easy way…about Indexes

I’ve seen a lot of people using and downloading my bat-file from my original post here and some of you have even tried some of my more advanced topics here.

There is something worth mentioning with the attached Install-WindowsImage.ps1 that I’ve received feedback from some of you out there. If you would like to install any other Operating System SKU then the default at Index 1 in my bat-file you’ll have to list the images first.

Let’s take a closer look at Install-WindowsImage.ps1:

image

As you can see someone has already been kind enough to provide us with an easy way to list the available images. So if I go ahead and run the example on my own system I get the following:

image

So imagine if I wanted the DataCenter Edition (Full Installation) instead of Standard (Full Installation) I’d have to change the Index in the provided CreateBootableVHD_v2.bat file:

image

Hope that clear any issues with Indexes you might have.

Enjoy!

Hugo