The perils of using “.Tests” as a file mask for unit tests in TFS Build

Background

As some of you might already have read here I’m currently working with a large BizTalk implementation that has been going on for more or less a year.

So there’s already tons of unit tests in place and one of the first things I’d hade to do is setup some CI-builds to verify that each check-in keeps the integrity of the source control tree intact and run unit tests.

The Challenge

The hardest challenge when I start coaching a team with devs with no automated builds this late in the game is:

  • Talking about the value that automation brings to their day to day development work
  • Introducing automation without changing the already established routines, i.e. keeping balance between the new and established routines

Here is a team that already established a routine for creating unit test projects for instance where they name all their unit test projects [Customer].[BTSSystem].[BTSArtifact].Tests. Not going to change this routine right? I’m going to use it in my CI-builds instead right?

I create a CI-build (read more here) that looks for the file mask .Tests (we’re using TFS 2008) and off we go!

The Error

Some days later I check back in (I’m not at the customer every day, otherwise I would constantly monitor the builds)  to follow up how the builds are going and I found one very odd build that was constantly “partially succeeded”. Learning from my earlier mistakes a read my blog post here about “partially succeeded” builds and found nothing that could help me identify the error. Our build log looked like this:

Using “TestToolsTask” task from assembly “E:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll”.
Task “TestToolsTask”
Command:
E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo
/runconfig:”E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings”
/searchpathroot:”E:\Blds\73\Binaries\Debug”
/resultsfileroot:”E:\Blds\73\TestResults”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Maps.Tests.dll”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Orchestration.Tests.dll”
/publish:”http://sometfs:8080/”
/publishbuild:”vstfs:///Build/Build/6829″
/teamproject:”[Customer].
/platform:”Any CPU”
/flavor:”Debug”
The “TestToolsTask” task is using “MSTest.exe” from “E:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe”.
Loading E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings…
Loading E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Maps.Tests.dll…
Loading E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].Orchestration.Tests.dll…
Starting execution…
TESTTOOLSTASK : warning : The disabled test ‘SomeTest’ was removed from the test run. [E:\Blds\73\BuildType\TFSBuild.proj]

Results               Top Level Tests
——-               —————
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest1
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest2
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest3
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest4
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest5
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest6
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest7
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest8
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest9
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest10
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest11
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest12
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest13
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest14
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest15
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest16
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest17
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest18
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest19
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest20
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest21
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest22
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest23
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest24
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest25
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest26
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest27
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest28
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest29
Passed                [Customer].[BtsApp].Maps.Tests.SomeTest30
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest1
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest2
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest3
Passed                [Customer].[BtsApp].Orchestration.Tests.SomeTest4
34/34 test(s) Passed

Summary
——-
Test Run Warning.
Passed  34
———-
Total   34
Results file:  E:\Blds\73\TestResults\SOME-TFS 2011-04-01 15_29_30_Any CPU_Debug.trx
Test Settings: Local

Run has the following issue(s):
Waiting to publish…
Publishing results of test run SOME-TFS 2011-04-01 15:29:30_Any CPU_Debug to http://sometfs:8080/…
..Publish completed successfully.
Command:
C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe /nologo
/runconfig:”E:\Blds\73\Sources\Dev\[Customer].Dev.CI.[BtsApp].testsettings”
/searchpathroot:”E:\Blds\73\Binaries\Debug”
/resultsfileroot:”E:\Blds\73\TestResults”
/testcontainer:”E:\Blds\73\Binaries\Debug\[Customer].[BtsApp].OrchestrationsHelper.Tests.dll”
/publish:”http://sometfs:8080/”
/publishbuild:”vstfs:///Build/Build/6829″
/teamproject:”[Customer].
/platform:”Any CPU”
/flavor:”Debug”
The “TestToolsTask” task is using “MSTest.exe” from “C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe”.

Microsoft (R) Test Execution Command Line Tool Version 9.0.21022.8
Copyright (c) Microsoft Corporation. All rights reserved.

The type initializer for ‘Microsoft.VisualStudio.TestTools.Utility.LicenseHelper’ threw an exception.
For switch syntax, type “MSTest /help”
MSBUILD : warning MSB6006: “MSTest.exe” exited with code 1. [E:\Blds\73\BuildType\TFSBuild.proj]
The previous error was converted to a warning because the task was called with ContinueOnError=true.
Build continuing because “ContinueOnError” on the task “TestToolsTask” is set to “true”.
Done executing task “TestToolsTask” — FAILED.

As you see all the tests are passing but after publishing the build calls the Visual Studio 2008 version of MSTest.exe. And this got me puzzled. But the error message The type initializer for ‘Microsoft.VisualStudio.TestTools.Utility.LicenseHelper’ threw an exception. got me really thinking…someone even suggested that I was messing with MS licensing, no WAY I would ever do that!

The Solution

The solution was really simple, someone in the devteam had created an ordinary BizTalk orchestration project named [Customer].[BtsApp].OrchestrationsHelper.Tests but because of that it WAS NOT a unit test project so that’s why the build returned that error.

The best solution would have been to rename the project but after asking the developers that renaming would have costs us a lot of time and refactoring so we decided to exclude that particular assembly from our tests. (details here)

Lessons learned

  • You should start to think about automated build processes from day one in a dev project.
  • Partially succeeded builds can also be a symptom of trying to execute “regular” assemblies through MSTest.

Enjoy!

Hugo


Generating BizTalk binding files with Excel automation in a TFS Build

This is yet another of those things I really need to blog about so that I don’t ever forget this personally

Background

So the background is as follows:

  • A large BizTalk 2010 implementation at a large private Swedish Enterprise customer.
  • Visual Studio 2010 and TFS 2008
  • Developers have already been working more or less 1 year before I arrive at the scene.
  • My mission is to automate the whole build/release process.
  • The devs have written an Excel macro that takes values from the Excel WorkSheet and together with a template-file for BizTalk bindings it generates all the different binding-files for the specified environments in the Excel Worksheet. I’ve seen a lot’s of different ways to do this before and this is yet another.
  • The Excel macro is called from a vbs-file.
  • The vbs-file is called from a powershell script, yes I see a lot of improvement here.

The challenge

So I started my journey today with the hopes of just hooking up the existing scripts for generating the BizTalk binding-files with a TFS build. But for some reason no files were generated but I could see that Excel started and so forth.

After I while I decided to leave the Excel macro thingee and rewrite the complete Excel macro in C#, and I did and it worked just fine.

I hooked up my little console app with my build and got the following message:

System.Runtime.InteropServices.COMException (0x800A03EC): Microsoft Office Excel cannot access the file ‘c:\temp\test.xls’. There are several possible reasons:

• The file name or path does not exist.
• The file is being used by another program.
• The workbook you are trying to save has the same name as a currently open workbook.

Finally! some error I could search for.

The solution

I found the solution here and it stipulates that you should add the following folder structure to the server where you’re running Excel automation:

  • Windows 2008 Server x64
      Please create this folder: C:\Windows\SysWOW64\config\systemprofile\Desktop
  • Windows 2008 Server x86
      Please create this folder: C:\Windows\System32\config\systemprofile\Desktop

It almost felt silly after some 4 hours of rewriting an Excel macro to C# to just add a the “Desktop” folder under C:\Windows\SysWOW64\config\systemprofile\ and the build just worked perfectly!

Warning read this official Microsoft kb about support for this here.

Lessons learned

  • VBA code can be very messy
  • You should start to think about automated build processes from day one in a dev project.
  • I never thought adding a folder could solve this

Cheers!

Hugo

Yet another post about Team Build and the 260+ char limit

So this is definitely not a new topic and there has been lots of post about it. But I’d like to make sure that you don’t find yourself in the situation where you receive the following error message:

”The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.”

As many already have stated this is not a limit in Team Build but a Windows limit, you can read more about that here.

Even though many devs out there are brutally aware of this hard limit I tend to see the same mistakes appear at customers over and over again.

Lets start out with some basics:

  • Don’t create source control folder hierarchies that mimic your namespace naming or any other convention. That leads to deep hierarchies that will get you closer to the 260+ limit.
    image
  • Instead try to keep your source control hierarchy as flat as possible, use the solution folders feature in Visual Studio to represent your hierarchy instead.

    image
    See how the Source and Tests hierarchy are used as Solution Folders instead in Visual Studio without creating any extra folder level in source control:
    image

  • Keep your workspace folder mappings short and consistent like “C:\ws\”
  • Keep your namespaces short and sweet and remember that you don’t need to name your solution/project folders the same as your namespace (although I know it simplifies the daily work a lot Blinkar)

And here are some advanced strategies:

  • Use $(BuildDefinitionId) instead of $(BuildDefinitionPath) when you configure your build agents.
    image
    The picture below shows the output in folder structures between these two approaches using the same build definition but different agent settings. The pictures are taken from a TFS 2010 installation for TFS 2008 there is no $(BuildAgentId) folder.
    image
  • In TFS 2008 shorten the “SourcesSubdirectory”, “BinariesSubdirectory” and “TestResultsSubdirectory” configuration in the TfsBuildService.exe.config.
  • In TFS 2010 shorten the assignment in the string for the “Initialize Sources Directory”, “Initialize Binaries Directory” and “Initialize TestResults Directory” in your DefaultTemplate.xaml as the picture shows:
    image
  • In TFS 2010 if you’re using the UpgradeTemplate.xaml you can substitute the name for the directories in the Advanced section like the picture shows:
    image
  • Create a custom Check-In policy that prevents users from checking in structures in source control that are longer then some number set by your administrator. Read more about it here.

This should prevent you from hitting the 260+ hard limit for a while but It’s no silver bullet unfortunately so I hope that a more complete solution will rise soon.

Enjoy,

Hugo