How to configure an automated test run with TFS 2010 and Test Manager

Some of you may have read my earlier post on the associated automation in TFS and Test Manager here. I got some comments that made me realize that more people could benefit from a complete step by step guide on this subject. As such this post will be very long but with a lot of pictures so I hope you stick with me.

Environment

First of all you’ll need a TFS environment, I downloaded my environment from Brian Keller here and all the screenshots you’ll see will be from Brian’s image.

Test Controller

You’ll need to start a Test Controller and register team project collection controller like so:

Test Controller

Configuring an automated Test Environment

To be able to run automated test runs we need to configure an automated test environment first. Startup your Test Manager and change to the Lab Center by clicking on the dropdown:

Testing Center - Lab Center

In the Lab Center click on the Controllers tab marked in red:
Controllers Tab

Make sure that your Test Controller shows up and has the status Ready:
Controllers Tab 2

Next step is to return to the Lab Tab marked in red and configure an Environment:

Lab Tab

On the New dropdown choose “New physical environment” like such:

New physical environment

Give your Environment a good name following your specific naming conventions and make sure that the Controller is correctly assigned. Click the button “Next” when you’re ready.

New physical environment - Name and location

On the “Machines” page click on your controller and then click on the button “Add to environment” marked in red.

New physical environment - Add to environment

As you do that your machine will appear on the left side of the screen. To continue you need to give the machine a specific role to make it easier for people to understand the role of the machine. I gave the machine the “Server” role like the picture below. When you’re ready should be able to click the next button.

New physical environment - Edit Role

On the “Machine Properties” page you could fill in more details regarding your machine but I simply skipped this step for now and left it clean.

New physical environment - Machine Properties

On the summary page I just clicked the Finished button and got on with next step.

New physical environment - Summary

When you’re finished make sure you see a green ready symbol like so:

Environments - Ready

Now we need to configure our automated test settings by clicking on the “Test Settings” tab marked in red below:

Test Settings Tab

Click the “New” button to create the automated test settings:

Test Settings Manager - New

Give the test settings a name, description and make sure that you choose automated in the selection section. Click the “Next” button when you’re finished.

image

On the Roles page make sure you click the role you choose earlier when you created your environment, for me that would be the “Server” role, otherwise you will not be able to click the “Next” button. Automated Test Run - Roles

If you have several roles you’ll need to chose which one you’re going to use for your automated tests, In my case I didn’t need to change this as I have only one role.

Automated Test Run - Select Roles

Click the “Next” button when you’re done.

On the “Data and Diagnostics” page configure anything you need and click the “Next” button, In my case I left all the defaults and clicked the “Next” button.

On the “Advanced” page you can configure even more advanced configurations, in my case I just clicked the “Next” button until the Summary page appeared.

image

So click the “Finish” button and lets go on with this adventure. Your result should be another entry in your Test Settings list:

Test Settings Manager - Automated

Turning the manual test into automated in Testing Center

Lets go back to the Testing Center by clicking the dropdown like so:

Switching back to Testing Center

Make sure you’re on the “Plan” tab and click the “Properties” link marked in red below. Here I’m assuming that you already have created a new suite, if you haven’t you should read this post.

Properties

On the somewhat huge properties page (you can easily get overwhelmed with all that information but I’ll lead the way) you should make sure that you choose your test settings in the Automated runs section, in my case I chose “Automated Test Run” created earlier.

Test settings - Automated Test Run

Next you choose your Test environment, in my case I chose “Automated Test Environment” created earlier.

Test environment - Automated Test Environment

Next you’ll have to assign a build for the automated tests. Assuming that you already created a build in this step otherwise you’ll need to create one. The section you’re looking for should look like so:

Builds

That’s it now click the Save and Close button and lets test this baby…

Change to the “Test” tab and start your automated test cases with the “Run” button:

Run Tests

If you’ve done your job right you should see the following very nice summary page:

Test Run Summary

Hope this will help someone out there with this rather straightforward but long blog post.

Cheers!

Hugo

How come the add button in Associated Automation isn’t visible in Microsoft Test Manager?

This year is off to a great start with fantastic first week and an even better last day of that week working with my fellow Avega Coach, Marcus Hammarberg .

Our work yesterday is clearly summarized by Marcus in this post and I think Marcus has made some real groundbreaking thoughts on SpecFlow together with TFS and Visual Studio and I’m very glad to have helped out some parts. A shout out goes to Håkan Forss as he gave both of us some valuable feedback too.

So what’s this post about you might ask? Well during our work we stumbled onto a gotcha with the way Test Manager and associated automation works.

As you can see in the picture below I’ve created a Test Case that some tester later gets to run.

1

Suppose the tester runs this Test Case and finds a bug. Now we want to assign this bug to a developer who fixes the bug and associates a test with the test case in such way that the test case gets automated.

So we change the automation status to Planned as you can see in the picture below:

2

Problem

As you can see the tester can’t set Automation Status in Test Manager to Automated because that state is missing from the list and if we zoom to the right side of the tab control we find that there’s no way of associating any tests either.
2.5

Solution

Being that this was the first time I looked a this particular feature of Test Manager I must say that I got a little intrigued to say the least. After some Internet searching we still came up with nothing but when we opened up the same Test Case in Visual Studio we finally saw the add button (marked with … in the picture below):

2.51

Digging deeper

So it got very clear to me then that this behavior was by design but I still felt that I needed to go behind the scenes to see how this is solved because in the end Test Case is just your average Work Item Type which is defined in XML.

Looking at the Automation Status field in the Test Case WIT XML you’ll see this:

3

As you can see in the XML you’ll never see the automated state if you haven’t assigned a Microsoft.VSTS.TCM.AutomatedTestId value which is exactly what you do when you associate a test with the add button like the picture below:

7

Looking at the Associated Automation tab in the Test Case WIT you’ll see this:

4

As you can see the Associated Automation tab implemented with a control called AssociatedAutomationControl. After some investigation with my favorite reflection utility .Net Reflector I found this section marked in red:

10

It looks like depending on which ServiceProvider that is used when creating the AssociatedAutomationControl the Add button will be visible or not, TADA!

To sum things up:

  1. You can’t associate automation with unit tests within Test Manager
  2. You can associate automation with unit tests within Visual Studio
  3. All this is because the ServiceProvider associated with the current AssociatedAutomationControl controls the visibility of the Add button in the Associate Automation tab.

Cheers!

Hugo