Up until recently we (Microsoft) recommended that you instrument functional test runs from VSTS like so:

We now recommend you do this instead:

Will my existing test runs continue to work?

Yes they will, but you should consider moving to the v2 VSTest task in the future as you will be missing out on new features.

Why has this change been made?

There are two main reasons.

  1. The old test agent deployment happened over WinRM, which is difficult to set up securely over the internet. (especially if the deploying agent is in VSTS and your test agent is in your infrastructure)
  2. The VSTS agent only requires an outbound connection to VSTS and can run tests like before, as long as the necessary tools are installed.

What are the benefits of the change?

  • You can now use one agent for everything
  • You can now use agent pools to manage persistent test agents
    • You can also use deployment groups if you like (there is no difference when it comes to the approach)
  • The VSTest task follows the rules already set in the agent phase settings, causing less confusion about the scale out settings for test agents

Are there any drawbacks?

If you are running on a version of TFS that does not have agent phases in build definitions, then you won’t be able to run functional tests on a different set of agents during your build. In this particular case it is recommended to continue using the previous set of tasks.

How can I deploy test agent machines on the fly now?

  • Continue to deploy machines as required
    • If you deploy your machines in Azure, you can use the VSTS agent extension to deploy the latest VSTS agent on the fly
      • Let it join an agent pool of test agents
    • If you are deploying machines outside of Azure, use automation to pull and install the latest VSTS agent.
      • Let it join an agent pool of test agents
    • The VSTS agent is an orchestrator, which means that you need to make sure all required testing tools are on the machine (use an image or desired state configuration)
  • Add an agent phase after your deployment
    • Select the pool of test agents
    • Select your preferred scale out options (concurrency, etc.)
  • Add the VSTest task to the agent phase and select the required binaries and apply any test filtering as required
  • (optional) Run a REST call to clean out the pool
  • (optional) Tear the machines down as before