Many development teams across the world use Selenium to automatically test changes to their web applications.
I often get asked about ways to integrate Selenium tests into a VSTS pipeline.
Here is how.
Before we begin: Understandstand what VSTS is and what it isn’t
VSTS’ build and release system is an orchestration system which allows you to run virtually any build, test, or release tool. VSTS is not a comprehensive automated testing system. There is a “Test” section in VSTS for manual testing. If you automate any tests in the test area there is a way of linking the test cases to an automated test execution. The auotmated tests will – however – always be carried out by a third test runner.
#1 Build and stage your binaries
Usually, UI tests run against an application that has been deployed into a environment. This implies that the test run would happen after the build process.
In the build process below I am building a libarary of Selenium tests. Note how I have removed the unit test task, as I am only building the library that contains the UI tests. If your solution includes unit tests, you can use filtering to only run unit tests. Find out more about VSTest parameters: https://msdn.microsoft.com/en-us/library/jj155796.aspx (Other runners will have different sets of parameters)
Note: If your current build process is creating a web deploy zip file, then you will need to produce a test dll as well as the zip file so that it can be run after deployment.
#2 Deploy your environment
Deploy your environment just as you usually would. This part does not change. You will continue to consume artifacts as normal.
If your Selenium binaries are built separately, be sure to include them as artefacts in your release.
#3 Run your tests
You can use a rest runner (like VSTest) to execute your tests after the release. We used to advise a test agent deployment over WinRM, but this guidance has now changed. Instead we recommend the use of agent phases in combination with the standard VSTest task. Your agent phase will determine which, and how many agents to target and in what manner. (concurrent, sequential)
Note that VSTS hosted agents run in non-interactive mode, which means that you will not be able to install or run any GUI-based browser applications. You can still run Selenium tests but only using headless browsers like PhantomJS, or headless Chrome.
Once again you can use the various options for filtering test cases to only run a subset of tests on a certain agent type. Find out more: https://msdn.microsoft.com/en-us/library/dd286683.aspx
The screenshot shows a test run on a headless agent pool that only runs headless Selenium tests.