What do you like best?
Of all services explored, Sauce Labs offers the single best solution for integrating deep automated browser testing in a CI/CD pipeline, and integrates best with Jenkins. The Jenkins UI plugin offers a way to easily configure browser settings for tests being executed continuously. The Sauce Connect Proxy is a necessary technology for companies who have firewalled testing environments, as well as localhost testing. Test reports (that also integrate seamlessly with Jenkins jobs) are also detailed and incredibly useful. Browser support is quick to be updated as new versions of browsers are released. The custom capabilities that can be provided to Sauce Labs VMs during automated testing sessions are a valuable option for experimental or niche testing.
What do you dislike?
The Jenkins Sauce OnDemand plugin could have more options available to tap into more of the custom capabilities Sauce Labs actually supports. Current, in order to tap into these capabilities, a programatic solution is required from test code, rather than being able to configure them via the Jenkins plugin UI. For example, things like desktop resolution, browser automation binary versions, remote Selenium version, and browser console logs. These are all things it would be nice to be able to customize straight from the Jenkins plugin UI. In our experience, we've also had some issues with the Sauce Connect Proxy on our Jenkins servers failing to start, which makes the optimal CI/CD pipeline come to a halt. User account management needs an overhauls, allowing for user groups, rather than just a hierarchy structure. Test job queuing is first-come-first-serve, meaning there is no in-built way to queue batches of jobs from, say, different Jenkins servers/accounts.
Recommendations to others considering the product
If your need is architecting a CI/CD pipeline with left-shifted continuous real browser testing, then there is simply no better service available than Sauce Labs to integrate with this desired model. They are a smaller company than their leading competitor, but they are rapidly innovating in the right direction, and lead the charge in continuous automated testing.
What business problems are you solving with the product? What benefits have you realized?
Continuous, deep, left-shifted testing executed in a CI/CD pipeline as a quality gate for build promotion into higher environments. Access to browsers, platforms, and devices for manual or automated testing we do not have access to physically. A platform for executing real browser tests that we cannot implement or maintain internally in a cost-effective manner.