Massive Reduction of Selenium Costs with Sauce Labs
At Okta we provide enterprise grade identity management from the cloud. Specifically, Okta delivers single sign-on across all your web applications, centralize user management and control, and integration with Active Directory. We invest heavily in test automation and validation. A lot of great testing tools are utilized, including Selenium. We currently have about 1000 Selenium tests in our production test system. These tests are deployed and run in the cloud with our trusted partner; Sauce Labs. Currently, Selenium tests run continuously on check-in across master and release branches against our supported browser/ version/ OS matrix. This matrix currently includes Firefox 3.6 on Linux, Firefox 7 (latest stable version) on Windows, Internet Explorer 8 on Windows, Internet Explorer 9 on Windows, Google Chrome on Windows, Safari 4 on Windows and Safari 5 on OSX. (Safari 5 on OSX is run locally until the up and coming Mac VM feature is released by Sauce Labs.) The decision to use Sauce Labs as the Selenium client rather than running tests internally on a roll your own Selenium Grid configuration was methodically considered by our engineering team.
Even before we began to consider working with Sauce Labs, internally a general question would come up every now and again: why bother with Selenium? The answer was simple: we have no choice. Because of our business focus, we must follow industry best practices with test automation across all levels of the stack including the GUI in order to ensure excellent product software quality. Yet, there is no escaping that web application test automation with Selenium is expensive. Engineers at our company report spending anywhere from 10 - 60% of their time writing, reviewing and maintaining Selenium tests. But the cost of not implementing Selenium testing is beyond high---it’s a game-ender. Without constant deep and broad test feedback developers are unable to introduce changes safely and responsibly at the constantly accelerating pace achieved by the world's best software companies---our competitors and your competitors. But we've managed to reduce these costs significantly with Sauce Labs. What we've found through our trial investigation and now full deployment is that Sauce Labs saves us tremendous time, and thus tremendous money.
Test Maintenance Cost
By far, the most enormous cost and savings opportunity is the time it takes for our development team to resolve test failures and maintain our Selenium test base. With Sauce Labs each test job page is directly linked from our test results and output logs and contains a full video, the entire set of selenium commands executed with corresponding screen-shots and time to complete each command, among other things. Debugging and resolving test failures previously took minutes to hours. It can now be done in seconds. Multiply that by the cost of employing a team of senior software engineers and we can easily see the savings in cold, hard cash each month.
Opportunity Cost
The faster development can proceed while remaining above a safe line of quality, the more features we can produce, the more customers we can please, the more money we can make. The less of our precious time we spend on testing, the more we can spend applying our core competency, creating value for our customers. All of the time savings gained intensify the focus on our product opportunity, and the ability to attack it with blazing fast speed and excellent code quality.
Feedback Delay Cost
The massively parallel test execution that Sauce Labs offers allows up to decrease the time for test results from hours to minutes. This allows us to get feedback quickly on recent changes. Like an organism trying to survive in a hostile environment, the faster we can get feedback about our actions, the faster we can modify our behavior to perform optimally in our environment, the better chance of our survival. In all areas of our business and culture, we try to approach the "constant feedback loop ideal".
Expert Cost
There are critical savings in Sauce Labs support and advice during new test development and new browser/os platform deployment. It would be impossible to hire the expertise Sauce Labs has built within their support team. Without Sauce Labs how do we answer the endless questions the come up? Is there a memory leak in our new selenium server causing long running test non-determinism? What are the quirks and workarounds for IE9? What are the most performant element selection techniques? When and how should we upgrade from Selenium Version 1 to Selenium Version 2? Our developers would spend hours to days sorting through forums, building open source libraries, trying to reproduce or work around problems. Now this process takes minutes - the time it takes for the experts who live and breath Selenium every day, all year long, to respond to our help email with the most up-to-date solutions. This is a huge time savings for our developers, and thus a huge cost savings. Architecturally they provide a stream of industry mentoring and guidance in best practices and cutting edge techniques. We could try to replicate what they have built both with staffing and tool sets. Would we? No - because it's not part of our core competency and the effort involved would never be prioritized over product features, as our board correctly dictates.
Administrative Cost
Aside from the savings on hardware and IT administration gained by handing over VM management and creation to Sauce Labs, which is in itself enough to even our balance sheets, there are much more important time savings which make Sauce Labs incredibly valuable for any company seriously invested in front-end test automation.
Related Links: