 |
When you are a startup you want to move as fast as possible. That is the top priority.
At the same time, you can’t have a product that frequently breaks. Otherwise, you won’t be able to retain customers and you won’t even be able to understand if you have product-market fit.
So you absolutely have to test. But testing takes ultra-precious time and inhibits your ability to move fast.
The Problem
If you test manually (yourself/developers/hire people), you’ll struggle to scale your test coverage AND your business will suffer. The speed of manually re-testing every change is just too time-prohibitive.
You tested something and you got a bug? Too bad! Now you have to test it again after the bug gets fixed. Manually testing everything drains a lot of resources from your dev team or forces you to hire testing resources when you may prefer to invest in adding more software engineers.
But what about test automation? Why don’t we write automated tests???
Test Automation Is Broken
The trick here is that there are multiple types of tests. Sure you’d want to write unit tests since it is a great way to speed up your development. Developers can test certain functions without having to run the whole sha-bang. Integration tests are much worse than that but still tolerable in terms of effort.
But don’t you want to make sure that the whole thing works? Create end-to-end tests? Somebody/something needs to check that your checkout flow checks out, right?
Well, good luck with that. It takes on average a couple of days to write a complex automated regression test. And just setting up a Selenium testing environment will take you a couple of weeks!
And when was the last time your developers wrote Selenium tests? Never? Well, their unit-testing skills are not going to help much since it is a different kind of beast and requires a different set of skills.
OK, well you decided to push through by forcing your devs to write automated tests and you made it. Congratulations, but now your developers hate you for making them write those nasty & boring end-to-end tests. Win? Nope. Not only does it take about 30% of their time to write those tests, but now too much of their time is spent on supporting those tests! Yep, this is what they do now. Fix bugs in their own tests and change tests when code is changing. Full freaking time!
Your devs are now stuck fixing blockers instead of executing on your P1 roadmap stories.
But wait, that’s not all. Your perfectly polished beautiful life-saving CI is now flaky and doesn’t work most of the time.
Wait what??? You can’t execute your favorite end-to-end tests in CI? Too flaky? Oops.
Moreover, you can’t even make your automated tests part of your CD. They are just not stable enough to be a semaphore! Your devs are now spending a bunch of time investigating issues that aren’t actually issues. So now you’re back to manual testing and investigating everything again instead of building.
Hold on, you say. Doesn’t my end-to-end test suite now run faster than the manual testing I used to do? I can do more testing, right? Well sure… Until it breaks. And then you have to figure out why those tests broke. Was it a real bug? Or, yet again, do you need to manually update those stubborn & fragile automated tests? I bet it is the latter.
But your team already spent countless hours trying to figure it out. And your VP of Sales can’t help you anymore as she used to for manual testing since you have to know how to code to understand why on Earth this thing threw this weird exception.
OMG, but I MUST test, you scream, what do I do???
The Solution
We believe any testing system in 2020 must adhere to the following four bold principles:
1. No setup.
It must just work out of the box. No painful onboarding and countless hours wasted helping new people learn how to set up a testing environment.
2. It must generate tests for you when possible.
Yes, I get it, not every situation can be covered for you automatically. But when it can – it should be. And when it can’t generate them, it should be outrageously easy to create good tests, even for people who can’t write code.
3. It must never fail without a good reason.
Seriously. Flakiness must not be tolerated. It’s needlessly inefficient. Most of the reasons are addressable like timeouts, random page load times, and, most importantly, reliance on XPath.
4. The system must make it trivial to maintain your tests.
And do it for you when it can. You should spend as little time as possible reacting to breaking changes in your application. Got a big change on the first steps affecting hundreds of tests? No problem – the system should figure out that it is a single issue, not hundreds, and let you address it in place. With just a couple of button clicks and maybe an edit or two.
We believe that a great software testing system should help you to move faster, not slower. And this is why and how we built testRigor.
