Why did we kill our test environment?

CI/CD Jun 17, 2019

Yes we did indeed completely removed test environment from our processed and de-provisioned it.

The story goes the following way:

The Problem

We started like everyone else with people writing and committing code to the code base and then eventually we would deploy it to the test environment and after validation it would go to production.

And yes, for the record, we do use TestRigor to test TestRigor. Well, that wasn't enough. Because TestRigor tests only deviations from previous build. And if some untested code gets into test, TesRigor more often than not will just update baseline thinking that this is how things should work.

Well, it wasn't always working as it should have. And even more, deployment to test to test things before it gets to production proved to be painful experience. It was sporadic and time consuming.

De-facto what was happening is if there was a blocker-level issue we would deploy it to production ASAP anyway. Yes, without testing. And guess what - it did broke the production.

More over the quality overall wasn't that great, since everyone was relaxed thinking that it WILL be tested on test environment before production.

And, on top of it, reviewing new changes on test proved to be tedious because they had a tendency to come in batches several people finishing several large features at the same time.

Needless to say we were struggling with quality (sic!). And we can't afford it obviously since we ARE a quality testing tool!


After some time of painful struggle it was decided to get rid of test environment all together. We embraced full CD and setup master build after merge to go directly to production.


What was the effect of this?

Controversially - we got to WAAAY better quality in production!

But why??

Well, I believe several things:

  1. We started to test it in production. Nothing is closer to production than production.
  2. Most importantly, developers now got a sense of ownership. We implemented a process where you need to deploy AND test your changes on your (temporary) dev environment. And that was super effective to drive quality up.

Check out what we envision:

the system is not public yet, however if you want it - just shoot me an email to sales at testrigor.com

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.