Kenneth's Corner

Engineering, data, and testing

Things I've unlearned in Test

- Posted in Testing by

One of the things that stuck with me from my time at Amazon was a willingness to constantly evaluate and challenge (or re-confirm) generally accepted best practices. Even their own leadership principles stated that they were only there "unless you know better" principles. As a result, my approach to Test Engineering has changed quite a bit since my first years at Microsoft, and I realized I've "unlearned" some things. Below are a few quick examples.

The role of Test/QA is to find bugs and validate requirements

This view is too specific and only captures the first part of the story: the visibility component. A highly-functioning Test organization adds value in two ways: 1) Test adds visibility into how the product behaves, and 2) Test makes the development process more efficient.

Finding bugs and validating requirements is part of adding visibility; however, so is reporting behavior under different circumstances, tracking performance, and documenting product configurations, setup, upgrades, etc. Test Engineering should know as much as possible about what's going out the door, bugs and all, and should relentlessly seek to understand the product being tested and how customers use and see that product.

Test Engineering makes the development process more efficient by providing quick, helpful feedback and assisting with driving testability and testing upstream in the development process. With good testing, developers can move faster with higher confidence, and your team and your company win.

Test harnesses should be highly architected, structured, and organized

I've seen a lot of over-complicated and incomprehensible tests. It isn't easy to make clear, simple tests, but those add the most value. Write tests in whichever ways best achieve that. Simple tests are more easily democratized. Ramp-up time is shorter. Failures due to complicated tests are less likely.

Make it easy to write tests, and write simple tests. Don't over-architect.

Tests should be written in the language of the thing being tested

Tests should be written in the language(s) that best facilitate testing.

It's important to use an industry-standard test framework

There are often many competing test frameworks. More important than a test framework: using the product through testing. That's what matters most. Focus on coverage, not framework fluff.

Don't repeat yourself

I used to take this to an extreme, generating broadly-encompassing test case generators to hit many input permutations with just a few lines of code. However, I'd often spend more time - usually much more - working on the perfect test case generator than it would have taken to create a base set of inputs and copy/paste/edit until I had everything I needed. So, instead of having inputs close to the test case, they were abstracted, took longer to write, and often had bugs in the generation code that took even more time to fix. Worst of all, I was the only one who could understand or maintain the code when it was done.

This isn't to say you shouldn't abstract standard functionality into libraries. But instead, when it comes to testing, obscuring the test interactions (parts of the test case) through abstractions with the product under test can hinder comprehension, take more time, and otherwise be fraught with enough peril to destroy any value-add the abstractions would otherwise create.

Track your test cases in a test case tracker

Test case trackers are often very expensive - hundreds of dollars a year per user - and hold your work (the test cases) hostage. Additionally, time spent futzing with a test case repo is NOT spent testing or adding other value to your team and company.

Let your automation document your test cases. Focus on that.

Don't reinvent the wheel

I've watched highly talented engineers spend significantly more time wrangling and debugging third-party libraries that do 80% of what needs to be done (and consist of 80% of things that aren't needed) than it would have taken to do what would have been required with custom code.

How much time have you spent in your career investigating how to do something in 3rd-party code that could have been coded directly without the library? If you do this a lot, you might need to "reinvent the wheel" more.

Conclusion

When it comes to testing, simpler is usually better. So, much of what I've "unlearned" are simplifications of process and tooling. Make it easy to write tests, and remove impediments that keep you from testing early and often.