Using test coverage

2017-03-27

Non-declarative code and test coverage

When testing non-declarative code, it's important to enable branching detection if your tool allows it, to ensure that you'll test every way your code can proceed.

Check the test coverage reports and ensure that every line helps implementing a feature and needs to be there. If you have tested all the relevant features and a line is still not being tested, then that line is not needed to enforce the features of your application and can be deleted. This is very useful for detecting what lines of code can be deleted after a refactoring... However, you can only feel confidence in this if all behaviors of your application are somehow tested. This is why it's important to test all the branches (even that very simple and obvious branch that produces a 404 if the object is not in the database).

Declarative code and test coverage

I believe declarative code does not need to be tested (although it can be). The non-declarative code responsible for making that declarative code work has already been tested where it was implemented (assuming you performed due diligence on your dependencies and they are thoroughly tested). Therefore, once you declare it correctly, it won't stop working unless the dependency changes its API (a dependency here can be something internal to the code, like other module, or an external library), and changes to the API should come with all the required changes to the dependent code anyway. If it happens to an external library, the users need to check the API changes and make the necessary adjustments before deploying the update.

Changes to declarative code should appear in code reviews and, if some declarative code changes with no corresponding change to the non-declarative code, this should obviously raise questions: why did this change, since the API remains the same? Manual tests should also be done to cover this.

QA, manual tests, exploratory tests

A comprehensive automated test suite is not a substitute for manual testing. Even if, theoretically, a code well-tested should have all bases covered, you never know if you forgot some edge case. One of the objectives of your development team should be to fix all bugs found in QA and write the unit tests to guarantee that will never happen again, and over time your code will be more and more thoroughly tested. However, you'll never be able to guarantee that all edge cases have been tested.

This is even more true for UI tests with real users, since only human testers can identify things about aesthetics and usability.

False sense of security

Some developers argue that 100% test coverage give a false sense of security, because this can lead to developers believing they can make any changes and not manually test thoroughly because the automated test suite will catch it. This is actually a problem with the developer, not the test coverage concept. Re-educate developers to test new features and all features adjacent manually with an exploratory mindset.