False positives and semantic versioning

As part of stabilizing an upcoming release, I always dog food a beta package against the 12000 unit tests in one of our bigger projects. In the early days, that would surface all kinds of edge cases I never thought of. In every single case, the first thing I would do is to add a new unit test to Fluent Assertions to make sure that edge case would be covered from that point on. But during the last couple of releases, finding a failing unit test would be pretty unique. What an unpleasant surprise it was when I encountered about 150 failing unit tests when running against a beta of v3.4.

The cause of this was FA's most powerful extension method ShouldBeEquivalentTo. It performs a recursive comparison of two object graphs. To determine which properties have to be included in the comparison, by default it's supposed to use the compile-time type (a.k.a. the declared type) of the objects in the graph. However, in early releases this didn't behave entirely consistent. Both me and top contributor Adam Voss have worked on the internals many times thereby slowly improving the consistency with every release. In v3.4 we both applied some more improvements, but those unfortunately surfaced a false-positive from an earlier release.

In our particular case, we were comparing a collection of events, declared as Event, with another collection. Because of that existing bug (I'm not sure when it was introduced), it would still use the run-time type of the particular event during the comparison. The call to ShouldBeEquivalentTo excluded the properties of the Event base-class. And since v3.4 will now include the properties defined by Event class only, the net result is the following InvalidOperationException:

No members were found for comparison. Please specify some members to include in the comparison or choose a more meaningful assertion.

This particular exception can be solved by using the IncludingAllRuntimeProperties option. But be prepared for unit tests that suddenly fail on some nested property's value that didn't match the expectation. In all cases where this happened in our code base, the unit test failed correctly. In other words, we had some pretty serious false positives.

So what does this have to do with semantic versioning? Well, Fluent Assertions' release strategy is based on semantic versioning. This requires me to carefully think about the version number increment and how that affects the changes I make in a particular release. To elaborate on that a bit, assume the current version is v3.4.0. If I change some internal logic without affecting the public API and/or fix a bug in a backwards compatible fashion, I'm supposed to increment the version to v3.4.1. If instead, I'm adding new extension methods, additional overloads or marking APIs obsolete, the version should get bumped to v3.5. And finally, if I would drop those obsolete APIs or a particular .NET framework variant, I must change the version to 4.0. In short, the versioning strategy is not based on a marketing decision, but purely derived from the changes you made. By strictly adhering to this, people using v3.3 should be able to upgrade to any other v3.x version with confidence.

So looking back at this false-positive, the question remains whether this release demands a major, minor or patch increment. Since this release also includes new backwards-compatible API changes, it's going to be at least a minor increment. Treating it as a patch release is out of the question. However, when somebody updates their FA version from v3.3 to v3.4, chances are that it'll break some unit tests. You could say that is a breaking change they want to postpone to a later point in time. On the other hand, we're talking about a false positive here. I guess people would like to know that their production code contains a problem not previously covered by Fluent Assertions. Having said that, I've decided to treat this fix as a normal bug fix.

Considering this story, I can imagine more gray area exist in the semantic versioning realm. I couldn't find a nice spot to discuss those topics, not on Gitter nor on Jabber. So until somebody else comes up with a better place, I've created a Gitter room. Please join me here if you have questions about semantic versioning.