Why software innovation is not a free pass for doing whatever you want

Somebody on Twitter recently posed the question whether innovation in software and agile development can co-exist or not. To remove any misinterpretation - something quite common for Twitter discussion - I asked him to clarify what he meant with the word 'innovation'. In short, any kind of software development that is difficult to estimate. Examples included such things as investigating the applicability of a new library or framework, the introduction of a new tool, troubleshooting bugs in production, bringing down debt, or improving the performance of a system. He argued that the inherently creative nature of these things makes them impossible to estimate and unsuitable for use with agile methodologies like Scrum or Kanban.

Having worked on all of the above at some point in my career, I'm very well aware of that creative part. I've wasted hours of my time trying to explain managers and product owners that comparing software development with industrial manufacturing is so wrong in so many ways. However, none of this gives a software developer the right to forfeit a process and just do whatever they feel like. Unless you're working on some kind of research project with enormous amounts of budgets, don't give me that "I don't need any mental structure" excuse. Most professional software teams are under some kind of budget, be it money or just development capacity. Either way, somebody is paying (for) you, so it is not unreasonable for that person to understand what they will get out of it. They may not understand every aspect of it, but if you can't reason about what you're doing in some form, I think you're being unprofessional.

Don't get me wrong, I don't like unnecessary ceremony any more than you do. But I do believe that tracking whatever you're working on is valuable to both your team as yourself. In my opinion, User Stories have many characteristics that can help you with this. They are supposed to be estimable (in particular when applying the INVEST principle), so you can use them to build up some historical data about your team as well as to maintain a burn-down chart. And the mere fact that you need to estimate something forces you to think about what you're going to do. In my experience, just discussing the scope of a story with your team surfaces a lot of aspects and edge cases that help you refine the purpose of the story. Defining the stories using the who, what and why pattern can really help here. Understanding who is going to benefit from the work and what they are trying to solve or accomplish is crucial. Because of its low threshold, I prefer Microsoft OneNote to structure my thoughts. Again, if you can't at least formulate your thought process in a couple of OneNote check marks, I sincerely doubt you are ready to do any coding. And don't worry too much about the granularity of those estimates. Small, medium and large is precise enough for most projects.

Even if you're working on something that is difficult to estimate I would still expect you schedule a time-boxed story. In other words, you assign a fixed number of days (or whatever units you estimate in), you define a prioritized list of goals you want to achieve, and re-asses after that fixed amount of time has passed. If you don't, my experience is that you end up in an ever-lasting research mode without a clear goal or a well-defined purpose. That fixed amount of time forces you to re-evaluate your findings from a high level make a conscious decision on how to continue from there. By sharing your results up to that point, you and your team might even realize that the path you were following should be abandoned. And that's exactly what the true nature of agile development is about. Trying a way of working, reflecting on that, and acting upon those results. And remember, it's an open process in which everybody can contribute. So if time-boxing doesn't work for you or your team, find another way and try again. Just keep embracing the strengths of agile development, regardless of the uncertain or creative nature of your work. Don't accept any excuses otherwise.

So what do you think? Do you agree that innovative software development is no excuse for just doing whatever you want? I'd love to hear your thoughts by commenting below. Oh, and follow me at @ddoomen to get regular updates on my everlasting quest for better solutions.