Incremental design isn’t always the best way to tackle really hard design problems. Sometimes you need to explore the territory ahead before you commit to a solution. I have a client who has been rolling out a complex business system over several years. They have several production releases a year. Yet at the same time they’ve also investigated design solutions for future releases. Over the past year, they’ve worked out some key design ideas and implemented several prototypes that demonstrate how to solve some complex processing. Some of these prototypes have worked their way into production; some have been stopped after lessons were learned; others are still in the "speculative pipeline". How have they managed to successfully balance this speculative work along with production release work?

A major key to their success was to not tie any speculative design work to any specific production release. Keeping this work off the production schedule has let them explore some ideas that required some heavy mental lifting, research and prototyping. But they have been disciplined about this work, too. Speculation isn’t another name for "noodling around". Here are some practices that seemed critical to their success: Defining concrete, realistic scenarios to drive design efforts. Having a knowledgeable leader/practical visionary set goals and monitor progress. Tackling thinking and design work in managed increments. Measuring progress. Knowing when to stop pursuing an idea. Delivering value—whether it is a description of an algorithm, a partially thought out object model, a working prototype, a summary of lessons learned, or all of the above—on a periodic basis.

I think teams can be successful at speculative design if they take a disciplined approach. If you have found ways to successfully balance future design while at the same time incrementally developing production software, I’d like to hear from you. What worked, what didn’t, and how did you find the right balance?