Design (Un)Certainty and Decision Trees

Billy Vaughn Koen, in The Discussion of The Method: Conducting the Engineer’s Approach to Problem Solving, says, “the absolute value of a heuristic is not established by conflict, but depends upon its usefulness in a specific context.”

Heuristics often compete and conflict with each other. Frequently I use examples I find in presentations or blog posts to illustrate this point.

For example, I took this photo of a presentation by Michiel Overeem at DDD Europe where he surveyed various approaches people employed to update event stores and their event schemas.

Five different alternatives for updating event stores

Five different alternatives for updating event stores


Event stores are often used (not exclusively) in event-sourced architectures, where instead of storing data in traditional databases, changes to application state are stored as a sequence of events in event stores. For an introduction to event sourced architectures see Martin Fowler’s article. An event store is comprised of a collection of uniquely identified event streams which contain collections of events. Events are typed, uniquely identified and contain a set of attribute/value pairs.

Michiel found five different ways designers addressed schema updates, each with different tradeoffs and constraints. The numbers across the top of the slide indicate the number of times each approached was used across 24 different people surveyed. Several used more than one approach (if you add up the x/24 count at the top of the slide there are more than 24 updates). Simply because you successfully updated your event store using one approach doesn’t mean you must do it the same way the next time. It is up to us as designers to sort things out and decide what to do next based on the current context. The nature of the schema change, the amount of data to be updated, the sensitivity of that data, and the amount of traffic that runs through apps that use the data all play into deciding which approach or approaches to take.

The “Weak schema” approach allows for additional data to be passed in an event record. The “upcaster” approach transforms incoming event data into the new format. The “copy transform” approach makes a copy and then updates that copy. Michiel found that these were the most common. “Versioned events” and “In-Place” updates were infrequently applied.

I always want to know more about what drives designers to choose a particular heuristic over another. So I was happy to read the research paper Michiel and colleagues wrote for SANER 2017(the International Conference on Software Analysis, Evolution and Reengineering) appropriately titled The Dark Side of Event Sourcing: Managing Data Conversion. Their paper, which gives a fuller treatment of options for updating event stores and their data schemas, can be found here. In the paper they characterized updates as being either basic (simple) or complex. Updates could be made to events, event streams, or the event store itself.

Basic event updates included adding or deleting an attribute, or changing the name or value of an attribute. Complex event updates included merging or splitting attributes.

Basic stream update operations were adding, deleting, or renaming events. Complex Stream updates involved merging or splitting events, or moving attributes between events.

Basic store updates were adding, deleting or renaming a stream. Complex store updates were merging and splitting streams or moving an event from one stream to another.

An example of a simple event update might be deleting a discountCode attribute. An example of a complex event update might be splitting an address attribute into various subparts (street, number, etc.).

Importantly, their paper offered a decision framework for selecting an appropriate update strategy which walks through a set of decisions leading to approaches for updating the data, the applications that use that data, or both. I’ve recreated the decision tree they used to summarize their framework below:

Decision Tree for Upgrading an Event Store System from The Dark Side of Event Sourcing: Managing Data Conversion

Decision Tree for Upgrading an Event Store System from The Dark Side of Event Sourcing: Managing Data Conversion


The authors also tested out their decision framework with three experts. Those experts noted that in their experiences they found complex schema updates to be rare. Hmm, does that mean that half of the decision tree isn’t very useful?

Also, they offered that instead of updating event schema they could instead use compensating techniques to accomplish similar results. For example, they might write code to create a new projection used by queries which contained an event attribute split into subparts—no need to update the event itself. Should the left half of their decision tree be expanded to include other compensating techniques (such as rewriting projection code logic)? Perhaps.

Also, the experts preferred copy or in-place transformations instead of approaches that involved writing and maintaining lots of conversion code. I wonder, did they do this even in the case of “basic” updates. If so, what led them to make this decision?

Decision trees are a good way to capture design options. But they can also be deceptive about their coverage of the problem/solution space. A decision table is considered balanced or complete if it includes every possible combination of input variables. We programmers are used to writing precise, balanced decision structures in our code based on a complete set of parameters. But the “variables” that go into deciding what event schema update approach aren’t so well-defined.

To me, initially this decision framework seemed comprehensive. But on further reflection, I believe it could benefit from reflecting the heuristics used by more designers and maintainers of production event-sourced systems. While this framework was a good first cut, the decision tree for schema updates needs more details (and rework) to capture more factors that go into schema update design. I don’t know of an easy way to capture these heuristics except through interviews, conversations, and observing what people actually do (rather than what they say they prefer).

For example, none of the experts preferred event record versioning, and yet, if it were done carefully, maybe event versions would require less maintenance. Is there ever a good time for “converting” an old event version to a newer one? And, if you need to delete an attribute on an event because it is no longer available or relevant, what are the implications of deprecating, rather than deleting it? Should it always be the event consumers’ responsibility to infer values for missing (deleted) attributes? Or is it ever advantageous to create and pass along default values for deleted attributes?

This led me to consider the myriad heuristics that go into making any data schema change (not just for event records): Under what conditions is it better to do a quick, cheap, easy to implement update to a schema—for example deciding to pack on additional attributes, having a policy to never delete any?

People in the database world have written much about gritty schema update approaches. While event stores are a relatively new technology, heuristics for data migration are not. But in order to benefit from these insights, we need to reach back in time and recover and repurpose these heuristics from those technologies to event stores.Those who build production event sourced architectures have likely done this or else they’ve had to learn some practical data schema evolution heuristics through experience (and trial and error). As Billy Vaughn Koen states, “If this context changes, the heuristic may become uninteresting and disappear from view awaiting, perhaps, a change of fortune in the future.” He wasn’t talking about software architecture and design heuristics per se, but he could have been.

Being Agile About Documenting and Communicating Architecture


Software architects and developers often need to defend, critique, define, or explain many different aspects of the design a complex system. Yet agile teams favor direct communications over documentation. Do we still need to document our designs?

Of course we do.

We won’t always be around to directly communicate our design to all current and future stakeholders. Personally, I’ve never found working code (or tests) to be the best expression of a software design. Tests express expectations about observable system behavior (not about the design choices we made in implementing that behavior). And the code doesn’t capture what we were thinking at the time we wrote that code or the ideas we considered and discarded as we got that code fully functioning. Neither tests nor code capture all our constraints and working assumptions or our hopes and aspirations for that code.

So what kind of design documentation should we create and how much documentation should we create, for whom, and when? And what is “good” enough documentation?
Eoin Woods gave a talk at XP 2016 titled, “Capturing Design (When You Really Have To)” that got me to revisit my own beliefs on the topic and to think about the state of the current agile practices on documenting architecture.

One take away from Eoin’s talk is to consider the primary purpose of any design description: is it primarily to immediately communicate or to be a long-term record? If your primary goal is to communicate on the fly, then Eoin claims that your documentation should be short lived, tailored to your audience, throwaway, and informal. On the other hand, design descriptions as records are likely to be long-lived, preserve information, be maintainable and organized, and more formal (or well-defined).

Since we are charged to deliver value with our working software, it is often hard to pay attention to any perceived efforts at “slowing down” to describe our systems as we build them. But if we’re building software that is expected to live long (and prosper), it makes sense to invest in documenting some aspects of that system—if nothing more to serve as breadcrumbs useful to those working on the system in the future or to our future selves.

So what can we do to keep design descriptions useful, relevant, unambiguous, and up-to-date? Eoin argues that to be palatable to agile projects, design documentation should be minimal, useful, and significant. It should explain what is important about the design and why it is important; what design decisions we made (and when), and what are the major system pieces, their responsibilities and key interactions. Because of my Responsibility-Driven Design values and roots, I like that he considers system elements and their responsibilities as being minimally useful information descriptions of a system. But to me this just a starting point to get an initial sense of what a system is and does and does not do. There certainly is room for more description, and more details when warranted.

And that gets us back to pragmatics. A design description isn’t the first thing that developers think of doing (not everyone is a visual thinker nor a writer). I know I’m atypical because, early in my engineering career, I enjoyed spending 3 weeks writing a document on my universal linker worked and how to extend it and its limitations. I was nearly as happy producing that document as I was in designing and implementing that linker. It pleased me that sustaining engineering found that document useful years after I had left for another job.

So for the rest of you who don’t find it natural to create documentation, here’s some advice from Eoin:

  • Do it sprint-by-sprint (or little by little). Don’t do it all at once.
  • Be aware of tradeoffs between fidelity and maintainability. The more precise it is, the harder descriptions will be to keep up to date.
  • Know the precision needed by your document’s users. If they need details, they need details. The more details, the more effort required to keep them up to date.
  • Consider linking design descriptions to your code (more on that in another blog post)
  • Create a “commons” where design descriptions are accessible and shared
  • Focus on the “gaps”—describe things that are poorly understood
  • Always ask what’s good enough. Don’t settle for less when more is needed or more when less is needed.

To this list I would add: if design descriptions are important to your company and your product or project, make it known. Explain why design documentation is important, respond to questions and challenges of that commitment, and then give people the support they need to create these kinds of descriptions. Let them perform experiments and build consensus around what is needed.

Be creative and incremental. One company I know made short video recordings of designers and architects giving short talks about why things worked the way they did. They were really short—five minutes or less. Another team created lightweight architecture documentation as they enhanced and made architectural improvements to the 300+ working applications they had to support. It was essential for them that there be more than just the code as the knowledge about these systems was getting lost and decaying over time. Rather than throw up their hands and give up, they just created enough design documentation using simple templates and only as new initiatives were started.

Find a willing documenter. Sometimes a new person (who is new to the system and to the company) is a good person to pair with another old hand to create a high-level description of the system as part of “getting their feet wet.” But don’t just stick them with the documentation. Have them write code and tests, too. From the start.

What does karaoke have to do with being agile?

Last week I attended XP 2015 in Helsinki, Finland. This is my second time there…I hope to go next year, too. It is a unique blend of researchers and students along with people working in big and small companies doing all kinds and flavors of agile development from all over the world. 37 countries were represented. We had 11 experience reports (Ken Power and I were track co-chairs of experience reports. More about them in another post).

Lots of learning. We had fun, too.

One of the highlights of the conference banquet was Presentation Karaoke. 6 brave souls gave 3 minute impromptu speeches. Every 30 seconds a new, unknown slide appeared.

Llewellyn Falco’s PowerPoint karaoke at XP 2015


Here’s video of parts of a presentation by Llewellyn Falco.

Avraham Poupko won the competition by popular vote. Pardon the shaky camera…I was laughing and not holding my cell phone very steady ….

So what’s connection between PowerPoint karaoke and agile development? Each speaker ostensibly spoke on a conference theme. Mostly, they had fun. High performing agile teams, like these speakers, know how to roll with the punches. Sure there’s a theme, a plan, a backlog of work items. But sometimes requirements change and you need to adapt on the spot. Rather than throw up your hands or panic, you need to make things work. It is impossible to go with the flow if you are in a panic. Like improv, agility demands that you accept change. Unanticipated things happen. You accept them and you adjust.

Some speakers made connections between slides. Avraham spotted what he thought were mangoes on the head of someone on one slide and incorporated them throughout his talk. Giovanni summarized all the key points at the end (it is a wonder he remembered them…but again, he wasn’t in a panic). Llewellyn was master at dramatic pauses before finding something to say. Their presentations were great.

Next time you are faced with unexpected change, take some cues from these improvisers: pause, gather yourself, take stock of the situation and then figure out what you need to do. Don’t panic. You’ll get through it. And maybe with a more relaxed attitude you’ll even have some fun.

When in Rome…

I attended my first XP conference in Rome in May. As they say, “when in Rome, do as the Romans do.” The actual quote attributed to St. Ambrose is, “si fueris Romae, Romano vivito more; si fueris alibi, vivito sicut ib,” or “if you should be in Rome, live in the Roman manner; if you should be elsewhere, live as they do there.”

As Italians do, I enjoyed good food, good company, and great wine.

I gave a workshop on Understanding Design Complexity (using commonality-variability analysis) and a tutorial on Agile Architecture Values and Practices.

I also sampled research AND non-research sessions in equal measure. Unlike other agile conferences I’ve attended, research is a prominent part of this conference. I listened to several research presentations and volunteered to be a commenteer for one research paper.

The XP 2014 paper acceptance rate was somewhat selective, with over 50% of the submissions rejected. Research topics were wide-ranging including a case study on UX Design, a survey of user story size and estimation accuracy, another on agile development practices, a case study on visualizing testing, another on agile and lean values, and one comparing scripted with exploratory testing. Short papers touched on agile organizational transformations, Randoori Coding Dojos, and how expertise is located on agile projects. In addition, four experience reports were published (in contrast, 27 experience reports will be published and presented this year at the Agile Conference).

If the research papers I sampled are an indicator, PhD students seem to be busy doing empirical studies on agile practices, processes, and values. If you go to the Open University’s website you’ll see these topics listed under their Empirical Studies PhD program: The emergence of Agile software development, the role of physicality and co-location in agile software development, and XP and end user development

Agile software development is being studied and data being collected. The paper I commenteered, “Why We Need a Granularity Concept for User Stories” by Olga Liskin and her colleagues, reported on results gleaned from surveying developers (who self-selected themselves as agile developers) working on both private and open source projects on GitHub. I had three short minutes after the presentation to carry on a dialog with Olga about their findings. Fortunately we also had more time to discuss her work over lunch.

This paper raised as many questions in my mind as it answered. On small projects (10 people or less), 55% of the respondents said they did not estimate their stories on a daily basis. Is this an affirmation of the No Estimates movement, or just how people work on certain kinds of projects? Open source projects are quite different from product development. (Not all of the GitHub projects were open source ones, but still…). Depending on your project, you may simply work off a backlog, not necessarily do any estimates to forecast how much you can accomplish. Several developers I know only \break down their work into identifiable tasks. Effort estimates (as long as they know how to do the work) aren’t that important. They’re done when they are done. And if you build shippable, workable software each sprint, well, you always have something potentially useful to deliver.

Here’s just a sampling of what I’d like to know: For those who estimated, how did story size correlate to estimation accuracy? And what happens when stories are split? Are estimates for split stories more accurate? And just how important is estimation accuracy to those who make estimates? It just something they did to get a rough idea, was it “required” of them, or did they effectively use estimates to plan and make forecasts? Did people who estimate improve their accuracy over time? It seems that if you are learning how to use a new technology, once you’ve spun up, estimates should be more accurate. But how long does it take to get up to speed on your estimates?

Not surprisingly, the authors found that the smaller the story size and the better known the technology was, the more accurate estimates were. Research results often confirm the obvious (but it is still it is nice to have some empirical evidence to back up our intuitions).

It their conclusions the authors don’t recommend a “best story size”. I’m happy they didn’t. They didn’t have enough evidence. Conventional wisdom says stories should fit into sprints (comfortably). Makes me want to know more about the accuracy of larger stories. It seems reasonable, that the more work involved, the more possibility you’ll miss something that may influence the accuracy of your estimate. And you also may fudge on your estimates just so you make sure a story fits inside a sprint (because you don’t want to split it). But estimates are just estimates. They shouldn’t be expected to be 100% accurate. How do people behave differently when estimation accuracy is rewarded (or worse yet, punished)? Estimates are just estimates. You learn to recalibrate your efforts when a task is harder or easier than expected.

The authors cautioned that developers’ views about story size and estimates need to be balanced by others’ concerns. Too many stories can be burden a product owner. Developers in dysfunctional organizations might pad estimates, just so they have some slack (does anyone knowingly pad estimates? I’d like to hear from you.).

So is it better to bundle up small, related stories and estimate them as a single unit? Maybe. Back in the days when I had to estimate work, I didn’t like tasks being too small. If they were, my manager would look at them more closely (I’m not sure why). I remember once telling a manager, you can ask what we will be doing every day and how long each task will take, or we will guarantee that we will deliver all the features above the cut line in the next two weeks. But you can’t have both daily accuracy and predictability. She backed off on knowing exactly what we were doing every day (as long as we weren’t stressed out). This was long before the Agile software development movement, but it still seems relevant. Our small team worked off a prioritized list of features. It didn’t matter who did what task, whenever a task was finished, the next one was picked up. And we finished our work on schedule…because we wanted to make our commitment.

Here’s one parting thought about empirical studies. I’m very wary of biases that work their way into them. Those who answered the agile estimation survey might be very different from those who did not. Self-reporting of any thing we do (whether it be software estimation, the amount of food we eat, or how much we exercise) is notoriously inaccurate. We underestimate our weight, overestimate our capabilities, and don’t remember accurately. More accurate evidence is obtained through field studies where people are observed working. I wish more software empirical researchers could have opportunities to work directly agile teams and spend significant time getting to know them and how they work (in addition to just asking them questions).

Las Vegas….gambling on agile?

OK, I want a catchy title… But I also want to tell you about the upcoming Better Software Conference and Agile Development Practices in Las Vegas June 6-11 where I’ll be presenting a one-day tutorial on Writing Effective Agile Use Cases. No I am not co-opting Alistair Cockburn’s bestselling Writing Effective Use Cases….this title was suggested by the conference organizers. They think it is more catchy than what I proposed: Writing Agile Use Cases.

I hope to share with you effective techniques I’ve picked up for writing use cases in an agile development environment. While I am not a believer in writing for writing’s sake, I happen to find that writing documentation on agile projects to not be intrinsically evil. I’ve worked with several agile teams to trim down their project documentation, write effectively, and to focus on what matters to them. Not every user story should be part of a use case description. And not every user story needs to be documented. But I believe in the power of the written word when it is effective, streamlined, and to the point. And I find that use cases that focus on usability and defining user tasks to be well-received by agile teams. Especially if they are done in context with usability experimention, wizard of oz prototyping, and lightweight user interface specification. So come join me in Las Vegas. Or ask me about agile use case writing workshops…where we hone our writing skills and write project-specific use cases.

Three in a Row

I just got home from a 3-in-a-row conference marathon. First I went to the Patterns of Programming Language Conference or PLoP in Nashville, where our paper Dynamic Factory, co-authored with Leon Welickiand Joe Yoder, was shepherded in a writers’ workshop. This was my first PLoP where I actually had fun. Our writers workshop group was a sharp group and gave everyone excellent feedback. The first time I had a paper workshopped at PLoP a couple of years ago I was disappointed about the quality of the feedback. Speaking with more seasoned patterns conference attendees, I learned that workshopping is hard work–and not always does it work out well. It takes a combination of a skilled workshop leader, workshop attendees who’ve critically read their papers, and a good mix of insight and discussion. Sometimes, even with the best intentions a workshop doesn’t deliver good feedback to every paper. When everything comes together the results are amazing.

Then it was on to OOPSLA where 750 geeky souls met to discuss programming languages, design, practice, and what the next software trends might be. At OOPSLA I presented a keynote, What Drives Design? that I’d spent the last five weeks working on. I talked about the main ideas behind Responsibility-Driven Design (the first, “driven” method) and then compared and contrasted several different design methods. For the record, I characterized TDD as “design between the keystrokes” and Feature Driven Development as “Design first, program next.” Truth be told there is a lot of design that happens when people practice TDD. But refactoring isn’t the same as initial design ideating. It still begs the question, when do you do an initial design (if ever)? Is it always at the keyboard? Scott Amber, with Agile Model Driven Development interjects a “model when you need to” front end to TDD practices. Not specifying what you should model, but giving people the framework to plug in non-programming design efforts when they think they need them.

Last week I was at SD Best Practices in Boston–the consummate conference for those seeking technical trends and practical advice. I presented my tutorial,The Art of Telling Your Design Story, and a talk on Giving and Taking Design Criticism. At the conference I enjoyed hearing Neal Ford take a swipe at the inefficiencies/unnecessary clutter of implementing design patterns in traditional (read non-dynamic) languages. Neal is a compelling speaker. At the end of his talk I was convinced that Ruby doesn’t have anything over Smalltalk. Neal confirmed my thoughts about Smalltalk. Still, I plan on taking a deep dive into Ruby this winter.

I’m somewhat concerned when DSL-makers tack on behaviors to any class of thing in order to have a fluent language. While that may work in the short run, I’m betting meta-programming tricks and behavior hacks to basic classes won’t scale. Call me a traditionalist, but I remember the clutter of extra behavior tacked on to Objects, Integers and Strings causing real problems in Smalltalk. It wasn’t considered good practice then…and I’m not sure I’d make the same “fluent language” tradeoffs Neal wood. Still, his talk got me thinking.

I also enjoyed hearing Rick Brenner’s Organizational Politics for Those Who Hate Politics. While I don’t have office politics to contend with in my own business, I do encounter lots in my clients. SD Best Practices is a conference where you can blend geek topics with management or professional development topics and come away feeling well-rounded.

Round the world in 28 days


I’ve just returned from four weeks travel to Athens, The Netherlands, Australia (Brisbane and Sydney), and Las Vegas. It is good to be home in Sherwood sleeping in my own bed. This trip was a combination of vacation, work, and geeky holiday. I and spoke at three different conferences in three weeks. JAOO Brisbane and Sydney was an opportunity to hear Erik Meijer give a great talk about why fundamentalist functional programming languages (think Haskell) solve the problems we procedural and oo language programmers just sweep under the rug. And then I got to grill Erik on why he thinks that declaring types to include side effects so important to writing good programs. Where else does a geeky woman get to hang out and talk shop with other software geeks? I snuck in some sight seeing too. The picture is of me taking rental bike across the Brisbane Harbor on a ferry. A bike ride with Dave Thomas and Kresten Thorup was about the only sunny day we had in Brisbane. The rains came to Australia just in time for JAOO.

In Greece I saw lots of ruins, attended the annual IEEE Software planning meeting, and ate lots of Greek salads, simply prepared fish, and drink thick coffee. They call it Greek coffee, but a few years ago even the Greeks called it Turkish coffee. But one highlight I won’t forget is hearing Linda Rising and her husband Karl sing “Take me out to the Ball Game” at the ampitheater at Epidaurus, Greece. They volunteered to demonstrate the phenomenal acoustics. It gave me goose bumps. Constructed in 500 BCE, the ampitheater perfectly amplifies sound from on stage to everywhere in the theater. You can whisper stage center and people in the back row can hear you perfectly. And sound is amplified back to you, too. Truly an engineering marvel, the acoustics are because of the location and how the ampitheater was carved into the rocky hillside.

I’ll be sliding back into a more normal work routine, but before the magic of this wonderfult trip fades, I hope to share some thoughts and reflections and experiences over the next few days.

Lessons Learned from Architecture Reviews

Last year I talked about lessons learned from architecture reviews at JAOO 2007 and Markus Voelter from Software Engineering radio interviewed me. You can listen to our conversation.

I’ve experienced both sides of reviewing. Early in my engineering career I presented the design of a universal linker to an external reviewer. I was scared to death that the reviewer would pick apart my design. I was relieved when my design was blessed, but annoyed when the reviewer doubted my ability to deliver on time. (I delivered, but in hindsight I could see why he and my management doubted me–as a new engineer I was working solo on a critical project…I just didn’t know what I couldn’t do).

These days, complex IT systems are rarely understood by a single engineer or architect. Teams come together to create complex software systems. Technical challenges can be enormous and the “tricky bits” involve the subtle interplay of business and technical design decisions. The focus is on achieving overall business objectives, not the optimal design of a single component. Yet poorly designed interfaces, sluggishly performing services, or crappily constructed components can cause enormous grief. Design still matters.

Agile Open Northwest 2008


Last year we held an open space in Portland at the Kennedy School focused around the theme, “Agile for real.” The response was so positive that we’re picking up the conversation again at the Seattle convention center March 18 and 19. I’m happy to get to renew connections with folks I met last year and excited to meet some new folks and hear their agile experiences. Attendance is limited to 100. Registration is filling fast. We’ve kept the price low ($100) so that even if your company can’t afford to send you, you might still attend. Most who’ve registered so far are from Seattle, but this is a northwest regional conference. So expect a few of us from Portland and elsewhere in the northwest to show up, too. And maybe even some folks from further away. We hope to see you there!