Architecture Patterns for the Rest of Us

Recently I have been reading and enjoying Robert Hanmer’s new book, Pattern-Oriented Software Architecture for Dummies. Disclaimer: In the past I have shied away from the books for dummy series, except when I wanted to learn about something clearly out of the realm of my expertise. Even then, just the notion of reading a book for dummies has stopped me from several dummy book purchases. Good grief! I didn’t know what I have been missing.

This book is not theoretical or dry. The prose is a pleasure to read. And it goes into way more depth than you might expect. Instead of simplifying the Patterns of Software Architecture book that is also on my bookshelf, I’d say it adds and complements it with clear explanation of benefits and liabilities of each pattern, step-by-step guides to implementing each architectural pattern and more. As an extra bonus, the first two parts of the book contain some of the clearest writing about what patterns are and how they can be used or discovered that I’ve seen.

I wish Bob Hanmer would write more patterns for the Dummy series. He knows his subject. He has good, solid examples. And he doesn’t insult your intelligence (In contrast, I find that the Head First books are definitely not my cup of tea…I don’t want to play games or have patterns trivialized). Bob has an easy, engaging style of writing. The graphics and illustrations are compelling In fact, I reproduced a couple of the graphics about finding patterns in a lecture, with credit to Bob of course, in a lecture I gave last week to my enterprise application design students.

This is a good book. If you’ve wondered about software architecture patterns and styles, read this book. Buy it. And tell your software buddies about it, too.

Why Domain Modeling?

One barrier to considering rich domain model architectures is a misconception about the value or purpose of a domain model. To some, creating a domain model seems a throwback to earlier days where design and modeling were perceived to be discrete, lengthy, and mostly unproductive activities.

When object technology was young, several notable authors made a strong distinction between object-oriented analysis and object-oriented design and programming. Ostensibly, during object-oriented analysis you analyzed a task that you wanted to automate and developed an underlying conceptual (object) model of that domain. You produced a set of task descriptions and a model of objects that included representations of domain concepts and showed how these objects interacted to accomplish some work. But you couldn’t directly implement these objects. During object-oriented design you refined this analysis model to consider implementation and technology constraints. Only then, after finishing design, would you write your program. The implication was that any model you produced during analysis or design needed extensive manipulation and refinement before you could write your program.

But even in those early days, many of us blurred the lines between object-oriented analysis, design and programming. In practice, how we worked was often quite different than suggested by the popular literature of the time. We might analyze the problem, quickly sketch out some design ideas and then implement then. We might use CRC cards to model our objects (which we would then discard). There weren’t distinct gaps between analyzing the problem and designing and implementing a solution. Sometimes different people did analysis while others did design and programming; but many times a developer would do all these activities. Sometimes, we created permanent representations of some models (in addition to our code). It depended on the situation and the need.

These days, I rarely see anyone produce detailed object analysis or design models. In fact, design and modeling have gotten somewhat of a bad wrap. Good object design is deemed too difficult for “average” programmers, and there isn’t time apart from coding to think about the domain and come up with any models.

The most common object models I see are created for one of two purposes: small conceptual models constructed to gain an understanding of significant new functionality; or informal design sketches intended to provide a quick overview to newcomers or non-code reading folks who need to “know more” about the software. A lack of modeling (unless you consider code or tests to be models—I don’t) is prevalent whether or not the team is following agile practices. The most common models I do see are very detailed E-R models that are more implementation specifications than models. They don’t leave out any details making it hard find the important bits.

Understanding and describing a domain and creating any model of it in any form takes a back seat to most development activities.

But if your software is complex, rapidly changing, and strategic and you aren’t doing any domain modeling, you may be missing out on something really important. If your software is complex enough, you can greatly benefit from domain modeling and thoughtfully doing some Domain-Driven Design activities.

For example, Domain Driven Design’s strategic design is a conscientious effort to create common understanding between business visionaries, domain experts and developers. Initial high-level domain discussions lead to understanding what is central to the problem (the core domain) and the relationships between all the important parts (sub-domains) that it interacts with. Gaining such consensus helps you focus your best design efforts and structure (or restructure) your software to enable it to sustainably grow and evolve.

But it doesn’t stop there. If you buy into domain modeling, you also commit to developing a deep shared understanding of the problem domain along with your code. Your mission isn’t to just deliver working functionality, but to embed domain knowledge in your working solution. Your code will have objects that represent domain concepts. You’ll be picky about how you name classes and methods so they accurately reflect the language of the domain. You will have ongoing discussions with domain experts and jointly discuss and refine your understanding of the domain. Along the way you may sketch and revise domain models. You’ll strive to identify, preserve, strengthen and make explicit the connections between the business problem and your code. When you refactor your design as you gain deeper understanding, you won’t forget to reflect the domain in your code. Your domain model lives and evolves along with the code.

As a consequence, there isn’t that big disconnect between what you code and what the business talks about. And that can be a powerful force for even closer collaborations between developers and domain experts.

Domain-Driven Design Applied

Vaughn Vernon’s new book, Implementing Domain-Driven Design, is full of advice for those who want to apply design principles and patterns to implementing rich domain model applications.

Check out my interview with Vaughn for InformIT. We discuss why he wrote this book and some important new Domain-Driven Design patterns and their architectural impact.

There is much to learn about good design of complex software from this book. In future blog posts I plan to unpack some of the meatier domain-driven design concepts and reflect on how best to apply them to designing enterprise applications.

Evangelizing New Software Architecture Ideas and Practices

Last December I spoke at the YOW Conferences in Australia on Why We Need Architects (and Architecture) on Agile Projects.

I wanted convince agile developers that emergent design doesn’t guarantee good software architecture. And often, you need to pay extra attention to architecture, especially if you are working on a large project.

There can be many reasons for paying attention to architecture: Meaningful progress may be blocked by architectural flaws. Some intricate tricky technical stuff may need to be worked out before you can implement functionality that relies on it. Critical components outside your control may need to be integrated. Achieving performance targets may be difficult and you need to explore what you can do before choosing among expensive or hard to reverse alternatives. Many other developers may depend on some new technical bit working well (whether it be infrastructure, that particular NoSQL database, or that unproven framework). Some design conventions need to be established before a large number of developers start whacking away at gobs of user stories.
I characterized the role of an agile architect as being hands-on; a steward of sustainable development, and someone who balances technical concerns with other perspectives. One difference between large agile and small agile projects is that you often need to do more significant wayfinding and architectural risk mitigation on larger projects.

I hoped to inspire agile developers to care more about sustainable architecture and to consider picking up some more architecture practices.

Unfortunately, my talk sorely disappointed a thoughtful architect who faces an entirely different dilemma: He needs to convince non-agile architects to adapt agile architectural practices. And my talk didn’t give him any arguments that would persuade them.

My first reaction to his rant was to want to shout: Give up! It is impossible to convince anyone to adopt new way of working that conflict with his or her deeply held values.

But then again, how do new ways of working ever take hold in an organization? By having some buzz around them. By being brought in (naively or conscientiously) by leaders and instigators who know how to build organizational support for new ideas. By being new and sexy instead of dull and boring. By demonstrating results. By capturing people’s imagination or assuaging their fears. Surreptitiously, quietly replacing older practices when reasons for doing them are no longer remembered. When the old guard dies out or gives up.

Attitudes rarely change through compelling discussions or persuasive argumentation. I look to Mary Lynn Manns and Linda Rising’s Fearless Change: Patterns for Introducing New Ideas for inspiration.

I take stock of how much energy I want to invest in changing attitudes and how much investment people have in the way they are doing things now. I don’t think of myself as a professional change agent. Yet, as a consultant I am often brought in when organizations (not necessarily every person, mind you) want to do things differently.
People generally aren’t receptive to new ideas or practices or technologies when they feel threatened, dismissed, disrespected, underappreciated, or misunderstood. I am successful at introducing new techniques when they are presented as ways to reduce or manage risks or increase productivity or reliability or improve performance or whatever hot button the people who I am exposing these new ways of working are receptive to. Labeling techniques as “agile” or “lean” may create a buzz in those that already receptive. But the reaction can be almost allergic in those who are not. The last thing I want do is to foster divisiveness. Labels do that. If I get people comfortable taking just that next small step, that is often enough for them to carry on and make even more significant changes. Changing practices takes patience and persistence. At the end of the day I can only convince, demonstrate and empathize; I cannot compel people to make changes.

Thanks to Joe and Jason…this blog is live again

It has been months since I’ve posted a blog entry. Why? Initially, life got busy. Then, in October my wordpress site was hacked. making it impossible to see or use my admin console. I went searching for fixes to this problem on the internet and at the wordpress community site, but didn’t find a whole lot of useful information. I was advised to reinstall a later version of wordpress and clear out my plugins…

I tried those suggested fixes. But I took conservative measures. Instead of reinstalling over my existing directories, I cleared out my plugins and only reuploaded portions of my wordpress installation. That did not fix my problem. I then backed up my wordpress database and tried a new installation of all wordpress files. I installed my new wordpress files into a separate directory (just to test whether it would fix my blank admin page problem). This did not work either. Little did I know that wordpress kindly redirects you back to your known directory path (which is conveniently stored in in the wordpress database) so it was accessing my corrupted wordpress installation directory structure, not the new installation.

If I’d only overwritten my existing wordpress directory with a fresh installation that would’ve fixed the problem. But I didn’t. Why? Because I thought I was being conservative by pointing a new wordpress installation of wordpress files to my database (and leaving the old installation intact). Big mistake.

I spent a day sleuthing my “blank” admin screen. I turned on php debugging. WordPress was getting redirected to a non-existent file (supposedly), but it seemed that the file was in my blog directory path. What was going on? I wasn’t sure. Never did figure out why that happens. (I really want to know about paths during execution of wordpress).

Next, I rooted around in all my directories after perusing some scanty but helpful documentation on the path and file structure of wordpress installations. When I did, I discovered a few suspicious php files in my wp-contents directory. I renamed they (hoping they wouldn’t be invoked, and that maybe they were causing the problem). But no luck. Gr.

Finally, I turned to my colleague and good friend Joe Yoder of The Refactory for help. Joe, who in addition to being a world class designer and Adaptive Object guru, also hosts a fair number of websites. He, along with his colleague Jason Frye, fixed my problem within the hour. First they installed wordpress and my database contents on their servers to check that my database wasn’t corrupted (wordpress ran just fine). Then they FTPed a fresh version of wordpress onto my server, and presto! I was back in business. They left me my old, corrupted version of wordpress in a newly renamed directory, just so it’s there in case I want to ever go figure out what was wrong.

I learned a few lessons during this very painful process: Advice on wordpress troubleshooting you find online is pretty sketchy. The “real” experts out there are sysadmin folks who’ve done enough wordpress installations and troubleshooting to know what to look for…and when they can, they reinstall a new version of wordpress rather than try to hack on a partially working installation. Not everyone has a Jason or a Joe to lean on. I’m glad I did. Thanks, guys!

I’ve been told by another friend that the wordpress’ execution model is pretty complex. Lots of paths and looking in different places…But after this experience, I’d be happy to help someone who is a guru, more fully document how wordpress works, what the database table structure and contents are, what the contents of subdirectories and when they are invoked, etc., etc. I’m serious about this.

Why?

I like to know how things work. I’m a designer, after all. I like to know how software works. I’d also like it if people who had wordpress installation problems had an easier time than I did (I got pretty discouraged). In my explorations, I uncovered many frustrated people who were asking for help and didn’t get very far. It doesn’t have to be that way.

If a deeply wordpress savvy person would like to talk about co-writing a small guide on “principles” of operation of wordpress, I’m game. I like to write. And explain how things work. Simply. So people don’t just have to “guess” and hack in as they struggle to fix problems. And with such a guide, I believe life would be for those who want to blog as well as know what’s going on, under the hood.

Evangelizing New (Software Architecture) Ideas and Practices

Last December I spoke at the YOW Conferences in Australia on Why We Need Architects (and Architecture) on Agile Projects.

I wanted convince agile developers that emergent design doesn’t guarantee good software architecture. And often, you need to pay extra attention to architecture, especially if you are working on a large project.

There can be many reasons for paying attention to architecture: Meaningful progress may be blocked by architectural flaws. Some intricate tricky technical stuff may need to be worked out before you can implement functionality that relies on it. Critical components outside your control may need to be integrated. Achieving performance targets may be difficult and you need to explore what you can do before choosing among expensive or hard to reverse alternatives. Many other developers may depend on some new technical bit working well (whether it be infrastructure, that particular NoSQL database, or that unproven framework). Some design conventions need to be established before a large number of developers start whacking away at gobs of user stories.

I characterized the role of an agile architect as being hands-on; a steward of sustainable development, and someone who balances technical concerns with other perspectives. One difference between large agile and small agile projects is that you often need to do more significant wayfinding and architectural risk mitigation on larger projects.

I hoped to inspire agile developers to care more about sustainable architecture and to consider picking up some more architecture practices.

Unfortunately, my talk sorely disappointed a thoughtful architect who faces an entirely different dilemma: He needs to convince non-agile architects to adapt agile architectural practices. And my talk didn’t give him any arguments that would persuade them.

My first reaction to his rant was to want to shout: Give up! It is impossible to convince anyone to adopt new way of working that conflict with his or her deeply held values.

But then again, how do new ways of working ever take hold in an organization? By having some buzz around them. By being brought in (naively or conscientiously) by leaders and instigators who know how to build organizational support for new ideas. By being new and sexy instead of dull and boring. By demonstrating results. By capturing people’s imagination or assuaging their fears. Surreptitiously, quietly replacing older practices when reasons for doing them are no longer remembered. When the old guard dies out or gives up.

Attitudes rarely change through compelling discussions or persuasive argumentation. I look to Mary Lynn Manns and Linda Rising’s Fearless Change: Patterns for Introducing New Ideas for inspiration.

I take stock of how much energy I want to invest in changing attitudes and how much investment people have in the way they are doing things now. I don’t think of myself as a professional change agent. Yet, as a consultant I am often brought in when organizations (not necessarily every person, mind you) want to do things differently.
People generally aren’t receptive to new ideas or practices or technologies when they feel threatened, dismissed, disrespected, underappreciated, or misunderstood. I am successful at introducing new techniques when they are presented as ways to reduce or manage risks or increase productivity or reliability or improve performance or whatever hot button the people who I am exposing these new ways of working are receptive to. Labeling techniques as “agile” or “lean” may create a buzz in those that already receptive. But the reaction can be almost allergic in those who are not. The last thing I want do is to foster divisiveness. Labels do that. If I get people comfortable taking just that next small step, that is often enough for them to carry on and make even more significant changes. Changing practices takes patience and persistence. At the end of the day I can only convince, demonstrate and empathize; I cannot compel people to make changes.

How far should you look ahead?

I’m a consultant. So an easy answer would be, “It depends.”

My real answer is more nuanced: only look ahead far enough to see some significant issue and a plausible way to resolve it. If you want to get better, practice looking ahead in the context of your work and its established rhythms and natural cycles.

If your organization doesn’t value planning and you want to look ahead, start by taking small steps. The reason to look ahead is to check for some pre-work or preparation that is likely to save time or money.

If you are a tech lead, try looking ahead in the context of release planning. See what architecture risks you spot (or new design requirements) as new features are scoped. Take notes on what you are concerned about. Spend some time speculating on what you should be doing architecturally to support the next release. Have a conversation with a trusted colleague about your concerns. Identify concrete follow-on activities specifically targeted at addressing your biggest concerns.

If you are an agile developer, you may not want to look further ahead than the current sprint. If you aren’t already doing so, try some iteration modeling for a couple of hours at the beginning of an iteration. Grab a handful of related story cards for a new feature and sketch out some design ideas. Draw sequence diagrams on a white board. Jot down class or component responsibilities on CRC cards. Noodle around. Give yourself the luxury of exploring with your co-workers how things might work. Don’t expect a final design, just a plausible one that will be fleshed out and revised as you implement those stories.

If your organization tends to look too far ahead, take actions to personally shift your focus away from endless planning. Winston Churchill said, “It is always wise to look ahead, but difficult to look further than you can see.” Looking too far ahead is also a big waste of time. Remedy this time suck by limiting the scope of what you plan for. Find opportunities to build and measure things.Learn to perform small experiments, make small decisions, communicate your results and move things along.

Colleagues and co-workers I’d classify as over-eager forward-looking designers typically build up comfortable infrastructure they think will be needed as a matter of course before they settle in to the programming task at hand. A good indicator that they’ve gone overboard is how others react to their work. One unhappy forward-looking architect remarked recently, “Not many people appreciate what I am trying to accomplish.”

You may not be aware that you look too far ahead. One indicator: you view most programming tasks as an opportunity to build or try out a cool new framework or create your own special DSL. (If you are part of an infrastructure team, then, fine, that is your job. Don’t embellish your creations. And make sure others can actually use your stuff.) If you are an application developer, nudge yourself towards refactoring and finding abstractions through concrete scenarios. Hold yourself back from writing speculative code.

I look to my colleague, Joe Yoder, as someone who takes a balanced approach when creating infrastructure or suggesting novel architectures. Joe likes to build flexible, adaptable systems, in particular Adaptive Object-Model architectures. He has years of experience doing this. He’s enthusiastic when opportunities come up to build these kinds of systems, but knows that this architecture style isn’t a good fit for every rapidly changing system. And he doesn’t build every part of such a system this way; only the parts that require extensive, ongoing customizations.

How far do you look ahead? Do you think you’ve found a good balance?

Agile Architecture Myth #5: Never Look Ahead

Stop it! Stop anticipating! Get coding. Get that next user story working. Don’t. Ever. Anticipate.

This seems to be a common agile mantra these days. Since we allegedly aren’t good at anticipating how code will evolve, well frack it, let’s just stop all speculation.

Instead, just do your level best to code for what you know right now. Let the design emerge. Keep it clean. Period.

But is that the best we can do?

Underlying these beliefs is the assumption that you will get burned if you look ahead because you are bad at anticipating change or the need for design flexibility. Since you never get it right, why ever bother looking ahead?

Well, who says we are bad at looking ahead? Is it really true?

The fact is we all make design mistakes, whether we look ahead or not. But that doesn’t mean looking ahead is the cause of more mistakes. (Sometimes it even prevents mistakes).

I believe that you can and should look ahead. And that most developers, given half a chance, are pretty good at incorporating past experiences and making anticipatory design choices. We might even say that an “experienced developer” is one who has the ability to use past experiences to help make good forward-looking decisions.

When you refactor code to reduce dependencies you are making an on-the-spot judgment call about what part of your code is more stable and what is more likely to change. You make these decisions based on concrete evidence. You are only speculating a little bit (things keep changing here, so I’ll try this).

Experienced designers developers look ahead at a larger scale. And the longer they’ve been at it, the more they incorporate past experiences into daily looking just a little bit ahead. You learn to anticipate where the design wrinkles will be.

If you’re less experienced, this seems like magic. How can you ever know when you’ve made an appropriate anticipatory design decision when the payoff only comes sometime in the future? Well, you don’t ever know for certain.

But you get better at looking ahead by practicing it in a safe, supportive development environment. One where you are encouraged to look around and look ahead. You get to see what more senior people do. And, if you are lucky, when you ask them why they are doing something they take the time to explain their reasoning.

Looking ahead isn’t antithetical to refactoring. In fact, it might even help you avoid some rework. Over time, we learn to make even more reasonable judgment calls based on experience, wisdom, and gut instinct. (And sometimes people think that we aren’t thinking ahead because we can fluidly make on on-the the-spot design changes).

If you never practice looking ahead, well, you might just be doomed to refactoring your way to a good design. For me, that simply doesn’t cut it. And it can simply be too painful and inefficient for large, complex projects.

Recently, a thoughtful agile coach and architect told me:

“When I start a project, I try to figure out what is the minimum architecture we need to do for that context. I start from a few key things:
What are the requirements in terms of performance, scalability, reliability, security, localization, functionality etc.?

What deployment structure best fits these requirements?

What platforms and frameworks facilitate the development of the application?

What don’t we know how to do at this point? How can we find out?”

Asking these questions helps him, “define a basic but good architecture. One that ensures the right [architectural] features, minimizes time/costs and reduces risks.” That’s just enough upfront speculation to establish a baseline architecture.

But being ever thoughtful, he also asked, “I am wondering however: Should I do more? In what circumstances?”

Under certain circumstances it may be appropriate to do even more upfront design work.

Agile heresy?

I think not.

When we tackle really big unknowns it makes sense to explore more before making costly investments. Experimentation backed up by evidence-based decision making makes our designs better. Looking ahead can save us time.

But still, too many are deceived into believing in those vividly scary look-ahead failure stories. Then they overreact, pulling back from any speculative design.

My advice: Don’t overlook the many times when you have successfully looked ahead and saved time and effort. Not looking ahead is symptomatic of inexperience.

What is Pragmatic Testing All About?

-Rebecca Wirfs-Brock and Joe Yoder

We’re excited to be offering a new course, Pragmatic Test-Driven Development, with a practical bent to testing. Test-driven development (TDD) traditionally means writing unit and acceptance tests along with production code. Some agile thought leaders push a hardnosed test-first practice, insisting that you always write a failing test before you write any production code. Then, in short programming and testing cycles, evolve your test as you complete your code.

One common objection to TDD done this way is that writing tests this way doesn’t seem “natural”. Instead, you may want to write a coherent chunk of code before writing any tests. We agree with that. In our course we explore several different ways you can do Test-Driven Development, making TDD much more approachable to a wider audience.

We believe that you should pay attention to testing and that it should be an integral part of your daily programming practice. But we don’t insist that you always evolve tests along with your code in many short cycles. Some find it more natural to outline some tests first, and use those test scenarios to guide them as they write code. Once they’ve completed a “good enough” implementation that supports the test scenarios, they then write those tests and incrementally fix any bugs as they go. As long as you don’t write hundreds of lines of code without any testing, we think this style is OK too (and we don’t insist on one style over the other).

As long as you check in your production code along with unit tests that pass, your programming is being “driven” by tests.

We believe that it’s not so important to test first as it is to test frequently! Testing should be integral to your daily coding and design rhythms. And your production code should only be checked in along with its passing tests.

There’s a lot more to becoming proficient at TDD. Developing automated test suites, refactoring and reworking tests to eliminate duplication, and testing for exceptional conditions, are just a few. And acceptance tests, smoke tests, integration, performance and load tests support incremental development, too. If all this testing sounds like too much work, well, let’s get practical. Testing shouldn’t be done just for testing’s sake. Instead, the tests you write should give you leverage to confidently change and evolve your code base and validate the requirements of the system.. That’s why it is important to know what to test, what not to test, and when to stop testing.

An Architect’s Dilemna: Should I Rework or Exploit Legacy Architecture?

I recently spoke with an architect has been tuning up a legacy system that is built out of a patchwork quilt of technologies. As a consequence of its age and lack of common design approaches, the system is difficult to maintain. Error and event logs are written (in fact, many are), but they are inconsistent and scattered. It is extremely hard to collect data from and troubleshoot the system when things go wrong.

The architect has instigated many architectural improvements to this system, but one that to me was absolutely brilliant was to not insist that the system be reworked to use a single common logging mechanism. Instead, logs were redirected to a NoSQL database that could then be intelligently queried to troubleshoot problems as they arose.

Rather than dive in and “fix” legacy code to be consistent, this was a “splice and intelligently interpret” solution that had minimal impact on working code. Yet this fairly simple fix made the lives of those troubleshooting the system much easier. No longer did they have to dig through various logs by hand. They could stare and compare a stream of correlated event data.

Early in my career I was often frustrated by discrepancies in systems I worked on. I envisioned a better world where the design conventions were consistently followed. I took pride in cleaning up crufty code. And in the spirit of redesigning for that new, improved world, I’d fix any inconsistencies that were under my control.

At a large scale, my individual clean up efforts would be entirely impractical. Complex software isnâ’t the byproduct of a single mind. Often, it simply isn’t practical to rework large systems make things consistent. It is far easier to spot and fix system warts early in their life than later after myriad cowpaths have been paved and initial good design ideas have become warped and obsfucated. Making significant changes in legacy systems requires skill, tenacity, and courage. But sometimes you can avoid making significant changes if you twist the way you think about the problem.

If your infrastructure causes problems, find ways to fix it. Better yet (and here’s the twist): find ways to avoid or exploit its limitations. Solving a problem by avoiding major rework is equally as rewarding as cleaning up cruft. Even if it leaves a poor design intact. Such fixes breathe life into systems that by all measures should have been scrapped long ago. Fashioning fixes that don’t force the core of a fragile architecture to be revised is a real engineering accomplishment. In an ideal world I’d like time to clean up crufty systems and make them better. But not if I can get significant improvement with far less effort. Engineering, after all, is the art of making intelligent tradeoffs.