What is Pragmatic Testing All About?

-Rebecca Wirfs-Brock and Joe Yoder

We’re excited to be offering a new course, Pragmatic Test-Driven Development, with a practical bent to testing. Test-driven development (TDD) traditionally means writing unit and acceptance tests along with production code. Some agile thought leaders push a hardnosed test-first practice, insisting that you always write a failing test before you write any production code. Then, in short programming and testing cycles, evolve your test as you complete your code.

One common objection to TDD done this way is that writing tests this way doesn’t seem “natural.” Instead, you may want to write a coherent chunk of code before writing any tests. We agree with that. In our course we explore several different ways you can do Test-Driven Development, making TDD much more approachable to a wider audience.

We believe that you should pay attention to testing and that it should be an integral part of your daily programming practice. But we don’t insist that you always evolve tests along with your code in many short cycles. Some find it more natural to outline some tests first, and use those test scenarios to guide them as they write code. Once they’ve completed a “good enough” implementation that supports the test scenarios, they then write those tests and incrementally fix any bugs as they go. As long as you don’t write hundreds of lines of code without any testing, we think this style is OK too (and we don’t insist on one style over the other).

As long as you check in your production code along with unit tests that pass, your programming is being “driven” by tests.

We believe that it’s not so important to test first as it is to test frequently! Testing should be integral to your daily coding and design rhythms. And your production code should only be checked in along with its passing tests.

There’s a lot more to becoming proficient at TDD. Developing automated test suites, refactoring and reworking tests to eliminate duplication, and testing for exceptional conditions, are just a few. And acceptance tests, smoke tests, integration, performance and load tests support incremental development, too. If all this testing sounds like too much work, well…let’s get practical. Testing shouldn’t be done just for testing’s sake. Instead, the tests you write should give you leverage to confidently change and evolve your code base and validate the requirements of the system.. That’s why it is important to know what to test, what not to test, and when to stop testing.

Posted in Uncategorized | Leave a comment

An Architect’s Dilemna: Should I Rework or Exploit Legacy Architecture?

I recently spoke with an architect has been tuning up a legacy system that is built out of a patchwork quilt of technologies. As a consequence of its age and lack of common design approaches, the system is difficult to maintain. Error and event logs are written (in fact, many are), but they are inconsistent and scattered. It is extremely hard to collect data from and troubleshoot the system when things go wrong.

The architect has instigated many architectural improvements to this system, but one that to me was absolutely brilliant was to not insist that the system be reworked to use a single common logging mechanism. Instead, logs were redirected to a NoSQL database that could then be intelligently queried to troubleshoot problems as they arose.

Rather than dive in and “fix” legacy code to be consistent, this was a “splice and intelligently interpret” solution that had minimal impact on working code. Yet this fairly simple fix made the lives of those troubleshooting the system much easier. No longer did they have to dig through various logs by hand. They could stare and compare a stream of correlated event data.

Early in my career I was often frustrated by discrepancies in systems I worked on. I envisioned a better world where the design conventions were consistently followed. I took pride in cleaning up crufty code. And in the spirit of redesigning for that new, improved world, I’d fix any inconsistencies that were under my control.

At a large scale, my individual clean up efforts would be entirely impractical. Complex software isn’t the byproduct of a single mind. Often, it simply isn’t practical to rework large systems make things consistent. It is far easier to spot and fix system warts early in their life than later after myriad cowpaths have been paved and initial good design ideas have become warped and obsfucated. Making significant changes in legacy systems requires skill, tenacity, and courage. But sometimes you can avoid making significant changes if you twist the way you think about the problem.

If your infrastructure causes problems, find ways to fix it. Better yet (and here’s the twist): find ways to avoid or exploit its limitations. Solving a problem by avoiding major rework is equally as rewarding as cleaning up cruft. Even if it leaves a poor design intact. Such fixes breathe life into systems that by all measures should have been scrapped long ago. Fashioning fixes that don’t force the core of a fragile architecture to be revised is a real engineering accomplishment. In an ideal world I’d like time to clean up crufty systems and make them better. But not if I can get significant improvement with far less effort. Engineering, after all, is the art of making intelligent tradeoffs.

Posted in Software Architecture, Software Design | 2 Comments

Agile Architecture Myths #4 Because you are agile you can change your system fast!

Agile designers embrace change. But that doesn’t mean change is always easy. Some things are harder to change than others. So it is good to know how to explain this to impatient product stakeholders, program managers, or product owners when they ask you to handle a new requirement that to them appears to be easy but isn’t.

Joe Yoder and Brian Foote, of the Big Ball of Mud fame, provide insights into ways systems can change without too much friction. They drew inspiration from Stuart Brand’s How Buildings Learn. Brand explains that buildings are made of components organized into shearing layers. He identifies six layers: the site, the structure, the skin, the services, the space plan, and physical stuff in the building.

Each shearing layer has its own value and speed of change, or pace. According to Brand, buildings are able to adapt because faster changing layers (e.g. the services layers and spaces) are purposefully designed so to not be obstructed by slower changing layers. If you design your building well, it is fairly easy to change the plumbing. Much easier than revising the foundation. And it is even easier to rearrange the furniture. Sometimes designers go to extra efforts to make a component easier to change. For example, most conference centers are designed so that sliding panels form walls that allow inside space to be quickly modified.

Brand’s ideas shouldn’t be surprising to software developers who follow good design practices that enable us to adapt our software: keep systems modular, remove unnecessary dependencies between components, and hide implementation details behind stable interfaces.

Foote and Yoder’s advice for avoiding tangled, hard-to-change software is to “Factor your system so that artifacts that change at similar rates are together.” They also present a chart of typical layers in a software system and their rates of change:

Frequently, we are asked to support new functionality that requires us to make changes deep in our system. We are asked to tinker with the underlying (supposedly slower changing) layers that the rest of our software relies upon. And often, we do achieve this miraculous feat of engineering because interfaces between layers were stable and performed adequately. We got away with tinkering with the foundations without serious disruption. But sometimes we aren’t so lucky. A new requirement might demand significantly more capabilities of our underlying layers. These types of changes require significant architectural rework. And no matter how matter how agile we are, major rework requires more effort.

Because we are agile, we recognize that change is inevitable. But embracing change doesn’t make it easier—just expected. I’d be interested in hearing your thoughts about Foote and Yoder’s shearing layers and ways you’ve found to ease the pain of making significant software changes.

Posted in Agile, Requirements, Software Architecture, Software Design, Uncategorized | Leave a comment

Re-thinking Thinking and Planning

In the tutorial, Hooray We’re Agile Testers! What’s Next?, Janet Gregory apologized a couple of times for saying upfront thinking or planning. I know Janet wanted to let the audience know that she isn’t a fan of massive test plans or documents written way ahead…But her remarks got me wondering. Why in the agile community is it a taboo to recommend or admit to doing any upfront thinking or planning?

When you incrementally build production code and tests you do come to a deeper understanding about your software’s capabilities and what your stakeholders want. As a consequence, if you are thoughtful and reactive, it’s natural to adjust and adapt to feedback. But it’s also natural to do some upfront thinking [there, I went and said “upfront thinking”, not just thinking or speculation, and I was cringing ever so slightly as I wrote those words] before expending a lot of time and effort. Sometimes you need to think about and discuss what you should be doing so you don’t waste time doing the wrong things.

As someone who embraces agile values, I expect to readjust my ideas and plans as I learn more. I get it that too much upfront anything results in much wasted effort. But there’s a distinction I’d like to make between too much and enough thinking and preparation.

If you have an agile mindset, you recognize that plans have limits. You let go of any illusion that you’re in control of your destiny simply because you have a plan. You are open to change. But being responsive to change doesn’t obviate the benefits of planning. Especially if your project has to mesh together the work of several teams.

I’m tired of having to apologize for upfront thinking, effort expended in creating a project or product roadmap, defining an initial product landing zones, or exploring options. Give thinking a chance. And find the right balance.

Posted in Agile | Leave a comment

Who Defines (or Redefines) Landing Zone Criteria?

Who should be in on discussions that set landing zone criteria? Because most landing zone have architectural implications, someone knowledgeable about the system architecture, in addition to the product owner and other key stakeholders should have a lot to say in vetting a landing zone.

Someone who has depth, breadth, and vision, is an ideal candidate for crafting an initial cut. But even if you are brilliant, I suggest you fine-tune your landing zone with a small, informed group. If you have lots of stakeholders who want to chime in, give each stakeholder group a voice in identifying qualities and values they find particularly relevant. And ask a representative from each stakeholder group to join in on a landing zone discussion. At a landing zone review, expect healthy discussion. Experts are usually highly opinionated as well as passionate.

You might even want to facilitate your discussions.

I find it much more effective to have an informed facilitator guide landing zone discussions, than a dispassionate, uniformed professional facilitator. An ideal landing zone meeting facilitator should know about the program or product but need not be the “authority” or definitive “expert”. It’s more important that they know the landscape and they are good at gaining consensus and getting the best out of individuals who hold strong opinions. Possibilities: chief business architects, quality leads, the program or product manager, yes, even a software architect.

Sometimes a facilitator needs to step out of that role and offer informed opinions. I find this highly desirable, as long as this shift is made clear. “Hang on, do you mind if I take a stab at explaining what I think are more reasonable targets?”

Minimum, target and acceptable values should be agreed upon by the group and it might take some discussion to reach mutual understanding and consensus. For example, someone might initially propose a set of landing zone values based on historical trends and extrapolation. The software architect could push back with values based on prototyping experiments and new benchmark data. The group might end up adjusting targets because that evidence was compelling. Or…they might agree on tentative values that need to be firmed by an expert. Hammering out numbers just to finish the landing zone isn’t the goal. Instead, you want to shape ideas for what you think will make your product a success based on the best evidence you have, backed up by experience and tempered by group wisdom. To effectively do this, people need to come to the discussion with mutual respect, trust and no hidden agendas.

And if you are agile, recognize that your landing zone can and should recalibrated once you learn more about what’s possible.

Posted in Agile, Requirements, Software Architecture | Tagged , | Leave a comment

Landing Zone Targets: Precision, Specificity, and Wiggle Room

A landing zone is a set of criteria used to monitor and characterize the “releasability” of a product. Landing zones allow you to take product features and system qualities and trade them off against each other to determine what an acceptable product has to be. Almost always these tradeoffs have architectural implications. If you’ve done something similar in the past, the criteria you should use to define your landing zone may be obvious. But for first time landing zone builders, I recommend you task someone who knows about the product to take a first cut at establishing landing zone criteria that is then reviewed and vetted by a small, informed group.

A business architect, product owner, or lead engineer might prepare a “proposed landing zone” of reasonable values for landing zone criteria that are questioned, challenged, and then reviewed by a small group. On one program I was involved with, the chief business architect made this initial cut. He was a former techno geek who knew his technical limits. More important, he had deep business knowledge, product vision, and had a keen sense about where to be precise and where there should be a lot of flexibility in the landing zone values.

Some transaction criteria were very precise. Since they were in the business of processing a lot of transactions, they knew their past and knew were they needed to improve (based on projected increases in transaction volumes). For example, that transaction throughput target for a particular business process was based on extrapolations from the existing implementation (taking into account the new architecture and system deployment capabilities). This is a purposefully obfuscated example:

Example Landing Zone Attribute
Characteristic Minimum Target Outstanding
Payment processing transactions per day 3,250,000 4,000,000 5,500,000

Some targets for explicit user tasks were very specific (one had a target of less than 4 hours with no errors, and an outstanding goal of 1 business day). On the other hand, many other landing zone criteria were only generally categorized as requiring either a patch, a new system release, or online update support. The definitions for what was a patch, a release or an online update were nailed down so that there was no ambiguity in what they meant.

For example, a patch was defined as a localized solution that took a month or less to implement and deploy. The goal was eventually to get closer to a week than a month, but they started out modestly. On the other hand, a release required coordination among several teams and an entire system redeployment. An online update was something a user could accomplish via an appropriate tool.

So, for example, the landing zone criteria for reconfiguring a workflow associated with a specific data update stream had minimal and target values of “release” and an outstanding value of “online update”.

When defining a landing zone for an agile product or program, carefully consider how precise you need to be and how many criteria are in your zone. Less precision allows for more wiggle room. Without enough constraints, however, it’s hard to know what is good enough. The more precise landing zone criteria are, the easier it is to tell whether you are on track to meet them. But if those landing zone criteria are too narrowly defined, there’s a danger of ignoring broader architecture and design concerns in order to focus only on specifically achieving targets.

We live in a world where there needs to be a balance. I’ll write more about who might be best suited to defining and redefining landing zones in another post.

Posted in Agile, Requirements, Software Architecture, Software Design | Tagged , | Leave a comment

Agile Landing Zones

In my last post I introduced the idea of landing zones, a set of criteria used to monitor and characterize the “releasability” of a product. A landing zone contains system attributes that must be achieved to declare “success”, along with their minimally acceptable, ideal target, and outstanding values.

If you’ve unfamiliar with landing zones, one concern you might have is the size of a landing zone. How many discrete system characteristics can you reasonably expect to track? At first blush, you might expect that the more complex a product or system, the more attributes should be tracked.

But an agile mindset leads me to a different answer: You don’t want to keep your eye on too many dials and gauges. It’s simply too difficult to monitor that many constraints for most projects. Yet you don’t want to have too skimpy of a landing zone either. A handful of disconnected system attributes isn’t worth much. You want to capture the essential success criteria in your landing zone. Things that are both important and measurable.

If you have more than a few attributes, it can be helpful to organize them according to category: e.g. cost, performance, data quality, reliability, usability, etc.). One large multi-year program I know of had a separate landing zone for each functional area (which was comprised of dozens of applications and services). In each functional area, dozens of individual landing zone characteristics were tracked.

Establishing and monitoring landing zone for a phased-release or multi-year product or program can be a big challenge. How can you establish accurate target values for what’s going to be delivered two years from now? You may not be able to. But you do your best given what you know now and what you can reasonably project for the near future.

Agile teams believe it is better to plan to replan than to over specify the future. If you embrace agile values you need to get comfortable with periodically re-planning and readjusting your targets. You should strive to firmly nail down parts of a landing zone that you expect to achieve over the next few months, leaving the rest of your landing zone purposefully sketchy.

And when new evidence comes to light, you likely will need to readjust your landing zone. What initially appeared to be achievable or reasonable targets can shift in light of new facts or market changes. No one wants to deliver yesterday’s product to today’s market. Landing zones, like release criteria can and do change. When they do, people have to make informed decisions to readjust targets.

For instance, you may have worked hard to meet some early achieved landing zone targets, only to find out that your early decisions had negative consequences on future work. You may have created some technical debt that either needs to be paid off (with interest) in order to achieve your next targets. But given time or budget constraints, you may decide to recalibrate your landing zone (and set expectations lower).

A landing zone is rarely cast in concrete. Things that can wiggle around and get more defined over time should be allowed to do so. But as your system architecture emerges and requirements inevitably change…it still helps to have concrete targets to shoot for. And the more precise the landing zone criteria are, the easier they are to design for and verify. I’ll say more on that in my next post.

Posted in Agile, Requirements | Leave a comment

Introducing Landing Zones

On an aircraft carrier, the landing zone describes a small section of deck that a pilot must touch down in to land the plane safely. By analogy, a landing zone for a product describes a range of measurable attributes that your product must deliver to achieve the product vision.

Landing zones are useful for products as well as complex projects and programs. For such complex systems it can be difficult to define “good enough to ship” without considering a lot of different factors and making tradeoffs between them. Recently I have been introducing landing zones as one technique for getting a “bigger picture” on agile projects and programs. I also find landing zones helpful in identifying architecture and design risks and potential required innovations.

I first learned about landing zones from Erik Simmons. Erik is s responsible for creating, implementing, and spreading requirements engineering practices at Intel. Long ago, he and his colleagues taught requirements classes at Oregon Graduate Institute. That’s where I met Erik and learned about landing zones. Since then I’ve helped several clients use them for managing complex programs and projects. They are particularly useful when many requirements needed to be considered and it is important not to lose sight of make or break requirements.

Online information about product or program landing zones is scarce (the only other references I found were a brief glossary definition in Tom Gilb’s Competitive Engineering: A Handbook for Systems Engineering, Requirements Engineering, and Software Engineering Using Planguage.

At first glance a landing zone seems nothing more than a glorified table. Each row in the landing zone represents a measurable requirement. Each requirement has a range of acceptable values labeled Minimum, Target, and Outstanding. The goal is to have each requirement within this range at the end of development. Inside the range is the desired value, labeled Target. Minimum, Target, and Outstanding are relative to your budget and timeframe.

Here’s an example of a landing zone for a loan processing system (all the examples I am using are concocted for simplicity’s sake…any relation to landing zones for real projects is coincidental):

Landing Zone for a Loan System
Attribute Minimum Target Outstanding
Adding new loan agreement 2 weeks 24 hours 12 hours
Add new product 3 weeks 2 weeks 1 week
Adjust loan terms 4 days 2 days 1 day
Access loan risk 1 day 6 hours 10 minutes
Assign loan servicer 1 month 1 week 1 day

And another for a smart phone (this one was cooked up by comparing competitive benchmark data):

Landing Zone for Smart Phone
Attribute Minimum Target Outstanding
Battery life – standby 300 hours 320 hours 420 hours
Battery life – in use 270 minutes 300 minutes 380 minutes
Footprint in inches 2.5 x 4.8 x .57 2.4 x 4.6 x .4 2.31 x 4.5 x .37
Screen size (pixels) 600 x 400 600 x 400 960 x 640
Digital camera resolution 8 MP 8 MP 9 MP
Weight 5 oz. 4.8 oz. 4.4 oz.

A landing zone is similar to release criteria, except it allows for tolerances in acceptable values. There isn’t one number you are aiming for; you have a range of values for each product attribute or characteristic you are targeting. This gives you some flexibility in defining what’s “good enough.” You’ll note on the smart phone that the minimum acceptable screen size is exactly the same as the target. This is not uncommon. It just means that there is no margin between your target and what is minimally acceptable. Sometimes the variance between minimum, target and outstanding can be small (this is when you know what your target is and how to achieve it, and are willing to accept only marginally less).

When you have little wiggle room in meeting requirements, you might simply want to define acceptance criteria with hard and fast numbers that simply must be met. I find it helpful to define landing zones for those product attributes that have some degree of flexibility in their outcome.

I like the way landing zones can help bring focus to a lot of complexity. If you are building something really big, you can roll up your product’s success to a few dozen things to monitor (instead of hundreds). In contrast to a list of release criteria, a landing zone also allows you to see a bigger picture and make sense of it: When one attribute is edging below its minimum, what is happening with the others? Are they trending below minimum, too? If so, you have a big problem with achieving your overall product goals. No, and you have a landing zone which allows you to achieve a successful product/system launch even if every requirement isn’t exactly on target.

Most important, landing zones allow you to make tradeoffs in multiple dimensions. The art is in understanding the tolerance for those attributes that define your landing zone, and then selecting reasonable values for minimum, target and outstanding.
If you are defining a landing zone for a new system or product, it may require you to do some research, experimentation, and prototyping to determine appropriate attributes and their values. If you are replacing an existing system, you probably know what capabilities need to be improved (and your minimum values are likely at least as good as the current system you are replacing).

If you have competitors, you will most likely benchmark their products as part of investigating what you can reasonably expect to achieve. For inspiration, look at comparative product reviews. For example see http://cell-phones.toptenreviews.com/smartphones/. Simple criteria, if met, are checked; other cells have explicit values.

Yet what’s good enough? A good landing zone allows for some flexibility in meeting goals without forcing you to accept unreasonable compromises. If all your landing zone attributes fall in the minimum acceptable category, do you still have a viable product? By definition, yes, your product is minimally acceptable. But that doesn’t guarantee its success. It means you have “landed”. You didn’t miss the aircraft carrier, but you are perilously close to the edge. You’d like to be in the target zone for most of the attributes.

In my next post I’ll write about using landing zones on agile projects and how architects can and should be involved in defining, vetting, and recalibrating landing zones.
If you’ve had experiences with landing zones I’d like to hear from you.

Posted in Agile, Requirements, Software Architecture | Leave a comment

Software Architecture Stewardship

On agile teams, architects do more than design and implement the interesting tricky bits. They typically balance a wide range of concerns: short-term goals, overall system design integrity, risks versus efforts, design expediency…

The successful agile architects I know aren’t ivory tower experts.

They take a leadership role in defining how the system is structured, organized and implemented, as well as how it evolves. They make sure there isn’t a hairball of component dependencies. They are hands-on and engaged in day-to-day development work. They actively ensure that the system is designed in a way that will sustain ongoing, incremental development. They are comfortable writing and refactoring code and figuring out how to fix things and improve the architecture. When things are “broken”, they often step in to help.

An example of this that stands out for me is the story an architect told of how he helped improve the performance of an underperforming website. It was way too slow, and doomed to be even slower unless they reworked the architecture. Over the period of two weeks he worked with a small team to analyze the performance and then refactor the design. He wasn’t a superhero, he just applied his know how, working with the team who knew the deep technical bits. He succeeded in turning this design around because he knew they could improve things once they measured performance, found out where the real bottlenecks were and worked to clean them up (using good design principles and practices). After finding out where bottlenecks were, they first restructured some of the JavaScript code to eliminate several extraneous trips to the server. Then, they cleaned up a couple of service interfaces, cached some data on the service, and eliminated some of the more complex queries. Same functionality; only now the website performed up to snuff. He didn’t view himself as a superhero; just as someone who did what was needed get things back on track. He believed that they could clean things up if he spurred this activity with the right mindset and a fresh pair of eyes…and they did. (Not every attempt at architecture rework has such immediate payoffs….I recognize this, yet still admire his attitude and skill).

The definition of a steward is “someone who manages another’s property or financial affairs; one who administers anything as the agent of another or others”. I like this definition. To me, stewardship has deep implications about the role an architect ideally plays on a team. It means you pay attention, take care, and are responsible for the creation and upkeep of the software. You are responsible for safekeeping the architecture, yet you don’t own the architecture. Sure, you may work out key design details of the architecture, but you are a team player. The system’s success and sustainable development is more important than your own individual technical contributions. While being a technical leader, you also value teamwork. You don’t expect to be the only one coding or designing the challenging or interesting parts. You do what you can each and every day to sustain the system’s architecture.

Posted in Agile, Software Architecture | Leave a comment

Agile Architecture Myths #3 Good Architecture Emerges

Last time I left the cap off of the toothpaste, a small blob of toothpaste flowed onto the counter. No planning; it just emerged.

Now I know that emergent software architecture is another thing entirely. We can’t anticipate everything about our software’s architecture before we start building it. It is impossible to get design “right” the first or the second or the third time. That’s why it is so important to iterate. Yet I don’t like to think that good software architecture simply emerges. It’s a bit more complicated than that.

Several synonyms for emerge leave me feeling uncomfortable. I’d rather not have my architecture “materialize”, “loom” or just “crop up”! Emergent behaviors are a reality with complex software. But emergence doesn’t equate to goodness, no matter how hard you wish it did.

Yet I’m not a fan of big upfront architecture theoretical architecture, either. Emergent behavior can be an opportunity. I like the sense of learning and anticipation in these synonyms: “become known”, “become visible” and “come into view”.
The architecture of complex systems simply has to unfold. We make architecturally relevant decisions based on what we know at any given point in time. And when we don’t know how to proceed we experiment (don’t tell your boss that, call it a design spike instead).

Our architectural ideas should change as we write more code and build more of a system. The many micro-decisions we make along the way should lead us to change our minds and our architecture. We shouldn’t have to live forever with bad choices. That’s the beauty of iterative and agile practices. We can fix and repair things. So this is my way of thinking about how good software architecture comes into being:

Good architecture doesn’t emerge; it evolves.

It’s deceptive to say, “good architecture emerges.” I find that good architecture rarely “emerges.” We aren’t magicians who “materialize” good architecture. Good architecture involves hard work, reflection, and paying attention to details. Ideas for architectural improvements emerge from coding. And as long as you have the skills and chops and know-how to make significant changes when they are needed you can improve your architecture. It takes skill to keep your software’s architecture on a good path. Refactoring and redesign is essential. Only when you have the courage (and permission) to refactor and rework your design will your architecture keep pace with your software

Posted in Agile, Software Architecture, Uncategorized | 5 Comments