Rebecca’s Web – Architects' Reactions to Signals

In Critical Software Redesign: Creating the Environment for Large Scale Change, we presented the story of the TaxGlobe company as a series of signals. We invited you, the reader, to consider how you would react to each signal, as if you were an architect at TaxGlobe. This is of course only an exercise. In reality, the signals wouldn’t be listed in writing, and you’d have to observe them over time, without the benefit of hindsight.

While writing that essay, we asked Avraham Poupko, Xin Yao, and Michael Keeling, three experienced architects, to do the same exercise, and to capture their reactions in writing. We present their comments, as they offer a peek into their brains.

Avraham Poupko

Avraham is the Head of Strategic Planning and Product at Toga Networks.

1/ A customer tells us one of our calculations is wrong. Our tax lawyer looks into it, and gets a developer to fix the bug.

Avraham: These mistakes happen. No big deal. If this happens a second time, I think we need to be more disciplined about the calculations. The third time, I think it’s time for a review of how we create the calculations. The review might uncover the signals. It might show that since developers are not tax experts, errors happen to frequently. It might also be time for a review of several of the more elaborate calculations.

2/ Another customer tells us that a calculation is wrong, but our tax lawyers believe it’s correct. The customer is a bit annoyed about it.

Avraham: Lets keep an eye open on this. If this repeats another once or twice, it might be worth having a discussion of how we go from regulations to calculations. The customer might be wrong, but perhaps it is their prerogative to be wrong.

3/ Another customer asks us to “reinterpret” a calculation because they think the tax law could be interpreted differently, to their advantage.

Avraham: Here I noticed that there is a gap between the tax code and the interpretation of it. We never made the explicit assumption that the regulation and the interpretation are the same. But that assumption was implied. Maybe we need to think about interpretation, as well as who gets to interpret it. In hindsight this should have been a pivot point. This is where we could have realised that our model is missing an essential element.

4/ We add a few customer-specific interpretations of the tax law into the code, using a conditional to switch between the normal calculation and modified one. We’re using hardcoded customer IDs to do the branching.

Avraham: After 4 such customers within a short while – two of whom had contradicting interpretations of the very same rule, we realize that a serious re-refactoring is in order. But we are still refactoring. Not rewriting or remodeling.

5/ Sales catches on that we can do custom interpretations. They’re also aware that it’s pretty easy to branch on customer ID, so they use it in their sales pitch.

Avraham: I wish sales would not do that. They are claiming flexibility that we do not actually have. It sometime feels to me like sales is claiming that our product can help "beat the system." Is this even legal?

6/ As we get more of them, doing these customizations becomes harder. We need to be more careful about not breaking the regular flow. And the code becomes harder to understand.

Avraham: Slow down. We only accept customizations if at least two customers ask for the same customization. And we only release customizations once a quarter.

7/ Management complains that we spend more and more time on each customization, and that lead time is becoming very unpredictable.

Avraham: It dawns on me that this is about more than refactoring. It might be time to change the model. Maybe we need to take interpretation more seriously. I raise this with management and they ask if I actually considered the cost implications of such a change.

8/ Sales is still offering these customizations as a cheap upsell, but in reality we lose money on them as the cost increases.

Avraham: I asked sales to stop doing that until we have had a chance to think about things. Why are they doing that?

9/ The testers are finding it difficult to keep up with testing the customizations.

Avraham: We start investing in testing infrastructure and methods. We end up writing new scripts and invest time in testing the testing. By now it is clear that we are sinking costs into maintaining a bad model. Not all is lost. We can still change the model and re-write major parts of code without it being too painful.

10/ We do a small refactor where a bit of code in our 3000 line getTaxes() method is extracted into a calculateRule12() method. Occasionally more of these small refactors happened, but there was never time to do major untangling of the code. If something worked, it was rarely refactored.

Avraham: Too late. We will never be able to cheaply change our model. Our only hope might be to wait for an opportunity large enough to justify a rewrite or at least a fundamental refactor.

11/ As we expand into more countries, our own tax lawyers ask for more time to learn the intricacies of the country-specific tax rules. They now understand better how much a different interpretation of a rule can impact the total tax cost.

Avraham: This might be the big opportunity we were waiting for. It will be expensive, but it might be justified.

12/ Production for one of our major customers has moved to Brazil, so they want us to handle Brazil tax law in more depth than we do now. But we have difficulty recruiting tax law experts there, and our existing lawyers admit they underestimated the complexity of those laws.

Avraham: The window of opportunity is closing rapidly. This is a signal that we must change the model and change it now.

13/ Eventually, one of the developers builds a small DSL that allows customers to change the interpretations of some of the rules in Brazil. Customers love it and ask us to make the DSL more powerful.

Avraham: It is a good thing this developer did not wait for permission to do this. He should be promoted for doing such a smart thing. Then he should be fired to taking two years to think of this.

14/ Later, the customer asks for a similar DSL for their other locales.

Avraham: Predictable. Time to start preparing the new infrastructure now.

15/ A customer requests a custom engineering effort: They want to recompute taxes for the prior year (based on their own interpretations of several tax rules). They’re willing to pay really good money for it, so we assume they expect to save a lot of money this way.

Avraham: Not sure this is a recurring use case. This might be a one-off that does not justify a new type of product.

16/ Because sales now has learned that we can charge high prices for these large customizations, they are putting more focus on this and the tax savings it can bring.

Avraham: We need to think and very carefully define what parts of customization we let the customer do, and what parts we do. We are not a tax company, we are a software company. The cost of added complexity needs to be very carefully considered

17/ Engineers complain that the current design isn’t very nice to work with. They want to move to a new tech stack.

Avraham: Engineers always want to move to a new stack. Let’s find out if this is a problem with the stack. Or a problem with the architecture of the product.

18/ There’s disagreement in TaxGlobe on whether we should go back to offering a single one-size-fits-all product (and potentially sacrificing a few large customers), or whether we should be fully focused on customized implementations. Right now we get a lot of revenue from custom work, but we also provide a DSL. Both sides of the argument feel that this contradiction amounts to shooting ourselves in the foot. The DSL steals business from custom work, and custom work takes away focus on and incentive for building a truly powerful DSL.

Avraham: This is a discussion that needs to be decided by the business. Here, the role of the architect is to provide leadership with the various options and costs, so that leadership can decide. This is a complex discussion that involves VP R&D, CTO, VP marketing and many others.

19/ People in engineering are advocating that there should be “standard” rules, and rules that are open to customization, just so they can keep things straight.

Avraham: Engineers often wish the world was simpler. But they don’t get to prescribe the way the world is. They must operate within the world. What the engineers can do, is work with domain experts to discover which rules are variable and which rules don’t change and can be standardised.

20/ Customers are reporting more bugs. The support people have difficulty evaluating the bugs, because they don’t understand which rules are being applied at which time.

Avraham: This might not be a model change. This might just be a call for richer debugging and versioning. So that every trace file comes along with details on what DSL was used and what version of the rules.

21/ The market leader for ERP releases a competing product called ShipTax3000. So far we’re not losing customers to them, as it doesn’t have the same amount of customizability. Still, they are gaining market share quickly because of their existing dominance.

Avraham: Customers often like software that reflects their domain model. This might be a great opportunity to capitalise on the fact that we are customisable. This must presented to the customer in a way that shows that our product ‘speaks their language’.

22/ Some customers complain about the performance. It is taking longer to process tax calculations, especially if there are a lot of custom variations — even if that specific customer doesn’t use the variations.

Avraham: This is a place for optimisation. It is not a major model issue. When it comes to tax calculation, performance might not be as important as flexibility and accuracy. I certainly would not let performance come at the expense of other needs.

23/ Customers are still willing to pay well for customized interpretations, but we can’t deliver them fast enough anymore. The customers are on a deadline to submit taxes, and they have no use for interpretations that are already outdated by the time they get them.

Avraham: This can kill us. Any customer that tries out a competitor might never come back. This one requires real strategic thinking. Analysis of our abilities, our customers and our code. An entire workshop or even a semester can be devoted to this item

24/ The customers don’t just want us to hardcode their interpretations any more. What they want, is to be able to simulate the calculations with a different set of interpretations, and only if the interpretations saves them money, they want to commit to them.

Avraham: This is a really neat feature. If we had done the architecture correctly until now, this would not be a major issue

Xin Yao

Xin is an independent consultant, after having been Chief Software Architect at Danske Bank.

1/ A customer tells us one of our calculations is wrong. Our tax lawyer looks into it, and gets a developer to fix the bug.

2/ Another customer tells us that a calculation is wrong, but our tax lawyers believe it’s correct. The customer is a bit annoyed about it.

3/ Another customer asks us to “reinterpret” a calculation because they think the tax law could be interpreted differently, to their advantage.

Xin: The first three are weak signals I would miss catching. No. 3 can raise a flag, but unless I am a 100% dedicated coding architect in the team, I would likely not worry and consider this a “one off”.

Xin: At this point code scanning tools like Sonar would come out with “high cognitive load” or alike. This is a stressor level signal I always watch for. Obviously the code no longer lives up to the “Single Responsibility Principle”. I would then check lines of code for the “calculation” part and the “interpretation part” to assess their relative business weight. And if I am not personally stressed by other tasks, I would advocate for a developer session first to see if we can refactor.

And refactoring effort might lead to discovery missing domain concepts such as “calculation” vs. “interpretation” vs. “customization”.

I’d check if the hardcoded customer IDs part is properly covered by automated tests.

5/ Sales catches on that we can do custom interpretations. They’re also aware that it’s pretty easy to branch on customer ID, so they use it in their sales pitch.

Xin: This part can happen at an architect’s blind angle.

Assuming the team practices Scrum, and I am at their refinement session or sprint planning session, I would likely sense that more and more epics/stories of the custom interpretation nature get prioritized. And developers would mention “Oh, this again, we hardcoded already for three customers, is this way to go?”. But perhaps I am making too many rosy assumptions about a well-functioning agile process.

6/ As we get more of them, doing these customizations becomes harder. We need to be more careful about not breaking the regular flow. And the code becomes harder to understand.

Xin: This is where a healthy developer team (not under stress and multitasking) would start discussing the symptoms. Depending on the company culture, they may attempt to fix it themselves or bring it to an architect’s or manager’s attention. A mature developer team, or an architect, would ask for a cross-functional deep-dive session with the business.

7/ Management complains that we spend more and more time on each customization, and that lead time is becoming very unpredictable.

Xin: Again, a mature and well-paced team would ask for thinking time for “isolating” the effect of code targeting customization. But a stressed or dysfunctional team could be too busy to take action

8/ Sales is still offering these customizations as a cheap upsell, but in reality we lose money on them as the cost increases.

Xin: This can be a lagging metric that only became evident far too late. If this is a hunch of some developer, it could get dismissed.

9/ The testers are finding it difficult to keep up with testing the customizations.

Xin: Sounds like the reality of many dev teams

Xin: Instead of treating it as a smart fix, I would lead an effort to reverse engineer the model inside the DSL. Are there shared abstractions with the developer-coded customizations? Are there conflicting concepts? Are there potential common ground building blocks? A conversation could lead us to a good rabbit hole to experiment with more modeling and design options.

14/ Later, the customer asks for a similar DSL for their other locales.

16/ Because sales now has learned that we can charge high prices for these large customizations, they are putting more focus on this and the tax savings it can bring.

Xin: 15 and 16 are clear leverage points for an engineering leader to argue for in-tandem pivoting of the problem and solution space at once. We have a potentially larger core domain that brings in more bucks. This justifies investment in design, and modernization (no. 17). We have now both a burning technical platform and lucrative product opportunity to leverage. Winning architecture politics could be asking for a design sprint. We could also get green light for a design spike, redefining a product vision, reintegrating isolated world views with e.g. event storming, and hard-core domain remodeling for high cohesion and low coupling.

17/ Engineers complain that the current design isn’t very nice to work with. They want to move to a new tech stack.

Xin: In my experience, crossroad like this is a wonderful opportunity for refocusing energy. Unfortunately, the reality can also be that the customization and DSL product teams have each evolved into a fiefdom by itself, playing organizational politics against each other. Even that could lead to positive pivoting of separating them from each other, leaving two code bases with duplicate logic, but potentially more understandable and maintainable individually. The crucial thing here is for some tough questions to be asked across sociotechnical system boundaries. Eventually if it hurts sufficiently much, something will happen. But if it is an intellectual discussion of optimizing logic or code base, maybe with me as the boundary-spanning architect shouting for change, I am not giving myself high odds of success.

19/ People in engineering are advocating that there should be “standard” rules, and rules that are open to customization, just so they can keep things straight.

20/ Customers are reporting more bugs. The support people have difficulty evaluating the bugs, because they don’t understand which rules are being applied at which time.

Xin: Did we survive until now? Miraculous. If the customers are so articulate about what they want, even the most insensitive executive floor would start formulating a new product strategy. Let’s hope it is not too late.

Xin’s reflection on this exercise:

Somewhere along this list of signals, there is a shift of context from "complicated" to "complex." A discerning architect or engineering leader needs to be able to sense this. While commenting numerically one by one, I could easily follow my inclination to solve the initial signals as an analytical problem ("complicated" - sense, analyze, respond). Gradually I approached the customization and DSL dilemma which is clearly a matter of unpredictability, that only experimentation and emergence could possibly solve. There I got frustrated because I couldn't come up with an "expert answer," so for a brief moment I was falling into the complacency trap of attempting to solve it like a complicated problem. But here is where the product modeling and software modeling need to merge organically into each other, a bit of the liminal struggle.

Michael Keeling

Michael is Staff Software Engineer at Kiavi, and author of “Design It! from Programmer to Software Architect”.

1/ A customer tells us one of our calculations is wrong. Our tax lawyer looks into it, and gets a developer to fix the bug.

Michael: Let's fix the bug.

2/ Another customer tells us that a calculation is wrong, but our tax lawyers believe it’s correct. The customer is a bit annoyed about it.

Michael: Interesting... so these things can be interpreted in different ways. How do we define correctness? How often does our understanding change? Keep an eye out for trends of knowledge / abstraction gaps.

3/ Another customer asks us to “reinterpret” a calculation because they think the tax law could be interpreted differently, to their advantage.

Michael: This is pointing to a gap in our understanding of the domain. Third bug in however much time that points to a lack of malleability for this code. Are we solving the right problem?

Michael: OK, let's run a quick and dirty experiment to unblock sales and start gaining some experience. In this case I think this could go to production, if it can be done cheap/fast enough and is reversible.

5/ Sales catches on that we can do custom interpretations. They’re also aware that it’s pretty easy to branch on customer ID, so they use it in their sales pitch.

Michael: Uh oh... maybe we shouldn't have put that experiment in production. Time to course correct, this product-line-per-customer approach isn't going to hold.

6/ As we get more of them, doing these customizations becomes harder. We need to be more careful about not breaking the regular flow. And the code becomes harder to understand.

Michael: Woof, it's too late. We screwed up. We need to figure out how to dig out. Are there other underlying cultural problems that got us to this point? Why did we continue branching? Start building a case for a small crew to explore this problem and figure out some solutions.

7/ Management complains that we spend more and more time on each customization, and that lead time is becoming very unpredictable.

Michael: Time to reduce WIP (editor’s note: Work In Progress, a metric for the amount of tasks being worked on concurrently) and increase focus on getting things done, not just doing things. How can we protect the people who are exploring while fighting these fires?

8/ Sales is still offering these customizations as a cheap upsell, but in reality we lose money on them as the cost increases.

Michael: Time to pull the plug and circle the wagons. This isn't working. We're at a point where we have enough data to justify a pivot and investment — assuming we have a solution to pitch...

9/ The testers are finding it difficult to keep up with testing the customizations.

Michael: How did we reach this point? The writing was on the wall! Since this is where we are, it seems like we've got to buckle up and fight the fires. What are the testers testing? What are they finding? Let's try using this quality data to increase testing responsibilities on the dev teams and reduce escaped defect to Testing.

Michael: How much work is required? How much time might we need? Let's get an estimate and figure out how to make the time for this work. Focusing on refactoring — this is good. How can we amplify this good? Let's create space for somebody to try a spike redesign here.

Michael: This tells me we are about to stumble into a nest of change and the whole world is about to get different for us. But also, this could be the break we desperately need. If things are about to fundamentally change and if business stakeholders are about to really understand this, it means we may have allies to support a redesign.

Michael: We have to design for change and release frequently. Focus on reducing time to change and quick feedback loops. We're not going to rewrite the system. I haven't seen anything here to imply this can't be salvaged, but can we use Brazil as the motivating example we need to figure out how to fix the rest of the system? Idea: what if we fork the system and allow the Brazil fork to be more bold in its changes? We'd need to consolidate at some point, but we need to make some room for change here.

Michael: Cool, rapid feedback loops. OK.

14/ Later, the customer asks for a similar DSL for their other locales.

Michael: "Consolidation comes later" is here... We already have a working example. What is the most difficult locale? What would it take to extend the DSL to that locale? I have a hunch that we'll need a v2 of the DSL. We need to test what we've got and figure out how much more work might be needed. Also, are we solving the right problem, before we get too far along down this solution?

Michael: The market seems to be telling us we're in the wrong business.

16/ Because sales now has learned that we can charge high prices for these large customizations, they are putting more focus on this and the tax savings it can bring.

Michael: ...and the market responds

17/ Engineers complain that the current design isn’t very nice to work with. They want to move to a new tech stack.

Michael: I'm sensing tensions between needs and capabilities. Let's explore what this would look like. Find space for a couple of spikes and start building a development plan. Further exploration — is the design the problem or are there other obstacles for which moving to the new tech stack are one possible solution?

Michael: Woof. When the business is divided then nobody wins. My take is that there's an integrated/whole solution in which we drive down costs through a customisable solution while pricing customization at market rates. This sounds like a situation in which there might be a win-win but not everyone has all the information and we need to explore some new scenarios.

19/ People in engineering are advocating that there should be “standard” rules, and rules that are open to customization, just so they can keep things straight.

Michael: This is likely coming from a place of pain. What problems are they experiencing? What is the root cause? If we have the time, this is worth a quick exploration.

20/ Customers are reporting more bugs. The support people have difficulty evaluating the bugs, because they don’t understand which rules are being applied at which time.

Michael: Firefighting mode again. We need to gain some experience to define the problem so we can solve it.

Michael: Here it comes...

Michael: It seems like we need a short term solution to relieve some of these pressures. If we do this right we can create space to figure out how to address root cause problems. What are the options? How do they compare to one another? How much effort and time is needed to truly address these problems?

Michael: We're toast! This is telling us that speed of change is an important system quality. If we couldn't make a case before to switch to a more changeable solution, then this as clear a signal as there ever will be. How quickly do customers want change? How fast can we make those changes with the current approach vs. alternatives? What are the alternatives?

Michael: Time for some hard decisions... There's a lot to explore now. New business models? How does our software fit those business models? What can we salvage? One software system? Multiple? Both design and operational questions.

Michael’s reflections on this exercise:

I seemed to use the word "exploration" over "experiment" naturally, but I think the general concept is approximately the same as what was described in the essay as an experiment.

Sometimes asking the right questions is the key.

As tensions increased, my musings and focus seemed to naturally shift from asking questions that need to be explored to figuring out (running experiments?) for creating space to answer these other questions.

There seems to be a natural progression:

Early signals make it easy to identify questions to explore / experiment.
Without change, tension increases and I sense that it becomes increasingly challenging to have the space to run experiments.
Without further change, we enter a firefighting stage in which the team is responding. I assume there is no space for experimentation in this situation so all effort goes into returning to stability and creating space to resolve the immediate tensions.

I clearly could sense something, the signals in the case. I used words like "sense" and "telling." Much of my musings only captured the reaction — the actions I would take in response to a signal without clearly articulating the signal itself. This seems interesting. Knowing "how you know" and clearly describing it is rather difficult. Taking shortcuts is so much easier.

Sometimes my reaction to a signal was doing something — course correcting, based on past experiences. Other times it was exploring to figure out what to do — an experiment.