In this episode, Andrew Harmel-Law, joined by Andrea Magnorsky and Kenny (Baas) Schwegler, discusses “ghost decisions,” which are fundamental architectural choices that are often undocumented, implicit, or even forgotten. These decisions can cast a long shadow, influencing everything from technology choices to team structures.
Key Takeaways
- Implicit Decisions: Andrew shares his experience with projects where he found that the most fundamental architectural decisions had already been made, often implicitly or as a result of legacy choices. These are often not explicitly architectural decisions, but rather things like the product being built or the team structure, which have a significant architectural impact.
- Reverse-Engineering ADRs: To address these “ghost decisions,” one team Andrew worked with began reverse-engineering Architecture Decision Records (ADRs). They documented historical decisions, including the three different options they selected and the reasons why they were rejected at the time. This provided clarity on the original constraints and allowed the team to revisit decisions when the context had changed, such as a startup growing into a large scale-up.
- Documenting Disagreements: The episode also touches on the challenge of documenting the human factors behind decisions, such as conflicts and power imbalances. Andrew suggests using phrases like “
- adopted despite” or “rejected despite” in ADRs to acknowledge opposing viewpoints and ensure all perspectives are represented in the official documentation. This approach can help people feel acknowledged and provides valuable context for new team members.
- The “Telephone Game”: Kenny highlights the dangers of relying on undocumented decisions, which can lead to a “telephone game” where information is distorted or lost as people join and leave the team. Without a clear record, it becomes difficult to understand why certain choices were made, making it harder to evolve the codebase.
Transcript
hello and welcome to virtual TDD we are doing
2
:today a stories on facilitating
software design and architecture.
3
:In this episode, the third one
of our series, we are with,
4
:Kenny, Andrew, and myself.
5
:We're gonna start today with
Andrew telling us a story about
6
:how they have been facilitating
software design and architecture.
7
:Over to you, Andrew.
8
:Andrew Harmel-Law: Thanks
Andrea, and thanks for, having me
9
:Kenny as well, and both of you.
10
:It's super cool.
11
:I did manage to join the first one,
but it's super cool to join this one.
12
:The story that I wanted to share is kind
of, it's an amalgam of lots of different
13
:experiences I've had with facilitating
software architecture and design.
14
:Um,
15
:so if people recognize different bits
of this, 'cause I'm a consultant,
16
:and I work for ThoughtWorks.
17
:people recognize different bits
of this, that's because I see this
18
:kind of thing over and over again.
19
:But the thing I wanted to talk about and
kind of maybe get your thoughts on is.
20
:Every time I've, I think almost
every single time I've started a
21
:software project, come on board.
22
:Even if I come on board very early at
the beginning, I'm always trying to
23
:find like the first or the, the most
fundamental architectural decisions and
24
:trying to figure out what those are.
25
:'cause they've happened already.
26
:Right?
27
:And maybe they were implicit or maybe
they were, just like legacy decisions
28
:from like, the history of, why the
project got to the point it's got to.
29
:And what's interesting to me is the
fact that those decisions which are
30
:kind of fundamental or underpinning
or very early kind of stage decisions,
31
:typically have a very long shadow and
they have a very long effect on things.
32
:And what's most interesting
to me is the fact that don't.
33
:It, it's, it's not even like everybody
knows they're there and people don't even
34
:acknowledge the fact that they exist.
35
:one example, I went to one place and,
I've been to multiple places, but for
36
:the benefits of this story, I went
to a client and they were working on
37
:a rewrite and a rebuild of things.
38
:And so therefore it was
a rewrite and a rebuild.
39
:So everyone was kind of treating it
as if it was greenfield and if it
40
:was completely fresh, things which
kind of touched and informed the
41
:fundamentals of what we were doing.
42
:so those kind of things were almost like
non architectural decisions, but decisions
43
:which have an architectural impact.
44
:So for example, when people were picking
like, what's the thing we're gonna build?
45
:Or what's the product
we're gonna build, right?
46
:That kind of frames the landscape
and within which you're doing,
47
:and it kind of dictates where
the edges and the boundaries and
48
:the scope of your landscape is.
49
:typically when I join,
someone's already figured out.
50
:How many people you'll need in
teams and what teams you'll need
51
:and all these things kind of Right.
52
:And again, like as, like, I
don't need to convince anyone
53
:that, that, that has an impact.
54
:Um, but what's interesting is, is to
figure out what those things have been.
55
:And typically when I have come in
and like, and trying to figure out
56
:architecturally, 'cause we need
things to be quite black and white.
57
:Right.
58
:Quite, very distinct.
59
:Right.
60
:Kind of in and out.
61
:Yes and no.
62
:All of these kind of things.
63
:So from an architectural decision
perspective, once or twice I've done
64
:this, where I've come in and we'll
end up having conversations and I'll
65
:eventually say this feels like there's
a previous proceeding decision.
66
:The time that this worked the best was
I started somewhere and there was a
67
:bunch of decisions which were kind of.
68
:Historical decisions, which, you
know, were not maybe made the most
69
:optimal decisions, they were the
decisions that had been taken.
70
:And that was where everything was and
that was where everything was headed.
71
:so after a big, long conversation with one
of the architects that I was working with.
72
:They offered.
73
:And so this was their idea, not my idea.
74
:They offered to kind of write down
those big decisions like reverse
75
:engineer the ADRs, kind of like filling
in the history, you know, 'cause
76
:like adr, I, definitely feel anywhere
kinda like an immutable change log of
77
:what you're doing in your software.
78
:But we were missing the initial
one or two or three things.
79
:So they wrote down these ADRs around
about when we move these teams and when
80
:we decided to pick this technology.
81
:And they even recorded like, these are
the three different options we selected
82
:and these are the reasons that we thought
were bad at the time, but we still did
83
:it because these were the circumstances
and it was interesting for them to
84
:kind of go back and do that kind of
archeological digging back through stuff.
85
:But it also, gave a lot of clarity
to the constraints that were there
86
:and why those constraints existed.
87
:And then that allowed people to kind of
challenge them if we were like, right,
88
:we can't do this because of this, and
they're like, well, the reason we didn't
89
:do this at the start was because if
you look in the a ADR we didn't do it
90
:because we didn't have enough people,
or we didn't, you know, we like product
91
:management, didn't think this was as
high a priority as this other thing.
92
:And then we could say, right now
we're looking at this decision again.
93
:'cause you can kind of go back
and refactor some of these
94
:fundamental decisions or like
not, or revisit, not refactor
95
:With that clarity as to why, 'cause
people kind of think that some of these
96
:decisions are, untouchable, right?
97
:You can't go back to them.
98
:But if you go back and remind yourself
of the context, then you can say,
99
:this context has totally changed.
100
:Like we've, we were a 12
person startup, right?
101
:And now we're a 200 person scale up.
102
:So now we do have people
to do stuff, right?
103
:Or there was another decision, I think
this one's in my book, where the client
104
:wanted to move really fast and they were
basing everything that they did on AWS.
105
:But there was one thing that
they wanted, there was a chip set
106
:that was only available on Azure.
107
:So therefore they took this decision
to do something knowing that they,
108
:their strategically target platform,
that they didn't have this chip
109
:set that they wanted available.
110
:Then they could go and say, right,
for this small decision, we're gonna
111
:take this thing and go to GCP and
it'll be like a bounded decision.
112
:So it kind of freed them up and it
reminded them what the actual constraints
113
:of their previous decisions were as
opposed to these kind of not very clear.
114
:Implicit decisions setting.
115
:Artificial kind of boundaries on things.
116
:And that was super important.
117
:The reason we did it at that time was
because the client I was working for
118
:was a scale up, so they needed all of
this clarity to make it clear to all
119
:of the new people joining why we're
doing this, why we're doing that.
120
:I think having done loads of things
since then, I think it's a kind of
121
:approach, which could work really
well, like these fundamental decisions.
122
:Like if you're working in a multi-tenant
system, what do we mean by a tenant?
123
:What's the boundaries of our
systems, et cetera, et cetera.
124
:What things, what's our core domain?
125
:What are the things we build?
126
:What are the pieces we don't
build, et cetera, et cetera.
127
:so loads of those things are typically
implicit, but you can pull them
128
:out and make them explicit, even
if they're kind of underpinning.
129
:So that's my experience.
130
:And I was wondering like what you're,
I guess maybe you've seen this kind
131
:of thing before as well, there.
132
:Kenny (Baas) Schwegler: Well, it
really reminds me of some workshop
133
:I've did in the past called.
134
:Appreciative Inquiry, which had nothing
to do with decisions, but everything
135
:to do with like events or decisions,
And it, really reminded me about
136
:that, And now I'm thinking about,
oh, it would be really well to do an
137
:event storming, like event storming.
138
:What decisions have we done?
139
:So far and what are
decisions that we still make?
140
:you can 100% sort of
like events from that.
141
:What are the events that decision took
place even though they're not right.
142
:so that's my first thought.
143
:And then with appreciative inquiry,
what we did then was looking at
144
:what are some good and bad decision
and there's a difference, right?
145
:What are some.
146
:Good and badly made decisions and
what were some good decision that
147
:had bad outcomes because right,
the resulting there, there, there's
148
:a difference between the two.
149
:But I think you could definitely
like storm that out in a whole group.
150
:That was my first thought.
151
:And then you can take, which
decision do we want to take with us?
152
:Which decision do we need to change?
153
:And then you can do sort of like a
reset maybe what what you're saying.
154
:Then write some of these
stuff down as an as.
155
:Okay, we took this decision, this is
what we know, and then just move on.
156
:Andrew Harmel-Law: Yeah,
157
:like baseline or like snapshot in
like an event driven system, right?
158
:You're like, we don't, we
could go all the way back.
159
:But then the further we go back,
we're into like people remembering
160
:and like half of those people or
maybe all of those people left, right?
161
:Like I see things before we're like,
nobody can quite remember what the
162
:decision was or why it was taken.
163
:It's just, we use this
software to do this.
164
:And you're like, why?
165
:No one can even remember
why this happens anymore.
166
:Andrea Magnorsky: that was kind of my
question as you were talking about this,
167
:is there a particular trigger that made
you want to, like, is it more about
168
:like, I found I, Andrew found it a good
practice too when I start the place, try
169
:to make this explicit to make your life
as in making change easier basically, or
170
:is there particular signals that you are
looking for to implement this strategy?
171
:Andrew Harmel-Law: That's a good question.
172
:So I think now, because lots of my
practice has kind of come about from,
173
:like, we were talking about this a bit
before, started recording Andrea, right?
174
:Like, I don't think there's,
there's a lot of this in my book.
175
:I had a, not a fight with O'Reilly,
but I was trying to make, when I
176
:wrote my book, I was like, I don't
want this to be like a recipe.
177
:Do these things and then you will get
success because you kind of don't,
178
:it's more like, watch out for these
things and then you'll probably
179
:be able to figure out what to do.
180
:So.
181
:came about because it was a
specific client and we were there
182
:specifically to help them scale up.
183
:So they'd made loads of decisions
super fast and they figured out what
184
:their problem solution fit was, and
their product market fit was right.
185
:They'd figured that out,
but they'd run really fast.
186
:So the people who were there remembered,
although the different people who
187
:remembered different aspects, like
the reason we picked Amazon over
188
:GCP is this, and if you spoke to
someone else in the data team,
189
:they'll be like, we hate those guys.
190
:We, lost the battle.
191
:Whereas like the people who wanted
AWS don't think anyone lost.
192
:They're just like, we
picked the best thing.
193
:So that was kind of like to
to just for that purpose.
194
:It was like keep having to explain
the same thing over and over again
195
:to every new person who joins we just
want to explain and we want to say,
196
:we know it's not perfect and we know
everyone wasn't happy and we know
197
:we may regret it because at the time
it was good, but now it isn't good.
198
:Um,
199
:Kenny (Baas) Schwegler: That because
that's something interesting you, they,
200
:we need to re-explain everything and
there's a telephone game in there.
201
:Andrew Harmel-Law: totally.
202
:Kenny (Baas) Schwegler: the
decision, and this is what I
203
:see many times as well, right?
204
:Some person made decision,
never wrote it down.
205
:And then when you come in, you
talk to one person, they say, A,
206
:and you talk to one person B, and
there's already a conflict, right?
207
:And then when you ask is,
okay, who made the decision?
208
:When is it made?
209
:Nobody would say, ah, well,
you know, it's over there.
210
:I made that decision
and this is the reason.
211
:And it, it's never one coherent story
when, when you don't write 'em down.
212
:Andrew Harmel-Law: totally.
213
:Kenny (Baas) Schwegler: That reminds
me of there's a total telephone game
214
:going on there and then ranking.
215
:Andrew Harmel-Law: Yeah.
216
:And because this, this is, and that
goes to two points that kinda, I think
217
:we don't worry about enough, but make
a big deal difference because build
218
:software, I don't think I've ever seen.
219
:The software being maintained by a team
with the same team that built it, right?
220
:We move around a lot, right?
221
:We do a bunch of stuff.
222
:We arrive somewhere.
223
:I mean, I know I'm a consultant, right?
224
:So I do this, this is my job.
225
:But even then, there's very infrequently
someone who's been there forever.
226
:And then there's the danger of, you
know, like Alberta brand Le would call
227
:'em, like them becoming the dungeon
master and all this kind of stuff, right?
228
:But most people move around, like
very few people are living in
229
:the code bases that they created.
230
:They're living in the code bases
that someone else created, and
231
:they're trying to, and if they
don't know the reasons why.
232
:like you said, that you don't get,
decisions are good and bad, and that's
233
:a, what Diana and Ian would say, right?
234
:Architecture, like you do the trade
offs and all that kinda stuff.
235
:There is no best decision, or
sorry, there's no right decision.
236
:It's the best one given the circumstances
and what you know, et cetera, et cetera.
237
:And then you hope that decision gets
into code, which maybe may, doesn't,
238
:maybe doesn't happen, but like, this
is like the thing, like the explaining,
239
:like you're saying Kenny, right?
240
:Like people go.
241
:They go like, you told me this thing
when I joined, or I read these pages
242
:on the wiki and then I looked at the
code base and I'm having difficulty
243
:understanding, at least if you have some
ADRs, at least for significant things,
244
:you can explain like the historical forces
that were affecting this kind of thing.
245
:Which helps people like raise the empathy
for the code base and each other and
246
:all of the other different things and
understanding they can reverse engineer
247
:the mental landscape of what was going on.
248
:As to why things are like they
are and then they know what they
249
:can maybe try and fix or maybe try
and change or maybe try and evolve
250
:And I think
251
:because I've talked about this elsewhere,
I think at New Craft, like there's
252
:like different bits of code bases can
have like strong ego power, right?
253
:Like it can feel like you shouldn't
touch it because someone who's probably.
254
:a senior executive, like wrote
these parts of the code and
255
:we all know we can't touch it.
256
:And you're like, but why?
257
:Like they don't live there anymore.
258
:It's like they built this
thing and then they've left and
259
:gone and done something else.
260
:Andrea Magnorsky: Some people, some
places that they still, they, like
261
:I know of, I've been there and I'm
sure you've probably been there too,
262
:of places that they left and they're
theoretically in some other position.
263
:But in reality, if you
touch that, their baby.
264
:You know, these people will
cry and will come down and be
265
:like, oh, you touched my baby.
266
:What is the story with this?
267
:Andrew Harmel-Law: Yeah,
268
:interesting and I think like,
I've seen this too, right?
269
:Like if you write that down, 'cause people
forget, like they're like, oh, I did the
270
:best thing and I built the best thing.
271
:But they never, I don't think, when
you speak to most people when they're
272
:doing things, they're like, no, there
were a bunch of forces and these
273
:are the things that we were taking
account of at the time and stuff.
274
:Andrea Magnorsky: actually, this is thing.
275
:So I'm gonna answer this last, but
how do you write down about the power
276
:imbalances, in ADRs in a useful way?
277
:Andrew Harmel-Law: See,
I don't think I do.
278
:I think this is something I
want to think about more is I'm
279
:thinking about it more and more.
280
:It's definitely not in the book, I mean.
281
:I think it should be in the book,
so yeah, but it's not, I think it's,
282
:Kenny (Baas) Schwegler: This is
a good question because I face
283
:this myself, and I'm very curious
about other people's thoughts.
284
:we maybe can do a whole
episode about this.
285
:Like, there are some personal reasons
that are very, let's be honest, we are
286
:in a patriarchal or in a male-dominant
industry where if you're dealing with
287
:emotions, It's very hard to talk about.
288
:Most of the time.
289
:I find it very hard.
290
:Well, I find it easier now, but I've been
to a lot of, therapies to talk about that.
291
:But, I've had situation where, you
know, there's a clash and there's a
292
:lot been said about an architecture
decision writing that down felt very.
293
:Hard for them to like write down.
294
:So going into the conflict management,
should we write down the conflicts
295
:and the power play that was there?
296
:Because that's relevant context, but
it's also very hard to write that down.
297
:A how do people interpreted
that and B, it's very fragile.
298
:Andrea Magnorsky: It's
extremely context aware.
299
:So my question really was about
that, the things like, it's important
300
:because if there was a conflict and
even the decision, the thing that you
301
:were explaining earlier, Andrew, that
thing of, oh, we chose AWS and the
302
:data, people are like, oh, we actually
preferred GCP or something else.
303
:And it's like all those
people that did even.
304
:That should be in the a DR, but it
probably isn't this is why it's so
305
:important because when you revisit
this decision, people that have
306
:a memory will be like, oh, we're
gonna get them back, the staff.
307
:Or maybe not.
308
:Or maybe they'll be like, you
know what, actually we live with
309
:AWS and it's, it's all right.
310
:We always just wondered, and so.
311
:My general take is to say there
was a conflict and not, and it's
312
:like, it might be worth talking
to people in data and blah.
313
:The, these were the two points,
but, it's not always possible
314
:to write that down safely.
315
:And that's where Kenny's comment
come about, the safety of,
316
:Andrew Harmel-Law: Yeah.
317
:Kenny (Baas) Schwegler: We
318
:Andrea Magnorsky: of who's writing what.
319
:Kenny (Baas) Schwegler: yeah, we should
definitely dive deeper, but I think
320
:you just portrayed a heuristic, right?
321
:What if something on a personal
conflict, like we call it, we call it
322
:personal based and relationship based.
323
:So what if there's a
relationship based conflict?
324
:Do we write that down?
325
:But I think it's a good heuristic
to just write down, there was
326
:a personal conflict, just.
327
:Talk with these people.
328
:I think that that would be a good
heuristic already to, to help people.
329
:Like what, if you deal with that, do
this for now and maybe in the next
330
:episode we could, definitely dive deeper
331
:Andrew Harmel-Law: there's one thing
actually I forgot about this in my
332
:book says, there's someone at my
current client who's using this, and
333
:it's really good because there's a
lot, there's where I'm at the moment,
334
:there's a lot of teams with a lot of
people trying to move really fast.
335
:Not everyone agrees with
every single decision.
336
:And, but there's a thing I've
suggested, and I've seen it
337
:occasionally in ADRs for when you've
got the pros and cons of each option.
338
:You can say adopted despite, right?
339
:So we take this, even though we
know that this is a downside, right?
340
:So we've taken this option, even
though this is a downside or rejected
341
:despite, and I've seen that work
well where people, they at least
342
:feel acknowledged and they feel
acknowledged not in the comment section,
343
:they feel acknowledged
in the body of the thing.
344
:So you can say, right, I disagree
with you, but I'm still gonna
345
:acknowledge that kind of stuff.
346
:And like you say, Kenny.
347
:It doesn't need to be like these
people fell out with each other, but
348
:this was a disagreement and stuff.
349
:There's bits in my book about
coalescent argumentation.
350
:So it's like trying to figure out
where the area of disagreement is,
351
:because when there's disagreements
frequently it's like, right, I don't
352
:like Andrea, so therefore any idea that
Andrea comes up with is terrible, which
353
:let's be honest, is not gonna be true.
354
:But maybe there is a thing where we're
like, right, this bit we disagree.
355
:And then when we, when you zoom in on it.
356
:So writing about ADRs can be good.
357
:And again, my big thing,
small ADRs, right?
358
:Because if it's a big a DR then
there's a, there's the, the
359
:chances of a clash is far bigger.
360
:If you break that up into smaller
pieces, you're like, this comes from
361
:Reinert, and it's like principles
of product development flow.
362
:Within every big decision,
there's one or two bad ones.
363
:you break a big decision
into 20 small decisions, then
364
:like 18 of them will be good.
365
:And then two of them might
be bad or hard, right?
366
:But at least you can then focus and
then you can have a discussion and a
367
:disagreement about something specific
as opposed to, I will block your whole
368
:thing because I don't like the fact
that you're going to use Node for this
369
:tiny little, some library I don't like
370
:Kenny (Baas) Schwegler: think next session
we could definitely invite Gien talking.
371
:About decision making, because
decision making theory, there's
372
:also a lot about what we want.
373
:So I think that's an interesting, how
would we write that down, what we want?
374
:So that's a personal interest.
375
:Andrew Harmel-Law: Yeah.
376
:Andrea Magnorsky: I
think we're on time-ish,
377
:Kenny (Baas) Schwegler: Yep.
378
:Andrea Magnorsky: so maybe we
should do some closing comments.
379
:Maybe a little summary wouldn't be amiss.
380
:So it feels like.
381
:Andrew told us a story about
surfacing, implicit, and very
382
:important, decisions that happened.
383
:That's a good way to kick off things.
384
:Then we talked about how, conflict,
how we could maybe include
385
:conflict or not in this, in ADRs.
386
:Anything else?
387
:I, I mean, there's more
than that obviously,
388
:Kenny (Baas) Schwegler: I think another
heuristic here would be, write down past
389
:decisions that cause conflict, right?
390
:If there's a lot of confusion.
391
:just start writing them down
and, and reset them in a way.
392
:Andrew Harmel-Law: Yeah.
393
:Because decisions are made
at a point in time, right.
394
:It's what you knew and
thought and felt at the time.
395
:Andrea Magnorsky: So is
that, a goodbye for now
396
:Kenny (Baas) Schwegler: yeah.
397
:Andrea Magnorsky: for this episode?
398
:Well, thank you very much
and see you next time.
399
:People listening to facilitating
software architecture and design.