There's a fantastic comment on the article written by Jessamyn West (Tom's daughter):
I saw this zipping by on Twitter. Heya I’m Tom West’s daughter and I always pop onto threads about SOANM to bang the drum of good work-life balance as well. My dad was an amazing man and we got along well, but he was a pretty distant dad and basically left the raising of my sister and me to my mom who might have wanted to do something professionally as well.
The things he did at work, and his and Kidder’s ability to talk about them, were quite interesting and have lessons that can be relevant in many different timelines and lifestyles. But you know, I wound up moving to Vermont and helping people use technology to solve problems in their lives. You need all the things–technology, an understanding of people, and a life–and my dad possibly only had two of those at any one time. Thanks for writing this.
Wow, I've worked with her professionally, and had no idea she was connected to a book I read and loved when I was young. Always interesting to hear more of the story around a company and a family.
I struggled with this a few times: “When a person signed up to do a job for him, he would in turn trust that person to accomplish it; he wouldn’t break it down into little pieces and make the task small, easy and dull.”
I’m not particularly talented, far from it actually, but many times I’ve worked in teams with non-junior people who were tasked to do some portion of a project, and simply delivered garbage: a terribly over engineered solution in a component where over engineering was actually very dangerous (e.g. doing distributed systems using several small bespoke moving parts rather than a single solid leader election mechanism, or even better getting away with something leaderless and more elegantly stateless, to give a recent example).
In those occurrences, I inevitably “broke it down to pieces” by keeping pointing out all the exact scenarios where the code delivered would fail, until the other person really had to finally move to a solution much more similar to the one I had in mind (much much simpler). Sometimes this required an embarrassing feedback loop of 10+ round trips, where every time I’d just look at the newly delivered code say “this doesn’t work in a X scenario”. And I was not in a better position from a knowledge perspective, we were essentially peers.
I felt like a prick and definitely felt resentment from the other party (especially since code reviews are public), but at the same time, how could I have acted differently? The end result was really orders of magnitude better, more robust and smaller.
> I inevitably “broke it down to pieces” by keeping pointing out all the exact scenarios where the code delivered would fail, ... I felt like a prick and definitely felt resentment from the other party (especially since code reviews are public)
There's the problem, right there. Using a code review to expose a design flaw.
I worked at my first startup starting in 1988, not long after SOANM came out. Most of the engineers at this company read the book. I was struck by the similarity between the situations and dynamics described in the book, and what I was observing first-hand.
Our approach to engineering was academic in nature. This was remarkable because there was only one PhD in the bunch -- me, the most junior person -- and everyone but me had a spectacular track record of designing, building and delivering innovative products.
We started with whiteboard discussions. These would go on for hours at a time, across multiple days. When we had the outlines of a workable idea, someone would go and write a design document. And that got reviewed extensively, over a few days. The "breaker" sorts of analyses got done at this point.
So by the time we started implementing, we knew we had a very solid design. From that point on, bugs were just bugs. They weren't these initial cracks that revealed a rotten foundation.
I thought that this approach worked incredibly well. And I was dismayed to observe that it wasn't practiced elsewhere. I did try to institute this approach a few times, but it doesn't really take hold unless you have other senior people who want to adopt it. And at my last startup, we had smart but young and inexperienced developers who were bought into Agile, and code reviews substituting for design reviews. The predictable (and predicted) hilarity ensued.
Yeah, these days what you are describing is simply utopy, at least according to my experience spanning a single decade across two startups and two large companies.
Engineers, once they are hired and passed the interview, are mostly seen as a commodity, in the sense that, in the eyes of management, every non-junior engineer should exactly be able to come up with the correct idea (modulo a short phase of code review, just in case), so the output they produce can just be as equally produced by a peer (just like a commodity): in that world view, design meetings spanning multiple days don't do much, and they just make you lose time against the current jira sprint.
I would love to work in a place where management understood that collective intelligence is almost always superior than the initiative of a single person, and thus it's very important to have a design phase where as many engineers as possible are involved.
Every single time I had to lead a significant project, I always spent so much time involving as many people as I could find to discuss the idea, constantly self doubting my ideas. Incidentally, very few significant bugs have occurred when the project was delivered, in contrast with the example I was quoting, where the thing was developed completely hermetically by another non-junior engineer, who failed to account for many non-to-rare corner cases.
Effective agile teams practice swarming by minimizing the work in progress "inventory" and focusing all team members on a very small number of stories at a time. You can sell it to management in terms of adopting agile best practices and reducing the cost of delay.
I wish I knew the answer to this, but I don’t. The closest I’ve come to succeeding at this is to have a reputation as a “breaker”. This really started in undergrad algorithms; I wasn’t great at finding solutions, but I had a knack for finding edge cases to break solutions others had come up with. Sometimes it just needed a tweak to work, sometimes it was “throw it all away and start over”. For the few times I’ve done regular office-style work over the years, that’s come in handy. I feel like a dick a too, but I try to be really gentle and end up with people showing up on their own to “run something by me” before building it out. Saves everyone frustration!
There have been colleagues though who do exactly what you’ve described. That brutally painful loop of breaking pieces of an overly complicated solution over and over. My unfortunate confession is that I usually end up giving up and let them fail on their own. I’m always willing to help, but if someone doesn’t want help, I won’t spend much energy forcing it on them.
Yes! Finally, a word — "breaker" — for my greatest talent, which I still haven't found a way to monetize in my 70 years and 8 months on the planet. I can unerringly and quickly find the FAIL point in most devices/methods/ideas/setups/systems/websites etc. in a savant-like fashion. Perhaps in the future I'll find someone who will pay me for my quirky skill.
Software QA. Not particularly prestigious but the pay is passable, more so if you can code well enough to automate tests or dig into the codebase to isolate the bug.
security related positions, such as application penetration testing, physical pen testing, etc. all require the "security mindset" which is essentially the skill of thinking adversarily and breaking things.
IMHO, Essentially you need to let them experience the issue you foresee first hand, by letting them fail. That way they can learn the same lessons you learned previously.
In my experience, it doesn’t work that way at all.
What happens is that the code will be released, some user will hit the bug in the field, and the issue will be escalated to the engineer, who will proceed to spend entire days or weeks to troubleshoot the problem, and then just put a dirty patch that fixes the specific issue, leaving all the other corner conditions still exposed and ready to explode. At that point, major redesign of the solution, which would have been possible at the code review phase, will just be deemed too risky by both the engineer and the management.
On top of this, Management, usually ignorant to concepts such as “over engineered solution”, will praise the heroic effort of the engineer who spent days troubleshooting the issue, clueless of the fact that it was a completely self inflicted wound because the engineer decided that the best way to calculate 2+2 is to really do sqrt(2^4) and then realized that the user really wanted to do 3+3, so they’re going to put a special case to evaluate 3+3 as sqrt(3^3+3^2) rather than just doing the damn x+y.
Nobody will even realize that the issue could have been completely prevented in the first place by adopting a solution that was much easier to reason with on the whiteboard, rather than a house of cards of custom logic.
That's a little easier to do for software, harder to do in the context of building hardware, which is what the book was about.
Circa 1980, around the time of the events in the book, the time delays involved in making design changes to boards built from MSI building blocks were bad enough. Physical wire mods to a board took hours or days. New boards could be made in weeks or months. There was also some flexibility because parts of that design used PALs, which were easily reprogrammed.
But in today's world the time cost of a re-spin of a CPU or a highly integrated SOC chip is huge. Months for simple changes, years for complete re-designs. Just look at the time estimates for hardware changes to x86 chips to mitigate Spectre and Meltdown. Some months for new microcode; many years for new silicon.
Nevertheless, it's perhaps easier to let things happen and afterward think "I told you so". Because, unfortunately at most companies, if you try to impart those lessons earlier in the process, e.g. at design reviews, you quickly get the reputation of not being a "team player". Sounds crazy, but that's simple human nature.
Only lazy, sloppy developers fail by repeating the same mistakes as others. Unless they're working on something truly novel (unlikely) there are books and articles out there which explain the problems to avoid.
When the coder is a peer and you’re not their mentor, manager, or lead, I normally keep code review comments to suggestions or questions - “What happens here in X scenario? Maybe we should refactor this part out into a stateless microservice?” That avoids the resentment and makes you look less like a control freak, but still shows your competency/foresight. If they choose not to take your advice, too bad for them, but I try to make sure they end up personally fixing the inevitable problems and hopefully learning from it.
Other than that I haven’t found any great solution, but I have found it stressful and counter-productive to try and do EVERYONE’S thinking for them. They do not learn in that case, they just continue to outsource their thinking to you.
e: also 10+ rounds is excessive for a peer review imo, after 2 or 3 I’d be having a higher level design chat with their lead/manager.
Other useful phrases: "Here, I was expecting to see...", "This code made me look at this other code and they cannot both be correct...", and "It took me a while to see why this is correct. We could save the next reader time by..."
Excellent code reviews take time and attention, which is why you need auto code formatting and $lang proverbs to take the uncontroversial stuff off the table early. Then you still have brain power left to give value from the review.
Also: work hard to earn a reputation as a fair but tough reviewer. Then when you need to be a prick, people will know it's important and let their ego down.
The quote you quoted says "signed up to do a job for him", but then what you describe is people being "tasked to do some portion of a project".
Is there a difference in that? If I sign up to do a job, that's quite different to having something assigned to me. Things I sign up for are things I have chosen, based on my abilities and enthusiasms; things that are assigned to me.. well, not so much.
That was my observation as well. I find the quality of work goes up dramatically when people are invested personally in the success of their part of it. "Signing up", to me, is about committing to that sort of investment, and it's the signal I look for to know if a team is going to be healthy. Not having that signal is a two-way street, though -- sometimes the answer is to restructure the work overall so that the pieces align better with what team members will sign up for.
I'd assume that the key element here that they don't talk about is that he didn't keep people on his team who he couldn't trust to accomplish the tasks he set them.
Giving feedback doesn't necessitate micromanagement (which I'm using as a synonym for demanding a particular "how" in addition to an assignment of "what"). Would your experience have been better if you'd given the same high-level feedback as what you've shared with us? For example: "Hey, Non-Junior Person, your approach appears to be overengineered. For this application, overengineering isn't a matter of taste; it's a matter of safety. Please come up with a simpler approach."
I've only seen it work when there's a combination of competence and ownership. i.e. someone from above can't just say 'this is yours, figure it out', the other party has to be able to do it and accept the ownership of it. Even when they can and do, there will inevitably be things they get wrong and/or miss but when those are pointed out they, assuming they've accepted ownership, will understand without being asked that it is for them to correct. If you try do it before someone is able or willing, it can turn into a train wreck. Also, I've found it much harder to do in larger organizations for a variety of reasons.
I can't quite decide if this sounds great or awful. If code hasn't been adequately verified and/or tested, helping someone address those omissions (which would require fixing the bugs they make manifest) would be great. OTOH, merely "pointing out" or saying "this doesn't work" sounds not so great, and "similar to the one I had in mind" rings a huge NIH alarm bell. Instead of wondering whether you were being a prick, I'd highly recommend asking your other peers for suggestions on making the process less adversarial.
over-engineering is real and is slowing things to a grind around you everyday.
my conclusion is that people don’t really like simple things and they believe that complex=good and it’s a sign of how good/smart you are. so people are suckers for complexity. nobody is going to receive a medal or a promotion for well written and simple code that “just works”. you won’t know it’s there.
now, the reaction you’re describing comes from the “i’ve seem some shit” syndrome where not only you can see the layers upon layers of unnecessary crud but you can also see how it’s going to break in the future. it’s terrifying.
A few years later, in 1986 to be specific, I started a job at an OCR company. My cube was directly across from the office of Carl Alsing, the guy who managed "the microkids". I didn't put two and two together at first. Alsing was one of the people who interviewed me for the job, and I got to know him a little bit for a couple months before someone told me about the book connection.
So I went to the library, got the book out, and read it again. When Kidder got to the part where he introduces Alsing, it was amazing. In a paragraph he captured details about Alsing that I hadn't even realized myself (eg, his personality projects the image of a smaller man, though he was actually decently tall).
The other thing he got right was Alsing wasn't much for schedules and tended to work on things that interested him instead.
Anyway, having verified that Kidder had indeed captured a lot of truth about Alsing, it gave me trust that the rest of the book was similarly accurate, and not just sexed up in the process of storytelling.
I loved this book as a kid, although once I got into the work force and began to appreciate the value of a work-life balance and the basically unbrigdeable gap in incentives between management and labor, I started to look at the ludicrous demands the company placed on its engineers with a more jaundiced eye.
My perception of the team leader hero, Tom West, was also permanently altered by an essay written by his daughter Jessamyn (a quasi famous technologist in her own right), who described him as basically a depressed alcoholic.
I've always considered myself lucky that one of first minicomputers I got to know and love was a DG Eclipse. I had the experience of reading the book in college and learning the weird, wonderful, and amazing AOS/VS operating system at the same time.
In the late 1980s I got to learn a ton about an operating system that had threads (tasks), virtual memory, shared memory, rich filesystem ACLs, symbolic links, and a bizarre shell language that forced you to do all your looping via recursion.
I learned how to program in C on that machine, I wrote some of my first assembly on that machine, and I spent hours pouring over thick black three-ring binders filled with manuals on OS usage, system calls, and the hardware architecture.
Friends and I competed to learn arcane details on how the system worked. My main claim to fame was I figured out how to share memory between unrelated processes, and how to use the atomic-test-and-set instruction to safely synchronize access to it.
One time while we were studying the virtual memory system, a friend wrote a program to see what would happen if you allocated the entire 4G address space of virtual memory and then randomly touched different pages of it. Predictably, the storm of page faults ground the machine to a standstill, causing our lazy sysadmin to have to put down his his copy of Datamation and address the gathering throng outside his door.
I have this recurring dream that I'm back at college for some event, and I wander into the Computer Center and look around. In the very back, there is a dusty 'Dasher' D210 terminal next to some stacked up old furniture -- plugged in and emitting a dim green glow. It's hooked up to our MV/10000 Eclipse, which I imagine is still in the basement of the Library Building humming along after all these years. I try to log in, but I can't remember any of my old accounts and passwords.
We had a MV/8000 at college, and it was a really nice machine. We had compilers for Pascal, C, COBOL, and oddly enough, Ada. You could cross-link these languages, so you'd have a front end written in COBOL that would make use of a Pascal back end. And the bindings just worked.
You knew when an Ada compilation job had been started because the other terminals would start getting sluggish...
I took a valuable lesson from the book that has served me well over the years.
At one point the CEO injects a critical piece of wisdom by telling the engineers they are not allowed to use a mode switch to preserve backwards compatibility.
In any system, the presence of mode switches often points to where things will go wrong. If a system has two modes, then inevitably, one of them will be more unloved, untested, under documented and unreliable than the other.
OTOH, in the context of a distributed system that has to be upgraded live and the deployment of new code has to be carefully staged across hours or days (e.g. a storage cluster with thousands of machines), such mode switches can be absolutely essential. The process can look something like this.
* Roll out the code with support for a new feature
* Flip the mode switch on a few nodes to use the new feature
* Wait a while, watch for anomalies
* Enable the new feature on more nodes, until it's on all
* Disable support for the old feature (another mode switch)
* Eventually, roll out new code with the old feature entirely removed
It's not quite a whole-system mode switch as in SoaNM, but that's kind of the point. In a distributed system actions do not affect the whole system all at once, so different rules apply.
I just read the book not too long ago myself. Thoroughly enjoyed it. The book that got me going was Hackers by Steven Levy which I've ready easily 20x. This book is just as good. I also think reading this much later in life gave me a fine appreciation as someone in Management of developers, politics, dealing with teams, trusts, motivations, etc. It's actually stunning how relevant the story still applies even tho it was written before most of us were born.
I read bits from the book in The Atlantic (I think) before I ever saw an MV. Some years later, my wife read the book for a class in her MBA program, and I read the whole thing.
A couple of points struck me then that may be of interest to the HN readership.
First, the instruction set originally proposed by the architect was very VAX-like. It would probably have been great for those writing in assembler. But the end was approaching VAX-like instruction sets.
Second, retaining the old Nova/Eclipse instruction set meant that one kept three general-purpose registers (well, sort of general-purpose: by convention, I think the stack pointer was in AC3). One also had no register-relative byte addressing.
I liked the Novas and Eclipses, and learned a lot working on them. But the RISC-based UNIX machines took over from the minicomputers pretty quickly.
[Edit: I don't think I've used an MV since about 1995--take any remarks with a grain of salt.]
It was fun reading the book as I was embedded in all the blue, black and red of wire wrapping my own boards for a mini-computer I designed and built for a company back then. There were times when I wondered if I was doing the right thing and to see that others did the same as I made me feel much better.
I read this book when I was in college and it had a profound impact on me - it was one of the primary motivators for me to get into computer science. Then when I came to the US for grad school in the mid 1980s, I got a programming job on campus in a lab and was shown around the computer room by my supervisor. And there I came face to face with the DG MV8000 and gave me a tremendous thrill. For the next couple of years when I worked on that machine (learning C among other things), I used to imagine the scenes from the book.
I saw this zipping by on Twitter. Heya I’m Tom West’s daughter and I always pop onto threads about SOANM to bang the drum of good work-life balance as well. My dad was an amazing man and we got along well, but he was a pretty distant dad and basically left the raising of my sister and me to my mom who might have wanted to do something professionally as well.
The things he did at work, and his and Kidder’s ability to talk about them, were quite interesting and have lessons that can be relevant in many different timelines and lifestyles. But you know, I wound up moving to Vermont and helping people use technology to solve problems in their lives. You need all the things–technology, an understanding of people, and a life–and my dad possibly only had two of those at any one time. Thanks for writing this.