My Disaster

In the past, learning and teaching debugging was, perhaps, only somewhat more haphazard than it is now. Learning from a mentor, a teammate, trial-and-error at 4 am before a customer demo. I’ve pulled double all-nighters. Painful, with a stomach-churning donut-induced sugar high.

This “approach to problem-solving” - it is really such a big deal?

Maybe this wouldn’t be such a big deal if we still had four-year product development cycles. A hint of this new reality emerged in the early 90’s as a simple pressure to “compress the schedule a bit.” But now we’re shoehorning all that work into 18 months or less, and we’ve unfortunately done nothing to shrink the debugging cycle along with it.

Another big deal is that customers want neat new products, but they’re also insisting on high quality. And now they’re not afraid to demand last-minute changes, dictate terms, or drop you for another supplier seemingly on a whim.

But we must recognize that short schedules are permanent and just another constraint in an extremely complex dance that squeezes us all: the customer, the management, and the technical teams. How we address the challenge becomes the differentiator.

It’s all about problem solving.

For several years I worked at major telecommunications companies as an embedded systems engineer and technical manager, designing mobile phones and leading mobile phone engineering teams. In the late 1990’s the telecom industry was in decline and 2000 was a terrible year; industry reports screamed of “carnage” and “tumbling stocks.” It was “survival of the fittest” for companies large and small in the increasingly unforgiving economy driven by quarterly reports.

As it happens, we were late and the customer was screaming about poor product performance. I was on one of several daily status conference calls with engineers and managers from two continents and three times zones when reinforcements arrived. Five developers stood at my door ready for their first assignments. My management absolutely supported me by giving me these talented developers, but they were inexperienced in low level embedded systems and new to this product. How was I going to quickly and effectively train them without letting the schedule slip any further? Should I drop everything to train them myself? Should I have the new engineers shadow the more experienced engineers? As an emergency problem solver myself, I knew the worse thing to do was to saddle my already stressed-out experts with novice engineers to mentor. But I might not have a choice.

So How Can We Fix This?

What did I do with my new engineers? Without realizing it, I forced them to become more methodical. I couldn’t focus on all the problems at once so I started asking tough questions about symptoms and behaviors. That kept folks busy until they got stuck again and came back to see me. Ping-ponging developers back and forth between my office and the lab made me feel a little guilty, but I didn’t know the answers to most of their questions without investigating them myself.

But something wonderful happened. Many of the developers got smart - they started coming prepared with answers to my questions, even if the root cause of the bug still eluded them. I still remember with great fondness a day when one of my engineers anticipated several steps in my thought process. In response to each of the successive questions I posed, he proudly whipped out one plot after another to prove he had identified, implemented, and validated a bug fix. Today, he is an experienced and respected embedded engineer at a major corporation, and I value the time I worked with him.

So what’s the answer? Where do people learn to solve problems? No formal methodology is taught in schools. Formal on-the-job mentoring is an excellent source of training, but it is time-consuming and expensive.

So I wondered, could technical folks learn debugging skills the way I mentored my team? Each bug my engineers attacked was actually a mystery solved through methodical questioning. Something about that process had worked, because after a time, the ping-ponging between my office and the lab declined. The team became more adept and flushing out the bugs more quickly and we got a handle on the infestation.

As they say, the proof was in the pudding.