I wonder if the author is right; using a slightly wrong capacitor for the clock (keeping in mind that they're usually +/- 10% or worse from the factory) may produce a different oscillation speed, but usually not an outright failure. These chips accept a pretty wide range of clock frequencies.

And in the comments, they note:

"I replaced the 18pF capacitors on one of the non-working boards with 10pF capacitors, but it still doesn’t boot or respond to the debugger. That surprised me – I thought that was the answer! It was a rush rework job and I made a bit of a mess of it, including accidentally desoldering and reinstalling the crystal, so I’ll try again later with another board. But it appears that the capacitor value may not have been the issue after all. Either 10pF is a bad value, or it’s the crystal itself that’s at fault, or I’ve failed somewhere in my troubleshooting reasoning. Hmm."

In my experience, mystery stability problems are often caused by capacitors, but of a different kind: decoupling capacitors on the power supply pins. If there's not enough of them to keep up with the noise originating from motors or digital switching, I'd expect that exact issue. Intermittent "impossible" CPU states on some boards, no rhyme or reason (because sometimes, that +/- 10% saves you and sometimes it does not). I'd try more caps and possibly some ferrite beads.

Not an electrical engineer, but I was 2 sentences in when I declared to myself "its the external clock source" and then a physical shudder as I remembered how many times a contract manufacturer has used some aging crystal or third party component src and not updated the matching capacitors, causing very similar issues on projects I've worked on in the past. The very very worst is the aging xtals - a warm wind can make them work for an undetermined amount of time, which is bad if the factory is running warm around 26C or so and you end up shipping the batch to Europe where its early winter...

Oh this reminds me of when I was working on embedded Linux devices for vehicles.

We had a new batch of hardware come in that had a failure rate of about 50%. Normally it was on the order of < 1%. Our CM was very good at troubleshooting.

Nothing had changed in the BOM so we were left scratching our heads.

Hooked up the JTAG debugger to see where it was failing to start, and the CPU wasn't even coming up. Power rails looked good, but the CPU just wasn't booting.

Eventually we discovered that the supplier had given us a batch of crystals that were slightly more sensitive to the capacitance, and our design was just on the margin of working.

Lowering the caps to the proper values according to the xtal's datasheet got everything working again.

Oh, wow, BMOW is still going. It's been quite a long time since I've read. Maybe I should catch up.

I've got a standard check-list of tests to run before any deep analysis on faulty circuit behaviour:

1. Check the power rails are correct and stable (at the target devices, not the PSU source)

2. Check the resets are correct (polarity, level, sequencing) and reaching where they're needed

3. Check the clock is toggling cleanly (no jitter, has clean monotonic waveform)

4. General signal integrity and setup/hold of signals that are related to the issue

That catches a lot of basic issues. If those are all clean, then you go deeper.

To the article author's suspicion of crystals, I have seen crystal oscillators fail (stopping toggling) as ambient temperature ramps up and down; that's a nasty one to catch and prove, but it can happen. Changing vendor was the only solution there.

I've had the same happen with a production run, except the crystals were from the same manufacturer and the same part number, just different batch. It was fun to track down.

When I got to my current workplace I was surprised to see we always, invariably (unless it's really impossible with the chip) use an oscillator not a crystal. But every time I see one of these posts I think about the difference in cost for us relative to the math of capacitance and debugging, it's just not worth it. And of course the first board I worked that we basically built to print and had a crystal integral to the design... They got the capacitance wrong and it didn't go.

Heh, when I read the symptoms:

“Some exhibited “haunted” behavior, seemingly jumping to random sections of the mcu program code, outputting messages on the display that made no sense given the context. One of them appeared to work in slow motion, with LED blinking and display updates noticeably more sluggish than normal,”

my first thought was that it smelled like a clock issue.

Some of the nastier issues I have had the pleasure to debug included (a) traces that had microcracks which affected analog readings when the PCB heated up after prolonged usage (QC issue from the PCB fab) and (b) a (suspected) ESD strike that gradually took out several components in the weeks following as I was investigating the device while new problems kept popping up. Marginally stable composite amplifiers have also caused some headaches over the years.

Hardware really is hard.