When Your Auto-Healing Hides the Bug

You add a watchdog because a device keeps going offline.

The watchdog power cycles it. The device comes back. Everything turns green again.

Problem solved.

Except six weeks later you still have no idea why the thing keeps failing.

That is basically what I built.

I have five ESP32 BLE proxies feeding Bermuda presence into Home Assistant. One of them started dropping offline regularly, so I added a simple recovery automation: if it stayed down for ten minutes, cut its Zigbee plug for fifteen seconds and bring it back.

I capped it at three attempts per day and let it run.

Three days later it had fixed nothing.

Worse, I eventually realized the automation had been erasing the evidence I needed to figure out what was wrong.

The watchdog was winning the race

ESPHome's debug component can expose the last reset reason from an ESP32: watchdog resets, software resets, power-on resets and other reboot causes.

That was exactly the information I wanted.

Looking through the outage history, though, most of this node's failures lasted around ten minutes.

My watchdog threshold was also ten minutes.

So the sequence looked something like this:

1. ESP32 disappears.
2. It stays offline for several minutes.
3. My ten-minute watchdog fires.
4. Home Assistant cuts power.
5. ESP32 boots again.
6. Reset reason now says the equivalent of a power-on reset.

Which was completely accurate.

I had just power-cycled it.

Instead of learning what happened before the outage, I was replacing the previous reboot evidence with evidence of my own intervention.

I spent three days collecting that data. Most of what I collected was me.

I made the watchdog slower

My first change was simple: ten minutes became sixty.

There is nothing special about sixty minutes. I am not claiming an ESP32 that has been offline for an hour is proven dead. It is just where I decided to switch priorities.

For shorter outages, I want to observe what happens without Home Assistant interfering. If the node reboots and returns by itself, the reset reason may tell me why that reboot happened.

Once the outage reaches an hour, I stop waiting and choose recovery.

So the threshold stopped meaning:

How quickly can I get this device back?

For this device it now means:

How long am I willing to leave this failure untouched before availability matters more than diagnosis?

I still cap automatic recovery attempts. Three per day is my limit. At that point I would rather investigate the device than have Home Assistant repeatedly bounce the power and hide the pattern.

A simplified excerpt of the recovery automation looks like this. The daily counter reset and the helper definitions live elsewhere and are not shown, so this is not complete copy-paste YAML:

triggers:
  - trigger: state
    entity_id: sensor.esp32_livingroom_reset_reason
    to: "unavailable"
    for: "01:00:00"

conditions:
  - condition: state
    entity_id: sensor.esp32_livingroom_reset_reason
    state: "unavailable"
    for: "01:00:00"

  - condition: numeric_state
    entity_id: input_number.esp32_livingroom_cycle_attempts
    below: 3

actions:
  - action: logbook.log
    data:
      name: BLE Proxy
      message: >-
        Home Assistant is initiating a power cycle after
        one hour offline.

  - action: input_number.increment
    target:
      entity_id: input_number.esp32_livingroom_cycle_attempts

  - action: switch.turn_off
    target:
      entity_id: switch.livingroom_ble_power

  - delay: "00:00:15"

  - action: switch.turn_on
    target:
      entity_id: switch.livingroom_ble_power

Notice what I am not doing there anymore.

I am not trying to read the reset reason immediately before cutting power.

That was mistake number two.

My safeguard never worked

Originally I had a logbook.log action immediately before the power cut.

The idea seemed airtight: read the reset reason, write it somewhere permanent, then reboot the ESP32. Even if the reboot cleared the live value, I had already saved it.

Except I hadn't.

Once Home Assistant loses communication with the ESPHome node, its entities become unavailable. Asking states() for the reset-reason sensor at that point gives me unavailable, not the previous useful value.

And when was my logging automation running?

After the device had already been offline for an hour.

So every attempt to preserve the reset reason happened long after Home Assistant had lost access to it.

The automation worked perfectly. It wrote a log entry every time. The information inside the log entry was useless.

I checked the recorder history from one particularly long outage. For that node's entities, the meaningful sequence was basically this:

21:08:12   unavailable
08:55:50   real value after reconnect

There was no hidden last-known value waiting in the middle for the automation to retrieve.

The capture needed to happen when the ESP came back.

Capture on reconnect instead

The useful point is when the reset-reason entity returns from unavailable and has a real value again. That is now the capture point.

triggers:
  - trigger: state
    entity_id: sensor.esp32_livingroom_reset_reason
    from: "unavailable"

actions:
  - wait_template: >-
      {{ has_value("sensor.esp32_livingroom_reset_reason") }}
    timeout: "00:03:00"
    continue_on_timeout: false

  - action: logbook.log
    data:
      name: BLE Proxy
      message: >-
        ESP32 reconnected. Reset reason:
        {{ states("sensor.esp32_livingroom_reset_reason") }}

I separately log when Home Assistant initiates a power cycle. That gives me two pieces of evidence instead of trying to infer everything later from one field:

• Home Assistant says whether it deliberately cycled the power.
• ESPHome says why the ESP thinks it last rebooted.

If Home Assistant pulled the plug and the node comes back reporting a power-on reset, that result is expected. It is not evidence about the original failure.

If the node returned without my recovery automation touching it and reports a watchdog or another reboot cause, that is much more interesting.

Then I discovered four nodes had almost no diagnostics

While sorting this out I found another problem.

I had five ESP32 BLE proxies. Only one had ESPHome's debug component configured. I had added it to that node during an earlier investigation and never rolled the change out to the others.

Then one of the other proxies disappeared for about 36 hours before eventually returning.

That should have been a useful failure to study.

Instead I discovered the device had no reset-reason sensor at all.

Adding one means flashing new firmware. Flashing the device means rebooting it. So I could add the diagnostic tooling, but I could not retroactively recover whatever useful state might have existed before that reboot.

That failure was gone.

The ESPHome side is not complicated:

debug:
  update_interval: 60s

text_sensor:
  - platform: debug
    device:
      name: "ESP32 Office Device Info"

    reset_reason:
      name: "ESP32 Office Reset Reason"

sensor:
  - platform: debug
    free:
      name: "ESP32 Office Heap Free"
      entity_category: diagnostic

    block:
      name: "ESP32 Office Heap Max Block"
      entity_category: diagnostic

    fragmentation:
      name: "ESP32 Office Heap Fragmentation"
      entity_category: diagnostic

    loop_time:
      name: "ESP32 Office Loop Time"
      entity_category: diagnostic

One easy thing to miss is the standalone debug: section at the top. The debug sensor platforms depend on the component being configured, and the compile error does not spell that out clearly.

I am keeping the heap diagnostics now as well.

reset_reason tells me why the ESP says it last rebooted. Heap free, largest free block, fragmentation and loop timing give me context about what the device was doing while it was still alive.

I would not look at a rising fragmentation number and declare I had found the crash cause. It is another signal. ESPHome exposes fragmentation directly, so there is no need to infer it from a shrinking largest-free-block the way older guides do. If memory behavior changes consistently before failures, that gives me something concrete to investigate instead of guessing about power supplies, Wi-Fi and bad boards.

More importantly, those diagnostics are now on all five nodes.

The best time to add post-mortem instrumentation is before the post mortem.

Auto-healing changes the system you are debugging

That was the piece I had missed.

I treated the watchdog as something outside the failure: ESP breaks, watchdog fixes ESP.

But once the watchdog resets the device, clears state, changes counters or destroys a crash artifact, it is no longer an outside observer.

It is part of the system.

That does not mean auto-recovery is bad. I still want Home Assistant to recover devices that are genuinely stuck. It means recovery has a cost.

For each watchdog now, I want to know three things.

What evidence disappears when this runs?

If the answer is reset reason, crash data or another post-mortem field, I want that understood before I automate the reset.

Can I capture the useful information at a point where it is actually available?

For this ESP32 problem, that point turned out to be reconnect, not the moment before the power cut. The instinct to log a field just before you clear it is the wrong instinct, because by then Home Assistant has usually already lost access to the value.

How long should I observe before I interfere?

A ten-minute outage and a sixty-minute outage do not have to mean the same thing operationally. The recovery threshold is also an observation window.

Some things should still be restarted immediately. Some should be given time to fail in a useful way. And some failures are better surfaced to a person than automatically hidden.

---

My dashboard looked healthier when the watchdog was aggressive. The system was not healthier. I had just become much better at making the evidence disappear.

Sometimes "fix it faster" and "figure out why it broke" really are opposite goals.