# Testing the Recovery Firmware, Rollback and Crash-Loop Fallback

This guide covers testing on hardware of:

- the EMS-ESP Recovery firmware in the factory `boot` partition (16MB boards only);
- installing the recovery from the EMS-ESP WebUI;
- the bootloader rollback of a new firmware that fails to start;
- the fallback to the recovery after repeated crashes;
- Ethernet and static IP support in the recovery.

It's written for an ESP32 16MB board with PSRAM (`s_16M_P`, e.g. a BBQKees E32V2). Other 16MB boards (`s_16M`, `s3_16M_P`) follow the same steps with their matching PlatformIO envs; Ethernet is E32V2 only. Keep a USB serial cable connected with a serial monitor open throughout, as most of the checks are log lines.

## 0. Preparation

Build everything:

```bash
pio run -e s_16M_P -e crashtest_s_16M_P -e recovery_s_16M_P
```

The firmware files are written to `build/firmware/`:

| File | What it is |
| --- | --- |
| `EMS-ESP-<version>-ESP32-16MB+.bin` | EMS-ESP |
| `EMS-ESP-CrashTest-<version>-ESP32-16MB+.bin` | crash test build |
| `EMS-ESP-Recovery-<version>-ESP32-16MB+.bin` | the recovery |

Each has a matching `.md5` file.

Then:

1. Run the unit tests with `pio run -e native-test -t exec`. All tests should pass.
2. Optionally, check the recovery UI against mock data first: `cd interface && pnpm recovery`, then open <http://localhost:3001>.
3. Open the serial monitor with `pio device monitor -e s_16M_P`. Recovery log lines start with `[recovery]`.

The crash counter is kept in RTC memory. It survives crashes and software restarts but not a power cycle, so don't unplug the board during the crash tests.

## 1. Baseline: EMS-ESP over serial, no recovery

1. Flash with `pio run -e s_16M_P -t upload`. This also writes the current bootloader, which has rollback enabled. Boards flashed with an older bootloader don't roll back until they've been flashed over serial once.
2. In the WebUI, open **Settings → Version**. Check that:
   - the **Recovery** row shows "not installed";
   - the firmware partition list doesn't include `boot`.
3. Wait more than 2 minutes. "New firmware confirmed as working" should **not** appear in the log, because a serial flash isn't in the pending-verify state.

## 2. Install the recovery from the EMS-ESP WebUI

1. On **Settings → Version**, upload `EMS-ESP-Recovery-…bin`. Expect:
   - a "recovery installed" toast;
   - **no** restart;
   - the Recovery row now shows the recovery version;
   - the install logged.
2. Repeat, uploading the `.md5` first and then the `.bin`. The MD5 check should pass.
3. Negative tests:
   - Upload the recovery built for another chip, e.g. the ESP32-S3 file on this ESP32. It should be rejected, and nothing should be written.
   - Start a recovery upload and close the browser tab halfway through. EMS-ESP should keep running and the Recovery row should show "not installed". Reinstall it properly afterwards.
4. Restart EMS-ESP normally. It should boot EMS-ESP again, not the recovery, because installing the recovery doesn't change the boot partition.

## 3. Enter the recovery on request

1. Enter the recovery in one of these ways:
   - in the console (telnet or serial) as admin, run `restart boot`;
   - long-press the button.
2. On the serial monitor, check that:
   - `[recovery] EMS-ESP Recovery v…` is logged;
   - LittleFS is mounted;
   - the access point has started;
   - Ethernet or WiFi connects.
3. Join the access point. Its name is the AP SSID plus `-recovery` (default `ems-esp-recovery`), and it uses the AP password (default `ems-esp-neo`). Open <http://192.168.4.1>; a captive portal page may open by itself. Alternatively, use the device's normal IP or `http://<hostname>.local` from the LAN.
4. Sign in with an EMS-ESP admin account (default `admin`/`admin`). A non-admin user must be refused.
5. On the dashboard, check that:
   - there's **no** red reason box, because the restart was requested;
   - `Recovery (boot)` has the "running" chip;
   - the installed firmware shows the correct version;
   - the System section shows the chip, flash, MAC, access point, and the active network (Ethernet or WiFi) with its IP.
6. Click **Restart**. It should come back into the recovery, still without a reason box, since the reason is only shown once.
7. Click **Start** on the EMS-ESP slot. EMS-ESP should boot, and the Version page should still show the recovery version.

## 4. Upload firmware from the recovery

1. Enter the recovery again with `restart boot`.
2. Check the "upload target" chip:
   - if there's an empty slot, that slot is chosen;
   - if both slots hold a firmware, the one with the older install date is chosen, so the newest firmware is kept.
3. Upload the `.md5`, then the EMS-ESP `.bin`. EMS-ESP should start from that slot. About 2 minutes after it boots, "New firmware confirmed as working" should appear in the log.
4. The recovery must refuse these uploads:
   - a recovery `.bin`: a 409 with a message that it can't install itself;
   - a settings `.json`: a 406;
   - an EMS-ESP `.bin` for another chip: rejected as incompatible.
5. In **Write to**, pick the other slot and upload there. That should also work.

## 5. Bootloader rollback (crash test mode 1)

`platformio.ini` ships with `-D EMSESP_CRASH_TEST=1`.

1. Boot a working EMS-ESP and let it run for at least 2 minutes so it's confirmed.
2. Upload `EMS-ESP-CrashTest-…bin` from the **WebUI**, not over serial.
3. On the serial monitor you should see:
   - `Crash test, aborting`, a panic, and a restart;
   - the bootloader rolling back to the previous slot, which then starts normally.
4. In the EMS-ESP log (serial, or **Status → System Log**) you should see:
   - `Restarted after a crash (1 in a row)`;
   - `Firmware in partition appX failed to start and was rolled back`.
5. Enter the recovery with `restart boot`. Check that:
   - the red box says the firmware in appX failed to start;
   - that slot has the "failed to start" chip;
   - that slot is the upload target.
6. Click **Start** on the failed slot. The dialog should show a warning; cancel it.
7. Start the good slot.

If step 3 ends in a crash loop instead of a rollback, the bootloader on the board doesn't have rollback enabled. Flash over serial as in section 1, step 1.

## 6. Crash-loop fallback to the recovery (crash test mode 2)

1. In `platformio.ini`, change `-D EMSESP_CRASH_TEST=1` to `=2`, then run `pio run -e crashtest_s_16M_P`.
2. Upload the crash test build from the WebUI. It runs normally.
3. Wait for "New firmware confirmed as working", about 2 minutes after boot.
4. Restart it from the WebUI. It should crash 5 times in a row, logging the "N in a row" count each time.
5. After the 5th crash you should see `Crashed 5 times in a row, restarting into the boot partition`, and the recovery starts.
6. In the recovery:
   - the red box should say "EMS-ESP crashed repeatedly and switched to the recovery";
   - after restarting the recovery once, the box should be gone.
7. Clean up: in **Write to**, pick the crash test slot **by hand** and upload a good EMS-ESP, or Start the good slot.
   - Switching to the recovery wipes the per-slot rollback states, so the default upload target may be the slot with the good firmware.
8. Change the flag back to `=1`.

Optional, without a recovery installed:

1. Erase the boot partition with `esptool --chip esp32 erase-region 0x10000 0x480000`. Older esptool versions use `erase_region`.
2. Repeat steps 2 to 4. It should keep crashing past 5, and needs to be reflashed over serial.

## 7. Network in the recovery

| Test | Setup | Expected in the recovery |
| --- | --- | --- |
| WiFi with static IP | Enable a static IP in the EMS-ESP Network settings, then `restart boot` | Same IP as EMS-ESP, and the WiFi row ends with ", static IP" |
| Ethernet | Cable plugged in, then `restart boot` | Log shows "Ethernet started" and "Ethernet connected, IP …". The Ethernet row shows "connected", and WiFi shows "not used, Ethernet is connected" |
| Ethernet with static IP | As above, with a static IP set | Ethernet uses the static IP and WiFi isn't started |
| No cable | Unplugged, then `restart boot` | Ethernet shows "no cable connected" and WiFi shows "waiting for Ethernet". After about 15 seconds the log shows "No Ethernet connection" and WiFi connects |
| Cable plugged in later | Plug in after WiFi has joined | Ethernet gets an IP as well, and WiFi stays connected |
| Access point | Any of the above | `ems-esp-recovery` is always reachable on 192.168.4.1 |

## 8. Other checks

- **Online update during the pending window:** after a WebUI upgrade, start an online update from the Version page within 2 minutes. It should install, not fail.
- **4MB boards:** flash `s_4M` and check that:
  - there's no Recovery row;
  - a normal WebUI upgrade still gets confirmed after 2 minutes.
- **Translations:** switch the WebUI language and check the Recovery row and toast.
- **CI:** run the dev release workflow manually (`workflow_dispatch`) and check that the `EMS-ESP-Recovery-*` files for all three 16MB boards are among the release assets.

## Reset to a clean state

`pio run -e s_16M_P -t upload` restores EMS-ESP at any point. It leaves the recovery in `boot` and the settings untouched.