OPNsense Serial Console Recovery — Single-User Mode¶

Procedure for recovering the Owl gateway (DEC700, FreeBSD) when all remote access is lost and the root password is unknown or has been changed.

Covers the specific failure mode where the serial console presents a login: prompt (not the OPNsense menu) and neither root nor joe passwords work — for example, after an Ansible run incorrectly overwrites user configuration in config.xml.

Why `login:` Instead of the OPNsense Menu¶

On a running OPNsense system, the root user's shell is /usr/local/sbin/opnsense-console. When you log in via serial as root, the console menu appears. The raw login: prompt is normal FreeBSD behavior — it appears at the same TTY, before authentication succeeds.

If you see login: and the password doesn't work, root's password was changed. Common cause: the puzzle.opnsense users Ansible module reconstructed config.xml user entries from inventory, and OPNsense applied the change, updating /etc/master.passwd with a new (or cleared) hash.

Recovery Overview¶

FreeBSD single-user mode drops to a root shell without requiring a password. This is the standard recovery path for a locked-out system. After reaching the shell:

Restore /conf/config.xml from a pre-damage backup (preferred), or
Reset the root password with passwd root and re-enable access

Step-by-Step: Single-User Mode Recovery¶

1. Connect to the serial console¶

From a host with a USB-serial adapter connected to the DEC700:

# macOS (luna or triton)
ls /dev/cu.usbserial-*         # Find device name
screen /dev/cu.usbserial-XXXX 115200
# Disconnect: Ctrl-A then K → Y

Settings: 115200 baud, 8N1, no flow control.

2. Reboot the DEC700¶

If you can't reboot from the console (no shell, password unknown), power-cycle the unit physically. The DEC700 has no IPMI or remote management.

3. Interrupt the boot loader¶

Watch the serial terminal immediately after power-on. The FreeBSD boot loader pauses for a few seconds with a countdown:

  ______               ____   ______ ____
  ...

  Booting [/boot/kernel/kernel]...
  Autoboot in 3 seconds. [Space] to pause

Press spacebar before the countdown reaches 0 to enter the boot loader menu.

If you miss the window: Power-cycle again and watch more carefully. The pause is ~3 seconds. On serial, the text may scroll quickly.

4. Select single-user mode¶

At the boot loader menu:

  1. Boot [multi-user]                (default)
  2. Boot [single-user]
  3. Escape to loader prompt
  4. Reboot

  Option? (default: 1):

Type 2 and press Enter.

If you see the FreeBSD loader prompt (OK) instead of a numbered menu, type:
boot -s

5. Mount the filesystem read-write¶

Single-user mode starts with the root filesystem mounted read-only. At the # prompt:

/sbin/mount -u /          # Remount root read-write
/sbin/mount -a            # Mount all filesystems (including /conf)

Verify:

mount | grep conf         # Should show /conf as rw
ls /conf/config.xml       # Should exist

6. Restore from a pre-run snapshot (preferred)¶

If a pre-damage config.xml is reachable — on triton, a USB drive, or served over HTTP from a LAN host — restore it directly:

Option A — USB drive:

# Insert FAT32 USB drive into DEC700
ls /dev/da*               # Find USB device (e.g., /dev/da0)
mount -t msdosfs /dev/da0s1 /mnt
ls /mnt/                  # Confirm config file is present

# Back up damaged config, restore good one
cp /conf/config.xml /conf/config.xml.damaged-$(date +%Y%m%d)
cp /mnt/config-pre-run-20260221-223129.xml /conf/config.xml
chown root:wheel /conf/config.xml
chmod 640 /conf/config.xml
umount /mnt

Option B — HTTP fetch from a LAN host:

# On luna or triton: serve the snapshot
cd ~/.config/scandora/backups/owl
python3 -m http.server 8080

# On Owl (single-user shell):
fetch http://10.7.1.20:8080/config-pre-run-20260221-223129.xml \
  -o /conf/config.xml.new
cp /conf/config.xml /conf/config.xml.damaged-$(date +%Y%m%d)
mv /conf/config.xml.new /conf/config.xml
chown root:wheel /conf/config.xml
chmod 640 /conf/config.xml

Option C — SCP from triton (if networking works in single-user):

# Start network in single-user mode
dhclient igb0             # Or set static: ifconfig igb0 10.7.0.1 netmask 255.255.0.0

# SCP from triton (triton needs the file first)
# Copy snapshot to triton: scp ~/.config/scandora/backups/owl/config-pre-run-*.xml joe@10.7.1.20:/tmp/
scp joe@10.7.1.20:/tmp/config-pre-run-20260221-223129.xml /conf/config.xml

7. Alternatively: reset root password only¶

If you can't get the pre-damage config but want to restore remote access first:

passwd root
# Enter new password twice
# Choose a strong password and save it immediately in 1Password

Then reboot. You'll get the OPNsense console menu, where you can:

Use option 8 (Shell) to restore config.xml via any method above
Use option 3 (Reset root password) if you change your mind again

8. Reboot normally¶

After restoring the config:

exit
# OR
reboot

The system will complete a normal boot cycle. Watch for the OPNsense menu or wait ~60 seconds for services to initialize before attempting remote access.

9. Verify recovery¶

From luna, after boot completes:

# SSH (ZeroTier)
ssh joe@192.168.194.10 echo "SSH OK"

# API health
curl -sk -u "API_KEY:API_SECRET" \
  https://10.7.0.1/api/diagnostics/interface/getInterfaceNames | python3 -m json.tool

# WAN SSH
ssh joe@46.110.77.34 echo "WAN SSH OK"

Check that joe's web GUI access is restored:

ssh joe@192.168.194.10 "sudo cat /conf/config.xml" | \
  grep -A5 "<user>" | grep -E "name|priv|authorized_keys"

After Recovery: Clear sshlockout¶

If luna's ZeroTier IP was banned by OPNsense's SSH login protection (triggered by repeated failed attempts), clear it from the Owl shell:

# Check what's banned
pfctl -t sshlockout -T show

# Flush the table (removes ALL bans — acceptable after a secured restore)
pfctl -t sshlockout -T flush

# Or remove a specific IP
pfctl -t sshlockout -T delete 192.168.194.236   # luna's ZT IP

After Recovery: Fix IaC Root Cause¶

The Ansible users module (puzzle.opnsense) overwrote joe's config with incomplete inventory data. Before running the users tag against production again:

1. Add all SSH keys to owl.yml¶

# cloud/ansible/inventory/owl.yml
opn_users:
  - username: "joe"
    ssh_keys: |
      ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOddEa4ruGxiqYUWu3ZCHAeO/bQDSYO2i7GCgu/rBP+b joe@luna
      # + triton public key
      # + saturn public key

2. Understand privilege management¶

The priv field (controls web GUI access) may not be supported by puzzle.opnsense. If privileges are stripped by the module, manage them via config.xml instead:

# Verify joe's current priv in config.xml
ssh joe@192.168.194.10 "sudo grep -A10 '<user>' /conf/config.xml | grep priv"

If puzzle.opnsense always clears priv, add a dedicated config.xml task to set it back — or use --skip-tags users in production runs until the root cause is fully understood.

3. Pre-flight checklist before future full runs¶

# Always confirm the pre-run snapshot exists
ls -la ~/.config/scandora/backups/owl/config-pre-run-*.xml | tail -3

# Run users tag standalone in --check mode first
./scripts/run-opnsense.sh owl --tags users --check

# Review what would change before applying

Reference: Pre-Damage Snapshots¶

run-opnsense.sh saves a snapshot before every production run:

~/.config/scandora/backups/owl/config-pre-run-YYYYMMDD-HHMMSS.xml

Listing available snapshots:

ls -lt ~/.config/scandora/backups/owl/config-pre-run-*.xml | head -10

The snapshot from 2026-02-21 22:31:29 (config-pre-run-20260221-223129.xml) predates the damage caused by the 2026-02-21 production run.