↓Skip to main content

When "My IP Keeps Changing" Means Your Disk Is Dying

·10 mins
Projects Raspberry-Pi NTP GPS Homelab Linux Chrony Debugging
An iceberg: the small visible tip labelled 'my NTP server's IP keeps changing', with a huge hidden mass below whose deepest layer is a cracked, dying USB flash drive
Table of Contents

TL;DR: My GPS-disciplined NTP server kept changing its IP address. Four layers down, the real problem was a dying USB flash drive silently corrupting my root filesystem. Here’s the full descent — and why “the IP keeps changing” was the least interesting thing wrong.

The failure chain #

   SYMPTOM        "My NTP server's IP keeps changing"   (.225 -> .232 -> .254)
   |
   v
  +--------------------------------------------------------------+
  | Layer 1  DHCP identity churn                                 |
  |          dhclient (-i -I) sends a DUID client-id, not the MAC|
  |          -> DUID not persisted -> new identity, many x/min   |
  +--------------------------------------------------------------+
   |
   v
  +--------------------------------------------------------------+
  | Layer 2  Root filesystem is READ-ONLY (emergency_ro)         |
  |          -> dhclient can't save its DUID                      |
  +--------------------------------------------------------------+
   |
   v
  +--------------------------------------------------------------+
  | Layer 3  ext4 error, "clean with errors", errors=remount-ro  |
  |          an EIO on the disk froze the filesystem read-only    |
  +--------------------------------------------------------------+
   |
   v
  +--------------------------------------------------------------+
  | Layer 4  The USB flash drive is DYING                        |
  |          ~2 MB/s reads, silent bit-rot, corrupt on copy      |
  +--------------------------------------------------------------+

The symptom #

I run a little Raspberry Pi 4 called TimePi — a GPS + PPS stratum 1 NTP server (DietPi, chrony, a GPS on the UART, PPS on GPIO18) that the rest of my network syncs to. If you’ve read Part 1, you’ve met it. It’s supposed to sit quietly in the garage forever.

It had developed an alarming habit: its IPv4 address kept changing on the LAN, even though I’d set a DHCP reservation for it. And not gently — this wasn’t a new address on each reboot. The box had been up for weeks, yet the IP was hopping around multiple times a minute. Run ip addr twice in a row and you’d catch it mid-jump: .225… refresh… .232… refresh… .254. For a box that everything else on the network points at by a fixed address, that’s a problem.

Seemed simple. It was not simple.

Layer 1: the DHCP reservation that wasn’t honoured #

First stop: how is the address assigned? DHCP reservation, wired only, SSH reachable. Good. So why isn’t the router honouring the reservation?

The MAC was rock-stable, so it wasn’t MAC randomisation. But the running DHCP client looked like this:

$ ps -eo args | grep '[d]hclient'
dhclient -4 -v -i -pf ... -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0

Those -i -I flags — DietPi’s defaults — make dhclient identify itself with an RFC 4361 DUID-based client-identifier (DHCP option 61) instead of its MAC. And the DUID file it’s supposed to persist? Empty. So it generated a new identity for essentially every DHCP exchange it made — and this broken client was making them constantly, many times a minute. Each time, the router saw a brand-new device and leased out yet another pool address — sailing right past my MAC-based reservation, over and over.

Root cause found! Except… why couldn’t it persist the DUID?

Layer 2: a read-only root filesystem #

$ mount | grep ' / '
/dev/sda2 on / type ext4 (rw,noatime,lazytime,emergency_ro)

emergency_ro. The kernel had hit a filesystem error and remounted root read-only to protect it (errors=remount-ro). And that closed the loop with Layer 1: with a read-only disk, dhclient could never write its DUID, so every DHCP transaction started fresh with a new identity — a different IP again and again, minute after minute. The wandering IP was just a symptom of a read-only root filesystem.

But why was it read-only?

Layer 3: ext4 corruption #

The superblock told the story:

$ sudo dumpe2fs -h /dev/sda2       # abridged
Filesystem state:  clean with errors
First error:       __ext4_find_entry  (EIO)

An I/O error had corrupted the filesystem, and errors=remount-ro did exactly what it promises — froze the whole thing read-only to stop further damage. The box hadn’t rebooted in weeks, so fsck had never had a chance to run and clear it. Damaged, and stuck that way.

Which begs the real question: what threw an I/O error in the first place?

Layer 4: a dying flash drive #

The rootfs lived on a cheap USB flash stick — and that stick, not DHCP and not ext4, was the villain at the bottom of the iceberg. It was failing. The ext4 error was just the first tremor of a drive quietly rotting from underneath; everything above it was a symptom of a symptom. I didn’t have the full, ugly proof yet — that came later, when I tried to back it up — but this was as deep as the rabbit hole went.

Now I just had to climb back out.

The repair that locked me out #

The fix for a “clean with errors” filesystem is a reboot so fsck can run. So I rebooted.

It didn’t come back.

fsck couldn’t auto-repair the damage and dropped to an emergency maintenance shell — which, by design, doesn’t bring up networking. On a headless box in the garage, that means it’s just… gone. Lesson relearned: never reboot a box with a known-bad root filesystem unless you have console access.

So I pulled the USB stick and brought it to my Mac.

macOS, ext4, and a genuinely obscure quirk #

macOS can’t read ext4, so: brew install e2fsprogs. Then a face-plant that cost me two attempts:

$ sudo e2fsck -fy /dev/rdisk4s2    # "unable to set superblock flags"  (fails)
$ sudo e2fsck -fy /dev/disk4s2     # works perfectly

On macOS, e2fsck couldn’t write to the raw device (/dev/rdisk...) but was fine on the buffered one (/dev/disk...). If you ever repair a Linux disk on a Mac and hit that error, drop the r.

The repair itself was enormous — thousands of cleared directory entries, salvaged directories, orphans reconnected to lost+found, mostly across /usr. A clean second pass confirmed it. Back in the Pi, it booted read-write, and — because the filesystem could finally persist state — it even grabbed its reserved .26.

Actually fixing the IP (the right way) #

Filesystem repaired, but I didn’t want to rely on the DUID persisting on a flaky disk. So I pinned a deterministic DHCP client-identifier — the classic 01 + MAC form that a normal client sends and a MAC reservation matches:

# /etc/dhcp/dhclient.conf
send dhcp-client-identifier 01:xx:xx:xx:xx:xx:xx;

Because it’s a static value in config, it produces the same identity every boot even if the disk stops persisting runtime state. Then I verified it on the wire — no tcpdump on the box, so a tiny Python raw-socket sniffer confirmed the DISCOVER/REQUEST carried 01:MAC and the lease came back as .26. Problem actually solved.

The proof: a backup against the clock #

Back to Layer 4 — this is where I got the proof. While backing things up, the flash read at ~2 MB/s. Healthy USB flash does 20–100+. Then the real tell: an e2image capture came back corrupt even though the device fsck’d clean — meaning the flash was returning bad data on some reads without reporting an error. This stick was on its way out.

Before it fully died, I took a proper, verified backup — and rather than guess what mattered, I diffed the whole filesystem against dpkg’s manifest to find everything not owned by a package (the genuinely custom bits): configs, the two Prometheus exporters, the systemd units, the GPS/PPS overlay in config.txt. That surfaced a gap I’d otherwise have missed (/boot/firmware/dietpi.txt lives on the boot partition, not /etc).

Migrating to an SSD — and the silent corruption trap #

I grabbed a 128 GB USB SSD and plugged it straight into the Pi (so it showed up as /dev/sdb), then cloned the live, repaired system onto it. I went file-level with rsync rather than a block image for two reasons: on a failing disk, rsync announces every file it can’t read, and cloning the running system preserved the exact box — same hostname, SSH host keys, Tailscale identity.

But “just rsync” skips a few steps. First, a fresh partition table and filesystems — a small FAT boot partition and an ext4 root filling the rest:

$ sudo sfdisk /dev/sdb <<'EOF'
label: dos
label-id: 0xa1b2c3d4
/dev/sdb1 : start=8192, size=524288, type=c    # 256 MB FAT (boot/firmware)
/dev/sdb2 : start=532480,            type=83   # ext4 root, rest of disk
EOF
$ sudo mkfs.vfat -F 32 -n boot /dev/sdb1
$ sudo mkfs.ext4 -L rootfs /dev/sdb2

Then the copy. Root and /boot/firmware are separate mounts, so they get separate passes — and the flags matter:

$ sudo mount /dev/sdb2 /mnt/ssd
$ sudo rsync -aHAXx --numeric-ids / /mnt/ssd/
$ sudo mount /dev/sdb1 /mnt/ssd/boot/firmware
$ sudo rsync -rt /boot/firmware/ /mnt/ssd/boot/firmware/

-a archive, -H keep hardlinks, -A/-X preserve ACLs and extended attributes, --numeric-ids so ownership survives — and crucially -x, stay on one filesystem. Without -x, rsync would wander into /proc, /sys, and the separate /boot/firmware mount; with it, root copies cleanly and boot gets its own pass.

Finally, a fresh partition table means new PARTUUIDs. The old stick was 45395152-*; the SSD is a1b2c3d4-* (the label-id I set above). If I don’t tell it, the Pi hunts for the old disk and won’t boot — so I rewrote the two files that reference it:

$ sudo sed -i 's/45395152/a1b2c3d4/g' /mnt/ssd/etc/fstab
$ sudo sed -i 's/45395152/a1b2c3d4/g' /mnt/ssd/boot/firmware/cmdline.txt

So: partition, format, two rsyncs, a two-line PARTUUID fixup — then shut down, pull the stick, and boot from the SSD. Not just rsync, but close.

During that root copy, rsync had flagged 86 I/O errors — files it couldn’t read off the dying flash. Mostly non-critical, and a retry pass recovered the important ones. Job done, I thought.

Except the SSD then segfaulted apt. Huh?

A full integrity scan told the real story:

$ sudo dpkg -V                     # checksum every packaged file
??5??????   /usr/lib/aarch64-linux-gnu/libapt-pkg.so.7.0.0
...                                # ~1,800 files flagged

The 5 is an md5 mismatch. The failing flash hadn’t just thrown read errors — it had handed back garbage of the correct length, which rsync copied without complaint (checksums matched garbage-to-garbage). libapt-pkg was quietly corrupt, so apt died. rsync and fsck won’t catch this; dpkg -V will.

The heal, once I understood it, was clean:

  1. wget a pristine libapt-pkg .deb straight from the Debian pool and swap it in -> apt works again.
  2. Map every damaged file to its package (35 of them), and apt-get install --reinstall the lot -> pristine files from the repo.
  3. dpkg -V -> 0 remaining damage.

The kernel, bootloader, and EEPROM were untouched, so the migration was safe. New PARTUUIDs, fstab/cmdline.txt updated, and the Pi booted off the SSD as the same machine — same hostname, SSH keys, Tailscale identity.

The GPS comes home #

One last scare: after migration, chronyc showed no GPS fix and stratum 2. Panic? No — I’d had the Pi indoors with the antenna unplugged the whole time. Back in the garage, antenna reconnected:

$ gpspipe -w -n 5            # gpsd now reports a 3D fix
$ sudo ppstest /dev/pps0     # clean 1 Hz assert pulses
$ chronyc tracking
Reference ID : 50505300 (PPS)
Stratum      : 1
Leap status  : Normal

Stratum 1, PPS-disciplined, back in business.

What I took away from it #

  • The reported symptom is rarely the disease. “The IP keeps changing” was four layers above the actual fault (a dying disk). Follow the why down.
  • Don’t run a rootfs on cheap USB flash. It doesn’t fail loudly — it corrupts silently.
  • rsync and fsck don’t detect bit-rot that returns as valid-length garbage. dpkg -V does. On Debian, package checksums are a superpower for verifying an install.
  • Cloning off a failing disk carries the corruption with it. Verify the clone, and heal from a clean source (the repo), not the dying original.
  • Pin a deterministic DHCP client-ID if DietPi/dhclient is defeating your reservation with a DUID.
  • Never reboot a box with a known-bad root filesystem without console access.
  • Take the boring backup before you need it — and verify it’s actually complete.

TimePi is now humming along on a proper SSD, holding its reserved address, disciplined by GPS and PPS to the microsecond. The old SanDisk stick is in a drawer as a rollback — and as a small memento of an afternoon that went much deeper than “why does my IP keep changing?”