Summary
Deleting one Kea DHCPv4 subnet caused every subnet after it in the list to be assigned a
new subnet-id. Kea's lease database keys on subnet_id, so on the following start all
leases in the shifted subnets failed the sanity check and were discarded. Clients received
different addresses.
Nothing failed at the time of the change — the subnet was removed, Kea restarted cleanly,
the GUI reported success. The consequences appeared later, as leases expired and clients
rebound to different addresses. In our case this silently broke a two-node cluster whose
nodes take their addresses from DHCP.
A second, possibly unrelated symptom followed about two hours later on the same host and is
documented separately at the end.
Below is what we did, in what order, and what the system recorded. We have deliberately not
attempted to diagnose the code or propose a fix.
The same failure outside OPNsense
We have hit exactly this failure mode before, running Kea directly on another platform
without OPNsense. In that environment the Kea configuration is maintained by hand, and the
remedy was simply to assign every subnet a fixed subnet-id and never change it again.
Since doing that, adding or removing subnets has had no effect on the remaining ones and the
lease database has stayed valid across restarts.
We mention it up front only as an operational data point: the same symptom, and pinning the
ids made it go away for good.
What we did
The task was to move a network from a tagged VLAN interface to the untagged (native)
interface. On this hardware the bridge member interfaces carry different addresses, so the
gateway address had to be provided by a CARP virtual IP. Creating that VIP required a free
CARP vhid, and the one we wanted was already held by a network that was no longer in use.
To free it we removed that unused network, which deleted its DHCPv4 subnet.
The deleted subnet had no active DHCP clients. Removing it is what invalidated the
leases of the subnets listed after it — none of the six rejected leases belonged to the
subnet that was deleted; all six belonged to two other, actively used subnets. From the
operator's point of view, cleaning up an unused network took down clients in networks that
had not been touched.
The remainder of the session was the rest of that migration: creating the VIP, adding a
VLAN, reassigning interfaces, deleting the VLAN, and configuring CARP.
Configuration changelog
Taken from the OPNsense configuration revision history in /conf/backup/ and confirmed
against /var/log/audit/. All times CEST (Europe/Warsaw, +0200); the operator's address
is redacted.
2026-08-03 13:05:50 | root@<internal> | /api/kea/dhcpv4/del_subnet/<uuid> made changes
2026-08-03 13:05:53 | root@<internal> | /api/kea/dhcpv4/set made changes
2026-08-03 13:23:42 | root@<internal> | /api/interfaces/vip_settings/add_item/ made changes
2026-08-03 13:27:00 | root@<internal> | /api/interfaces/vip_settings/set_item/<uuid> made changes
2026-08-03 15:21:55 | root@<internal> | /api/interfaces/vlan_settings/add_item/ made changes
2026-08-03 15:24:03 | root@<internal> | /interfaces_assign.php made changes
2026-08-03 15:24:37 | root@<internal> | /api/interfaces/vlan_settings/del_item/<uuid> made changes
2026-08-03 15:25:13 | root@<internal> | /api/interfaces/vip_settings/set_item/<uuid> made changes
2026-08-03 15:57:10 | root@<internal> | /api/interfaces/vip_settings/set_item/<uuid> made changes
2026-08-03 15:57:39 | root@<internal> | /api/interfaces/vip_settings/add_item/ made changes
The relevant action is the first one: a single DHCPv4 subnet was deleted. It sat at
position 3 of 7 in the list. Nothing else about Kea was touched — the configuration
diff between the previous revision and 13:05:50 contains only the removal of that one
<subnet4> block, and the subnet count stayed at 6 for the rest of the session.
Logs
Addresses and VLAN interface names below are anonymised; structure, ordering, ids and
message text are verbatim. The mapping is consistent throughout this report:
| Subnet in report |
subnet-id before |
after |
Interface in report |
10.10.1.0/24 |
1 |
1 |
vtnet0_vlan101 |
10.10.2.0/24 |
2 |
2 |
vtnet0_vlan102 |
10.10.3.0/24 — deleted |
3 |
— |
vtnet0_vlan103 |
10.10.4.0/24 |
4 |
3 |
vtnet0_vlan104 |
10.10.5.0/24 |
5 |
4 |
vtnet0_vlan105 |
10.10.6.0/24 |
6 |
5 |
vtnet0_vlan106 |
10.10.7.96/27 |
7 |
6 |
vtnet0_vlan107 |
All lease sanity failures
Six in total. Kea loads leases from two files (kea-leases4.csv.2 and kea-leases4.csv)
and rejects the affected leases on both passes. Every reported shift matches the table
above — 10.10.6.x went from 6 to 5, 10.10.5.x went from 5 to 4:
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="37"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.6.22 with subnet-id 6 failed subnet-id checks (the lease should have subnet-id 5).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="38"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.5.22 with subnet-id 5 failed subnet-id checks (the lease should have subnet-id 4).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="40"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.6.22 with subnet-id 6 failed subnet-id checks (the lease should have subnet-id 5).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="41"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.5.22 with subnet-id 5 failed subnet-id checks (the lease should have subnet-id 4).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="42"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.5.26 with subnet-id 5 failed subnet-id checks (the lease should have subnet-id 4).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="43"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.6.22 with subnet-id 6 failed subnet-id checks (the lease should have subnet-id 5).
Five leases survived the load:
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="44"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 5 leases, extended info sanity checks modified 0 / updated 0 leases and 0 leases have relay or remote id
Full Kea restart block at 13:05:53
Complete sequence, from shutdown of the previous process (pid 48106) to DHCP4_STARTED of
the new one (pid 71948). Note DHCP4_CONFIG_COMPLETE ... added IPv4 subnets: 6 — one fewer
than before the change, as expected — and the DHCPSRV_CFGMGR_NEW_SUBNET4 lines showing
the surviving subnets in their new order.
54 lines — click to expand
2026-08-03T13:05:53+02:00 kea-dhcp4 48106 - [meta sequenceId="1"] INFO [kea-dhcp4.commands] COMMAND_RECEIVED Received command 'shutdown'
2026-08-03T13:05:53+02:00 kea-dhcp4 48106 - [meta sequenceId="2"] INFO [kea-dhcp4.dhcp4] DHCP4_SHUTDOWN server shutdown
2026-08-03T13:05:53+02:00 kea-dhcp4 48106 - [meta sequenceId="3"] INFO [kea-dhcp4.host-cmds-hooks] HOST_CMDS_DEINIT_OK unloading Host Commands hooks library successful
2026-08-03T13:05:53+02:00 kea-dhcp4 48106 - [meta sequenceId="4"] INFO [kea-dhcp4.lease-cmds-hooks] LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful
2026-08-03T13:05:53+02:00 kea-dhcp4 48106 - [meta sequenceId="5"] INFO [kea-dhcp4.hooks] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_host_cmds.so successfully closed
2026-08-03T13:05:53+02:00 kea-dhcp4 48106 - [meta sequenceId="6"] INFO [kea-dhcp4.hooks] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="7"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="8"] WARN [kea-dhcp4.dhcp4] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="9"] INFO [kea-dhcp4.hooks] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully closed
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="10"] INFO [kea-dhcp4.hooks] HOOKS_LIBRARY_CLOSED hooks library /usr/local/lib/kea/hooks/libdhcp_host_cmds.so successfully closed
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="11"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.1.0/24 with params: valid-lifetime=4000
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="12"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.2.0/24 with params: valid-lifetime=4000
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="13"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.4.0/24 with params: valid-lifetime=4000
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="14"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.5.0/24 with params: valid-lifetime=4000
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="15"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.6.0/24 with params: valid-lifetime=4000
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="16"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_NEW_SUBNET4 a new subnet has been added to configuration: 10.10.7.96/27 with params: valid-lifetime=4000
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="17"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="18"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="19"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_ADD_IFACE listening on interface vtnet0_vlan104
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="20"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_ADD_IFACE listening on interface vtnet0_vlan102
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="21"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_ADD_IFACE listening on interface vtnet0_vlan105
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="22"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_ADD_IFACE listening on interface vtnet0_vlan107
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="23"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_ADD_IFACE listening on interface vtnet0_vlan101
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="24"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_ADD_IFACE listening on interface vtnet0_vlan106
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="25"] INFO [kea-dhcp4.commands] COMMAND_ACCEPTOR_START Starting to accept connections via unix domain socket bound to /var/run/kea/kea4-ctrl-socket
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="26"] INFO [kea-dhcp4.lease-cmds-hooks] LEASE_CMDS_INIT_OK loading Lease Commands hooks library successful
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="27"] INFO [kea-dhcp4.hooks] HOOKS_LIBRARY_LOADED hooks library /usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully loaded
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="28"] INFO [kea-dhcp4.host-cmds-hooks] HOST_CMDS_INIT_OK loading Host Commands hooks library successful
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="29"] INFO [kea-dhcp4.hooks] HOOKS_LIBRARY_LOADED hooks library /usr/local/lib/kea/hooks/libdhcp_host_cmds.so successfully loaded
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="30"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_MGR_BACKENDS_REGISTERED the following lease backend types are available: memfile
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="31"] INFO [kea-dhcp4.hosts] HOSTS_BACKENDS_REGISTERED the following host backend types are available:
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="32"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_FORENSIC_BACKENDS_REGISTERED the following forensic backend types are available:
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="33"] INFO [kea-dhcp4.database] CONFIG_BACKENDS_REGISTERED the following config backend types are available:
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="34"] INFO [kea-dhcp4.dhcp4] DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: added IPv4 subnets: 6; DDNS: disabled
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="35"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_DB opening memory file lease database: persist=true type=memfile universe=4
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="36"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file /var/db/kea/kea-leases4.csv.2
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="37"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.6.22 with subnet-id 6 failed subnet-id checks (the lease should have subnet-id 5).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="38"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.5.22 with subnet-id 5 failed subnet-id checks (the lease should have subnet-id 4).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="39"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file /var/db/kea/kea-leases4.csv
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="40"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.6.22 with subnet-id 6 failed subnet-id checks (the lease should have subnet-id 5).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="41"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.5.22 with subnet-id 5 failed subnet-id checks (the lease should have subnet-id 4).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="42"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.5.26 with subnet-id 5 failed subnet-id checks (the lease should have subnet-id 4).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="43"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_LEASE_SANITY_FAIL The lease 10.10.6.22 with subnet-id 6 failed subnet-id checks (the lease should have subnet-id 5).
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="44"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 5 leases, extended info sanity checks modified 0 / updated 0 leases and 0 leases have relay or remote id
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="45"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to 3600 sec
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="46"] WARN [kea-dhcp4.dhcpsrv] DHCPSRV_MULTIPLE_RAW_SOCKETS_PER_IFACE current configuration will result in opening multiple broadcast capable sockets on some interfaces and some DHCP messages may be duplicated
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="47"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.10.1.0/24
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="48"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.10.2.0/24
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="49"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.10.4.0/24
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="50"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.10.5.0/24
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="51"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.10.6.0/24
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="52"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_CFGMGR_USE_ALLOCATOR using the iterative allocator for V4 leases in subnet 10.10.7.96/27
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="53"] INFO [kea-dhcp4.dhcp4] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 1, queue size: 64
2026-08-03T13:05:53+02:00 kea-dhcp4 71948 - [meta sequenceId="54"] INFO [kea-dhcp4.dhcp4] DHCP4_STARTED Kea DHCPv4 server version 3.0.3 started
Client side
The affected hosts are Debian with ISC dhclient. Lease history for one of them shows the
address changing after the event, with no client-side configuration change:
/var/lib/dhcp/dhclient.<iface>.leases
fixed-address 10.10.6.22; 3 occurrences, all before 13:05
fixed-address 10.10.6.20; 5 occurrences, all after
The address released by that host was subsequently leased to a different machine, so
reverting the client to its previous address was not possible.
Impact
Both nodes of a two-node cluster take their ring addresses from DHCP pools. Both were
renumbered by this single subnet deletion, their cluster configuration silently stopped
matching reality, and quorum was lost. Nothing in the cluster logs pointed back at the
firewall; the connection was only found by correlating timestamps with the OPNsense
revision history.
Lease time is the OPNsense default (valid-lifetime 4000 s, ~67 minutes), so from the
operator's perspective the failure appeared roughly an hour after the change that caused it.
Environment
OPNsense 26.4.1p1_3 (amd64)
FreeBSD 14.3-RELEASE-p16, stable/26.1-n272152-9b6eef552f24
kea 3.0.3
Two-node CARP HA pair; Kea runs on the master only. Everything above happened on the master
and is not related to HA.
Second symptom — kea-dhcp4 stopped serving while appearing healthy
Reported here because it happened on the same host within the same session; it may well be
a separate problem, possibly upstream.
Roughly two hours after the subnet deletion, kea-dhcp4 stopped answering DHCP entirely
while remaining alive. One thread spun at 100% CPU, the other five stayed idle:
PID LWP STAT TIME %CPU COMMAND
71948 103703 R 71:05.83 100.0 /usr/local/sbin/kea-dhcp4 -c .../kea-dhcp4.conf
71948 183746 I 0:00.00 0.0
71948 183747 I 0:00.00 0.0
71948 183748 I 0:00.00 0.0
71948 183749 I 0:00.00 0.0
71948 183750 I 0:00.02 0.0
The daemon kept its UDP/67 sockets bound on every interface and the firewall answered ICMP
and served all other services normally, so from the outside nothing looked wrong for about
80 minutes while no client could obtain or renew a lease.
Log line count per time window
The silence is total — not reduced traffic, but nothing at all:
13:05 -> 54 (restart described above)
15:1x -> 51 (last served transactions)
15:2x -> 0
15:3x -> 0
16:0x -> 2 (Lease File Cleanup only)
16:39 -> 100 (manual restart)
16:40 -> 44 (clients recovering)
The last transaction served was at 15:18:45, the next entry of any kind is internal
housekeeping at 16:05:54:
2026-08-03T15:18:45+02:00 kea-dhcp4 71948 - [meta sequenceId="5"] INFO [kea-dhcp4.packets] DHCP4_PACKET_SEND ... trying to send packet DHCPACK (type 5) from 10.10.1.2:67 to 10.10.1.21:68 on interface vtnet0_vlan101
2026-08-03T16:05:54+02:00 kea-dhcp4 71948 - [meta sequenceId="1"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_LFC_START starting Lease File Cleanup
2026-08-03T16:05:54+02:00 kea-dhcp4 71948 - [meta sequenceId="2"] INFO [kea-dhcp4.dhcpsrv] DHCPSRV_MEMFILE_LFC_EXECUTE executing Lease File Cleanup using: /usr/local/sbin/kea-lfc -4 -x /var/db/kea/kea-leases4.csv.2 ...
A client renewal was due at approximately 15:32 and does not appear at all, so the daemon
had stopped processing before then.
Correlation worth noting
The process id did not change (71948 throughout, started 13:05), so Kea was never restarted
during this period — it stopped serving in place. The window in which it stopped responding,
between 15:18:45 and the missing 15:32 renewal, contains these operations from the changelog
above:
15:21:55 /api/interfaces/vlan_settings/add_item/
15:24:03 /interfaces_assign.php
15:24:37 /api/interfaces/vlan_settings/del_item/<uuid>
These were deliberate operator actions, not background activity: a VLAN interface was
created, interfaces were reassigned, and a VLAN interface was deleted — the continuation of
the migration described at the top of this report. They were performed from the GUI while
kea-dhcp4 was running and holding raw sockets bound per interface
(DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw, six DHCPSRV_CFGMGR_ADD_IFACE
entries in the startup block above).
Note that these operations did not change the Kea configuration in any way — the DHCPv4
subnet list is byte-identical before and after them, and the subnet count stays at 6. So
this symptom is not a repeat of the first one; whatever stopped the daemon, it was not a
Kea reconfiguration.
The daemon also runs with DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 1, queue size: 64, so a single blocked worker stops all processing.
We cannot say whether removing an interface from under a running daemon is the cause, only
that it is what happened inside the window.
service kea restart restored service immediately and the new process has been stable
since. We did not capture a backtrace before restarting, so we cannot offer one — if this is
worth pursuing and it recurs, tell us what to collect and we will.
Summary
Deleting one Kea DHCPv4 subnet caused every subnet after it in the list to be assigned a
new
subnet-id. Kea's lease database keys onsubnet_id, so on the following start allleases in the shifted subnets failed the sanity check and were discarded. Clients received
different addresses.
Nothing failed at the time of the change — the subnet was removed, Kea restarted cleanly,
the GUI reported success. The consequences appeared later, as leases expired and clients
rebound to different addresses. In our case this silently broke a two-node cluster whose
nodes take their addresses from DHCP.
A second, possibly unrelated symptom followed about two hours later on the same host and is
documented separately at the end.
Below is what we did, in what order, and what the system recorded. We have deliberately not
attempted to diagnose the code or propose a fix.
The same failure outside OPNsense
We have hit exactly this failure mode before, running Kea directly on another platform
without OPNsense. In that environment the Kea configuration is maintained by hand, and the
remedy was simply to assign every subnet a fixed
subnet-idand never change it again.Since doing that, adding or removing subnets has had no effect on the remaining ones and the
lease database has stayed valid across restarts.
We mention it up front only as an operational data point: the same symptom, and pinning the
ids made it go away for good.
What we did
The task was to move a network from a tagged VLAN interface to the untagged (native)
interface. On this hardware the bridge member interfaces carry different addresses, so the
gateway address had to be provided by a CARP virtual IP. Creating that VIP required a free
CARP vhid, and the one we wanted was already held by a network that was no longer in use.
To free it we removed that unused network, which deleted its DHCPv4 subnet.
The deleted subnet had no active DHCP clients. Removing it is what invalidated the
leases of the subnets listed after it — none of the six rejected leases belonged to the
subnet that was deleted; all six belonged to two other, actively used subnets. From the
operator's point of view, cleaning up an unused network took down clients in networks that
had not been touched.
The remainder of the session was the rest of that migration: creating the VIP, adding a
VLAN, reassigning interfaces, deleting the VLAN, and configuring CARP.
Configuration changelog
Taken from the OPNsense configuration revision history in
/conf/backup/and confirmedagainst
/var/log/audit/. All times CEST (Europe/Warsaw,+0200); the operator's addressis redacted.
The relevant action is the first one: a single DHCPv4 subnet was deleted. It sat at
position 3 of 7 in the list. Nothing else about Kea was touched — the configuration
diff between the previous revision and 13:05:50 contains only the removal of that one
<subnet4>block, and the subnet count stayed at 6 for the rest of the session.Logs
Addresses and VLAN interface names below are anonymised; structure, ordering, ids and
message text are verbatim. The mapping is consistent throughout this report:
subnet-idbefore10.10.1.0/24vtnet0_vlan10110.10.2.0/24vtnet0_vlan10210.10.3.0/24— deletedvtnet0_vlan10310.10.4.0/24vtnet0_vlan10410.10.5.0/24vtnet0_vlan10510.10.6.0/24vtnet0_vlan10610.10.7.96/27vtnet0_vlan107All lease sanity failures
Six in total. Kea loads leases from two files (
kea-leases4.csv.2andkea-leases4.csv)and rejects the affected leases on both passes. Every reported shift matches the table
above —
10.10.6.xwent from 6 to 5,10.10.5.xwent from 5 to 4:Five leases survived the load:
Full Kea restart block at 13:05:53
Complete sequence, from shutdown of the previous process (pid 48106) to
DHCP4_STARTEDofthe new one (pid 71948). Note
DHCP4_CONFIG_COMPLETE ... added IPv4 subnets: 6— one fewerthan before the change, as expected — and the
DHCPSRV_CFGMGR_NEW_SUBNET4lines showingthe surviving subnets in their new order.
54 lines — click to expand
Client side
The affected hosts are Debian with ISC
dhclient. Lease history for one of them shows theaddress changing after the event, with no client-side configuration change:
The address released by that host was subsequently leased to a different machine, so
reverting the client to its previous address was not possible.
Impact
Both nodes of a two-node cluster take their ring addresses from DHCP pools. Both were
renumbered by this single subnet deletion, their cluster configuration silently stopped
matching reality, and quorum was lost. Nothing in the cluster logs pointed back at the
firewall; the connection was only found by correlating timestamps with the OPNsense
revision history.
Lease time is the OPNsense default (
valid-lifetime4000 s, ~67 minutes), so from theoperator's perspective the failure appeared roughly an hour after the change that caused it.
Environment
Two-node CARP HA pair; Kea runs on the master only. Everything above happened on the master
and is not related to HA.
Second symptom —
kea-dhcp4stopped serving while appearing healthyReported here because it happened on the same host within the same session; it may well be
a separate problem, possibly upstream.
Roughly two hours after the subnet deletion,
kea-dhcp4stopped answering DHCP entirelywhile remaining alive. One thread spun at 100% CPU, the other five stayed idle:
The daemon kept its UDP/67 sockets bound on every interface and the firewall answered ICMP
and served all other services normally, so from the outside nothing looked wrong for about
80 minutes while no client could obtain or renew a lease.
Log line count per time window
The silence is total — not reduced traffic, but nothing at all:
The last transaction served was at 15:18:45, the next entry of any kind is internal
housekeeping at 16:05:54:
A client renewal was due at approximately 15:32 and does not appear at all, so the daemon
had stopped processing before then.
Correlation worth noting
The process id did not change (71948 throughout, started 13:05), so Kea was never restarted
during this period — it stopped serving in place. The window in which it stopped responding,
between 15:18:45 and the missing 15:32 renewal, contains these operations from the changelog
above:
These were deliberate operator actions, not background activity: a VLAN interface was
created, interfaces were reassigned, and a VLAN interface was deleted — the continuation of
the migration described at the top of this report. They were performed from the GUI while
kea-dhcp4was running and holding raw sockets bound per interface(
DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type raw, sixDHCPSRV_CFGMGR_ADD_IFACEentries in the startup block above).
Note that these operations did not change the Kea configuration in any way — the DHCPv4
subnet list is byte-identical before and after them, and the subnet count stays at 6. So
this symptom is not a repeat of the first one; whatever stopped the daemon, it was not a
Kea reconfiguration.
The daemon also runs with
DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 1, queue size: 64, so a single blocked worker stops all processing.We cannot say whether removing an interface from under a running daemon is the cause, only
that it is what happened inside the window.
service kea restartrestored service immediately and the new process has been stablesince. We did not capture a backtrace before restarting, so we cannot offer one — if this is
worth pursuing and it recurs, tell us what to collect and we will.