Root cause: peers were keyed only by (IP, listen port). An IPv6 host holds several addresses at once, so one node appeared as several peers and could not recognise its own addresses.
Protocol:
- Added a random per-run node identity (localNodeId), advertised as a length-guarded trailing field in HELLO and ACK_HELLO — backwards compatible with peers that omit it
- Added peerNodeId to tcp_connection_t; peers are now identified by this rather than by an endpoint
- ACK_HELLO is now sent before any decision to drop the connection, so a rejected dialer learns whose address it reached instead of retrying forever
Self-connection:
- Identity match → close the connection and record that endpoint permanently as our own
- Self-endpoint set preemptively seeded from getifaddrs() at startup, so a node knows its own addresses before ever dialing one
- Node_ConnectPeer refuses self endpoints, which also kills the echo-back chain that could exhaust connection slots
Duplicate connections and churn:
- Dedup by identity per direction — one inbound and one outbound per physical peer, regardless of how many addresses it has
- Moved dial history out of the peer table, so striking a peer no longer resets its connect-retry cooldown (this was the loop engine: strike → re-learn via gossip → redial next tick)
- Node_HasOtherInboundFrom / Node_HasLiveConnectionTo now ignore connections that are already tearing down, and match on identity as well as endpoint
Gossip hygiene:
- Never hand a peer its own other addresses in a PEERS reply (matched on identity, not just the socket address)
- Reject unusable endpoints: link-local without scope id, unspecified, multicast, site-local (loopback stays allowed for local testing)
- Normalise IPv4-mapped IPv6 so one host cannot occupy two entries
Two bugs found along the way:
- All nodes drew the same identity — random_eight_byte() comes from srand(time(NULL)), so processes started in the same second produced identical values. Added random_secure_eight_byte() (/dev/urandom) for the identity, and mixed the pid into the seed so connection IDs stop colliding too
- Identity dedup initially left inbound-only nodes mute — broadcasts traverse outbound connections only, so suppressing a dial-back because an inbound existed would have silenced such a node. Corrected to per-direction
Other:
- peers output now shows each entry's node identity and the node's own endpoints
- Added _DEFAULT_SOURCE to the build so getifaddrs() stays visible on glibc
Reclaim outbound slots on peer disconnect:
- TcpClient_ThreadProc fired on_disconnect but never cleared the outbound
slot, closed the fd, joined the io thread, or freed the connection, so a
dead peer permanently held its outboundClients[] slot. After MAX_CONS (32)
churned connections the node could make no new outbound connections, and
leaked fds/threads/memory. (The inbound side already self-reclaimed.)
- Add a reaper (Node_ReapDeadOutbound) on the maintenance thread: under
outboundLock it detaches dead (disconnect-notified) slots, then joins the
io thread and destroys/frees each connection outside the lock.
- Guard against use-after-free with a pin count on tcp_connection_t
(TcpConnection_Pin/Unpin). The only cross-thread consumer holding a raw
connection pointer across a blocking op is the `sync` command (via
Node_GetBestOutboundPeer); it now pins the peer and unpins when done, and
the reaper skips pinned connections. Discovery's snapshots run on the
reaper's own thread, so they need no pin.
- Node_GetBestOutboundPeer/GetClientList/GetPeerEndpoints skip
disconnect-notified connections so a dead peer is never handed out.
- Node_Destroy stops+joins the maintenance thread before tearing down
outbound clients, so the reaper can't race shutdown.
Strike disconnected peers from the discovery peer list:
- Add NodeDiscovery_RemovePeer + Node_HandlePeerDisconnect: on disconnect,
remove the peer from the known-peer table once no live connection (inbound
or outbound) to its listen endpoint remains (Node_HasLiveConnectionTo;
disconnect-notified conns don't count, so both directions dropping at once
is handled). Wired into Node_Server_OnDisconnect and Node_Client_OnDisconnect.
- Implement NodeDiscovery engine: known-peer table (DynArr) with a
per-tick seed/ping/query/connect state machine driven by the node
maintenance thread; bounded multi-hop crawl (FANOUT peers per node,
hop-capped) that connects to reachable peers lowest-ping-first
- Add GET_PEERS/PEERS TCP opcodes for peer-list exchange, handled on
both inbound and outbound connections
- Measure UDP round-trip time and pass it to the on_pong callback
(previously the send timestamp was only used for retries)
- Advertise each node's listen port in HELLO/ACK_HELLO and store it
per-connection, so inbound-only peers and non-default ports are
discoverable (length-guarded parse; wire-compatible with old peers)
- Wire a udp_node_t + node_discovery_t into net_node_t: init/start in
Node_Create, tick in the maintenance loop, teardown in Node_Destroy
(stop UDP before destroying discovery to avoid callback races)
- Add Node_ConnListenEndpoint / Node_GetPeerEndpoints helpers to derive
peers' listen endpoints (outbound: dialed port; inbound: advertised),
with IPv4-mapped-IPv6 normalization and IP+port dedup
- Match ping pong/timeout callbacks by peer address (the UDP layer owns
the nonce), fixing discovered peers stuck UNREACHABLE
- Ping the peer's listen port instead of the ephemeral TCP source port
(fixes the original stub so pongs actually return)
- Add `peers` CLI command to dump the discovery table (endpoint/hop/
state/ping)
- Add discovery tunables to constants.h (fanout, max hops, target
connections, timeouts, caps)
dcrubro