Fix multi-homed / IPv6 peer identity: nodes adding themselves and reconnect churn
Root cause: peers were keyed only by (IP, listen port). An IPv6 host holds several addresses at once, so one node appeared as several peers and could not recognise its own addresses. Protocol: - Added a random per-run node identity (localNodeId), advertised as a length-guarded trailing field in HELLO and ACK_HELLO — backwards compatible with peers that omit it - Added peerNodeId to tcp_connection_t; peers are now identified by this rather than by an endpoint - ACK_HELLO is now sent before any decision to drop the connection, so a rejected dialer learns whose address it reached instead of retrying forever Self-connection: - Identity match → close the connection and record that endpoint permanently as our own - Self-endpoint set preemptively seeded from getifaddrs() at startup, so a node knows its own addresses before ever dialing one - Node_ConnectPeer refuses self endpoints, which also kills the echo-back chain that could exhaust connection slots Duplicate connections and churn: - Dedup by identity per direction — one inbound and one outbound per physical peer, regardless of how many addresses it has - Moved dial history out of the peer table, so striking a peer no longer resets its connect-retry cooldown (this was the loop engine: strike → re-learn via gossip → redial next tick) - Node_HasOtherInboundFrom / Node_HasLiveConnectionTo now ignore connections that are already tearing down, and match on identity as well as endpoint Gossip hygiene: - Never hand a peer its own other addresses in a PEERS reply (matched on identity, not just the socket address) - Reject unusable endpoints: link-local without scope id, unspecified, multicast, site-local (loopback stays allowed for local testing) - Normalise IPv4-mapped IPv6 so one host cannot occupy two entries Two bugs found along the way: - All nodes drew the same identity — random_eight_byte() comes from srand(time(NULL)), so processes started in the same second produced identical values. Added random_secure_eight_byte() (/dev/urandom) for the identity, and mixed the pid into the seed so connection IDs stop colliding too - Identity dedup initially left inbound-only nodes mute — broadcasts traverse outbound connections only, so suppressing a dial-back because an inbound existed would have silenced such a node. Corrected to per-direction Other: - peers output now shows each entry's node identity and the node's own endpoints - Added _DEFAULT_SOURCE to the build so getifaddrs() stays visible on glibc
This commit is contained in:
@@ -85,8 +85,12 @@ void Node_GetClientList(net_node_t* node, tcp_connection_t** outClients, size_t*
|
||||
// Returns non-zero on success (usable endpoint with a known, non-zero port), zero otherwise.
|
||||
int Node_ConnListenEndpoint(const tcp_connection_t* conn, struct sockaddr_storage* out);
|
||||
|
||||
// Returns the node identity advertised by a connection's peer, or 0 if it is not known yet.
|
||||
uint64_t Node_ConnPeerNodeId(const tcp_connection_t* conn);
|
||||
|
||||
// Fills outEndpoints with the listen endpoints of all current connections (inbound + outbound),
|
||||
// deduped by IP+port. Returns the number of endpoints written (<= maxOut).
|
||||
size_t Node_GetPeerEndpoints(net_node_t* node, struct sockaddr_storage* outEndpoints, size_t maxOut);
|
||||
// deduped by IP+port, and outNodeIds (optional, may be NULL) with the matching peer identities.
|
||||
// Returns the number of endpoints written (<= maxOut).
|
||||
size_t Node_GetPeerEndpoints(net_node_t* node, struct sockaddr_storage* outEndpoints, uint64_t* outNodeIds, size_t maxOut);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -27,6 +27,18 @@ void NodeDiscovery_OnPeersReceived(node_discovery_t* disc, tcp_connection_t* fro
|
||||
// logically disconnected (no remaining connection to it).
|
||||
void NodeDiscovery_RemovePeer(node_discovery_t* disc, const struct sockaddr_storage* endpoint);
|
||||
|
||||
// Record the node identity behind an endpoint (learned from a completed HELLO/ACK_HELLO). Entries
|
||||
// carrying an identity we are already connected to are skipped by the connect picker, which is what
|
||||
// stops a multi-homed peer from being dialed once per address it is reachable on.
|
||||
void NodeDiscovery_NoteIdentity(node_discovery_t* disc, const struct sockaddr_storage* endpoint, uint64_t nodeId);
|
||||
|
||||
// Mark an endpoint as one of our own, permanently. Self endpoints are never added to the known-peer
|
||||
// table, never pinged and never dialed. Seeded from the local interface addresses at creation and
|
||||
// extended whenever a handshake turns out to come from ourselves.
|
||||
void NodeDiscovery_MarkSelfEndpoint(node_discovery_t* disc, const struct sockaddr_storage* endpoint);
|
||||
// Returns non-zero if the endpoint is known to be one of our own.
|
||||
int NodeDiscovery_IsSelfEndpoint(node_discovery_t* disc, const struct sockaddr_storage* endpoint);
|
||||
|
||||
// Dump the known-peer table to stdout (for the CLI `peers` command).
|
||||
void NodeDiscovery_PrintPeers(node_discovery_t* disc);
|
||||
|
||||
|
||||
@@ -11,4 +11,9 @@ uint16_t random_two_byte(void);
|
||||
uint32_t random_four_byte(void);
|
||||
uint64_t random_eight_byte(void);
|
||||
|
||||
// Draws from the OS entropy pool instead of the srand()-seeded generator, which repeats across
|
||||
// processes started within the same second. Use this wherever a value must be unique between nodes
|
||||
// (e.g. the node identity). Never returns 0.
|
||||
uint64_t random_secure_eight_byte(void);
|
||||
|
||||
#endif
|
||||
|
||||
@@ -17,6 +17,10 @@ extern const char* chainDataDir;
|
||||
extern unsigned short listenPort;
|
||||
extern bool echoPeersEnabled;
|
||||
extern bool forceOrphanReorgEnabled;
|
||||
// Random per-run identity of this node, advertised in HELLO/ACK_HELLO. A host can be reachable
|
||||
// under many addresses (especially over IPv6), so an (ip, port) endpoint is not a peer identity:
|
||||
// this nonce is what lets us recognise our own connections and a peer we already talk to.
|
||||
extern uint64_t localNodeId;
|
||||
|
||||
// Global synchronization primitives for runtime state
|
||||
extern pthread_rwlock_t chainLock; // protects chain structure and related mutations
|
||||
|
||||
@@ -31,6 +31,11 @@ struct tcp_connection_t {
|
||||
// For OUTBOUND connections the peerAddr port already is the listen port; this matters for INBOUND peers.
|
||||
uint16_t peerListenPort;
|
||||
|
||||
// Peer's advertised node identity (learned from HELLO/ACK_HELLO). 0 until known / peer too old
|
||||
// to advertise one. Unlike the peer address, this is stable across all of a multi-homed peer's
|
||||
// endpoints, so it is what identifies the node behind this connection.
|
||||
uint64_t peerNodeId;
|
||||
|
||||
pthread_t ioThread;
|
||||
pthread_mutex_t sendLock;
|
||||
pthread_mutex_t stateLock;
|
||||
|
||||
Reference in New Issue
Block a user