Reported symptom: node A (4656 blocks, last 10 mined on its own fork) syncing
from node B (5646 blocks) crawled for minutes while B re-served the whole chain
several times over. A's log was thousands of "Queued orphan BLOCK_DATA" lines
interleaved with "timed out fetching block N, giving up", sliding forward 8
blocks at a time and never reorging.
== Root cause ==
The sync loop had no channel for FETCH_BLOCK replies. It inferred that a
requested block had arrived purely from the chain growing:
if ((uint64_t)Chain_Size(chain) > h) { /* block h was applied */ }
But the reply is handled on the peer's io thread by Node_ParseAndAcceptBlock,
which correctly identifies a block whose prevHash points at the peer's branch
and files it in the orphan pool -- the chain does not grow. So a block that
ARRIVED AND WAS CORRECTLY ORPHANED was indistinguishable from a lost packet.
Every one of the 990 blocks therefore cost MAX_SYNC_RETRIES re-requests plus a
SYNC_REQUEST_TIMEOUT_MS stall before the window slid on and did it again. That
is both the crawl and the repeated serving on the peer.
Neither existing fork probe could rescue it:
* The divergence check sits downstream of a SUCCESSFUL append -- it only runs
after Chain_GetBlockCopy(chain, h) succeeds. When the fork begins at the
very first requested height nothing ever lands, so RequestForkWindow was
never reached from there.
* The no-progress fallback only runs after the entire window range drains,
i.e. after timing out through all 990 blocks, and is capped at 3 rounds.
== Delivery receipts ==
Node_NoteBlockDelivered / Node_TakeBlockDelivery / Node_ResetBlockDeliveries,
backed by a 512-slot ring with its own mutex. Recorded from the BLOCK_DATA
handler only -- BLOCK_DATA is sent solely in reply to a FETCH_BLOCK, so a
receipt means "the peer answered", independently of whether the block could
join our chain. A receipt for a height already present is refreshed in place,
so a retried request cannot leave a stale one for the next window to consume.
Receipts carry a status rather than a bool:
NODE_DELIVERY_APPENDED joined our chain
NODE_DELIVERY_DUPLICATE we already held this exact block -- common ground
NODE_DELIVERY_ORPHANED competing branch; now pooled
NODE_DELIVERY_REJECTED failed validation
DUPLICATE is what lets a backwards fork walk terminate; a plain "appended"
boolean cannot tell "we already have this" from "this is on their branch",
which is precisely the signal the walk needs.
The sync loop now treats any non-APPENDED delivery as the divergence signal and
probes immediately, instead of retrying something retrying cannot fix. Bounded
by MAX_FORK_PROBE_ROUNDS -- without that the window resets to the same height
and re-probes forever -- and the counter resets on real progress so a branch
that forks again further on can still be followed.
== RequestForkWindow is now an actual walk ==
It previously fired REORG_FETCH_DEPTH (128) requests and slept a flat
SYNC_REQUEST_TIMEOUT_MS. Two problems:
* The sleep was a race against the peer's serving rate. At the ~10 blocks/s
observed in testing, a 128-block window cannot land in 5s, so the
OrphanPool_AttemptAttach that followed ran against a half-filled pool and
reported a failure that was not real.
* It always asked for the full depth. On a shallow fork the overwhelming
majority came back as duplicates that were discarded without even entering
the pool -- a wasted full block send each.
It now descends in batches of MAX_PARALLEL_FETCHES, waits on that batch's
receipts rather than guessing, and stops at the first height the peer returns
that we already hold: that block is the fork point and everything below it is
shared. It returns whether common ground was found, so callers no longer
attempt an attach that cannot possibly link.
Measured on the reported scenario: 16 requests instead of 129, locating the
fork point at height 4645 (fork = blocks 4646..4655, exactly the 10 mined).
== IBD_TIP_AGE_BLOCKS 500 -> 20 ==
The reorg penalty is served by LOCAL chain growth, so a node that does not mine
can never serve it -- its tip does not move, and the only way it could grow is
by adopting the branch the penalty is gating. The IBD exemption is that node's
only route back, so at 500 block times it had to sit stalled for ~12.5 hours.
20 block times is ~30 minutes at a 90s target, far beyond normal Poisson block
spacing (a gap that long has probability ~e^-20), so a node genuinely following
the tip will not trip it.
This is what flipped the live test from "initialSync=no" to "initialSync=yes"
and let the reorg be adopted at all. Noted in the constant's comment: the
exemption is all-or-nothing, so lowering it further widens that hole.
== Reorg failures are now legible ==
Chain_ReplaceBranch broke silently for fork-point-beyond-tip, work computation
failure, insufficient work, snapshot failure and candidate copy failure alike,
so "not adopted" could not be diagnosed from a log. Each now says which. The
not-heavier case reports candidate vs incumbent block counts, because the
common cause is a branch that is still arriving -- a partial branch is
genuinely lighter -- and that reads very differently from a peer actually on a
weaker chain.
The sync loop's "Reorg candidate not adopted (lighter branch, or still serving
its reorg penalty)" is replaced by "Reorg not completed on this pass; branch
stays pooled for retry". The old wording was actively misleading: live testing
showed it firing on reorgs that the 1Hz maintenance thread completed a second
later.
== Verification ==
Against a real peer at 5646 blocks, from a real local chain of 4656 with a
10-block fork:
reported receipts only all changes
wall-clock minutes 123s 116s
timed-out fetches 1 per block 0 0
fork-walk requests n/a 129 16
reorg completed never maintenance first pass
final height 4656 5646 5646
fullverify - Chain OK Chain OK
The 116s is not a meaningful speedup over 123s and should not be read as one:
990 blocks at ~10 blocks/s is ~100s, so both are bound by the peer's serving
throughput rather than by anything in the sync loop. The substantive wins are
the 129 -> 16 fork-walk reduction, the reorg completing deterministically
instead of by luck, and logs that no longer report failure on success.
Also exercised: a synthetic partitioned fork (10 vs 60 blocks) confirming zero
timeouts and bounded probe rounds when the branch is correctly refused by the
penalty.
== Known limits, unchanged ==
Chain_ReplaceBranch adopts whatever is pooled at attach time (18 blocks in the
live run); the remainder arrives through normal windowed sync afterwards. That
bounds a single reorg by MAX_ORPHAN_BLOCKS.
124 lines
5.3 KiB
C
124 lines
5.3 KiB
C
#ifndef NET_NODE_H
|
|
#define NET_NODE_H
|
|
|
|
#ifndef _WIN32
|
|
// POSIX
|
|
#include <tcpd/tcpconnection.h>
|
|
#include <tcpd/tcpclient.h>
|
|
#include <tcpd/tcpserver.h>
|
|
#endif
|
|
|
|
#include <constants.h>
|
|
#include <packettype.h>
|
|
#include <udpd/udpnode.h>
|
|
|
|
// Forward declaration - the discovery state is defined in nodediscovery.c (opaque here).
|
|
typedef struct node_discovery node_discovery_t;
|
|
|
|
#include <stddef.h>
|
|
|
|
#include <dynarr.h>
|
|
#include <dynset.h>
|
|
|
|
#include <pthread.h>
|
|
|
|
#include <block/block.h>
|
|
#include <block/chain.h>
|
|
#include <block/transaction.h>
|
|
#include <stdatomic.h>
|
|
|
|
typedef struct {
|
|
tcp_server_t* server;
|
|
tcp_client_t outboundClients[MAX_CONS];
|
|
size_t outboundCount;
|
|
// Dedup cache for recently seen block hashes (canonical 32-byte hash)
|
|
DynSet* seenBlocks;
|
|
// Protects seenBlocks
|
|
pthread_mutex_t seenLock;
|
|
// Protects outboundClients snapshots and peerBlockHeight writes
|
|
pthread_mutex_t outboundLock;
|
|
void (*on_connect)(tcp_connection_t* conn, void* user);
|
|
void (*on_data)(tcp_connection_t* conn, const unsigned char* data, size_t len, void* user);
|
|
void (*on_disconnect)(tcp_connection_t* conn, void* user);
|
|
void* callbackUser;
|
|
// Maintenance thread for periodic tasks (orphan attach, pruning, metrics)
|
|
pthread_t maintenanceThread;
|
|
// Cross-thread stop flag: written by Node_Destroy on the main thread, read by the maintenance
|
|
// thread's loop condition. `volatile` stops the compiler hoisting the load but provides neither
|
|
// atomicity nor ordering, so this has to be a real atomic (and TSan rightly flagged it).
|
|
_Atomic int maintenanceRunning;
|
|
int maintenanceIntervalMs;
|
|
// UDP ping/pong daemon (latency oracle) and peer discovery state
|
|
udp_node_t* udpNode;
|
|
node_discovery_t* discovery;
|
|
} net_node_t;
|
|
|
|
net_node_t* Node_Create();
|
|
void Node_Destroy(net_node_t* node);
|
|
|
|
void Node_SetCallbacks(
|
|
net_node_t* node,
|
|
void (*on_connect)(tcp_connection_t* conn, void* user),
|
|
void (*on_data)(tcp_connection_t* conn, const unsigned char* data, size_t len, void* user),
|
|
void (*on_disconnect)(tcp_connection_t* conn, void* user),
|
|
void* user
|
|
);
|
|
|
|
int Node_ConnectPeer(net_node_t* node, const char* ip, unsigned short port);
|
|
int Node_ConnectStartupPeers(net_node_t* node, const char** ips, const unsigned short* ports, size_t peersCount);
|
|
|
|
int Node_SendPacket(net_node_t* node, tcp_connection_t* conn, packet_type_t packetType, const void* payload, size_t payloadLen);
|
|
int Node_BroadcastTransaction(net_node_t* node, signed_transaction_t* tx, tcp_connection_t* excludeNode);
|
|
|
|
// Helpers for outbound peer selection and block broadcast
|
|
int Node_GetBestOutboundPeer(net_node_t* node, tcp_connection_t** outConn, uint64_t* outHeight);
|
|
|
|
/**
|
|
* Delivery receipts for windowed sync.
|
|
*
|
|
* A FETCH_BLOCK reply is handled on the peer's io thread and may legitimately never reach the
|
|
* chain: a block belonging to a competing branch is filed in the orphan pool instead. A sync loop
|
|
* that infers arrival from the chain growing therefore cannot tell "arrived but forked" from "lost
|
|
* in transit", so it re-requests until it times out. Against a peer on a fork that costs one full
|
|
* retry-and-timeout cycle for EVERY block, which is why syncing to a forked peer used to crawl.
|
|
*
|
|
* DUPLICATE is what makes a backwards fork walk terminate: it means we already hold exactly that
|
|
* block, so the two chains agree at that height and there is no reason to keep descending.
|
|
**/
|
|
typedef enum {
|
|
NODE_DELIVERY_APPENDED = 0, // joined our chain
|
|
NODE_DELIVERY_DUPLICATE = 1, // we already held this exact block -- common ground
|
|
NODE_DELIVERY_ORPHANED = 2, // belongs to a competing branch; now in the orphan pool
|
|
NODE_DELIVERY_REJECTED = 3 // failed validation
|
|
} node_delivery_status_t;
|
|
|
|
void Node_NoteBlockDelivered(uint64_t height, node_delivery_status_t status);
|
|
bool Node_TakeBlockDelivery(uint64_t height, node_delivery_status_t* outStatus);
|
|
void Node_ResetBlockDeliveries(void);
|
|
void Node_BroadcastChainRange(net_node_t* node, size_t startHeightInclusive, tcp_connection_t* sourceConn);
|
|
|
|
// Callback logic
|
|
void Node_Server_OnConnect(tcp_connection_t* client);
|
|
void Node_Server_OnData(tcp_connection_t* client);
|
|
void Node_Server_OnDisconnect(tcp_connection_t* client);
|
|
void Node_Client_OnConnect(tcp_connection_t* client);
|
|
void Node_Client_OnData(tcp_connection_t* client);
|
|
void Node_Client_OnDisconnect(tcp_connection_t* client);
|
|
|
|
void Node_GetClientList(net_node_t* node, tcp_connection_t** outClients, size_t* outCount);
|
|
|
|
// Computes a connection's peer listen endpoint (IP + advertised/dialed listen port) into *out.
|
|
// Outbound: the dialed peerAddr port already is the listen port. Inbound: uses peerListenPort.
|
|
// Returns non-zero on success (usable endpoint with a known, non-zero port), zero otherwise.
|
|
int Node_ConnListenEndpoint(const tcp_connection_t* conn, struct sockaddr_storage* out);
|
|
|
|
// Returns the node identity advertised by a connection's peer, or 0 if it is not known yet.
|
|
uint64_t Node_ConnPeerNodeId(const tcp_connection_t* conn);
|
|
|
|
// Fills outEndpoints with the listen endpoints of all current connections (inbound + outbound),
|
|
// deduped by IP+port, and outNodeIds (optional, may be NULL) with the matching peer identities.
|
|
// Returns the number of endpoints written (<= maxOut).
|
|
size_t Node_GetPeerEndpoints(net_node_t* node, struct sockaddr_storage* outEndpoints, uint64_t* outNodeIds, size_t maxOut);
|
|
|
|
#endif
|