dcrubro/skalacoin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Let a node that is behind adopt the peer's branch, and fix two bugs that only appear once a reorg actually applies

Browse Source 
Reported: a node at height N+10 on its own short fork, with the peer at N+500,
could never sync. It located the fork point correctly, pooled the branch, and
then refused it forever. At 90s blocks a node is 960 blocks behind after a day
offline, so this is the normal case, not an edge case.

== Why it could never recover ==

The reorg delay was served by LOCAL chain growth alone:

    elapsed = tipHeight - observedTip

That only makes sense for a node whose tip is advancing. A node that is behind
has a frozen tip precisely because it is rejecting the branch, so elapsed stays
0 forever while penalty(5) = 42. Both escape hatches also fail: mining out of it
means extending a fork nobody accepts, and the IBD exemption never fires for a
node that is mining, because mining keeps its tip fresh.

Being 490 blocks behind is being behind, not a reorg contest. Refusing to adopt
protects nothing while the node falls a further 40 blocks behind every hour.

The delay is now satisfiable by EITHER side making progress:

    localGrowth = tipHeight - observedTip
    branchLead  = candidateTip - tipHeight        (0 if not ahead)
    elapsed     = max(localGrowth, branchLead)

A branch already extending `penalty` blocks past our tip has demonstrated
exactly what the delay asks for, and every one of those blocks carries PoW we
validated ourselves. Requiring us to independently produce the same amount is
demanding the same proof twice. Only VALIDATED blocks count -- a peer's
advertised height is not evidence and never reaches this code.

It degrades correctly in both directions: a mining node's tip advances, so an
attacker must outpace it by penalty blocks; a node that is only observing
follows the heaviest chain, which is what an observer should do.

== The window has to keep pulling the branch ==

Every forked delivery reset nextReq back to our own tip, so the window
re-requested the same eight heights forever and the pool never grew past the
window size -- branchLead could not rise even in principle. The backward walk
now runs ONCE to establish linkage, then the window marches forward pooling the
branch, retrying adoption per window-full with a final attempt when it drains.

== sync force ==

Operator override that skips the delay for one sync, for a node whose chain is
known to be the wrong one. Threaded explicitly (Chain_ReplaceBranch gains
bypassPenalty, OrphanPool_AttemptAttachForced) rather than through a global, so
nothing a peer sends can reach it. Linkage, work comparison and atomicity still
apply -- it waives only the waiting, and says so loudly in the log.

== Bug found in testing: PoW is branch-relative ==

A block's epoch seed is the last block of the previous epoch ON ITS OWN BRANCH.
Validating a competing branch's block against OUR epoch seed does not merely
fail to resolve: when the chains diverge before the boundary it resolves to the
WRONG seed and rejects a perfectly valid block. Any fork spanning an epoch
boundary was therefore impossible to assemble -- the branch could not grow past
the boundary block, so branchLead stalled one short of it.

The receive path now does self-contained checks only (Block_HasValidStructure:
merkle, transactions, vote, non-empty). Proof of work moved to
Chain_AddBlockLocked, at the point a block joins the chain, where the branch
context is real -- the rollback has put its ancestors in place by then. That is
where the invariant belongs and it removes a duplicate check rather than adding
one. Needs Chain_DagParamsForHeightLocked, because Chain_AddBlockLocked already
holds chainLock for writing and the lock is not recursive.

Consequence worth knowing: the orphan pool can now hold blocks whose work has
not been verified, bounded by MAX_ORPHAN_BLOCKS (512). Each still had to pass
merkle and full transaction/signature validation, and none can reach the chain
unverified.

This bug also affected plain forward sync across block 350000; it was masked
because appending keeps the boundary block in the chain.

== Bug found in testing: stale DAG accepted as current ==

Block_PowHashHeavy matched on epoch index and size but not the seed. A DAG's
content is a function of (seed, size); the epoch index is a label for it. A
reorg is exactly the event that changes the seed while leaving index and size
untouched, so mid-apply the miner's context still held a DAG built from the
PRE-reorg seed, the guard passed, and a valid block was hashed against the wrong
lanes. g_dagSeed now records what each DAG was generated from and both
Block_EnsureAutolykos2Dag and Block_PowHashHeavy compare it.

== Bug found in testing: double free on the failed-apply path ==

SIGABRT in the allocator: free_tiny_botch -> DynArr_destroy -> Block_Destroy ->
Chain_FreeBlockArray -> Chain_ReplaceBranch.

DynArr_push_back stores the struct BY VALUE, so the chain's element and the
caller's block_t share one transactions pointer. Three places free that array
through the chain's copy -- Chain_ClearBlocks, Chain_RollbackToHeightLocked and
Chain_SaveToFile -- and each NULLs only the chain's side, leaving any caller
wrapper dangling. Whether a caller then had to use free() or Block_Destroy() was
a convention carried in comments at every call site plus a consumed-count passed
into Chain_FreeBlockArray. Chain_ReplaceBranch reset that count to 0 after
rolling back a failed apply, which told the cleanup to Block_Destroy exactly the
blocks whose arrays the rollback had just freed.

Rather than fix the count, the aliasing is now safe by construction:
Chain_AddBlockLocked clears the CALLER's transactions pointer immediately after
the push. Since DynArr_destroy(NULL) is a no-op, free(wrapper) and
Block_Destroy(wrapper) become equivalent and both safe regardless of what later
frees the chain's copy. The consumed-count parameter and both counters are gone
-- the thing that could be got wrong no longer exists -- and all call sites are
unified on Block_Destroy.

Placement is deliberate: immediately after the push, not at the end on success.
The ledger pass can fail with the block already in the chain, returning false to
a caller that destroys its wrapper on failure -- OrphanPool_ExtendTip does
exactly that, a third live instance not yet triggered.

Two follow-ons the refactor forced, both improvements anyway: MineAndAppendBlock
read the coinbase for its log line after the add (hoisted above it), and the
success log printed the caller's block rather than the chain's copy.

== Also ==

The deferral line is rate-limited. The maintenance thread retries pooled
branches once a second and elapsed only changes when something moves, so it
printed an identical line every second -- forever, on a node that is not mining.
It now reports each distinct situation once.

== Verification ==

Synthetic fork, node A 5 deep, node B ~60 ahead, EPOCH_LENGTH=8 so the branch
crosses three epoch boundaries:

  branchLead climbs 8 -> 16 -> 28 -> 34, crosses penalty(5)=42
  Adopted competing branch of 50 block(s) at fork height 20
  sync complete: localHeight=70
  Chain OK

Repeated under AddressSanitizer: adopted 47 blocks, 0 ASan errors on both nodes.
This matters because the refactor rewrites the exact cleanup path the SIGABRT
came from, and a double free that no longer aborts would otherwise pass silently.

Unit suites pass, including a new assertion "heavy path refuses a DAG built from
a different seed" -- the direct regression for the stale-DAG bug.

Harness note: each node needs its OWN wallet. With a shared one both pay the
same coinbase address, produce identical merkle roots, and at easy difficulty
mine byte-identical blocks -- the fork test silently became a catch-up test.

== Still untested ==

The restore-after-failed-apply path is no longer naturally reachable now that
the two bugs above are fixed, so it needs deliberate corruption to exercise.
Test B (branch only slightly ahead must still DEFER), test C (sync force), and a
TSan pass over the changed paths are outstanding.
This commit is contained in:
dcrubro
2026-08-02 19:36:09 +02:00
parent 01c44731ef
commit 5eaf0b699c
8 changed files with 349 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/main.c
+98 -50
View File
@@ -478,11 +478,8 @@ static bool MineAndAppendBlock(blockchain_t* chain,
return false;
}
if (!Chain_AddBlock(chain, block)) {
fprintf(stderr, "failed to append block to chain\n");
return false;
}
// Read the coinbase BEFORE handing the block to the chain. Chain_AddBlock takes ownership of
// the transaction array and clears our pointer to it, so this has to happen first.
uint64_t coinbaseAmount = 0;
if (block->transactions && DynArr_size(block->transactions) > 0) {
signed_transaction_t* firstTx = (signed_transaction_t*)DynArr_at(block->transactions, 0);
@@ -491,6 +488,11 @@ static bool MineAndAppendBlock(blockchain_t* chain,
}
}
if (!Chain_AddBlock(chain, block)) {
fprintf(stderr, "failed to append block to chain\n");
return false;
}
/* Debug proof removed: miner printed proof that coinbase == baseReward + totalFees during debugging. */
// After successfully appending a block, attempt to attach any orphans.
@@ -922,7 +924,7 @@ int main(int argc, char* argv[]) {
char supplyStr[80];
Uint256ToDecimal(&currentSupply, supplyStr, sizeof(supplyStr));
printf("Current chain has %zu blocks, total supply %s\n", Chain_Size(chain), supplyStr);
printf("Commands: mine <x>, send <address> <amount> [fee], txpooldetail <txhash>, balance [address], connect <ipv4> [port], peers, sync (requires nodes), flushchain, fullverify, blockdetail <block number>, wipechain, genaddr, exit\n");
printf("Commands: mine <x>, send <address> <amount> [fee], txpooldetail <txhash>, balance [address], connect <ipv4> [port], peers, sync [force] (requires nodes), dagvote <grow|hold|down>, flushchain, fullverify, blockdetail <block number>, wipechain, genaddr, exit\n");
char line[1024];
while (true) {
@@ -1025,7 +1027,7 @@ int main(int argc, char* argv[]) {
break;
}
free(block); // Chain stores block by value and owns copied transaction array.
Block_Destroy(block); // Chain_AddBlock already took the transaction array.
// Broadcast newly mined block to outbound peers
if (node) {
@@ -1132,7 +1134,7 @@ int main(int argc, char* argv[]) {
FlushChainAndSheet(chain, chainDataDir, currentSupply, currentReward);
free(block);
Block_Destroy(block); // the chain took the transaction array; this frees the wrapper
if (node) {
Node_BroadcastChainRange(node, Chain_Size(chain) - 1, NULL);
}
@@ -1162,6 +1164,24 @@ int main(int argc, char* argv[]) {
continue;
}
// `sync force` skips the reorg delay penalty for this sync only. The penalty is served
// by local chain growth, so a node that is neither mining nor stale enough to count as
// catching up cannot clear it on its own -- this is the operator's way out when they
// know their branch is the wrong one. Work comparison and linkage still apply.
bool forceSync = false;
{
const char* syncArg = strtok(NULL, " \t");
if (syncArg && strcmp(syncArg, "force") == 0) {
forceSync = true;
} else if (syncArg) {
printf("usage: sync [force]\n");
continue;
}
}
if (forceSync) {
printf("sync force: the reorg delay penalty will be skipped for this sync\n");
}
// Choose the best outbound peer by advertised height
tcp_connection_t* peerConn = NULL;
uint64_t peerHeight = 0;
@@ -1174,6 +1194,12 @@ int main(int argc, char* argv[]) {
bool madeProgressOverall = false;
int forkProbes = 0;
// Fork state for this sync. The backward walk runs once; after that the window keeps
// marching forward so the competing branch accumulates in the pool, which is what lets
// its length past our tip satisfy the reorg delay.
bool forkPointLocated = false;
uint64_t pooledSinceAttach = 0;
// Drop receipts left over from an earlier sync, so a stale one cannot be mistaken for
// an answer to a request this run has not sent yet.
Node_ResetBlockDeliveries();
@@ -1285,7 +1311,7 @@ int main(int argc, char* argv[]) {
printf("Divergence at height %" PRIu64 "; probing for the fork point\n", h);
RequestForkWindow(node, peerConn, h);
size_t reattached = OrphanPool_AttemptAttach(chain);
size_t reattached = OrphanPool_AttemptAttachForced(chain, forceSync);
if (reattached > 0) {
printf("Reorg attached %zu block(s) from the peer's branch\n", reattached);
} else {
@@ -1334,51 +1360,62 @@ int main(int argc, char* argv[]) {
// and made the peer re-serve the whole chain several times over.
node_delivery_status_t deliveryStatus = NODE_DELIVERY_REJECTED;
if (Node_TakeBlockDelivery(h, &deliveryStatus) && deliveryStatus != NODE_DELIVERY_APPENDED) {
if (forkProbes >= MAX_FORK_PROBE_ROUNDS) {
printf("block %" PRIu64 " is on a branch we cannot join after %d probe(s); "
"giving up on this peer\n", h, forkProbes);
inFlight = 0;
nextReq = end; // stop refilling the window; this peer is unreachable by extension
break;
}
// Locate the fork point ONCE. After that the branch just needs to keep
// accumulating: the delay is satisfied by the branch extending past our tip
// (see Chain_ReplaceBranch), so the window must march FORWARD pooling
// blocks. Resetting it to our tip on every forked delivery -- which is what
// this used to do -- re-requested the same eight heights forever, so the
// pool never grew past the window size and a node that was far behind could
// never accumulate enough of the branch to adopt it.
if (!forkPointLocated) {
forkProbes++;
printf("block %" PRIu64 " arrived but does not extend our chain; "
"probing for the fork point\n", h);
forkProbes++;
printf("block %" PRIu64 " arrived but does not extend our chain; "
"probing for the fork point (round %d/%d)\n",
h, forkProbes, MAX_FORK_PROBE_ROUNDS);
// Pull a window BELOW the divergence so the pool can walk prevHash back to
// the common ancestor. Without this the fork point is never requested at
// all, because the window only ever moves forward from our own tip.
const bool foundCommon = RequestForkWindow(node, peerConn, h);
size_t reattached = foundCommon ? OrphanPool_AttemptAttach(chain) : 0;
if (reattached > 0) {
printf("Reorg attached %zu block(s) from the peer's branch\n", reattached);
forkProbes = 0; // real progress; allow probing again if it forks further on
} else if (!foundCommon) {
printf("No shared block found with this peer; its branch cannot be linked to ours\n");
} else {
// Deliberately not phrased as a failure: the branch stays pooled and the
// 1Hz maintenance thread retries it, which is usually what completes a
// reorg whose blocks were still arriving when this pass ran.
printf("Reorg not completed on this pass; branch stays pooled for retry\n");
}
// Restart the window against whatever our tip is now.
nextReq = Chain_Size(chain);
inFlight = 0;
if (Chain_Size(chain) > 0) {
block_t* tip = NULL;
if (Chain_GetBlockCopy(chain, Chain_Size(chain) - 1, &tip) && tip) {
Block_CalculateHash(tip, expectedPrevHash);
Block_Destroy(tip);
forkPointLocated = RequestForkWindow(node, peerConn, h);
if (!forkPointLocated) {
printf("No shared block found with this peer within %" PRIu64
" blocks; its branch cannot be linked to ours\n", REORG_FETCH_DEPTH);
inFlight = 0;
nextReq = end; // this peer is unreachable by extension
break;
}
} else {
memset(expectedPrevHash, 0, sizeof(expectedPrevHash));
}
// Delivered, so drop it from the in-flight set and let the window advance.
for (int j = i; j < inFlight - 1; ++j) {
requestedHeights[j] = requestedHeights[j + 1];
retryCount[j] = retryCount[j + 1];
sentAtMs[j] = sentAtMs[j + 1];
}
inFlight--;
pooledSinceAttach++;
progressed = true;
// Retry adoption as the branch grows, rather than once per probe. Each
// attempt walks the pool, so do it per window-full instead of per block.
if (pooledSinceAttach >= (uint64_t)maxInFlight) {
pooledSinceAttach = 0;
size_t reattached = OrphanPool_AttemptAttachForced(chain, forceSync);
if (reattached > 0) {
printf("Reorg attached %zu block(s) from the peer's branch\n", reattached);
// The chain moved; realign the window and the expected parent hash.
nextReq = Chain_Size(chain);
inFlight = 0;
forkPointLocated = false; // a further divergence would need a new walk
if (Chain_Size(chain) > 0) {
block_t* tip = NULL;
if (Chain_GetBlockCopy(chain, Chain_Size(chain) - 1, &tip) && tip) {
Block_CalculateHash(tip, expectedPrevHash);
Block_Destroy(tip);
}
} else {
memset(expectedPrevHash, 0, sizeof(expectedPrevHash));
}
}
}
break;
}
@@ -1411,6 +1448,17 @@ int main(int argc, char* argv[]) {
}
}
// The window drained. Anything pooled since the last attempt has not been tried yet --
// without this a branch whose final partial batch never reached the retry threshold
// would sit in the pool unadopted until the maintenance thread happened to pick it up.
if (pooledSinceAttach > 0) {
pooledSinceAttach = 0;
size_t reattached = OrphanPool_AttemptAttachForced(chain, forceSync);
if (reattached > 0) {
printf("Reorg attached %zu block(s) from the peer's branch\n", reattached);
}
}
// After the window completes, check progress and possibly refresh peer height.
// This flag tracks THIS iteration only: it used to be set once and never cleared, so
// after a single productive pass the "no progress -> stop" guard below could never fire
@@ -1445,7 +1493,7 @@ int main(int argc, char* argv[]) {
peerHeight, newLocal);
const bool foundCommon = RequestForkWindow(node, peerConn, newLocal);
size_t attached = foundCommon ? OrphanPool_AttemptAttach(chain) : 0;
size_t attached = foundCommon ? OrphanPool_AttemptAttachForced(chain, forceSync) : 0;
if (attached > 0) {
printf("Fork probe adopted %zu block(s) from the peer's branch\n", attached);
continue;