Part IV · Surviving the long run This chapter answers the questions real users actually ask.
Every chapter so far has been “how the system works.” This one switches viewpoint: where exactly is your transaction between the moment you send it and the moment it lands in a block?
Four questions:
Why hasn’t my transaction confirmed?
Why isn’t it first-come, first-served?
I underpaid — can I fix it?
Why can one confirmation still disappear?
12.1 The mempool: a waiting basket¶
After broadcast a transaction does not go straight into a block. It waits in the mempool of many nodes.
Here is a fact most people don’t know:
There is no single global mempool. Every node has its own, usually similar, never guaranteed identical.
Chapter 7’s sentence lands here: the mempool is “I heard about this”; the blockchain is “the network finally accepted this.”
# A transaction here has four fields: id, size, fee, and which inputs it spends.
mempool = [
{"txid": "txA", "size": 500, "fee": 1000, "inputs": ["coinA"]},
{"txid": "txB", "size": 200, "fee": 700, "inputs": ["coinB"]},
{"txid": "txC", "size": 150, "fee": 900, "inputs": ["coinC"]},
{"txid": "txD", "size": 300, "fee": 450, "inputs": ["coinD"]},
{"txid": "txE", "size": 250, "fee": 300, "inputs": ["coinE"]},
]
def feerate(tx):
'''Fee rate = fee / size.'''
return tx["fee"] / tx["size"]
for tx in mempool:
print(tx["txid"], "fee=", tx["fee"], "size=", tx["size"], "feerate=", round(feerate(tx), 2))
txA fee= 1000 size= 500 feerate= 2.0
txB fee= 700 size= 200 feerate= 3.5
txC fee= 900 size= 150 feerate= 6.0
txD fee= 450 size= 300 feerate= 1.5
txE fee= 300 size= 250 feerate= 1.2
mempool is a list of dictionaries. No other structure.
Now look at feerate — this is the chapter’s first counterintuitive point.
Miners do not care about the total fee. They care about the rate: how much they earn per unit of block space.
Compare txA and txC in the output:
txA fee=1000 size=500 feerate=2.0
txC fee=900 size=150 feerate=6.0txA pays more in total (1000 > 900), and txC gets in first.
Because txA occupies more than three times the room. A miner is not selling a service. He is selling block space. What you pay is not “the price of a transaction” — it is rent per byte.
(Which is why your wallet quotes sat/vB — satoshis per byte — rather than a total.)
12.2 How a miner packs a block¶
def pick_block(mempool, max_size):
'''Pack highest-fee-rate first; skip anything that doesn't fit.'''
picked = []
used = 0
for tx in sorted(mempool, key=feerate, reverse=True):
if used + tx["size"] <= max_size:
picked.append(tx)
used += tx["size"]
return picked
def show_block(picked):
total_fee = sum(tx["fee"] for tx in picked)
total_size = sum(tx["size"] for tx in picked)
print("packed into the block:", [tx["txid"] for tx in picked])
print("total size:", total_size, " total fees:", total_fee)
picked = pick_block(mempool, max_size=650)
show_block(picked)
print("left out:", [tx["txid"] for tx in mempool if tx not in picked])
packed into the block: ['txC', 'txB', 'txD']
total size: 650 total fees: 2050
left out: ['txA', 'txE']
Notice the if used + tx["size"] <= max_size: — anything that doesn’t fit is skipped and
the loop continues rather than stopping.
Which means a transaction with a mediocre fee rate but a small size can get in simply by fitting into the gap left over.
(This is really a knapsack problem, and pick_block is a greedy approximation.
Real Bitcoin is greedy too — the optimal solution is NP-hard and a miner has seconds.
This is the first place in the book where we settle for good-enough because optimal is
unavailable: the line between engineering and mathematics.)
# Experiment: the tighter the block, the more low-fee-rate transactions wait.
for max_size in [400, 650, 1000, 1400]:
picked = pick_block(mempool, max_size)
print(f"block space {max_size:>4} ->", [tx["txid"] for tx in picked])
block space 400 -> ['txC', 'txB']
block space 650 -> ['txC', 'txB', 'txD']
block space 1000 -> ['txC', 'txB', 'txA']
block space 1400 -> ['txC', 'txB', 'txA', 'txD', 'txE']
Those four lines are the whole fee market:
More people want in than there is room.
Miners take the highest rate first.
Anyone who wants speed raises their rate.No authority sets the price. Competition for finite block space produces it.
12.3 Speeding up, method one: RBF¶
Your fee rate was too low and the transaction is stuck. Now what?
RBF = Replace-By-Fee:
I sent a low-fee transaction.
It hasn't confirmed.
I send a new transaction spending the same coin, with a higher fee.
Nodes may replace the old one with the new one.Note this does not add money to the old transaction. It broadcasts a more expensive replacement that pushes the original out.
def conflicts(tx1, tx2):
'''Two transactions spending the same input conflict -- both cannot be mined.'''
return bool(set(tx1["inputs"]) & set(tx2["inputs"]))
def add_with_rbf(pool, new_tx):
'''Simplified RBF: if the new tx conflicts with an old one and pays more, replace it.'''
new_pool = []
replaced = []
for old in pool:
if conflicts(old, new_tx):
if new_tx["fee"] > old["fee"]:
replaced.append(old)
else:
return pool, "rejected: the replacement does not pay enough"
else:
new_pool.append(old)
new_pool.append(new_tx)
if replaced:
return new_pool, "replaced " + ", ".join(tx["txid"] for tx in replaced)
return new_pool, "added to the mempool"
low = {"txid": "pay_low", "size": 250, "fee": 200, "inputs": ["coinZ"]}
high = {"txid": "pay_high", "size": 250, "fee": 1200, "inputs": ["coinZ"]}
pool = mempool + [low]
print("before:", [tx["txid"] for tx in pool])
pool, msg = add_with_rbf(pool, high)
print(msg)
print("after :", [tx["txid"] for tx in pool])
before: ['txA', 'txB', 'txC', 'txD', 'txE', 'pay_low']
replaced pay_low
after : ['txA', 'txB', 'txC', 'txD', 'txE', 'pay_high']
Look at what conflicts checks: whether two transactions spend the same input.
Go back to chapter 6 — two transactions spending the same input is a double-spend.
So what RBF does, literally, is deliberately initiate a double-spend, where one version pays more and the nodes are willing to swap.
RBF is not a double-spend loophole. RBF is double-spending, in the one form the rules have chosen to accept.
This is exactly why a merchant cannot safely accept a zero-confirmation payment: an unconfirmed transaction can be replaced by its sender at any moment.
12.4 Speeding up, method two: CPFP¶
Only the sender can use RBF — it requires spending the same input, which requires the key.
So can a recipient who sees an underpaying transaction do anything? Yes.
CPFP = Child-Pays-For-Parent:
The parent's fee is too low; miners aren't interested.
But a child transaction spends the parent's output and pays a very high fee.
The child cannot confirm before the parent does.
So a miner who wants the child's fee has to include both.parent = {"txid": "parent_low", "size": 500, "fee": 100, "inputs": ["coinP"], "depends_on": []}
child = {"txid": "child_high", "size": 120, "fee": 1100, "inputs": ["parent_low:0"], "depends_on": ["parent_low"]}
def package_feerate(txs):
return sum(tx["fee"] for tx in txs) / sum(tx["size"] for tx in txs)
print("parent alone, fee rate:", round(feerate(parent), 2))
print("child alone, fee rate :", round(feerate(child), 2), "-- but it depends on the parent")
print("parent+child package :", round(package_feerate([parent, child]), 2))
threshold = 1.5
print("miner's minimum fee rate:", threshold)
print("is the parent enough on its own?", feerate(parent) >= threshold)
print("are they enough together? ", package_feerate([parent, child]) >= threshold)
parent alone, fee rate: 0.2
child alone, fee rate : 9.17 -- but it depends on the parent
parent+child package : 1.94
miner's minimum fee rate: 1.5
is the parent enough on its own? False
are they enough together? True
The key is package_feerate: it treats parent and child as a single unit.
The parent alone rates 0.2 and misses the threshold. Together they rate 1.94 and clear it. A miner who wants the child’s 1100 has no choice but to carry the parent along.
In one line:
RBF : replace the old transaction with a more expensive one. (sender only)
CPFP: use a high-fee child to drag a low-fee parent into a block. (anyone)12.5 What a miner picks determines the block’s fingerprint¶
Now chapters 5, 10 and 11 come together.
import hashlib
def H(s): return hashlib.sha256(s.encode()).hexdigest()
def merkle_root(leaves):
if not leaves:
return H("")
level = [H(x) for x in leaves]
while len(level) > 1:
if len(level) % 2 == 1:
level.append(level[-1])
level = [H(level[i] + level[i+1]) for i in range(0, len(level), 2)]
return level[0]
def block_hash(prev_hash, txids, nonce=0):
root = merkle_root(txids)
return H(prev_hash + root + str(nonce)), root
txids1 = [tx["txid"] for tx in pick_block(mempool, max_size=650)]
txids2 = [tx["txid"] for tx in pick_block(pool, max_size=650)] # the mempool after RBF
h1, root1 = block_hash("prev", txids1)
h2, root2 = block_hash("prev", txids2)
print("originally packed:", txids1)
print("Merkle root:", root1[:18], " block hash:", h1[:18])
print()
print("packed after RBF :", txids2)
print("Merkle root:", root2[:18], " block hash:", h2[:18])
print("same root?", root1 == root2)
originally packed: ['txC', 'txB', 'txD']
Merkle root: b816616688b7bc9774 block hash: af27b53088d67a249c
packed after RBF : ['txC', 'pay_high', 'txB']
Merkle root: 6a6e48bc642766d9e2 block hash: 4795431218b0877474
same root? False
This is the first place several threads in the book fully converge:
Fees decide which transactions a miner picks (chapter 10)
|
The transaction set determines the Merkle root (chapter 11)
|
The Merkle root goes into the block header (chapter 11)
|
The block header must be re-mined (chapter 5)So every time a miner swaps a transaction, mining restarts from scratch. Which explains something practical: miners generally do not churn their transaction set mid-block — every change throws away the work done so far.
12.6 Reorgs: why one confirmation can vanish¶
Chapter 7’s rule says nodes follow the valid chain with the most accumulated work.
If your transaction went into a block on one chain, and another chain later becomes heavier, nodes switch. That switch is a chain reorganization, or reorg.
def chain_work(chain):
return sum(block["work"] for block in chain)
def all_txs(chain):
seen = []
for block in chain:
seen += block["txs"]
return seen
genesis = {"name": "A", "txs": [], "work": 1}
block_B = {"name": "B", "txs": ["tx1"], "work": 1}
block_C_with_pay = {"name": "C", "txs": ["pay_to_shop"], "work": 1}
# What you saw first: pay_to_shop is in block C, one confirmation.
chain_seen_first = [genesis, block_B, block_C_with_pay]
# Later, the other side gets longer -- and it does not contain pay_to_shop.
block_C_other = {"name": "C_other", "txs": ["tx2"], "work": 1}
block_D_other = {"name": "D_other", "txs": ["tx3"], "work": 1}
chain_later_wins = [genesis, block_B, block_C_other, block_D_other]
winner = chain_later_wins if chain_work(chain_later_wins) > chain_work(chain_seen_first) else chain_seen_first
dropped = [tx for tx in all_txs(chain_seen_first) if tx not in all_txs(winner)]
print("chain seen first :", [b["name"] for b in chain_seen_first], "work =", chain_work(chain_seen_first))
print("chain that wins :", [b["name"] for b in chain_later_wins], "work =", chain_work(chain_later_wins))
print("nodes now follow :", [b["name"] for b in winner])
print("dropped back to the mempool:", dropped)
chain seen first : ['A', 'B', 'C'] work = 3
chain that wins : ['A', 'B', 'C_other', 'D_other'] work = 4
nodes now follow : ['A', 'B', 'C_other', 'D_other']
dropped back to the mempool: ['pay_to_shop']
pay_to_shop is gone.
It had one confirmation. The merchant may already have shipped. Then another chain won and that transaction was thrown back into the mempool — as if it had never happened.
Note that nobody cheated here. Both chains are valid; network latency simply had different miners working on different branches, and the heavier one won. A reorg is not an attack. It is part of the design.
What a confirmation actually is¶
0 confirmations: in the mempool, not in a block
1 confirmation : in a block, but shallow
2 confirmations: another block on top
6 confirmations: buried deeper -- reorging it out requires far more workA confirmation is a measure of how deeply a transaction is buried in history.
And an attacker is not “editing one transaction.” He is racing every miner on Earth.
One confirmation means the attacker is one block behind. Six means six. If he is slower than the honest network, the further behind he falls, the lower his chance of ever catching up.
The whitepaper’s final section computes exactly this: given z blocks behind, what is the probability the attacker catches up?
The answer decays exponentially in z. So “six confirmations” is not a magic number. It is simply a depth at which the probability has become small enough to accept.
And where that depth should be depends on the amount. Chapter 8’s opportunity cost returns here: one confirmation is fine for a coffee; six may not be enough for a house.
(Remember section 5.2? Block intervals have enormous variance. So “six confirmations” does not mean “one hour” — it might be twenty minutes or it might be two. That geometric distribution again.)
YOUR TURN¶
# YOUR TURN:
# 1) Set max_size to 300. Which transactions still fit, and why isn't it the highest fee rate?
# 2) Change low's fee to 2000 and run add_with_rbf again. Rejected? Why?
# 3) Add a second high-fee child to the parent. What does the package fee rate become?
# 4) Add one more block to chain_seen_first (making it 4).
# How long must the competing chain be to reorg it away?
# -- that difference is exactly what "one more confirmation" buys you.
12.7 The ledger¶
| What you wrote | Field |
|---|---|
mempool — each node’s own waiting basket | Networking + node policy |
feerate — rent per byte | Economics |
pick_block() — greedy packing | Economics + algorithms (approximate knapsack) |
add_with_rbf() — the double-spend the rules accept | Node policy |
package_feerate() — CPFP | Miner economics |
merkle_root() — transaction set to block fingerprint | Cryptography (chapter 11’s) |
| Chain reorganization | Consensus |
This is the book’s most cross-cutting chapter: user experience, miner economics, node policy, Merkle structure and longest-chain consensus all meet here.
It is also the chapter closest to the lived experience of using Bitcoin. Every “speed up transaction” button, every estimated confirmation time, every “why hasn’t this arrived yet” traces back to this material.
Next¶
Switching to the real curve, secp256k1.
The close of Part IV, and something of a ceremony.
You will replace the toy curve from chapters 2 and 3 — mod 23, twenty-nine points — with the one Bitcoin actually uses, and find that almost nothing in your code has to change.
Exactly one thing must. And that one thing is chapter 2’s complexity intuition, finally put to work.