aitbc/docs/development/9_block-production-runbook.md

Block Production Operational Runbook

Architecture Overview

Clients → RPC /sendTx → Mempool → PoA Proposer → Block (with Transactions)
                                      ↓
                              Circuit Breaker
                              (graceful degradation)

Configuration

Setting	Default	Env Var	Description
block_time_seconds	2	BLOCK_TIME_SECONDS	Block interval
max_block_size_bytes	1,000,000	MAX_BLOCK_SIZE_BYTES	Max block size (1 MB)
max_txs_per_block	500	MAX_TXS_PER_BLOCK	Max transactions per block
min_fee	0	MIN_FEE	Minimum fee to accept into mempool
mempool_backend	memory	MEMPOOL_BACKEND	"memory" or "database"
mempool_max_size	10,000	MEMPOOL_MAX_SIZE	Max pending transactions
circuit_breaker_threshold	5	CIRCUIT_BREAKER_THRESHOLD	Failures before circuit opens
circuit_breaker_timeout	30	CIRCUIT_BREAKER_TIMEOUT	Seconds before half-open retry

Mempool Backends

In-Memory (default)

• Fast, no persistence
• Lost on restart
• Suitable for devnet/testnet

Database-backed (SQLite)

• Persistent across restarts
• Shared between services via file
• Set MEMPOOL_BACKEND=database

Monitoring Metrics

Block Production

• blocks_proposed_total — Total blocks proposed
• chain_head_height — Current chain height
• last_block_tx_count — Transactions in last block
• last_block_total_fees — Total fees in last block
• block_build_duration_seconds — Time to build last block
• block_interval_seconds — Time between blocks

Mempool

• mempool_size — Current pending transaction count
• mempool_tx_added_total — Total transactions added
• mempool_tx_drained_total — Total transactions included in blocks
• mempool_evictions_total — Transactions evicted (low fee)

Circuit Breaker

• circuit_breaker_state — 0=closed, 1=open
• circuit_breaker_trips_total — Times circuit breaker opened
• blocks_skipped_circuit_breaker_total — Blocks skipped due to open circuit

RPC

• rpc_send_tx_total — Total transaction submissions
• rpc_send_tx_success_total — Successful submissions
• rpc_send_tx_rejected_total — Rejected (fee too low, validation)
• rpc_send_tx_failed_total — Failed (mempool unavailable)

Troubleshooting

Empty blocks (tx_count=0)

1. Check mempool size: GET /metrics → mempool_size
2. Verify transactions are being submitted: rpc_send_tx_total
3. Check if fees meet minimum: rpc_send_tx_rejected_total
4. Verify block size limits aren't too restrictive

Circuit breaker open

1. Check circuit_breaker_state metric (1 = open)
2. Review logs for repeated failures
3. Check database connectivity
4. Wait for timeout (default 30s) for automatic half-open retry
5. If persistent, restart the node

Mempool full

1. Check mempool_size vs MEMPOOL_MAX_SIZE
2. Low-fee transactions are auto-evicted
3. Increase MEMPOOL_MAX_SIZE or raise MIN_FEE

High block build time

1. Check block_build_duration_seconds
2. Reduce MAX_TXS_PER_BLOCK if too slow
3. Consider database mempool for large volumes
4. Check disk I/O if using SQLite backend

Transaction not included in block

1. Verify transaction was accepted: check tx_hash in response
2. Check fee is competitive (higher fee = higher priority)
3. Check transaction size vs MAX_BLOCK_SIZE_BYTES
4. Transaction may be queued — check mempool_size