aitbc/docs/blockchain/blockchain_synchronization_issues_and_fixes.md

Blockchain Synchronization Issues and Fixes

Overview

This document describes the blockchain synchronization issues discovered between the AITBC nodes (aitbc and aitbc1) and the fixes implemented to resolve them.

Network Configuration

• Leader Node (aitbc): 10.1.223.93:8006 - Produces blocks, should NOT run chain-sync service
• Follower Node (aitbc1): 10.1.223.40:8006 - Imports blocks from leader, runs chain-sync service
• RPC Port: 8006 on both nodes
• Database Location: /var/lib/aitbc/data/ait-mainnet/chain.db

Service Configuration Requirements

Chain-Sync Service

The aitbc-blockchain-sync service should only run on follower nodes, not on the leader node.

Why: The chain-sync service imports blocks from a remote RPC endpoint. When running on the leader node:

• It would try to import blocks it just created locally
• This causes hash conflict warnings in the RPC logs
• It's redundant since the leader already has all blocks
• It creates unnecessary load on the system

Configuration:

• Leader (aitbc): aitbc-blockchain-sync service should be disabled
• Followers (aitbc1, gitea-runner): aitbc-blockchain-sync service should be enabled

How to disable on leader node:

systemctl disable --now aitbc-blockchain-sync

How to enable on follower node:

systemctl enable --now aitbc-blockchain-sync

Issues Identified

Issue 1: Rate Limiting on Import Block

Description

The /rpc/importBlock RPC endpoint had a 1-second minimum rate limit that significantly slowed down block synchronization when importing large numbers of blocks.

Location

File: /opt/aitbc/apps/blockchain-node/src/aitbc_chain/rpc/router.py

Original Code

@router.post("/importBlock", summary="Import a block")
async def import_block(block_data: dict) -> Dict[str, Any]:
    """Import a block into the blockchain"""
    global _last_import_time
    
    async with _import_lock:
        try:
            # Rate limiting: max 1 import per second
            current_time = time.time()
            time_since_last = current_time - _last_import_time
            if time_since_last < 1.0:  # 1 second minimum between imports
                await asyncio.sleep(1.0 - time_since_last)
            
            _last_import_time = current.time()
            # ... rest of implementation

Impact

• Synchronizing 4,000+ blocks would take over an hour
• Follower node would fall behind during high-throughput periods
• Manual testing was severely slowed down

Fix Applied

Temporarily disabled the rate limit for development/testing:

# Rate limiting: max 1 import per second
current_time = time.time()
time_since_last = current_time - _last_import_time
if False:  # time_since_last < 1.0:  # 1 second minimum between imports
    await asyncio.sleep(1.0 - time_since_last)

_last_import_time = time.time()

Status

• Applied: April 10, 2026
• Type: Temporary workaround
• Recommended Action: Implement proper rate limiting with configuration option

---

Issue 2: Blocks-Range Endpoint Missing Transaction Data

Description

The /rpc/blocks-range RPC endpoint returns block metadata but does not include the actual transaction data. This causes follower nodes to import "empty" blocks when using the standard P2P synchronization mechanism.

Location

File: /opt/aitbc/apps/blockchain-node/src/aitbc_chain/rpc/router.py

Current Implementation

@router.get("/blocks-range", summary="Get blocks in height range")
async def get_blocks_range(start: int = 0, end: int = 10) -> Dict[str, Any]:
    """Get blocks in a height range"""
    with session_scope() as session:
        from ..config import settings as cfg
        blocks = session.exec(
            select(Block).where(
                Block.chain_id == cfg.chain_id,
                Block.height >= start,
                Block.height <= end,
            ).order_by(Block.height.asc())
        ).all()
        return {
            "success": True,
            "blocks": [
                {
                    "height": b.height, 
                    "hash": b.hash, 
                    "timestamp": b.timestamp.isoformat(), 
                    "tx_count": b.tx_count  # Only metadata
                } 
                for b in blocks
            ],
            "count": len(blocks),
        }

Impact

• Follower nodes import blocks without transactions
• Account balances don't update correctly
• Smart contract state doesn't propagate
• Cross-node communication fails

Evidence

When querying the database after sync:

# Block metadata shows tx_count = 1
block = session.exec(select(Block).where(Block.height == 26931)).first()
# Result: {'tx_count': 0, ...}  # Transactions not imported

Workaround Applied

Instead of relying on /rpc/blocks-range, the agent daemon directly queries the local SQLite database:

from sqlmodel import create_engine, Session, select
from aitbc_chain.models import Transaction

engine = create_engine("sqlite:////var/lib/aitbc/data/ait-mainnet/chain.db")
with Session(engine) as session:
    txs = session.exec(select(Transaction).where(...)).all()

Recommended Fix

Update /rpc/blocks-range to include transaction data:

@router.get("/blocks-range", summary="Get blocks in height range")
async def get_blocks_range(start: int = 0, end: int = 10, include_tx: bool = True) -> Dict[str, Any]:
    """Get blocks in a height range"""
    with session_scope() as session:
        from ..config import settings as cfg
        blocks = session.exec(
            select(Block).where(
                Block.chain_id == cfg.chain_id,
                Block.height >= start,
                Block.height <= end,
            ).order_by(Block.height.asc())
        ).all()
        
        result_blocks = []
        for b in blocks:
            block_data = {
                "height": b.height, 
                "hash": b.hash, 
                "timestamp": b.timestamp.isoformat(), 
                "tx_count": b.tx_count
            }
            
            if include_tx:
                txs = session.exec(
                    select(Transaction).where(Transaction.block_height == b.height)
                ).all()
                block_data["transactions"] = [tx.model_dump() for tx in txs]
            
            result_blocks.append(block_data)
        
        return {
            "success": True,
            "blocks": result_blocks,
            "count": len(blocks),
        }

Status

• Identified: April 10, 2026
• Workaround: Direct database queries
• Recommended Action: Implement proper fix in /rpc/blocks-range

---

Issue 3: Chain Sync Service Broadcasting Instead of Importing

Description

The chain_sync.py service on the follower node was configured to broadcast its local blocks to the genesis node instead of importing from it, causing it to propagate empty blocks.

Location

File: /opt/aitbc/apps/blockchain-node/src/aitbc_chain/chain_sync.py

Configuration Issue

The sync service had incorrect source/target configuration:

• Expected behavior: Import blocks from genesis node
• Actual behavior: Broadcast local blocks to genesis node

Log Evidence

Apr 10 13:03:19 aitbc python3[25724]: INFO:__main__:Broadcasted block 26921
Apr 10 13:03:19 aitbc python3[25724]: INFO:__main__:Broadcasted block 26922

Fix Applied

Manual synchronization using custom script:

# /tmp/sync_once.py
import requests

def main():
    # Get head from aitbc1
    r = requests.get("http://10.1.223.40:8006/rpc/head")
    target_height = r.json()["height"]
    
    # Get local head
    r_local = requests.get("http://localhost:8006/rpc/head")
    local_height = r_local.json()["height"]
    
    # Import missing blocks
    for h in range(local_height + 1, target_height + 1):
        block_r = requests.get(f"http://10.1.223.40:8006/rpc/blocks-range?start={h}&end={h}")
        if block_r.status_code == 200:
            blocks = block_r.json().get("blocks", [])
            if blocks:
                res = requests.post("http://localhost:8006/rpc/importBlock", json=blocks[0])
                print(f"Imported block {h}: {res.json()}")

Status

• Fixed: April 10, 2026 (manual sync)
• Recommended Action: Review chain_sync.py configuration

---

Issue 4: Missing RPC Endpoints

Description

Several expected RPC endpoints are not implemented, requiring workarounds for transaction queries.

Missing Endpoints

1. /rpc/transactions?address={addr} - Query transactions by address
2. /rpc/transaction/{hash} - Get specific transaction details

Impact

• aitbc-cli wallet transactions command fails with "Not Found"
• Agent messaging requires direct database queries
• Transaction status checking is limited

Workaround

Direct database queries using SQLModel:

from sqlmodel import create_engine, Session, select
from aitbc_chain.models import Transaction

engine = create_engine("sqlite:////var/lib/aitbc/data/ait-mainnet/chain.db")
with Session(engine) as session:
    txs = session.exec(select(Transaction).where(...)).all()

Recommended Fix

Implement missing RPC endpoints in router.py:

@router.get("/transactions", summary="Get transactions for address")
async def get_transactions(address: str, limit: int = 50) -> Dict[str, Any]:
    """Get transactions for a specific address"""
    with session_scope() as session:
        txs = session.exec(
            select(Transaction).where(
                Transaction.sender == address
            ).order_by(Transaction.timestamp.desc()).limit(limit)
        ).all()
        return {
            "success": True,
            "transactions": [tx.model_dump() for tx in txs],
            "count": len(txs)
        }

@router.get("/transactions/{tx_hash}", summary="Get transaction by hash")
async def get_transaction(tx_hash: str) -> Dict[str, Any]:
    """Get specific transaction details"""
    with session_scope() as session:
        tx = session.exec(
            select(Transaction).where(Transaction.tx_hash == tx_hash)
        ).first()
        if tx:
            return {"success": True, "transaction": tx.model_dump()}
        return {"success": False, "error": "Transaction not found"}

Status

• Identified: April 10, 2026
• Workaround: Direct database queries
• Recommended Action: Implement missing endpoints

---

Manual Synchronization Procedure

Step 1: Stop Services

systemctl stop aitbc-blockchain-node aitbc-blockchain-sync
ssh aitbc1 'systemctl stop aitbc-blockchain-node'

Step 2: Copy Database Files

# Copy chain.db, chain.db-wal, chain.db-shm from aitbc1 to aitbc
scp aitbc1:/var/lib/aitbc/data/ait-mainnet/chain.db* /var/lib/aitbc/data/ait-mainnet/
chown aitbc:aitbc /var/lib/aitbc/data/ait-mainnet/chain.db*

Step 3: Restart Services

ssh aitbc1 'systemctl start aitbc-blockchain-node'
systemctl start aitbc-blockchain-node aitbc-blockchain-sync

Step 4: Verify Synchronization

NODE_URL=http://localhost:8006 ./aitbc-cli blockchain height
ssh aitbc1 'NODE_URL=http://localhost:8006 /opt/aitbc/aitbc-cli blockchain height'

---

Monitoring Synchronization Status

Watch Script

#!/bin/bash
while true; do
  clear
  echo "Watching block sync on both nodes..."
  echo ""
  echo "Genesis (aitbc1) Block Height:"
  NODE_URL=http://10.1.223.40:8006 /opt/aitbc/aitbc-cli blockchain height
  echo ""
  echo "Follower (aitbc) Block Height:"
  NODE_URL=http://localhost:8006 /opt/aitbc/aitbc-cli blockchain height
  echo ""
  sleep 5
done

Check Logs

# Follower node sync logs
journalctl -u aitbc-blockchain-sync -f

# Genesis node logs
ssh aitbc1 'journalctl -u aitbc-blockchain-node -f'

---

Performance Metrics

Before Fixes

• Sync Rate: ~1 block/second (due to rate limiting)
• Time to sync 4,000 blocks: ~1 hour
• Block Import Success: 0% (empty blocks due to missing transaction data)

After Fixes

• Sync Rate: ~50 blocks/second (rate limit disabled)
• Time to sync 4,000 blocks: ~80 seconds (manual sync)
• Block Import Success: 100% (manual DB copy)

---

Recommendations

Short-term

1. Implement proper rate limiting configuration - Allow disabling rate limit via config
2. Fix /rpc/blocks-range endpoint - Include transaction data by default
3. Implement missing transaction endpoints - /rpc/transactions and /rpc/transactions/{hash}
4. Review chain_sync.py configuration - Ensure correct source/target setup

Long-term

1. Implement proper P2P block propagation - Include transactions in gossip protocol
2. Add synchronization monitoring - Automated alerts for sync failures
3. Implement state reconciliation - Periodic full-state verification
4. Add transaction replay protection - Better nonce management

---

References

Related Documentation

• Cross-Node Communication Guide
• RPC API Documentation
• Chain Sync Service

Source Files

• /opt/aitbc/apps/blockchain-node/src/aitbc_chain/rpc/router.py
• /opt/aitbc/apps/blockchain-node/src/aitbc_chain/chain_sync.py
• /opt/aitbc/apps/blockchain-node/src/aitbc_chain/sync.py

---

Last Updated: 2026-04-28 Version: 1.1 Status: Active Issues Documented