aitbc/skills/aitbc-blockchain-troubleshooting.md

name, description, category

name	description	category
aitbc-blockchain-troubleshooting	Blockchain troubleshooting including sync issues, P2P problems, service failures, data corruption, and blockchain recovery operations	troubleshooting

AITBC Blockchain Troubleshooting Skill

Trigger Conditions

Activate when user requests blockchain troubleshooting: sync issues, P2P problems, service failures, data corruption, or blockchain recovery operations.

Purpose

Diagnose and troubleshoot AITBC blockchain issues including synchronization failures, P2P network problems, service failures, and data corruption.

Prerequisites

• SSH access to all nodes (aitbc, aitbc1, gitea-runner)
• Systemd services operational or accessible for debugging
• Log access at /var/log/aitbc/
• Data directory at /var/lib/aitbc/
• CLI accessible at /opt/aitbc/aitbc-cli

Operations

1. Initial Diagnosis

# Check service status on all nodes
systemctl status aitbc-blockchain-node.service
ssh aitbc1 'systemctl status aitbc-blockchain-node.service'
ssh gitea-runner 'systemctl status aitbc-blockchain-node.service'

# Check blockchain RPC health
curl http://localhost:8006/health
curl http://10.1.223.40:8006/health
curl http://aitbc1:8006/health

# Check P2P network status
netstat -an | grep 7070
ssh aitbc1 'netstat -an | grep 7070'

2. Blockchain Sync Issues

# Check blockchain height on all nodes
./aitbc-cli chain
ssh aitbc1 'cd /opt/aitbc && ./aitbc-cli chain'
ssh gitea-runner 'cd /opt/aitbc && ./aitbc-cli chain'

# Check mempool status
./aitbc-cli mempool status
ssh aitbc1 'cd /opt/aitbc && ./aitbc-cli mempool status'

# Check P2P connections
./aitbc-cli network
ssh aitbc1 'cd /opt/aitbc && ./aitbc-cli network'

2.1 Genesis Block Mismatch Issues

Symptoms:

• "Unhandled import case" errors during bulk sync
• Nodes unable to sync blocks for a specific chain
• Different genesis block hashes across nodes

Diagnosis:

# Check genesis block hashes across nodes
sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "SELECT chain_id, height, hash FROM block WHERE height=0"
ssh aitbc1 'sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "SELECT chain_id, height, hash FROM block WHERE height=0"'

# Check RPC bootstrap logs
journalctl -u aitbc-blockchain-node.service | grep -i "RPC bootstrap"

# Verify RPC endpoint is accessible
curl http://aitbc1:8006/rpc/genesis_allocations?chain_id=ait-testnet

Solution - Force RPC Bootstrap:

# Stop blockchain service
sudo systemctl stop aitbc-blockchain-node.service

# Delete genesis block from database
sqlite3 /var/lib/aitbc/data/<chain_id>/chain.db "DELETE FROM block WHERE chain_id='<chain_id>' AND height=0"

# Restart service to trigger RPC bootstrap
sudo systemctl start aitbc-blockchain-node.service

# Verify RPC bootstrap worked
journalctl -u aitbc-blockchain-node.service | grep -i "RPC bootstrap"

How RPC Bootstrap Works:

• Nodes attempt RPC bootstrap when genesis block is missing
• Fetches genesis block data (allocations, hash, state_root) from trusted peers
• Creates genesis block using RPC-provided data for consistency
• Falls back to local creation if RPC bootstrap fails

Configuration Requirements:

• default_peer_rpc_url must be set in blockchain.env
• Points to trusted peer with correct genesis block
• Multiple peers can be configured

3. P2P Network Problems

# Check P2P node IDs
cat /etc/aitbc/.env | grep p2p_node_id
ssh aitbc1 'cat /etc/aitbc/.env | grep p2p_node_id'

# Generate unique node IDs if duplicates found
/opt/aitbc/scripts/utils/generate_unique_node_ids.py

# Restart blockchain services
systemctl restart aitbc-blockchain-p2p.service
ssh aitbc1 'systemctl restart aitbc-blockchain-p2p.service'

4. Service Failures

# Check service logs
journalctl -u aitbc-blockchain-node.service -n 100
journalctl -u aitbc-blockchain-p2p.service -n 100
ssh aitbc1 'journalctl -u aitbc-blockchain-node.service -n 100'

# Check application logs
tail -f /var/log/aitbc/blockchain-node.log
tail -f /var/log/aitbc/blockchain-p2p.log
ssh aitbc1 'tail -f /var/log/aitbc/blockchain-node.log'

5. Data Corruption

# Check database integrity
sqlite3 /var/lib/aitbc/data/blockchain.db "PRAGMA integrity_check;"

# Check CoW status (Btrfs)
lsattr /var/lib/aitbc

# Disable CoW if enabled
chattr +C /var/lib/aitbc

# Enable WAL mode
sqlite3 /var/lib/aitbc/data/blockchain.db "PRAGMA journal_mode=WAL;"

6. Recovery Operations

# Stop blockchain services
systemctl stop aitbc-blockchain-node.service aitbc-blockchain-p2p.service
ssh aitbc1 'systemctl stop aitbc-blockchain-node.service aitbc-blockchain-p2p.service'

# Backup current data
cp -r /var/lib/aitbc/data /var/lib/aitbc/data.backup

# Restore from backup if needed
systemctl stop aitbc-blockchain-node.service
rm -rf /var/lib/aitbc/data/*
cp -r /var/lib/aitbc/data.backup/* /var/lib/aitbc/data/
systemctl start aitbc-blockchain-node.service

# Restart services
systemctl start aitbc-blockchain-node.service aitbc-blockchain-p2p.service
ssh aitbc1 'systemctl start aitbc-blockchain-node.service aitbc-blockchain-p2p.service'

7. Communication Test

# Run full communication test
./scripts/blockchain-communication-test.sh --full --debug

# Verify all services are healthy
curl http://localhost:8006/health
curl http://aitbc1:8006/health
curl http://10.1.223.40:8001/health
curl http://10.1.223.40:8011/health

# Check blockchain sync
NODE_URL=http://10.1.223.40:8006 ./aitbc-cli blockchain height
NODE_URL=http://aitbc1:8006 ./aitbc-cli blockchain height

Common Pitfalls

1. Duplicate P2P Node IDs: Check for duplicate p2p_node_id in /etc/aitbc/.env - generate unique IDs
2. Btrfs CoW Corruption: Disable CoW on /var/lib/aitbc with chattr +C
3. SQLite Corruption: Enable WAL mode and check database integrity
4. Port Mismatches: Coordinator API is on port 8011 (not 8000)
5. Service Start Order: Ensure P2P service starts before blockchain-node service
6. Network Connectivity: Verify P2P port 7070 is open and accessible
7. Data Directory Permissions: Ensure proper permissions on /var/lib/aitbc/data

Verification Checklist

• All blockchain services running
• Blockchain heights match across nodes
• P2P connections established (port 7070)
• RPC endpoints responding (port 8006)
• No duplicate P2P node IDs
• Database integrity check passes
• CoW disabled on data directory
• WAL mode enabled for database

CLI Tool Preference

• Primary CLI: /opt/aitbc/aitbc-cli is the single CLI entry point
• RPC URL: Default is http://localhost:8006
• Coordinator API: Port 8011 (not 8000)