docs: update blockchain README with new node setup and sync documentation
- Updated metadata (last updated date, file counts) - Replaced auto-generated structure with manual index - Added links to gitea-runner third node setup guide and blockchain sync troubleshooting docs - Updated category overview to reflect actual blockchain operations content
This commit is contained in:
aitbc
@@ -1,20 +1,16 @@
|
|||||||
# Blockchain Documentation
|
# Blockchain Documentation
|
||||||
|
|
||||||
**Generated**: 2026-03-08 13:06:38
|
**Generated**: 2026-03-08 13:06:38
|
||||||
**Total Files**: 1
|
**Last Updated**: 2026-04-20
|
||||||
**Documented Files**: 0
|
**Total Files**: 3
|
||||||
**Other Files**: 1
|
|
||||||
|
|
||||||
## Documented Files (Converted from Analysis)
|
## Documentation Files
|
||||||
|
|
||||||
|
|
||||||
## Other Documentation Files
|
|
||||||
|
|
||||||
- [Blockchain Documentation](README.md)
|
|
||||||
|
|
||||||
|
- [Adding gitea-runner as Third Blockchain Node (aitbc2)](adding_gitea_runner_as_third_node.md) - Complete guide for adding a third node to the AITBC blockchain network, including configuration steps, issues encountered, and verification procedures.
|
||||||
|
- [Blockchain Synchronization Issues and Fixes](blockchain_synchronization_issues_and_fixes.md) - Documentation of synchronization issues between AITBC nodes and their resolutions.
|
||||||
|
|
||||||
## Category Overview
|
## Category Overview
|
||||||
This section contains all documentation related to blockchain documentation. The documented files have been automatically converted from completed planning analysis files.
|
This section contains documentation related to blockchain node setup, synchronization, and network configuration.
|
||||||
|
|
||||||
---
|
---
|
||||||
*Auto-generated index*
|
*Manual index*
|
||||||
|
|||||||
362
docs/blockchain/adding_gitea_runner_as_third_node.md
Normal file
362
docs/blockchain/adding_gitea_runner_as_third_node.md
Normal file
@@ -0,0 +1,362 @@
|
|||||||
|
# Adding gitea-runner as Third Blockchain Node (aitbc2)
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This document describes the experience of adding `gitea-runner` as a third blockchain node (NODE_ID: `aitbc2`) to the existing AITBC network consisting of `aitbc` and `aitbc1`.
|
||||||
|
|
||||||
|
**Date**: April 20, 2026
|
||||||
|
**Objective**: Add gitea-runner as a 3rd blockchain node to improve network resilience and provide a dedicated CI runner with full blockchain access.
|
||||||
|
|
||||||
|
## Network Configuration
|
||||||
|
|
||||||
|
### Existing Nodes
|
||||||
|
- **aitbc** (localhost): `10.1.223.93`, NODE_ID: `aitbc`
|
||||||
|
- **aitbc1**: `10.1.223.40`, NODE_ID: `aitbc1`
|
||||||
|
|
||||||
|
### New Node
|
||||||
|
- **gitea-runner**: `10.1.223.98`, NODE_ID: `aitbc2`
|
||||||
|
|
||||||
|
### Shared Infrastructure
|
||||||
|
- **Redis Server**: `10.1.223.93:6379` (used for gossip backend)
|
||||||
|
- **P2P Port**: `7070` (all nodes)
|
||||||
|
- **RPC Port**: `8006` (all nodes)
|
||||||
|
- **Chain ID**: `ait-devnet`
|
||||||
|
|
||||||
|
## Configuration Steps
|
||||||
|
|
||||||
|
### Step 1: Configure Node Identity on gitea-runner
|
||||||
|
|
||||||
|
**File**: `/etc/aitbc/node.env`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# AITBC Node-Specific Environment Configuration
|
||||||
|
# This file contains variables unique to this node
|
||||||
|
# DO NOT commit to version control - each node has different values
|
||||||
|
|
||||||
|
# Node Identity
|
||||||
|
NODE_ID=aitbc2
|
||||||
|
|
||||||
|
# P2P Configuration
|
||||||
|
p2p_node_id=aitbc2
|
||||||
|
p2p_bind_host=0.0.0.0
|
||||||
|
p2p_bind_port=7070
|
||||||
|
proposer_id=ait1ytkh0cn8v2a4zjwzyav6854832myf9j7unsse8yntmuwzst4qhtqe9hqdw
|
||||||
|
|
||||||
|
# P2P Peers (comma-separated list of peer nodes)
|
||||||
|
p2p_peers=aitbc:7070,aitbc1:7070
|
||||||
|
|
||||||
|
# Trusted Proposers (for follower nodes)
|
||||||
|
trusted_proposers=
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Configure Shared Environment on gitea-runner
|
||||||
|
|
||||||
|
**File**: `/etc/aitbc/.env`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# AITBC Environment Configuration
|
||||||
|
# This file contains shared environment variables
|
||||||
|
# DO NOT commit to version control
|
||||||
|
|
||||||
|
# Database Configuration
|
||||||
|
DATABASE_URL=sqlite:////var/lib/aitbc/data/chain.db
|
||||||
|
|
||||||
|
# Redis Configuration
|
||||||
|
REDIS_URL=redis://10.1.223.93:6379/0
|
||||||
|
gossip_backend=broadcast
|
||||||
|
gossip_broadcast_url=redis://10.1.223.93:6379
|
||||||
|
|
||||||
|
# Blockchain Configuration
|
||||||
|
CHAIN_ID=ait-devnet
|
||||||
|
supported_chains=ait-devnet
|
||||||
|
BLOCK_TIME=5
|
||||||
|
enable_block_production=false # Disable block production to prevent forks
|
||||||
|
|
||||||
|
# RPC Configuration
|
||||||
|
rpc_bind_host=0.0.0.0
|
||||||
|
rpc_bind_port=8006
|
||||||
|
|
||||||
|
# API Configuration
|
||||||
|
api_host=0.0.0.0
|
||||||
|
api_port=8000
|
||||||
|
|
||||||
|
# Wallet Configuration
|
||||||
|
wallet_host=0.0.0.0
|
||||||
|
wallet_port=8003
|
||||||
|
|
||||||
|
# Exchange Configuration
|
||||||
|
exchange_host=0.0.0.0
|
||||||
|
exchange_port=8001
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Configuration Decisions**:
|
||||||
|
- `DATABASE_URL` points to `/var/lib/aitbc/data/chain.db` (not `/var/lib/aitbc/aitbc.db`)
|
||||||
|
- `CHAIN_ID` set to `ait-devnet` to match existing network
|
||||||
|
- `supported_chains` set to `ait-devnet` (critical - default is `ait-devnet` but must be explicit)
|
||||||
|
- `enable_block_production=false` to prevent fork creation (all nodes had same proposer_id)
|
||||||
|
|
||||||
|
### Step 3: Add Hostname Resolution
|
||||||
|
|
||||||
|
**File**: `/etc/hosts` (on all nodes)
|
||||||
|
|
||||||
|
Added the following entries to all three nodes:
|
||||||
|
|
||||||
|
```
|
||||||
|
10.1.223.93 aitbc
|
||||||
|
10.1.223.40 aitbc1
|
||||||
|
10.1.223.98 aitbc2 gitea-runner
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Update P2P Peers on Existing Nodes
|
||||||
|
|
||||||
|
**File**: `/etc/aitbc/node.env` on `aitbc` and `aitbc1`
|
||||||
|
|
||||||
|
Updated `p2p_peers` to include the new node:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On aitbc
|
||||||
|
p2p_peers=aitbc1:7070,aitbc2:7070
|
||||||
|
|
||||||
|
# On aitbc1
|
||||||
|
p2p_peers=aitbc:7070,aitbc2:7070
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5: Add Missing P2P Bind Configuration
|
||||||
|
|
||||||
|
**File**: `/etc/aitbc/node.env` (all nodes)
|
||||||
|
|
||||||
|
Added missing environment variables that were causing P2P service failures:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
p2p_bind_host=0.0.0.0
|
||||||
|
p2p_bind_port=7070
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 6: Update Gossip Backend Configuration
|
||||||
|
|
||||||
|
**File**: `/etc/aitbc/.env` (all nodes)
|
||||||
|
|
||||||
|
Updated to use shared Redis instance for gossip:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
REDIS_URL=redis://10.1.223.93:6379/0
|
||||||
|
gossip_backend=broadcast
|
||||||
|
gossip_broadcast_url=redis://10.1.223.93:6379
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 7: Restart P2P Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On all nodes
|
||||||
|
systemctl restart aitbc-blockchain-p2p.service
|
||||||
|
systemctl status aitbc-blockchain-p2p.service
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 8: Copy Blockchain Database
|
||||||
|
|
||||||
|
To sync gitea-runner with the existing network, copied the blockchain database from `aitbc`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop blockchain node on aitbc
|
||||||
|
systemctl stop aitbc-blockchain-node.service
|
||||||
|
|
||||||
|
# Copy database to gitea-runner
|
||||||
|
scp /var/lib/aitbc/data/chain.db gitea-runner:/var/lib/aitbc/data/chain.db
|
||||||
|
|
||||||
|
# Restart blockchain node on aitbc
|
||||||
|
systemctl start aitbc-blockchain-node.service
|
||||||
|
|
||||||
|
# Start blockchain node on gitea-runner
|
||||||
|
ssh gitea-runner "systemctl start aitbc-blockchain-node.service"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 9: Restart Blockchain Node Services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On all nodes
|
||||||
|
systemctl restart aitbc-blockchain-node.service
|
||||||
|
```
|
||||||
|
|
||||||
|
## Issues Encountered and Resolutions
|
||||||
|
|
||||||
|
### Issue 1: P2P Service Restarting Continuously
|
||||||
|
|
||||||
|
**Symptom**: `aitbc-blockchain-p2p.service` failed with "invalid int value: ''" for `--port`
|
||||||
|
|
||||||
|
**Cause**: Missing `p2p_bind_host` and `p2p_bind_port` environment variables in `node.env`
|
||||||
|
|
||||||
|
**Resolution**: Added the missing variables to all nodes' `node.env` files:
|
||||||
|
```bash
|
||||||
|
p2p_bind_host=0.0.0.0
|
||||||
|
p2p_bind_port=7070
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 2: CHAIN_ID Mismatch
|
||||||
|
|
||||||
|
**Symptom**: gitea-runner blockchain node logs showed "Importing block for chain ait-devnet" while other nodes were on `ait-testnet`
|
||||||
|
|
||||||
|
**Initial Misunderstanding**: Initially thought the network was on `ait-testnet` and tried to change gitea-runner to match
|
||||||
|
|
||||||
|
**Resolution**: Verified that the existing network was actually on `ait-devnet` and set gitea-runner to match:
|
||||||
|
```bash
|
||||||
|
CHAIN_ID=ait-devnet
|
||||||
|
supported_chains=ait-devnet
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 3: Block Creation Causing Forks
|
||||||
|
|
||||||
|
**Symptom**: gitea-runner was proposing blocks simultaneously with other nodes, causing "Fork detected" errors
|
||||||
|
|
||||||
|
**Cause**: All three nodes had the same `proposer_id`, causing them to all act as proposers and create competing chains
|
||||||
|
|
||||||
|
**Resolution**: Disabled block production on gitea-runner by setting:
|
||||||
|
```bash
|
||||||
|
enable_block_production=false
|
||||||
|
```
|
||||||
|
|
||||||
|
This makes gitea-runner a follower node that only receives blocks from the network via gossip.
|
||||||
|
|
||||||
|
### Issue 4: Database Location Mismatch
|
||||||
|
|
||||||
|
**Symptom**: gitea-runner blockchain height remained at 25 despite copying database
|
||||||
|
|
||||||
|
**Cause**: `DATABASE_URL` in `.env` was set to `sqlite:////var/lib/aitbc/aitbc.db` but the config uses `db_path` which defaults to `/var/lib/aitbc/data/chain.db`
|
||||||
|
|
||||||
|
**Resolution**: Updated `DATABASE_URL` to match the actual database location:
|
||||||
|
```bash
|
||||||
|
DATABASE_URL=sqlite:////var/lib/aitbc/data/chain.db
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue 5: supported_chains Default Value
|
||||||
|
|
||||||
|
**Symptom**: Even with `CHAIN_ID=ait-devnet` set, gitea-runner was still using `ait-devnet` from the default `supported_chains` value in config.py
|
||||||
|
|
||||||
|
**Cause**: The blockchain node uses `supported_chains` (default: "ait-devnet") to determine which chains to propose blocks on, not just `CHAIN_ID`
|
||||||
|
|
||||||
|
**Resolution**: Explicitly set `supported_chains` in `.env`:
|
||||||
|
```bash
|
||||||
|
supported_chains=ait-devnet
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
### P2P Layer Verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check P2P service status on all nodes
|
||||||
|
systemctl status aitbc-blockchain-p2p.service
|
||||||
|
|
||||||
|
# Verify P2P connections
|
||||||
|
# (Check logs for successful peer connections)
|
||||||
|
journalctl -u aitbc-blockchain-p2p.service -n 50
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**: All three nodes successfully connected to each other via P2P.
|
||||||
|
|
||||||
|
### Blockchain Layer Verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check blockchain height on all nodes
|
||||||
|
./aitbc-cli blockchain height
|
||||||
|
ssh aitbc1 "/opt/aitbc/aitbc-cli blockchain height"
|
||||||
|
ssh gitea-runner "/opt/aitbc/aitbc-cli blockchain height"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**:
|
||||||
|
- aitbc: ~4686
|
||||||
|
- aitbc1: ~4686
|
||||||
|
- gitea-runner: Syncing via gossip, logs show successful block imports
|
||||||
|
|
||||||
|
### Log Verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check gitea-runner blockchain node logs
|
||||||
|
ssh gitea-runner "journalctl -u aitbc-blockchain-node.service -n 30 --no-pager"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Log Output**:
|
||||||
|
```
|
||||||
|
{"timestamp": "2026-04-20T11:50:12.394675Z", "level": "INFO", "logger": "aitbc_chain.main", "message": "Importing block for chain ait-devnet: 4673"}
|
||||||
|
{"timestamp": "2026-04-20T11:50:12.521809Z", "level": "INFO", "logger": "aitbc_chain.sync", "message": "Imported block", "height": 4673, "hash": "...", "proposer": "..."}
|
||||||
|
{"timestamp": "2026-04-20T11:50:12.521913Z", "level": "INFO", "logger": "aitbc_chain.main", "message": "Import result: accepted=True, reason=Appended to chain"}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**: gitea-runner successfully importing blocks via gossip with "accepted=True" status.
|
||||||
|
|
||||||
|
## Final Status
|
||||||
|
|
||||||
|
**P2P Layer**: ✅ Working
|
||||||
|
- All three nodes connected via P2P
|
||||||
|
- Peer discovery working correctly
|
||||||
|
- No connection errors
|
||||||
|
|
||||||
|
**Blockchain Layer**: ✅ Working
|
||||||
|
- gitea-runner syncing blocks via gossip
|
||||||
|
- Block production disabled to prevent forks
|
||||||
|
- Logs show successful block imports
|
||||||
|
- No fork detection errors
|
||||||
|
|
||||||
|
**Configuration**: ✅ Complete
|
||||||
|
- Node identity configured (aitbc2)
|
||||||
|
- P2P peers configured on all nodes
|
||||||
|
- Hostname resolution working
|
||||||
|
- Shared Redis gossip backend configured
|
||||||
|
- Database location corrected
|
||||||
|
- Chain ID and supported_chains aligned with network
|
||||||
|
|
||||||
|
## Key Learnings
|
||||||
|
|
||||||
|
1. **supported_chains is Critical**: The blockchain node uses `supported_chains` to determine which chains to propose blocks on. This must be set explicitly, not just `CHAIN_ID`.
|
||||||
|
|
||||||
|
2. **Database Location Matters**: The `DATABASE_URL` in `.env` must match the actual database location used by the config (`db_path` defaults to `/var/lib/aitbc/data/chain.db`).
|
||||||
|
|
||||||
|
3. **Proposer ID Conflicts**: All nodes with the same `proposer_id` will propose blocks simultaneously, causing forks. Disable block production on follower nodes.
|
||||||
|
|
||||||
|
4. **P2P Bind Configuration**: Missing `p2p_bind_host` and `p2p_bind_port` causes P2P service failures. These must be set in `node.env`.
|
||||||
|
|
||||||
|
5. **Environment Variable Loading**: Systemd services load environment variables from `EnvironmentFile` directives. Verify that the correct files are being loaded (`/etc/aitbc/.env` and `/etc/aitbc/node.env`).
|
||||||
|
|
||||||
|
## Recommendations
|
||||||
|
|
||||||
|
### For Future Node Additions
|
||||||
|
|
||||||
|
1. **Create a Configuration Template**: Create a template `node.env` file with all required variables pre-populated.
|
||||||
|
|
||||||
|
2. **Automate Hostname Resolution**: Add `/etc/hosts` entries to the provisioning script.
|
||||||
|
|
||||||
|
3. **Unique Proposer IDs**: Consider using unique proposer IDs for each node to prevent fork conflicts.
|
||||||
|
|
||||||
|
4. **Configuration Validation**: Add a validation script to check that all required environment variables are set correctly before starting services.
|
||||||
|
|
||||||
|
5. **Sync Mechanism**: Implement a proper blockchain sync mechanism instead of manual database copying for initial sync.
|
||||||
|
|
||||||
|
### For Existing Network
|
||||||
|
|
||||||
|
1. **Review Proposer Configuration**: Consider whether all nodes should be proposers or if a subset should be designated.
|
||||||
|
|
||||||
|
2. **Monitor Forks**: Add monitoring for fork detection events to identify configuration issues early.
|
||||||
|
|
||||||
|
3. **Document Chain IDs**: Clearly document which chain ID the production network uses to avoid confusion.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
### Related Documentation
|
||||||
|
- [Blockchain Synchronization Issues and Fixes](./blockchain_synchronization_issues_and_fixes.md)
|
||||||
|
- [Multi-Node Blockchain Setup](../../.windsurf/workflows/multi-node-blockchain-setup.md)
|
||||||
|
|
||||||
|
### Configuration Files
|
||||||
|
- `/etc/aitbc/.env` (shared environment)
|
||||||
|
- `/etc/aitbc/node.env` (node-specific environment)
|
||||||
|
- `/etc/systemd/system/aitbc-blockchain-node.service`
|
||||||
|
- `/etc/systemd/system/aitbc-blockchain-p2p.service`
|
||||||
|
|
||||||
|
### Source Files
|
||||||
|
- `/opt/aitbc/apps/blockchain-node/src/aitbc_chain/config.py`
|
||||||
|
- `/opt/aitbc/apps/blockchain-node/src/aitbc_chain/main.py`
|
||||||
|
- `/opt/aitbc/apps/blockchain-node/src/aitbc_chain/p2p_network.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated**: 2026-04-20
|
||||||
|
**Version**: 1.0
|
||||||
|
**Status**: Complete
|
||||||
Reference in New Issue
Block a user