dc27332538
Some checks failed
Deploy to Testnet / deploy-testnet (push) Successful in 1m12s
Documentation Validation / validate-docs (push) Failing after 10s
Documentation Validation / validate-policies-strict (push) Successful in 4s
Node Failover Simulation / failover-test (push) Failing after 3s
Multi-Node Stress Testing / stress-test (push) Successful in 3s
Cross-Node Transaction Testing / transaction-test (push) Successful in 3s
- Update documentation to reflect consolidation of production.env into blockchain.env - Change file references in configuration management, agent training, and release docs - Update troubleshooting and setup instructions to use blockchain.env - Remove references to deprecated 3-file structure (blockchain.env, node.env, production.env) - Document completed consolidation benefits and migration steps
105 lines
4.0 KiB
Markdown
105 lines
4.0 KiB
Markdown
---
|
|
name: aitbc-node-coordination
|
|
description: Cross-node operations including synchronization, coordination, messaging, and multi-node status checks between genesis and follower nodes
|
|
category: operations
|
|
---
|
|
|
|
# AITBC Node Coordination Skill
|
|
|
|
## Trigger Conditions
|
|
Activate when user requests cross-node operations: synchronization, coordination, messaging, or multi-node status checks.
|
|
|
|
## Purpose
|
|
Coordinate cross-node operations, synchronize blockchain state, and manage inter-node messaging between genesis and follower nodes.
|
|
|
|
## Prerequisites
|
|
- SSH access configured between genesis (aitbc) and follower (aitbc1) with key-based authentication
|
|
- Blockchain nodes operational on both nodes via systemd services
|
|
- P2P mesh network active on port 7070 with peer configuration
|
|
- Unique node IDs configured (proposer_id and p2p_node_id in `/etc/aitbc/node.env`)
|
|
- Git synchronization configured between nodes at `/opt/aitbc/.git`
|
|
|
|
## Operations
|
|
|
|
### Check Multi-Node Status
|
|
```bash
|
|
# Check all three nodes
|
|
cd /opt/aitbc
|
|
echo "=== Genesis ===" && git status --short && git rev-parse --short HEAD
|
|
echo "=== Follower ===" && ssh aitbc1 'cd /opt/aitbc && git status --short && git rev-parse --short HEAD'
|
|
echo "=== Gitea-Runner ===" && ssh gitea-runner 'cd /opt/aitbc && git status --short && git rev-parse --short HEAD'
|
|
```
|
|
|
|
### Sync All Nodes from Genesis
|
|
```bash
|
|
# 1. Commit and push from genesis
|
|
cd /opt/aitbc
|
|
git add . && git commit -m "feat: description" && git push origin main
|
|
|
|
# 2. Pull on follower
|
|
ssh aitbc1 'cd /opt/aitbc && git pull origin main'
|
|
|
|
# 3. Pull on gitea-runner
|
|
ssh gitea-runner 'cd /opt/aitbc && git pull origin main'
|
|
```
|
|
|
|
### Handle Sync Conflicts
|
|
```bash
|
|
# If git pull fails on remote node
|
|
ssh aitbc1 'cd /opt/aitbc && git checkout --force . && git clean -fd && git pull origin main'
|
|
ssh gitea-runner 'cd /opt/aitbc && git checkout --force . && git clean -fd && git pull origin main'
|
|
```
|
|
|
|
### Service Restart After Sync
|
|
```bash
|
|
# Restart services that need code updates
|
|
ssh aitbc1 'systemctl restart aitbc-agent-coordinator.service'
|
|
ssh aitbc1 'systemctl restart aitbc-blockchain-node.service'
|
|
ssh gitea-runner 'systemctl restart aitbc-blockchain-node.service'
|
|
```
|
|
|
|
### Check Blockchain Sync Status
|
|
```bash
|
|
# Check blockchain height on all nodes
|
|
./aitbc-cli chain
|
|
ssh aitbc1 'cd /opt/aitbc && ./aitbc-cli chain'
|
|
ssh gitea-runner 'cd /opt/aitbc && ./aitbc-cli chain'
|
|
```
|
|
|
|
### Check Node Health
|
|
```bash
|
|
# Check service status on all nodes
|
|
systemctl status aitbc-blockchain-node.service
|
|
ssh aitbc1 'systemctl status aitbc-blockchain-node.service'
|
|
ssh gitea-runner 'systemctl status aitbc-blockchain-node.service'
|
|
```
|
|
|
|
## Common Pitfalls
|
|
|
|
1. **SSH Connectivity Issues:** Verify SSH keys are configured at `/root/.ssh/` for passwordless access
|
|
2. **Git Conflicts:** Use `--force` flag with caution, prefer manual resolution
|
|
3. **P2P Handshake Rejection:** Check for duplicate p2p_node_id, run `/opt/aitbc/scripts/utils/generate_unique_node_ids.py`
|
|
4. **Service Restart Failures:** Check systemd logs: `journalctl -u aitbc-blockchain-node.service -n 50`
|
|
5. **Sync Partial Failure:** Identify which sync type failed (blockchain, mempool, configuration, git)
|
|
|
|
## Verification Checklist
|
|
- [ ] SSH connectivity to all nodes verified
|
|
- [ ] Git status consistent across all nodes
|
|
- [ ] Blockchain heights match across nodes
|
|
- [ ] P2P mesh network operational (port 7070)
|
|
- [ ] Services running on all nodes
|
|
- [ ] Node IDs unique (no duplicate p2p_node_id)
|
|
|
|
## Node Architecture
|
|
- **Genesis Node** (localhost): `/opt/aitbc` - Primary development node
|
|
- **Follower Node** (aitbc1): `/opt/aitbc` - Secondary blockchain node
|
|
- **Gitea-Runner Node** (gitea-runner): `/opt/aitbc` - CI/CD runner node (also hosts aitbc2 blockchain)
|
|
|
|
## Git Remote Strategy
|
|
- **Primary Remote:** `origin` (Gitea) - Daily development operations
|
|
- **Secondary Remote:** `github` - Milestone releases only
|
|
|
|
## CLI Tool Preference
|
|
- **Primary CLI:** `/opt/aitbc/aitbc-cli` is the single CLI entry point
|
|
- **SSH Access:** Use `ssh aitbc1` for follower node, `ssh gitea-runner` for CI/CD node
|