docs: update cross-node communication guides for systemd service management

- Add environment variables section (GENESIS_IP, FOLLOWER_IP, RPC_PORT, CHAIN_ID) - Document systemd service management for agent daemon - Update wallet configuration with password file paths - Add service management commands (start, status, restart, logs) - Document service file locations and wrapper scripts - Update troubleshooting section with systemd commands - Replace hardcoded IPs with environment variables - Add service health checks to prerequisites - Update version to 2
2026-05-09 17:27:26 +02:00
parent 81fea29941
commit f6f6a41191
5 changed files with 301 additions and 95 deletions
--- a/docs/MASTER_INDEX.md
+++ b/docs/MASTER_INDEX.md
@@ -94,6 +94,10 @@ Documentation about the documentation system itself
  - [Agent Global AI](agents/AGENT_GLOBAL_AI.md)
  - [Agent Explorer](agents/AGENT_EXPLORER.md)

+### 📦 **Applications Documentation**
+- **[Apps](apps/)** - Applications documentation (72 items)
+  - Application-level services, components, and integrations
+
 ### 🏗️ Architecture Documentation
 - **[Architecture](architecture/)** - System architecture and design patterns (10 items)

--- a/docs/hermes/guides/openclaw_cross_node_communication.md
+++ b/docs/hermes/guides/openclaw_cross_node_communication.md
@@ -11,6 +11,14 @@ This guide documents the successful implementation and testing of cross-node age
 - **Follower Node (aitbc)**: `10.1.223.93:8006` - Secondary blockchain node
 - **RPC Port**: `8006` on both nodes

+**Environment Variables**:
+```bash
+export GENESIS_IP="10.1.223.40"  # aitbc1 (genesis node)
+export FOLLOWER_IP="10.1.223.93"  # aitbc (follower node)
+export RPC_PORT="8006"
+export CHAIN_ID="ait-mainnet"
+```
+
 ### Communication Mechanism

 Agents communicate by embedding messages in blockchain transaction payloads:
@@ -32,28 +40,54 @@ Agents communicate by embedding messages in blockchain transaction payloads:

 ### Agent Daemon Architecture

-The autonomous agent daemon (`agent_daemon4.py`) runs on the follower node and:
+The autonomous agent daemon is managed as a systemd service (`aitbc-agent-daemon.service`) and runs on both nodes. The daemon:

-1. **Polls Blockchain State**: Queries the local SQLite database (`chain.db`) for incoming transactions
-2. **Filters Messages**: Identifies transactions addressed to the agent's wallet
-3. **Parses Payloads**: Extracts message content from transaction payloads
-4. **Autonomous Replies**: Constructs and broadcasts reply transactions
-5. **Cryptographic Signing**: Uses wallet private keys for transaction signing
+1. **Systemd Service**: Managed by systemd for reliable operation and automatic restart
+2. **Polls Blockchain State**: Queries the local SQLite database (`chain.db`) for incoming transactions
+3. **Filters Messages**: Identifies transactions addressed to the agent's wallet
+4. **Parses Payloads**: Extracts message content from transaction payloads
+5. **Autonomous Replies**: Constructs and broadcasts reply transactions
+6. **Cryptographic Signing**: Uses wallet private keys for transaction signing
+
+**Service Management**:
+```bash
+# Start the agent daemon
+sudo systemctl start aitbc-agent-daemon.service
+
+# Check service status
+sudo systemctl status aitbc-agent-daemon.service
+
+# View service logs
+sudo journalctl -u aitbc-agent-daemon -f
+
+# Restart the service
+sudo systemctl restart aitbc-agent-daemon.service
+```
+
+**Service Configuration**:
+- **Service File**: `/etc/systemd/system/aitbc-agent-daemon.service`
+- **Wrapper Script**: `/opt/aitbc/scripts/wrappers/aitbc-agent-daemon-wrapper.py`
+- **Main Script**: `/opt/aitbc/apps/agent-coordinator/scripts/agent_daemon.py`
+- **Wallet**: `temp-agent`
+- **Agent Address**: `ait1d18e286fc0c12888aca94732b5507c8787af71a5`
+- **Password File**: `/var/lib/aitbc/keystore/.agent_daemon_password`
+- **RPC URL**: `http://localhost:8006`
+- **Poll Interval**: 2 seconds

 ## Implementation Details

 ### Wallet Configuration

 #### Genesis Node (aitbc1)
- **Wallet**: `temp-agent2`
- **Address**: `ait16af0b743fd6a2d3e2e2f28a820066706aa5813b5`
- **Password**: `temp123`
+- **Wallet**: `temp-agent`
+- **Address**: `ait1d18e286fc0c12888aca94732b5507c8787af71a5`
+- **Password File**: `/var/lib/aitbc/keystore/.agent_daemon_password`
 - **Balance**: 49,990 AIT (after funding)

 #### Follower Node (aitbc)
 - **Wallet**: `temp-agent`
 - **Address**: `ait1d18e286fc0c12888aca94732b5507c8787af71a5`
- **Password**: `temp123`
+- **Password File**: `/var/lib/aitbc/keystore/.agent_daemon_password`
 - **Balance**: 0 AIT (sends zero-fee messages)

 ### Transaction Signing Process
@@ -257,30 +291,47 @@ Location: `/opt/aitbc/.windsurf/workflows/hermes-cross-node-communication.md`

 ### Agent Daemon Not Starting
 ```bash
-# Check logs
-ssh aitbc1 'cat /tmp/agent_daemon4.log'
+# Check service status
+sudo systemctl status aitbc-agent-daemon.service
+
+# Check service logs
+sudo journalctl -u aitbc-agent-daemon -n 50
+
+# Start the service
+sudo systemctl start aitbc-agent-daemon.service

 # Verify wallet decryption
-ssh aitbc1 '/opt/aitbc/venv/bin/python -c "from scripts import decrypt_wallet; print(decrypt_wallet(...))"'
+/opt/aitbc/venv/bin/python -c "
+from pathlib import Path
+import json
+keystore_path = Path('/var/lib/aitbc/keystore/temp-agent.json')
+with open(keystore_path) as f:
+    data = json.load(f)
+    print('Wallet loaded successfully')
+"
 ```

 ### Sync Issues
 ```bash
-# Manual sync script
-python /tmp/sync_once.py
+# Check block heights on both nodes
+NODE_URL=http://${GENESIS_IP}:${RPC_PORT} ./aitbc-cli blockchain height
+NODE_URL=http://${FOLLOWER_IP}:${RPC_PORT} ./aitbc-cli blockchain height

-# Check block heights
-NODE_URL=http://localhost:8006 ./aitbc-cli blockchain height
-ssh aitbc1 'NODE_URL=http://localhost:8006 /opt/aitbc/aitbc-cli blockchain height'
+# Check sync status
+curl http://${GENESIS_IP}:${RPC_PORT}/rpc/head
+curl http://${FOLLOWER_IP}:${RPC_PORT}/rpc/head
 ```

 ### Transaction Not Mining
 ```bash
 # Check mempool
-curl http://localhost:8006/rpc/mempool
+curl http://localhost:${RPC_PORT}/rpc/mempool

 # Verify nonce uniqueness
 # Ensure nonces are unique per sender
+
+# Check blockchain node status
+sudo systemctl status aitbc-blockchain-node.service
 ```

 ## References
@@ -291,12 +342,13 @@ curl http://localhost:8006/rpc/mempool
 - [Blockchain Operations](../../blockchain/)

 ### Source Code
- Agent Daemon: `/tmp/agent_daemon4.py`
- Ping Script: `/tmp/send_ping2.py`
+- Agent Daemon Service: `/etc/systemd/system/aitbc-agent-daemon.service`
+- Agent Daemon Wrapper: `/opt/aitbc/scripts/wrappers/aitbc-agent-daemon-wrapper.py`
+- Agent Daemon Script: `/opt/aitbc/apps/agent-coordinator/scripts/agent_daemon.py`
 - Training Script: `/opt/aitbc/scripts/training/hermes_cross_node_comm.sh`

 ---

-**Last Updated**: 2026-04-10
-**Version**: 1.0
-**Status**: Production Tested
+**Last Updated**: 2026-05-09
+**Version**: 2.0
+**Status**: Production Tested - Updated for systemd service management
--- a/docs/hermes/training/cross_node_communication_training.md
+++ b/docs/hermes/training/cross_node_communication_training.md
@@ -9,14 +9,25 @@ This training module teaches hermes agents how to establish, verify, and utilize
 ### System Requirements
 - AITBC blockchain nodes synchronized and communicating on port 8006
 - Both nodes operational (genesis node `aitbc1` and follower node `aitbc`)
+- systemd service management available
 - Funded wallets on both nodes for transaction fees
 - Python 3.13+ with cryptography library
 - SQLModel for database access

+### Environment Variables
+Set these environment variables before running training:
+```bash
+export GENESIS_IP="10.1.223.40"  # aitbc1 (genesis node)
+export FOLLOWER_IP="10.1.223.93"  # aitbc (follower node)
+export RPC_PORT="8006"
+export CHAIN_ID="ait-mainnet"
+```
+
 ### Wallet Configuration
- **Genesis Node (aitbc1)**: `temp-agent2` wallet with AIT for fees
+- **Genesis Node (aitbc1)**: `temp-agent` wallet with AIT for fees
 - **Follower Node (aitbc)**: `temp-agent` wallet for message sending
- Both wallets should be created with known passwords
+- **Agent Address**: `ait1d18e286fc0c12888aca94732b5507c8787af71a5`
+- **Password File**: `/var/lib/aitbc/keystore/.agent_daemon_password`

 ## Training Workflow

@@ -26,14 +37,14 @@ This training module teaches hermes agents how to establish, verify, and utilize

 **Commands**:
 ```bash
-# Genesis Node (aitbc1: 10.1.223.40)
-NODE_URL=http://10.1.223.40:8006 ./aitbc-cli agent create \
+# Genesis Node (aitbc1: $GENESIS_IP)
+NODE_URL=http://${GENESIS_IP}:${RPC_PORT} ./aitbc-cli agent create \
  --name "hermes-genesis-commander" \
  --description "Primary coordinator agent on genesis node" \
  --verification full

-# Follower Node (aitbc: 10.1.223.93)
-NODE_URL=http://localhost:8006 ./aitbc-cli agent create \
+# Follower Node (aitbc: $FOLLOWER_IP)
+NODE_URL=http://${FOLLOWER_IP}:${RPC_PORT} ./aitbc-cli agent create \
  --name "hermes-follower-worker" \
  --description "Worker agent on follower node" \
  --verification full
@@ -88,7 +99,7 @@ def create_tx(private_bytes, from_addr, to_addr, amount, fee, payload):
 priv = decrypt_wallet("/var/lib/aitbc/keystore/temp-agent.json", "temp123")
 tx = create_tx(priv, "ait1d18e286fc0c12888aca94732b5507c8787af71a5",
              "ait16af0b743fd6a2d3e2e2f28a820066706aa5813b5", 0, 10, "ping")
-response = requests.post("http://10.1.223.40:8006/rpc/transaction", json=tx)
+response = requests.post(f"http://${GENESIS_IP}:${RPC_PORT}/rpc/transaction", json=tx)
 print("Ping sent:", response.json())
 ```

@@ -96,44 +107,37 @@ print("Ping sent:", response.json())

 **Objective**: The follower agent must listen for and decode messages.

-**Agent Daemon Implementation**:
-```python
-# agent_daemon.py
-import time
-from sqlmodel import create_engine, Session, select
-from aitbc_chain.models import Transaction
+**Agent Daemon Service**:
+The agent daemon is managed as a systemd service for reliable operation:

-MY_ADDRESS = "ait16af0b743fd6a2d3e2e2f28a820066706aa5813b5"
-engine = create_engine("sqlite:////var/lib/aitbc/data/ait-mainnet/chain.db")
+```bash
+# Start the agent daemon service
+sudo systemctl start aitbc-agent-daemon.service

-processed_txs = set()
+# Check service status
+sudo systemctl status aitbc-agent-daemon.service

-while True:
-    with Session(engine) as session:
-        txs = session.exec(
-            select(Transaction).where(Transaction.recipient == MY_ADDRESS)
-        ).all()
-        
-        for tx in txs:
-            if tx.id in processed_txs: continue
-            processed_txs.add(tx.id)
-            
-            # Parse payload
-            data = ""
-            if hasattr(tx, "tx_metadata") and tx.tx_metadata:
-                if isinstance(tx.tx_metadata, dict):
-                    data = tx.tx_metadata.get("payload", "")
-            elif hasattr(tx, "payload") and tx.payload:
-                if isinstance(tx.payload, dict):
-                    data = tx.payload.get("payload", "")
-            
-            # Process message
-            if "ping" in str(data):
-                print(f"Received ping from {tx.sender}")
-                # Send pong reply
-    time.sleep(2)
+# View service logs
+sudo journalctl -u aitbc-agent-daemon -f
 ```

+**Service Configuration**:
+- **Service Name**: `aitbc-agent-daemon.service`
+- **Wallet**: `temp-agent`
+- **Agent Address**: `ait1d18e286fc0c12888aca94732b5507c8787af71a5`
+- **Password File**: `/var/lib/aitbc/keystore/.agent_daemon_password`
+- **RPC URL**: `http://localhost:8006`
+- **Poll Interval**: 2 seconds
+- **Trigger Message**: `ping`
+- **Reply Message**: `pong`
+- **Database Path**: `/var/lib/aitbc/data/${CHAIN_ID}/chain.db`
+
+**Agent Daemon Implementation**:
+The actual agent daemon script is located at:
+`/opt/aitbc/apps/agent-coordinator/scripts/agent_daemon.py`
+
+It polls the blockchain database for incoming transactions addressed to the agent wallet and automatically replies to trigger messages.
+
 ### Module 4: Distributed Task Execution

 **Objective**: Combine AI job submission with cross-node agent coordination.
@@ -218,35 +222,65 @@ Genesis Node: Received "pong" in Block 26952
 ### Workarounds
 - Custom Python scripts for transaction creation
 - Direct database queries for message retrieval
- Autonomous agent daemon for message handling
+- Systemd-managed agent daemon for message handling

 ## Troubleshooting

 ### Agent Daemon Not Starting
 ```bash
-# Check logs
-ssh aitbc1 'cat /tmp/agent_daemon4.log'
+# Check service status
+sudo systemctl status aitbc-agent-daemon.service

-# Verify wallet access
-ssh aitbc1 '/opt/aitbc/venv/bin/python -c "from scripts import decrypt_wallet"'
+# Check service logs
+sudo journalctl -u aitbc-agent-daemon -n 50
+
+# Start the service
+sudo systemctl start aitbc-agent-daemon.service
+
+# Restart the service
+sudo systemctl restart aitbc-agent-daemon.service
+```
+
+### Wallet Access Issues
+```bash
+# Verify wallet file exists
+ls -la /var/lib/aitbc/keystore/temp-agent.json
+
+# Verify password file exists
+ls -la /var/lib/aitbc/keystore/.agent_daemon_password
+
+# Test wallet decryption
+/opt/aitbc/venv/bin/python -c "
+from pathlib import Path
+import json
+keystore_path = Path('/var/lib/aitbc/keystore/temp-agent.json')
+with open(keystore_path) as f:
+    data = json.load(f)
+    print('Wallet loaded successfully')
+"
 ```

 ### Transactions Not Mining
 ```bash
 # Check mempool
-curl http://localhost:8006/rpc/mempool
+curl http://localhost:${RPC_PORT}/rpc/mempool

 # Verify nonce uniqueness
 # Ensure nonces are unique per sender
+
+# Check blockchain node status
+sudo systemctl status aitbc-blockchain-node.service
 ```

 ### Sync Issues
 ```bash
-# Manual sync
-python /tmp/sync_once.py
+# Check block heights on both nodes
+NODE_URL=http://${GENESIS_IP}:${RPC_PORT} ./aitbc-cli blockchain height
+NODE_URL=http://${FOLLOWER_IP}:${RPC_PORT} ./aitbc-cli blockchain height

-# Check block heights
-NODE_URL=http://localhost:8006 ./aitbc-cli blockchain height
+# Check sync status
+curl http://${GENESIS_IP}:${RPC_PORT}/rpc/head
+curl http://${FOLLOWER_IP}:${RPC_PORT}/rpc/head
 ```

 ## Related Documentation
@@ -271,6 +305,6 @@ Implement reliable message acknowledgment protocol for critical communications.

 ---

-**Last Updated**: 2026-04-10
-**Version**: 1.0
-**Status**: Production Tested
+**Last Updated**: 2026-05-09
+**Version**: 2.0
+**Status**: Production Tested - Updated for systemd service management
--- a/scripts/training/hermes_cross_node_comm.sh
+++ b/scripts/training/hermes_cross_node_comm.sh
@@ -2,7 +2,8 @@
 #
 # hermes Cross-Node Communication Training Module
 # Teaches and validates agent-to-agent communication across the AITBC blockchain
-# Nodes: Genesis (10.1.223.40:8006) and Follower (<aitbc1-ip>:8006)
+# Nodes: Genesis ($GENESIS_IP:$PORT) and Follower ($FOLLOWER_IP:$PORT)
+# Uses systemd-managed agent daemon service
 #

 set -e
@@ -11,9 +12,10 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"

 # Configuration
-GENESIS_IP="10.1.223.40"
-FOLLOWER_IP="<aitbc1-ip>"  # To be replaced during live training
-PORT=8006
+GENESIS_IP="${GENESIS_IP:-10.1.223.40}"
+FOLLOWER_IP="${FOLLOWER_IP:-10.1.223.93}"
+PORT="${RPC_PORT:-8006}"
+CHAIN_ID="${CHAIN_ID:-ait-mainnet}"
 CLI_PATH="${CLI_PATH:-${REPO_ROOT}/aitbc-cli}"

 # Colors for output
@@ -41,28 +43,43 @@ log_error() {
    echo -e "${RED}✗ $1${NC}"
 }

+log_warning() {
+    echo -e "${YELLOW}⚠ $1${NC}"
+}
+
 check_prerequisites() {
    log_step "Checking Prerequisites"
    
+    # Check environment variables
+    log_step "Environment Variables"
+    log_success "GENESIS_IP: ${GENESIS_IP}"
+    log_success "FOLLOWER_IP: ${FOLLOWER_IP}"
+    log_success "PORT: ${PORT}"
+    log_success "CHAIN_ID: ${CHAIN_ID}"
+    
    if ! curl -s -f "http://${GENESIS_IP}:${PORT}/health" > /dev/null; then
        log_error "Genesis node unreachable at ${GENESIS_IP}:${PORT}"
        exit 1
    fi
-    log_success "Genesis node active"
-    
-    # Try to auto-detect follower IP if placeholder is still present
-    if [[ "${FOLLOWER_IP}" == "<aitbc1-ip>" ]]; then
-        # Try to resolve aitbc1 hostname
-        FOLLOWER_IP=$(ping -c 1 aitbc1 | head -n 1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' || echo "localhost")
-        log_step "Auto-detected Follower IP: ${FOLLOWER_IP}"
-    fi
+    log_success "Genesis node reachable"
    
    if ! curl -s -f "http://${FOLLOWER_IP}:${PORT}/health" > /dev/null; then
-        log_warning "Follower node unreachable at ${FOLLOWER_IP}:${PORT}. Using localhost as fallback for training purposes."
-        FOLLOWER_IP="127.0.0.1"
-    else
-        log_success "Follower node active"
+        log_error "Follower node unreachable at ${FOLLOWER_IP}:${PORT}"
+        exit 1
    fi
+    log_success "Follower node reachable"
+    
+    # Check agent daemon service
+    if ! systemctl is-active --quiet aitbc-agent-daemon.service; then
+        log_warning "Agent daemon service not running, attempting to start..."
+        sudo systemctl start aitbc-agent-daemon.service
+        sleep 2
+        if ! systemctl is-active --quiet aitbc-agent-daemon.service; then
+            log_error "Failed to start agent daemon service"
+            exit 1
+        fi
+    fi
+    log_success "Agent daemon service running"
 }

 run_module1_registration() {
@@ -160,6 +177,12 @@ main() {
    echo -e "${CYAN}======================================================${NC}"
    echo -e "${CYAN}   hermes Cross-Node Communication Training Module  ${NC}"
    echo -e "${CYAN}======================================================${NC}"
+    echo -e "${CYAN}Configuration:${NC}"
+    echo -e "  Genesis Node: ${GENESIS_IP}:${PORT}"
+    echo -e "  Follower Node: ${FOLLOWER_IP}:${PORT}"
+    echo -e "  Chain ID: ${CHAIN_ID}"
+    echo -e "  Agent Daemon Service: aitbc-agent-daemon.service"
+    echo -e "${CYAN}======================================================${NC}"
    
    check_prerequisites
    run_module1_registration
@@ -174,9 +197,15 @@ main() {
    echo "✓ Transaction Broadcasting"
    echo "✓ Message Retrieval and Parsing"
    echo "✓ Cross-Node AI Job Coordination"
+    echo "✓ Systemd Service Management"
    
    echo -e "\n${GREEN}hermes agent has successfully completed Cross-Node Communication Training!${NC}"
-    echo "The agent is now certified to coordinate tasks across aitbc and aitbc1 nodes."
+    echo "The agent is now certified to coordinate tasks across ${GENESIS_IP} and ${FOLLOWER_IP} nodes."
+    echo ""
+    echo "Service Management Commands:"
+    echo "  Check status: sudo systemctl status aitbc-agent-daemon.service"
+    echo "  View logs: sudo journalctl -u aitbc-agent-daemon -f"
+    echo "  Restart service: sudo systemctl restart aitbc-agent-daemon.service"
 }

 main
--- a/test_hierarchical_config.py
+++ b/test_hierarchical_config.py
@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+"""Test hierarchical config implementation"""
+
+import sys
+import os
+import tempfile
+from pathlib import Path
+
+# Add the aitbc module to path
+sys.path.insert(0, '/opt/aitbc')
+
+def test_hierarchical_config():
+    print("Testing Hierarchical Configuration System...")
+    
+    try:
+        from aitbc.hierarchical_config import load_config, ValidatedAITBCConfig, create_config_template
+        print("✓ Imports successful")
+    except ImportError as e:
+        print(f"✗ Import failed: {e}")
+        return False
+    
+    # Test 1: Default loading
+    try:
+        config = load_config()
+        print(f"✓ Default config loaded: env={config.environment}, port={config.port}")
+    except Exception as e:
+        print(f"✗ Default config failed: {e}")
+        return False
+    
+    # Test 2: File-based config
+    try:
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f:
+            f.write('''
+environment: production
+port: 9000
+debug: false
+log_level: WARNING
+secret_key: "test-secret"
+jwt_secret: "test-jwt"
+''')
+            temp_path = f.name
+        
+        config = load_config(config_file=temp_path)
+        print(f"✓ File config loaded: env={config.environment}, port={config.port}")
+        os.unlink(temp_path)
+    except Exception as e:
+        print(f"✗ File config failed: {e}")
+        return False
+    
+    # Test 3: Validation
+    try:
+        # Set up for production validation
+        os.environ['AITBC_ENVIRONMENT'] = 'production'
+        os.environ['AITBC_SECRET_KEY'] = 'test-key'
+        os.environ['AITBC_JWT_SECRET'] = 'test-jwt'
+        config = ValidatedAITBCConfig()
+        print('✓ Production validation passed')
+        
+        # Should fail without secrets
+        del os.environ['AITBC_SECRET_KEY']
+        try:
+            config = ValidatedAITBCConfig()
+            print('✗ Validation should have failed')
+            return False
+        except Exception:
+            print('✓ Validation correctly rejected missing secret')
+            os.environ['AITBC_SECRET_KEY'] = 'test-key'  # restore
+            
+    except Exception as e:
+        print(f'✗ Validation test failed: {e}')
+        return False
+
+    # Test 4: Templates
+    try:
+        dev = create_config_template('development')
+        prod = create_config_template('production')
+        print(f'✓ Templates created: dev_debug={dev["debug"]}, prod_workers={prod["workers"]}')
+    except Exception as e:
+        print(f'✗ Template test failed: {e}')
+        return False
+        
+    print("✓ All hierarchical config tests passed!")
+    return True
+
+if __name__ == "__main__":
+    success = test_hierarchical_config()
+    sys.exit(0 if success else 1)