aitbc/docs/1_project/aitbc.md

AITBC Platform Deployment Guide

Overview

This guide provides comprehensive deployment instructions for the AITBC (AI Trading Blockchain Compute) platform, including infrastructure requirements, service configurations, and troubleshooting procedures. Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.

System Requirements

Hardware Requirements

• CPU: 4+ cores recommended
• Memory: 8GB+ RAM minimum, 16GB+ recommended
• Storage: 50GB+ available space
• Network: Stable internet connection with 100Mbps+ bandwidth
• GPU: Not required (aitbc server has no GPU access)
• Mining: No miner service needed

Software Requirements

• Operating System: Debian 13 Trixie (primary) or Ubuntu 22.04+ (alternative)
• Python: 3.13.5+ (strictly enforced - platform requires 3.13+ features)
• Node.js: 22+ (current tested: v22.22.x)
• Database: SQLite (default) or PostgreSQL (production)

Network Requirements

• Core Services Ports: 8000-8003 (must be available)
  • Port 8000: Coordinator API
  • Port 8001: Exchange API
  • Port 8002: Blockchain Node (internal)
  • Port 8003: Blockchain RPC
• Enhanced Services Ports: 8010-8017 (must be available)
  • Port 8010: Multimodal GPU Service
  • Port 8011: GPU Multimodal Service
  • Port 8012: Modality Optimization Service
  • Port 8013: Adaptive Learning Service
  • Port 8014: Marketplace Enhanced Service
  • Port 8015: OpenClaw Enhanced Service
  • Port 8016: Web UI Service
  • Port 8017: Geographic Load Balancer
• Firewall: Managed by firehol on at1 host (container networking handled by incus)
• SSL/TLS: Recommended for production deployments

Architecture Overview

AITBC Platform Architecture (Updated March 4, 2026)
├── Core Services (8000-8003)
│   ├── Coordinator API (Port 8000) ✅ PRODUCTION READY
│   ├── Exchange API (Port 8001) ✅ PRODUCTION READY
│   ├── Blockchain Node (Port 8002) ✅ PRODUCTION READY
│   └── Blockchain RPC (Port 8003) ✅ PRODUCTION READY
├── Enhanced Services (8010-8017)
│   ├── Multimodal GPU (Port 8010) ✅ PRODUCTION READY (CPU-only mode)
│   ├── GPU Multimodal (Port 8011) ✅ PRODUCTION READY (CPU-only mode)
│   ├── Modality Optimization (Port 8012) ✅ PRODUCTION READY
│   ├── Adaptive Learning (Port 8013) ✅ PRODUCTION READY
│   ├── Marketplace Enhanced (Port 8014) ✅ PRODUCTION READY
│   ├── OpenClaw Enhanced (Port 8015) ✅ PRODUCTION READY
│   ├── Web UI (Port 8016) ✅ PRODUCTION READY
│   └── Geographic Load Balancer (Port 8017) ✅ PRODUCTION READY
└── Infrastructure
    ├── Database (SQLite/PostgreSQL)
    ├── Monitoring & Logging
    ├── Security & Authentication
    └── Container Support (0.0.0.0 binding)

Deployment Steps

Phase 1: Environment Setup

1.1 System Preparation

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install required packages
sudo apt install -y python3.13 python3.13-venv python3-pip nodejs npm nginx sqlite3

# Create aitbc user
sudo useradd -m -s /bin/bash aitbc
sudo usermod -aG sudo aitbc

1.2 Directory Structure

# Create AITBC directory structure (standardized)
sudo mkdir -p /opt/aitbc/{apps,config,logs,scripts,backups}
sudo chown -R aitbc:aitbc /opt/aitbc

1.3 Code Deployment

# Clone or copy AITBC codebase
cd /opt/aitbc
# Option 1: Git clone
git clone https://github.com/oib/AITBC.git .
# Option 2: Copy from existing installation
# scp -r /path/to/aitbc/* aitbc@target:/opt/aitbc/

# Set permissions (standardized)
sudo chown -R aitbc:aitbc /opt/aitbc
sudo chmod -R 755 /opt/aitbc

Phase 2: Service Configuration

2.1 Python Environment Setup

# Coordinator API Environment (Python 3.13+ required)
cd /opt/aitbc/apps/coordinator-api
python3.13 -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn sqlalchemy aiosqlite sqlmodel pydantic pydantic-settings httpx aiofiles python-jose passlib bcrypt prometheus-client slowapi websockets numpy

# Enhanced Services Environment (CPU-only mode)
cd /opt/aitbc/apps/coordinator-api  # Enhanced services use same environment
source .venv/bin/activate
pip install aiohttp asyncio  # Additional dependencies for enhanced services
# Note: GPU-related packages (CUDA, torch) not installed - no GPU access

2.2 Environment Configuration

# Coordinator API Environment (Production)
cd /opt/aitbc/apps/coordinator-api
cat > .env << 'EOF'
MINER_API_KEYS=["production_key_32_characters_long_minimum"]
DATABASE_URL=sqlite:///./aitbc_coordinator.db
LOG_LEVEL=INFO
ENVIRONMENT=production
API_HOST=0.0.0.0
API_PORT=8000
WORKERS=4
# Note: No miner service needed - configuration kept for compatibility
EOF

# Set permissions
chmod 600 .env
chown aitbc:aitbc .env

2.3 Systemd Service Installation

# Copy service files (updated for new port logic)
sudo cp -r /opt/aitbc/systemd/* /etc/systemd/system/
sudo systemctl daemon-reload

# Enable core services
sudo systemctl enable aitbc-coordinator-api.service
sudo systemctl enable aitbc-blockchain-node.service
sudo systemctl enable aitbc-blockchain-rpc.service
sudo systemctl enable aitbc-exchange-api.service
sudo systemctl enable aitbc-explorer.service

# Enable enhanced services
sudo systemctl enable aitbc-multimodal-gpu.service
sudo systemctl enable aitbc-multimodal.service
sudo systemctl enable aitbc-modality-optimization.service
sudo systemctl enable aitbc-adaptive-learning.service
sudo systemctl enable aitbc-marketplace-enhanced.service
sudo systemctl enable aitbc-openclaw-enhanced.service
sudo systemctl enable aitbc-web-ui.service
sudo systemctl enable aitbc-loadbalancer-geo.service

Phase 3: Service Deployment

3.1 Core Services Startup

# Start core services in order
sudo systemctl start aitbc-blockchain-node.service
sleep 5
sudo systemctl start aitbc-blockchain-rpc.service
sleep 3
sudo systemctl start aitbc-coordinator-api.service
sleep 3
sudo systemctl start aitbc-exchange-api.service
sleep 3
sudo systemctl start aitbc-explorer.service

3.2 Enhanced Services Startup

# Start enhanced services (new port logic)
sudo systemctl start aitbc-multimodal-gpu.service      # Port 8010
sudo systemctl start aitbc-multimodal.service          # Port 8011
sudo systemctl start aitbc-modality-optimization.service # Port 8012
sudo systemctl start aitbc-adaptive-learning.service     # Port 8013
sudo systemctl start aitbc-marketplace-enhanced.service # Port 8014
sudo systemctl start aitbc-openclaw-enhanced.service     # Port 8015
sudo systemctl start aitbc-web-ui.service               # Port 8016
sudo systemctl start aitbc-loadbalancer-geo.service      # Port 8017

3.3 Service Verification

# Check service status
sudo systemctl list-units --type=service --state=running | grep aitbc

# Test core endpoints
curl -X GET "http://localhost:8000/v1/health"    # Coordinator API
curl -X GET "http://localhost:8001/"               # Exchange API
curl -X GET "http://localhost:8003/rpc/head"      # Blockchain RPC

# Test enhanced endpoints (CPU-only mode)
curl -X GET "http://localhost:8010/health"        # Multimodal GPU (CPU mode)
curl -X GET "http://localhost:8011/health"        # GPU Multimodal (CPU mode)
curl -X GET "http://localhost:8012/health"        # Modality Optimization
curl -X GET "http://localhost:8013/health"        # Adaptive Learning
curl -X GET "http://localhost:8014/health"        # Marketplace Enhanced
curl -X GET "http://localhost:8015/health"        # OpenClaw Enhanced
curl -X GET "http://localhost:8016/health"        # Web UI
curl -X GET "http://localhost:8017/health"        # Geographic Load Balancer

Phase 4: Production Configuration

4.1 Security Configuration

# Note: AITBC servers run in incus containers on at1 host
# Firewall is managed by firehol on at1, not ufw in containers
# Container networking is handled by incus with appropriate port forwarding

# Secure sensitive files
chmod 600 /opt/aitbc/apps/coordinator-api/.env
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

4.2 Performance Optimization

# Database optimization
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 10000;
PRAGMA temp_store = MEMORY;
EOF

# System limits
echo "aitbc soft nofile 65536" | sudo tee -a /etc/security/limits.conf
echo "aitbc hard nofile 65536" | sudo tee -a /etc/security/limits.conf

# Network optimization
echo "net.core.somaxconn = 1024" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.tcp_max_syn_backlog = 1024" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

4.3 Monitoring Setup

# Create comprehensive monitoring script (updated for new port logic)
cat > /opt/aitbc/scripts/monitor-services.sh << 'EOF'
#!/bin/bash
echo "AITBC Service Monitor - $(date)"
echo "================================"

# Service status
echo "Service Status:"
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"

# Core endpoint health
echo -e "\nCore Services Health:"
for port in 8000 8001 8003; do
    status=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$port/health" 2>/dev/null)
    if [ "$status" = "200" ]; then
        echo "Port $port: ✅ Healthy"
    else
        echo "Port $port: ❌ Unhealthy ($status)"
    fi
done

# Enhanced endpoint health
echo -e "\nEnhanced Services Health:"
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
    status=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$port/health" 2>/dev/null)
    if [ "$status" = "200" ]; then
        echo "Port $port: ✅ Healthy"
    else
        echo "Port $port: ❌ Unhealthy ($status)"
    fi
done

# System resources
echo -e "\nSystem Resources:"
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
echo "Disk: $(df -h / | tail -1 | awk '{print $3"/"$2}')"

# Port usage verification
echo -e "\nPort Usage:"
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)" | sort
EOF

chmod +x /opt/aitbc/scripts/monitor-services.sh
chown aitbc:aitbc /opt/aitbc/scripts/monitor-services.sh

Troubleshooting

Common Issues

Service Not Starting

# Check service logs
sudo journalctl -u aitbc-coordinator-api.service -n 50

# Check Python environment (must be 3.13+)
cd /opt/aitbc/apps/coordinator-api
source .venv/bin/activate
python --version  # Should show 3.13.x

# Check permissions
ls -la /opt/aitbc/apps/coordinator-api/

Database Issues

# Check database file
ls -la /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Test database connection
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db ".tables"

# Recreate database if corrupted
mv /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db.backup

Port Conflicts (New Port Logic)

# Check port usage (new port logic)
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"

# Kill conflicting processes
sudo fuser -k 8000/tcp  # Core services
sudo fuser -k 8010/tcp  # Enhanced services

# Restart services
sudo systemctl restart aitbc-coordinator-api.service

Container Access Issues

# Test 0.0.0.0 binding (for container access)
curl -s http://localhost:8017/health  # Should work
curl -s http://10.1.223.1:8017/health  # Should work from containers

# Check service binding
sudo netstat -tlnp | grep :8017  # Should show 0.0.0.0:8017

Permission Issues

# Fix file ownership (standardized)
sudo chown -R aitbc:aitbc /opt/aitbc

# Fix file permissions
sudo chmod -R 755 /opt/aitbc
chmod 600 /opt/aitbc/apps/coordinator-api/.env

Performance Issues

High Memory Usage

# Check memory usage
free -h
ps aux --sort=-%mem | head -10

# Optimize Python processes
# Reduce worker count in service files
# Implement database connection pooling

High CPU Usage

# Check CPU usage
top
ps aux --sort=-%cpu | head -10

# Optimize database queries
# Add database indexes
# Implement caching

Maintenance

Daily Tasks

# Service health check (updated for new port logic)
/opt/aitbc/scripts/monitor-services.sh

# Log rotation
sudo logrotate -f /etc/logrotate.d/aitbc

# Backup database
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db /opt/aitbc/backups/aitbc_coordinator_$(date +%Y%m%d).db

Weekly Tasks

# System updates
sudo apt update && sudo apt upgrade -y

# Service restart
sudo systemctl restart aitbc-*.service

# Performance review
/opt/aitbc/scripts/monitor-services.sh > /opt/aitbc/logs/weekly_$(date +%Y%m%d).log

Monthly Tasks

# Security updates
sudo apt update && sudo apt upgrade -y

# Database maintenance
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db "VACUUM;"

# Log cleanup
find /opt/aitbc/logs -name "*.log" -mtime +30 -delete

Scaling Considerations

Horizontal Scaling

• Load balancer configuration (Port 8017)
• Multiple service instances
• Database clustering
• CDN implementation

Vertical Scaling

• Resource allocation increases
• Performance optimization
• Caching strategies
• Database tuning

Security Best Practices

Network Security

• Firewall configuration
• SSL/TLS implementation
• VPN access for management
• Network segmentation

Application Security

• Environment variable protection
• API rate limiting
• Input validation
• Regular security audits

Data Security

• Database encryption
• Backup encryption
• Access control
• Audit logging

Backup and Recovery

Automated Backup Script

cat > /opt/aitbc/scripts/backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/opt/aitbc/backups"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup database
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db

# Backup configuration
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/

# Backup scripts
tar -czf $BACKUP_DIR/scripts_$DATE.tar.gz /opt/aitbc/scripts/

# Backup service configurations
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service

# Clean old backups (keep 7 days)
find $BACKUP_DIR -name "*.db" -mtime +7 -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $DATE"
EOF

chmod +x /opt/aitbc/scripts/backup.sh
chown aitbc:aitbc /opt/aitbc/scripts/backup.sh

Recovery Procedures

# Stop services
sudo systemctl stop aitbc-*.service

# Restore database
cp /opt/aitbc/backups/aitbc_coordinator_YYYYMMDD.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Restore configuration
tar -xzf /opt/aitbc/backups/config_YYYYMMDD.tar.gz -C /

# Restore service configurations
tar -xzf /opt/aitbc/backups/services_YYYYMMDD.tar.gz -C /
sudo systemctl daemon-reload

# Start services
sudo systemctl start aitbc-*.service

Monitoring and Alerting

Key Metrics

• Service uptime (all 12 services)
• API response times
• Database performance
• System resource usage
• Error rates

Alerting Thresholds

• Service downtime > 5 minutes
• API response time > 1 second
• CPU usage > 80%
• Memory usage > 90%
• Disk usage > 85%

Production Deployment Checklist

✅ Pre-Deployment

• Python 3.13+ installed and verified
• All required ports available (8000-8003, 8010-8017)
• System requirements met
• Dependencies installed
• Network configuration verified

✅ Deployment

• Codebase copied to /opt/aitbc
• Virtual environments created (Python 3.13+)
• Dependencies installed
• Environment variables configured
• Service files installed (new port logic)
• Services enabled and started

✅ Post-Deployment

• All 12 services running
• Core API endpoints responding (8000-8003)
• Enhanced services endpoints responding (8010-8017)
• Database operational
• Container access working (0.0.0.0 binding)
• Monitoring working
• Backup system active
• Security configured

✅ Testing

• Health endpoints responding for all services
• API functionality verified
• Database operations working
• External access via proxy working
• SSL certificates valid
• Performance acceptable
• Container connectivity verified

Documentation References

• Service Configuration Guide
• Security Hardening Guide
• Performance Optimization Guide
• Troubleshooting Guide
• Enhanced Services Guide
• Port Logic Implementation

---

Version: 2.0 (Updated for new port logic)
Last Updated: 2026-03-04
Maintainer: AITBC Development Team
Status: ✅ PRODUCTION READY
GPU Access: None (CPU-only mode)
Miner Service: Not needed