refactor(ports): standardize service ports to 8000-8017 range and update CORS configurations across all services

- Update CORS allow_origins in blockchain-node app.py and gossip relay.py to use new port logic (8000-8016) - Update coordinator-api config.py and config_pg.py with standardized port ranges and service labels - Update coordinator-api health check script from port 18000 to 8000 - Update geo_load_balancer.py to use configurable host/port (default 0.0.0.0:8017) - Update agent_security.py sandbox
2026-03-04 15:43:17 +01:00
parent 3df0a9ed62
commit 5534226895
57 changed files with 9690 additions and 126 deletions
--- a/docs/1_project/aitbc1.md
+++ b/docs/1_project/aitbc1.md
@@ -0,0 +1,584 @@
+# AITBC1 Deployment Notes
+
+## Overview
+
+This document contains specific deployment notes and considerations for deploying the AITBC platform on the **aitbc1** server. These notes complement the general deployment guide with server-specific configurations and troubleshooting. **Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
+
+## Server Specifications
+
+### **aitbc1 Server Details**
+- **Hostname**: aitbc
+- **IP Address**: 10.1.223.1 (container IP)
+- **Operating System**: Debian 13 Trixie (primary development environment)
+- **Access Method**: SSH via aitbc-cascade proxy
+- **Web Root**: `/var/www/html/`
+- **Nginx Configuration**: Two-tier setup with SSL termination
+- **Container Support**: Incus containers with 0.0.0.0 binding for container access
+
+### **Network Architecture**
+```
+Internet → aitbc-cascade (Proxy) → aitbc (Container)
+         SSL Termination        Application Server
+         Port 443/80            Port 8000-8003, 8010-8017
+```
+
+## Pre-Deployment Checklist
+
+### **✅ Server Preparation**
+- [ ] SSH access confirmed via aitbc-cascade
+- [ ] System packages updated
+- [ ] aitbc user created with sudo access
+- [ ] Directory structure created
+- [ ] Firewall rules configured
+- [ ] Python 3.13+ installed and verified
+- [ ] Container networking configured
+- [ ] GPU access confirmed as not available
+- [ ] Miner service requirements confirmed as not needed
+
+### **✅ Network Configuration**
+- [ ] Port forwarding configured on aitbc-cascade
+- [ ] SSL certificates installed on proxy
+- [ ] DNS records configured
+- [ ] Load balancer rules set
+- [ ] Container access configured (0.0.0.0 binding)
+
+### **✅ Storage Requirements**
+- [ ] Minimum 50GB free space available
+- [ ] Backup storage allocated
+- [ ] Log rotation configured
+- [ ] Database storage planned
+
+## Deployment Issues & Solutions
+
+### **🔥 Issue 1: Python Version Compatibility**
+
+**Problem**: aitbc1 may have Python 3.10 instead of required 3.13+
+
+**Solution**:
+```bash
+# Check current Python version
+python3 --version
+
+# Install Python 3.13 if not available
+sudo apt update
+sudo apt install -y python3.13 python3.13-venv python3.13-dev
+
+# Update alternatives
+sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.13 1
+```
+
+**Verification**:
+```bash
+python3 --version  # Should show 3.13.x
+```
+
+### **🔥 Issue 1b: Node.js Version Compatibility**
+
+**Current Status**: Node.js v22.22.x (tested and compatible)
+
+**Note**: Current Node.js version v22.22.x meets the minimum requirement of 22.0.0 and is fully compatible with AITBC platform.
+
+### **🔥 Issue 1c: Operating System Compatibility**
+
+**Current Status**: Debian 13 Trixie (development environment)
+
+**Note**: Development environment is running Debian 13 Trixie, which is newer than the minimum requirement of Debian 11+ and fully supported for AITBC development. This is the primary development environment for the AITBC platform.
+
+### **🔥 Issue 2: Port Conflicts with Existing Services**
+
+**Problem**: Ports 8000-8008 may be in use by existing applications
+
+**Solution**:
+```bash
+# Check port usage (new port logic)
+sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"
+
+# Kill conflicting processes if needed
+sudo fuser -k 8000/tcp  # Core services
+sudo fuser -k 8010/tcp  # Enhanced services
+
+# Alternative: Use different ports in service files
+# Edit /etc/systemd/system/aitbc-*.service files
+# Change --port 8000 to --port 9000, etc.
+```
+
+**Port Mapping for aitbc1 (Updated)**:
+```
+Core Services:
+- Coordinator API: 8000 ✅
+- Exchange API: 8001 ✅
+- Blockchain RPC: 8003 ✅
+
+Enhanced Services:
+- Multimodal GPU: 8010 ✅ (CPU-only mode)
+- GPU Multimodal: 8011 ✅ (CPU-only mode)
+- Modality Optimization: 8012 ✅
+- Adaptive Learning: 8013 ✅
+- Marketplace Enhanced: 8014 ✅
+- OpenClaw Enhanced: 8015 ✅
+- Web UI: 8016 ✅
+- Geographic Load Balancer: 8017 ✅
+```
+
+### **🔥 Issue 3: Database Permission Issues**
+
+**Problem**: SQLite database file permissions preventing access
+
+**Solution**:
+```bash
+# Fix database ownership (standardized)
+sudo chown aitbc:aitbc /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
+
+# Fix database permissions
+sudo chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
+
+# Ensure directory permissions
+sudo chmod 755 /opt/aitbc/apps/coordinator-api/
+```
+
+### **🔥 Issue 4: Systemd Service Failures**
+
+**Problem**: Services failing to start due to missing dependencies
+
+**Solution**:
+```bash
+# Check service status
+sudo systemctl status aitbc-coordinator-api.service
+
+# Check service logs
+sudo journalctl -u aitbc-coordinator-api.service -n 50
+
+# Common fixes:
+# 1. Install missing Python packages
+cd /opt/aitbc/apps/coordinator-api
+source .venv/bin/activate
+pip install missing-package
+
+# 2. Fix environment variables
+echo "ENVIRONMENT=production" >> .env
+
+# 3. Fix working directory
+sudo systemctl edit aitbc-coordinator-api.service
+# Add: WorkingDirectory=/opt/aitbc/apps/coordinator-api
+```
+
+### **🔥 Issue 5: Nginx Proxy Configuration**
+
+**Problem**: Requests not properly forwarded from aitbc-cascade to aitbc
+
+**Solution**:
+```bash
+# On aitbc-cascade, check proxy configuration
+cat /etc/nginx/sites-available/aitbc-proxy.conf
+
+# Ensure upstream configuration includes aitbc
+upstream aitbc_backend {
+    server 10.1.223.1:8000;  # Coordinator API
+    server 10.1.223.1:8001;  # Exchange API
+    server 10.1.223.1:8003;  # Blockchain RPC
+    # Add enhanced services ports
+    server 10.1.223.1:8010;  # Multimodal GPU
+    server 10.1.223.1:8011;  # GPU Multimodal
+    server 10.1.223.1:8012;  # Modality Optimization
+    server 10.1.223.1:8013;  # Adaptive Learning
+    server 10.1.223.1:8014;  # Marketplace Enhanced
+    server 10.1.223.1:8015;  # OpenClaw Enhanced
+    server 10.1.223.1:8016;  # Web UI
+    server 10.1.223.1:8017;  # Geographic Load Balancer
+}
+
+# Reload nginx configuration
+sudo nginx -t && sudo systemctl reload nginx
+```
+
+### **🔥 Issue 6: SSL Certificate Issues**
+
+**Problem**: SSL certificates not properly configured for aitbc domain
+
+**Solution**:
+```bash
+# On aitbc-cascade, check certificate status
+sudo certbot certificates
+
+# Renew or obtain certificate
+sudo certbot --nginx -d aitbc.bubuit.net
+
+# Test SSL configuration
+curl -I https://aitbc.bubuit.net
+```
+
+## aitbc-Specific Configurations
+
+### **Environment Variables**
+```bash
+# /opt/aitbc/apps/coordinator-api/.env
+MINER_API_KEYS=["aitbc_production_key_32_characters_long"]
+DATABASE_URL=sqlite:///./aitbc_coordinator.db
+LOG_LEVEL=INFO
+ENVIRONMENT=production
+API_HOST=0.0.0.0
+API_PORT=8000
+WORKERS=2  # Reduced for aitbc resources
+SERVER_NAME=aitbc.bubuit.net
+# Note: No miner service needed - configuration kept for compatibility
+```
+
+### **Service Configuration Adjustments**
+```bash
+# aitbc-coordinator-api.service adjustments
+# Edit: /etc/systemd/system/aitbc-coordinator-api.service
+
+[Service]
+User=aitbc
+Group=aitbc
+WorkingDirectory=/opt/aitbc/apps/coordinator-api
+Environment=PYTHONPATH=src
+EnvironmentFile=/opt/aitbc/apps/coordinator-api/.env
+ExecStart=/opt/aitbc/apps/coordinator-api/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### **Resource Limits for aitbc**
+```bash
+# /etc/systemd/system/aitbc-coordinator-api.service
+[Service]
+# Add resource limits
+MemoryMax=2G
+CPUQuota=200%
+TasksMax=100
+```
+
+## Performance Optimization for aitbc
+
+### **Database Optimization**
+```bash
+# SQLite optimization for aitbc
+sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
+PRAGMA synchronous = NORMAL;
+PRAGMA cache_size = 5000;  # Reduced for aitbc
+PRAGMA temp_store = MEMORY;
+PRAGMA journal_mode = WAL;
+PRAGMA busy_timeout = 30000;
+EOF
+```
+
+### **System Resource Limits**
+```bash
+# /etc/security/limits.conf additions for aitbc
+aitbc soft nofile 4096
+aitbc hard nofile 4096
+aitbc soft nproc 512
+aitbc hard nproc 512
+```
+
+### **Network Optimization**
+```bash
+# /etc/sysctl.conf additions for aitbc
+net.core.somaxconn = 512
+net.ipv4.tcp_max_syn_backlog = 512
+net.ipv4.ip_local_port_range = 1024 65535
+```
+
+## Monitoring Setup for aitbc
+
+### **Custom Monitoring Script**
+```bash
+# /opt/aitbc/scripts/monitor-aitbc.sh
+#!/bin/bash
+echo "AITBC Monitor - $(date)"
+echo "========================"
+
+# Service status
+echo "Service Status:"
+systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"
+
+# Resource usage
+echo -e "\nResource Usage:"
+echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
+echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
+echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"
+
+# Network connectivity
+echo -e "\nNetwork Test:"
+curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/v1/health" | grep -q "200" && echo "Coordinator API: ✅" || echo "Coordinator API: ❌"
+curl -s -o /dev/null -w "%{http_code}" "http://localhost:8001/" | grep -q "200" && echo "Exchange API: ✅" || echo "Exchange API: ❌"
+curl -s -o /dev/null -w "%{http_code}" "http://localhost:8003/rpc/head" | grep -q "200" && echo "Blockchain RPC: ✅" || echo "Blockchain RPC: ❌"
+
+# Enhanced services health (CPU-only mode)
+echo -e "\nEnhanced Services:"
+for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
+    status=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:$port/health" 2>/dev/null)
+    if [ "$status" = "200" ]; then
+        service_name="Port $port"
+        case $port in
+            8010|8011) service_name="$service_name (CPU-only)" ;;
+        esac
+        echo "$service_name: ✅"
+    else
+        echo "Port $port: ❌ ($status)"
+    fi
+done
+
+# Database status
+echo -e "\nDatabase Status:"
+if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
+    size=$(du -h /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db | cut -f1)
+    echo "Database: ✅ ($size)"
+else
+    echo "Database: ❌ (Missing)"
+fi
+
+# Container access test
+echo -e "\nContainer Access Test:"
+curl -s -o /dev/null -w "%{http_code}" "http://10.1.223.1:8017/health" | grep -q "200" && echo "Container Access: ✅" || echo "Container Access: ❌"
+EOF
+
+chmod +x /opt/aitbc/scripts/monitor-aitbc.sh
+```
+
+## Backup Strategy for aitbc
+
+### **Automated Backup Script**
+```bash
+# /opt/aitbc/scripts/backup-aitbc.sh
+#!/bin/bash
+BACKUP_DIR="/opt/aitbc/backups"
+DATE=$(date +%Y%m%d_%H%M%S)
+RETENTION_DAYS=7
+
+# Create backup directory
+mkdir -p $BACKUP_DIR
+
+# Backup database
+if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
+    cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
+    echo "Database backed up: aitbc_coordinator_$DATE.db"
+fi
+
+# Backup configuration
+tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/ 2>/dev/null
+echo "Configuration backed up: config_$DATE.tar.gz"
+
+# Backup service files
+tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
+echo "Service files backed up: services_$DATE.tar.gz"
+
+# Backup enhanced services scripts
+tar -czf $BACKUP_DIR/enhanced-services_$DATE.tar.gz /opt/aitbc/scripts/*service*.py 2>/dev/null
+echo "Enhanced services backed up: enhanced-services_$DATE.tar.gz"
+
+# Clean old backups
+find $BACKUP_DIR -name "*.db" -mtime +$RETENTION_DAYS -delete
+find $BACKUP_DIR -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete
+
+echo "Backup completed: $DATE"
+echo "Retention period: $RETENTION_DAYS days"
+EOF
+
+chmod +x /opt/aitbc/scripts/backup-aitbc.sh
+```
+
+## Troubleshooting aitbc Specific Issues
+
+### **Issue: Services Not Starting After Reboot**
+```bash
+# Check if services are enabled
+systemctl list-unit-files | grep aitbc
+
+# Enable services for auto-start
+sudo systemctl enable aitbc-coordinator-api.service
+sudo systemctl enable aitbc-blockchain-node.service
+sudo systemctl enable aitbc-blockchain-rpc.service
+sudo systemctl enable aitbc-exchange-api.service
+
+# Enable enhanced services
+sudo systemctl enable aitbc-multimodal-gpu.service
+sudo systemctl enable aitbc-multimodal.service
+sudo systemctl enable aitbc-modality-optimization.service
+sudo systemctl enable aitbc-adaptive-learning.service
+sudo systemctl enable aitbc-marketplace-enhanced.service
+sudo systemctl enable aitbc-openclaw-enhanced.service
+sudo systemctl enable aitbc-web-ui.service
+sudo systemctl enable aitbc-loadbalancer-geo.service
+```
+
+### **Issue: High Memory Usage**
+```bash
+# Check memory usage
+free -h
+ps aux --sort=-%mem | head -10
+
+# Reduce worker count in service files
+# Edit ExecStart line: --workers 1 instead of --workers 4
+```
+
+### **Issue: Database Locking**
+```bash
+# Check for database locks
+sudo lsof /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
+
+# Restart services to release locks
+sudo systemctl restart aitbc-coordinator-api.service
+```
+
+### **Issue: Network Connectivity**
+```bash
+# Test local connectivity
+curl -X GET "http://localhost:8000/v1/health"
+
+# Test external connectivity via proxy
+curl -X GET "http://aitbc.bubuit.net/health"
+
+# Check proxy configuration
+ssh aitbc-cascade "cat /etc/nginx/sites-available/aitbc-proxy.conf"
+```
+
+### **Issue: Container Access Problems**
+```bash
+# Test 0.0.0.0 binding
+curl -s http://localhost:8017/health  # Should work
+curl -s http://10.1.223.1:8017/health  # Should work from containers
+
+# Check service binding
+sudo netstat -tlnp | grep :8017  # Should show 0.0.0.0:8017
+
+# Test from other containers
+# From another container: curl http://aitbc:8017/health
+```
+
+## Security Considerations for aitbc
+
+### **Firewall Configuration**
+```bash
+# Configure UFW on aitbc (if not using firehol)
+sudo ufw allow ssh
+sudo ufw allow 8000/tcp
+sudo ufw allow 8001/tcp
+sudo ufw allow 8003/tcp
+sudo ufw allow 8010/tcp
+sudo ufw allow 8011/tcp
+sudo ufw allow 8012/tcp
+sudo ufw allow 8013/tcp
+sudo ufw allow 8014/tcp
+sudo ufw allow 8015/tcp
+sudo ufw allow 8016/tcp
+sudo ufw allow 8017/tcp
+sudo ufw --force enable
+```
+
+### **File Permissions**
+```bash
+# Secure sensitive files
+chmod 600 /opt/aitbc/apps/coordinator-api/.env
+chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
+chmod 755 /opt/aitbc/apps/coordinator-api/
+```
+
+### **Access Control**
+```bash
+# Restrict SSH access to specific users
+echo "AllowUsers aitbc" | sudo tee -a /etc/ssh/sshd_config
+sudo systemctl restart ssh
+```
+
+## Deployment Validation Checklist
+
+### **✅ Pre-Deployment**
+- [ ] Server access confirmed
+- [ ] System requirements met
+- [ ] Python 3.13+ installed and verified
+- [ ] Dependencies installed
+- [ ] Network configuration verified
+- [ ] Container networking configured
+- [ ] GPU access confirmed as not available
+- [ ] Miner service requirements confirmed as not needed
+
+### **✅ Deployment**
+- [ ] Codebase copied to /opt/aitbc
+- [ ] Virtual environments created (Python 3.13+)
+- [ ] Dependencies installed
+- [ ] Environment variables configured
+- [ ] Service files installed (new port logic)
+- [ ] Services enabled and started
+
+### **✅ Post-Deployment**
+- [ ] All 12 services running
+- [ ] Core API endpoints responding (8000-8003)
+- [ ] Enhanced services endpoints responding (8010-8017)
+- [ ] Database operational
+- [ ] Container access working (0.0.0.0 binding)
+- [ ] Monitoring working
+- [ ] Backup system active
+- [ ] Security configured
+- [ ] GPU services confirmed running in CPU-only mode
+- [ ] Miner service confirmed as not needed
+
+### **✅ Testing**
+- [ ] Health endpoints responding for all services
+- [ ] API functionality verified
+- [ ] Database operations working
+- [ ] External access via proxy working
+- [ ] SSL certificates valid
+- [ ] Performance acceptable
+- [ ] Container connectivity verified
+- [ ] Geographic load balancer accessible from containers
+- [ ] GPU services confirmed operating in CPU-only mode
+- [ ] No miner service requirements confirmed
+
+## Rollback Procedures
+
+### **Service Rollback**
+```bash
+# Stop all services
+sudo systemctl stop aitbc-*.service
+
+# Restore previous configuration
+sudo cp /etc/systemd/system/aitbc-*.service.backup /etc/systemd/system/
+sudo systemctl daemon-reload
+
+# Restore database
+cp /opt/aitbc/backups/aitbc_coordinator_PREV_DEPLOY.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
+
+# Start services
+sudo systemctl start aitbc-*.service
+```
+
+### **Full System Rollback**
+```bash
+# Restore from backup
+cd /opt/aitbc
+tar -xzf /opt/aitbc/backups/full_backup_YYYYMMDD.tar.gz
+
+# Restart services
+sudo systemctl restart aitbc-*.service
+```
+
+## Contact Information
+
+### **Support Contacts**
+- **Primary Admin**: aitbc-admin
+- **Network Admin**: aitbc-network
+- **Security Team**: aitbc-security
+
+### **Emergency Procedures**
+1. Check service status: `systemctl status aitbc-*`
+2. Review logs: `journalctl -u aitbc-coordinator-api.service`
+3. Run monitoring: `/opt/aitbc/scripts/monitor-aitbc.sh`
+4. Check container access: `curl http://10.1.223.1:8017/health`
+5. Verify GPU services running in CPU-only mode
+6. Confirm no miner service is needed
+7. Contact support if issues persist
+
+---
+
+**Server**: aitbc (Container)  
+**Environment**: Production  
+**GPU Access**: None (CPU-only mode)  
+**Miner Service**: Not needed  
+**Last Updated**: 2026-03-04  
+**Maintainer**: AITBC Operations Team  
+**Status**: ✅ PRODUCTION READY