aitbc/docs/1_project/aitbc1.md

# AITBC1 Deployment Notes

## Overview

This document contains specific deployment notes and considerations for deploying the AITBC platform on the **aitbc1** server. These notes complement the general deployment guide with server-specific configurations and troubleshooting. **Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**

## Server Specifications

### **aitbc1 Server Details**
- **Hostname**: aitbc
- **IP Address**: 10.1.223.1 (container IP)
- **Operating System**: Debian 13 Trixie (primary development environment)
- **Access Method**: SSH via aitbc-cascade proxy
- **Web Root**: `/var/www/html/`
- **Nginx Configuration**: Two-tier setup with SSL termination
- **Container Support**: Incus containers with 0.0.0.0 binding for container access

### **Network Architecture**
```
Internet → aitbc-cascade (Proxy) → aitbc (Container)
         SSL Termination        Application Server
         Port 443/80            Port 8000-8003, 8010-8017
```

## Pre-Deployment Checklist

### **✅ Server Preparation**
- [ ] SSH access confirmed via aitbc-cascade
- [ ] System packages updated
- [ ] aitbc user created with sudo access
- [ ] Directory structure created
- [ ] Firewall rules configured
- [ ] Python 3.13+ installed and verified
- [ ] Container networking configured
- [ ] GPU access confirmed as not available
- [ ] Miner service requirements confirmed as not needed

### **✅ Network Configuration**
- [ ] Port forwarding configured on aitbc-cascade
- [ ] SSL certificates installed on proxy
- [ ] DNS records configured
- [ ] Load balancer rules set
- [ ] Container access configured (0.0.0.0 binding)

### **✅ Storage Requirements**
- [ ] Minimum 50GB free space available
- [ ] Backup storage allocated
- [ ] Log rotation configured
- [ ] Database storage planned

## Deployment Issues & Solutions

### **🔥 Issue 1: Python Version Compatibility**

**Problem**: aitbc1 may have Python 3.10 instead of required 3.13+

**Solution**:
```bash
# Check current Python version
python3 --version

# Install Python 3.13 if not available
sudo apt update
sudo apt install -y python3.13 python3.13-venv python3.13-dev

# Update alternatives
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.13 1
```

**Verification**:
```bash
python3 --version  # Should show 3.13.x
```

### **🔥 Issue 1b: Node.js Version Compatibility**

**Current Status**: Node.js v22.22.x (tested and compatible)

**Note**: Current Node.js version v22.22.x meets the minimum requirement of 22.0.0 and is fully compatible with AITBC platform.

### **🔥 Issue 1c: Operating System Compatibility**

**Current Status**: Debian 13 Trixie (development environment)

**Note**: Development environment is running Debian 13 Trixie, which is newer than the minimum requirement of Debian 11+ and fully supported for AITBC development. This is the primary development environment for the AITBC platform.

### **🔥 Issue 2: Port Conflicts with Existing Services**

**Problem**: Ports 8000-8008 may be in use by existing applications

**Solution**:
```bash
# Check port usage (new port logic)
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"

# Kill conflicting processes if needed
sudo fuser -k 8000/tcp  # Core services
sudo fuser -k 8010/tcp  # Enhanced services

# Alternative: Use different ports in service files
# Edit /etc/systemd/system/aitbc-*.service files
# Change --port 8000 to --port 9000, etc.
```

**Port Mapping for aitbc1 (Updated)**:
```
Core Services:
- Coordinator API: 8000 ✅
- Exchange API: 8001 ✅
- Blockchain RPC: 8003 ✅

Enhanced Services:
- Multimodal GPU: 8010 ✅ (CPU-only mode)
- GPU Multimodal: 8011 ✅ (CPU-only mode)
- Modality Optimization: 8012 ✅
- Adaptive Learning: 8013 ✅
- Marketplace Enhanced: 8014 ✅
- OpenClaw Enhanced: 8015 ✅
- Web UI: 8016 ✅
- Geographic Load Balancer: 8017 ✅
```

### **🔥 Issue 3: Database Permission Issues**

**Problem**: SQLite database file permissions preventing access

**Solution**:
```bash
# Fix database ownership (standardized)
sudo chown aitbc:aitbc /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Fix database permissions
sudo chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Ensure directory permissions
sudo chmod 755 /opt/aitbc/apps/coordinator-api/
```

### **🔥 Issue 4: Systemd Service Failures**

**Problem**: Services failing to start due to missing dependencies

**Solution**:
```bash
# Check service status
sudo systemctl status aitbc-coordinator-api.service

# Check service logs
sudo journalctl -u aitbc-coordinator-api.service -n 50

# Common fixes:
# 1. Install missing Python packages
cd /opt/aitbc/apps/coordinator-api
source .venv/bin/activate
pip install missing-package

# 2. Fix environment variables
echo "ENVIRONMENT=production" >> .env

# 3. Fix working directory
sudo systemctl edit aitbc-coordinator-api.service
# Add: WorkingDirectory=/opt/aitbc/apps/coordinator-api
```

### **🔥 Issue 5: Nginx Proxy Configuration**

**Problem**: Requests not properly forwarded from aitbc-cascade to aitbc

**Solution**:
```bash
# On aitbc-cascade, check proxy configuration
cat /etc/nginx/sites-available/aitbc-proxy.conf

# Ensure upstream configuration includes aitbc
upstream aitbc_backend {
    server 10.1.223.1:8000;  # Coordinator API
    server 10.1.223.1:8001;  # Exchange API
    server 10.1.223.1:8003;  # Blockchain RPC
    # Add enhanced services ports
    server 10.1.223.1:8010;  # Multimodal GPU
    server 10.1.223.1:8011;  # GPU Multimodal
    server 10.1.223.1:8012;  # Modality Optimization
    server 10.1.223.1:8013;  # Adaptive Learning
    server 10.1.223.1:8014;  # Marketplace Enhanced
    server 10.1.223.1:8015;  # OpenClaw Enhanced
    server 10.1.223.1:8016;  # Web UI
    server 10.1.223.1:8017;  # Geographic Load Balancer
}

# Reload nginx configuration
sudo nginx -t && sudo systemctl reload nginx
```

### **🔥 Issue 6: SSL Certificate Issues**

**Problem**: SSL certificates not properly configured for aitbc domain

**Solution**:
```bash
# On aitbc-cascade, check certificate status
sudo certbot certificates

# Renew or obtain certificate
sudo certbot --nginx -d aitbc.bubuit.net

# Test SSL configuration
curl -I https://aitbc.bubuit.net
```

## aitbc-Specific Configurations

### **Environment Variables**
```bash
# /opt/aitbc/apps/coordinator-api/.env
MINER_API_KEYS=["aitbc_production_key_32_characters_long"]
DATABASE_URL=sqlite:///./aitbc_coordinator.db
LOG_LEVEL=INFO
ENVIRONMENT=production
API_HOST=0.0.0.0
API_PORT=8000
WORKERS=2  # Reduced for aitbc resources
SERVER_NAME=aitbc.bubuit.net
# Note: No miner service needed - configuration kept for compatibility
```

### **Service Configuration Adjustments**
```bash
# aitbc-coordinator-api.service adjustments
# Edit: /etc/systemd/system/aitbc-coordinator-api.service

[Service]
User=aitbc
Group=aitbc
WorkingDirectory=/opt/aitbc/apps/coordinator-api
Environment=PYTHONPATH=src
EnvironmentFile=/opt/aitbc/apps/coordinator-api/.env
ExecStart=/opt/aitbc/apps/coordinator-api/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

### **Resource Limits for aitbc**
```bash
# /etc/systemd/system/aitbc-coordinator-api.service
[Service]
# Add resource limits
MemoryMax=2G
CPUQuota=200%
TasksMax=100
```

## Performance Optimization for aitbc

### **Database Optimization**
```bash
# SQLite optimization for aitbc
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 5000;  # Reduced for aitbc
PRAGMA temp_store = MEMORY;
PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 30000;
EOF
```

### **System Resource Limits**
```bash
# /etc/security/limits.conf additions for aitbc
aitbc soft nofile 4096
aitbc hard nofile 4096
aitbc soft nproc 512
aitbc hard nproc 512
```

### **Network Optimization**
```bash
# /etc/sysctl.conf additions for aitbc
net.core.somaxconn = 512
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.ip_local_port_range = 1024 65535
```

## Monitoring Setup for aitbc

### **Custom Monitoring Script**
```bash
# /opt/aitbc/scripts/monitor-aitbc.sh
#!/bin/bash
echo "AITBC Monitor - $(date)"
echo "========================"

# Service status
echo "Service Status:"
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"

# Resource usage
echo -e "\nResource Usage:"
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"

# Network connectivity
echo -e "\nNetwork Test:"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/v1/health" | grep -q "200" && echo "Coordinator API: ✅" || echo "Coordinator API: ❌"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8001/" | grep -q "200" && echo "Exchange API: ✅" || echo "Exchange API: ❌"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8003/rpc/head" | grep -q "200" && echo "Blockchain RPC: ✅" || echo "Blockchain RPC: ❌"

# Enhanced services health (CPU-only mode)
echo -e "\nEnhanced Services:"
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
    status=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:$port/health" 2>/dev/null)
    if [ "$status" = "200" ]; then
        service_name="Port $port"
        case $port in
            8010|8011) service_name="$service_name (CPU-only)" ;;
        esac
        echo "$service_name: ✅"
    else
        echo "Port $port: ❌ ($status)"
    fi
done

# Database status
echo -e "\nDatabase Status:"
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
    size=$(du -h /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db | cut -f1)
    echo "Database: ✅ ($size)"
else
    echo "Database: ❌ (Missing)"
fi

# Container access test
echo -e "\nContainer Access Test:"
curl -s -o /dev/null -w "%{http_code}" "http://10.1.223.1:8017/health" | grep -q "200" && echo "Container Access: ✅" || echo "Container Access: ❌"
EOF

chmod +x /opt/aitbc/scripts/monitor-aitbc.sh
```

## Backup Strategy for aitbc

### **Automated Backup Script**
```bash
# /opt/aitbc/scripts/backup-aitbc.sh
#!/bin/bash
BACKUP_DIR="/opt/aitbc/backups"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup database
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
    cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
    echo "Database backed up: aitbc_coordinator_$DATE.db"
fi

# Backup configuration
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/ 2>/dev/null
echo "Configuration backed up: config_$DATE.tar.gz"

# Backup service files
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
echo "Service files backed up: services_$DATE.tar.gz"

# Backup enhanced services scripts
tar -czf $BACKUP_DIR/enhanced-services_$DATE.tar.gz /opt/aitbc/scripts/*service*.py 2>/dev/null
echo "Enhanced services backed up: enhanced-services_$DATE.tar.gz"

# Clean old backups
find $BACKUP_DIR -name "*.db" -mtime +$RETENTION_DAYS -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete

echo "Backup completed: $DATE"
echo "Retention period: $RETENTION_DAYS days"
EOF

chmod +x /opt/aitbc/scripts/backup-aitbc.sh
```

## Troubleshooting aitbc Specific Issues

### **Issue: Services Not Starting After Reboot**
```bash
# Check if services are enabled
systemctl list-unit-files | grep aitbc

# Enable services for auto-start
sudo systemctl enable aitbc-coordinator-api.service
sudo systemctl enable aitbc-blockchain-node.service
sudo systemctl enable aitbc-blockchain-rpc.service
sudo systemctl enable aitbc-exchange-api.service

# Enable enhanced services
sudo systemctl enable aitbc-multimodal-gpu.service
sudo systemctl enable aitbc-multimodal.service
sudo systemctl enable aitbc-modality-optimization.service
sudo systemctl enable aitbc-adaptive-learning.service
sudo systemctl enable aitbc-marketplace-enhanced.service
sudo systemctl enable aitbc-openclaw-enhanced.service
sudo systemctl enable aitbc-web-ui.service
sudo systemctl enable aitbc-loadbalancer-geo.service
```

### **Issue: High Memory Usage**
```bash
# Check memory usage
free -h
ps aux --sort=-%mem | head -10

# Reduce worker count in service files
# Edit ExecStart line: --workers 1 instead of --workers 4
```

### **Issue: Database Locking**
```bash
# Check for database locks
sudo lsof /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Restart services to release locks
sudo systemctl restart aitbc-coordinator-api.service
```

### **Issue: Network Connectivity**
```bash
# Test local connectivity
curl -X GET "http://localhost:8000/v1/health"

# Test external connectivity via proxy
curl -X GET "http://aitbc.bubuit.net/health"

# Check proxy configuration
ssh aitbc-cascade "cat /etc/nginx/sites-available/aitbc-proxy.conf"
```

### **Issue: Container Access Problems**
```bash
# Test 0.0.0.0 binding
curl -s http://localhost:8017/health  # Should work
curl -s http://10.1.223.1:8017/health  # Should work from containers

# Check service binding
sudo netstat -tlnp | grep :8017  # Should show 0.0.0.0:8017

# Test from other containers
# From another container: curl http://aitbc:8017/health
```

## Security Considerations for aitbc

### **Firewall Configuration**
```bash
# Configure UFW on aitbc (if not using firehol)
sudo ufw allow ssh
sudo ufw allow 8000/tcp
sudo ufw allow 8001/tcp
sudo ufw allow 8003/tcp
sudo ufw allow 8010/tcp
sudo ufw allow 8011/tcp
sudo ufw allow 8012/tcp
sudo ufw allow 8013/tcp
sudo ufw allow 8014/tcp
sudo ufw allow 8015/tcp
sudo ufw allow 8016/tcp
sudo ufw allow 8017/tcp
sudo ufw --force enable
```

### **File Permissions**
```bash
# Secure sensitive files
chmod 600 /opt/aitbc/apps/coordinator-api/.env
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
chmod 755 /opt/aitbc/apps/coordinator-api/
```

### **Access Control**
```bash
# Restrict SSH access to specific users
echo "AllowUsers aitbc" | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart ssh
```

## Deployment Validation Checklist

### **✅ Pre-Deployment**
- [ ] Server access confirmed
- [ ] System requirements met
- [ ] Python 3.13+ installed and verified
- [ ] Dependencies installed
- [ ] Network configuration verified
- [ ] Container networking configured
- [ ] GPU access confirmed as not available
- [ ] Miner service requirements confirmed as not needed

### **✅ Deployment**
- [ ] Codebase copied to /opt/aitbc
- [ ] Virtual environments created (Python 3.13+)
- [ ] Dependencies installed
- [ ] Environment variables configured
- [ ] Service files installed (new port logic)
- [ ] Services enabled and started

### **✅ Post-Deployment**
- [ ] All 12 services running
- [ ] Core API endpoints responding (8000-8003)
- [ ] Enhanced services endpoints responding (8010-8017)
- [ ] Database operational
- [ ] Container access working (0.0.0.0 binding)
- [ ] Monitoring working
- [ ] Backup system active
- [ ] Security configured
- [ ] GPU services confirmed running in CPU-only mode
- [ ] Miner service confirmed as not needed

### **✅ Testing**
- [ ] Health endpoints responding for all services
- [ ] API functionality verified
- [ ] Database operations working
- [ ] External access via proxy working
- [ ] SSL certificates valid
- [ ] Performance acceptable
- [ ] Container connectivity verified
- [ ] Geographic load balancer accessible from containers
- [ ] GPU services confirmed operating in CPU-only mode
- [ ] No miner service requirements confirmed

## Rollback Procedures

### **Service Rollback**
```bash
# Stop all services
sudo systemctl stop aitbc-*.service

# Restore previous configuration
sudo cp /etc/systemd/system/aitbc-*.service.backup /etc/systemd/system/
sudo systemctl daemon-reload

# Restore database
cp /opt/aitbc/backups/aitbc_coordinator_PREV_DEPLOY.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Start services
sudo systemctl start aitbc-*.service
```

### **Full System Rollback**
```bash
# Restore from backup
cd /opt/aitbc
tar -xzf /opt/aitbc/backups/full_backup_YYYYMMDD.tar.gz

# Restart services
sudo systemctl restart aitbc-*.service
```

## Contact Information

### **Support Contacts**
- **Primary Admin**: aitbc-admin
- **Network Admin**: aitbc-network
- **Security Team**: aitbc-security

### **Emergency Procedures**
1. Check service status: `systemctl status aitbc-*`
2. Review logs: `journalctl -u aitbc-coordinator-api.service`
3. Run monitoring: `/opt/aitbc/scripts/monitor-aitbc.sh`
4. Check container access: `curl http://10.1.223.1:8017/health`
5. Verify GPU services running in CPU-only mode
6. Confirm no miner service is needed
7. Contact support if issues persist

---

**Server**: aitbc (Container)  
**Environment**: Production  
**GPU Access**: None (CPU-only mode)  
**Miner Service**: Not needed  
**Last Updated**: 2026-03-04  
**Maintainer**: AITBC Operations Team  
**Status**: ✅ PRODUCTION READY