aitbc/docs/1_project/aitbc1.md

AITBC1 Deployment Notes

Overview

This document contains specific deployment notes and considerations for deploying the AITBC platform on the aitbc1 server. These notes complement the general deployment guide with server-specific configurations and troubleshooting. Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.

Server Specifications

aitbc1 Server Details

• Hostname: aitbc
• IP Address: 10.1.223.1 (container IP)
• Operating System: Debian 13 Trixie (primary development environment)
• Access Method: SSH via aitbc-cascade proxy
• Web Root: /var/www/html/
• Nginx Configuration: Two-tier setup with SSL termination
• Container Support: Incus containers with 0.0.0.0 binding for container access

Network Architecture

Internet → aitbc-cascade (Proxy) → aitbc (Container)
         SSL Termination        Application Server
         Port 443/80            Port 8000-8003, 8010-8017

Pre-Deployment Checklist

✅ Server Preparation

• SSH access confirmed via aitbc-cascade
• System packages updated
• aitbc user created with sudo access
• Directory structure created
• Firewall rules configured
• Python 3.13+ installed and verified
• Container networking configured
• GPU access confirmed as not available
• Miner service requirements confirmed as not needed

✅ Network Configuration

• Port forwarding configured on aitbc-cascade
• SSL certificates installed on proxy
• DNS records configured
• Load balancer rules set
• Container access configured (0.0.0.0 binding)

✅ Storage Requirements

• Minimum 50GB free space available
• Backup storage allocated
• Log rotation configured
• Database storage planned

Deployment Issues & Solutions

🔥 Issue 1: Python Version Compatibility

Problem: aitbc1 may have Python 3.10 instead of required 3.13+

Solution:

# Check current Python version
python3 --version

# Install Python 3.13 if not available
sudo apt update
sudo apt install -y python3.13 python3.13-venv python3.13-dev

# Update alternatives
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.13 1

Verification:

python3 --version  # Should show 3.13.x

🔥 Issue 1b: Node.js Version Compatibility

Current Status: Node.js v22.22.x (tested and compatible)

Note: Current Node.js version v22.22.x meets the minimum requirement of 22.0.0 and is fully compatible with AITBC platform.

🔥 Issue 1c: Operating System Compatibility

Current Status: Debian 13 Trixie (development environment)

Note: Development environment is running Debian 13 Trixie, which is newer than the minimum requirement of Debian 11+ and fully supported for AITBC development. This is the primary development environment for the AITBC platform.

🔥 Issue 2: Port Conflicts with Existing Services

Problem: Ports 8000-8008 may be in use by existing applications

Solution:

# Check port usage (new port logic)
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"

# Kill conflicting processes if needed
sudo fuser -k 8000/tcp  # Core services
sudo fuser -k 8010/tcp  # Enhanced services

# Alternative: Use different ports in service files
# Edit /etc/systemd/system/aitbc-*.service files
# Change --port 8000 to --port 9000, etc.

Port Mapping for aitbc1 (Updated):

Core Services:
- Coordinator API: 8000 ✅
- Exchange API: 8001 ✅
- Blockchain RPC: 8003 ✅

Enhanced Services:
- Multimodal GPU: 8010 ✅ (CPU-only mode)
- GPU Multimodal: 8011 ✅ (CPU-only mode)
- Modality Optimization: 8012 ✅
- Adaptive Learning: 8013 ✅
- Marketplace Enhanced: 8014 ✅
- OpenClaw Enhanced: 8015 ✅
- Web UI: 8016 ✅
- Geographic Load Balancer: 8017 ✅

🔥 Issue 3: Database Permission Issues

Problem: SQLite database file permissions preventing access

Solution:

# Fix database ownership (standardized)
sudo chown aitbc:aitbc /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Fix database permissions
sudo chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Ensure directory permissions
sudo chmod 755 /opt/aitbc/apps/coordinator-api/

🔥 Issue 4: Systemd Service Failures

Problem: Services failing to start due to missing dependencies

Solution:

# Check service status
sudo systemctl status aitbc-coordinator-api.service

# Check service logs
sudo journalctl -u aitbc-coordinator-api.service -n 50

# Common fixes:
# 1. Install missing Python packages
cd /opt/aitbc/apps/coordinator-api
source .venv/bin/activate
pip install missing-package

# 2. Fix environment variables
echo "ENVIRONMENT=production" >> .env

# 3. Fix working directory
sudo systemctl edit aitbc-coordinator-api.service
# Add: WorkingDirectory=/opt/aitbc/apps/coordinator-api

🔥 Issue 5: Nginx Proxy Configuration

Problem: Requests not properly forwarded from aitbc-cascade to aitbc

Solution:

# On aitbc-cascade, check proxy configuration
cat /etc/nginx/sites-available/aitbc-proxy.conf

# Ensure upstream configuration includes aitbc
upstream aitbc_backend {
    server 10.1.223.1:8000;  # Coordinator API
    server 10.1.223.1:8001;  # Exchange API
    server 10.1.223.1:8003;  # Blockchain RPC
    # Add enhanced services ports
    server 10.1.223.1:8010;  # Multimodal GPU
    server 10.1.223.1:8011;  # GPU Multimodal
    server 10.1.223.1:8012;  # Modality Optimization
    server 10.1.223.1:8013;  # Adaptive Learning
    server 10.1.223.1:8014;  # Marketplace Enhanced
    server 10.1.223.1:8015;  # OpenClaw Enhanced
    server 10.1.223.1:8016;  # Web UI
    server 10.1.223.1:8017;  # Geographic Load Balancer
}

# Reload nginx configuration
sudo nginx -t && sudo systemctl reload nginx

🔥 Issue 6: SSL Certificate Issues

Problem: SSL certificates not properly configured for aitbc domain

Solution:

# On aitbc-cascade, check certificate status
sudo certbot certificates

# Renew or obtain certificate
sudo certbot --nginx -d aitbc.bubuit.net

# Test SSL configuration
curl -I https://aitbc.bubuit.net

aitbc-Specific Configurations

Environment Variables

# /opt/aitbc/apps/coordinator-api/.env
MINER_API_KEYS=["aitbc_production_key_32_characters_long"]
DATABASE_URL=sqlite:///./aitbc_coordinator.db
LOG_LEVEL=INFO
ENVIRONMENT=production
API_HOST=0.0.0.0
API_PORT=8000
WORKERS=2  # Reduced for aitbc resources
SERVER_NAME=aitbc.bubuit.net
# Note: No miner service needed - configuration kept for compatibility

Service Configuration Adjustments

# aitbc-coordinator-api.service adjustments
# Edit: /etc/systemd/system/aitbc-coordinator-api.service

[Service]
User=aitbc
Group=aitbc
WorkingDirectory=/opt/aitbc/apps/coordinator-api
Environment=PYTHONPATH=src
EnvironmentFile=/opt/aitbc/apps/coordinator-api/.env
ExecStart=/opt/aitbc/apps/coordinator-api/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Resource Limits for aitbc

# /etc/systemd/system/aitbc-coordinator-api.service
[Service]
# Add resource limits
MemoryMax=2G
CPUQuota=200%
TasksMax=100

Performance Optimization for aitbc

Database Optimization

# SQLite optimization for aitbc
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 5000;  # Reduced for aitbc
PRAGMA temp_store = MEMORY;
PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 30000;
EOF

System Resource Limits

# /etc/security/limits.conf additions for aitbc
aitbc soft nofile 4096
aitbc hard nofile 4096
aitbc soft nproc 512
aitbc hard nproc 512

Network Optimization

# /etc/sysctl.conf additions for aitbc
net.core.somaxconn = 512
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.ip_local_port_range = 1024 65535

Monitoring Setup for aitbc

Custom Monitoring Script

# /opt/aitbc/scripts/monitor-aitbc.sh
#!/bin/bash
echo "AITBC Monitor - $(date)"
echo "========================"

# Service status
echo "Service Status:"
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"

# Resource usage
echo -e "\nResource Usage:"
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"

# Network connectivity
echo -e "\nNetwork Test:"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/v1/health" | grep -q "200" && echo "Coordinator API: ✅" || echo "Coordinator API: ❌"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8001/" | grep -q "200" && echo "Exchange API: ✅" || echo "Exchange API: ❌"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8003/rpc/head" | grep -q "200" && echo "Blockchain RPC: ✅" || echo "Blockchain RPC: ❌"

# Enhanced services health (CPU-only mode)
echo -e "\nEnhanced Services:"
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
    status=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:$port/health" 2>/dev/null)
    if [ "$status" = "200" ]; then
        service_name="Port $port"
        case $port in
            8010|8011) service_name="$service_name (CPU-only)" ;;
        esac
        echo "$service_name: ✅"
    else
        echo "Port $port: ❌ ($status)"
    fi
done

# Database status
echo -e "\nDatabase Status:"
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
    size=$(du -h /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db | cut -f1)
    echo "Database: ✅ ($size)"
else
    echo "Database: ❌ (Missing)"
fi

# Container access test
echo -e "\nContainer Access Test:"
curl -s -o /dev/null -w "%{http_code}" "http://10.1.223.1:8017/health" | grep -q "200" && echo "Container Access: ✅" || echo "Container Access: ❌"
EOF

chmod +x /opt/aitbc/scripts/monitor-aitbc.sh

Backup Strategy for aitbc

Automated Backup Script

# /opt/aitbc/scripts/backup-aitbc.sh
#!/bin/bash
BACKUP_DIR="/opt/aitbc/backups"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup database
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
    cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
    echo "Database backed up: aitbc_coordinator_$DATE.db"
fi

# Backup configuration
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/ 2>/dev/null
echo "Configuration backed up: config_$DATE.tar.gz"

# Backup service files
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
echo "Service files backed up: services_$DATE.tar.gz"

# Backup enhanced services scripts
tar -czf $BACKUP_DIR/enhanced-services_$DATE.tar.gz /opt/aitbc/scripts/*service*.py 2>/dev/null
echo "Enhanced services backed up: enhanced-services_$DATE.tar.gz"

# Clean old backups
find $BACKUP_DIR -name "*.db" -mtime +$RETENTION_DAYS -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete

echo "Backup completed: $DATE"
echo "Retention period: $RETENTION_DAYS days"
EOF

chmod +x /opt/aitbc/scripts/backup-aitbc.sh

Troubleshooting aitbc Specific Issues

Issue: Services Not Starting After Reboot

# Check if services are enabled
systemctl list-unit-files | grep aitbc

# Enable services for auto-start
sudo systemctl enable aitbc-coordinator-api.service
sudo systemctl enable aitbc-blockchain-node.service
sudo systemctl enable aitbc-blockchain-rpc.service
sudo systemctl enable aitbc-exchange-api.service

# Enable enhanced services
sudo systemctl enable aitbc-multimodal-gpu.service
sudo systemctl enable aitbc-multimodal.service
sudo systemctl enable aitbc-modality-optimization.service
sudo systemctl enable aitbc-adaptive-learning.service
sudo systemctl enable aitbc-marketplace-enhanced.service
sudo systemctl enable aitbc-openclaw-enhanced.service
sudo systemctl enable aitbc-web-ui.service
sudo systemctl enable aitbc-loadbalancer-geo.service

Issue: High Memory Usage

# Check memory usage
free -h
ps aux --sort=-%mem | head -10

# Reduce worker count in service files
# Edit ExecStart line: --workers 1 instead of --workers 4

Issue: Database Locking

# Check for database locks
sudo lsof /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Restart services to release locks
sudo systemctl restart aitbc-coordinator-api.service

Issue: Network Connectivity

# Test local connectivity
curl -X GET "http://localhost:8000/v1/health"

# Test external connectivity via proxy
curl -X GET "http://aitbc.bubuit.net/health"

# Check proxy configuration
ssh aitbc-cascade "cat /etc/nginx/sites-available/aitbc-proxy.conf"

Issue: Container Access Problems

# Test 0.0.0.0 binding
curl -s http://localhost:8017/health  # Should work
curl -s http://10.1.223.1:8017/health  # Should work from containers

# Check service binding
sudo netstat -tlnp | grep :8017  # Should show 0.0.0.0:8017

# Test from other containers
# From another container: curl http://aitbc:8017/health

Security Considerations for aitbc

Firewall Configuration

# Configure UFW on aitbc (if not using firehol)
sudo ufw allow ssh
sudo ufw allow 8000/tcp
sudo ufw allow 8001/tcp
sudo ufw allow 8003/tcp
sudo ufw allow 8010/tcp
sudo ufw allow 8011/tcp
sudo ufw allow 8012/tcp
sudo ufw allow 8013/tcp
sudo ufw allow 8014/tcp
sudo ufw allow 8015/tcp
sudo ufw allow 8016/tcp
sudo ufw allow 8017/tcp
sudo ufw --force enable

File Permissions

# Secure sensitive files
chmod 600 /opt/aitbc/apps/coordinator-api/.env
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
chmod 755 /opt/aitbc/apps/coordinator-api/

Access Control

# Restrict SSH access to specific users
echo "AllowUsers aitbc" | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart ssh

Deployment Validation Checklist

✅ Pre-Deployment

• Server access confirmed
• System requirements met
• Python 3.13+ installed and verified
• Dependencies installed
• Network configuration verified
• Container networking configured
• GPU access confirmed as not available
• Miner service requirements confirmed as not needed

✅ Deployment

• Codebase copied to /opt/aitbc
• Virtual environments created (Python 3.13+)
• Dependencies installed
• Environment variables configured
• Service files installed (new port logic)
• Services enabled and started

✅ Post-Deployment

• All 12 services running
• Core API endpoints responding (8000-8003)
• Enhanced services endpoints responding (8010-8017)
• Database operational
• Container access working (0.0.0.0 binding)
• Monitoring working
• Backup system active
• Security configured
• GPU services confirmed running in CPU-only mode
• Miner service confirmed as not needed

✅ Testing

• Health endpoints responding for all services
• API functionality verified
• Database operations working
• External access via proxy working
• SSL certificates valid
• Performance acceptable
• Container connectivity verified
• Geographic load balancer accessible from containers
• GPU services confirmed operating in CPU-only mode
• No miner service requirements confirmed

Rollback Procedures

Service Rollback

# Stop all services
sudo systemctl stop aitbc-*.service

# Restore previous configuration
sudo cp /etc/systemd/system/aitbc-*.service.backup /etc/systemd/system/
sudo systemctl daemon-reload

# Restore database
cp /opt/aitbc/backups/aitbc_coordinator_PREV_DEPLOY.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db

# Start services
sudo systemctl start aitbc-*.service

Full System Rollback

# Restore from backup
cd /opt/aitbc
tar -xzf /opt/aitbc/backups/full_backup_YYYYMMDD.tar.gz

# Restart services
sudo systemctl restart aitbc-*.service

Contact Information

Support Contacts

• Primary Admin: aitbc-admin
• Network Admin: aitbc-network
• Security Team: aitbc-security

Emergency Procedures

1. Check service status: systemctl status aitbc-*
2. Review logs: journalctl -u aitbc-coordinator-api.service
3. Run monitoring: /opt/aitbc/scripts/monitor-aitbc.sh
4. Check container access: curl http://10.1.223.1:8017/health
5. Verify GPU services running in CPU-only mode
6. Confirm no miner service is needed
7. Contact support if issues persist

---

Server: aitbc (Container)
Environment: Production
GPU Access: None (CPU-only mode)
Miner Service: Not needed
Last Updated: 2026-03-04
Maintainer: AITBC Operations Team
Status: ✅ PRODUCTION READY