refactor(ports): standardize service ports to 8000-8017 range and update CORS configurations across all services

- Update CORS allow_origins in blockchain-node app.py and gossip relay.py to use new port logic (8000-8016)
- Update coordinator-api config.py and config_pg.py with standardized port ranges and service labels
- Update coordinator-api health check script from port 18000 to 8000
- Update geo_load_balancer.py to use configurable host/port (default 0.0.0.0:8017)
- Update agent_security.py sandbox
This commit is contained in:
oib
2026-03-04 15:43:17 +01:00
parent 3df0a9ed62
commit 5534226895
57 changed files with 9690 additions and 126 deletions

View File

@@ -1,10 +1,10 @@
# AITBC Infrastructure Documentation
> Last updated: 2026-03-04
> Last updated: 2026-03-04 (Updated for new port logic and production-ready codebase)
## Overview
Two-tier architecture: **incus host (at1)** runs the reverse proxy with SSL termination, forwarding all `aitbc.bubuit.net` traffic to the **aitbc container** which runs nginx + all services.
Two-tier architecture: **incus host (at1)** runs the reverse proxy with SSL termination, forwarding all `aitbc.bubuit.net` traffic to the **aitbc container** which runs nginx + all services. **Updated for new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
```
Internet → aitbc.bubuit.net (HTTPS :443)
@@ -17,19 +17,28 @@ Internet → aitbc.bubuit.net (HTTPS :443)
│ aitbc-proxy.conf │
│ │
│ ┌────────────────────────────────────────┐ │
│ │ Container: aitbc (10.1.223.93) │ │
│ │ Container: aitbc (10.1.223.1) │ │
│ │ Access: ssh aitbc-cascade │ │
│ │ OS: Debian 13 Trixie │ │
│ │ Node.js: 22+ │ │
│ │ Python: 3.13.5+ │ │
│ │ GPU Access: None (CPU-only mode) │ │
│ │ Miner Service: Not needed │ │
│ │ │ │
│ │ Nginx (:80) → routes to services: │ │
│ │ / → static website │ │
│ │ /explorer/ → Vite SPA │ │
│ │ /marketplace/ → Vite SPA │ │
│ │ /Exchange → :3002 (Python) │ │
│ │ /docs/ → static HTML │ │
│ │ /wallet/ → :8002 (daemon) │ │
│ │ /api/ → :8000 (coordinator)│ │
│ │ /rpc/ → :9080 (blockchain) │ │
│ │ /admin/ → :8000 (coordinator)│ │
│ │ /api/exchange/ → :8001 (exchange) │ │
│ │ /rpc/ → :8003 (blockchain) │ │
│ │ /app/ → :8016 (web ui) │ │
│ │ /api/gpu/ → :8010 (multimodal) │ │
│ │ /api/gpu-multimodal/ → :8011 │ │
│ │ /api/optimization/ → :8012 │ │
│ │ /api/learning/ → :8013 │ │
│ │ /api/marketplace-enhanced/ → :8014 │ │
│ │ /api/openclaw/ → :8015 │ │
│ │ /health → 200 OK │ │
│ │ │ │
│ │ Config: /etc/nginx/sites-enabled/ │ │
@@ -38,11 +47,37 @@ Internet → aitbc.bubuit.net (HTTPS :443)
└──────────────────────────────────────────────┘
```
## Port Logic Implementation (March 4, 2026)
### **Core Services (8000-8003)**
- **Port 8000**: Coordinator API ✅ PRODUCTION READY
- **Port 8001**: Exchange API ✅ PRODUCTION READY
- **Port 8002**: Blockchain Node (internal) ✅ PRODUCTION READY
- **Port 8003**: Blockchain RPC ✅ PRODUCTION READY
### **Enhanced Services (8010-8017)**
- **Port 8010**: Multimodal GPU Service ✅ PRODUCTION READY (CPU-only mode)
- **Port 8011**: GPU Multimodal Service ✅ PRODUCTION READY (CPU-only mode)
- **Port 8012**: Modality Optimization Service ✅ PRODUCTION READY
- **Port 8013**: Adaptive Learning Service ✅ PRODUCTION READY
- **Port 8014**: Marketplace Enhanced Service ✅ PRODUCTION READY
- **Port 8015**: OpenClaw Enhanced Service ✅ PRODUCTION READY
- **Port 8016**: Web UI Service ✅ PRODUCTION READY
- **Port 8017**: Geographic Load Balancer ✅ PRODUCTION READY
### **Legacy Ports (Decommissioned)**
- **Port 8080**: No longer used by AITBC
- **Port 9080**: Successfully decommissioned
- **Port 8009**: No longer in use
## Incus Host (at1)
### Host Details
- **Hostname**: `at1` (primary development workstation)
- **Environment**: Windsurf development environment
- **OS**: Debian 13 Trixie (development environment)
- **Node.js**: 22+ (current tested: v22.22.x)
- **Python**: 3.13.5+ (minimum requirement, strictly enforced)
- **GPU Access**: **Primary GPU access location** - all GPU workloads must run on at1
- **Architecture**: x86_64 Linux with CUDA GPU support
@@ -50,11 +85,11 @@ Internet → aitbc.bubuit.net (HTTPS :443)
| Service | Port | Process | Python Version | Purpose | Status |
|---------|------|---------|----------------|---------|--------|
| Mock Coordinator | 8090 | python3 | 3.11+ | Development/testing API endpoint | systemd: aitbc-mock-coordinator.service |
| Blockchain Node | N/A | python3 | 3.11+ | Local blockchain node | systemd: aitbc-blockchain-node.service |
| Blockchain Node RPC | 9080 | python3 | 3.11+ | RPC API for blockchain | systemd: aitbc-blockchain-rpc.service |
| GPU Miner Client | N/A | python3 | 3.11+ | GPU mining client | systemd: aitbc-gpu-miner.service |
| Local Development Tools | Varies | python3 | 3.11+ | CLI tools, scripts, testing | Manual/venv |
| Mock Coordinator | 8090 | python3 | 3.13.5+ | Development/testing API endpoint | systemd: aitbc-mock-coordinator.service |
| Blockchain Node | N/A | python3 | 3.13.5+ | Local blockchain node | systemd: aitbc-blockchain-node.service |
| Blockchain Node RPC | 8003 | python3 | 3.13.5+ | RPC API for blockchain | systemd: aitbc-blockchain-rpc.service |
| Local Development Tools | Varies | python3 | 3.13.5+ | CLI tools, scripts, testing | Manual/venv |
| **Note**: GPU Miner Client removed - no miner service needed on aitbc server
### Systemd Services (Host)
@@ -63,14 +98,15 @@ All services are configured as systemd units but currently inactive:
```bash
# Service files location: /etc/systemd/system/
aitbc-blockchain-node.service # Blockchain node main process
aitbc-blockchain-rpc.service # RPC API on port 9080
aitbc-gpu-miner.service # GPU mining client
aitbc-blockchain-rpc.service # RPC API on port 8003
aitbc-mock-coordinator.service # Mock coordinator on port 8090
# Note: aitbc-gpu-miner.service removed - no miner service needed
```
**Service Details:**
- **Working Directory**: `/opt/aitbc/` (standard path for all services)
- **Python Environment**: `/opt/aitbc/.venv/bin/python`
- **Python Environment**: `/opt/aitbc/.venv/bin/python` (Python 3.13.5+)
- **Node.js Environment**: System Node.js 22+ (current tested: v22.22.x)
- **User**: oib
- **Restart Policy**: always (with 5s delay)
@@ -137,6 +173,10 @@ ls -la /opt/aitbc/systemd # Should show symlink to windsurf system
├── apps/coordinator-api -> /home/oib/windsurf/aitbc/apps/coordinator-api/
├── scripts -> /home/oib/windsurf/aitbc/scripts/
└── systemd -> /home/oib/windsurf/aitbc/systemd/
# Node.js environment
node --version # Should show v22.22.x
npm --version # Should show compatible version
```
**Note**: Services use individual symlinks to specific app directories, while development uses the full Windsurf project workspace.
@@ -148,6 +188,8 @@ ls -la /opt/aitbc/ # Should show individual symlinks, not s
ls -la /opt/aitbc/apps/blockchain-node # Should point to windsurf project
python3 --version # Should show Python 3.13.5
ls -la /home/oib/windsurf/aitbc/.venv/bin/python # Check development venv
node --version # Should show v22.22.x
npm --version # Should show compatible version
# Test symlink resolution
readlink -f /opt/aitbc/apps/blockchain-node # Should resolve to windsurf project path
@@ -185,7 +227,7 @@ server {
**Purged legacy configs** (2026-02-14):
- `aitbc-website-new.conf` — served files directly from host, bypassing container. Deleted.
## Container: aitbc (10.1.223.93)
## Container: aitbc (10.1.223.1)
### Access
```bash
@@ -194,10 +236,19 @@ ssh aitbc-cascade # Direct SSH to container
**GPU Access**: No GPU passthrough. All GPU workloads must run on **at1** (Windsurf development host), not inside incus containers.
**Miner Service**: Not needed - aitbc server operates in CPU-only mode.
**Host Proxies (for localhost GPU clients)**
- `127.0.0.1:18000` → container `127.0.0.1:8000` (coordinator/marketplace API)
- Use this to submit offers/bids/contracts/mining requests from localhost GPU miners/dev clients.
**Container Services (Updated March 4, 2026)**
- **12 Services**: All 12 services operational with new port logic
- **Core Services**: 8000-8003 (Coordinator, Exchange, Blockchain Node, RPC)
- **Enhanced Services**: 8010-8017 (GPU services in CPU-only mode, Web UI, Load Balancer)
- **0.0.0.0 Binding**: All services bind to 0.0.0.0 for container access
- **Production Ready**: All services marked as production ready
## Container: aitbc1 (10.1.223.40) — New Dev Server
### Access
@@ -207,7 +258,10 @@ ssh aitbc1-cascade # Direct SSH to aitbc1 container (incus)
### Notes
- Purpose: secondary AITBC dev environment (incus container)
- Host: 10.1.223.40 (Debian trixie), accessible via new SSH alias `aitbc1-cascade`
- Host: 10.1.223.40 (Debian 13 Trixie), accessible via new SSH alias `aitbc1-cascade`
- OS: Debian 13 Trixie (development environment)
- Node.js: 22+ (current tested: v22.22.x)
- Python: 3.13.5+ (minimum requirement, strictly enforced)
- Proxy device: incus proxy on host maps 127.0.0.1:18001 → 127.0.0.1:8000 inside container
- AppArmor profile: unconfined (incus raw.lxc)
- Use same deployment patterns as `aitbc` (nginx + services) once provisioned
@@ -236,36 +290,50 @@ ssh aitbc1-cascade # Direct SSH to aitbc1 container (incus)
### Services
| Service | Port | Process | Python Version | Public URL |
|---------|------|---------|----------------|------------|
| Nginx (web) | 80 | nginx | N/A | https://aitbc.bubuit.net/ |
| Coordinator API | 8000 | python (uvicorn) | 3.13.5 | /api/ → /v1/ |
| Blockchain Node RPC | 9080 | python3 | 3.13.5 | /rpc/ |
| Wallet Daemon | 8002 | python | 3.13.5 | /wallet/ |
| Trade Exchange | 3002 | python (server.py) | 3.13.5 | /Exchange |
| Exchange API | 8085 | python | 3.13.5 | /api/trades/*, /api/orders/* |
| Service | Port | Process | Python Version | Public URL | Status |
|---------|------|---------|----------------|------------|--------|
| Nginx (web) | 80 | nginx | N/A | https://aitbc.bubuit.net/ | ✅ |
| Coordinator API | 8000 | python (uvicorn) | 3.13.5 | /api/ → /v1/ | ✅ |
| Exchange API | 8001 | python (uvicorn) | 3.13.5 | /api/exchange/* | ✅ |
| Blockchain Node | 8002 | python3 | 3.13.5 | Internal | ✅ |
| Blockchain RPC | 8003 | python3 | 3.13.5 | /rpc/ | ✅ |
| Multimodal GPU | 8010 | python | 3.13.5 | /api/gpu/* | ✅ (CPU-only) |
| GPU Multimodal | 8011 | python | 3.13.5 | /api/gpu-multimodal/* | ✅ (CPU-only) |
| Modality Optimization | 8012 | python | 3.13.5 | /api/optimization/* | ✅ |
| Adaptive Learning | 8013 | python | 3.13.5 | /api/learning/* | ✅ |
| Marketplace Enhanced | 8014 | python | 3.13.5 | /api/marketplace-enhanced/* | ✅ |
| OpenClaw Enhanced | 8015 | python | 3.13.5 | /api/openclaw/* | ✅ |
| Web UI | 8016 | python | 3.13.5 | /app/ | ✅ |
| Geographic Load Balancer | 8017 | python | 3.13.5 | /api/loadbalancer/* | ✅ |
**Python 3.13.5 Upgrade Complete** (2026-02-23):
**Python 3.13.5 and Node.js 22+ Upgrade Complete** (2026-03-04):
- All services upgraded to Python 3.13.5
- Node.js upgraded to 22+ (current tested: v22.22.x)
- Virtual environments updated and verified
- API routing fixed for external access
- Services fully operational with enhanced performance
- New port logic implemented: Core Services (8000+), Enhanced Services (8010+)
- GPU services configured for CPU-only mode
- Miner service removed - not needed
- 0.0.0.0 binding enabled for container access
### Python Environment Details
All Python services in the AITBC container run on **Python 3.13.5** with isolated virtual environments:
```bash
# Container: aitbc (10.1.223.93)
/opt/coordinator-api/.venv/ # Coordinator API (uvicorn, FastAPI)
/opt/blockchain-node/.venv/ # Blockchain Node 1 (aitbc_chain)
/opt/blockchain-node-2/.venv/ # Blockchain Node 2 (aitbc_chain)
/opt/exchange/.venv/ # Exchange API (Flask/specific framework)
# Container: aitbc (10.1.223.1)
/opt/aitbc/apps/coordinator-api/.venv/ # Coordinator API (uvicorn, FastAPI)
/opt/aitbc/apps/blockchain-node/.venv/ # Blockchain Node 1 (aitbc_chain)
/opt/aitbc/apps/exchange/.venv/ # Exchange API (Flask/specific framework)
# Note: Standardized /opt/aitbc structure for all services
```
**Verification Commands:**
```bash
ssh aitbc-cascade "python3 --version" # Should show Python 3.13.5
ssh aitbc-cascade "node --version" # Should show v22.22.x
ssh aitbc-cascade "npm --version" # Should show compatible version
ssh aitbc-cascade "ls -la /opt/*/.venv/bin/python" # Check venv symlinks
ssh aitbc-cascade "curl -s http://127.0.0.1:8000/v1/health" # Coordinator API health
curl -s https://aitbc.bubuit.net/api/v1/health # External API access
@@ -275,30 +343,42 @@ curl -s https://aitbc.bubuit.net/api/v1/health # External API access
Config: `/etc/nginx/sites-enabled/aitbc.bubuit.net`
| Route | Target | Type |
|-------|--------|------|
| `/` | static files (`/var/www/aitbc.bubuit.net/`) | try_files |
| `/explorer/` | Vite SPA (`/var/www/aitbc.bubuit.net/explorer/`) | try_files |
| `/marketplace/` | Vite SPA (`/var/www/aitbc.bubuit.net/marketplace/`) | try_files |
| `/docs/` | static HTML (`/var/www/aitbc.bubuit.net/docs/`) | alias |
| `/Exchange` | proxy → `127.0.0.1:3002` | proxy_pass |
| `/exchange` | 301 → `/Exchange` | redirect |
| `/api/` | proxy → `127.0.0.1:8000/` | proxy_pass |
| `/api/explorer/` | proxy → `127.0.0.1:8000/v1/explorer/` | proxy_pass |
| `/api/users/` | proxy → `127.0.0.1:8000/v1/users/` | proxy_pass |
| `/api/trades/recent` | proxy → `127.0.0.1:8085` | proxy_pass |
| `/api/orders/orderbook` | proxy → `127.0.0.1:8085` | proxy_pass |
| `/admin/` | proxy → `127.0.0.1:8000/v1/admin/` | proxy_pass |
| `/rpc/` | proxy → `127.0.0.1:9080` | proxy_pass |
| `/wallet/` | proxy → `127.0.0.1:8002` | proxy_pass |
| `/v1/` | proxy → `10.1.223.1:8090` (mock coordinator) | proxy_pass |
| `/health` | 200 OK | direct |
| `/Marketplace` | 301 → `/marketplace/` | redirect (legacy) |
| `/BrowserWallet` | 301 → `/docs/browser-wallet.html` | redirect (legacy) |
| Route | Target | Type | Status |
|-------|--------|------|--------|
| `/` | static files (`/var/www/aitbc.bubuit.net/`) | try_files | ✅ |
| `/explorer/` | Vite SPA (`/var/www/aitbc.bubuit.net/explorer/`) | try_files | ✅ |
| `/marketplace/` | Vite SPA (`/var/www/aitbc.bubuit.net/marketplace/`) | try_files | ✅ |
| `/docs/` | static HTML (`/var/www/aitbc.bubuit.net/docs/`) | alias | ✅ |
| `/api/` | proxy → `127.0.0.1:8000/` | proxy_pass | ✅ |
| `/api/explorer/` | proxy → `127.0.0.1:8000/v1/explorer/` | proxy_pass | ✅ |
| `/api/users/` | proxy → `127.0.0.1:8000/v1/users/` | proxy_pass | ✅ |
| `/api/exchange/` | proxy → `127.0.0.1:8001/` | proxy_pass | ✅ |
| `/api/trades/recent` | proxy → `127.0.0.1:8001/trades/recent` | proxy_pass | ✅ |
| `/api/orders/orderbook` | proxy → `127.0.0.1:8001/orders/orderbook` | proxy_pass | ✅ |
| `/admin/` | proxy → `127.0.0.1:8000/v1/admin/` | proxy_pass | ✅ |
| `/rpc/` | proxy → `127.0.0.1:8003` | proxy_pass | ✅ |
| `/app/` | proxy → `127.0.0.1:8016` | proxy_pass | ✅ |
| `/api/gpu/` | proxy → `127.0.0.1:8010` | proxy_pass | ✅ (CPU-only) |
| `/api/gpu-multimodal/` | proxy → `127.0.0.1:8011` | proxy_pass | ✅ (CPU-only) |
| `/api/optimization/` | proxy → `127.0.0.1:8012` | proxy_pass | ✅ |
| `/api/learning/` | proxy → `127.0.0.1:8013` | proxy_pass | ✅ |
| `/api/marketplace-enhanced/` | proxy → `127.0.0.1:8014` | proxy_pass | ✅ |
| `/api/openclaw/` | proxy → `127.0.0.1:8015` | proxy_pass | ✅ |
| `/api/loadbalancer/` | proxy → `127.0.0.1:8017` | proxy_pass | ✅ |
| `/health` | 200 OK | direct | ✅ |
| `/Marketplace` | 301 → `/marketplace/` | redirect (legacy) | ✅ |
| `/BrowserWallet` | 301 → `/docs/browser-wallet.html` | redirect (legacy) | ✅ |
**API Routing Fixed** (2026-02-23):
**API Routing Updated** (2026-03-04):
- Updated `/api/` proxy_pass from `http://127.0.0.1:8000/v1/` to `http://127.0.0.1:8000/`
- Updated Exchange API routes to port 8001 (new port logic)
- Updated RPC route to port 8003 (new port logic)
- Added Enhanced Services routes (8010-8017)
- Added Web UI route to port 8016
- Added Geographic Load Balancer route to port 8017
- Removed legacy routes (Exchange, wallet, mock coordinator)
- External API access now working: `https://aitbc.bubuit.net/api/v1/health``{"status":"ok","env":"dev"}`
- All GPU services marked as CPU-only mode
### Web Root (`/var/www/aitbc.bubuit.net/`)
@@ -332,26 +412,24 @@ Config: `/etc/nginx/sites-enabled/aitbc.bubuit.net`
### Data Storage (container)
```
/opt/coordinator-api/ # Coordinator application
/opt/aitbc/apps/coordinator-api/ # Coordinator application
├── src/coordinator.db # Main database
└── .venv/ # Python environment
/opt/blockchain-node/ # Blockchain Node 1
/opt/aitbc/apps/blockchain-node/ # Blockchain Node 1
├── data/chain.db # Chain database
└── .venv/ # Python environment
/opt/blockchain-node-2/ # Blockchain Node 2
├── data/chain2.db # Chain database
└── .venv/ # Python environment
/opt/exchange/ # Exchange API
/opt/aitbc/apps/exchange/ # Exchange API
├── data/ # Exchange data
└── .venv/ # Python environment
```
### Configuration (container)
- Node 1: `/opt/blockchain-node/src/aitbc_chain/config.py`
- Node 2: `/opt/blockchain-node-2/src/aitbc_chain/config.py`
- Node 1: `/opt/aitbc/apps/blockchain-node/src/aitbc_chain/config.py`
- Coordinator API: `/opt/aitbc/apps/coordinator-api/.env`
- Exchange API: `/opt/aitbc/apps/exchange/.env`
- Enhanced Services: Environment variables in respective service files
## Remote Site (ns3)
@@ -359,16 +437,19 @@ Config: `/etc/nginx/sites-enabled/aitbc.bubuit.net`
- **IP**: 95.216.198.140
- **Access**: `ssh ns3-root`
- **Bridge**: incusbr0 `192.168.100.1/24`
- **Port forwarding**: firehol (8000, 8081, 8082, 9080 → 192.168.100.10)
- **Port forwarding**: firehol (8000, 8001, 8003, 8010-8017 → 192.168.100.10)
- **Updated**: Port logic aligned with main aitbc server
### Container (ns3/aitbc)
- **IP**: 192.168.100.10
- **Domain**: aitbc.keisanki.net
- **Access**: `ssh ns3-root``incus shell aitbc`
- **Blockchain Node 3**: RPC on port 8082
- **Blockchain Node 3**: RPC on port 8003 (updated port logic)
- **GPU Access**: None (CPU-only mode)
- **Miner Service**: Not needed
```bash
curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC
curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC (port 8003)
```
## Cross-Site Synchronization
@@ -377,8 +458,9 @@ curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC
- **Method**: RPC-based polling every 10 seconds
- **Features**: Transaction propagation, height detection, block import
- **Endpoints**:
- Local: https://aitbc.bubuit.net/rpc/ (Node 1, port 9080)
- Remote: http://aitbc.keisanki.net/rpc/ (Node 3, port 8082)
- Local: https://aitbc.bubuit.net/rpc/ (Node 1, port 8003)
- Remote: http://aitbc.keisanki.net/rpc/ (Node 3, port 8003)
- **Updated**: All nodes using new port logic (8003 for RPC)
- **Consensus**: PoA with 2s block intervals
- **P2P**: Not connected yet; nodes maintain independent chain state
@@ -414,15 +496,23 @@ ssh aitbc-cascade "systemctl restart coordinator-api"
```bash
# From at1 (via container)
ssh aitbc-cascade "curl -s http://localhost:8000/v1/health"
ssh aitbc-cascade "curl -s http://localhost:9080/rpc/head | jq .height"
ssh aitbc-cascade "curl -s http://localhost:8003/rpc/head | jq .height"
# Test enhanced services
ssh aitbc-cascade "curl -s http://localhost:8010/health" # Multimodal GPU (CPU-only)
ssh aitbc-cascade "curl -s http://localhost:8017/health" # Geographic Load Balancer
# From internet (Python 3.13.5 upgraded services)
curl -s https://aitbc.bubuit.net/health
curl -s https://aitbc.bubuit.net/api/v1/health # ✅ Fixed API routing
curl -s https://aitbc.bubuit.net/api/explorer/blocks
# Test enhanced services externally
curl -s https://aitbc.bubuit.net/api/gpu/health
curl -s https://aitbc.bubuit.net/api/loadbalancer/health
# Remote site
ssh ns3-root "curl -s http://192.168.100.10:8082/rpc/head | jq .height"
ssh ns3-root "curl -s http://192.168.100.10:8003/rpc/head | jq .height"
# Python version verification
ssh aitbc-cascade "python3 --version" # Python 3.13.5
@@ -432,8 +522,12 @@ ssh aitbc-cascade "python3 --version" # Python 3.13.5
```bash
# Container systemd logs
ssh aitbc-cascade "journalctl -u coordinator-api --no-pager -n 20"
ssh aitbc-cascade "journalctl -u aitbc-blockchain-node-1 --no-pager -n 20"
ssh aitbc-cascade "journalctl -u aitbc-coordinator-api --no-pager -n 20"
ssh aitbc-cascade "journalctl -u aitbc-blockchain-node --no-pager -n 20"
# Enhanced services logs
ssh aitbc-cascade "journalctl -u aitbc-multimodal-gpu --no-pager -n 20"
ssh aitbc-cascade "journalctl -u aitbc-loadbalancer-geo --no-pager -n 20"
# Container nginx logs
ssh aitbc-cascade "tail -20 /var/log/nginx/aitbc.bubuit.net.error.log"
@@ -450,9 +544,11 @@ sudo tail -20 /var/log/nginx/error.log
- HTTP → HTTPS redirect (Certbot managed)
### CORS
- Coordinator API: localhost origins only (3000, 8080, 8000, 8011)
- Exchange API: localhost origins only
- Blockchain Node: localhost origins only
- Coordinator API: localhost origins only (8000-8003, 8010-8017)
- Exchange API: localhost origins only (8000-8003, 8010-8017)
- Blockchain Node: localhost origins only (8000-8003, 8010-8017)
- Enhanced Services: localhost origins only (8010-8017)
- **Updated**: New port logic reflected in CORS policies
### Authentication
- Coordinator API: `X-Api-Key` header required
@@ -467,9 +563,15 @@ sudo tail -20 /var/log/nginx/error.log
```bash
# Coordinator API
JWT_SECRET=<secret>
DATABASE_URL=postgresql://user:pass@host/db
DATABASE_URL=sqlite:///./aitbc_coordinator.db
MINER_API_KEYS=["production_key_32_characters_long_minimum"]
# Note: No miner service needed - configuration kept for compatibility
# Exchange API
SESSION_SECRET=<secret>
WALLET_ENCRYPTION_KEY=<key>
# Enhanced Services
HOST=0.0.0.0 # For container access
PORT=8010-8017 # Enhanced services port range
```

577
docs/1_project/aitbc.md Normal file
View File

@@ -0,0 +1,577 @@
# AITBC Platform Deployment Guide
## Overview
This guide provides comprehensive deployment instructions for the AITBC (AI Trading Blockchain Compute) platform, including infrastructure requirements, service configurations, and troubleshooting procedures. **Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
## System Requirements
### **Hardware Requirements**
- **CPU**: 4+ cores recommended
- **Memory**: 8GB+ RAM minimum, 16GB+ recommended
- **Storage**: 50GB+ available space
- **Network**: Stable internet connection with 100Mbps+ bandwidth
- **GPU**: Not required (aitbc server has no GPU access)
- **Mining**: No miner service needed
### **Software Requirements**
- **Operating System**: Debian 13 Trixie (primary) or Ubuntu 22.04+ (alternative)
- **Python**: 3.13.5+ (strictly enforced - platform requires 3.13+ features)
- **Node.js**: 22+ (current tested: v22.22.x)
- **Database**: SQLite (default) or PostgreSQL (production)
### **Network Requirements**
- **Core Services Ports**: 8000-8003 (must be available)
- Port 8000: Coordinator API
- Port 8001: Exchange API
- Port 8002: Blockchain Node (internal)
- Port 8003: Blockchain RPC
- **Enhanced Services Ports**: 8010-8017 (must be available)
- Port 8010: Multimodal GPU Service
- Port 8011: GPU Multimodal Service
- Port 8012: Modality Optimization Service
- Port 8013: Adaptive Learning Service
- Port 8014: Marketplace Enhanced Service
- Port 8015: OpenClaw Enhanced Service
- Port 8016: Web UI Service
- Port 8017: Geographic Load Balancer
- **Firewall**: Managed by firehol on at1 host (container networking handled by incus)
- **SSL/TLS**: Recommended for production deployments
## Architecture Overview
```
AITBC Platform Architecture (Updated March 4, 2026)
├── Core Services (8000-8003)
│ ├── Coordinator API (Port 8000) ✅ PRODUCTION READY
│ ├── Exchange API (Port 8001) ✅ PRODUCTION READY
│ ├── Blockchain Node (Port 8002) ✅ PRODUCTION READY
│ └── Blockchain RPC (Port 8003) ✅ PRODUCTION READY
├── Enhanced Services (8010-8017)
│ ├── Multimodal GPU (Port 8010) ✅ PRODUCTION READY (CPU-only mode)
│ ├── GPU Multimodal (Port 8011) ✅ PRODUCTION READY (CPU-only mode)
│ ├── Modality Optimization (Port 8012) ✅ PRODUCTION READY
│ ├── Adaptive Learning (Port 8013) ✅ PRODUCTION READY
│ ├── Marketplace Enhanced (Port 8014) ✅ PRODUCTION READY
│ ├── OpenClaw Enhanced (Port 8015) ✅ PRODUCTION READY
│ ├── Web UI (Port 8016) ✅ PRODUCTION READY
│ └── Geographic Load Balancer (Port 8017) ✅ PRODUCTION READY
└── Infrastructure
├── Database (SQLite/PostgreSQL)
├── Monitoring & Logging
├── Security & Authentication
└── Container Support (0.0.0.0 binding)
```
## Deployment Steps
### **Phase 1: Environment Setup**
#### 1.1 System Preparation
```bash
# Update system packages
sudo apt update && sudo apt upgrade -y
# Install required packages
sudo apt install -y python3.13 python3.13-venv python3-pip nodejs npm nginx sqlite3
# Create aitbc user
sudo useradd -m -s /bin/bash aitbc
sudo usermod -aG sudo aitbc
```
#### 1.2 Directory Structure
```bash
# Create AITBC directory structure (standardized)
sudo mkdir -p /opt/aitbc/{apps,config,logs,scripts,backups}
sudo chown -R aitbc:aitbc /opt/aitbc
```
#### 1.3 Code Deployment
```bash
# Clone or copy AITBC codebase
cd /opt/aitbc
# Option 1: Git clone
git clone https://github.com/oib/AITBC.git .
# Option 2: Copy from existing installation
# scp -r /path/to/aitbc/* aitbc@target:/opt/aitbc/
# Set permissions (standardized)
sudo chown -R aitbc:aitbc /opt/aitbc
sudo chmod -R 755 /opt/aitbc
```
### **Phase 2: Service Configuration**
#### 2.1 Python Environment Setup
```bash
# Coordinator API Environment (Python 3.13+ required)
cd /opt/aitbc/apps/coordinator-api
python3.13 -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn sqlalchemy aiosqlite sqlmodel pydantic pydantic-settings httpx aiofiles python-jose passlib bcrypt prometheus-client slowapi websockets numpy
# Enhanced Services Environment (CPU-only mode)
cd /opt/aitbc/apps/coordinator-api # Enhanced services use same environment
source .venv/bin/activate
pip install aiohttp asyncio # Additional dependencies for enhanced services
# Note: GPU-related packages (CUDA, torch) not installed - no GPU access
```
#### 2.2 Environment Configuration
```bash
# Coordinator API Environment (Production)
cd /opt/aitbc/apps/coordinator-api
cat > .env << 'EOF'
MINER_API_KEYS=["production_key_32_characters_long_minimum"]
DATABASE_URL=sqlite:///./aitbc_coordinator.db
LOG_LEVEL=INFO
ENVIRONMENT=production
API_HOST=0.0.0.0
API_PORT=8000
WORKERS=4
# Note: No miner service needed - configuration kept for compatibility
EOF
# Set permissions
chmod 600 .env
chown aitbc:aitbc .env
```
#### 2.3 Systemd Service Installation
```bash
# Copy service files (updated for new port logic)
sudo cp -r /opt/aitbc/systemd/* /etc/systemd/system/
sudo systemctl daemon-reload
# Enable core services
sudo systemctl enable aitbc-coordinator-api.service
sudo systemctl enable aitbc-blockchain-node.service
sudo systemctl enable aitbc-blockchain-rpc.service
sudo systemctl enable aitbc-exchange-api.service
sudo systemctl enable aitbc-explorer.service
# Enable enhanced services
sudo systemctl enable aitbc-multimodal-gpu.service
sudo systemctl enable aitbc-multimodal.service
sudo systemctl enable aitbc-modality-optimization.service
sudo systemctl enable aitbc-adaptive-learning.service
sudo systemctl enable aitbc-marketplace-enhanced.service
sudo systemctl enable aitbc-openclaw-enhanced.service
sudo systemctl enable aitbc-web-ui.service
sudo systemctl enable aitbc-loadbalancer-geo.service
```
### **Phase 3: Service Deployment**
#### 3.1 Core Services Startup
```bash
# Start core services in order
sudo systemctl start aitbc-blockchain-node.service
sleep 5
sudo systemctl start aitbc-blockchain-rpc.service
sleep 3
sudo systemctl start aitbc-coordinator-api.service
sleep 3
sudo systemctl start aitbc-exchange-api.service
sleep 3
sudo systemctl start aitbc-explorer.service
```
#### 3.2 Enhanced Services Startup
```bash
# Start enhanced services (new port logic)
sudo systemctl start aitbc-multimodal-gpu.service # Port 8010
sudo systemctl start aitbc-multimodal.service # Port 8011
sudo systemctl start aitbc-modality-optimization.service # Port 8012
sudo systemctl start aitbc-adaptive-learning.service # Port 8013
sudo systemctl start aitbc-marketplace-enhanced.service # Port 8014
sudo systemctl start aitbc-openclaw-enhanced.service # Port 8015
sudo systemctl start aitbc-web-ui.service # Port 8016
sudo systemctl start aitbc-loadbalancer-geo.service # Port 8017
```
#### 3.3 Service Verification
```bash
# Check service status
sudo systemctl list-units --type=service --state=running | grep aitbc
# Test core endpoints
curl -X GET "http://localhost:8000/v1/health" # Coordinator API
curl -X GET "http://localhost:8001/" # Exchange API
curl -X GET "http://localhost:8003/rpc/head" # Blockchain RPC
# Test enhanced endpoints (CPU-only mode)
curl -X GET "http://localhost:8010/health" # Multimodal GPU (CPU mode)
curl -X GET "http://localhost:8011/health" # GPU Multimodal (CPU mode)
curl -X GET "http://localhost:8012/health" # Modality Optimization
curl -X GET "http://localhost:8013/health" # Adaptive Learning
curl -X GET "http://localhost:8014/health" # Marketplace Enhanced
curl -X GET "http://localhost:8015/health" # OpenClaw Enhanced
curl -X GET "http://localhost:8016/health" # Web UI
curl -X GET "http://localhost:8017/health" # Geographic Load Balancer
```
### **Phase 4: Production Configuration**
#### 4.1 Security Configuration
```bash
# Note: AITBC servers run in incus containers on at1 host
# Firewall is managed by firehol on at1, not ufw in containers
# Container networking is handled by incus with appropriate port forwarding
# Secure sensitive files
chmod 600 /opt/aitbc/apps/coordinator-api/.env
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
```
#### 4.2 Performance Optimization
```bash
# Database optimization
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 10000;
PRAGMA temp_store = MEMORY;
EOF
# System limits
echo "aitbc soft nofile 65536" | sudo tee -a /etc/security/limits.conf
echo "aitbc hard nofile 65536" | sudo tee -a /etc/security/limits.conf
# Network optimization
echo "net.core.somaxconn = 1024" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.tcp_max_syn_backlog = 1024" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```
#### 4.3 Monitoring Setup
```bash
# Create comprehensive monitoring script (updated for new port logic)
cat > /opt/aitbc/scripts/monitor-services.sh << 'EOF'
#!/bin/bash
echo "AITBC Service Monitor - $(date)"
echo "================================"
# Service status
echo "Service Status:"
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"
# Core endpoint health
echo -e "\nCore Services Health:"
for port in 8000 8001 8003; do
status=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$port/health" 2>/dev/null)
if [ "$status" = "200" ]; then
echo "Port $port: ✅ Healthy"
else
echo "Port $port: ❌ Unhealthy ($status)"
fi
done
# Enhanced endpoint health
echo -e "\nEnhanced Services Health:"
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
status=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$port/health" 2>/dev/null)
if [ "$status" = "200" ]; then
echo "Port $port: ✅ Healthy"
else
echo "Port $port: ❌ Unhealthy ($status)"
fi
done
# System resources
echo -e "\nSystem Resources:"
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
echo "Disk: $(df -h / | tail -1 | awk '{print $3"/"$2}')"
# Port usage verification
echo -e "\nPort Usage:"
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)" | sort
EOF
chmod +x /opt/aitbc/scripts/monitor-services.sh
chown aitbc:aitbc /opt/aitbc/scripts/monitor-services.sh
```
## Troubleshooting
### **Common Issues**
#### Service Not Starting
```bash
# Check service logs
sudo journalctl -u aitbc-coordinator-api.service -n 50
# Check Python environment (must be 3.13+)
cd /opt/aitbc/apps/coordinator-api
source .venv/bin/activate
python --version # Should show 3.13.x
# Check permissions
ls -la /opt/aitbc/apps/coordinator-api/
```
#### Database Issues
```bash
# Check database file
ls -la /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
# Test database connection
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db ".tables"
# Recreate database if corrupted
mv /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db.backup
```
#### Port Conflicts (New Port Logic)
```bash
# Check port usage (new port logic)
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"
# Kill conflicting processes
sudo fuser -k 8000/tcp # Core services
sudo fuser -k 8010/tcp # Enhanced services
# Restart services
sudo systemctl restart aitbc-coordinator-api.service
```
#### Container Access Issues
```bash
# Test 0.0.0.0 binding (for container access)
curl -s http://localhost:8017/health # Should work
curl -s http://10.1.223.1:8017/health # Should work from containers
# Check service binding
sudo netstat -tlnp | grep :8017 # Should show 0.0.0.0:8017
```
#### Permission Issues
```bash
# Fix file ownership (standardized)
sudo chown -R aitbc:aitbc /opt/aitbc
# Fix file permissions
sudo chmod -R 755 /opt/aitbc
chmod 600 /opt/aitbc/apps/coordinator-api/.env
```
### **Performance Issues**
#### High Memory Usage
```bash
# Check memory usage
free -h
ps aux --sort=-%mem | head -10
# Optimize Python processes
# Reduce worker count in service files
# Implement database connection pooling
```
#### High CPU Usage
```bash
# Check CPU usage
top
ps aux --sort=-%cpu | head -10
# Optimize database queries
# Add database indexes
# Implement caching
```
## Maintenance
### **Daily Tasks**
```bash
# Service health check (updated for new port logic)
/opt/aitbc/scripts/monitor-services.sh
# Log rotation
sudo logrotate -f /etc/logrotate.d/aitbc
# Backup database
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db /opt/aitbc/backups/aitbc_coordinator_$(date +%Y%m%d).db
```
### **Weekly Tasks**
```bash
# System updates
sudo apt update && sudo apt upgrade -y
# Service restart
sudo systemctl restart aitbc-*.service
# Performance review
/opt/aitbc/scripts/monitor-services.sh > /opt/aitbc/logs/weekly_$(date +%Y%m%d).log
```
### **Monthly Tasks**
```bash
# Security updates
sudo apt update && sudo apt upgrade -y
# Database maintenance
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db "VACUUM;"
# Log cleanup
find /opt/aitbc/logs -name "*.log" -mtime +30 -delete
```
## Scaling Considerations
### **Horizontal Scaling**
- Load balancer configuration (Port 8017)
- Multiple service instances
- Database clustering
- CDN implementation
### **Vertical Scaling**
- Resource allocation increases
- Performance optimization
- Caching strategies
- Database tuning
## Security Best Practices
### **Network Security**
- Firewall configuration
- SSL/TLS implementation
- VPN access for management
- Network segmentation
### **Application Security**
- Environment variable protection
- API rate limiting
- Input validation
- Regular security audits
### **Data Security**
- Database encryption
- Backup encryption
- Access control
- Audit logging
## Backup and Recovery
### **Automated Backup Script**
```bash
cat > /opt/aitbc/scripts/backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/opt/aitbc/backups"
DATE=$(date +%Y%m%d_%H%M%S)
# Create backup directory
mkdir -p $BACKUP_DIR
# Backup database
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
# Backup configuration
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/
# Backup scripts
tar -czf $BACKUP_DIR/scripts_$DATE.tar.gz /opt/aitbc/scripts/
# Backup service configurations
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
# Clean old backups (keep 7 days)
find $BACKUP_DIR -name "*.db" -mtime +7 -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete
echo "Backup completed: $DATE"
EOF
chmod +x /opt/aitbc/scripts/backup.sh
chown aitbc:aitbc /opt/aitbc/scripts/backup.sh
```
### **Recovery Procedures**
```bash
# Stop services
sudo systemctl stop aitbc-*.service
# Restore database
cp /opt/aitbc/backups/aitbc_coordinator_YYYYMMDD.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
# Restore configuration
tar -xzf /opt/aitbc/backups/config_YYYYMMDD.tar.gz -C /
# Restore service configurations
tar -xzf /opt/aitbc/backups/services_YYYYMMDD.tar.gz -C /
sudo systemctl daemon-reload
# Start services
sudo systemctl start aitbc-*.service
```
## Monitoring and Alerting
### **Key Metrics**
- Service uptime (all 12 services)
- API response times
- Database performance
- System resource usage
- Error rates
### **Alerting Thresholds**
- Service downtime > 5 minutes
- API response time > 1 second
- CPU usage > 80%
- Memory usage > 90%
- Disk usage > 85%
## Production Deployment Checklist
### **✅ Pre-Deployment**
- [ ] Python 3.13+ installed and verified
- [ ] All required ports available (8000-8003, 8010-8017)
- [ ] System requirements met
- [ ] Dependencies installed
- [ ] Network configuration verified
### **✅ Deployment**
- [ ] Codebase copied to /opt/aitbc
- [ ] Virtual environments created (Python 3.13+)
- [ ] Dependencies installed
- [ ] Environment variables configured
- [ ] Service files installed (new port logic)
- [ ] Services enabled and started
### **✅ Post-Deployment**
- [ ] All 12 services running
- [ ] Core API endpoints responding (8000-8003)
- [ ] Enhanced services endpoints responding (8010-8017)
- [ ] Database operational
- [ ] Container access working (0.0.0.0 binding)
- [ ] Monitoring working
- [ ] Backup system active
- [ ] Security configured
### **✅ Testing**
- [ ] Health endpoints responding for all services
- [ ] API functionality verified
- [ ] Database operations working
- [ ] External access via proxy working
- [ ] SSL certificates valid
- [ ] Performance acceptable
- [ ] Container connectivity verified
## Documentation References
- [Service Configuration Guide](./service-configuration.md)
- [Security Hardening Guide](./security-hardening.md)
- [Performance Optimization Guide](./performance-optimization.md)
- [Troubleshooting Guide](./troubleshooting.md)
- [Enhanced Services Guide](./enhanced-services.md)
- [Port Logic Implementation](./port-logic.md)
---
**Version**: 2.0 (Updated for new port logic)
**Last Updated**: 2026-03-04
**Maintainer**: AITBC Development Team
**Status**: ✅ PRODUCTION READY
**GPU Access**: None (CPU-only mode)
**Miner Service**: Not needed

584
docs/1_project/aitbc1.md Normal file
View File

@@ -0,0 +1,584 @@
# AITBC1 Deployment Notes
## Overview
This document contains specific deployment notes and considerations for deploying the AITBC platform on the **aitbc1** server. These notes complement the general deployment guide with server-specific configurations and troubleshooting. **Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
## Server Specifications
### **aitbc1 Server Details**
- **Hostname**: aitbc
- **IP Address**: 10.1.223.1 (container IP)
- **Operating System**: Debian 13 Trixie (primary development environment)
- **Access Method**: SSH via aitbc-cascade proxy
- **Web Root**: `/var/www/html/`
- **Nginx Configuration**: Two-tier setup with SSL termination
- **Container Support**: Incus containers with 0.0.0.0 binding for container access
### **Network Architecture**
```
Internet → aitbc-cascade (Proxy) → aitbc (Container)
SSL Termination Application Server
Port 443/80 Port 8000-8003, 8010-8017
```
## Pre-Deployment Checklist
### **✅ Server Preparation**
- [ ] SSH access confirmed via aitbc-cascade
- [ ] System packages updated
- [ ] aitbc user created with sudo access
- [ ] Directory structure created
- [ ] Firewall rules configured
- [ ] Python 3.13+ installed and verified
- [ ] Container networking configured
- [ ] GPU access confirmed as not available
- [ ] Miner service requirements confirmed as not needed
### **✅ Network Configuration**
- [ ] Port forwarding configured on aitbc-cascade
- [ ] SSL certificates installed on proxy
- [ ] DNS records configured
- [ ] Load balancer rules set
- [ ] Container access configured (0.0.0.0 binding)
### **✅ Storage Requirements**
- [ ] Minimum 50GB free space available
- [ ] Backup storage allocated
- [ ] Log rotation configured
- [ ] Database storage planned
## Deployment Issues & Solutions
### **🔥 Issue 1: Python Version Compatibility**
**Problem**: aitbc1 may have Python 3.10 instead of required 3.13+
**Solution**:
```bash
# Check current Python version
python3 --version
# Install Python 3.13 if not available
sudo apt update
sudo apt install -y python3.13 python3.13-venv python3.13-dev
# Update alternatives
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.13 1
```
**Verification**:
```bash
python3 --version # Should show 3.13.x
```
### **🔥 Issue 1b: Node.js Version Compatibility**
**Current Status**: Node.js v22.22.x (tested and compatible)
**Note**: Current Node.js version v22.22.x meets the minimum requirement of 22.0.0 and is fully compatible with AITBC platform.
### **🔥 Issue 1c: Operating System Compatibility**
**Current Status**: Debian 13 Trixie (development environment)
**Note**: Development environment is running Debian 13 Trixie, which is newer than the minimum requirement of Debian 11+ and fully supported for AITBC development. This is the primary development environment for the AITBC platform.
### **🔥 Issue 2: Port Conflicts with Existing Services**
**Problem**: Ports 8000-8008 may be in use by existing applications
**Solution**:
```bash
# Check port usage (new port logic)
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"
# Kill conflicting processes if needed
sudo fuser -k 8000/tcp # Core services
sudo fuser -k 8010/tcp # Enhanced services
# Alternative: Use different ports in service files
# Edit /etc/systemd/system/aitbc-*.service files
# Change --port 8000 to --port 9000, etc.
```
**Port Mapping for aitbc1 (Updated)**:
```
Core Services:
- Coordinator API: 8000 ✅
- Exchange API: 8001 ✅
- Blockchain RPC: 8003 ✅
Enhanced Services:
- Multimodal GPU: 8010 ✅ (CPU-only mode)
- GPU Multimodal: 8011 ✅ (CPU-only mode)
- Modality Optimization: 8012 ✅
- Adaptive Learning: 8013 ✅
- Marketplace Enhanced: 8014 ✅
- OpenClaw Enhanced: 8015 ✅
- Web UI: 8016 ✅
- Geographic Load Balancer: 8017 ✅
```
### **🔥 Issue 3: Database Permission Issues**
**Problem**: SQLite database file permissions preventing access
**Solution**:
```bash
# Fix database ownership (standardized)
sudo chown aitbc:aitbc /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
# Fix database permissions
sudo chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
# Ensure directory permissions
sudo chmod 755 /opt/aitbc/apps/coordinator-api/
```
### **🔥 Issue 4: Systemd Service Failures**
**Problem**: Services failing to start due to missing dependencies
**Solution**:
```bash
# Check service status
sudo systemctl status aitbc-coordinator-api.service
# Check service logs
sudo journalctl -u aitbc-coordinator-api.service -n 50
# Common fixes:
# 1. Install missing Python packages
cd /opt/aitbc/apps/coordinator-api
source .venv/bin/activate
pip install missing-package
# 2. Fix environment variables
echo "ENVIRONMENT=production" >> .env
# 3. Fix working directory
sudo systemctl edit aitbc-coordinator-api.service
# Add: WorkingDirectory=/opt/aitbc/apps/coordinator-api
```
### **🔥 Issue 5: Nginx Proxy Configuration**
**Problem**: Requests not properly forwarded from aitbc-cascade to aitbc
**Solution**:
```bash
# On aitbc-cascade, check proxy configuration
cat /etc/nginx/sites-available/aitbc-proxy.conf
# Ensure upstream configuration includes aitbc
upstream aitbc_backend {
server 10.1.223.1:8000; # Coordinator API
server 10.1.223.1:8001; # Exchange API
server 10.1.223.1:8003; # Blockchain RPC
# Add enhanced services ports
server 10.1.223.1:8010; # Multimodal GPU
server 10.1.223.1:8011; # GPU Multimodal
server 10.1.223.1:8012; # Modality Optimization
server 10.1.223.1:8013; # Adaptive Learning
server 10.1.223.1:8014; # Marketplace Enhanced
server 10.1.223.1:8015; # OpenClaw Enhanced
server 10.1.223.1:8016; # Web UI
server 10.1.223.1:8017; # Geographic Load Balancer
}
# Reload nginx configuration
sudo nginx -t && sudo systemctl reload nginx
```
### **🔥 Issue 6: SSL Certificate Issues**
**Problem**: SSL certificates not properly configured for aitbc domain
**Solution**:
```bash
# On aitbc-cascade, check certificate status
sudo certbot certificates
# Renew or obtain certificate
sudo certbot --nginx -d aitbc.bubuit.net
# Test SSL configuration
curl -I https://aitbc.bubuit.net
```
## aitbc-Specific Configurations
### **Environment Variables**
```bash
# /opt/aitbc/apps/coordinator-api/.env
MINER_API_KEYS=["aitbc_production_key_32_characters_long"]
DATABASE_URL=sqlite:///./aitbc_coordinator.db
LOG_LEVEL=INFO
ENVIRONMENT=production
API_HOST=0.0.0.0
API_PORT=8000
WORKERS=2 # Reduced for aitbc resources
SERVER_NAME=aitbc.bubuit.net
# Note: No miner service needed - configuration kept for compatibility
```
### **Service Configuration Adjustments**
```bash
# aitbc-coordinator-api.service adjustments
# Edit: /etc/systemd/system/aitbc-coordinator-api.service
[Service]
User=aitbc
Group=aitbc
WorkingDirectory=/opt/aitbc/apps/coordinator-api
Environment=PYTHONPATH=src
EnvironmentFile=/opt/aitbc/apps/coordinator-api/.env
ExecStart=/opt/aitbc/apps/coordinator-api/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
```
### **Resource Limits for aitbc**
```bash
# /etc/systemd/system/aitbc-coordinator-api.service
[Service]
# Add resource limits
MemoryMax=2G
CPUQuota=200%
TasksMax=100
```
## Performance Optimization for aitbc
### **Database Optimization**
```bash
# SQLite optimization for aitbc
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
PRAGMA synchronous = NORMAL;
PRAGMA cache_size = 5000; # Reduced for aitbc
PRAGMA temp_store = MEMORY;
PRAGMA journal_mode = WAL;
PRAGMA busy_timeout = 30000;
EOF
```
### **System Resource Limits**
```bash
# /etc/security/limits.conf additions for aitbc
aitbc soft nofile 4096
aitbc hard nofile 4096
aitbc soft nproc 512
aitbc hard nproc 512
```
### **Network Optimization**
```bash
# /etc/sysctl.conf additions for aitbc
net.core.somaxconn = 512
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.ip_local_port_range = 1024 65535
```
## Monitoring Setup for aitbc
### **Custom Monitoring Script**
```bash
# /opt/aitbc/scripts/monitor-aitbc.sh
#!/bin/bash
echo "AITBC Monitor - $(date)"
echo "========================"
# Service status
echo "Service Status:"
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"
# Resource usage
echo -e "\nResource Usage:"
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"
# Network connectivity
echo -e "\nNetwork Test:"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/v1/health" | grep -q "200" && echo "Coordinator API: ✅" || echo "Coordinator API: ❌"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8001/" | grep -q "200" && echo "Exchange API: ✅" || echo "Exchange API: ❌"
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8003/rpc/head" | grep -q "200" && echo "Blockchain RPC: ✅" || echo "Blockchain RPC: ❌"
# Enhanced services health (CPU-only mode)
echo -e "\nEnhanced Services:"
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
status=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:$port/health" 2>/dev/null)
if [ "$status" = "200" ]; then
service_name="Port $port"
case $port in
8010|8011) service_name="$service_name (CPU-only)" ;;
esac
echo "$service_name: ✅"
else
echo "Port $port: ❌ ($status)"
fi
done
# Database status
echo -e "\nDatabase Status:"
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
size=$(du -h /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db | cut -f1)
echo "Database: ✅ ($size)"
else
echo "Database: ❌ (Missing)"
fi
# Container access test
echo -e "\nContainer Access Test:"
curl -s -o /dev/null -w "%{http_code}" "http://10.1.223.1:8017/health" | grep -q "200" && echo "Container Access: ✅" || echo "Container Access: ❌"
EOF
chmod +x /opt/aitbc/scripts/monitor-aitbc.sh
```
## Backup Strategy for aitbc
### **Automated Backup Script**
```bash
# /opt/aitbc/scripts/backup-aitbc.sh
#!/bin/bash
BACKUP_DIR="/opt/aitbc/backups"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7
# Create backup directory
mkdir -p $BACKUP_DIR
# Backup database
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
echo "Database backed up: aitbc_coordinator_$DATE.db"
fi
# Backup configuration
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/ 2>/dev/null
echo "Configuration backed up: config_$DATE.tar.gz"
# Backup service files
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
echo "Service files backed up: services_$DATE.tar.gz"
# Backup enhanced services scripts
tar -czf $BACKUP_DIR/enhanced-services_$DATE.tar.gz /opt/aitbc/scripts/*service*.py 2>/dev/null
echo "Enhanced services backed up: enhanced-services_$DATE.tar.gz"
# Clean old backups
find $BACKUP_DIR -name "*.db" -mtime +$RETENTION_DAYS -delete
find $BACKUP_DIR -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete
echo "Backup completed: $DATE"
echo "Retention period: $RETENTION_DAYS days"
EOF
chmod +x /opt/aitbc/scripts/backup-aitbc.sh
```
## Troubleshooting aitbc Specific Issues
### **Issue: Services Not Starting After Reboot**
```bash
# Check if services are enabled
systemctl list-unit-files | grep aitbc
# Enable services for auto-start
sudo systemctl enable aitbc-coordinator-api.service
sudo systemctl enable aitbc-blockchain-node.service
sudo systemctl enable aitbc-blockchain-rpc.service
sudo systemctl enable aitbc-exchange-api.service
# Enable enhanced services
sudo systemctl enable aitbc-multimodal-gpu.service
sudo systemctl enable aitbc-multimodal.service
sudo systemctl enable aitbc-modality-optimization.service
sudo systemctl enable aitbc-adaptive-learning.service
sudo systemctl enable aitbc-marketplace-enhanced.service
sudo systemctl enable aitbc-openclaw-enhanced.service
sudo systemctl enable aitbc-web-ui.service
sudo systemctl enable aitbc-loadbalancer-geo.service
```
### **Issue: High Memory Usage**
```bash
# Check memory usage
free -h
ps aux --sort=-%mem | head -10
# Reduce worker count in service files
# Edit ExecStart line: --workers 1 instead of --workers 4
```
### **Issue: Database Locking**
```bash
# Check for database locks
sudo lsof /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
# Restart services to release locks
sudo systemctl restart aitbc-coordinator-api.service
```
### **Issue: Network Connectivity**
```bash
# Test local connectivity
curl -X GET "http://localhost:8000/v1/health"
# Test external connectivity via proxy
curl -X GET "http://aitbc.bubuit.net/health"
# Check proxy configuration
ssh aitbc-cascade "cat /etc/nginx/sites-available/aitbc-proxy.conf"
```
### **Issue: Container Access Problems**
```bash
# Test 0.0.0.0 binding
curl -s http://localhost:8017/health # Should work
curl -s http://10.1.223.1:8017/health # Should work from containers
# Check service binding
sudo netstat -tlnp | grep :8017 # Should show 0.0.0.0:8017
# Test from other containers
# From another container: curl http://aitbc:8017/health
```
## Security Considerations for aitbc
### **Firewall Configuration**
```bash
# Configure UFW on aitbc (if not using firehol)
sudo ufw allow ssh
sudo ufw allow 8000/tcp
sudo ufw allow 8001/tcp
sudo ufw allow 8003/tcp
sudo ufw allow 8010/tcp
sudo ufw allow 8011/tcp
sudo ufw allow 8012/tcp
sudo ufw allow 8013/tcp
sudo ufw allow 8014/tcp
sudo ufw allow 8015/tcp
sudo ufw allow 8016/tcp
sudo ufw allow 8017/tcp
sudo ufw --force enable
```
### **File Permissions**
```bash
# Secure sensitive files
chmod 600 /opt/aitbc/apps/coordinator-api/.env
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
chmod 755 /opt/aitbc/apps/coordinator-api/
```
### **Access Control**
```bash
# Restrict SSH access to specific users
echo "AllowUsers aitbc" | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart ssh
```
## Deployment Validation Checklist
### **✅ Pre-Deployment**
- [ ] Server access confirmed
- [ ] System requirements met
- [ ] Python 3.13+ installed and verified
- [ ] Dependencies installed
- [ ] Network configuration verified
- [ ] Container networking configured
- [ ] GPU access confirmed as not available
- [ ] Miner service requirements confirmed as not needed
### **✅ Deployment**
- [ ] Codebase copied to /opt/aitbc
- [ ] Virtual environments created (Python 3.13+)
- [ ] Dependencies installed
- [ ] Environment variables configured
- [ ] Service files installed (new port logic)
- [ ] Services enabled and started
### **✅ Post-Deployment**
- [ ] All 12 services running
- [ ] Core API endpoints responding (8000-8003)
- [ ] Enhanced services endpoints responding (8010-8017)
- [ ] Database operational
- [ ] Container access working (0.0.0.0 binding)
- [ ] Monitoring working
- [ ] Backup system active
- [ ] Security configured
- [ ] GPU services confirmed running in CPU-only mode
- [ ] Miner service confirmed as not needed
### **✅ Testing**
- [ ] Health endpoints responding for all services
- [ ] API functionality verified
- [ ] Database operations working
- [ ] External access via proxy working
- [ ] SSL certificates valid
- [ ] Performance acceptable
- [ ] Container connectivity verified
- [ ] Geographic load balancer accessible from containers
- [ ] GPU services confirmed operating in CPU-only mode
- [ ] No miner service requirements confirmed
## Rollback Procedures
### **Service Rollback**
```bash
# Stop all services
sudo systemctl stop aitbc-*.service
# Restore previous configuration
sudo cp /etc/systemd/system/aitbc-*.service.backup /etc/systemd/system/
sudo systemctl daemon-reload
# Restore database
cp /opt/aitbc/backups/aitbc_coordinator_PREV_DEPLOY.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
# Start services
sudo systemctl start aitbc-*.service
```
### **Full System Rollback**
```bash
# Restore from backup
cd /opt/aitbc
tar -xzf /opt/aitbc/backups/full_backup_YYYYMMDD.tar.gz
# Restart services
sudo systemctl restart aitbc-*.service
```
## Contact Information
### **Support Contacts**
- **Primary Admin**: aitbc-admin
- **Network Admin**: aitbc-network
- **Security Team**: aitbc-security
### **Emergency Procedures**
1. Check service status: `systemctl status aitbc-*`
2. Review logs: `journalctl -u aitbc-coordinator-api.service`
3. Run monitoring: `/opt/aitbc/scripts/monitor-aitbc.sh`
4. Check container access: `curl http://10.1.223.1:8017/health`
5. Verify GPU services running in CPU-only mode
6. Confirm no miner service is needed
7. Contact support if issues persist
---
**Server**: aitbc (Container)
**Environment**: Production
**GPU Access**: None (CPU-only mode)
**Miner Service**: Not needed
**Last Updated**: 2026-03-04
**Maintainer**: AITBC Operations Team
**Status**: ✅ PRODUCTION READY