refactor(ports): standardize service ports to 8000-8017 range and update CORS configurations across all services
- Update CORS allow_origins in blockchain-node app.py and gossip relay.py to use new port logic (8000-8016) - Update coordinator-api config.py and config_pg.py with standardized port ranges and service labels - Update coordinator-api health check script from port 18000 to 8000 - Update geo_load_balancer.py to use configurable host/port (default 0.0.0.0:8017) - Update agent_security.py sandbox
This commit is contained in:
@@ -1,10 +1,10 @@
|
||||
# AITBC Infrastructure Documentation
|
||||
|
||||
> Last updated: 2026-03-04
|
||||
> Last updated: 2026-03-04 (Updated for new port logic and production-ready codebase)
|
||||
|
||||
## Overview
|
||||
|
||||
Two-tier architecture: **incus host (at1)** runs the reverse proxy with SSL termination, forwarding all `aitbc.bubuit.net` traffic to the **aitbc container** which runs nginx + all services.
|
||||
Two-tier architecture: **incus host (at1)** runs the reverse proxy with SSL termination, forwarding all `aitbc.bubuit.net` traffic to the **aitbc container** which runs nginx + all services. **Updated for new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
|
||||
|
||||
```
|
||||
Internet → aitbc.bubuit.net (HTTPS :443)
|
||||
@@ -17,19 +17,28 @@ Internet → aitbc.bubuit.net (HTTPS :443)
|
||||
│ aitbc-proxy.conf │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────┐ │
|
||||
│ │ Container: aitbc (10.1.223.93) │ │
|
||||
│ │ Container: aitbc (10.1.223.1) │ │
|
||||
│ │ Access: ssh aitbc-cascade │ │
|
||||
│ │ OS: Debian 13 Trixie │ │
|
||||
│ │ Node.js: 22+ │ │
|
||||
│ │ Python: 3.13.5+ │ │
|
||||
│ │ GPU Access: None (CPU-only mode) │ │
|
||||
│ │ Miner Service: Not needed │ │
|
||||
│ │ │ │
|
||||
│ │ Nginx (:80) → routes to services: │ │
|
||||
│ │ / → static website │ │
|
||||
│ │ /explorer/ → Vite SPA │ │
|
||||
│ │ /marketplace/ → Vite SPA │ │
|
||||
│ │ /Exchange → :3002 (Python) │ │
|
||||
│ │ /docs/ → static HTML │ │
|
||||
│ │ /wallet/ → :8002 (daemon) │ │
|
||||
│ │ /api/ → :8000 (coordinator)│ │
|
||||
│ │ /rpc/ → :9080 (blockchain) │ │
|
||||
│ │ /admin/ → :8000 (coordinator)│ │
|
||||
│ │ /api/exchange/ → :8001 (exchange) │ │
|
||||
│ │ /rpc/ → :8003 (blockchain) │ │
|
||||
│ │ /app/ → :8016 (web ui) │ │
|
||||
│ │ /api/gpu/ → :8010 (multimodal) │ │
|
||||
│ │ /api/gpu-multimodal/ → :8011 │ │
|
||||
│ │ /api/optimization/ → :8012 │ │
|
||||
│ │ /api/learning/ → :8013 │ │
|
||||
│ │ /api/marketplace-enhanced/ → :8014 │ │
|
||||
│ │ /api/openclaw/ → :8015 │ │
|
||||
│ │ /health → 200 OK │ │
|
||||
│ │ │ │
|
||||
│ │ Config: /etc/nginx/sites-enabled/ │ │
|
||||
@@ -38,11 +47,37 @@ Internet → aitbc.bubuit.net (HTTPS :443)
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Port Logic Implementation (March 4, 2026)
|
||||
|
||||
### **Core Services (8000-8003)**
|
||||
- **Port 8000**: Coordinator API ✅ PRODUCTION READY
|
||||
- **Port 8001**: Exchange API ✅ PRODUCTION READY
|
||||
- **Port 8002**: Blockchain Node (internal) ✅ PRODUCTION READY
|
||||
- **Port 8003**: Blockchain RPC ✅ PRODUCTION READY
|
||||
|
||||
### **Enhanced Services (8010-8017)**
|
||||
- **Port 8010**: Multimodal GPU Service ✅ PRODUCTION READY (CPU-only mode)
|
||||
- **Port 8011**: GPU Multimodal Service ✅ PRODUCTION READY (CPU-only mode)
|
||||
- **Port 8012**: Modality Optimization Service ✅ PRODUCTION READY
|
||||
- **Port 8013**: Adaptive Learning Service ✅ PRODUCTION READY
|
||||
- **Port 8014**: Marketplace Enhanced Service ✅ PRODUCTION READY
|
||||
- **Port 8015**: OpenClaw Enhanced Service ✅ PRODUCTION READY
|
||||
- **Port 8016**: Web UI Service ✅ PRODUCTION READY
|
||||
- **Port 8017**: Geographic Load Balancer ✅ PRODUCTION READY
|
||||
|
||||
### **Legacy Ports (Decommissioned)**
|
||||
- **Port 8080**: No longer used by AITBC
|
||||
- **Port 9080**: Successfully decommissioned
|
||||
- **Port 8009**: No longer in use
|
||||
|
||||
## Incus Host (at1)
|
||||
|
||||
### Host Details
|
||||
- **Hostname**: `at1` (primary development workstation)
|
||||
- **Environment**: Windsurf development environment
|
||||
- **OS**: Debian 13 Trixie (development environment)
|
||||
- **Node.js**: 22+ (current tested: v22.22.x)
|
||||
- **Python**: 3.13.5+ (minimum requirement, strictly enforced)
|
||||
- **GPU Access**: **Primary GPU access location** - all GPU workloads must run on at1
|
||||
- **Architecture**: x86_64 Linux with CUDA GPU support
|
||||
|
||||
@@ -50,11 +85,11 @@ Internet → aitbc.bubuit.net (HTTPS :443)
|
||||
|
||||
| Service | Port | Process | Python Version | Purpose | Status |
|
||||
|---------|------|---------|----------------|---------|--------|
|
||||
| Mock Coordinator | 8090 | python3 | 3.11+ | Development/testing API endpoint | systemd: aitbc-mock-coordinator.service |
|
||||
| Blockchain Node | N/A | python3 | 3.11+ | Local blockchain node | systemd: aitbc-blockchain-node.service |
|
||||
| Blockchain Node RPC | 9080 | python3 | 3.11+ | RPC API for blockchain | systemd: aitbc-blockchain-rpc.service |
|
||||
| GPU Miner Client | N/A | python3 | 3.11+ | GPU mining client | systemd: aitbc-gpu-miner.service |
|
||||
| Local Development Tools | Varies | python3 | 3.11+ | CLI tools, scripts, testing | Manual/venv |
|
||||
| Mock Coordinator | 8090 | python3 | 3.13.5+ | Development/testing API endpoint | systemd: aitbc-mock-coordinator.service |
|
||||
| Blockchain Node | N/A | python3 | 3.13.5+ | Local blockchain node | systemd: aitbc-blockchain-node.service |
|
||||
| Blockchain Node RPC | 8003 | python3 | 3.13.5+ | RPC API for blockchain | systemd: aitbc-blockchain-rpc.service |
|
||||
| Local Development Tools | Varies | python3 | 3.13.5+ | CLI tools, scripts, testing | Manual/venv |
|
||||
| **Note**: GPU Miner Client removed - no miner service needed on aitbc server
|
||||
|
||||
### Systemd Services (Host)
|
||||
|
||||
@@ -63,14 +98,15 @@ All services are configured as systemd units but currently inactive:
|
||||
```bash
|
||||
# Service files location: /etc/systemd/system/
|
||||
aitbc-blockchain-node.service # Blockchain node main process
|
||||
aitbc-blockchain-rpc.service # RPC API on port 9080
|
||||
aitbc-gpu-miner.service # GPU mining client
|
||||
aitbc-blockchain-rpc.service # RPC API on port 8003
|
||||
aitbc-mock-coordinator.service # Mock coordinator on port 8090
|
||||
# Note: aitbc-gpu-miner.service removed - no miner service needed
|
||||
```
|
||||
|
||||
**Service Details:**
|
||||
- **Working Directory**: `/opt/aitbc/` (standard path for all services)
|
||||
- **Python Environment**: `/opt/aitbc/.venv/bin/python`
|
||||
- **Python Environment**: `/opt/aitbc/.venv/bin/python` (Python 3.13.5+)
|
||||
- **Node.js Environment**: System Node.js 22+ (current tested: v22.22.x)
|
||||
- **User**: oib
|
||||
- **Restart Policy**: always (with 5s delay)
|
||||
|
||||
@@ -137,6 +173,10 @@ ls -la /opt/aitbc/systemd # Should show symlink to windsurf system
|
||||
├── apps/coordinator-api -> /home/oib/windsurf/aitbc/apps/coordinator-api/
|
||||
├── scripts -> /home/oib/windsurf/aitbc/scripts/
|
||||
└── systemd -> /home/oib/windsurf/aitbc/systemd/
|
||||
|
||||
# Node.js environment
|
||||
node --version # Should show v22.22.x
|
||||
npm --version # Should show compatible version
|
||||
```
|
||||
|
||||
**Note**: Services use individual symlinks to specific app directories, while development uses the full Windsurf project workspace.
|
||||
@@ -148,6 +188,8 @@ ls -la /opt/aitbc/ # Should show individual symlinks, not s
|
||||
ls -la /opt/aitbc/apps/blockchain-node # Should point to windsurf project
|
||||
python3 --version # Should show Python 3.13.5
|
||||
ls -la /home/oib/windsurf/aitbc/.venv/bin/python # Check development venv
|
||||
node --version # Should show v22.22.x
|
||||
npm --version # Should show compatible version
|
||||
|
||||
# Test symlink resolution
|
||||
readlink -f /opt/aitbc/apps/blockchain-node # Should resolve to windsurf project path
|
||||
@@ -185,7 +227,7 @@ server {
|
||||
**Purged legacy configs** (2026-02-14):
|
||||
- `aitbc-website-new.conf` — served files directly from host, bypassing container. Deleted.
|
||||
|
||||
## Container: aitbc (10.1.223.93)
|
||||
## Container: aitbc (10.1.223.1)
|
||||
|
||||
### Access
|
||||
```bash
|
||||
@@ -194,10 +236,19 @@ ssh aitbc-cascade # Direct SSH to container
|
||||
|
||||
**GPU Access**: No GPU passthrough. All GPU workloads must run on **at1** (Windsurf development host), not inside incus containers.
|
||||
|
||||
**Miner Service**: Not needed - aitbc server operates in CPU-only mode.
|
||||
|
||||
**Host Proxies (for localhost GPU clients)**
|
||||
- `127.0.0.1:18000` → container `127.0.0.1:8000` (coordinator/marketplace API)
|
||||
- Use this to submit offers/bids/contracts/mining requests from localhost GPU miners/dev clients.
|
||||
|
||||
**Container Services (Updated March 4, 2026)**
|
||||
- **12 Services**: All 12 services operational with new port logic
|
||||
- **Core Services**: 8000-8003 (Coordinator, Exchange, Blockchain Node, RPC)
|
||||
- **Enhanced Services**: 8010-8017 (GPU services in CPU-only mode, Web UI, Load Balancer)
|
||||
- **0.0.0.0 Binding**: All services bind to 0.0.0.0 for container access
|
||||
- **Production Ready**: All services marked as production ready
|
||||
|
||||
## Container: aitbc1 (10.1.223.40) — New Dev Server
|
||||
|
||||
### Access
|
||||
@@ -207,7 +258,10 @@ ssh aitbc1-cascade # Direct SSH to aitbc1 container (incus)
|
||||
|
||||
### Notes
|
||||
- Purpose: secondary AITBC dev environment (incus container)
|
||||
- Host: 10.1.223.40 (Debian trixie), accessible via new SSH alias `aitbc1-cascade`
|
||||
- Host: 10.1.223.40 (Debian 13 Trixie), accessible via new SSH alias `aitbc1-cascade`
|
||||
- OS: Debian 13 Trixie (development environment)
|
||||
- Node.js: 22+ (current tested: v22.22.x)
|
||||
- Python: 3.13.5+ (minimum requirement, strictly enforced)
|
||||
- Proxy device: incus proxy on host maps 127.0.0.1:18001 → 127.0.0.1:8000 inside container
|
||||
- AppArmor profile: unconfined (incus raw.lxc)
|
||||
- Use same deployment patterns as `aitbc` (nginx + services) once provisioned
|
||||
@@ -236,36 +290,50 @@ ssh aitbc1-cascade # Direct SSH to aitbc1 container (incus)
|
||||
|
||||
### Services
|
||||
|
||||
| Service | Port | Process | Python Version | Public URL |
|
||||
|---------|------|---------|----------------|------------|
|
||||
| Nginx (web) | 80 | nginx | N/A | https://aitbc.bubuit.net/ |
|
||||
| Coordinator API | 8000 | python (uvicorn) | 3.13.5 | /api/ → /v1/ |
|
||||
| Blockchain Node RPC | 9080 | python3 | 3.13.5 | /rpc/ |
|
||||
| Wallet Daemon | 8002 | python | 3.13.5 | /wallet/ |
|
||||
| Trade Exchange | 3002 | python (server.py) | 3.13.5 | /Exchange |
|
||||
| Exchange API | 8085 | python | 3.13.5 | /api/trades/*, /api/orders/* |
|
||||
| Service | Port | Process | Python Version | Public URL | Status |
|
||||
|---------|------|---------|----------------|------------|--------|
|
||||
| Nginx (web) | 80 | nginx | N/A | https://aitbc.bubuit.net/ | ✅ |
|
||||
| Coordinator API | 8000 | python (uvicorn) | 3.13.5 | /api/ → /v1/ | ✅ |
|
||||
| Exchange API | 8001 | python (uvicorn) | 3.13.5 | /api/exchange/* | ✅ |
|
||||
| Blockchain Node | 8002 | python3 | 3.13.5 | Internal | ✅ |
|
||||
| Blockchain RPC | 8003 | python3 | 3.13.5 | /rpc/ | ✅ |
|
||||
| Multimodal GPU | 8010 | python | 3.13.5 | /api/gpu/* | ✅ (CPU-only) |
|
||||
| GPU Multimodal | 8011 | python | 3.13.5 | /api/gpu-multimodal/* | ✅ (CPU-only) |
|
||||
| Modality Optimization | 8012 | python | 3.13.5 | /api/optimization/* | ✅ |
|
||||
| Adaptive Learning | 8013 | python | 3.13.5 | /api/learning/* | ✅ |
|
||||
| Marketplace Enhanced | 8014 | python | 3.13.5 | /api/marketplace-enhanced/* | ✅ |
|
||||
| OpenClaw Enhanced | 8015 | python | 3.13.5 | /api/openclaw/* | ✅ |
|
||||
| Web UI | 8016 | python | 3.13.5 | /app/ | ✅ |
|
||||
| Geographic Load Balancer | 8017 | python | 3.13.5 | /api/loadbalancer/* | ✅ |
|
||||
|
||||
**Python 3.13.5 Upgrade Complete** (2026-02-23):
|
||||
**Python 3.13.5 and Node.js 22+ Upgrade Complete** (2026-03-04):
|
||||
- All services upgraded to Python 3.13.5
|
||||
- Node.js upgraded to 22+ (current tested: v22.22.x)
|
||||
- Virtual environments updated and verified
|
||||
- API routing fixed for external access
|
||||
- Services fully operational with enhanced performance
|
||||
- New port logic implemented: Core Services (8000+), Enhanced Services (8010+)
|
||||
- GPU services configured for CPU-only mode
|
||||
- Miner service removed - not needed
|
||||
- 0.0.0.0 binding enabled for container access
|
||||
|
||||
### Python Environment Details
|
||||
|
||||
All Python services in the AITBC container run on **Python 3.13.5** with isolated virtual environments:
|
||||
|
||||
```bash
|
||||
# Container: aitbc (10.1.223.93)
|
||||
/opt/coordinator-api/.venv/ # Coordinator API (uvicorn, FastAPI)
|
||||
/opt/blockchain-node/.venv/ # Blockchain Node 1 (aitbc_chain)
|
||||
/opt/blockchain-node-2/.venv/ # Blockchain Node 2 (aitbc_chain)
|
||||
/opt/exchange/.venv/ # Exchange API (Flask/specific framework)
|
||||
# Container: aitbc (10.1.223.1)
|
||||
/opt/aitbc/apps/coordinator-api/.venv/ # Coordinator API (uvicorn, FastAPI)
|
||||
/opt/aitbc/apps/blockchain-node/.venv/ # Blockchain Node 1 (aitbc_chain)
|
||||
/opt/aitbc/apps/exchange/.venv/ # Exchange API (Flask/specific framework)
|
||||
# Note: Standardized /opt/aitbc structure for all services
|
||||
```
|
||||
|
||||
**Verification Commands:**
|
||||
```bash
|
||||
ssh aitbc-cascade "python3 --version" # Should show Python 3.13.5
|
||||
ssh aitbc-cascade "node --version" # Should show v22.22.x
|
||||
ssh aitbc-cascade "npm --version" # Should show compatible version
|
||||
ssh aitbc-cascade "ls -la /opt/*/.venv/bin/python" # Check venv symlinks
|
||||
ssh aitbc-cascade "curl -s http://127.0.0.1:8000/v1/health" # Coordinator API health
|
||||
curl -s https://aitbc.bubuit.net/api/v1/health # External API access
|
||||
@@ -275,30 +343,42 @@ curl -s https://aitbc.bubuit.net/api/v1/health # External API access
|
||||
|
||||
Config: `/etc/nginx/sites-enabled/aitbc.bubuit.net`
|
||||
|
||||
| Route | Target | Type |
|
||||
|-------|--------|------|
|
||||
| `/` | static files (`/var/www/aitbc.bubuit.net/`) | try_files |
|
||||
| `/explorer/` | Vite SPA (`/var/www/aitbc.bubuit.net/explorer/`) | try_files |
|
||||
| `/marketplace/` | Vite SPA (`/var/www/aitbc.bubuit.net/marketplace/`) | try_files |
|
||||
| `/docs/` | static HTML (`/var/www/aitbc.bubuit.net/docs/`) | alias |
|
||||
| `/Exchange` | proxy → `127.0.0.1:3002` | proxy_pass |
|
||||
| `/exchange` | 301 → `/Exchange` | redirect |
|
||||
| `/api/` | proxy → `127.0.0.1:8000/` | proxy_pass |
|
||||
| `/api/explorer/` | proxy → `127.0.0.1:8000/v1/explorer/` | proxy_pass |
|
||||
| `/api/users/` | proxy → `127.0.0.1:8000/v1/users/` | proxy_pass |
|
||||
| `/api/trades/recent` | proxy → `127.0.0.1:8085` | proxy_pass |
|
||||
| `/api/orders/orderbook` | proxy → `127.0.0.1:8085` | proxy_pass |
|
||||
| `/admin/` | proxy → `127.0.0.1:8000/v1/admin/` | proxy_pass |
|
||||
| `/rpc/` | proxy → `127.0.0.1:9080` | proxy_pass |
|
||||
| `/wallet/` | proxy → `127.0.0.1:8002` | proxy_pass |
|
||||
| `/v1/` | proxy → `10.1.223.1:8090` (mock coordinator) | proxy_pass |
|
||||
| `/health` | 200 OK | direct |
|
||||
| `/Marketplace` | 301 → `/marketplace/` | redirect (legacy) |
|
||||
| `/BrowserWallet` | 301 → `/docs/browser-wallet.html` | redirect (legacy) |
|
||||
| Route | Target | Type | Status |
|
||||
|-------|--------|------|--------|
|
||||
| `/` | static files (`/var/www/aitbc.bubuit.net/`) | try_files | ✅ |
|
||||
| `/explorer/` | Vite SPA (`/var/www/aitbc.bubuit.net/explorer/`) | try_files | ✅ |
|
||||
| `/marketplace/` | Vite SPA (`/var/www/aitbc.bubuit.net/marketplace/`) | try_files | ✅ |
|
||||
| `/docs/` | static HTML (`/var/www/aitbc.bubuit.net/docs/`) | alias | ✅ |
|
||||
| `/api/` | proxy → `127.0.0.1:8000/` | proxy_pass | ✅ |
|
||||
| `/api/explorer/` | proxy → `127.0.0.1:8000/v1/explorer/` | proxy_pass | ✅ |
|
||||
| `/api/users/` | proxy → `127.0.0.1:8000/v1/users/` | proxy_pass | ✅ |
|
||||
| `/api/exchange/` | proxy → `127.0.0.1:8001/` | proxy_pass | ✅ |
|
||||
| `/api/trades/recent` | proxy → `127.0.0.1:8001/trades/recent` | proxy_pass | ✅ |
|
||||
| `/api/orders/orderbook` | proxy → `127.0.0.1:8001/orders/orderbook` | proxy_pass | ✅ |
|
||||
| `/admin/` | proxy → `127.0.0.1:8000/v1/admin/` | proxy_pass | ✅ |
|
||||
| `/rpc/` | proxy → `127.0.0.1:8003` | proxy_pass | ✅ |
|
||||
| `/app/` | proxy → `127.0.0.1:8016` | proxy_pass | ✅ |
|
||||
| `/api/gpu/` | proxy → `127.0.0.1:8010` | proxy_pass | ✅ (CPU-only) |
|
||||
| `/api/gpu-multimodal/` | proxy → `127.0.0.1:8011` | proxy_pass | ✅ (CPU-only) |
|
||||
| `/api/optimization/` | proxy → `127.0.0.1:8012` | proxy_pass | ✅ |
|
||||
| `/api/learning/` | proxy → `127.0.0.1:8013` | proxy_pass | ✅ |
|
||||
| `/api/marketplace-enhanced/` | proxy → `127.0.0.1:8014` | proxy_pass | ✅ |
|
||||
| `/api/openclaw/` | proxy → `127.0.0.1:8015` | proxy_pass | ✅ |
|
||||
| `/api/loadbalancer/` | proxy → `127.0.0.1:8017` | proxy_pass | ✅ |
|
||||
| `/health` | 200 OK | direct | ✅ |
|
||||
| `/Marketplace` | 301 → `/marketplace/` | redirect (legacy) | ✅ |
|
||||
| `/BrowserWallet` | 301 → `/docs/browser-wallet.html` | redirect (legacy) | ✅ |
|
||||
|
||||
**API Routing Fixed** (2026-02-23):
|
||||
**API Routing Updated** (2026-03-04):
|
||||
- Updated `/api/` proxy_pass from `http://127.0.0.1:8000/v1/` to `http://127.0.0.1:8000/`
|
||||
- Updated Exchange API routes to port 8001 (new port logic)
|
||||
- Updated RPC route to port 8003 (new port logic)
|
||||
- Added Enhanced Services routes (8010-8017)
|
||||
- Added Web UI route to port 8016
|
||||
- Added Geographic Load Balancer route to port 8017
|
||||
- Removed legacy routes (Exchange, wallet, mock coordinator)
|
||||
- External API access now working: `https://aitbc.bubuit.net/api/v1/health` → `{"status":"ok","env":"dev"}`
|
||||
- All GPU services marked as CPU-only mode
|
||||
|
||||
### Web Root (`/var/www/aitbc.bubuit.net/`)
|
||||
|
||||
@@ -332,26 +412,24 @@ Config: `/etc/nginx/sites-enabled/aitbc.bubuit.net`
|
||||
### Data Storage (container)
|
||||
|
||||
```
|
||||
/opt/coordinator-api/ # Coordinator application
|
||||
/opt/aitbc/apps/coordinator-api/ # Coordinator application
|
||||
├── src/coordinator.db # Main database
|
||||
└── .venv/ # Python environment
|
||||
|
||||
/opt/blockchain-node/ # Blockchain Node 1
|
||||
/opt/aitbc/apps/blockchain-node/ # Blockchain Node 1
|
||||
├── data/chain.db # Chain database
|
||||
└── .venv/ # Python environment
|
||||
|
||||
/opt/blockchain-node-2/ # Blockchain Node 2
|
||||
├── data/chain2.db # Chain database
|
||||
└── .venv/ # Python environment
|
||||
|
||||
/opt/exchange/ # Exchange API
|
||||
/opt/aitbc/apps/exchange/ # Exchange API
|
||||
├── data/ # Exchange data
|
||||
└── .venv/ # Python environment
|
||||
```
|
||||
|
||||
### Configuration (container)
|
||||
- Node 1: `/opt/blockchain-node/src/aitbc_chain/config.py`
|
||||
- Node 2: `/opt/blockchain-node-2/src/aitbc_chain/config.py`
|
||||
- Node 1: `/opt/aitbc/apps/blockchain-node/src/aitbc_chain/config.py`
|
||||
- Coordinator API: `/opt/aitbc/apps/coordinator-api/.env`
|
||||
- Exchange API: `/opt/aitbc/apps/exchange/.env`
|
||||
- Enhanced Services: Environment variables in respective service files
|
||||
|
||||
## Remote Site (ns3)
|
||||
|
||||
@@ -359,16 +437,19 @@ Config: `/etc/nginx/sites-enabled/aitbc.bubuit.net`
|
||||
- **IP**: 95.216.198.140
|
||||
- **Access**: `ssh ns3-root`
|
||||
- **Bridge**: incusbr0 `192.168.100.1/24`
|
||||
- **Port forwarding**: firehol (8000, 8081, 8082, 9080 → 192.168.100.10)
|
||||
- **Port forwarding**: firehol (8000, 8001, 8003, 8010-8017 → 192.168.100.10)
|
||||
- **Updated**: Port logic aligned with main aitbc server
|
||||
|
||||
### Container (ns3/aitbc)
|
||||
- **IP**: 192.168.100.10
|
||||
- **Domain**: aitbc.keisanki.net
|
||||
- **Access**: `ssh ns3-root` → `incus shell aitbc`
|
||||
- **Blockchain Node 3**: RPC on port 8082
|
||||
- **Blockchain Node 3**: RPC on port 8003 (updated port logic)
|
||||
- **GPU Access**: None (CPU-only mode)
|
||||
- **Miner Service**: Not needed
|
||||
|
||||
```bash
|
||||
curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC
|
||||
curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC (port 8003)
|
||||
```
|
||||
|
||||
## Cross-Site Synchronization
|
||||
@@ -377,8 +458,9 @@ curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC
|
||||
- **Method**: RPC-based polling every 10 seconds
|
||||
- **Features**: Transaction propagation, height detection, block import
|
||||
- **Endpoints**:
|
||||
- Local: https://aitbc.bubuit.net/rpc/ (Node 1, port 9080)
|
||||
- Remote: http://aitbc.keisanki.net/rpc/ (Node 3, port 8082)
|
||||
- Local: https://aitbc.bubuit.net/rpc/ (Node 1, port 8003)
|
||||
- Remote: http://aitbc.keisanki.net/rpc/ (Node 3, port 8003)
|
||||
- **Updated**: All nodes using new port logic (8003 for RPC)
|
||||
- **Consensus**: PoA with 2s block intervals
|
||||
- **P2P**: Not connected yet; nodes maintain independent chain state
|
||||
|
||||
@@ -414,15 +496,23 @@ ssh aitbc-cascade "systemctl restart coordinator-api"
|
||||
```bash
|
||||
# From at1 (via container)
|
||||
ssh aitbc-cascade "curl -s http://localhost:8000/v1/health"
|
||||
ssh aitbc-cascade "curl -s http://localhost:9080/rpc/head | jq .height"
|
||||
ssh aitbc-cascade "curl -s http://localhost:8003/rpc/head | jq .height"
|
||||
|
||||
# Test enhanced services
|
||||
ssh aitbc-cascade "curl -s http://localhost:8010/health" # Multimodal GPU (CPU-only)
|
||||
ssh aitbc-cascade "curl -s http://localhost:8017/health" # Geographic Load Balancer
|
||||
|
||||
# From internet (Python 3.13.5 upgraded services)
|
||||
curl -s https://aitbc.bubuit.net/health
|
||||
curl -s https://aitbc.bubuit.net/api/v1/health # ✅ Fixed API routing
|
||||
curl -s https://aitbc.bubuit.net/api/explorer/blocks
|
||||
|
||||
# Test enhanced services externally
|
||||
curl -s https://aitbc.bubuit.net/api/gpu/health
|
||||
curl -s https://aitbc.bubuit.net/api/loadbalancer/health
|
||||
|
||||
# Remote site
|
||||
ssh ns3-root "curl -s http://192.168.100.10:8082/rpc/head | jq .height"
|
||||
ssh ns3-root "curl -s http://192.168.100.10:8003/rpc/head | jq .height"
|
||||
|
||||
# Python version verification
|
||||
ssh aitbc-cascade "python3 --version" # Python 3.13.5
|
||||
@@ -432,8 +522,12 @@ ssh aitbc-cascade "python3 --version" # Python 3.13.5
|
||||
|
||||
```bash
|
||||
# Container systemd logs
|
||||
ssh aitbc-cascade "journalctl -u coordinator-api --no-pager -n 20"
|
||||
ssh aitbc-cascade "journalctl -u aitbc-blockchain-node-1 --no-pager -n 20"
|
||||
ssh aitbc-cascade "journalctl -u aitbc-coordinator-api --no-pager -n 20"
|
||||
ssh aitbc-cascade "journalctl -u aitbc-blockchain-node --no-pager -n 20"
|
||||
|
||||
# Enhanced services logs
|
||||
ssh aitbc-cascade "journalctl -u aitbc-multimodal-gpu --no-pager -n 20"
|
||||
ssh aitbc-cascade "journalctl -u aitbc-loadbalancer-geo --no-pager -n 20"
|
||||
|
||||
# Container nginx logs
|
||||
ssh aitbc-cascade "tail -20 /var/log/nginx/aitbc.bubuit.net.error.log"
|
||||
@@ -450,9 +544,11 @@ sudo tail -20 /var/log/nginx/error.log
|
||||
- HTTP → HTTPS redirect (Certbot managed)
|
||||
|
||||
### CORS
|
||||
- Coordinator API: localhost origins only (3000, 8080, 8000, 8011)
|
||||
- Exchange API: localhost origins only
|
||||
- Blockchain Node: localhost origins only
|
||||
- Coordinator API: localhost origins only (8000-8003, 8010-8017)
|
||||
- Exchange API: localhost origins only (8000-8003, 8010-8017)
|
||||
- Blockchain Node: localhost origins only (8000-8003, 8010-8017)
|
||||
- Enhanced Services: localhost origins only (8010-8017)
|
||||
- **Updated**: New port logic reflected in CORS policies
|
||||
|
||||
### Authentication
|
||||
- Coordinator API: `X-Api-Key` header required
|
||||
@@ -467,9 +563,15 @@ sudo tail -20 /var/log/nginx/error.log
|
||||
```bash
|
||||
# Coordinator API
|
||||
JWT_SECRET=<secret>
|
||||
DATABASE_URL=postgresql://user:pass@host/db
|
||||
DATABASE_URL=sqlite:///./aitbc_coordinator.db
|
||||
MINER_API_KEYS=["production_key_32_characters_long_minimum"]
|
||||
# Note: No miner service needed - configuration kept for compatibility
|
||||
|
||||
# Exchange API
|
||||
SESSION_SECRET=<secret>
|
||||
WALLET_ENCRYPTION_KEY=<key>
|
||||
|
||||
# Enhanced Services
|
||||
HOST=0.0.0.0 # For container access
|
||||
PORT=8010-8017 # Enhanced services port range
|
||||
```
|
||||
|
||||
577
docs/1_project/aitbc.md
Normal file
577
docs/1_project/aitbc.md
Normal file
@@ -0,0 +1,577 @@
|
||||
# AITBC Platform Deployment Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides comprehensive deployment instructions for the AITBC (AI Trading Blockchain Compute) platform, including infrastructure requirements, service configurations, and troubleshooting procedures. **Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
|
||||
|
||||
## System Requirements
|
||||
|
||||
### **Hardware Requirements**
|
||||
- **CPU**: 4+ cores recommended
|
||||
- **Memory**: 8GB+ RAM minimum, 16GB+ recommended
|
||||
- **Storage**: 50GB+ available space
|
||||
- **Network**: Stable internet connection with 100Mbps+ bandwidth
|
||||
- **GPU**: Not required (aitbc server has no GPU access)
|
||||
- **Mining**: No miner service needed
|
||||
|
||||
### **Software Requirements**
|
||||
- **Operating System**: Debian 13 Trixie (primary) or Ubuntu 22.04+ (alternative)
|
||||
- **Python**: 3.13.5+ (strictly enforced - platform requires 3.13+ features)
|
||||
- **Node.js**: 22+ (current tested: v22.22.x)
|
||||
- **Database**: SQLite (default) or PostgreSQL (production)
|
||||
|
||||
### **Network Requirements**
|
||||
- **Core Services Ports**: 8000-8003 (must be available)
|
||||
- Port 8000: Coordinator API
|
||||
- Port 8001: Exchange API
|
||||
- Port 8002: Blockchain Node (internal)
|
||||
- Port 8003: Blockchain RPC
|
||||
- **Enhanced Services Ports**: 8010-8017 (must be available)
|
||||
- Port 8010: Multimodal GPU Service
|
||||
- Port 8011: GPU Multimodal Service
|
||||
- Port 8012: Modality Optimization Service
|
||||
- Port 8013: Adaptive Learning Service
|
||||
- Port 8014: Marketplace Enhanced Service
|
||||
- Port 8015: OpenClaw Enhanced Service
|
||||
- Port 8016: Web UI Service
|
||||
- Port 8017: Geographic Load Balancer
|
||||
- **Firewall**: Managed by firehol on at1 host (container networking handled by incus)
|
||||
- **SSL/TLS**: Recommended for production deployments
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
AITBC Platform Architecture (Updated March 4, 2026)
|
||||
├── Core Services (8000-8003)
|
||||
│ ├── Coordinator API (Port 8000) ✅ PRODUCTION READY
|
||||
│ ├── Exchange API (Port 8001) ✅ PRODUCTION READY
|
||||
│ ├── Blockchain Node (Port 8002) ✅ PRODUCTION READY
|
||||
│ └── Blockchain RPC (Port 8003) ✅ PRODUCTION READY
|
||||
├── Enhanced Services (8010-8017)
|
||||
│ ├── Multimodal GPU (Port 8010) ✅ PRODUCTION READY (CPU-only mode)
|
||||
│ ├── GPU Multimodal (Port 8011) ✅ PRODUCTION READY (CPU-only mode)
|
||||
│ ├── Modality Optimization (Port 8012) ✅ PRODUCTION READY
|
||||
│ ├── Adaptive Learning (Port 8013) ✅ PRODUCTION READY
|
||||
│ ├── Marketplace Enhanced (Port 8014) ✅ PRODUCTION READY
|
||||
│ ├── OpenClaw Enhanced (Port 8015) ✅ PRODUCTION READY
|
||||
│ ├── Web UI (Port 8016) ✅ PRODUCTION READY
|
||||
│ └── Geographic Load Balancer (Port 8017) ✅ PRODUCTION READY
|
||||
└── Infrastructure
|
||||
├── Database (SQLite/PostgreSQL)
|
||||
├── Monitoring & Logging
|
||||
├── Security & Authentication
|
||||
└── Container Support (0.0.0.0 binding)
|
||||
```
|
||||
|
||||
## Deployment Steps
|
||||
|
||||
### **Phase 1: Environment Setup**
|
||||
|
||||
#### 1.1 System Preparation
|
||||
```bash
|
||||
# Update system packages
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Install required packages
|
||||
sudo apt install -y python3.13 python3.13-venv python3-pip nodejs npm nginx sqlite3
|
||||
|
||||
# Create aitbc user
|
||||
sudo useradd -m -s /bin/bash aitbc
|
||||
sudo usermod -aG sudo aitbc
|
||||
```
|
||||
|
||||
#### 1.2 Directory Structure
|
||||
```bash
|
||||
# Create AITBC directory structure (standardized)
|
||||
sudo mkdir -p /opt/aitbc/{apps,config,logs,scripts,backups}
|
||||
sudo chown -R aitbc:aitbc /opt/aitbc
|
||||
```
|
||||
|
||||
#### 1.3 Code Deployment
|
||||
```bash
|
||||
# Clone or copy AITBC codebase
|
||||
cd /opt/aitbc
|
||||
# Option 1: Git clone
|
||||
git clone https://github.com/oib/AITBC.git .
|
||||
# Option 2: Copy from existing installation
|
||||
# scp -r /path/to/aitbc/* aitbc@target:/opt/aitbc/
|
||||
|
||||
# Set permissions (standardized)
|
||||
sudo chown -R aitbc:aitbc /opt/aitbc
|
||||
sudo chmod -R 755 /opt/aitbc
|
||||
```
|
||||
|
||||
### **Phase 2: Service Configuration**
|
||||
|
||||
#### 2.1 Python Environment Setup
|
||||
```bash
|
||||
# Coordinator API Environment (Python 3.13+ required)
|
||||
cd /opt/aitbc/apps/coordinator-api
|
||||
python3.13 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install fastapi uvicorn sqlalchemy aiosqlite sqlmodel pydantic pydantic-settings httpx aiofiles python-jose passlib bcrypt prometheus-client slowapi websockets numpy
|
||||
|
||||
# Enhanced Services Environment (CPU-only mode)
|
||||
cd /opt/aitbc/apps/coordinator-api # Enhanced services use same environment
|
||||
source .venv/bin/activate
|
||||
pip install aiohttp asyncio # Additional dependencies for enhanced services
|
||||
# Note: GPU-related packages (CUDA, torch) not installed - no GPU access
|
||||
```
|
||||
|
||||
#### 2.2 Environment Configuration
|
||||
```bash
|
||||
# Coordinator API Environment (Production)
|
||||
cd /opt/aitbc/apps/coordinator-api
|
||||
cat > .env << 'EOF'
|
||||
MINER_API_KEYS=["production_key_32_characters_long_minimum"]
|
||||
DATABASE_URL=sqlite:///./aitbc_coordinator.db
|
||||
LOG_LEVEL=INFO
|
||||
ENVIRONMENT=production
|
||||
API_HOST=0.0.0.0
|
||||
API_PORT=8000
|
||||
WORKERS=4
|
||||
# Note: No miner service needed - configuration kept for compatibility
|
||||
EOF
|
||||
|
||||
# Set permissions
|
||||
chmod 600 .env
|
||||
chown aitbc:aitbc .env
|
||||
```
|
||||
|
||||
#### 2.3 Systemd Service Installation
|
||||
```bash
|
||||
# Copy service files (updated for new port logic)
|
||||
sudo cp -r /opt/aitbc/systemd/* /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Enable core services
|
||||
sudo systemctl enable aitbc-coordinator-api.service
|
||||
sudo systemctl enable aitbc-blockchain-node.service
|
||||
sudo systemctl enable aitbc-blockchain-rpc.service
|
||||
sudo systemctl enable aitbc-exchange-api.service
|
||||
sudo systemctl enable aitbc-explorer.service
|
||||
|
||||
# Enable enhanced services
|
||||
sudo systemctl enable aitbc-multimodal-gpu.service
|
||||
sudo systemctl enable aitbc-multimodal.service
|
||||
sudo systemctl enable aitbc-modality-optimization.service
|
||||
sudo systemctl enable aitbc-adaptive-learning.service
|
||||
sudo systemctl enable aitbc-marketplace-enhanced.service
|
||||
sudo systemctl enable aitbc-openclaw-enhanced.service
|
||||
sudo systemctl enable aitbc-web-ui.service
|
||||
sudo systemctl enable aitbc-loadbalancer-geo.service
|
||||
```
|
||||
|
||||
### **Phase 3: Service Deployment**
|
||||
|
||||
#### 3.1 Core Services Startup
|
||||
```bash
|
||||
# Start core services in order
|
||||
sudo systemctl start aitbc-blockchain-node.service
|
||||
sleep 5
|
||||
sudo systemctl start aitbc-blockchain-rpc.service
|
||||
sleep 3
|
||||
sudo systemctl start aitbc-coordinator-api.service
|
||||
sleep 3
|
||||
sudo systemctl start aitbc-exchange-api.service
|
||||
sleep 3
|
||||
sudo systemctl start aitbc-explorer.service
|
||||
```
|
||||
|
||||
#### 3.2 Enhanced Services Startup
|
||||
```bash
|
||||
# Start enhanced services (new port logic)
|
||||
sudo systemctl start aitbc-multimodal-gpu.service # Port 8010
|
||||
sudo systemctl start aitbc-multimodal.service # Port 8011
|
||||
sudo systemctl start aitbc-modality-optimization.service # Port 8012
|
||||
sudo systemctl start aitbc-adaptive-learning.service # Port 8013
|
||||
sudo systemctl start aitbc-marketplace-enhanced.service # Port 8014
|
||||
sudo systemctl start aitbc-openclaw-enhanced.service # Port 8015
|
||||
sudo systemctl start aitbc-web-ui.service # Port 8016
|
||||
sudo systemctl start aitbc-loadbalancer-geo.service # Port 8017
|
||||
```
|
||||
|
||||
#### 3.3 Service Verification
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl list-units --type=service --state=running | grep aitbc
|
||||
|
||||
# Test core endpoints
|
||||
curl -X GET "http://localhost:8000/v1/health" # Coordinator API
|
||||
curl -X GET "http://localhost:8001/" # Exchange API
|
||||
curl -X GET "http://localhost:8003/rpc/head" # Blockchain RPC
|
||||
|
||||
# Test enhanced endpoints (CPU-only mode)
|
||||
curl -X GET "http://localhost:8010/health" # Multimodal GPU (CPU mode)
|
||||
curl -X GET "http://localhost:8011/health" # GPU Multimodal (CPU mode)
|
||||
curl -X GET "http://localhost:8012/health" # Modality Optimization
|
||||
curl -X GET "http://localhost:8013/health" # Adaptive Learning
|
||||
curl -X GET "http://localhost:8014/health" # Marketplace Enhanced
|
||||
curl -X GET "http://localhost:8015/health" # OpenClaw Enhanced
|
||||
curl -X GET "http://localhost:8016/health" # Web UI
|
||||
curl -X GET "http://localhost:8017/health" # Geographic Load Balancer
|
||||
```
|
||||
|
||||
### **Phase 4: Production Configuration**
|
||||
|
||||
#### 4.1 Security Configuration
|
||||
```bash
|
||||
# Note: AITBC servers run in incus containers on at1 host
|
||||
# Firewall is managed by firehol on at1, not ufw in containers
|
||||
# Container networking is handled by incus with appropriate port forwarding
|
||||
|
||||
# Secure sensitive files
|
||||
chmod 600 /opt/aitbc/apps/coordinator-api/.env
|
||||
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
```
|
||||
|
||||
#### 4.2 Performance Optimization
|
||||
```bash
|
||||
# Database optimization
|
||||
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
|
||||
PRAGMA synchronous = NORMAL;
|
||||
PRAGMA cache_size = 10000;
|
||||
PRAGMA temp_store = MEMORY;
|
||||
EOF
|
||||
|
||||
# System limits
|
||||
echo "aitbc soft nofile 65536" | sudo tee -a /etc/security/limits.conf
|
||||
echo "aitbc hard nofile 65536" | sudo tee -a /etc/security/limits.conf
|
||||
|
||||
# Network optimization
|
||||
echo "net.core.somaxconn = 1024" | sudo tee -a /etc/sysctl.conf
|
||||
echo "net.ipv4.tcp_max_syn_backlog = 1024" | sudo tee -a /etc/sysctl.conf
|
||||
sudo sysctl -p
|
||||
```
|
||||
|
||||
#### 4.3 Monitoring Setup
|
||||
```bash
|
||||
# Create comprehensive monitoring script (updated for new port logic)
|
||||
cat > /opt/aitbc/scripts/monitor-services.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
echo "AITBC Service Monitor - $(date)"
|
||||
echo "================================"
|
||||
|
||||
# Service status
|
||||
echo "Service Status:"
|
||||
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"
|
||||
|
||||
# Core endpoint health
|
||||
echo -e "\nCore Services Health:"
|
||||
for port in 8000 8001 8003; do
|
||||
status=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$port/health" 2>/dev/null)
|
||||
if [ "$status" = "200" ]; then
|
||||
echo "Port $port: ✅ Healthy"
|
||||
else
|
||||
echo "Port $port: ❌ Unhealthy ($status)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Enhanced endpoint health
|
||||
echo -e "\nEnhanced Services Health:"
|
||||
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
|
||||
status=$(curl -s -o /dev/null -w "%{http_code}" "http://127.0.0.1:$port/health" 2>/dev/null)
|
||||
if [ "$status" = "200" ]; then
|
||||
echo "Port $port: ✅ Healthy"
|
||||
else
|
||||
echo "Port $port: ❌ Unhealthy ($status)"
|
||||
fi
|
||||
done
|
||||
|
||||
# System resources
|
||||
echo -e "\nSystem Resources:"
|
||||
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
|
||||
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
|
||||
echo "Disk: $(df -h / | tail -1 | awk '{print $3"/"$2}')"
|
||||
|
||||
# Port usage verification
|
||||
echo -e "\nPort Usage:"
|
||||
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)" | sort
|
||||
EOF
|
||||
|
||||
chmod +x /opt/aitbc/scripts/monitor-services.sh
|
||||
chown aitbc:aitbc /opt/aitbc/scripts/monitor-services.sh
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### **Common Issues**
|
||||
|
||||
#### Service Not Starting
|
||||
```bash
|
||||
# Check service logs
|
||||
sudo journalctl -u aitbc-coordinator-api.service -n 50
|
||||
|
||||
# Check Python environment (must be 3.13+)
|
||||
cd /opt/aitbc/apps/coordinator-api
|
||||
source .venv/bin/activate
|
||||
python --version # Should show 3.13.x
|
||||
|
||||
# Check permissions
|
||||
ls -la /opt/aitbc/apps/coordinator-api/
|
||||
```
|
||||
|
||||
#### Database Issues
|
||||
```bash
|
||||
# Check database file
|
||||
ls -la /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
|
||||
# Test database connection
|
||||
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db ".tables"
|
||||
|
||||
# Recreate database if corrupted
|
||||
mv /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db.backup
|
||||
```
|
||||
|
||||
#### Port Conflicts (New Port Logic)
|
||||
```bash
|
||||
# Check port usage (new port logic)
|
||||
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"
|
||||
|
||||
# Kill conflicting processes
|
||||
sudo fuser -k 8000/tcp # Core services
|
||||
sudo fuser -k 8010/tcp # Enhanced services
|
||||
|
||||
# Restart services
|
||||
sudo systemctl restart aitbc-coordinator-api.service
|
||||
```
|
||||
|
||||
#### Container Access Issues
|
||||
```bash
|
||||
# Test 0.0.0.0 binding (for container access)
|
||||
curl -s http://localhost:8017/health # Should work
|
||||
curl -s http://10.1.223.1:8017/health # Should work from containers
|
||||
|
||||
# Check service binding
|
||||
sudo netstat -tlnp | grep :8017 # Should show 0.0.0.0:8017
|
||||
```
|
||||
|
||||
#### Permission Issues
|
||||
```bash
|
||||
# Fix file ownership (standardized)
|
||||
sudo chown -R aitbc:aitbc /opt/aitbc
|
||||
|
||||
# Fix file permissions
|
||||
sudo chmod -R 755 /opt/aitbc
|
||||
chmod 600 /opt/aitbc/apps/coordinator-api/.env
|
||||
```
|
||||
|
||||
### **Performance Issues**
|
||||
|
||||
#### High Memory Usage
|
||||
```bash
|
||||
# Check memory usage
|
||||
free -h
|
||||
ps aux --sort=-%mem | head -10
|
||||
|
||||
# Optimize Python processes
|
||||
# Reduce worker count in service files
|
||||
# Implement database connection pooling
|
||||
```
|
||||
|
||||
#### High CPU Usage
|
||||
```bash
|
||||
# Check CPU usage
|
||||
top
|
||||
ps aux --sort=-%cpu | head -10
|
||||
|
||||
# Optimize database queries
|
||||
# Add database indexes
|
||||
# Implement caching
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### **Daily Tasks**
|
||||
```bash
|
||||
# Service health check (updated for new port logic)
|
||||
/opt/aitbc/scripts/monitor-services.sh
|
||||
|
||||
# Log rotation
|
||||
sudo logrotate -f /etc/logrotate.d/aitbc
|
||||
|
||||
# Backup database
|
||||
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db /opt/aitbc/backups/aitbc_coordinator_$(date +%Y%m%d).db
|
||||
```
|
||||
|
||||
### **Weekly Tasks**
|
||||
```bash
|
||||
# System updates
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Service restart
|
||||
sudo systemctl restart aitbc-*.service
|
||||
|
||||
# Performance review
|
||||
/opt/aitbc/scripts/monitor-services.sh > /opt/aitbc/logs/weekly_$(date +%Y%m%d).log
|
||||
```
|
||||
|
||||
### **Monthly Tasks**
|
||||
```bash
|
||||
# Security updates
|
||||
sudo apt update && sudo apt upgrade -y
|
||||
|
||||
# Database maintenance
|
||||
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db "VACUUM;"
|
||||
|
||||
# Log cleanup
|
||||
find /opt/aitbc/logs -name "*.log" -mtime +30 -delete
|
||||
```
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
### **Horizontal Scaling**
|
||||
- Load balancer configuration (Port 8017)
|
||||
- Multiple service instances
|
||||
- Database clustering
|
||||
- CDN implementation
|
||||
|
||||
### **Vertical Scaling**
|
||||
- Resource allocation increases
|
||||
- Performance optimization
|
||||
- Caching strategies
|
||||
- Database tuning
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### **Network Security**
|
||||
- Firewall configuration
|
||||
- SSL/TLS implementation
|
||||
- VPN access for management
|
||||
- Network segmentation
|
||||
|
||||
### **Application Security**
|
||||
- Environment variable protection
|
||||
- API rate limiting
|
||||
- Input validation
|
||||
- Regular security audits
|
||||
|
||||
### **Data Security**
|
||||
- Database encryption
|
||||
- Backup encryption
|
||||
- Access control
|
||||
- Audit logging
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### **Automated Backup Script**
|
||||
```bash
|
||||
cat > /opt/aitbc/scripts/backup.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/opt/aitbc/backups"
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Create backup directory
|
||||
mkdir -p $BACKUP_DIR
|
||||
|
||||
# Backup database
|
||||
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
|
||||
|
||||
# Backup configuration
|
||||
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/
|
||||
|
||||
# Backup scripts
|
||||
tar -czf $BACKUP_DIR/scripts_$DATE.tar.gz /opt/aitbc/scripts/
|
||||
|
||||
# Backup service configurations
|
||||
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
|
||||
|
||||
# Clean old backups (keep 7 days)
|
||||
find $BACKUP_DIR -name "*.db" -mtime +7 -delete
|
||||
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete
|
||||
|
||||
echo "Backup completed: $DATE"
|
||||
EOF
|
||||
|
||||
chmod +x /opt/aitbc/scripts/backup.sh
|
||||
chown aitbc:aitbc /opt/aitbc/scripts/backup.sh
|
||||
```
|
||||
|
||||
### **Recovery Procedures**
|
||||
```bash
|
||||
# Stop services
|
||||
sudo systemctl stop aitbc-*.service
|
||||
|
||||
# Restore database
|
||||
cp /opt/aitbc/backups/aitbc_coordinator_YYYYMMDD.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
|
||||
# Restore configuration
|
||||
tar -xzf /opt/aitbc/backups/config_YYYYMMDD.tar.gz -C /
|
||||
|
||||
# Restore service configurations
|
||||
tar -xzf /opt/aitbc/backups/services_YYYYMMDD.tar.gz -C /
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Start services
|
||||
sudo systemctl start aitbc-*.service
|
||||
```
|
||||
|
||||
## Monitoring and Alerting
|
||||
|
||||
### **Key Metrics**
|
||||
- Service uptime (all 12 services)
|
||||
- API response times
|
||||
- Database performance
|
||||
- System resource usage
|
||||
- Error rates
|
||||
|
||||
### **Alerting Thresholds**
|
||||
- Service downtime > 5 minutes
|
||||
- API response time > 1 second
|
||||
- CPU usage > 80%
|
||||
- Memory usage > 90%
|
||||
- Disk usage > 85%
|
||||
|
||||
## Production Deployment Checklist
|
||||
|
||||
### **✅ Pre-Deployment**
|
||||
- [ ] Python 3.13+ installed and verified
|
||||
- [ ] All required ports available (8000-8003, 8010-8017)
|
||||
- [ ] System requirements met
|
||||
- [ ] Dependencies installed
|
||||
- [ ] Network configuration verified
|
||||
|
||||
### **✅ Deployment**
|
||||
- [ ] Codebase copied to /opt/aitbc
|
||||
- [ ] Virtual environments created (Python 3.13+)
|
||||
- [ ] Dependencies installed
|
||||
- [ ] Environment variables configured
|
||||
- [ ] Service files installed (new port logic)
|
||||
- [ ] Services enabled and started
|
||||
|
||||
### **✅ Post-Deployment**
|
||||
- [ ] All 12 services running
|
||||
- [ ] Core API endpoints responding (8000-8003)
|
||||
- [ ] Enhanced services endpoints responding (8010-8017)
|
||||
- [ ] Database operational
|
||||
- [ ] Container access working (0.0.0.0 binding)
|
||||
- [ ] Monitoring working
|
||||
- [ ] Backup system active
|
||||
- [ ] Security configured
|
||||
|
||||
### **✅ Testing**
|
||||
- [ ] Health endpoints responding for all services
|
||||
- [ ] API functionality verified
|
||||
- [ ] Database operations working
|
||||
- [ ] External access via proxy working
|
||||
- [ ] SSL certificates valid
|
||||
- [ ] Performance acceptable
|
||||
- [ ] Container connectivity verified
|
||||
|
||||
## Documentation References
|
||||
|
||||
- [Service Configuration Guide](./service-configuration.md)
|
||||
- [Security Hardening Guide](./security-hardening.md)
|
||||
- [Performance Optimization Guide](./performance-optimization.md)
|
||||
- [Troubleshooting Guide](./troubleshooting.md)
|
||||
- [Enhanced Services Guide](./enhanced-services.md)
|
||||
- [Port Logic Implementation](./port-logic.md)
|
||||
|
||||
---
|
||||
|
||||
**Version**: 2.0 (Updated for new port logic)
|
||||
**Last Updated**: 2026-03-04
|
||||
**Maintainer**: AITBC Development Team
|
||||
**Status**: ✅ PRODUCTION READY
|
||||
**GPU Access**: None (CPU-only mode)
|
||||
**Miner Service**: Not needed
|
||||
584
docs/1_project/aitbc1.md
Normal file
584
docs/1_project/aitbc1.md
Normal file
@@ -0,0 +1,584 @@
|
||||
# AITBC1 Deployment Notes
|
||||
|
||||
## Overview
|
||||
|
||||
This document contains specific deployment notes and considerations for deploying the AITBC platform on the **aitbc1** server. These notes complement the general deployment guide with server-specific configurations and troubleshooting. **Updated for the new port logic implementation (8000-8003, 8010-8017) and production-ready codebase.**
|
||||
|
||||
## Server Specifications
|
||||
|
||||
### **aitbc1 Server Details**
|
||||
- **Hostname**: aitbc
|
||||
- **IP Address**: 10.1.223.1 (container IP)
|
||||
- **Operating System**: Debian 13 Trixie (primary development environment)
|
||||
- **Access Method**: SSH via aitbc-cascade proxy
|
||||
- **Web Root**: `/var/www/html/`
|
||||
- **Nginx Configuration**: Two-tier setup with SSL termination
|
||||
- **Container Support**: Incus containers with 0.0.0.0 binding for container access
|
||||
|
||||
### **Network Architecture**
|
||||
```
|
||||
Internet → aitbc-cascade (Proxy) → aitbc (Container)
|
||||
SSL Termination Application Server
|
||||
Port 443/80 Port 8000-8003, 8010-8017
|
||||
```
|
||||
|
||||
## Pre-Deployment Checklist
|
||||
|
||||
### **✅ Server Preparation**
|
||||
- [ ] SSH access confirmed via aitbc-cascade
|
||||
- [ ] System packages updated
|
||||
- [ ] aitbc user created with sudo access
|
||||
- [ ] Directory structure created
|
||||
- [ ] Firewall rules configured
|
||||
- [ ] Python 3.13+ installed and verified
|
||||
- [ ] Container networking configured
|
||||
- [ ] GPU access confirmed as not available
|
||||
- [ ] Miner service requirements confirmed as not needed
|
||||
|
||||
### **✅ Network Configuration**
|
||||
- [ ] Port forwarding configured on aitbc-cascade
|
||||
- [ ] SSL certificates installed on proxy
|
||||
- [ ] DNS records configured
|
||||
- [ ] Load balancer rules set
|
||||
- [ ] Container access configured (0.0.0.0 binding)
|
||||
|
||||
### **✅ Storage Requirements**
|
||||
- [ ] Minimum 50GB free space available
|
||||
- [ ] Backup storage allocated
|
||||
- [ ] Log rotation configured
|
||||
- [ ] Database storage planned
|
||||
|
||||
## Deployment Issues & Solutions
|
||||
|
||||
### **🔥 Issue 1: Python Version Compatibility**
|
||||
|
||||
**Problem**: aitbc1 may have Python 3.10 instead of required 3.13+
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check current Python version
|
||||
python3 --version
|
||||
|
||||
# Install Python 3.13 if not available
|
||||
sudo apt update
|
||||
sudo apt install -y python3.13 python3.13-venv python3.13-dev
|
||||
|
||||
# Update alternatives
|
||||
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.13 1
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
python3 --version # Should show 3.13.x
|
||||
```
|
||||
|
||||
### **🔥 Issue 1b: Node.js Version Compatibility**
|
||||
|
||||
**Current Status**: Node.js v22.22.x (tested and compatible)
|
||||
|
||||
**Note**: Current Node.js version v22.22.x meets the minimum requirement of 22.0.0 and is fully compatible with AITBC platform.
|
||||
|
||||
### **🔥 Issue 1c: Operating System Compatibility**
|
||||
|
||||
**Current Status**: Debian 13 Trixie (development environment)
|
||||
|
||||
**Note**: Development environment is running Debian 13 Trixie, which is newer than the minimum requirement of Debian 11+ and fully supported for AITBC development. This is the primary development environment for the AITBC platform.
|
||||
|
||||
### **🔥 Issue 2: Port Conflicts with Existing Services**
|
||||
|
||||
**Problem**: Ports 8000-8008 may be in use by existing applications
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check port usage (new port logic)
|
||||
sudo netstat -tlnp | grep -E ":(8000|8001|8003|8010|8011|8012|8013|8014|8015|8016|8017)"
|
||||
|
||||
# Kill conflicting processes if needed
|
||||
sudo fuser -k 8000/tcp # Core services
|
||||
sudo fuser -k 8010/tcp # Enhanced services
|
||||
|
||||
# Alternative: Use different ports in service files
|
||||
# Edit /etc/systemd/system/aitbc-*.service files
|
||||
# Change --port 8000 to --port 9000, etc.
|
||||
```
|
||||
|
||||
**Port Mapping for aitbc1 (Updated)**:
|
||||
```
|
||||
Core Services:
|
||||
- Coordinator API: 8000 ✅
|
||||
- Exchange API: 8001 ✅
|
||||
- Blockchain RPC: 8003 ✅
|
||||
|
||||
Enhanced Services:
|
||||
- Multimodal GPU: 8010 ✅ (CPU-only mode)
|
||||
- GPU Multimodal: 8011 ✅ (CPU-only mode)
|
||||
- Modality Optimization: 8012 ✅
|
||||
- Adaptive Learning: 8013 ✅
|
||||
- Marketplace Enhanced: 8014 ✅
|
||||
- OpenClaw Enhanced: 8015 ✅
|
||||
- Web UI: 8016 ✅
|
||||
- Geographic Load Balancer: 8017 ✅
|
||||
```
|
||||
|
||||
### **🔥 Issue 3: Database Permission Issues**
|
||||
|
||||
**Problem**: SQLite database file permissions preventing access
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Fix database ownership (standardized)
|
||||
sudo chown aitbc:aitbc /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
|
||||
# Fix database permissions
|
||||
sudo chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
|
||||
# Ensure directory permissions
|
||||
sudo chmod 755 /opt/aitbc/apps/coordinator-api/
|
||||
```
|
||||
|
||||
### **🔥 Issue 4: Systemd Service Failures**
|
||||
|
||||
**Problem**: Services failing to start due to missing dependencies
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl status aitbc-coordinator-api.service
|
||||
|
||||
# Check service logs
|
||||
sudo journalctl -u aitbc-coordinator-api.service -n 50
|
||||
|
||||
# Common fixes:
|
||||
# 1. Install missing Python packages
|
||||
cd /opt/aitbc/apps/coordinator-api
|
||||
source .venv/bin/activate
|
||||
pip install missing-package
|
||||
|
||||
# 2. Fix environment variables
|
||||
echo "ENVIRONMENT=production" >> .env
|
||||
|
||||
# 3. Fix working directory
|
||||
sudo systemctl edit aitbc-coordinator-api.service
|
||||
# Add: WorkingDirectory=/opt/aitbc/apps/coordinator-api
|
||||
```
|
||||
|
||||
### **🔥 Issue 5: Nginx Proxy Configuration**
|
||||
|
||||
**Problem**: Requests not properly forwarded from aitbc-cascade to aitbc
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# On aitbc-cascade, check proxy configuration
|
||||
cat /etc/nginx/sites-available/aitbc-proxy.conf
|
||||
|
||||
# Ensure upstream configuration includes aitbc
|
||||
upstream aitbc_backend {
|
||||
server 10.1.223.1:8000; # Coordinator API
|
||||
server 10.1.223.1:8001; # Exchange API
|
||||
server 10.1.223.1:8003; # Blockchain RPC
|
||||
# Add enhanced services ports
|
||||
server 10.1.223.1:8010; # Multimodal GPU
|
||||
server 10.1.223.1:8011; # GPU Multimodal
|
||||
server 10.1.223.1:8012; # Modality Optimization
|
||||
server 10.1.223.1:8013; # Adaptive Learning
|
||||
server 10.1.223.1:8014; # Marketplace Enhanced
|
||||
server 10.1.223.1:8015; # OpenClaw Enhanced
|
||||
server 10.1.223.1:8016; # Web UI
|
||||
server 10.1.223.1:8017; # Geographic Load Balancer
|
||||
}
|
||||
|
||||
# Reload nginx configuration
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
### **🔥 Issue 6: SSL Certificate Issues**
|
||||
|
||||
**Problem**: SSL certificates not properly configured for aitbc domain
|
||||
|
||||
**Solution**:
|
||||
```bash
|
||||
# On aitbc-cascade, check certificate status
|
||||
sudo certbot certificates
|
||||
|
||||
# Renew or obtain certificate
|
||||
sudo certbot --nginx -d aitbc.bubuit.net
|
||||
|
||||
# Test SSL configuration
|
||||
curl -I https://aitbc.bubuit.net
|
||||
```
|
||||
|
||||
## aitbc-Specific Configurations
|
||||
|
||||
### **Environment Variables**
|
||||
```bash
|
||||
# /opt/aitbc/apps/coordinator-api/.env
|
||||
MINER_API_KEYS=["aitbc_production_key_32_characters_long"]
|
||||
DATABASE_URL=sqlite:///./aitbc_coordinator.db
|
||||
LOG_LEVEL=INFO
|
||||
ENVIRONMENT=production
|
||||
API_HOST=0.0.0.0
|
||||
API_PORT=8000
|
||||
WORKERS=2 # Reduced for aitbc resources
|
||||
SERVER_NAME=aitbc.bubuit.net
|
||||
# Note: No miner service needed - configuration kept for compatibility
|
||||
```
|
||||
|
||||
### **Service Configuration Adjustments**
|
||||
```bash
|
||||
# aitbc-coordinator-api.service adjustments
|
||||
# Edit: /etc/systemd/system/aitbc-coordinator-api.service
|
||||
|
||||
[Service]
|
||||
User=aitbc
|
||||
Group=aitbc
|
||||
WorkingDirectory=/opt/aitbc/apps/coordinator-api
|
||||
Environment=PYTHONPATH=src
|
||||
EnvironmentFile=/opt/aitbc/apps/coordinator-api/.env
|
||||
ExecStart=/opt/aitbc/apps/coordinator-api/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### **Resource Limits for aitbc**
|
||||
```bash
|
||||
# /etc/systemd/system/aitbc-coordinator-api.service
|
||||
[Service]
|
||||
# Add resource limits
|
||||
MemoryMax=2G
|
||||
CPUQuota=200%
|
||||
TasksMax=100
|
||||
```
|
||||
|
||||
## Performance Optimization for aitbc
|
||||
|
||||
### **Database Optimization**
|
||||
```bash
|
||||
# SQLite optimization for aitbc
|
||||
sqlite3 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db << 'EOF'
|
||||
PRAGMA synchronous = NORMAL;
|
||||
PRAGMA cache_size = 5000; # Reduced for aitbc
|
||||
PRAGMA temp_store = MEMORY;
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA busy_timeout = 30000;
|
||||
EOF
|
||||
```
|
||||
|
||||
### **System Resource Limits**
|
||||
```bash
|
||||
# /etc/security/limits.conf additions for aitbc
|
||||
aitbc soft nofile 4096
|
||||
aitbc hard nofile 4096
|
||||
aitbc soft nproc 512
|
||||
aitbc hard nproc 512
|
||||
```
|
||||
|
||||
### **Network Optimization**
|
||||
```bash
|
||||
# /etc/sysctl.conf additions for aitbc
|
||||
net.core.somaxconn = 512
|
||||
net.ipv4.tcp_max_syn_backlog = 512
|
||||
net.ipv4.ip_local_port_range = 1024 65535
|
||||
```
|
||||
|
||||
## Monitoring Setup for aitbc
|
||||
|
||||
### **Custom Monitoring Script**
|
||||
```bash
|
||||
# /opt/aitbc/scripts/monitor-aitbc.sh
|
||||
#!/bin/bash
|
||||
echo "AITBC Monitor - $(date)"
|
||||
echo "========================"
|
||||
|
||||
# Service status
|
||||
echo "Service Status:"
|
||||
systemctl list-units --type=service --state=running | grep aitbc | wc -l | xargs echo "Running services:"
|
||||
|
||||
# Resource usage
|
||||
echo -e "\nResource Usage:"
|
||||
echo "Memory: $(free -h | grep Mem | awk '{print $3"/"$2}')"
|
||||
echo "CPU: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)%"
|
||||
echo "Disk: $(df -h / | tail -1 | awk '{print $5}')"
|
||||
|
||||
# Network connectivity
|
||||
echo -e "\nNetwork Test:"
|
||||
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/v1/health" | grep -q "200" && echo "Coordinator API: ✅" || echo "Coordinator API: ❌"
|
||||
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8001/" | grep -q "200" && echo "Exchange API: ✅" || echo "Exchange API: ❌"
|
||||
curl -s -o /dev/null -w "%{http_code}" "http://localhost:8003/rpc/head" | grep -q "200" && echo "Blockchain RPC: ✅" || echo "Blockchain RPC: ❌"
|
||||
|
||||
# Enhanced services health (CPU-only mode)
|
||||
echo -e "\nEnhanced Services:"
|
||||
for port in 8010 8011 8012 8013 8014 8015 8016 8017; do
|
||||
status=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:$port/health" 2>/dev/null)
|
||||
if [ "$status" = "200" ]; then
|
||||
service_name="Port $port"
|
||||
case $port in
|
||||
8010|8011) service_name="$service_name (CPU-only)" ;;
|
||||
esac
|
||||
echo "$service_name: ✅"
|
||||
else
|
||||
echo "Port $port: ❌ ($status)"
|
||||
fi
|
||||
done
|
||||
|
||||
# Database status
|
||||
echo -e "\nDatabase Status:"
|
||||
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
|
||||
size=$(du -h /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db | cut -f1)
|
||||
echo "Database: ✅ ($size)"
|
||||
else
|
||||
echo "Database: ❌ (Missing)"
|
||||
fi
|
||||
|
||||
# Container access test
|
||||
echo -e "\nContainer Access Test:"
|
||||
curl -s -o /dev/null -w "%{http_code}" "http://10.1.223.1:8017/health" | grep -q "200" && echo "Container Access: ✅" || echo "Container Access: ❌"
|
||||
EOF
|
||||
|
||||
chmod +x /opt/aitbc/scripts/monitor-aitbc.sh
|
||||
```
|
||||
|
||||
## Backup Strategy for aitbc
|
||||
|
||||
### **Automated Backup Script**
|
||||
```bash
|
||||
# /opt/aitbc/scripts/backup-aitbc.sh
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/opt/aitbc/backups"
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
RETENTION_DAYS=7
|
||||
|
||||
# Create backup directory
|
||||
mkdir -p $BACKUP_DIR
|
||||
|
||||
# Backup database
|
||||
if [ -f "/opt/aitbc/apps/coordinator-api/aitbc_coordinator.db" ]; then
|
||||
cp /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db $BACKUP_DIR/aitbc_coordinator_$DATE.db
|
||||
echo "Database backed up: aitbc_coordinator_$DATE.db"
|
||||
fi
|
||||
|
||||
# Backup configuration
|
||||
tar -czf $BACKUP_DIR/config_$DATE.tar.gz /opt/aitbc/config/ 2>/dev/null
|
||||
echo "Configuration backed up: config_$DATE.tar.gz"
|
||||
|
||||
# Backup service files
|
||||
tar -czf $BACKUP_DIR/services_$DATE.tar.gz /etc/systemd/system/aitbc-*.service
|
||||
echo "Service files backed up: services_$DATE.tar.gz"
|
||||
|
||||
# Backup enhanced services scripts
|
||||
tar -czf $BACKUP_DIR/enhanced-services_$DATE.tar.gz /opt/aitbc/scripts/*service*.py 2>/dev/null
|
||||
echo "Enhanced services backed up: enhanced-services_$DATE.tar.gz"
|
||||
|
||||
# Clean old backups
|
||||
find $BACKUP_DIR -name "*.db" -mtime +$RETENTION_DAYS -delete
|
||||
find $BACKUP_DIR -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete
|
||||
|
||||
echo "Backup completed: $DATE"
|
||||
echo "Retention period: $RETENTION_DAYS days"
|
||||
EOF
|
||||
|
||||
chmod +x /opt/aitbc/scripts/backup-aitbc.sh
|
||||
```
|
||||
|
||||
## Troubleshooting aitbc Specific Issues
|
||||
|
||||
### **Issue: Services Not Starting After Reboot**
|
||||
```bash
|
||||
# Check if services are enabled
|
||||
systemctl list-unit-files | grep aitbc
|
||||
|
||||
# Enable services for auto-start
|
||||
sudo systemctl enable aitbc-coordinator-api.service
|
||||
sudo systemctl enable aitbc-blockchain-node.service
|
||||
sudo systemctl enable aitbc-blockchain-rpc.service
|
||||
sudo systemctl enable aitbc-exchange-api.service
|
||||
|
||||
# Enable enhanced services
|
||||
sudo systemctl enable aitbc-multimodal-gpu.service
|
||||
sudo systemctl enable aitbc-multimodal.service
|
||||
sudo systemctl enable aitbc-modality-optimization.service
|
||||
sudo systemctl enable aitbc-adaptive-learning.service
|
||||
sudo systemctl enable aitbc-marketplace-enhanced.service
|
||||
sudo systemctl enable aitbc-openclaw-enhanced.service
|
||||
sudo systemctl enable aitbc-web-ui.service
|
||||
sudo systemctl enable aitbc-loadbalancer-geo.service
|
||||
```
|
||||
|
||||
### **Issue: High Memory Usage**
|
||||
```bash
|
||||
# Check memory usage
|
||||
free -h
|
||||
ps aux --sort=-%mem | head -10
|
||||
|
||||
# Reduce worker count in service files
|
||||
# Edit ExecStart line: --workers 1 instead of --workers 4
|
||||
```
|
||||
|
||||
### **Issue: Database Locking**
|
||||
```bash
|
||||
# Check for database locks
|
||||
sudo lsof /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
|
||||
# Restart services to release locks
|
||||
sudo systemctl restart aitbc-coordinator-api.service
|
||||
```
|
||||
|
||||
### **Issue: Network Connectivity**
|
||||
```bash
|
||||
# Test local connectivity
|
||||
curl -X GET "http://localhost:8000/v1/health"
|
||||
|
||||
# Test external connectivity via proxy
|
||||
curl -X GET "http://aitbc.bubuit.net/health"
|
||||
|
||||
# Check proxy configuration
|
||||
ssh aitbc-cascade "cat /etc/nginx/sites-available/aitbc-proxy.conf"
|
||||
```
|
||||
|
||||
### **Issue: Container Access Problems**
|
||||
```bash
|
||||
# Test 0.0.0.0 binding
|
||||
curl -s http://localhost:8017/health # Should work
|
||||
curl -s http://10.1.223.1:8017/health # Should work from containers
|
||||
|
||||
# Check service binding
|
||||
sudo netstat -tlnp | grep :8017 # Should show 0.0.0.0:8017
|
||||
|
||||
# Test from other containers
|
||||
# From another container: curl http://aitbc:8017/health
|
||||
```
|
||||
|
||||
## Security Considerations for aitbc
|
||||
|
||||
### **Firewall Configuration**
|
||||
```bash
|
||||
# Configure UFW on aitbc (if not using firehol)
|
||||
sudo ufw allow ssh
|
||||
sudo ufw allow 8000/tcp
|
||||
sudo ufw allow 8001/tcp
|
||||
sudo ufw allow 8003/tcp
|
||||
sudo ufw allow 8010/tcp
|
||||
sudo ufw allow 8011/tcp
|
||||
sudo ufw allow 8012/tcp
|
||||
sudo ufw allow 8013/tcp
|
||||
sudo ufw allow 8014/tcp
|
||||
sudo ufw allow 8015/tcp
|
||||
sudo ufw allow 8016/tcp
|
||||
sudo ufw allow 8017/tcp
|
||||
sudo ufw --force enable
|
||||
```
|
||||
|
||||
### **File Permissions**
|
||||
```bash
|
||||
# Secure sensitive files
|
||||
chmod 600 /opt/aitbc/apps/coordinator-api/.env
|
||||
chmod 600 /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
chmod 755 /opt/aitbc/apps/coordinator-api/
|
||||
```
|
||||
|
||||
### **Access Control**
|
||||
```bash
|
||||
# Restrict SSH access to specific users
|
||||
echo "AllowUsers aitbc" | sudo tee -a /etc/ssh/sshd_config
|
||||
sudo systemctl restart ssh
|
||||
```
|
||||
|
||||
## Deployment Validation Checklist
|
||||
|
||||
### **✅ Pre-Deployment**
|
||||
- [ ] Server access confirmed
|
||||
- [ ] System requirements met
|
||||
- [ ] Python 3.13+ installed and verified
|
||||
- [ ] Dependencies installed
|
||||
- [ ] Network configuration verified
|
||||
- [ ] Container networking configured
|
||||
- [ ] GPU access confirmed as not available
|
||||
- [ ] Miner service requirements confirmed as not needed
|
||||
|
||||
### **✅ Deployment**
|
||||
- [ ] Codebase copied to /opt/aitbc
|
||||
- [ ] Virtual environments created (Python 3.13+)
|
||||
- [ ] Dependencies installed
|
||||
- [ ] Environment variables configured
|
||||
- [ ] Service files installed (new port logic)
|
||||
- [ ] Services enabled and started
|
||||
|
||||
### **✅ Post-Deployment**
|
||||
- [ ] All 12 services running
|
||||
- [ ] Core API endpoints responding (8000-8003)
|
||||
- [ ] Enhanced services endpoints responding (8010-8017)
|
||||
- [ ] Database operational
|
||||
- [ ] Container access working (0.0.0.0 binding)
|
||||
- [ ] Monitoring working
|
||||
- [ ] Backup system active
|
||||
- [ ] Security configured
|
||||
- [ ] GPU services confirmed running in CPU-only mode
|
||||
- [ ] Miner service confirmed as not needed
|
||||
|
||||
### **✅ Testing**
|
||||
- [ ] Health endpoints responding for all services
|
||||
- [ ] API functionality verified
|
||||
- [ ] Database operations working
|
||||
- [ ] External access via proxy working
|
||||
- [ ] SSL certificates valid
|
||||
- [ ] Performance acceptable
|
||||
- [ ] Container connectivity verified
|
||||
- [ ] Geographic load balancer accessible from containers
|
||||
- [ ] GPU services confirmed operating in CPU-only mode
|
||||
- [ ] No miner service requirements confirmed
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### **Service Rollback**
|
||||
```bash
|
||||
# Stop all services
|
||||
sudo systemctl stop aitbc-*.service
|
||||
|
||||
# Restore previous configuration
|
||||
sudo cp /etc/systemd/system/aitbc-*.service.backup /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Restore database
|
||||
cp /opt/aitbc/backups/aitbc_coordinator_PREV_DEPLOY.db /opt/aitbc/apps/coordinator-api/aitbc_coordinator.db
|
||||
|
||||
# Start services
|
||||
sudo systemctl start aitbc-*.service
|
||||
```
|
||||
|
||||
### **Full System Rollback**
|
||||
```bash
|
||||
# Restore from backup
|
||||
cd /opt/aitbc
|
||||
tar -xzf /opt/aitbc/backups/full_backup_YYYYMMDD.tar.gz
|
||||
|
||||
# Restart services
|
||||
sudo systemctl restart aitbc-*.service
|
||||
```
|
||||
|
||||
## Contact Information
|
||||
|
||||
### **Support Contacts**
|
||||
- **Primary Admin**: aitbc-admin
|
||||
- **Network Admin**: aitbc-network
|
||||
- **Security Team**: aitbc-security
|
||||
|
||||
### **Emergency Procedures**
|
||||
1. Check service status: `systemctl status aitbc-*`
|
||||
2. Review logs: `journalctl -u aitbc-coordinator-api.service`
|
||||
3. Run monitoring: `/opt/aitbc/scripts/monitor-aitbc.sh`
|
||||
4. Check container access: `curl http://10.1.223.1:8017/health`
|
||||
5. Verify GPU services running in CPU-only mode
|
||||
6. Confirm no miner service is needed
|
||||
7. Contact support if issues persist
|
||||
|
||||
---
|
||||
|
||||
**Server**: aitbc (Container)
|
||||
**Environment**: Production
|
||||
**GPU Access**: None (CPU-only mode)
|
||||
**Miner Service**: Not needed
|
||||
**Last Updated**: 2026-03-04
|
||||
**Maintainer**: AITBC Operations Team
|
||||
**Status**: ✅ PRODUCTION READY
|
||||
Reference in New Issue
Block a user