759 lines
34 KiB
Markdown
759 lines
34 KiB
Markdown
# AITBC Infrastructure Documentation
|
||
|
||
> Last updated: 2026-03-10 (Updated nginx configuration with new port logic implementation)
|
||
|
||
## Overview
|
||
|
||
Two-tier architecture: **incus host (at1)** runs the reverse proxy with SSL termination, forwarding all `aitbc.bubuit.net` traffic to the **aitbc container** which runs nginx + all services. **Updated for port logic 8000+ implementation with unified numbering scheme and production-ready codebase.**
|
||
|
||
```
|
||
Internet → aitbc.bubuit.net (HTTPS :443)
|
||
│
|
||
▼
|
||
┌──────────────────────────────────────────────┐
|
||
│ Incus Host (at1 / localhost) │
|
||
│ Nginx reverse proxy (:443 SSL → :80) │
|
||
│ Config: /etc/nginx/sites-available/ │
|
||
│ aitbc-proxy.conf │
|
||
│ │
|
||
│ ┌────────────────────────────────────────┐ │
|
||
│ │ Container: aitbc (10.1.223.1) │ │
|
||
│ │ Access: ssh aitbc-cascade │ │
|
||
│ │ OS: Debian 13 Trixie │ │
|
||
│ │ Node.js: 24+ │ │
|
||
│ │ Python: 3.13.5+ │ │
|
||
│ │ GPU Access: None (CPU-only mode) │ │
|
||
│ │ Miner Service: Not needed │ │
|
||
│ │ │ │
|
||
│ │ Nginx (:80) → routes to services: │ │
|
||
│ │ / → static website │ │
|
||
│ │ /api/ → :8000 (coordinator)│ │
|
||
│ │ /exchange/ → :8001 (exchange) │ │
|
||
│ │ /rpc/ → :8006 (blockchain) │ │
|
||
│ │ /wallet/ → :8000 (wallet) │ │
|
||
│ │ /health → :8000 (health) │ │
|
||
│ │ /gpu/multimodal/ → :8010 │ │
|
||
│ │ /gpu/service/ → :8011 │ │
|
||
│ │ /optimization/ → :8012 │ │
|
||
│ │ /learning/ → :8013 │ │
|
||
│ │ /marketplace/enhanced/ → :8014 │ │
|
||
│ │ /openclaw/ → :8015 │ │
|
||
│ │ /explorer/ → :8016 │ │
|
||
│ │ /balancer/ → :8017 │ │
|
||
│ │ │ │
|
||
│ │ Config: /etc/nginx/sites-enabled/ │ │
|
||
│ │ aitbc.bubuit.net │ │
|
||
│ └────────────────────────────────────────┘ │
|
||
└──────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Port Logic Implementation (Updated March 10, 2026)
|
||
|
||
### **Core Services (8000-8001) - AT1 STANDARD REFERENCE**
|
||
- **Port 8000**: Coordinator API ✅ PRODUCTION READY
|
||
- **Port 8001**: Exchange API ✅ PRODUCTION READY (127.0.0.1 binding)
|
||
|
||
### **Blockchain Services (8005-8006) - AT1 STANDARD REFERENCE**
|
||
- **Port 8005**: Primary Blockchain Node ✅ PRODUCTION READY (aitbc-blockchain-node.service)
|
||
- **Port 8006**: Primary Blockchain RPC ✅ PRODUCTION READY (aitbc-blockchain-rpc.service)
|
||
|
||
### **Enhanced Services (8010-8017) - CPU-ONLY MODE**
|
||
- **Port 8010**: Multimodal GPU Service ✅ PRODUCTION READY (CPU-only mode)
|
||
- **Port 8011**: GPU Multimodal Service ✅ PRODUCTION READY (CPU-only mode)
|
||
- **Port 8012**: Modality Optimization Service ✅ PRODUCTION READY
|
||
- **Port 8013**: Adaptive Learning Service ✅ PRODUCTION READY
|
||
- **Port 8014**: Marketplace Enhanced Service ✅ PRODUCTION READY
|
||
- **Port 8015**: OpenClaw Enhanced Service ✅ PRODUCTION READY
|
||
- **Port 8016**: Blockchain Explorer Service ✅ PRODUCTION READY
|
||
- **Port 8017**: Geographic Load Balancer ✅ PRODUCTION READY
|
||
|
||
### **Mock & Test Services (8020-8029)**
|
||
- **Port 8020**: Mock Coordinator API ✅ TESTING READY
|
||
- **Port 8021**: Coordinator API (dev) ✅ TESTING READY
|
||
- **Port 8022**: Test Blockchain Node (localhost) ✅ TESTING READY
|
||
- **Port 8023**: Mock Exchange API ✅ TESTING READY
|
||
- **Port 8024**: Mock Blockchain RPC ✅ TESTING READY
|
||
- **Port 8025**: Development Blockchain Node ✅ TESTING READY (aitbc-blockchain-node-dev.service)
|
||
- **Port 8026**: Development Blockchain RPC ✅ TESTING READY (aitbc-blockchain-rpc-dev.service)
|
||
- **Port 8027**: Load Testing Endpoint ✅ TESTING READY
|
||
- **Port 8028**: Integration Test API ✅ TESTING READY
|
||
- **Port 8029**: Performance Monitor ✅ TESTING READY
|
||
|
||
### **Container Services (8080-8089) - LEGACY**
|
||
- **Port 8080**: Container Coordinator API (aitbc) ⚠️ LEGACY - Use port 8000-8003 range
|
||
- **Port 8081**: Container Blockchain Node 1 ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8082**: Container Exchange API ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8083**: Container Wallet Daemon ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8084**: Container Blockchain Node 2 ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8085**: Container Explorer UI ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8086**: Container Marketplace ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8087**: Container Miner Dashboard ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8088**: Container Load Balancer ⚠️ LEGACY - Use port 8010+ range
|
||
- **Port 8089**: Container Debug API ⚠️ LEGACY - Use port 8010+ range
|
||
|
||
### **Legacy Ports (Decommissioned)**
|
||
- **Port 8003**: Previously Primary Blockchain RPC - Decommissioned (moved to port 8006)
|
||
- **Port 8090**: No longer used by AITBC
|
||
- **Port 9080**: Successfully decommissioned
|
||
- **Port 8009**: No longer in use
|
||
|
||
## Incus Host (at1)
|
||
|
||
### Host Details
|
||
- **Hostname**: `at1` (primary development workstation)
|
||
- **Environment**: Windsurf development environment
|
||
- **OS**: Debian 13 Trixie (development environment)
|
||
- **Node.js**: 24+ (current tested: v24.14.x)
|
||
- **Python**: 3.13.5+ (minimum requirement, strictly enforced)
|
||
- **GPU Access**: **Primary GPU access location** - all GPU workloads must run on at1
|
||
- **Architecture**: x86_64 Linux with CUDA GPU support
|
||
|
||
### Services (Host)
|
||
|
||
| Service | Port | Process | Python Version | Purpose | Status |
|
||
|---------|------|---------|----------------|---------|--------|
|
||
| Coordinator API | 8000 | python3 | 3.13.5+ | Production coordinator API | systemd: aitbc-coordinator-api.service |
|
||
| Mock Coordinator | 8020 | python3 | 3.13.5+ | Development/testing API endpoint | systemd: aitbc-mock-coordinator.service |
|
||
| Blockchain Node | N/A | python3 | 3.13.5+ | Local blockchain node | systemd: aitbc-blockchain-node.service |
|
||
| Blockchain Node RPC | 8003 | python3 | 3.13.5+ | RPC API for blockchain | systemd: aitbc-blockchain-rpc.service |
|
||
| Local Development Tools | Varies | python3 | 3.13.5+ | CLI tools, scripts, testing | Manual/venv |
|
||
| **Note**: GPU Miner Client removed - no miner service needed on aitbc server |
|
||
| **Port Logic**: Production services use 8000-8019, Mock/Testing services use 8020+ |
|
||
|
||
### Systemd Services (Host)
|
||
|
||
All services are configured as systemd units but currently inactive:
|
||
|
||
```bash
|
||
# Service files location: /etc/systemd/system/
|
||
aitbc-coordinator-api.service # Production coordinator API on port 8000
|
||
aitbc-blockchain-node.service # Blockchain node main process
|
||
aitbc-blockchain-rpc.service # RPC API on port 8003
|
||
aitbc-mock-coordinator.service # Mock coordinator on port 8020
|
||
# Note: aitbc-gpu-miner.service removed - no miner service needed
|
||
```
|
||
|
||
**Service Details:**
|
||
- **Working Directory**: `/opt/aitbc/` (standard path for all services)
|
||
- **Python Environment**: `/opt/aitbc/.venv/bin/python` (Python 3.13.5+)
|
||
- **Node.js Environment**: System Node.js 24+ (current tested: v24.14.x)
|
||
- **User**: oib
|
||
- **Restart Policy**: always (with 5s delay)
|
||
|
||
### Standard Service Structure (/opt/aitbc)
|
||
|
||
On at1, `/opt/aitbc` uses individual symlinks to the Windsurf project directories:
|
||
|
||
```bash
|
||
/opt/aitbc/ # Service root with selective symlinks
|
||
├── apps/ # Symlinked app directories
|
||
│ ├── blockchain-explorer -> /home/oib/windsurf/aitbc/apps/blockchain-explorer/
|
||
│ ├── blockchain-node -> /home/oib/windsurf/aitbc/apps/blockchain-node/
|
||
│ ├── coordinator-api -> /home/oib/windsurf/aitbc/apps/coordinator-api/
|
||
│ ├── marketplace-web -> /home/oib/windsurf/aitbc/apps/marketplace-web/
|
||
│ ├── pool-hub -> /home/oib/windsurf/aitbc/apps/pool-hub/
|
||
│ ├── trade-exchange -> /home/oib/windsurf/aitbc/apps/trade-exchange/
|
||
│ ├── wallet-daemon -> /home/oib/windsurf/aitbc/apps/wallet-daemon/
|
||
│ └── zk-circuits -> /home/oib/windsurf/aitbc/apps/zk-circuits/
|
||
├── data/ # Local service data
|
||
├── logs/ # Local service logs
|
||
├── models/ # Local model storage
|
||
├── scripts -> /home/oib/windsurf/aitbc/scripts/ # Shared scripts
|
||
└── systemd -> /home/oib/windsurf/aitbc/systemd/ # Service definitions
|
||
```
|
||
|
||
**On aitbc/aitbc1 servers**: `/opt/aitbc` is symlinked to the git repo clone (`/opt/aitbc -> /path/to/aitbc-repo`) for complete repository access.
|
||
|
||
**Verification Commands:**
|
||
```bash
|
||
# Check service status
|
||
sc-status aitbc-blockchain-node.service aitbc-blockchain-rpc.service aitbc-gpu-miner.service aitbc-mock-coordinator.service
|
||
|
||
# Start services
|
||
sudo systemctl start aitbc-mock-coordinator.service
|
||
sudo systemctl start aitbc-blockchain-node.service
|
||
|
||
# Check logs
|
||
journalctl -u aitbc-mock-coordinator --no-pager -n 20
|
||
|
||
# Verify /opt/aitbc symlink structure
|
||
ls -la /opt/aitbc/ # Should show individual app symlinks
|
||
ls -la /opt/aitbc/apps/ # Should show all app symlinks
|
||
ls -la /opt/aitbc/scripts # Should show symlink to windsurf scripts
|
||
ls -la /opt/aitbc/systemd # Should show symlink to windsurf systemd
|
||
```
|
||
|
||
### Python Environment (at1)
|
||
|
||
**Development vs Service Environments**:
|
||
|
||
```bash
|
||
# Development environment (Windsurf project)
|
||
/home/oib/windsurf/aitbc/.venv/ # Development Python 3.13.5 environment
|
||
├── bin/python # Python executable
|
||
├── apps/ # Service applications
|
||
├── cli/ # CLI tools (12 command groups)
|
||
├── scripts/ # Development scripts
|
||
└── tests/ # Pytest suites
|
||
|
||
# Service environment (/opt/aitbc with symlinks)
|
||
/opt/aitbc/ # Service root with selective symlinks
|
||
├── apps/blockchain-node -> /home/oib/windsurf/aitbc/apps/blockchain-node/
|
||
├── apps/coordinator-api -> /home/oib/windsurf/aitbc/apps/coordinator-api/
|
||
├── scripts -> /home/oib/windsurf/aitbc/scripts/
|
||
└── systemd -> /home/oib/windsurf/aitbc/systemd/
|
||
|
||
# Node.js environment
|
||
node --version # Should show v24.14.x
|
||
npm --version # Should show compatible version
|
||
```
|
||
|
||
**Note**: Services use individual symlinks to specific app directories, while development uses the full Windsurf project workspace.
|
||
|
||
**Verification Commands:**
|
||
```bash
|
||
# Verify symlink structure
|
||
ls -la /opt/aitbc/ # Should show individual symlinks, not single repo symlink
|
||
ls -la /opt/aitbc/apps/blockchain-node # Should point to windsurf project
|
||
python3 --version # Should show Python 3.13.5
|
||
ls -la /home/oib/windsurf/aitbc/.venv/bin/python # Check development venv
|
||
node --version # Should show v24.14.x
|
||
npm --version # Should show compatible version
|
||
|
||
# Test symlink resolution
|
||
readlink -f /opt/aitbc/apps/blockchain-node # Should resolve to windsurf project path
|
||
readlink -f /opt/aitbc/scripts # Should resolve to windsurf scripts
|
||
```
|
||
|
||
### Nginx Reverse Proxy
|
||
|
||
The host runs a simple reverse proxy that forwards all traffic to the container. SSL is terminated here via Let's Encrypt.
|
||
|
||
- **Config**: `/etc/nginx/sites-available/aitbc-proxy.conf`
|
||
- **Enabled**: symlinked in `/etc/nginx/sites-enabled/`
|
||
- **SSL**: Let's Encrypt cert for `bubuit.net` (managed by Certbot)
|
||
- **Upstream**: `http://10.1.223.93` (container IP)
|
||
- **WebSocket**: supported (Upgrade/Connection headers forwarded)
|
||
|
||
```nginx
|
||
# /etc/nginx/sites-available/aitbc-proxy.conf (active)
|
||
server {
|
||
server_name aitbc.bubuit.net;
|
||
location / {
|
||
proxy_pass http://10.1.223.93;
|
||
proxy_set_header Host $host;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Upgrade $http_upgrade;
|
||
proxy_set_header Connection "upgrade";
|
||
}
|
||
listen 443 ssl; # managed by Certbot
|
||
ssl_certificate /etc/letsencrypt/live/bubuit.net/fullchain.pem;
|
||
ssl_certificate_key /etc/letsencrypt/live/bubuit.net/privkey.pem;
|
||
}
|
||
# HTTP → HTTPS redirect (managed by Certbot)
|
||
```
|
||
|
||
**Purged legacy configs** (2026-02-14):
|
||
- `aitbc-website-new.conf` — served files directly from host, bypassing container. Deleted.
|
||
|
||
## Container: aitbc (10.1.223.1)
|
||
|
||
### Access
|
||
```bash
|
||
ssh aitbc-cascade # Direct SSH to container
|
||
```
|
||
|
||
**GPU Access**: No GPU passthrough. All GPU workloads must run on **at1** (Windsurf development host), not inside incus containers.
|
||
|
||
**Miner Service**: Not needed - aitbc server operates in CPU-only mode.
|
||
|
||
**Host Proxies (for localhost GPU clients)**
|
||
- `127.0.0.1:18000` → container `127.0.0.1:8000` (coordinator/marketplace API)
|
||
- Use this to submit offers/bids/contracts/mining requests from localhost GPU miners/dev clients.
|
||
|
||
**Container Services (Updated March 5, 2026 - Port Logic 8000+)**
|
||
- **12 Services**: All 12 services operational with unified port logic
|
||
- **Core Services**: 8000-8003 (Coordinator, Exchange, Blockchain Node, RPC)
|
||
- **Enhanced Services**: 8010-8017 (GPU services in CPU-only mode, Web UI, Load Balancer)
|
||
- **Port Logic**: All services use 8000+ numbering scheme for consistency
|
||
- **0.0.0.0 Binding**: All services bind to 0.0.0.0 for container access
|
||
- **Production Ready**: All services marked as production ready
|
||
|
||
**Port Logic Breakdown:**
|
||
- **8000**: Coordinator API (main API gateway)
|
||
- **8001**: Cross-Chain Exchange API (Multi-chain trading operations)
|
||
- **8002**: Blockchain Node (P2P node service)
|
||
- **8003**: Blockchain RPC (JSON-RPC interface)
|
||
- **8007**: Blockchain Service (Transaction processing and consensus)
|
||
- **8008**: Network Service (P2P block propagation)
|
||
- **8016**: Blockchain Explorer (Data aggregation and web interface)
|
||
- **8010**: Multimodal GPU (AI processing)
|
||
- **8011**: GPU Multimodal (multi-modal AI)
|
||
- **8012**: Modality Optimization (AI optimization)
|
||
- **8013**: Adaptive Learning (machine learning)
|
||
- **8014**: Marketplace Enhanced (advanced marketplace)
|
||
- **8015**: OpenClaw Enhanced (agent marketplace)
|
||
- **8016**: Web UI (dashboard interface)
|
||
- **8017**: Geographic Load Balancer (traffic distribution)
|
||
|
||
## Container: aitbc1 (10.1.223.40) — New Dev Server
|
||
|
||
### Access
|
||
```bash
|
||
ssh aitbc1-cascade # Direct SSH to aitbc1 container (incus)
|
||
```
|
||
|
||
### Notes
|
||
- Purpose: secondary AITBC dev environment (incus container)
|
||
- Host: 10.1.223.40 (Debian 13 Trixie), accessible via new SSH alias `aitbc1-cascade`
|
||
- OS: Debian 13 Trixie (development environment)
|
||
- Node.js: 24+ (current tested: v24.14.x)
|
||
- Python: 3.13.5+ (minimum requirement, strictly enforced)
|
||
- Proxy device: incus proxy on host maps 127.0.0.1:18001 → 127.0.0.1:8000 inside container
|
||
- AppArmor profile: unconfined (incus raw.lxc)
|
||
- Use same deployment patterns as `aitbc` (nginx + services) once provisioned
|
||
- **GPU Access**: None. Run GPU-dependent tasks on **at1** (Windsurf development host) only.
|
||
|
||
**Host Proxies (for localhost GPU clients)**
|
||
- `127.0.0.1:18001` → container `127.0.0.1:8000` (coordinator/marketplace API)
|
||
- Use this to hit the second marketplace/coordinator from localhost GPU miners/dev clients.
|
||
- (Optional) Expose marketplace frontend for aitbc1 via an additional proxy/port if needed for UI tests.
|
||
- Health check suggestion: `curl -s http://127.0.0.1:18001/v1/health`
|
||
|
||
**at1 dual-miner/dual-client test (shared GPU)**
|
||
- Run two miners on **at1** (GPU shared), targeting each marketplace:
|
||
- Miner A → `http://127.0.0.1:18000`
|
||
- Miner B → `http://127.0.0.1:18001`
|
||
- Run two clients on **at1** for bids/contracts/Ollama answers:
|
||
- Client 1 → `http://127.0.0.1:18000`
|
||
- Client 2 → `http://127.0.0.1:18001`
|
||
- Use a shared dev chain so both marketplaces see the same on-chain events.
|
||
- Example commands (adjust to your scripts/flags):
|
||
- `miner --id miner-A --gpu 0 --api http://127.0.0.1:18000`
|
||
- `miner --id miner-B --gpu 0 --api http://127.0.0.1:18001`
|
||
- `client --id client-1 --api http://127.0.0.1:18000 --ollama-model <model>`
|
||
- `client --id client-2 --api http://127.0.0.1:18001 --ollama-model <model>`
|
||
|
||
|
||
### Services (Port Logic 8000+)
|
||
|
||
| Service | Port (8000+) | Process | Python Version | Public URL | Status |
|
||
|---------|-------------|---------|----------------|------------|--------|
|
||
| Nginx (web) | 80 | nginx | N/A | https://aitbc.bubuit.net/ | ✅ |
|
||
| Coordinator API | 8000 | python (uvicorn) | 3.13.5 | /api/ → /v1/ | ✅ |
|
||
| Exchange API | 8001 | python (uvicorn) | 3.13.5 | /api/exchange/* | ✅ |
|
||
| Blockchain Node | 8002 | python3 | 3.13.5 | Internal | ✅ |
|
||
| Blockchain RPC | 8003 | python3 | 3.13.5 | /rpc/ | ✅ |
|
||
| Multimodal GPU | 8010 | python | 3.13.5 | /api/gpu/* | ✅ (CPU-only) |
|
||
| GPU Multimodal | 8011 | python | 3.13.5 | /api/gpu-multimodal/* | ✅ (CPU-only) |
|
||
| Modality Optimization | 8012 | python | 3.13.5 | /api/optimization/* | ✅ |
|
||
| Adaptive Learning | 8013 | python | 3.13.5 | /api/learning/* | ✅ |
|
||
| Marketplace Enhanced | 8014 | python | 3.13.5 | /api/marketplace-enhanced/* | ✅ |
|
||
| OpenClaw Enhanced | 8015 | python | 3.13.5 | /api/openclaw/* | ✅ |
|
||
| Web UI | 8016 | python | 3.13.5 | /app/ | ✅ |
|
||
| Geographic Load Balancer | 8017 | python | 3.13.5 | /api/loadbalancer/* | ✅ |
|
||
|
||
**Python 3.13.5 and Node.js 24+ Upgrade Complete** (2026-03-05):
|
||
- All services upgraded to Python 3.13.5
|
||
- Node.js upgraded to 24+ (current tested: v24.14.x)
|
||
- Virtual environments updated and verified
|
||
- API routing fixed for external access
|
||
- Services fully operational with enhanced performance
|
||
- **Port Logic 8000+**: Unified numbering scheme implemented
|
||
- Core Services: 8000-8003 (Coordinator, Exchange, Blockchain, RPC)
|
||
- Enhanced Services: 8010-8017 (AI, GPU, Web UI, Load Balancer)
|
||
- GPU services configured for CPU-only mode
|
||
- Miner service removed - not needed
|
||
- 0.0.0.0 binding enabled for container access
|
||
|
||
### Python Environment Details
|
||
|
||
All Python services in the AITBC container run on **Python 3.13.5** with isolated virtual environments:
|
||
|
||
```bash
|
||
# Container: aitbc (10.1.223.1)
|
||
/opt/aitbc/apps/coordinator-api/.venv/ # Coordinator API (uvicorn, FastAPI)
|
||
/opt/aitbc/apps/blockchain-node/.venv/ # Blockchain Node 1 (aitbc_chain)
|
||
/opt/aitbc/apps/exchange/.venv/ # Exchange API (Flask/specific framework)
|
||
# Note: Standardized /opt/aitbc structure for all services
|
||
```
|
||
|
||
**Verification Commands:**
|
||
```bash
|
||
ssh aitbc-cascade "python3 --version" # Should show Python 3.13.5
|
||
ssh aitbc-cascade "node --version" # Should show v24.14.x
|
||
ssh aitbc-cascade "npm --version" # Should show compatible version
|
||
ssh aitbc-cascade "ls -la /opt/*/.venv/bin/python" # Check venv symlinks
|
||
ssh aitbc-cascade "curl -s http://127.0.0.1:8000/v1/health" # Coordinator API health
|
||
curl -s https://aitbc.bubuit.net/api/v1/health # External API access
|
||
```
|
||
|
||
### Nginx Routes (container)
|
||
|
||
Config: `/etc/nginx/sites-enabled/aitbc`
|
||
|
||
| Route | Target | Type | Status |
|
||
|-------|--------|------|--------|
|
||
| `/` | static files (`/var/www/html/`) | try_files | ✅ |
|
||
| `/api/` | proxy → `127.0.0.1:8000/v1/` | proxy_pass | ✅ |
|
||
| `/exchange/` | proxy → `127.0.0.1:8001/` | proxy_pass | ✅ |
|
||
| `/rpc/` | proxy → `127.0.0.1:8006/rpc/` | proxy_pass | ✅ |
|
||
| `/wallet/` | proxy → `127.0.0.1:8000/wallet/` | proxy_pass | ✅ |
|
||
| `/health` | proxy → `127.0.0.1:8000/v1/health` | proxy_pass | ✅ |
|
||
| `/gpu/multimodal/` | proxy → `127.0.0.1:8010/` | proxy_pass | ✅ (CPU-only) |
|
||
| `/gpu/service/` | proxy → `127.0.0.1:8011/` | proxy_pass | ✅ (CPU-only) |
|
||
| `/optimization/` | proxy → `127.0.0.1:8012/` | proxy_pass | ✅ |
|
||
| `/learning/` | proxy → `127.0.0.1:8013/` | proxy_pass | ✅ |
|
||
| `/marketplace/enhanced/` | proxy → `127.0.0.1:8014/` | proxy_pass | ✅ |
|
||
| `/openclaw/` | proxy → `127.0.0.1:8015/` | proxy_pass | ✅ |
|
||
| `/explorer/` | proxy → `127.0.0.1:8016/` | proxy_pass | ✅ |
|
||
| `/balancer/` | proxy → `127.0.0.1:8017/` | proxy_pass | ✅ |
|
||
|
||
**API Routing Updated** (2026-03-10):
|
||
- Updated nginx configuration to use new port logic from infrastructure documentation
|
||
- Updated RPC route from port 8003 to port 8006 (blockchain services)
|
||
- Updated Exchange API route to port 8001 (core services)
|
||
- Added Enhanced Services routes with correct port mappings (8010-8017)
|
||
- Simplified configuration for HTTP-only mode (SSL handled by host reverse proxy)
|
||
- External API access: `https://aitbc.bubuit.net/api/v1/health` → `{"status":"ok","env":"dev"}`
|
||
- All GPU services configured for CPU-only mode
|
||
|
||
### Web Root (`/var/www/html/`)
|
||
|
||
```
|
||
/var/www/html/
|
||
├── index.html # Main website
|
||
├── 404.html # Error page
|
||
└── static files # CSS, JS, images
|
||
```
|
||
|
||
### Data Storage (container)
|
||
|
||
```
|
||
/opt/aitbc/apps/coordinator-api/ # Coordinator application
|
||
├── src/coordinator.db # Main database
|
||
└── .venv/ # Python environment
|
||
|
||
/opt/aitbc/apps/blockchain-node/ # Blockchain Node 1
|
||
├── data/chain.db # Chain database
|
||
└── .venv/ # Python environment
|
||
|
||
/opt/aitbc/apps/exchange/ # Exchange API
|
||
├── data/ # Exchange data
|
||
└── .venv/ # Python environment
|
||
```
|
||
|
||
### Configuration (container)
|
||
- Node 1: `/opt/aitbc/apps/blockchain-node/src/aitbc_chain/config.py`
|
||
- Coordinator API: `/opt/aitbc/apps/coordinator-api/.env`
|
||
- Exchange API: `/opt/aitbc/apps/exchange/.env`
|
||
- Enhanced Services: Environment variables in respective service files
|
||
|
||
## Known Limitations and Compatibility Issues
|
||
|
||
### Concrete ML Python 3.13 Compatibility
|
||
|
||
**Status**: ⚠️ **Known Limitation**
|
||
**Severity**: 🟡 **Medium** (Functional limitation, no security impact)
|
||
**Date Identified**: March 5, 2026
|
||
|
||
#### Issue Description
|
||
The Coordinator API service logs a warning about Concrete ML not being installed due to Python version incompatibility:
|
||
|
||
```
|
||
WARNING:root:Concrete ML not installed; skipping Concrete provider. Concrete ML requires Python <3.13. Current version: 3.13.5
|
||
```
|
||
|
||
#### Technical Details
|
||
- **Affected Component**: Coordinator API FHE (Fully Homomorphic Encryption) Service
|
||
- **Root Cause**: Concrete ML library requires Python <3.13, but AITBC runs on Python 3.13.5
|
||
- **Impact**: Limited to Concrete ML FHE provider; TenSEAL provider continues to work normally
|
||
- **Current Status**: Service operates normally with TenSEAL provider only
|
||
|
||
#### Compatibility Matrix
|
||
| Python Version | Concrete ML Support | AITBC Status |
|
||
|---------------|-------------------|--------------|
|
||
| 3.8.x - 3.12.x | ✅ Supported | ❌ Not used |
|
||
| 3.13.x | ❌ Not Supported | ✅ Current version |
|
||
| 3.14+ | ❌ Unknown | ❌ Future consideration |
|
||
|
||
#### Functional Impact
|
||
- **FHE Operations**: ✅ **No Impact** - TenSEAL provides full FHE functionality
|
||
- **API Endpoints**: ✅ **No Impact** - All FHE endpoints work normally
|
||
- **Performance**: ✅ **No Impact** - TenSEAL performance is excellent
|
||
- **Security**: ✅ **No Impact** - Encryption schemes remain secure
|
||
|
||
#### Feature Limitations
|
||
- **Neural Network Compilation**: ❌ **Unavailable** - Concrete ML specific feature
|
||
- **Advanced ML Models**: ⚠️ **Limited** - Some complex models may require Concrete ML
|
||
- **Research Features**: ❌ **Unavailable** - Experimental Concrete ML features
|
||
|
||
#### Resolution Strategy
|
||
- **Short Term**: Continue with TenSEAL-only implementation (already in place)
|
||
- **Medium Term**: Monitor Concrete ML for Python 3.13 compatibility updates
|
||
- **Long Term**: Consider dual Python environment if business need arises
|
||
|
||
#### Related Documentation
|
||
- See `docs/12_issues/concrete-ml-compatibility.md` for detailed technical analysis
|
||
- Monitoring and alerting configured for service health
|
||
- No user-facing impact or action required
|
||
|
||
## Remote Site (ns3)
|
||
|
||
### Host (ns3-root)
|
||
- **IP**: 95.216.198.140
|
||
- **Access**: `ssh ns3-root`
|
||
- **Bridge**: incusbr0 `192.168.100.1/24`
|
||
- **Port forwarding**: firehol (8000, 8001, 8003, 8010-8017 → 192.168.100.10)
|
||
- **Updated**: Port logic aligned with main aitbc server
|
||
|
||
### Container (ns3/aitbc)
|
||
- **IP**: 192.168.100.10
|
||
- **Domain**: aitbc.keisanki.net
|
||
- **Access**: `ssh ns3-root` → `incus shell aitbc`
|
||
- **Blockchain Node 3**: RPC on port 8003 (updated port logic)
|
||
- **GPU Access**: None (CPU-only mode)
|
||
- **Miner Service**: Not needed
|
||
|
||
```bash
|
||
curl http://aitbc.keisanki.net/rpc/head # Node 3 RPC (port 8003)
|
||
```
|
||
|
||
## Cross-Site Synchronization
|
||
|
||
- **Status**: Active on all 3 nodes
|
||
- **Method**: RPC-based polling every 10 seconds
|
||
- **Features**: Transaction propagation, height detection, block import
|
||
- **Endpoints**:
|
||
- Local: https://aitbc.bubuit.net/rpc/ (Node 1, port 8003)
|
||
- Remote: http://aitbc.keisanki.net/rpc/ (Node 3, port 8003)
|
||
- **Updated**: All nodes using new port logic (8003 for RPC)
|
||
- **Consensus**: PoA with 2s block intervals
|
||
- **P2P**: Not connected yet; nodes maintain independent chain state
|
||
|
||
## Development Workspace (at1)
|
||
|
||
```
|
||
/home/oib/windsurf/aitbc/ # at1 Windsurf development workspace
|
||
├── apps/ # Application source (8 apps)
|
||
├── cli/ # CLI tools (12 command groups)
|
||
├── scripts/ # Organized scripts (8 subfolders)
|
||
│ ├── blockchain/ # Genesis, proposer, mock chain
|
||
│ ├── dev/ # Dev tools, local services
|
||
│ └── examples/ # Usage examples and simulations
|
||
├── tests/ # Pytest suites + verification scripts
|
||
├── docs/ # Markdown documentation (10 sections)
|
||
└── website/ # Public website source
|
||
```
|
||
|
||
### Deploying to Container
|
||
```bash
|
||
# Push website files
|
||
scp -r website/* aitbc-cascade:/var/www/aitbc.bubuit.net/
|
||
|
||
# Push app updates (blockchain-explorer serves its own interface)
|
||
# No separate deployment needed - blockchain-explorer handles both API and UI
|
||
|
||
# Restart a service
|
||
ssh aitbc-cascade "systemctl restart coordinator-api"
|
||
```
|
||
|
||
## Health Checks
|
||
|
||
```bash
|
||
# From at1 (via container)
|
||
ssh aitbc-cascade "curl -s http://localhost:8000/v1/health"
|
||
ssh aitbc-cascade "curl -s http://localhost:8003/rpc/head | jq .height"
|
||
|
||
# Test enhanced services
|
||
ssh aitbc-cascade "curl -s http://localhost:8010/health" # Multimodal GPU (CPU-only)
|
||
ssh aitbc-cascade "curl -s http://localhost:8017/health" # Geographic Load Balancer
|
||
|
||
# From internet (Python 3.13.5 upgraded services)
|
||
curl -s https://aitbc.bubuit.net/health
|
||
curl -s https://aitbc.bubuit.net/api/v1/health # ✅ Fixed API routing
|
||
curl -s https://aitbc.bubuit.net/api/explorer/blocks
|
||
|
||
# Test enhanced services externally
|
||
curl -s https://aitbc.bubuit.net/api/gpu/health
|
||
curl -s https://aitbc.bubuit.net/api/loadbalancer/health
|
||
|
||
# Remote site
|
||
ssh ns3-root "curl -s http://192.168.100.10:8003/rpc/head | jq .height"
|
||
|
||
# Python version verification
|
||
ssh aitbc-cascade "python3 --version" # Python 3.13.5
|
||
```
|
||
|
||
## Monitoring and Logging
|
||
|
||
```bash
|
||
# Container systemd logs
|
||
ssh aitbc-cascade "journalctl -u aitbc-coordinator-api --no-pager -n 20"
|
||
ssh aitbc-cascade "journalctl -u aitbc-blockchain-node --no-pager -n 20"
|
||
|
||
# Enhanced services logs
|
||
ssh aitbc-cascade "journalctl -u aitbc-multimodal-gpu --no-pager -n 20"
|
||
ssh aitbc-cascade "journalctl -u aitbc-loadbalancer-geo --no-pager -n 20"
|
||
|
||
# Container nginx logs
|
||
ssh aitbc-cascade "tail -20 /var/log/nginx/aitbc.bubuit.net.error.log"
|
||
|
||
# Host nginx logs
|
||
sudo tail -20 /var/log/nginx/error.log
|
||
```
|
||
|
||
## Security
|
||
|
||
### SSL/TLS
|
||
- Let's Encrypt certificate for `bubuit.net` (wildcard)
|
||
- SSL termination at incus host nginx
|
||
- HTTP → HTTPS redirect (Certbot managed)
|
||
|
||
### CORS
|
||
- Coordinator API: localhost origins only (8000-8003, 8010-8017)
|
||
- Exchange API: localhost origins only (8000-8003, 8010-8017)
|
||
- Blockchain Node: localhost origins only (8000-8003, 8010-8017)
|
||
- Enhanced Services: localhost origins only (8010-8017)
|
||
- **Updated**: New port logic reflected in CORS policies
|
||
|
||
### Authentication
|
||
- Coordinator API: `X-Api-Key` header required
|
||
- Exchange API: session-based (wallet address login, 24h expiry)
|
||
- JWT secrets from environment variables (fail-fast on startup)
|
||
|
||
### Encryption
|
||
- Wallet private keys: Fernet (AES-128-CBC) with PBKDF2-SHA256 key derivation
|
||
- Database credentials: parsed from `DATABASE_URL` env var
|
||
|
||
### Environment Variables
|
||
```bash
|
||
# Coordinator API
|
||
JWT_SECRET=<secret>
|
||
DATABASE_URL=sqlite:///./aitbc_coordinator.db
|
||
MINER_API_KEYS=["production_key_32_characters_long_minimum"]
|
||
# Note: No miner service needed - configuration kept for compatibility
|
||
|
||
# Exchange API
|
||
SESSION_SECRET=<secret>
|
||
WALLET_ENCRYPTION_KEY=<key>
|
||
|
||
# Enhanced Services
|
||
HOST=0.0.0.0 # For container access
|
||
PORT=8010-8017 # Enhanced services port range
|
||
```
|
||
|
||
### Container Access & Port Logic (Updated March 6, 2026)
|
||
|
||
#### **SSH-Based Container Access**
|
||
```bash
|
||
# Access aitbc container
|
||
ssh aitbc-cascade
|
||
|
||
# Access aitbc1 container
|
||
ssh aitbc1-cascade
|
||
|
||
# Check services in containers
|
||
ssh aitbc-cascade 'systemctl list-units | grep aitbc-'
|
||
ssh aitbc1-cascade 'systemctl list-units | grep aitbc-'
|
||
|
||
# Debug specific services
|
||
ssh aitbc-cascade 'systemctl status aitbc-coordinator-api'
|
||
ssh aitbc1-cascade 'systemctl status aitbc-wallet'
|
||
```
|
||
|
||
#### **Port Distribution Strategy - NEW STANDARD**
|
||
```bash
|
||
# === NEW STANDARD PORT LOGIC ===
|
||
|
||
# Core Services (8000-8003) - NEW STANDARD
|
||
- Port 8000: Coordinator API (local) ✅ NEW STANDARD
|
||
- Port 8001: Exchange API (local) ✅ NEW STANDARD
|
||
- Port 8002: Blockchain Node (local) ✅ NEW STANDARD
|
||
- Port 8003: Blockchain RPC (local) ✅ NEW STANDARD
|
||
|
||
# Blockchain Services (8004-8005) - PRODUCTION READY
|
||
- Port 8004: Primary Blockchain Node ✅ PRODUCTION READY (aitbc-blockchain-node.service)
|
||
- Port 8005: Blockchain RPC 2 ✅ PRODUCTION READY
|
||
|
||
# Level 2 Services (8010-8017) - NEW STANDARD
|
||
- Port 8010: Multimodal GPU Service ✅ NEW STANDARD
|
||
- Port 8011: GPU Multimodal Service ✅ NEW STANDARD
|
||
- Port 8012: Modality Optimization Service ✅ NEW STANDARD
|
||
- Port 8013: Adaptive Learning Service ✅ NEW STANDARD
|
||
- Port 8014: Marketplace Enhanced Service ✅ NEW STANDARD
|
||
- Port 8015: OpenClaw Enhanced Service ✅ NEW STANDARD
|
||
- Port 8016: Web UI Service ✅ NEW STANDARD
|
||
- Port 8017: Geographic Load Balancer ✅ NEW STANDARD
|
||
|
||
# Mock & Test Services (8020-8029) - NEW STANDARD
|
||
- Port 8020: Mock Coordinator API ✅ NEW STANDARD
|
||
- Port 8021: Coordinator API (dev) ✅ NEW STANDARD
|
||
- Port 8022: Test Blockchain Node (localhost) ✅ NEW STANDARD
|
||
- Port 8025: Development Blockchain Node ✅ NEW STANDARD (aitbc-blockchain-node-dev.service)
|
||
- Port 8026-8029: Additional testing services ✅ NEW STANDARD
|
||
|
||
# === LEGACY PORTS (DEPRECATED) ===
|
||
|
||
# Legacy Container Services (8080-8089) - DEPRECATED
|
||
- Port 8080-8089: All container services ⚠️ DEPRECATED - Use 8000+ and 8010+ ranges
|
||
```
|
||
|
||
#### **Service Naming Convention**
|
||
```bash
|
||
# === STANDARDIZED SERVICE NAMES ===
|
||
|
||
# Primary Production Services:
|
||
✅ aitbc-blockchain-node.service (port 8005) - Primary blockchain node
|
||
✅ aitbc-blockchain-rpc.service (port 8006) - Primary blockchain RPC (localhost + containers)
|
||
✅ aitbc-coordinator-api.service (port 8000) - Main coordinator API
|
||
✅ aitbc-exchange-api.service (port 8001) - Exchange API
|
||
✅ aitbc-wallet.service (port 8002) - Wallet Service (localhost + containers)
|
||
|
||
# Development/Test Services:
|
||
✅ aitbc-blockchain-node-dev.service (port 8025) - Development blockchain node
|
||
✅ aitbc-blockchain-rpc-dev.service (port 8026) - Development blockchain RPC
|
||
✅ aitbc-coordinator-api-dev.service (port 8021) - Development coordinator API
|
||
|
||
# Container Locations:
|
||
✅ localhost (at1): Primary services + development services
|
||
✅ aitbc container: Primary services + development services
|
||
✅ aitbc1 container: Primary services + development services
|
||
```
|
||
|
||
#### **Port Conflict Resolution**
|
||
```bash
|
||
# Updated port assignments - NO CONFLICTS:
|
||
# Local services use 8000-8003 range (core services)
|
||
# Blockchain services use 8004-8005 range (primary blockchain nodes)
|
||
# Level 2 services use 8010-8017 range (enhanced services)
|
||
# Mock & test services use 8020-8029 range (development services)
|
||
|
||
# Check port usage
|
||
netstat -tlnp | grep -E ":(800[0-5]|801[0-7]|802[0-9])"
|
||
ssh aitbc-cascade 'netstat -tlnp | grep -E ":(800[0-5]|801[0-7]|802[0-9])"
|
||
ssh aitbc1-cascade 'netstat -tlnp | grep -E ":(800[0-5]|801[0-7]|802[0-9])"
|
||
|
||
# Service Management Commands:
|
||
# Primary services:
|
||
systemctl status aitbc-blockchain-node.service # localhost
|
||
systemctl status aitbc-blockchain-rpc.service # localhost (port 8006)
|
||
systemctl status aitbc-wallet.service # localhost (port 8002)
|
||
ssh aitbc-cascade 'systemctl status aitbc-blockchain-node.service' # aitbc container
|
||
ssh aitbc1-cascade 'systemctl status aitbc-blockchain-node.service' # aitbc1 container
|
||
|
||
# Wallet services:
|
||
ssh aitbc-cascade 'systemctl status aitbc-wallet.service' # port 8002
|
||
ssh aitbc1-cascade 'systemctl status aitbc-wallet.service' # port 8002
|
||
|
||
# RPC services:
|
||
ssh aitbc-cascade 'systemctl status aitbc-blockchain-rpc.service' # port 8006
|
||
ssh aitbc1-cascade 'systemctl status aitbc-blockchain-rpc.service' # port 8006
|
||
ssh aitbc-cascade 'systemctl status aitbc-blockchain-rpc-dev.service' # port 8026
|
||
ssh aitbc1-cascade 'systemctl status aitbc-blockchain-rpc-dev.service' # port 8026
|
||
|
||
# Development services:
|
||
ssh aitbc-cascade 'systemctl status aitbc-blockchain-node-dev.service'
|
||
ssh aitbc1-cascade 'systemctl status aitbc-blockchain-node-dev.service'
|
||
```
|