aitbc/apps/docs/MICROSERVICES_TESTING_GUIDE.md

AITBC Microservices Testing Guide

This guide provides comprehensive testing procedures for the AITBC microservices architecture following the Coordinator-API monolith breakup.

Table of Contents

• Prerequisites
• Database Setup
• Service Installation
• Service Startup
• Individual Service Testing
• Gateway Integration Testing
• Expected Test Outcomes
• Troubleshooting

Prerequisites

System Requirements

• Python 3.13+
• PostgreSQL 14+
• Poetry (for dependency management)
• Systemd (for production deployment)
• curl or httpie (for testing endpoints)

Environment Setup

# Ensure you're in the AITBC root directory
cd /opt/aitbc

# Install dependencies
poetry install

# Verify PostgreSQL is running
sudo systemctl status postgresql

Database Setup

Each microservice requires a separate database. Run the setup scripts for each service you want to test.

GPU Service Database

sudo -u postgres psql -f apps/gpu-service/scripts/setup-database.sql

Expected output:

CREATE DATABASE
CREATE ROLE
GRANT

Marketplace Service Database

sudo -u postgres psql -f apps/marketplace-service/scripts/setup-database.sql

Trading Service Database

sudo -u postgres psql -f apps/trading-service/scripts/setup-database.sql

Governance Service Database

sudo -u postgres psql -f apps/governance-service/scripts/setup-database.sql

Verify Database Creation

sudo -u postgres psql -l

Expected databases:

• aitbc_gpu
• aitbc_marketplace
• aitbc_trading
• aitbc_governance

Service Installation

Install each service using Poetry:

# GPU Service
poetry install --with gpu-service

# Marketplace Service
poetry install --with marketplace-service

# Trading Service
poetry install --with trading-service

# Governance Service
poetry install --with governance-service

# API Gateway
poetry install --with api-gateway

Service Startup

Development Mode (Manual Startup)

Start services manually in separate terminal windows:

# Terminal 1: GPU Service
cd /opt/aitbc
python -m gpu_service.main

# Terminal 2: Marketplace Service
cd /opt/aitbc
python -m marketplace_service.main

# Terminal 3: Trading Service
cd /opt/aitbc
python -m trading_service.main

# Terminal 4: Governance Service
cd /opt/aitbc
python -m governance_service.main

# Terminal 5: API Gateway
cd /opt/aitbc
python -m api_gateway.main

Production Mode (Systemd)

For production deployment, use systemd services:

# Copy service files to systemd
sudo cp apps/gpu-service/gpu-service.service /etc/systemd/system/
sudo cp apps/marketplace-service/marketplace-service.service /etc/systemd/system/
sudo cp apps/trading-service/trading-service.service /etc/systemd/system/
sudo cp apps/governance-service/governance-service.service /etc/systemd/system/
sudo cp apps/api-gateway/api-gateway.service /etc/systemd/system/

# Reload systemd
sudo systemctl daemon-reload

# Enable and start services
sudo systemctl enable gpu-service marketplace-service trading-service governance-service api-gateway
sudo systemctl start gpu-service marketplace-service trading-service governance-service api-gateway

# Check service status
sudo systemctl status gpu-service
sudo systemctl status marketplace-service
sudo systemctl status trading-service
sudo systemctl status governance-service
sudo systemctl status api-gateway

Individual Service Testing

GPU Service Testing

Health Check

curl http://localhost:8101/health

Expected response:

{
  "status": "healthy",
  "service": "gpu-service"
}

GPU Status

curl http://localhost:8101/gpu/status

Expected response:

{
  "status": "operational",
  "service": "gpu-service",
  "message": "GPU service is running"
}

Get Consumer GPU Profiles

curl http://localhost:8101/v1/marketplace/edge-gpu/profiles

Expected response:

[
  {
    "profile_id": "consumer_nvidia_a100",
    "name": "NVIDIA A100",
    "architecture": "NVIDIA",
    "memory_gb": 80,
    "cuda_cores": 6912,
    "tensor_cores": 432,
    "compute_capability": "8.0",
    "typical_use_cases": ["ml_training", "inference", "hpc"]
  }
]

Marketplace Service Testing

Health Check

curl http://localhost:8102/health

Expected response:

{
  "status": "healthy",
  "service": "marketplace-service"
}

Get Marketplace Offers

curl http://localhost:8102/v1/marketplace/offers

Expected response:

[]

Trading Service Testing

Health Check

curl http://localhost:8104/health

Expected response:

{
  "status": "healthy",
  "service": "trading-service"
}

Get Trade Requests

curl http://localhost:8104/v1/trading/requests

Expected response:

[]

Governance Service Testing

Health Check

curl http://localhost:8105/health

Expected response:

{
  "status": "healthy",
  "service": "governance-service"
}

Get Governance Proposals

curl http://localhost:8105/v1/governance/proposals

Expected response:

[]

Gateway Integration Testing

Gateway Health Check

curl http://localhost:8080/health

Expected response:

{
  "status": "healthy",
  "service": "api-gateway"
}

Gateway Service Registry

curl http://localhost:8080/services

Expected response:

{
  "services": [
    {
      "name": "gpu",
      "url": "http://localhost:8101",
      "health_check": "/health",
      "routes": ["/gpu/*"]
    },
    {
      "name": "marketplace",
      "url": "http://localhost:8102",
      "health_check": "/health",
      "routes": ["/marketplace/*"]
    },
    {
      "name": "trading",
      "url": "http://localhost:8104",
      "health_check": "/health",
      "routes": ["/trading/*"]
    },
    {
      "name": "governance",
      "url": "http://localhost:8105",
      "health_check": "/health",
      "routes": ["/governance/*"]
    }
  ]
}

Test GPU Service Through Gateway

curl http://localhost:8080/gpu/health

Expected response:

{
  "status": "healthy",
  "service": "gpu-service"
}

curl http://localhost:8080/gpu/v1/marketplace/edge-gpu/profiles

Expected response:

[
  {
    "profile_id": "consumer_nvidia_a100",
    "name": "NVIDIA A100",
    "architecture": "NVIDIA",
    "memory_gb": 80,
    "cuda_cores": 6912,
    "tensor_cores": 432,
    "compute_capability": "8.0",
    "typical_use_cases": ["ml_training", "inference", "hpc"]
  }
]

Test Marketplace Service Through Gateway

curl http://localhost:8080/marketplace/health

Expected response:

{
  "status": "healthy",
  "service": "marketplace-service"
}

Test Trading Service Through Gateway

curl http://localhost:8080/trading/health

Expected response:

{
  "status": "healthy",
  "service": "trading-service"
}

Test Governance Service Through Gateway

curl http://localhost:8080/governance/health

Expected response:

{
  "status": "healthy",
  "service": "governance-service"
}

Expected Test Outcomes

Successful Test Indicators

• All health endpoints return {"status": "healthy", "service": "<service-name>"}
• Gateway successfully proxies requests to microservices
• Service registry shows all registered services
• Database connections are established without errors
• No 500 errors or connection refused messages

Failure Indicators

• Connection refused errors (service not running)
• 500 internal server errors (service error)
• Gateway routing failures (incorrect configuration)
• Database connection errors (database not set up)

Troubleshooting

Service Won't Start

Issue: Service fails to start with database connection error

Solution:

# Verify database exists
sudo -u postgres psql -l

# Recreate database if needed
sudo -u postgres psql -f apps/<service-name>/scripts/setup-database.sql

# Check service logs
sudo journalctl -u <service-name> -n 50

Gateway Can't Reach Service

Issue: Gateway returns connection refused when proxying to service

Solution:

# Verify service is running
curl http://localhost:<port>/health

# Check gateway service registry
curl http://localhost:8080/services

# Verify service URL in gateway configuration

Database Connection Errors

Issue: Service fails with "could not connect to server" error

Solution:

# Verify PostgreSQL is running
sudo systemctl status postgresql

# Check database credentials in service file
# Default: aitbc_<service>:password@localhost:5432/aitbc_<service>

# Test database connection manually
sudo -u postgres psql -d aitbc_<service>

Port Already in Use

Issue: Service fails to start with "Address already in use" error

Solution:

# Find process using the port
sudo lsof -i :<port>

# Kill the process if needed
sudo kill <pid>

# Or use a different port in the service configuration

Testing Checklist

Before considering testing complete:

• All databases created and accessible
• All services installed via Poetry
• All services start successfully in development mode
• All services start successfully in production mode (systemd)
• Health endpoints return healthy status for all services
• Gateway health check passes
• Gateway service registry shows all services
• Gateway successfully proxies requests to GPU service
• Gateway successfully proxies requests to Marketplace service
• Gateway successfully proxies requests to Trading service
• Gateway successfully proxies requests to Governance service
• No 500 errors in service logs
• No connection errors in gateway logs

Next Steps

After successful testing:

1. Deploy services to production environment
2. Configure monitoring and alerting
3. Set up log aggregation
4. Implement automated testing in CI/CD pipeline
5. Gradually migrate traffic from coordinator-api to microservices