Add comprehensive testing documentation for microservices

- Created MICROSERVICES_TESTING_GUIDE.md with detailed testing procedures
- Updated GPU service README with detailed testing steps and expected responses
- Updated Marketplace service README with detailed testing steps and expected responses
- Updated Trading service README with detailed testing steps and expected responses
- Updated Governance service README with detailed testing steps and expected responses
- Updated API gateway README with detailed testing steps and routing tests

This completes Phase 7b: Update service READMEs with detailed testing steps

apps/api-gateway/README.md

@@ -31,6 +31,139 @@ sudo systemctl start api-gateway
 sudo systemctl enable api-gateway
 ```
 
+## Testing
+
+### Prerequisites
+
+- All microservices installed via Poetry
+- At least one microservice running (e.g., GPU service)
+- Microservice database created (if testing that service)
+
+### Start Gateway (Development)
+
+```bash
+python -m api_gateway.main
+```
+
+### Health Check
+
+```bash
+curl http://localhost:8080/health
+```
+
+Expected response:
+```json
+{"status": "healthy", "service": "api-gateway"}
+```
+
+### Service Registry
+
+```bash
+curl http://localhost:8080/services
+```
+
+Expected response:
+```json
+{
+  "services": [
+    {
+      "name": "gpu",
+      "url": "http://localhost:8101",
+      "health_check": "/health",
+      "routes": ["/gpu/*"]
+    },
+    {
+      "name": "marketplace",
+      "url": "http://localhost:8102",
+      "health_check": "/health",
+      "routes": ["/marketplace/*"]
+    },
+    {
+      "name": "trading",
+      "url": "http://localhost:8104",
+      "health_check": "/health",
+      "routes": ["/trading/*"]
+    },
+    {
+      "name": "governance",
+      "url": "http://localhost:8105",
+      "health_check": "/health",
+      "routes": ["/governance/*"]
+    }
+  ]
+}
+```
+
+### Test Routing to GPU Service
+
+1. Start GPU service in another terminal:
+   ```bash
+   python -m gpu_service.main
+   ```
+
+2. Test through gateway:
+   ```bash
+   curl http://localhost:8080/gpu/health
+   ```
+
+Expected response:
+```json
+{"status": "healthy", "service": "gpu-service"}
+```
+
+### Test Routing to Marketplace Service
+
+1. Start Marketplace service in another terminal:
+   ```bash
+   python -m marketplace_service.main
+   ```
+
+2. Test through gateway:
+   ```bash
+   curl http://localhost:8080/marketplace/health
+   ```
+
+Expected response:
+```json
+{"status": "healthy", "service": "marketplace-service"}
+```
+
+### Test Routing to Trading Service
+
+1. Start Trading service in another terminal:
+   ```bash
+   python -m trading_service.main
+   ```
+
+2. Test through gateway:
+   ```bash
+   curl http://localhost:8080/trading/health
+   ```
+
+Expected response:
+```json
+{"status": "healthy", "service": "trading-service"}
+```
+
+### Test Routing to Governance Service
+
+1. Start Governance service in another terminal:
+   ```bash
+   python -m governance_service.main
+   ```
+
+2. Test through gateway:
+   ```bash
+   curl http://localhost:8080/governance/health
+   ```
+
+Expected response:
+```json
+{"status": "healthy", "service": "governance-service"}
+```
+
+For comprehensive testing procedures, see [MICROSERVICES_TESTING_GUIDE.md](../docs/MICROSERVICES_TESTING_GUIDE.md).
+
 ## Endpoints
 
 - `GET /health` - Health check

apps/docs/MICROSERVICES_TESTING_GUIDE.md

@@ -0,0 +1,535 @@
+# AITBC Microservices Testing Guide
+
+This guide provides comprehensive testing procedures for the AITBC microservices architecture following the Coordinator-API monolith breakup.
+
+## Table of Contents
+
+- [Prerequisites](#prerequisites)
+- [Database Setup](#database-setup)
+- [Service Installation](#service-installation)
+- [Service Startup](#service-startup)
+- [Individual Service Testing](#individual-service-testing)
+- [Gateway Integration Testing](#gateway-integration-testing)
+- [Expected Test Outcomes](#expected-test-outcomes)
+- [Troubleshooting](#troubleshooting)
+
+## Prerequisites
+
+### System Requirements
+
+- Python 3.13+
+- PostgreSQL 14+
+- Poetry (for dependency management)
+- Systemd (for production deployment)
+- curl or httpie (for testing endpoints)
+
+### Environment Setup
+
+```bash
+# Ensure you're in the AITBC root directory
+cd /opt/aitbc
+
+# Install dependencies
+poetry install
+
+# Verify PostgreSQL is running
+sudo systemctl status postgresql
+```
+
+## Database Setup
+
+Each microservice requires a separate database. Run the setup scripts for each service you want to test.
+
+### GPU Service Database
+
+```bash
+sudo -u postgres psql -f apps/gpu-service/scripts/setup-database.sql
+```
+
+Expected output:
+```
+CREATE DATABASE
+CREATE ROLE
+GRANT
+```
+
+### Marketplace Service Database
+
+```bash
+sudo -u postgres psql -f apps/marketplace-service/scripts/setup-database.sql
+```
+
+### Trading Service Database
+
+```bash
+sudo -u postgres psql -f apps/trading-service/scripts/setup-database.sql
+```
+
+### Governance Service Database
+
+```bash
+sudo -u postgres psql -f apps/governance-service/scripts/setup-database.sql
+```
+
+### Verify Database Creation
+
+```bash
+sudo -u postgres psql -l
+```
+
+Expected databases:
+- aitbc_gpu
+- aitbc_marketplace
+- aitbc_trading
+- aitbc_governance
+
+## Service Installation
+
+Install each service using Poetry:
+
+```bash
+# GPU Service
+poetry install --with gpu-service
+
+# Marketplace Service
+poetry install --with marketplace-service
+
+# Trading Service
+poetry install --with trading-service
+
+# Governance Service
+poetry install --with governance-service
+
+# API Gateway
+poetry install --with api-gateway
+```
+
+## Service Startup
+
+### Development Mode (Manual Startup)
+
+Start services manually in separate terminal windows:
+
+```bash
+# Terminal 1: GPU Service
+cd /opt/aitbc
+python -m gpu_service.main
+
+# Terminal 2: Marketplace Service
+cd /opt/aitbc
+python -m marketplace_service.main
+
+# Terminal 3: Trading Service
+cd /opt/aitbc
+python -m trading_service.main
+
+# Terminal 4: Governance Service
+cd /opt/aitbc
+python -m governance_service.main
+
+# Terminal 5: API Gateway
+cd /opt/aitbc
+python -m api_gateway.main
+```
+
+### Production Mode (Systemd)
+
+For production deployment, use systemd services:
+
+```bash
+# Copy service files to systemd
+sudo cp apps/gpu-service/gpu-service.service /etc/systemd/system/
+sudo cp apps/marketplace-service/marketplace-service.service /etc/systemd/system/
+sudo cp apps/trading-service/trading-service.service /etc/systemd/system/
+sudo cp apps/governance-service/governance-service.service /etc/systemd/system/
+sudo cp apps/api-gateway/api-gateway.service /etc/systemd/system/
+
+# Reload systemd
+sudo systemctl daemon-reload
+
+# Enable and start services
+sudo systemctl enable gpu-service marketplace-service trading-service governance-service api-gateway
+sudo systemctl start gpu-service marketplace-service trading-service governance-service api-gateway
+
+# Check service status
+sudo systemctl status gpu-service
+sudo systemctl status marketplace-service
+sudo systemctl status trading-service
+sudo systemctl status governance-service
+sudo systemctl status api-gateway
+```
+
+## Individual Service Testing
+
+### GPU Service Testing
+
+#### Health Check
+
+```bash
+curl http://localhost:8101/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "gpu-service"
+}
+```
+
+#### GPU Status
+
+```bash
+curl http://localhost:8101/gpu/status
+```
+
+Expected response:
+```json
+{
+  "status": "operational",
+  "service": "gpu-service",
+  "message": "GPU service is running"
+}
+```
+
+#### Get Consumer GPU Profiles
+
+```bash
+curl http://localhost:8101/v1/marketplace/edge-gpu/profiles
+```
+
+Expected response:
+```json
+[
+  {
+    "profile_id": "consumer_nvidia_a100",
+    "name": "NVIDIA A100",
+    "architecture": "NVIDIA",
+    "memory_gb": 80,
+    "cuda_cores": 6912,
+    "tensor_cores": 432,
+    "compute_capability": "8.0",
+    "typical_use_cases": ["ml_training", "inference", "hpc"]
+  }
+]
+```
+
+### Marketplace Service Testing
+
+#### Health Check
+
+```bash
+curl http://localhost:8102/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "marketplace-service"
+}
+```
+
+#### Get Marketplace Offers
+
+```bash
+curl http://localhost:8102/v1/marketplace/offers
+```
+
+Expected response:
+```json
+[]
+```
+
+### Trading Service Testing
+
+#### Health Check
+
+```bash
+curl http://localhost:8104/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "trading-service"
+}
+```
+
+#### Get Trade Requests
+
+```bash
+curl http://localhost:8104/v1/trading/requests
+```
+
+Expected response:
+```json
+[]
+```
+
+### Governance Service Testing
+
+#### Health Check
+
+```bash
+curl http://localhost:8105/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "governance-service"
+}
+```
+
+#### Get Governance Proposals
+
+```bash
+curl http://localhost:8105/v1/governance/proposals
+```
+
+Expected response:
+```json
+[]
+```
+
+## Gateway Integration Testing
+
+### Gateway Health Check
+
+```bash
+curl http://localhost:8080/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "api-gateway"
+}
+```
+
+### Gateway Service Registry
+
+```bash
+curl http://localhost:8080/services
+```
+
+Expected response:
+```json
+{
+  "services": [
+    {
+      "name": "gpu",
+      "url": "http://localhost:8101",
+      "health_check": "/health",
+      "routes": ["/gpu/*"]
+    },
+    {
+      "name": "marketplace",
+      "url": "http://localhost:8102",
+      "health_check": "/health",
+      "routes": ["/marketplace/*"]
+    },
+    {
+      "name": "trading",
+      "url": "http://localhost:8104",
+      "health_check": "/health",
+      "routes": ["/trading/*"]
+    },
+    {
+      "name": "governance",
+      "url": "http://localhost:8105",
+      "health_check": "/health",
+      "routes": ["/governance/*"]
+    }
+  ]
+}
+```
+
+### Test GPU Service Through Gateway
+
+```bash
+curl http://localhost:8080/gpu/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "gpu-service"
+}
+```
+
+```bash
+curl http://localhost:8080/gpu/v1/marketplace/edge-gpu/profiles
+```
+
+Expected response:
+```json
+[
+  {
+    "profile_id": "consumer_nvidia_a100",
+    "name": "NVIDIA A100",
+    "architecture": "NVIDIA",
+    "memory_gb": 80,
+    "cuda_cores": 6912,
+    "tensor_cores": 432,
+    "compute_capability": "8.0",
+    "typical_use_cases": ["ml_training", "inference", "hpc"]
+  }
+]
+```
+
+### Test Marketplace Service Through Gateway
+
+```bash
+curl http://localhost:8080/marketplace/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "marketplace-service"
+}
+```
+
+### Test Trading Service Through Gateway
+
+```bash
+curl http://localhost:8080/trading/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "trading-service"
+}
+```
+
+### Test Governance Service Through Gateway
+
+```bash
+curl http://localhost:8080/governance/health
+```
+
+Expected response:
+```json
+{
+  "status": "healthy",
+  "service": "governance-service"
+}
+```
+
+## Expected Test Outcomes
+
+### Successful Test Indicators
+
+- All health endpoints return `{"status": "healthy", "service": "<service-name>"}`
+- Gateway successfully proxies requests to microservices
+- Service registry shows all registered services
+- Database connections are established without errors
+- No 500 errors or connection refused messages
+
+### Failure Indicators
+
+- Connection refused errors (service not running)
+- 500 internal server errors (service error)
+- Gateway routing failures (incorrect configuration)
+- Database connection errors (database not set up)
+
+## Troubleshooting
+
+### Service Won't Start
+
+**Issue:** Service fails to start with database connection error
+
+**Solution:**
+```bash
+# Verify database exists
+sudo -u postgres psql -l
+
+# Recreate database if needed
+sudo -u postgres psql -f apps/<service-name>/scripts/setup-database.sql
+
+# Check service logs
+sudo journalctl -u <service-name> -n 50
+```
+
+### Gateway Can't Reach Service
+
+**Issue:** Gateway returns connection refused when proxying to service
+
+**Solution:**
+```bash
+# Verify service is running
+curl http://localhost:<port>/health
+
+# Check gateway service registry
+curl http://localhost:8080/services
+
+# Verify service URL in gateway configuration
+```
+
+### Database Connection Errors
+
+**Issue:** Service fails with "could not connect to server" error
+
+**Solution:**
+```bash
+# Verify PostgreSQL is running
+sudo systemctl status postgresql
+
+# Check database credentials in service file
+# Default: aitbc_<service>:password@localhost:5432/aitbc_<service>
+
+# Test database connection manually
+sudo -u postgres psql -d aitbc_<service>
+```
+
+### Port Already in Use
+
+**Issue:** Service fails to start with "Address already in use" error
+
+**Solution:**
+```bash
+# Find process using the port
+sudo lsof -i :<port>
+
+# Kill the process if needed
+sudo kill <pid>
+
+# Or use a different port in the service configuration
+```
+
+## Testing Checklist
+
+Before considering testing complete:
+
+- [ ] All databases created and accessible
+- [ ] All services installed via Poetry
+- [ ] All services start successfully in development mode
+- [ ] All services start successfully in production mode (systemd)
+- [ ] Health endpoints return healthy status for all services
+- [ ] Gateway health check passes
+- [ ] Gateway service registry shows all services
+- [ ] Gateway successfully proxies requests to GPU service
+- [ ] Gateway successfully proxies requests to Marketplace service
+- [ ] Gateway successfully proxies requests to Trading service
+- [ ] Gateway successfully proxies requests to Governance service
+- [ ] No 500 errors in service logs
+- [ ] No connection errors in gateway logs
+
+## Next Steps
+
+After successful testing:
+
+1. Deploy services to production environment
+2. Configure monitoring and alerting
+3. Set up log aggregation
+4. Implement automated testing in CI/CD pipeline
+5. Gradually migrate traffic from coordinator-api to microservices

apps/governance-service/README.md

@@ -53,24 +53,90 @@ sudo systemctl enable governance-service
 
 ## Testing
 
-To test the governance service end-to-end with the gateway:
+### Prerequisites
 
-1. Start the governance service:
+- PostgreSQL running and aitbc_governance database created
-   ```bash
+- Poetry dependencies installed
-   python -m governance_service.main
-   ```
 
-2. Start the API gateway:
+### Database Setup
+
+```bash
+sudo -u postgres psql -f scripts/setup-database.sql
+```
+
+### Start Service (Development)
+
+```bash
+python -m governance_service.main
+```
+
+### Health Check
+
+```bash
+curl http://localhost:8105/health
+```
+
+Expected response:
+```json
+{"status": "healthy", "service": "governance-service"}
+```
+
+### Governance Status
+
+```bash
+curl http://localhost:8105/governance/status
+```
+
+Expected response:
+```json
+{
+  "status": "operational",
+  "service": "governance-service",
+  "message": "Governance service is running"
+}
+```
+
+### Get Governance Proposals
+
+```bash
+curl http://localhost:8105/v1/governance/proposals
+```
+
+Expected response:
+```json
+[]
+```
+
+### Create Governance Proposal
+
+```bash
+curl -X POST http://localhost:8105/v1/governance/proposals \
+  -H "Content-Type: application/json" \
+  -d '{
+    "proposer_id": "test_proposer",
+    "title": "Test Proposal",
+    "description": "Test description",
+    "category": "general",
+    "voting_starts": "2026-05-01T00:00:00Z",
+    "voting_ends": "2026-05-08T00:00:00Z"
+  }'
+```
+
+### Test Through Gateway
+
+1. Start the API gateway:
    ```bash
    python -m api_gateway.main
    ```
 
-3. Test governance endpoints through the gateway:
+2. Test governance endpoints through the gateway:
    ```bash
-   curl http://localhost:8080/governance/v1/governance/proposals
    curl http://localhost:8080/governance/health
+   curl http://localhost:8080/governance/v1/governance/proposals
    ```
 
+For comprehensive testing procedures, see [MICROSERVICES_TESTING_GUIDE.md](../docs/MICROSERVICES_TESTING_GUIDE.md).
+
 ## Service Configuration
 
 - Port: 8105

apps/gpu-service/README.md

@@ -47,24 +47,86 @@ sudo systemctl enable gpu-service
 
 ## Testing
 
-To test the GPU service end-to-end with the gateway:
+### Prerequisites
 
-1. Start the GPU service:
+- PostgreSQL running and aitbc_gpu database created
-   ```bash
+- Poetry dependencies installed
-   python -m gpu_service.main
-   ```
 
-2. Start the API gateway:
+### Database Setup
+
+```bash
+sudo -u postgres psql -f scripts/setup-database.sql
+```
+
+### Start Service (Development)
+
+```bash
+python -m gpu_service.main
+```
+
+### Health Check
+
+```bash
+curl http://localhost:8101/health
+```
+
+Expected response:
+```json
+{"status": "healthy", "service": "gpu-service"}
+```
+
+### GPU Status
+
+```bash
+curl http://localhost:8101/gpu/status
+```
+
+Expected response:
+```json
+{
+  "status": "operational",
+  "service": "gpu-service",
+  "message": "GPU service is running"
+}
+```
+
+### Get Consumer GPU Profiles
+
+```bash
+curl http://localhost:8101/v1/marketplace/edge-gpu/profiles
+```
+
+Expected response:
+```json
+[
+  {
+    "profile_id": "consumer_nvidia_a100",
+    "name": "NVIDIA A100",
+    "architecture": "NVIDIA",
+    "memory_gb": 80,
+    "cuda_cores": 6912,
+    "tensor_cores": 432,
+    "compute_capability": "8.0",
+    "typical_use_cases": ["ml_training", "inference", "hpc"]
+  }
+]
+```
+
+### Test Through Gateway
+
+1. Start the API gateway:
    ```bash
    python -m api_gateway.main
    ```
 
-3. Test GPU endpoints through the gateway:
+2. Test GPU endpoints through the gateway:
    ```bash
-   curl http://localhost:8080/gpu/v1/marketplace/edge-gpu/profiles
    curl http://localhost:8080/gpu/health
+   curl http://localhost:8080/gpu/v1/marketplace/edge-gpu/profiles
    ```
 
+For comprehensive testing procedures, see [MICROSERVICES_TESTING_GUIDE.md](../docs/MICROSERVICES_TESTING_GUIDE.md).
+
 ## Service Configuration
 
 - Port: 8101

apps/marketplace-service/README.md

@@ -49,24 +49,88 @@ sudo systemctl enable marketplace-service
 
 ## Testing
 
-To test the marketplace service end-to-end with the gateway:
+### Prerequisites
 
-1. Start the marketplace service:
+- PostgreSQL running and aitbc_marketplace database created
-   ```bash
+- Poetry dependencies installed
-   python -m marketplace_service.main
-   ```
 
-2. Start the API gateway:
+### Database Setup
+
+```bash
+sudo -u postgres psql -f scripts/setup-database.sql
+```
+
+### Start Service (Development)
+
+```bash
+python -m marketplace_service.main
+```
+
+### Health Check
+
+```bash
+curl http://localhost:8102/health
+```
+
+Expected response:
+```json
+{"status": "healthy", "service": "marketplace-service"}
+```
+
+### Marketplace Status
+
+```bash
+curl http://localhost:8102/marketplace/status
+```
+
+Expected response:
+```json
+{
+  "status": "operational",
+  "service": "marketplace-service",
+  "message": "Marketplace service is running"
+}
+```
+
+### Get Marketplace Offers
+
+```bash
+curl http://localhost:8102/v1/marketplace/offers
+```
+
+Expected response:
+```json
+[]
+```
+
+### Create Marketplace Offer
+
+```bash
+curl -X POST http://localhost:8102/v1/marketplace/offers \
+  -H "Content-Type: application/json" \
+  -d '{
+    "provider": "test_provider",
+    "capacity": 100,
+    "price": 0.5,
+    "gpu_model": "NVIDIA A100"
+  }'
+```
+
+### Test Through Gateway
+
+1. Start the API gateway:
    ```bash
    python -m api_gateway.main
    ```
 
-3. Test marketplace endpoints through the gateway:
+2. Test marketplace endpoints through the gateway:
    ```bash
-   curl http://localhost:8080/marketplace/v1/marketplace/offers
    curl http://localhost:8080/marketplace/health
+   curl http://localhost:8080/marketplace/v1/marketplace/offers
    ```
 
+For comprehensive testing procedures, see [MICROSERVICES_TESTING_GUIDE.md](../docs/MICROSERVICES_TESTING_GUIDE.md).
+
 ## Service Configuration
 
 - Port: 8102

apps/trading-service/README.md

@@ -51,24 +51,88 @@ sudo systemctl enable trading-service
 
 ## Testing
 
-To test the trading service end-to-end with the gateway:
+### Prerequisites
 
-1. Start the trading service:
+- PostgreSQL running and aitbc_trading database created
-   ```bash
+- Poetry dependencies installed
-   python -m trading_service.main
-   ```
 
-2. Start the API gateway:
+### Database Setup
+
+```bash
+sudo -u postgres psql -f scripts/setup-database.sql
+```
+
+### Start Service (Development)
+
+```bash
+python -m trading_service.main
+```
+
+### Health Check
+
+```bash
+curl http://localhost:8104/health
+```
+
+Expected response:
+```json
+{"status": "healthy", "service": "trading-service"}
+```
+
+### Trading Status
+
+```bash
+curl http://localhost:8104/trading/status
+```
+
+Expected response:
+```json
+{
+  "status": "operational",
+  "service": "trading-service",
+  "message": "Trading service is running"
+}
+```
+
+### Get Trade Requests
+
+```bash
+curl http://localhost:8104/v1/trading/requests
+```
+
+Expected response:
+```json
+[]
+```
+
+### Create Trade Request
+
+```bash
+curl -X POST http://localhost:8104/v1/trading/requests \
+  -H "Content-Type: application/json" \
+  -d '{
+    "buyer_agent_id": "test_buyer",
+    "trade_type": "compute_resources",
+    "title": "Test Trade Request",
+    "description": "Test description"
+  }'
+```
+
+### Test Through Gateway
+
+1. Start the API gateway:
    ```bash
    python -m api_gateway.main
    ```
 
-3. Test trading endpoints through the gateway:
+2. Test trading endpoints through the gateway:
    ```bash
-   curl http://localhost:8080/trading/v1/trading/requests
    curl http://localhost:8080/trading/health
+   curl http://localhost:8080/trading/v1/trading/requests
    ```
 
+For comprehensive testing procedures, see [MICROSERVICES_TESTING_GUIDE.md](../docs/MICROSERVICES_TESTING_GUIDE.md).
+
 ## Service Configuration
 
 - Port: 8104