aitbc/docs/testing/e2e-test-plan.md

AITBC End-to-End Test Plan

Version: 1.0 Date: 2026-05-11 Status: Draft Purpose: Define comprehensive end-to-end testing strategy for AITBC platform

Current State

Existing Test Infrastructure

Integration Tests Location: /opt/aitbc/tests/integration/

Existing Test Files:

• test_full_workflow.py - Integration tests for job execution, payment flow, P2P sync, marketplace, security
• test_agent_coordinator.py - Agent coordinator integration tests (141KB)
• test_agent_coordinator_api.py - Agent coordinator API tests
• test_blockchain_nodes.py - Blockchain node integration tests
• test_staking_lifecycle.py - Staking lifecycle tests
• test_working_integration.py - Working integration tests
• test_basic_integration.py - Basic integration tests
• test_blockchain_simple.py - Simple blockchain tests
• test_blockchain_final.py - Final blockchain tests
• test_integration_simple.py - Simple integration tests

Current Limitations

1. Mock Clients: Most integration tests use mock clients rather than real services
2. Service Dependencies: Tests require running services (coordinator, blockchain, marketplace, wallet)
3. No True E2E: Tests don't span full user journeys from registration to completion
4. Environment Setup: No dedicated E2E test environment configuration
5. Test Data: No comprehensive test data fixtures for E2E scenarios

E2E Test Scope

Test Scenarios

1. User Registration and Wallet Creation

Objective: Verify complete user onboarding flow

Steps:

1. User registers via CLI
2. Wallet is created automatically
3. Private key is generated and stored securely
4. User receives wallet address
5. User can view wallet balance

Success Criteria:

• Registration completes without errors
• Wallet is created with valid address
• Private key is securely stored
• User can access wallet information

Prerequisites:

• Coordinator API running
• Wallet daemon running
• Blockchain node running

2. Job Submission and Processing

Objective: Verify complete job lifecycle

Steps:

1. User submits AI inference job via CLI
2. Job is queued in coordinator
3. Miner picks up job via polling
4. Miner processes job (GPU execution)
5. Miner submits result
6. User receives result
7. Payment is processed

Success Criteria:

• Job is successfully submitted
• Job transitions through states: QUEUED → ASSIGNED → PROCESSING → COMPLETED
• Result is returned to user
• Payment is transferred correctly
• Transaction is recorded on blockchain

Prerequisites:

• Coordinator API running
• GPU miner running
• Wallet daemon running
• Blockchain node running
• Marketplace service running

3. Payment and Receipt Generation

Objective: Verify payment flow and receipt generation

Steps:

1. Job is submitted with payment
2. Payment is escrowed in smart contract
3. Job completes successfully
4. Payment is released to miner
5. Receipt is generated
6. Receipt is stored on blockchain
7. User can retrieve receipt

Success Criteria:

• Payment is escrowed correctly
• Payment is released on job completion
• Receipt is generated with all required fields
• Receipt is stored on blockchain
• User can retrieve and verify receipt

Prerequisites:

• Coordinator API running
• Wallet daemon running
• Blockchain node running
• Smart contracts deployed

4. Miner Registration and Operation

Objective: Verify miner onboarding and operation

Steps:

1. Miner registers with coordinator
2. Miner provides GPU capabilities
3. Miner creates marketplace offer
4. Miner receives jobs
5. Miner processes jobs
6. Miner receives payments
7. Miner updates capabilities

Success Criteria:

• Miner registration succeeds
• Capabilities are recorded correctly
• Marketplace offer is created
• Jobs are assigned to miner
• Payments are received
• Capability updates succeed

Prerequisites:

• Coordinator API running
• GPU miner running
• Wallet daemon running
• Marketplace service running
• Blockchain node running

5. Agent Communication

Objective: Verify agent-to-agent communication

Steps:

1. Agent A registers with coordinator
2. Agent B registers with coordinator
3. Agent A sends message to Agent B
4. Agent B receives message
5. Agent B responds
6. Communication is encrypted
7. Message is logged on blockchain

Success Criteria:

• Agents register successfully
• Message is delivered
• Response is received
• Communication is encrypted
• Message is recorded on blockchain

Prerequisites:

• Coordinator API running
• Agent daemon running
• Blockchain node running
• Smart contracts deployed

6. Blockchain Transactions

Objective: Verify blockchain transaction processing

Steps:

1. User initiates transaction
2. Transaction is submitted to blockchain
3. Transaction is validated
4. Transaction is included in block
5. Block is propagated to network
6. Transaction is confirmed
7. Receipt is generated

Success Criteria:

• Transaction is submitted successfully
• Transaction is validated
• Transaction is included in block
• Block is propagated
• Transaction is confirmed
• Receipt is generated

Prerequisites:

• Blockchain node running (multiple nodes for network testing)
• Wallet daemon running
• Consensus mechanism operational

7. API Interactions

Objective: Verify API contract compliance

Steps:

1. Test all API endpoints
2. Verify request/response formats
3. Test authentication/authorization
4. Test error handling
5. Test rate limiting
6. Test pagination
7. Test filtering and sorting

Success Criteria:

• All endpoints respond correctly
• Request/response formats match API spec
• Authentication/authorization works correctly
• Errors are handled appropriately
• Rate limiting is enforced
• Pagination works correctly
• Filtering and sorting work correctly

Prerequisites:

• All API services running
• API documentation available
• Test users with different roles

Test Environment Setup

Infrastructure Requirements

Hardware

• Minimum: 4 CPU cores, 16GB RAM, 100GB storage
• Recommended: 8 CPU cores, 32GB RAM, 500GB storage
• GPU: NVIDIA GPU with CUDA support (for miner testing)

Software

• Operating System: Debian stable (bookworm)
• Python: 3.13 or 3.14
• PostgreSQL: 15 or later
• Redis: 7 or later

Services Required

Service	Port	Purpose	Status
Coordinator API	8011	Job management	Required
Blockchain Node	8080	Blockchain RPC	Required
Wallet Daemon	8081	Wallet management	Required
GPU Miner	-	Job processing	Required
Marketplace	8102	Service marketplace	Required
Exchange	8082	Trading platform	Required
Agent Coordinator	8011	Agent management	Required
PostgreSQL	5432	Database	Required
Redis	6379	Cache	Required

Service Orchestration (Systemd)

AITBC uses systemd for service orchestration. Services are managed via systemd unit files.

Starting Services

# Start PostgreSQL
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Start Redis
sudo systemctl start redis
sudo systemctl enable redis

# Start AITBC services
sudo systemctl start aitbc-blockchain-node
sudo systemctl start aitbc-coordinator-api
sudo systemctl start aitbc-marketplace
sudo systemctl start aitbc-exchange

# Check service status
sudo systemctl status aitbc-blockchain-node
sudo systemctl status aitbc-coordinator-api
sudo systemctl status aitbc-marketplace

Service Health Checks

# Check coordinator API
curl -s http://localhost:8011/v1/health

# Check blockchain node
curl -s http://localhost:8080/v1/health

# Check marketplace
curl -s http://localhost:8102/v1/health

Configuration

Environment Variables

# Coordinator API
COORDINATOR_URL=http://localhost:8011
CLIENT_API_KEY=test-api-key
ADMIN_API_KEY=test-admin-key

# Blockchain
BLOCKCHAIN_URL=http://localhost:8080
BLOCKCHAIN_DATA_DIR=/tmp/blockchain-test

# Wallet
WALLET_DAEMON_URL=http://localhost:8081
WALLET_DATA_DIR=/tmp/wallet-test

# Marketplace
MARKETPLACE_URL=http://localhost:8102

# Database
POSTGRES_URL=postgresql://aitbc:test@localhost:5432/aitbc_test
REDIS_URL=redis://localhost:6379/0

Test Data

• Test users with different roles
• Test wallets with pre-funded accounts
• Test blockchain state (genesis block)
• Test marketplace offers
• Test job templates

Test Implementation

Test Framework

Recommended Framework: pytest with pytest-asyncio

Additional Tools:

• httpx for HTTP client
• playwright for browser automation (if UI testing needed)
• docker-compose for service orchestration
• testcontainers for containerized test dependencies

Test Structure

tests/e2e/
├── conftest.py                 # E2E fixtures and configuration
├── test_user_registration.py   # User registration E2E tests
├── test_job_lifecycle.py       # Job submission and processing E2E tests
├── test_payment_flow.py        # Payment and receipt E2E tests
├── test_miner_operations.py    # Miner registration and operation E2E tests
├── test_agent_communication.py # Agent communication E2E tests
├── test_blockchain_transactions.py # Blockchain transaction E2E tests
├── test_api_compliance.py      # API contract compliance E2E tests
├── fixtures/                   # Test data fixtures
│   ├── users.py
│   ├── wallets.py
│   ├── jobs.py
│   └── blockchain.py
└── utils/                      # Test utilities
    ├── helpers.py
    └── assertions.py

Sample Test Structure

import pytest
import httpx
from datetime import datetime, timedelta

@pytest.mark.e2e
class TestJobLifecycle:
    """End-to-end test for complete job lifecycle"""
    
    @pytest.fixture
    async def client(self):
        """HTTP client for API calls"""
        async with httpx.AsyncClient() as client:
            yield client
    
    async def test_complete_job_execution(self, client):
        """Test complete job from submission to completion"""
        # 1. Submit job
        job_data = {
            "payload": {
                "job_type": "ai_inference",
                "parameters": {
                    "model": "gpt-4",
                    "prompt": "Test prompt",
                    "max_tokens": 100
                }
            },
            "ttl_seconds": 900
        }
        
        response = await client.post(
            "http://localhost:8011/v1/jobs",
            json=job_data,
            headers={"X-Api-Key": "test-api-key"}
        )
        assert response.status_code == 201
        job = response.json()
        job_id = job["job_id"]
        
        # 2. Wait for job assignment
        await self._wait_for_job_state(client, job_id, "ASSIGNED", timeout=60)
        
        # 3. Wait for job completion
        await self._wait_for_job_state(client, job_id, "COMPLETED", timeout=300)
        
        # 4. Verify result
        response = await client.get(
            f"http://localhost:8011/v1/jobs/{job_id}",
            headers={"X-Api-Key": "test-api-key"}
        )
        assert response.status_code == 200
        job = response.json()
        assert job["state"] == "COMPLETED"
        assert "result" in job
        
        # 5. Verify payment
        response = await client.get(
            f"http://localhost:8011/v1/jobs/{job_id}/payment",
            headers={"X-Api-Key": "test-api-key"}
        )
        assert response.status_code == 200
        payment = response.json()
        assert payment["status"] == "completed"
    
    async def _wait_for_job_state(self, client, job_id, expected_state, timeout):
        """Wait for job to reach expected state"""
        start = datetime.now()
        while (datetime.now() - start).total_seconds() < timeout:
            response = await client.get(
                f"http://localhost:8011/v1/jobs/{job_id}",
                headers={"X-Api-Key": "test-api-key"}
            )
            job = response.json()
            if job["state"] == expected_state:
                return
            await asyncio.sleep(1)
        
        pytest.fail(f"Job did not reach state {expected_state} within {timeout}s")

Test Execution

Manual Execution

# Run all E2E tests
pytest tests/e2e/ -v --tb=short

# Run specific test suite
pytest tests/e2e/test_job_lifecycle.py -v

# Run with timeout
pytest tests/e2e/ --timeout=600 -v

Automated Execution

CI/CD Integration:

• Run E2E tests nightly
• Run E2E tests on release candidates
• Run E2E tests after major changes

GitHub Actions Example:

name: E2E Tests

on:
  schedule:
    - cron: '0 2 * * *'  # Daily at 2 AM UTC
  workflow_dispatch:

jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.13'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest pytest-asyncio httpx
      - name: Start services
        run: docker-compose -f docker-compose.test.yml up -d
      - name: Wait for services
        run: ./scripts/wait-for-services.sh
      - name: Run E2E tests
        run: pytest tests/e2e/ -v --tb=short
      - name: Stop services
        if: always()
        run: docker-compose -f docker-compose.test.yml down

Test Data Management

Fixtures

User Fixtures:

• Regular user
• Admin user
• Miner user
• Agent user

Wallet Fixtures:

• Pre-funded wallets
• Empty wallets
• Wallets with staked tokens

Job Fixtures:

• Simple inference job
• Complex inference job
• Confidential job
• Batch jobs

Blockchain Fixtures:

• Genesis block
• Pre-populated accounts
• Sample transactions

Data Cleanup

Before Each Test:

• Reset database to known state
• Clear blockchain test data
• Reset cache

After Each Test:

• Clean up created resources
• Reset service states
• Verify no data leaks

Reporting

Test Report Format

# E2E Test Report

**Date:** [Date]
**Environment:** [Environment]
**Test Suite:** [Suite Name]

## Summary
- Total Tests: [Count]
- Passed: [Count]
- Failed: [Count]
- Skipped: [Count]
- Duration: [Time]

## Results by Scenario
| Scenario | Tests | Passed | Failed | Duration |
|----------|-------|--------|--------|----------|
| [Scenario 1] | [Count] | [Count] | [Count] | [Time] |
| [Scenario 2] | [Count] | [Count] | [Count] | [Time] |

## Failed Tests
### [Test Name]
- **Error:** [Error message]
- **Stack Trace:** [Trace]
- **Logs:** [Relevant logs]

## Recommendations
- [Recommendation 1]
- [Recommendation 2]

Metrics to Track

• Test execution time
• Test success rate
• Test flakiness
• Resource utilization during tests
• Service restart count

Maintenance

Regular Updates

• Weekly: Review test failures and flakiness
• Monthly: Update test data and fixtures
• Quarterly: Review and update test scenarios
• Annually: Full test suite review and refactoring

Test Maintenance Checklist

• Tests are up to date with API changes
• Test data is relevant and current
• Test environment matches production
• Test execution time is acceptable
• Test coverage is adequate
• Test documentation is current

Risks and Mitigations

Risks

1. Service Dependencies: Tests depend on multiple services

   • Mitigation: Use docker-compose for service orchestration, implement service health checks

2. Test Data Management: Managing test data across tests

   • Mitigation: Implement robust fixture system, use database transactions for rollback

3. Test Execution Time: E2E tests can be slow

   • Mitigation: Parallelize tests where possible, use test selection for targeted testing

4. Environment Differences: Test environment may not match production

   • Mitigation: Use production-like configuration, regular environment audits

5. Test Flakiness: E2E tests can be flaky due to timing issues

   • Mitigation: Implement proper waits and retries, use idempotent operations

Success Criteria

Before Production Deployment

• All critical E2E test scenarios implemented
• Test success rate >95%
• Test execution time <30 minutes
• Test environment matches production configuration
• Test data is comprehensive and current
• Tests integrated into CI/CD pipeline
• Test documentation is complete and current

Ongoing

• Tests run at least daily
• Test failures are investigated and resolved
• Test suite is reviewed and updated quarterly
• Test metrics are tracked and reported

Next Steps

Immediate (1-2 weeks)

1. Set up E2E test environment
2. Implement service orchestration (docker-compose)
3. Create test fixtures
4. Implement critical test scenarios (job lifecycle, payment flow)

Short-term (1 month)

1. Implement remaining test scenarios
2. Integrate tests into CI/CD
3. Set up test reporting
4. Document test procedures

Long-term (3 months)

1. Optimize test execution time
2. Implement parallel test execution
3. Add UI testing (if applicable)
4. Implement test data management system

Appendix

A. Service Startup Order

1. PostgreSQL
2. Redis
3. Blockchain Node
4. Wallet Daemon
5. Coordinator API
6. Marketplace
7. Exchange
8. GPU Miner
9. Agent Coordinator

B. Test Data Examples

Sample User:

{
  "user_id": "test-user-001",
  "email": "test@example.com",
  "role": "user",
  "wallet_address": "ait1testuser001"
}

Sample Job:

{
  "job_id": "test-job-001",
  "job_type": "ai_inference",
  "parameters": {
    "model": "gpt-4",
    "prompt": "Test prompt",
    "max_tokens": 100
  },
  "state": "QUEUED",
  "payment_amount": 100,
  "payment_currency": "AITBC"
}

C. Troubleshooting

Service Won't Start:

• Check logs: docker-compose logs [service]
• Verify configuration: docker-compose config
• Check port conflicts: netstat -tulpn

Test Times Out:

• Check service health: curl http://localhost:[port]/health
• Verify service dependencies: docker-compose ps
• Check for resource exhaustion: htop

Test Fails Intermittently:

• Review test logs for timing issues
• Increase wait times in tests
• Implement retries for flaky operations
• Check for race conditions

Approval

Role	Name	Date	Signature
QA Lead
Engineering Lead
DevOps Lead