aitbc/docs/12_issues/89_test.md

Cross-Container Multi-Chain Test Scenario

📋 Connected Resources

Testing Skill

For comprehensive testing capabilities and automated test execution, see the AITBC Testing Skill:

/windsurf/skills/test

Test Workflow

For step-by-step testing procedures and troubleshooting, see:

/windsurf/workflows/test

Tests Folder

Complete test suite implementation located at:

tests/
├── cli/                    # CLI command testing
├── integration/            # Service integration testing
├── e2e/                   # End-to-end workflow testing
├── unit/                  # Unit component testing
├── contracts/             # Smart contract testing
├── performance/           # Performance and load testing
├── security/              # Security vulnerability testing
├── conftest.py           # Test configuration and fixtures
└── run_all_tests.sh      # Comprehensive test runner

Multi-Chain Registration & Cross-Site Synchronization

Objective

Test the new multi-chain capabilities across the live system where:

1. One single node instance hosts multiple independent chains (ait-devnet, ait-testnet, ait-healthchain)
2. Nodes across aitbc and aitbc1 correctly synchronize independent chains using their chain_id

Test Architecture

┌─────────────────┐    HTTP/8082     ┌─────────────────┐    HTTP/8082     ┌─────────────────┐
│   localhost     │ ◄──────────────► │     aitbc       │ ◄──────────────► │     aitbc1      │
│  (Test Client)  │   (Direct RPC)   │  (Primary Node) │   (P2P Gossip)   │ (Secondary Node)│
│                 │                  │                 │                  │                 │
│                 │                  │  • ait-devnet   │                  │  • ait-devnet   │
│                 │                  │  • ait-testnet  │                  │  • ait-testnet  │
│                 │                  │  • ait-healthch │                  │  • ait-healthch │
└─────────────────┘                  └─────────────────┘                  └─────────────────┘

Automated Test Execution

Using the Testing Skill

# Execute multi-chain tests using the testing skill
skill test

# Run specific multi-chain test scenarios
python -m pytest tests/integration/test_multichain.py -v

# Run all tests including multi-chain scenarios
./scripts/testing/run_all_tests.sh

Using CLI for Testing

# Test CLI connectivity to multi-chain endpoints
cd /home/oib/windsurf/aitbc/cli
source venv/bin/activate

# Test health endpoint
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key health

# Test multi-chain status
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain chains

Test Phase 1: Multi-Chain Live Verification

1.1 Check Multi-Chain Status on aitbc

# Verify multiple chains are active on aitbc node
curl -s "http://127.0.0.1:8000/v1/health" | jq .supported_chains

# Expected response:
# [
#   "ait-devnet",
#   "ait-testnet",
#   "ait-healthchain"
# ]

# Alternative using CLI
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain chains

1.2 Verify Independent Genesis Blocks

# Get genesis for devnet
curl -s "http://127.0.0.1:8082/rpc/blocks/0?chain_id=ait-devnet" | jq .hash

# Get genesis for testnet (should be different from devnet)
curl -s "http://127.0.0.1:8082/rpc/blocks/0?chain_id=ait-testnet" | jq .hash

# Get genesis for healthchain (should be different from others)
curl -s "http://127.0.0.1:8082/rpc/blocks/0?chain_id=ait-healthchain" | jq .hash

# Alternative using CLI
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain genesis --chain-id ait-devnet
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain genesis --chain-id ait-testnet
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain genesis --chain-id ait-healthchain

Test Phase 2: Isolated Transaction Processing

2.1 Submit Transaction to Specific Chain

# Submit TX to healthchain
curl -s -X POST "http://127.0.0.1:8082/rpc/sendTx?chain_id=ait-healthchain" \
  -H "Content-Type: application/json" \
  -d '{"sender":"alice","recipient":"bob","payload":{"data":"medical_record"},"nonce":1,"fee":0,"type":"TRANSFER"}'

# Expected response:
# {
#   "tx_hash": "0x..."
# }

# Alternative using CLI
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain send \
  --chain-id ait-healthchain \
  --from alice \
  --to bob \
  --data "medical_record" \
  --nonce 1

2.2 Verify Chain Isolation

# Check mempool on healthchain (should have 1 tx)
curl -s "http://127.0.0.1:8082/rpc/mempool?chain_id=ait-healthchain"

# Check mempool on devnet (should have 0 tx)
curl -s "http://127.0.0.1:8082/rpc/mempool?chain_id=ait-devnet"

# Alternative using CLI
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain mempool --chain-id ait-healthchain
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain mempool --chain-id ait-devnet

Test Phase 3: Cross-Site Multi-Chain Synchronization

3.1 Verify Sync to aitbc1

# Wait for block proposal (interval is 2s)
sleep 5

# Check block on aitbc (Primary)
curl -s "http://127.0.0.1:8082/rpc/head?chain_id=ait-healthchain" | jq .

# Check block on aitbc1 (Secondary) - Should match exactly
ssh aitbc1-cascade "curl -s \"http://127.0.0.1:8082/rpc/head?chain_id=ait-healthchain\"" | jq .

# Alternative using CLI
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key blockchain head --chain-id ait-healthchain

Test Phase 4: Automated Test Suite Execution

4.1 Run Complete Test Suite

# Execute all tests including multi-chain scenarios
./scripts/testing/run_all_tests.sh

# Run specific multi-chain integration tests
python -m pytest tests/integration/test_multichain.py -v

# Run CLI tests with multi-chain support
python -m pytest tests/cli/test_cli_integration.py -v

4.2 Test Result Validation

# Generate test coverage report
python -m pytest tests/ --cov=. --cov-report=html

# View test results
open htmlcov/index.html

# Check specific test results
python -m pytest tests/integration/test_multichain.py::TestMultiChain::test_chain_isolation -v

Integration with Test Framework

Test Configuration

The multi-chain tests integrate with the main test framework through:

• conftest.py: Shared test fixtures and configuration
• test_cli_integration.py: CLI integration testing
• test_integration/: Service integration tests
• run_all_tests.sh: Comprehensive test execution

Environment Setup

# Set up test environment for multi-chain testing
export PYTHONPATH="/home/oib/windsurf/aitbc/cli:/home/oib/windsurf/aitbc/packages/py/aitbc-core/src:/home/oib/windsurf/aitbc/packages/py/aitbc-crypto/src:/home/oib/windsurf/aitbc/packages/py/aitbc-sdk/src:/home/oib/windsurf/aitbc/apps/coordinator-api/src:$PYTHONPATH"
export TEST_MODE=true
export TEST_DATABASE_URL="sqlite:///:memory:"
export _AITBC_NO_RICH=1

Mock Services

The test framework provides comprehensive mocking for:

• HTTP Clients: httpx.Client mocking for API calls
• Blockchain Services: Mock blockchain responses
• Multi-Chain Coordination: Mock chain synchronization
• Cross-Site Communication: Mock P2P gossip

Test Automation

Continuous Integration

# Automated test execution in CI/CD
name: Multi-Chain Tests
on: [push, pull_request]
jobs:
  multichain:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run Multi-Chain Tests
        run: |
          python -m pytest tests/integration/test_multichain.py -v
          python -m pytest tests/cli/test_cli_integration.py -v

Scheduled Testing

# Regular multi-chain test execution
0 2 * * * cd /home/oib/windsurf/aitbc && ./scripts/testing/run_all_tests.sh

Troubleshooting

Common Issues

• Connection Refused: Check if coordinator API is running
• Chain Not Found: Verify chain configuration
• Sync Failures: Check P2P network connectivity
• Test Failures: Review test logs and configuration

Debug Mode

# Run tests with debug output
python -m pytest tests/integration/test_multichain.py -v -s --tb=long

# Run specific test with debugging
python -m pytest tests/integration/test_multichain.py::TestMultiChain::test_chain_isolation -v -s --pdb

Service Status

# Check coordinator API status
curl -s "http://127.0.0.1:8000/v1/health"

# Check blockchain node status
curl -s "http://127.0.0.1:8082/rpc/status"

# Check CLI connectivity
python -m aitbc_cli --url http://127.0.0.1:8000 --api-key test-key health

Test Results and Reporting

Success Criteria

• ✅ All chains are active and accessible
• ✅ Independent genesis blocks for each chain
• ✅ Chain isolation is maintained
• ✅ Cross-site synchronization works correctly
• ✅ CLI commands work with multi-chain setup

Failure Analysis

• Connection Issues: Network connectivity problems
• Configuration Errors: Incorrect chain setup
• Synchronization Failures: P2P network issues
• CLI Errors: Command-line interface problems

Performance Metrics

• Test Execution Time: <5 minutes for full suite
• Chain Sync Time: <10 seconds for block propagation
• CLI Response Time: <200ms for command execution
• API Response Time: <100ms for health checks

Future Enhancements

Planned Improvements

• Visual Testing: Multi-chain visualization
• Load Testing: High-volume transaction testing
• Chaos Testing: Network partition testing
• Performance Testing: Scalability testing

Integration Points

• Monitoring: Real-time test monitoring
• Alerting: Test failure notifications
• Dashboard: Test result visualization
• Analytics: Test trend analysis