aitbc/.windsurf/workflows/archive/test-ai-operations.md

---
description: AI job submission, processing, and resource management testing module
title: AI Operations Testing Module
version: 1.0
---

# AI Operations Testing Module

This module covers AI job submission, processing, resource management, and AI service integration testing.

## Prerequisites

### Required Setup
- Working directory: `/opt/aitbc`
- Virtual environment: `/opt/aitbc/venv`
- CLI wrapper: `/opt/aitbc/aitbc-cli`
- Services running (Coordinator, Exchange, Blockchain RPC, Ollama)
- Basic Testing Module completed

### Environment Setup
```bash
cd /opt/aitbc
source venv/bin/activate
./aitbc-cli --version
```

## 1. AI Job Submission Testing

### Basic AI Job Submission
```bash
# Test basic AI job submission
echo "Testing basic AI job submission..."

# Submit inference job
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Generate a short story about AI" --payment 100 | grep -o "ai_job_[0-9]*")
echo "Submitted job: $JOB_ID"

# Check job status
echo "Checking job status..."
./aitbc-cli ai-ops --action status --job-id $JOB_ID

# Wait for completion and get results
echo "Waiting for job completion..."
sleep 10
./aitbc-cli ai-ops --action results --job-id $JOB_ID
```

### Advanced AI Job Types
```bash
# Test different AI job types
echo "Testing advanced AI job types..."

# Parallel AI job
./aitbc-cli ai-submit --wallet genesis-ops --type parallel --prompt "Parallel AI processing test" --payment 500

# Ensemble AI job
./aitbc-cli ai-submit --wallet genesis-ops --type ensemble --prompt "Ensemble AI processing test" --payment 600

# Multi-modal AI job
./aitbc-cli ai-submit --wallet genesis-ops --type multimodal --prompt "Multi-modal AI test" --payment 1000

# Resource allocation job
./aitbc-cli ai-submit --wallet genesis-ops --type resource-allocation --prompt "Resource allocation test" --payment 800

# Performance tuning job
./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Performance tuning test" --payment 1000
```

### Expected Results
- All job types should submit successfully
- Job IDs should be generated and returned
- Job status should be trackable
- Results should be retrievable upon completion

## 2. AI Job Monitoring Testing

### Job Status Monitoring
```bash
# Test job status monitoring
echo "Testing job status monitoring..."

# Submit test job
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Monitoring test job" --payment 100 | grep -o "ai_job_[0-9]*")

# Monitor job progress
for i in {1..10}; do
    echo "Check $i:"
    ./aitbc-cli ai-ops --action status --job-id $JOB_ID
    sleep 2
done
```

### Multiple Job Monitoring
```bash
# Test multiple job monitoring
echo "Testing multiple job monitoring..."

# Submit multiple jobs
JOB1=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Job 1" --payment 100 | grep -o "ai_job_[0-9]*")
JOB2=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Job 2" --payment 100 | grep -o "ai_job_[0-9]*")
JOB3=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Job 3" --payment 100 | grep -o "ai_job_[0-9]*")

echo "Submitted jobs: $JOB1, $JOB2, $JOB3"

# Monitor all jobs
for job in $JOB1 $JOB2 $JOB3; do
    echo "Status for $job:"
    ./aitbc-cli ai-ops --action status --job-id $job
done
```

## 3. Resource Management Testing

### Resource Status Monitoring
```bash
# Test resource status monitoring
echo "Testing resource status monitoring..."

# Check current resource status
./aitbc-cli resource status

# Monitor resource changes over time
for i in {1..5}; do
    echo "Resource check $i:"
    ./aitbc-cli resource status
    sleep 5
done
```

### Resource Allocation Testing
```bash
# Test resource allocation
echo "Testing resource allocation..."

# Allocate resources for AI operations
ALLOCATION_ID=$(./aitbc-cli resource allocate --agent-id test-ai-agent --cpu 2 --memory 4096 --duration 3600 | grep -o "alloc_[0-9]*")
echo "Resource allocation: $ALLOCATION_ID"

# Verify allocation
./aitbc-cli resource status

# Test resource deallocation
echo "Testing resource deallocation..."
# Note: Deallocation would be handled automatically when duration expires
```

### Resource Optimization Testing
```bash
# Test resource optimization
echo "Testing resource optimization..."

# Submit resource-intensive job
./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Resource optimization test with high resource usage" --payment 1500

# Monitor resource utilization during job
for i in {1..10}; do
    echo "Resource utilization check $i:"
    ./aitbc-cli resource status
    sleep 3
done
```

## 4. AI Service Integration Testing

### Ollama Integration Testing
```bash
# Test Ollama service integration
echo "Testing Ollama integration..."

# Check Ollama status
curl -sf http://localhost:11434/api/tags

# Test Ollama model availability
curl -sf http://localhost:11434/api/show/llama3.1:8b

# Test Ollama inference
curl -sf -X POST http://localhost:11434/api/generate \
    -H "Content-Type: application/json" \
    -d '{"model": "llama3.1:8b", "prompt": "Test inference", "stream": false}'
```

### Exchange API Integration
```bash
# Test Exchange API integration
echo "Testing Exchange API integration..."

# Check Exchange API status
curl -sf http://localhost:8001/health

# Test marketplace operations
./aitbc-cli market-list

# Test marketplace creation
./aitbc-cli market-create --type ai-inference --name "Test AI Service" --price 100 --description "Test service for AI operations" --wallet genesis-ops
```

### Blockchain RPC Integration
```bash
# Test Blockchain RPC integration
echo "Testing Blockchain RPC integration..."

# Check RPC status
curl -sf http://localhost:8006/rpc/health

# Test transaction submission
curl -sf -X POST http://localhost:8006/rpc/transaction \
    -H "Content-Type: application/json" \
    -d '{"from": "ait158ec7a0713f30ccfb1aac6bfbab71f36271c5871", "to": "ait141b3bae6eea3a74273ef3961861ee58e12b6d855", "amount": 1, "fee": 10}'
```

## 5. Advanced AI Operations Testing

### Complex Workflow Testing
```bash
# Test complex AI workflow
echo "Testing complex AI workflow..."

# Submit complex pipeline job
./aitbc-cli ai-submit --wallet genesis-ops --type parallel --prompt "Design and execute complex AI pipeline for medical diagnosis with ensemble validation and error handling" --payment 2000

# Monitor workflow execution
sleep 5
./aitbc-cli ai-ops --action status --job-id latest
```

### Multi-Modal Processing Testing
```bash
# Test multi-modal AI processing
echo "Testing multi-modal AI processing..."

# Submit multi-modal job
./aitbc-cli ai-submit --wallet genesis-ops --type multimodal --prompt "Process customer feedback with text sentiment analysis and image recognition" --payment 2500

# Monitor multi-modal processing
sleep 10
./aitbc-cli ai-ops --action status --job-id latest
```

### Performance Optimization Testing
```bash
# Test AI performance optimization
echo "Testing AI performance optimization..."

# Submit performance tuning job
./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Optimize AI model performance for sub-100ms inference latency with quantization and pruning" --payment 3000

# Monitor optimization process
sleep 15
./aitbc-cli ai-ops --action status --job-id latest
```

## 6. Error Handling Testing

### Invalid Job Submission Testing
```bash
# Test invalid job submission handling
echo "Testing invalid job submission..."

# Test missing required parameters
./aitbc-cli ai-submit --wallet genesis-ops --type inference 2>/dev/null && echo "ERROR: Missing prompt accepted" || echo "✅ Missing prompt properly rejected"

# Test invalid wallet
./aitbc-cli ai-submit --wallet invalid-wallet --type inference --prompt "Test" --payment 100 2>/dev/null && echo "ERROR: Invalid wallet accepted" || echo "✅ Invalid wallet properly rejected"

# Test insufficient payment
./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Test" --payment 1 2>/dev/null && echo "ERROR: Insufficient payment accepted" || echo "✅ Insufficient payment properly rejected"
```

### Invalid Job ID Testing
```bash
# Test invalid job ID handling
echo "Testing invalid job ID..."

# Test non-existent job
./aitbc-cli ai-ops --action status --job-id "non_existent_job" 2>/dev/null && echo "ERROR: Non-existent job accepted" || echo "✅ Non-existent job properly rejected"

# Test invalid job ID format
./aitbc-cli ai-ops --action status --job-id "invalid_format" 2>/dev/null && echo "ERROR: Invalid format accepted" || echo "✅ Invalid format properly rejected"
```

## 7. Performance Testing

### AI Job Throughput Testing
```bash
# Test AI job submission throughput
echo "Testing AI job throughput..."

# Submit multiple jobs rapidly
echo "Submitting 10 jobs rapidly..."
for i in {1..10}; do
    ./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Throughput test job $i" --payment 100
    echo "Submitted job $i"
done

# Monitor system performance
echo "Monitoring system performance during high load..."
for i in {1..10}; do
    echo "Performance check $i:"
    ./aitbc-cli resource status
    sleep 2
done
```

### Resource Utilization Testing
```bash
# Test resource utilization under load
echo "Testing resource utilization..."

# Submit resource-intensive jobs
for i in {1..5}; do
    ./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Resource utilization test $i" --payment 1000
    echo "Submitted resource-intensive job $i"
done

# Monitor resource utilization
for i in {1..15}; do
    echo "Resource utilization $i:"
    ./aitbc-cli resource status
    sleep 3
done
```

## 8. Automated AI Operations Testing

### Comprehensive AI Test Suite
```bash
#!/bin/bash
# automated_ai_tests.sh

echo "=== AI Operations Tests ==="

# Test basic AI job submission
echo "Testing basic AI job submission..."
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Automated test job" --payment 100 | grep -o "ai_job_[0-9]*")
[ -n "$JOB_ID" ] || exit 1

# Test job status monitoring
echo "Testing job status monitoring..."
./aitbc-cli ai-ops --action status --job-id $JOB_ID || exit 1

# Test resource status
echo "Testing resource status..."
./aitbc-cli resource status | jq -r '.cpu_utilization' || exit 1

# Test advanced AI job types
echo "Testing advanced AI job types..."
./aitbc-cli ai-submit --wallet genesis-ops --type multimodal --prompt "Automated multi-modal test" --payment 500 || exit 1

echo "✅ All AI operations tests passed!"
```

## 9. Integration Testing

### End-to-End AI Workflow Testing
```bash
# Test complete AI workflow
echo "Testing end-to-end AI workflow..."

# 1. Submit AI job
echo "1. Submitting AI job..."
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "End-to-end test: Generate a comprehensive analysis of AI workflow integration" --payment 500)

# 2. Monitor job progress
echo "2. Monitoring job progress..."
for i in {1..10}; do
    STATUS=$(./aitbc-cli ai-ops --action status --job-id $JOB_ID | grep -o '"status": "[^"]*"' | cut -d'"' -f4)
    echo "Job status: $STATUS"
    [ "$STATUS" = "completed" ] && break
    sleep 3
done

# 3. Retrieve results
echo "3. Retrieving results..."
./aitbc-cli ai-ops --action results --job-id $JOB_ID

# 4. Verify resource impact
echo "4. Verifying resource impact..."
./aitbc-cli resource status
```

## 10. Troubleshooting Guide

### Common AI Operations Issues

#### Job Submission Failures
```bash
# Problem: AI job submission failing
# Solution: Check wallet balance and service status
./aitbc-cli balance --wallet genesis-ops
./aitbc-cli resource status
curl -sf http://localhost:8000/health
```

#### Job Processing Stalled
```bash
# Problem: AI jobs not processing
# Solution: Check AI services and restart if needed
curl -sf http://localhost:11434/api/tags
sudo systemctl restart aitbc-ollama
```

#### Resource Allocation Issues
```bash
# Problem: Resource allocation failing
# Solution: Check resource availability
./aitbc-cli resource status
free -h
df -h
```

#### Performance Issues
```bash
# Problem: Slow AI job processing
# Solution: Check system resources and optimize
./aitbc-cli resource status
top -n 1
```

## 11. Success Criteria

### Pass/Fail Criteria
- ✅ AI job submission working for all job types
- ✅ Job status monitoring functional
- ✅ Resource management operational
- ✅ AI service integration working
- ✅ Advanced AI operations functional
- ✅ Error handling working correctly
- ✅ Performance within acceptable limits

### Performance Benchmarks
- Job submission time: <3 seconds
- Job status check: <1 second
- Resource status check: <1 second
- Basic AI job completion: <30 seconds
- Advanced AI job completion: <120 seconds
- Resource allocation: <2 seconds

---

**Dependencies**: [Basic Testing Module](test-basic.md)  
**Next Module**: [Advanced AI Testing](test-advanced-ai.md) or [Cross-Node Testing](test-cross-node.md)