aitbc/.windsurf/workflows/test-ai-operations.md

description, title, version

description	title	version
AI job submission, processing, and resource management testing module	AI Operations Testing Module	1.0

AI Operations Testing Module

This module covers AI job submission, processing, resource management, and AI service integration testing.

Prerequisites

Required Setup

• Working directory: /opt/aitbc
• Virtual environment: /opt/aitbc/venv
• CLI wrapper: /opt/aitbc/aitbc-cli
• Services running (Coordinator, Exchange, Blockchain RPC, Ollama)
• Basic Testing Module completed

Environment Setup

cd /opt/aitbc
source venv/bin/activate
./aitbc-cli --version

1. AI Job Submission Testing

Basic AI Job Submission

# Test basic AI job submission
echo "Testing basic AI job submission..."

# Submit inference job
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Generate a short story about AI" --payment 100 | grep -o "ai_job_[0-9]*")
echo "Submitted job: $JOB_ID"

# Check job status
echo "Checking job status..."
./aitbc-cli ai-ops --action status --job-id $JOB_ID

# Wait for completion and get results
echo "Waiting for job completion..."
sleep 10
./aitbc-cli ai-ops --action results --job-id $JOB_ID

Advanced AI Job Types

# Test different AI job types
echo "Testing advanced AI job types..."

# Parallel AI job
./aitbc-cli ai-submit --wallet genesis-ops --type parallel --prompt "Parallel AI processing test" --payment 500

# Ensemble AI job
./aitbc-cli ai-submit --wallet genesis-ops --type ensemble --prompt "Ensemble AI processing test" --payment 600

# Multi-modal AI job
./aitbc-cli ai-submit --wallet genesis-ops --type multimodal --prompt "Multi-modal AI test" --payment 1000

# Resource allocation job
./aitbc-cli ai-submit --wallet genesis-ops --type resource-allocation --prompt "Resource allocation test" --payment 800

# Performance tuning job
./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Performance tuning test" --payment 1000

Expected Results

• All job types should submit successfully
• Job IDs should be generated and returned
• Job status should be trackable
• Results should be retrievable upon completion

2. AI Job Monitoring Testing

Job Status Monitoring

# Test job status monitoring
echo "Testing job status monitoring..."

# Submit test job
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Monitoring test job" --payment 100 | grep -o "ai_job_[0-9]*")

# Monitor job progress
for i in {1..10}; do
    echo "Check $i:"
    ./aitbc-cli ai-ops --action status --job-id $JOB_ID
    sleep 2
done

Multiple Job Monitoring

# Test multiple job monitoring
echo "Testing multiple job monitoring..."

# Submit multiple jobs
JOB1=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Job 1" --payment 100 | grep -o "ai_job_[0-9]*")
JOB2=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Job 2" --payment 100 | grep -o "ai_job_[0-9]*")
JOB3=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Job 3" --payment 100 | grep -o "ai_job_[0-9]*")

echo "Submitted jobs: $JOB1, $JOB2, $JOB3"

# Monitor all jobs
for job in $JOB1 $JOB2 $JOB3; do
    echo "Status for $job:"
    ./aitbc-cli ai-ops --action status --job-id $job
done

3. Resource Management Testing

Resource Status Monitoring

# Test resource status monitoring
echo "Testing resource status monitoring..."

# Check current resource status
./aitbc-cli resource status

# Monitor resource changes over time
for i in {1..5}; do
    echo "Resource check $i:"
    ./aitbc-cli resource status
    sleep 5
done

Resource Allocation Testing

# Test resource allocation
echo "Testing resource allocation..."

# Allocate resources for AI operations
ALLOCATION_ID=$(./aitbc-cli resource allocate --agent-id test-ai-agent --cpu 2 --memory 4096 --duration 3600 | grep -o "alloc_[0-9]*")
echo "Resource allocation: $ALLOCATION_ID"

# Verify allocation
./aitbc-cli resource status

# Test resource deallocation
echo "Testing resource deallocation..."
# Note: Deallocation would be handled automatically when duration expires

Resource Optimization Testing

# Test resource optimization
echo "Testing resource optimization..."

# Submit resource-intensive job
./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Resource optimization test with high resource usage" --payment 1500

# Monitor resource utilization during job
for i in {1..10}; do
    echo "Resource utilization check $i:"
    ./aitbc-cli resource status
    sleep 3
done

4. AI Service Integration Testing

Ollama Integration Testing

# Test Ollama service integration
echo "Testing Ollama integration..."

# Check Ollama status
curl -sf http://localhost:11434/api/tags

# Test Ollama model availability
curl -sf http://localhost:11434/api/show/llama3.1:8b

# Test Ollama inference
curl -sf -X POST http://localhost:11434/api/generate \
    -H "Content-Type: application/json" \
    -d '{"model": "llama3.1:8b", "prompt": "Test inference", "stream": false}'

Exchange API Integration

# Test Exchange API integration
echo "Testing Exchange API integration..."

# Check Exchange API status
curl -sf http://localhost:8001/health

# Test marketplace operations
./aitbc-cli market-list

# Test marketplace creation
./aitbc-cli market-create --type ai-inference --name "Test AI Service" --price 100 --description "Test service for AI operations" --wallet genesis-ops

Blockchain RPC Integration

# Test Blockchain RPC integration
echo "Testing Blockchain RPC integration..."

# Check RPC status
curl -sf http://localhost:8006/rpc/health

# Test transaction submission
curl -sf -X POST http://localhost:8006/rpc/transaction \
    -H "Content-Type: application/json" \
    -d '{"from": "ait158ec7a0713f30ccfb1aac6bfbab71f36271c5871", "to": "ait141b3bae6eea3a74273ef3961861ee58e12b6d855", "amount": 1, "fee": 10}'

5. Advanced AI Operations Testing

Complex Workflow Testing

# Test complex AI workflow
echo "Testing complex AI workflow..."

# Submit complex pipeline job
./aitbc-cli ai-submit --wallet genesis-ops --type parallel --prompt "Design and execute complex AI pipeline for medical diagnosis with ensemble validation and error handling" --payment 2000

# Monitor workflow execution
sleep 5
./aitbc-cli ai-ops --action status --job-id latest

Multi-Modal Processing Testing

# Test multi-modal AI processing
echo "Testing multi-modal AI processing..."

# Submit multi-modal job
./aitbc-cli ai-submit --wallet genesis-ops --type multimodal --prompt "Process customer feedback with text sentiment analysis and image recognition" --payment 2500

# Monitor multi-modal processing
sleep 10
./aitbc-cli ai-ops --action status --job-id latest

Performance Optimization Testing

# Test AI performance optimization
echo "Testing AI performance optimization..."

# Submit performance tuning job
./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Optimize AI model performance for sub-100ms inference latency with quantization and pruning" --payment 3000

# Monitor optimization process
sleep 15
./aitbc-cli ai-ops --action status --job-id latest

6. Error Handling Testing

Invalid Job Submission Testing

# Test invalid job submission handling
echo "Testing invalid job submission..."

# Test missing required parameters
./aitbc-cli ai-submit --wallet genesis-ops --type inference 2>/dev/null && echo "ERROR: Missing prompt accepted" || echo "✅ Missing prompt properly rejected"

# Test invalid wallet
./aitbc-cli ai-submit --wallet invalid-wallet --type inference --prompt "Test" --payment 100 2>/dev/null && echo "ERROR: Invalid wallet accepted" || echo "✅ Invalid wallet properly rejected"

# Test insufficient payment
./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Test" --payment 1 2>/dev/null && echo "ERROR: Insufficient payment accepted" || echo "✅ Insufficient payment properly rejected"

Invalid Job ID Testing

# Test invalid job ID handling
echo "Testing invalid job ID..."

# Test non-existent job
./aitbc-cli ai-ops --action status --job-id "non_existent_job" 2>/dev/null && echo "ERROR: Non-existent job accepted" || echo "✅ Non-existent job properly rejected"

# Test invalid job ID format
./aitbc-cli ai-ops --action status --job-id "invalid_format" 2>/dev/null && echo "ERROR: Invalid format accepted" || echo "✅ Invalid format properly rejected"

7. Performance Testing

AI Job Throughput Testing

# Test AI job submission throughput
echo "Testing AI job throughput..."

# Submit multiple jobs rapidly
echo "Submitting 10 jobs rapidly..."
for i in {1..10}; do
    ./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Throughput test job $i" --payment 100
    echo "Submitted job $i"
done

# Monitor system performance
echo "Monitoring system performance during high load..."
for i in {1..10}; do
    echo "Performance check $i:"
    ./aitbc-cli resource status
    sleep 2
done

Resource Utilization Testing

# Test resource utilization under load
echo "Testing resource utilization..."

# Submit resource-intensive jobs
for i in {1..5}; do
    ./aitbc-cli ai-submit --wallet genesis-ops --type performance-tuning --prompt "Resource utilization test $i" --payment 1000
    echo "Submitted resource-intensive job $i"
done

# Monitor resource utilization
for i in {1..15}; do
    echo "Resource utilization $i:"
    ./aitbc-cli resource status
    sleep 3
done

8. Automated AI Operations Testing

Comprehensive AI Test Suite

#!/bin/bash
# automated_ai_tests.sh

echo "=== AI Operations Tests ==="

# Test basic AI job submission
echo "Testing basic AI job submission..."
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "Automated test job" --payment 100 | grep -o "ai_job_[0-9]*")
[ -n "$JOB_ID" ] || exit 1

# Test job status monitoring
echo "Testing job status monitoring..."
./aitbc-cli ai-ops --action status --job-id $JOB_ID || exit 1

# Test resource status
echo "Testing resource status..."
./aitbc-cli resource status | jq -r '.cpu_utilization' || exit 1

# Test advanced AI job types
echo "Testing advanced AI job types..."
./aitbc-cli ai-submit --wallet genesis-ops --type multimodal --prompt "Automated multi-modal test" --payment 500 || exit 1

echo "✅ All AI operations tests passed!"

9. Integration Testing

End-to-End AI Workflow Testing

# Test complete AI workflow
echo "Testing end-to-end AI workflow..."

# 1. Submit AI job
echo "1. Submitting AI job..."
JOB_ID=$(./aitbc-cli ai-submit --wallet genesis-ops --type inference --prompt "End-to-end test: Generate a comprehensive analysis of AI workflow integration" --payment 500)

# 2. Monitor job progress
echo "2. Monitoring job progress..."
for i in {1..10}; do
    STATUS=$(./aitbc-cli ai-ops --action status --job-id $JOB_ID | grep -o '"status": "[^"]*"' | cut -d'"' -f4)
    echo "Job status: $STATUS"
    [ "$STATUS" = "completed" ] && break
    sleep 3
done

# 3. Retrieve results
echo "3. Retrieving results..."
./aitbc-cli ai-ops --action results --job-id $JOB_ID

# 4. Verify resource impact
echo "4. Verifying resource impact..."
./aitbc-cli resource status

10. Troubleshooting Guide

Common AI Operations Issues

Job Submission Failures

# Problem: AI job submission failing
# Solution: Check wallet balance and service status
./aitbc-cli balance --wallet genesis-ops
./aitbc-cli resource status
curl -sf http://localhost:8000/health

Job Processing Stalled

# Problem: AI jobs not processing
# Solution: Check AI services and restart if needed
curl -sf http://localhost:11434/api/tags
sudo systemctl restart aitbc-ollama

Resource Allocation Issues

# Problem: Resource allocation failing
# Solution: Check resource availability
./aitbc-cli resource status
free -h
df -h

Performance Issues

# Problem: Slow AI job processing
# Solution: Check system resources and optimize
./aitbc-cli resource status
top -n 1

11. Success Criteria

Pass/Fail Criteria

• ✅ AI job submission working for all job types
• ✅ Job status monitoring functional
• ✅ Resource management operational
• ✅ AI service integration working
• ✅ Advanced AI operations functional
• ✅ Error handling working correctly
• ✅ Performance within acceptable limits

Performance Benchmarks

• Job submission time: <3 seconds
• Job status check: <1 second
• Resource status check: <1 second
• Basic AI job completion: <30 seconds
• Advanced AI job completion: <120 seconds
• Resource allocation: <2 seconds

---

Dependencies: Basic Testing Module
Next Module: Advanced AI Testing or Cross-Node Testing