aitbc/docs/10_plan/admin-test-scenarios.md

# Admin Commands Test Scenarios

## Overview

This document provides comprehensive test scenarios for the AITBC CLI admin commands, designed to validate system administration capabilities and ensure robust infrastructure management.

## Test Environment Setup

### Prerequisites
- AITBC CLI installed and configured
- Admin privileges or appropriate API keys
- Test environment with coordinator, blockchain node, and marketplace services
- Backup storage location available
- Network connectivity to all system components

### Environment Variables
```bash
export AITBC_ADMIN_API_KEY="your-admin-api-key"
export AITBC_BACKUP_PATH="/backups/aitbc-test"
export AITBC_LOG_LEVEL="info"
```

---

## Test Scenario Matrix

| Scenario | Command | Priority | Expected Duration | Dependencies |
|----------|---------|----------|-------------------|--------------|
| 13.1 | `admin backup` | High | 5-15 min | Storage space |
| 13.2 | `admin logs` | Medium | 1-2 min | Log access |
| 13.3 | `admin monitor` | High | 2-5 min | Monitoring service |
| 13.4 | `admin restart` | Critical | 1-3 min | Service control |
| 13.5 | `admin status` | High | 30 sec | All services |
| 13.6 | `admin update` | Medium | 5-20 min | Update server |
| 13.7 | `admin users` | Medium | 1-2 min | User database |

---

## Detailed Test Scenarios

### Scenario 13.1: System Backup Operations

#### Test Case 13.1.1: Full System Backup
```bash
# Command
aitbc admin backup --type full --destination /backups/aitbc-$(date +%Y%m%d) --compress

# Validation Steps
1. Check backup file creation: `ls -la /backups/aitbc-*`
2. Verify backup integrity: `aitbc admin backup --verify /backups/aitbc-20260305`
3. Check backup size and compression ratio
4. Validate backup contains all required components
```

#### Expected Results
- ✅ Backup file created successfully
- ✅ Checksum verification passes
- ✅ Backup size reasonable (< 10GB for test environment)
- ✅ All critical components included (blockchain, configs, user data)

#### Test Case 13.1.2: Incremental Backup
```bash
# Command
aitbc admin backup --type incremental --since "2026-03-04" --destination /backups/incremental

# Validation Steps
1. Verify incremental backup creation
2. Check that only changed files are included
3. Test restore from incremental backup
```

#### Expected Results
- ✅ Incremental backup created
- ✅ Significantly smaller than full backup
- ✅ Can be applied to full backup successfully

---

### Scenario 13.2: View System Logs

#### Test Case 13.2.1: Service-Specific Logs
```bash
# Command
aitbc admin logs --service coordinator --tail 50 --level info

# Validation Steps
1. Verify log output format
2. Check timestamp consistency
3. Validate log level filtering
4. Test with different services (blockchain, marketplace)
```

#### Expected Results
- ✅ Logs displayed in readable format
- ✅ Timestamps are current and sequential
- ✅ Log level filtering works correctly
- ✅ Different services show appropriate log content

#### Test Case 13.2.2: Live Log Following
```bash
# Command
aitbc admin logs --service all --follow --level warning

# Validation Steps
1. Start log following
2. Trigger a system event (e.g., submit a job)
3. Verify new logs appear in real-time
4. Stop following with Ctrl+C
```

#### Expected Results
- ✅ Real-time log updates
- ✅ New events appear immediately
- ✅ Clean termination on interrupt
- ✅ Warning level filtering works

---

### Scenario 13.3: System Monitoring Dashboard

#### Test Case 13.3.1: Basic Monitoring
```bash
# Command
aitbc admin monitor --dashboard --refresh 10 --duration 60

# Validation Steps
1. Verify dashboard initialization
2. Check all metrics are displayed
3. Validate refresh intervals
4. Test metric accuracy
```

#### Expected Results
- ✅ Dashboard loads successfully
- ✅ All key metrics visible (CPU, memory, disk, network)
- ✅ Refresh interval works as specified
- ✅ Metrics values are reasonable and accurate

#### Test Case 13.3.2: Alert Threshold Testing
```bash
# Command
aitbc admin monitor --alerts --threshold cpu:80 --threshold memory:90

# Validation Steps
1. Set low thresholds for testing
2. Generate load on system
3. Verify alert triggers
4. Check alert notification format
```

#### Expected Results
- ✅ Alert configuration accepted
- ✅ Alerts trigger when thresholds exceeded
- ✅ Alert messages are clear and actionable
- ✅ Alert history is maintained

---

### Scenario 13.4: Service Restart Operations

#### Test Case 13.4.1: Graceful Service Restart
```bash
# Command
aitbc admin restart --service coordinator --graceful --timeout 120

# Validation Steps
1. Verify graceful shutdown initiation
2. Check in-flight operations handling
3. Monitor service restart process
4. Validate service health post-restart
```

#### Expected Results
- ✅ Service shuts down gracefully
- ✅ In-flight operations completed or queued
- ✅ Service restarts successfully
- ✅ Health checks pass after restart

#### Test Case 13.4.2: Emergency Service Restart
```bash
# Command
aitbc admin restart --service blockchain-node --emergency --force

# Validation Steps
1. Verify immediate service termination
2. Check service restart speed
3. Validate service recovery
4. Test data integrity post-restart
```

#### Expected Results
- ✅ Service stops immediately
- ✅ Fast restart (< 30 seconds)
- ✅ Service recovers fully
- ✅ No data corruption or loss

---

### Scenario 13.5: System Status Overview

#### Test Case 13.5.1: Comprehensive Status Check
```bash
# Command
aitbc admin status --verbose --format json --output /tmp/system-status.json

# Validation Steps
1. Verify JSON output format
2. Check all services are reported
3. Validate status accuracy
4. Test with different output formats
```

#### Expected Results
- ✅ Valid JSON output
- ✅ All services included in status
- ✅ Status information is accurate
- ✅ Multiple output formats work

#### Test Case 13.5.2: Health Check Mode
```bash
# Command
aitbc admin status --health-check --comprehensive --report

# Validation Steps
1. Run comprehensive health check
2. Verify all components checked
3. Check health report completeness
4. Validate recommendations provided
```

#### Expected Results
- ✅ All components undergo health checks
- ✅ Detailed health report generated
- ✅ Issues identified with severity levels
- ✅ Actionable recommendations provided

---

### Scenario 13.6: System Update Operations

#### Test Case 13.6.1: Dry Run Update
```bash
# Command
aitbc admin update --component coordinator --version latest --dry-run

# Validation Steps
1. Verify update simulation runs
2. Check compatibility analysis
3. Review downtime estimate
4. Validate rollback plan
```

#### Expected Results
- ✅ Dry run completes successfully
- ✅ Compatibility issues identified
- ✅ Downtime accurately estimated
- ✅ Rollback plan is viable

#### Test Case 13.6.2: Actual Update (Test Environment)
```bash
# Command
aitbc admin update --component coordinator --version 2.1.0-test --backup

# Validation Steps
1. Verify backup creation
2. Monitor update progress
3. Validate post-update functionality
4. Test rollback if needed
```

#### Expected Results
- ✅ Backup created before update
- ✅ Update progresses smoothly
- ✅ Service functions post-update
- ✅ Rollback works if required

---

### Scenario 13.7: User Management Operations

#### Test Case 13.7.1: User Listing and Filtering
```bash
# Command
aitbc admin users --action list --role miner --status active --format table

# Validation Steps
1. Verify user list display
2. Test role filtering
3. Test status filtering
4. Validate output formats
```

#### Expected Results
- ✅ User list displays correctly
- ✅ Role filtering works
- ✅ Status filtering works
- ✅ Multiple output formats available

#### Test Case 13.7.2: User Creation and Management
```bash
# Command
aitbc admin users --action create --username testuser --role operator --email test@example.com

# Validation Steps
1. Create test user
2. Verify user appears in listings
3. Test user permission assignment
4. Clean up test user
```

#### Expected Results
- ✅ User created successfully
- ✅ User appears in system listings
- ✅ Permissions assigned correctly
- ✅ User can be cleanly removed

---

## Emergency Response Test Scenarios

### Scenario 14.1: Emergency Service Recovery

#### Test Case 14.1.1: Full System Recovery
```bash
# Simulate system failure
sudo systemctl stop aitbc-coordinator aitbc-blockchain aitbc-marketplace

# Emergency recovery
aitbc admin restart --service all --emergency --force

# Validation Steps
1. Verify all services stop
2. Execute emergency restart
3. Monitor service recovery sequence
4. Validate system functionality
```

#### Expected Results
- ✅ All services stop successfully
- ✅ Emergency restart initiates
- ✅ Services recover in correct order
- ✅ System fully functional post-recovery

---

## Performance Benchmarks

### Expected Performance Metrics

| Operation | Expected Time | Acceptable Range |
|-----------|---------------|------------------|
| Full Backup | 10 min | 5-20 min |
| Incremental Backup | 2 min | 1-5 min |
| Service Restart | 30 sec | 10-60 sec |
| Status Check | 5 sec | 2-10 sec |
| Log Retrieval | 2 sec | 1-5 sec |
| User Operations | 1 sec | < 3 sec |

### Load Testing Scenarios

#### High Load Backup Test
```bash
# Generate load while backing up
aitbc client submit --type inference --model llama3 --data '{"prompt":"Load test"}' &
aitbc admin backup --type full --destination /backups/load-test-backup

# Expected: Backup completes successfully under load
```

#### Concurrent Admin Operations
```bash
# Run multiple admin commands concurrently
aitbc admin status &
aitbc admin logs --tail 10 &
aitbc admin monitor --duration 30 &

# Expected: All commands complete without interference
```

---

## Test Automation Script

### Automated Test Runner
```bash
#!/bin/bash
# admin-test-runner.sh

echo "Starting AITBC Admin Commands Test Suite"

# Test configuration
TEST_LOG="/tmp/admin-test-$(date +%Y%m%d-%H%M%S).log"
FAILED_TESTS=0

# Test functions
test_backup() {
    echo "Testing backup operations..." | tee -a $TEST_LOG
    aitbc admin backup --type full --destination /tmp/test-backup --dry-run
    if [ $? -eq 0 ]; then
        echo "✅ Backup test passed" | tee -a $TEST_LOG
    else
        echo "❌ Backup test failed" | tee -a $TEST_LOG
        FAILED_TESTS=$((FAILED_TESTS + 1))
    fi
}

test_status() {
    echo "Testing status operations..." | tee -a $TEST_LOG
    aitbc admin status --format json > /tmp/status-test.json
    if [ $? -eq 0 ]; then
        echo "✅ Status test passed" | tee -a $TEST_LOG
    else
        echo "❌ Status test failed" | tee -a $TEST_LOG
        FAILED_TESTS=$((FAILED_TESTS + 1))
    fi
}

# Run all tests
test_backup
test_status

# Summary
echo "Test completed. Failed tests: $FAILED_TESTS" | tee -a $TEST_LOG
exit $FAILED_TESTS
```

---

## Troubleshooting Guide

### Common Issues and Solutions

#### Backup Failures
- **Issue**: Insufficient disk space
- **Solution**: Check available space with `df -h`, clear old backups

#### Service Restart Issues
- **Issue**: Service fails to restart
- **Solution**: Check logs with `aitbc admin logs --service <service> --level error`

#### Permission Errors
- **Issue**: Access denied errors
- **Solution**: Verify admin API key permissions and user role

#### Network Connectivity
- **Issue**: Cannot reach services
- **Solution**: Check network connectivity and service endpoints

### Debug Commands
```bash
# Check admin permissions
aitbc auth status

# Verify service connectivity
aitbc admin status --health-check

# Check system resources
aitbc admin monitor --duration 60

# Review recent errors
aitbc admin logs --level error --since "1 hour ago"
```

---

## Test Reporting

### Test Result Template
```markdown
# Admin Commands Test Report

**Date**: 2026-03-05
**Environment**: Test
**Tester**: [Your Name]

## Test Summary
- Total Tests: 15
- Passed: 14
- Failed: 1
- Success Rate: 93.3%

## Failed Tests
1. **Test Case 13.6.2**: Actual Update - Version compatibility issue
   - **Issue**: Target version not compatible with current dependencies
   - **Resolution**: Update dependencies first, then retry

## Recommendations
1. Implement automated dependency checking before updates
2. Add backup verification automation
3. Enhance error messages for better troubleshooting

## Next Steps
1. Fix failed test case
2. Implement recommendations
3. Schedule re-test
```

---

*Last updated: March 5, 2026*  
*Test scenarios version: 1.0*  
*Compatible with AITBC CLI version: 2.x*