oib/aitbc
1
0
Fork 0
Code Issues Pull Requests Actions 19 Packages Projects Releases Wiki Activity

Update SSH access patterns documentation and expand workflow integration test suite
Some checks failed
Cross-Node Transaction Testing / transaction-test (push) Has been cancelled
Details
Deploy to Testnet / deploy-testnet (push) Has been cancelled
Details
Documentation Validation / validate-docs (push) Has been cancelled
Details
Documentation Validation / validate-policies-strict (push) Has been cancelled
Details
Multi-Node Stress Testing / stress-test (push) Has been cancelled
Details
Coverage Phase 1 (70% Target) / test-coverage-70 (push) Has been cancelled
Details
Coverage Phase 2 (85% Target) / test-coverage-85 (push) Has been cancelled
Details
Python Tests / test-python (push) Has been cancelled
Details

Browse Source 
- ssh-access-patterns.md: Clarify ns3/aitbc container setup with correct paths and service names
  - Add container hostname verification command
  - Update paths: /etc/aitbc/blockchain.env, /opt/aitbc/apps/blockchain-node/
  - Fix service name: aitbc-blockchain-node (not aitbc-blockchain-node-3)
  - Add service restart and log viewing examples
- test_workflow.sh: Rewrite as comprehensive integration test suite
  - Add
This commit is contained in:
aitbc
2026-05-27 09:16:23 +02:00
parent 7f71d8a6c7
commit 2acb5ccc49
17 changed files with 5403 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
.windsurf/skills/ssh-access-patterns.md
View File

@@ -40,7 +40,7 @@ ssh gitea-runner "systemctl status aitbc-blockchain-node --no-pager"
``` ```
### ns3 (hosts hub.aitbc.bubuit.net incus container) ### ns3 (hosts hub.aitbc.bubuit.net incus container)
Direct SSH access. The hub.aitbc.bubuit.net service runs as an incus container on ns3. Direct SSH access. The hub.aitbc.bubuit.net service runs as an incus container named "aitbc" on ns3.
```bash ```bash
ssh ns3 ssh ns3
# Or execute single command # Or execute single command
@@ -51,19 +51,29 @@ ssh ns3 "incus exec aitbc -- bash"
# Or execute single command in container # Or execute single command in container
ssh ns3 "incus exec aitbc -- command" ssh ns3 "incus exec aitbc -- command"
# Check container hostname
ssh ns3 "incus exec aitbc -- hostname"
# Output: hub.aitbc.bubuit.net
# Container IP: 192.168.100.10 # Container IP: 192.168.100.10
# Access via container IP from ns3 # Access via container IP from ns3
ssh ns3 "curl http://192.168.100.10:8006/rpc/head" ssh ns3 "curl http://192.168.100.10:8006/rpc/head"
# Check environment configuration in container # Check environment configuration in container
ssh ns3 "incus exec aitbc -- cat /etc/aitbc/.env" ssh ns3 "incus exec aitbc -- cat /etc/aitbc/blockchain.env"
ssh ns3 "incus exec aitbc -- cat /etc/aitbc/node.env" ssh ns3 "incus exec aitbc -- cat /etc/aitbc/node.env"
# Check blockchain node configuration # Check blockchain node configuration
ssh ns3 "incus exec aitbc -- cat /opt/blockchain-node/src/aitbc_chain/config.py" ssh ns3 "incus exec aitbc -- cat /opt/aitbc/apps/blockchain-node/src/aitbc_chain/config.py"
# Check service status in container # Check service status in container
ssh ns3 "incus exec aitbc -- systemctl status aitbc-blockchain-node-3 --no-pager" ssh ns3 "incus exec aitbc -- systemctl status aitbc-blockchain-node --no-pager"
# Restart service in container
ssh ns3 "incus exec aitbc -- systemctl restart aitbc-blockchain-node"
# View service logs in container
ssh ns3 "incus exec aitbc -- journalctl -u aitbc-blockchain-node -n 50 --no-pager"
``` ```
## Important Notes ## Important Notes

420
docs/scenarios/48_config_profiles.md Normal file
View File

@@ -0,0 +1,420 @@
# Configuration Profiles for hermes Agents
**Level**: Beginner
**Prerequisites**: Basic CLI knowledge, AITBC CLI installed
**Estimated Time**: 15 minutes
**Last Updated**: 2026-05-27
**Version**: 1.0
## 🧭 **Navigation Path:**
**🏠 [Documentation Home](../README.md)** → **🎭 [Agent Scenarios](./README.md)** → *You are here*
**breadcrumb**: Home → Scenarios → Configuration Profiles
---
## 🎯 **See Also:**
- **📖 Previous Scenario**: [47 Cross Chain Atomic Swap](./47_cross_chain_atomic_swap.md)
- **📖 Next Scenario**: [49 Resource Management](./49_resource_management.md)
- **⚙️ Config Documentation**: [CLI Config Commands](../cli/CLI_DOCUMENTATION.md)
---
## 📚 **Scenario Overview**
This scenario demonstrates how hermes agents use configuration profiles to manage different AITBC environments. Profiles allow agents to quickly switch between development, staging, and production configurations without manually editing config files.
### **Use Case**
An hermes agent needs to:
- Switch between multiple coordinator environments (dev, staging, prod)
- Save frequently-used configurations for quick access
- Maintain separate profiles for different network topologies
- Share configuration profiles across team members
### **What You'll Learn**
- Save current configuration as a named profile
- List all available profiles
- Load a profile to switch configurations
- Delete unused profiles
- Understand profile storage location and format
### **Features Combined**
- **Profile Management**: Save, list, load, and delete configuration profiles
- **Environment Switching**: Quick switching between different coordinator URLs
- **File System Integration**: Profiles stored in `~/.config/aitbc/profiles/`
- **Security**: API keys are not saved in profiles (must be set separately)
---
## 📋 **Prerequisites**
### **Knowledge Required**
- Basic command-line interface usage
- Understanding of YAML configuration files
- AITBC CLI installed and accessible
### **System Requirements**
- AITBC CLI installed
- Write access to `~/.config/aitbc/profiles/` directory
- Coordinator URL configured (can be set via `aitbc config set`)
---
## 🚀 **Quick Start**
```bash
# Save current configuration as a profile
aitbc config profiles save development
# List all available profiles
aitbc config profiles list
# Load a profile
aitbc config profiles load production
# Delete a profile
aitbc config profiles delete old_profile
```
---
## 📖 **Detailed Steps**
### Step 1: Configure Base Settings
Before creating profiles, set up your base configuration:
```bash
# Set coordinator URL
aitbc config set coordinator_url http://127.0.0.1:18000
# Set timeout
aitbc config set timeout 30
# Verify configuration
aitbc config show
```
**Expected Output:**
```
coordinator_url: http://127.0.0.1:18000
api_key: None
timeout: 30
config_file: /home/user/.aitbc.yaml
```
### Step 2: Save Development Profile
Save your current configuration as a development profile:
```bash
aitbc config profiles save development
```
**Expected Output:**
```
Profile 'development' saved
```
**What happens:**
- Profile file created at `~/.config/aitbc/profiles/development.yaml`
- Current coordinator_url and timeout saved
- API key is NOT saved (security measure)
**Verify profile file:**
```bash
cat ~/.config/aitbc/profiles/development.yaml
```
**Expected file content:**
```yaml
coordinator_url: http://127.0.0.1:18000
timeout: 30
```
### Step 3: Configure and Save Production Profile
Switch to production settings and save as a profile:
```bash
# Set production coordinator URL
aitbc config set coordinator_url http://prod.example.com:18000
# Set production timeout
aitbc config set timeout 60
# Save as production profile
aitbc config profiles save production
```
**Expected Output:**
```
Coordinator URL set to: http://prod.example.com:18000
Timeout set to: 60s
Profile 'production' saved
```
### Step 4: List All Profiles
View all available profiles:
```bash
aitbc config profiles list
```
**Expected Output:**
```json
{
"profiles": [
{
"name": "development",
"coordinator_url": "http://127.0.0.1:18000",
"timeout": 30
},
{
"name": "production",
"coordinator_url": "http://prod.example.com:18000",
"timeout": 60
}
]
}
```
### Step 5: Load a Profile
Switch to a different environment by loading a profile:
```bash
aitbc config profiles load development
```
**Expected Output:**
```
Profile 'development' loaded
```
**What happens:**
- Profile content written to `.aitbc.yaml` in current directory
- Current configuration overwritten with profile values
- CLI now uses the loaded configuration
**Verify loaded configuration:**
```bash
aitbc config show
```
**Expected Output:**
```
coordinator_url: http://127.0.0.1:18000
api_key: None
timeout: 30
config_file: .aitbc.yaml
```
### Step 6: Delete a Profile
Remove an unused profile:
```bash
aitbc config profiles delete old_profile
```
**Expected Output:**
```
Delete profile 'old_profile'? [y/N]: y
Profile 'old_profile' deleted
```
**What happens:**
- Profile file removed from `~/.config/aitbc/profiles/`
- Confirmation prompt before deletion
- No impact on current configuration
---
## 🔧 **Advanced Usage**
### Profile with Different Network Topologies
Create profiles for different network configurations:
```bash
# Local development
aitbc config set coordinator_url http://localhost:18000
aitbc config profiles save local
# Testnet
aitbc config set coordinator_url http://testnet.aitbc.io:18000
aitbc config profiles save testnet
# Mainnet
aitbc config set coordinator_url http://mainnet.aitbc.io:18000
aitbc config profiles save mainnet
```
### Profile for Different Agent Roles
Create profiles specific to agent roles:
```bash
# GPU provider profile
aitbc config set coordinator_url http://gpu-hub.aitbc.io:18000
aitbc config set timeout 120
aitbc config profiles save gpu-provider
# AI job submitter profile
aitbc config set coordinator_url http://ai-hub.aitbc.io:18000
aitbc config set timeout 30
aitbc config profiles save ai-submitter
```
### Manual Profile Editing
Profiles can be manually edited for advanced configurations:
```bash
# Edit profile file directly
nano ~/.config/aitbc/profiles/custom.yaml
```
**Custom profile example:**
```yaml
coordinator_url: http://custom.example.com:18000
timeout: 45
custom_field: custom_value
```
---
## ⚠️ **Important Notes**
### Security Considerations
- **API keys are NOT saved in profiles** - must be set separately
- Profile files are stored in plain text YAML
- Ensure `~/.config/aitbc/` directory has appropriate permissions (600 for secrets)
### Profile Storage Location
- Profiles stored in: `~/.config/aitbc/profiles/`
- Profile file format: `<profile_name>.yaml`
- Config loaded to: `.aitbc.yaml` in current directory
### Profile Limitations
- Only saves `coordinator_url` and `timeout`
- Does not save API keys (security measure)
- Does not save environment variables
- Overwrites current config when loaded
---
## 🐛 **Troubleshooting**
### Profile not found
**Error:**
```
Error: Profile 'my_profile' not found
```
**Solution:**
```bash
# List available profiles
aitbc config profiles list
# Check if profiles directory exists
ls -la ~/.config/aitbc/profiles/
```
### Permission denied
**Error:**
```
Error: Permission denied when saving profile
```
**Solution:**
```bash
# Create profiles directory with correct permissions
mkdir -p ~/.config/aitbc/profiles
chmod 755 ~/.config/aitbc
chmod 755 ~/.config/aitbc/profiles
```
### Config file not created after load
**Error:**
Profile loads but `.aitbc.yaml` not created
**Solution:**
```bash
# Check current directory
pwd
# Ensure write permissions
ls -la .
# Manually create config file
aitbc config profiles load my_profile
```
### Profile contains unexpected values
**Issue:**
Loaded profile has different values than expected
**Solution:**
```bash
# Inspect profile file directly
cat ~/.config/aitbc/profiles/my_profile.yaml
# Re-save profile with current config
aitbc config set coordinator_url http://correct:18000
aitbc config profiles save my_profile
```
---
## 📊 **Testing**
Run the integration test script to verify profile operations:
```bash
# Run pytest tests
cd /opt/aitbc
pytest tests/cli/test_config_profiles.py -v
# Run bash integration test
scripts/testing/test_config_profiles.sh
```
**Expected test output:**
```
tests/cli/test_config_profiles.py::TestConfigProfilesIntegration::test_profiles_save_creates_file PASSED
tests/cli/test_config_profiles.py::TestConfigProfilesIntegration::test_profiles_list_multiple PASSED
tests/cli/test_config_profiles.py::TestConfigProfilesIntegration::test_profiles_load_creates_config PASSED
...
```
---
## 🎓 **Summary**
In this scenario, you learned:
- How to save current configuration as a named profile
- How to list and inspect available profiles
- How to load profiles to switch between environments
- How to delete unused profiles
- Profile storage location and security considerations
**Key Takeaways:**
- Profiles enable quick environment switching
- API keys are not saved in profiles (security)
- Profiles stored in `~/.config/aitbc/profiles/`
- Load overwrites current configuration
- Use profiles for different environments, roles, or network topologies
---
## 🔄 **Related Scenarios**
- **Scenario 01**: [Wallet Basics](./01_wallet_basics.md) - Basic wallet operations
- **Scenario 49**: [Resource Management](./49_resource_management.md) - Resource allocation
- **Scenario 50**: [Workflow Management](./50_workflow_management.md) - Workflow operations

525
docs/scenarios/49_resource_management.md Normal file
View File

@@ -0,0 +1,525 @@
# Resource Management for hermes Agents
**Level**: Intermediate
**Prerequisites**: Basic CLI knowledge, AITBC CLI installed, coordinator-api running
**Estimated Time**: 25 minutes
**Last Updated**: 2026-05-27
**Version**: 1.0
## 🧭 **Navigation Path:**
**🏠 [Documentation Home](../README.md)** → **🎭 [Agent Scenarios](./README.md)** → *You are here*
**breadcrumb**: Home → Scenarios → Resource Management
---
## 🎯 **See Also:**
- **📖 Previous Scenario**: [48 Configuration Profiles](./48_config_profiles.md)
- **📖 Next Scenario**: [50 Workflow Management](./50_workflow_management.md)
- **⚙️ Resource Documentation**: [CLI Resource Commands](../cli/CLI_DOCUMENTATION.md)
---
## 📚 **Scenario Overview**
This scenario demonstrates how hermes agents use resource management commands to allocate, monitor, and optimize compute resources (GPU, CPU, memory) within the AITBC ecosystem. Resource management enables efficient utilization of distributed compute resources for AI workloads.
### **Use Case**
An hermes agent needs to:
- Allocate GPU resources for AI job processing
- Monitor resource utilization and efficiency
- Release resources when no longer needed
- Optimize resource allocation for cost efficiency
- Track resource status across the network
### **What You'll Learn**
- Check resource status across the network
- Allocate resources for specific workloads
- List available resources by type
- Release resources back to the pool
- Monitor resource utilization metrics
- Optimize resource allocation
- Deallocate resources with confirmation
### **Features Combined**
- **Resource Allocation**: Allocate GPU, CPU, and memory resources
- **Status Monitoring**: Track resource availability and utilization
- **Resource Release**: Return resources to the pool
- **Utilization Tracking**: Monitor efficiency and performance
- **Optimization**: Improve resource allocation strategies
- **Deallocation**: Clean up unused resources
---
## 📋 **Prerequisites**
### **Knowledge Required**
- Basic command-line interface usage
- Understanding of compute resources (GPU, CPU, memory)
- AITBC CLI installed and accessible
- coordinator-api running (for API-integrated operations)
### **System Requirements**
- AITBC CLI installed
- coordinator-api running (http://127.0.0.1:18000)
- Access to distributed compute resources
---
## 🚀 **Quick Start**
```bash
# Check resource status
aitbc resource status
# Allocate GPU resources (experimental, requires --mock)
aitbc resource allocate --resource-type gpu --quantity 4 --mock
# List available resources (experimental, requires --mock)
aitbc resource list --mock
# Release resources (experimental, requires --mock)
aitbc resource release <resource_id> --mock
# Check utilization (experimental, requires --mock)
aitbc resource utilization --mock
# Deallocate resources
aitbc resource deallocate <resource_id>
```
---
## 📖 **Detailed Steps**
### Step 1: Check Resource Status
View status of all resources in the network:
```bash
aitbc resource status
```
**Expected Output:**
```
Resource Status:
- res_001: GPU (allocated) - 85.5% efficiency
- res_002: CPU (available) - idle
- res_003: GPU (allocated) - 92.1% efficiency
```
**JSON format:**
```bash
aitbc resource status --format json
```
**Expected JSON Output:**
```json
{
"resources": [
{
"id": "res_001",
"type": "gpu",
"status": "allocated",
"efficiency": "85.5%"
},
{
"id": "res_002",
"type": "cpu",
"status": "available"
},
{
"id": "res_003",
"type": "gpu",
"status": "allocated",
"efficiency": "92.1%"
}
]
}
```
**Check specific resource:**
```bash
aitbc resource status --resource-id res_001
```
**Expected Output:**
```
Resource Status for res_001:
Type: GPU
Status: allocated
Efficiency: 85.5%
Last Updated: 2026-05-27 08:30:00
```
### Step 2: Allocate Resources (Experimental)
Allocate GPU resources for AI workloads:
```bash
aitbc resource allocate --resource-type gpu --quantity 4 --mock
```
**Expected Output:**
```
Resource Allocation (MOCK MODE)
Allocating 4 GPU resources
Allocation ID: alloc_1716789123
Resource IDs: res_004, res_005, res_006, res_007
Status: allocated
```
**What happens:**
- 4 GPU resources allocated from available pool
- Unique allocation ID assigned
- Resources marked as allocated
- **Note**: This is an experimental command requiring `--mock` flag
**Allocate with parameters:**
```bash
aitbc resource allocate \
--resource-type gpu \
--quantity 8 \
--min-memory 32 \
--mock
```
**Expected Output:**
```
Resource Allocation (MOCK MODE)
Allocating 8 GPU resources with minimum 32GB memory
Allocation ID: alloc_1716789234
Resource IDs: res_008, res_009, res_010, res_011, res_012, res_013, res_014, res_015
Status: allocated
```
### Step 3: List Available Resources (Experimental)
View all available resources by type:
```bash
aitbc resource list --mock
```
**Expected Output:**
```
Available Resources (MOCK MODE):
GPU Resources:
- res_016: NVIDIA A100 (80GB) - available
- res_017: NVIDIA V100 (32GB) - available
CPU Resources:
- res_018: 64 cores - available
- res_019: 32 cores - available
```
**Filter by resource type:**
```bash
aitbc resource list --resource-type gpu --mock
```
### Step 4: Monitor Resource Utilization (Experimental)
Track resource efficiency and performance:
```bash
aitbc resource utilization --mock
```
**Expected Output:**
```
Resource Utilization (MOCK MODE):
Overall Utilization: 78.5%
GPU Utilization: 85.2%
CPU Utilization: 62.3%
Memory Utilization: 71.8%
Top Resources by Efficiency:
- res_003: 92.1% (GPU)
- res_001: 85.5% (GPU)
- res_020: 81.2% (CPU)
```
**JSON format:**
```bash
aitbc resource utilization --format json --mock
```
### Step 5: Optimize Resource Allocation (Experimental)
Improve resource allocation for better efficiency:
```bash
aitbc resource optimize --mock
```
**Expected Output:**
```
Resource Optimization (MOCK MODE)
Analyzing current allocation...
Optimization Recommendations:
1. Reallocate res_021 to workload A (potential gain: 15%)
2. Release res_022 (idle for 2 hours)
3. Consolidate GPU resources (potential savings: 20%)
Optimization applied: 3 recommendations
Expected efficiency improvement: 18.5%
```
### Step 6: Release Resources (Experimental)
Return resources to the available pool:
```bash
aitbc resource release res_004 --mock
```
**Expected Output:**
```
Resource Release (MOCK MODE)
Releasing resource res_004
Status: released
Resource returned to available pool
```
**Release multiple resources:**
```bash
aitbc resource release res_004 res_005 res_006 --mock
```
### Step 7: Deallocate Resources
Clean up resources with confirmation:
```bash
aitbc resource deallocate res_123
```
**Expected Output:**
```
Deallocate resource res_123? [y/N]: y
Resource deallocated successfully
Status: deallocated
Timestamp: 2026-05-27T08:30:00Z
```
**Force deallocation without confirmation:**
```bash
aitbc resource deallocate res_123 --force
```
**Expected Output:**
```
Resource deallocated successfully (forced)
Status: deallocated
Timestamp: 2026-05-27T08:30:00Z
```
---
## 🔧 **Advanced Usage**
### Batch Resource Allocation
Allocate resources for multiple workloads:
```bash
#!/bin/bash
# allocate_resources.sh
workloads=(
"gpu:4:training"
"gpu:2:inference"
"cpu:8:preprocessing"
)
for workload in "${workloads[@]}"; do
IFS=':' read -r type count name <<< "$workload"
echo "Allocating $count $type resources for $name"
aitbc resource allocate --resource-type "$type" --quantity "$count" --mock
done
```
### Resource Monitoring Loop
Continuously monitor resource utilization:
```bash
#!/bin/bash
# monitor_resources.sh
while true; do
clear
echo "=== Resource Utilization ==="
aitbc resource utilization --mock
echo ""
echo "Press Ctrl+C to exit"
sleep 10
done
```
### Automated Resource Cleanup
Release idle resources automatically:
```bash
#!/bin/bash
# cleanup_idle_resources.sh
# Get list of allocated resources
resources=$(aitbc resource status --format json | jq -r '.resources[] | select(.status=="allocated") | .id')
for res_id in $resources; do
# Check utilization (mock example)
util=$(aitbc resource utilization --resource-id "$res_id" --mock | jq -r '.utilization')
if [ "$util" -lt 10 ]; then
echo "Releasing idle resource: $res_id (utilization: $util%)"
aitbc resource release "$res_id" --mock
fi
done
```
---
## ⚠️ **Important Notes**
### Experimental Commands
- **allocate, list, release, utilization, optimize** are experimental
- Require `--mock` flag for testing
- May change in future versions
- Not recommended for production use yet
### Resource States
- **available**: Resource ready for allocation
- **allocated**: Resource currently in use
- **deallocated**: Resource released and cleaned up
- **maintenance**: Resource under maintenance
### Resource Types
- **gpu**: Graphics processing units for AI workloads
- **cpu**: Central processing units for general compute
- **memory**: RAM resources for data processing
- **storage**: Disk resources for data storage
### Coordinator-API Integration
- `status` and `deallocate` use coordinator-api
- Require coordinator-api running at http://127.0.0.1:18000
- API calls validated and tracked
- Network errors handled gracefully
---
## 🐛 **Troubleshooting**
### Experimental command without --mock flag
**Error:**
```
Error: EXPERIMENTAL command - use --mock flag for testing
```
**Solution:**
```bash
# Add --mock flag
aitbc resource allocate --resource-type gpu --quantity 4 --mock
```
### Resource not found
**Error:**
```
Error: Resource 'res_999' not found
```
**Solution:**
```bash
# List available resources
aitbc resource status
# Verify resource ID
aitbc resource status --resource-id res_123
```
### Coordinator-api unavailable
**Error:**
```
Error: coordinator-api unavailable at http://127.0.0.1:18000
```
**Solution:**
```bash
# Check coordinator-api status
curl http://127.0.0.1:18000/health
# Start coordinator-api if not running
systemctl start aitbc-coordinator-api
```
### Deallocation confirmation
**Issue:**
Deallocation requires confirmation but you want to automate
**Solution:**
```bash
# Use --force flag
aitbc resource deallocate res_123 --force
# Or pipe confirmation
echo 'y' | aitbc resource deallocate res_123
```
---
## 📊 **Testing**
Run the integration test script to verify resource operations:
```bash
# Run pytest tests
cd /opt/aitbc
pytest tests/cli/test_resource.py -v
# Run bash integration test
scripts/testing/test_resource.sh
```
**Expected test output:**
```
tests/cli/test_resource.py::TestResourceCommands::test_resource_status_all PASSED
tests/cli/test_resource.py::TestResourceCommands::test_resource_status_specific PASSED
tests/cli/test_resource.py::TestResourceCommands::test_resource_deallocate PASSED
tests/cli/test_resource.py::TestResourceCommands::test_resource_allocate_with_mock PASSED
...
```
---
## 🎓 **Summary**
In this scenario, you learned:
- How to check resource status across the network
- How to allocate resources for workloads (experimental)
- How to list available resources (experimental)
- How to monitor resource utilization (experimental)
- How to optimize resource allocation (experimental)
- How to release resources (experimental)
- How to deallocate resources with confirmation
**Key Takeaways:**
- Resource management enables efficient compute utilization
- Experimental commands require `--mock` flag
- Status and deallocation use coordinator-api
- Utilization tracking helps optimize performance
- Automated cleanup can reduce waste
- Force deallocation bypasses confirmation
---
## 🔄 **Related Scenarios**
- **Scenario 07**: [AI Job Submission](./07_ai_job_submission.md) - AI job resource usage
- **Scenario 09**: [GPU Listing](./09_gpu_listing.md) - GPU marketplace
- **Scenario 50**: [Workflow Management](./50_workflow_management.md) - Workflow resource orchestration
- **Scenario 48**: [Configuration Profiles](./48_config_profiles.md) - Config management

507
docs/scenarios/50_workflow_management.md Normal file
View File

@@ -0,0 +1,507 @@
# Workflow Management for hermes Agents
**Level**: Intermediate
**Prerequisites**: Basic CLI knowledge, AITBC CLI installed, coordinator-api running
**Estimated Time**: 20 minutes
**Last Updated**: 2026-05-27
**Version**: 1.0
## 🧭 **Navigation Path:**
**🏠 [Documentation Home](../README.md)** → **🎭 [Agent Scenarios](./README.md)** → *You are here*
**breadcrumb**: Home → Scenarios → Workflow Management
---
## 🎯 **See Also:**
- **📖 Previous Scenario**: [48 Configuration Profiles](./48_config_profiles.md)
- **📖 Next Scenario**: [51 Simulation Scenarios](./51_simulation_scenarios.md)
- **⚙️ Workflow Documentation**: [CLI Workflow Commands](../cli/CLI_DOCUMENTATION.md)
---
## 📚 **Scenario Overview**
This scenario demonstrates how hermes agents use workflow management to orchestrate complex multi-step operations. Workflows enable agents to execute predefined sequences of tasks such as GPU marketplace operations, AI job processing, and mining optimization.
### **Use Case**
An hermes agent needs to:
- Execute complex multi-step operations automatically
- Track workflow execution status and progress
- Stop running workflows when needed
- Manage multiple concurrent workflows
- Use dry-run mode to validate workflows before execution
### **What You'll Learn**
- List available workflows
- Run workflows with configuration
- Check workflow execution status
- Stop running workflows
- Use dry-run mode for validation
- Create custom workflow configurations
### **Features Combined**
- **Workflow Execution**: Run predefined and custom workflows
- **Status Tracking**: Monitor workflow progress and state
- **Workflow Control**: Start, stop, and manage workflows
- **Configuration Support**: Use config files for workflow parameters
- **Dry-Run Mode**: Validate workflows without execution
---
## 📋 **Prerequisites**
### **Knowledge Required**
- Basic command-line interface usage
- Understanding of YAML configuration files
- AITBC CLI installed and accessible
- coordinator-api running (for API-integrated workflows)
### **System Requirements**
- AITBC CLI installed
- coordinator-api running (http://127.0.0.1:18000)
- Write access to current directory for config files
---
## 🚀 **Quick Start**
```bash
# List available workflows
aitbc workflow list
# Run a workflow
aitbc workflow run gpu-marketplace
# Check workflow status
aitbc workflow status gpu-marketplace
# Stop a running workflow
aitbc workflow stop gpu-marketplace
```
---
## 📖 **Detailed Steps**
### Step 1: List Available Workflows
View all available workflows and their status:
```bash
aitbc workflow list
```
**Expected Output:**
```
Available workflows:
- gpu-marketplace: active (5 steps)
- ai-job-processing: active (3 steps)
- mining-optimization: inactive (4 steps)
```
**JSON format:**
```bash
aitbc workflow list --format json
```
**Expected JSON Output:**
```json
[
{
"name": "gpu-marketplace",
"status": "active",
"steps": 5
},
{
"name": "ai-job-processing",
"status": "active",
"steps": 3
},
{
"name": "mining-optimization",
"status": "inactive",
"steps": 4
}
]
```
### Step 2: Run Workflow (Dry Run)
Validate a workflow without executing it:
```bash
aitbc workflow run gpu-marketplace --dry-run
```
**Expected Output:**
```
Dry run for workflow gpu-marketplace
Would execute workflow without making changes
```
**What happens:**
- Workflow validation performed
- No actual execution
- Useful for testing workflow configuration
### Step 3: Run Workflow with Configuration
Create a workflow configuration file:
```bash
cat > workflow_config.yaml << EOF
gpu_count: 4
max_price: 100.0
min_memory_gb: 16
timeout: 300
EOF
```
Run workflow with configuration:
```bash
aitbc workflow run gpu-marketplace --config workflow_config.yaml
```
**Expected Output:**
```
Run workflow gpu-marketplace
Using config: workflow_config.yaml
Execution ID: wf_exec_1716789123
Status: Running
```
**What happens:**
- Workflow started with configuration
- Unique execution ID assigned
- Workflow begins executing steps
### Step 4: Check Workflow Status
Monitor workflow execution:
```bash
aitbc workflow status gpu-marketplace
```
**Expected Output:**
```
Get status for workflow gpu-marketplace
Status: Running
Last execution: 2026-05-27 08:30:45
Current step: 2/5
Progress: 40%
```
**What happens:**
- Current workflow state retrieved
- Execution progress displayed
- Step-by-step status shown
### Step 5: Stop Running Workflow
Terminate a running workflow:
```bash
aitbc workflow stop gpu-marketplace
```
**Expected Output:**
```
Stop workflow gpu-marketplace
Workflow stopped successfully
```
**What happens:**
- Workflow execution terminated
- Resources cleaned up
- Status updated to stopped
### Step 6: Run AI Job Processing Workflow
Execute AI job workflow:
```bash
aitbc workflow run ai-job-processing
```
**Expected Output:**
```
Run workflow ai-job-processing
Execution ID: wf_exec_1716789234
Status: Running
```
Check status:
```bash
aitbc workflow status ai-job-processing
```
**Expected Output:**
```
Get status for workflow ai-job-processing
Status: Completed
Last execution: 2026-05-27 08:35:12
Duration: 45 seconds
Jobs processed: 10
```
---
## 🔧 **Advanced Usage**
### Custom Workflow Names
Use custom workflow names with various formats:
```bash
# With dashes
aitbc workflow run my-custom-workflow
# With underscores
aitbc workflow run my_custom_workflow
# With dots
aitbc workflow run my.workflow
# CamelCase
aitbc workflow run MyWorkflow
```
### Workflow Configuration Parameters
Create detailed configuration files:
```yaml
# gpu_workflow_config.yaml
workflow:
name: gpu-marketplace
parallel: true
resources:
gpu:
count: 4
min_memory_gb: 16
architecture: "NVIDIA"
edge_optimized: true
pricing:
max_price: 100.0
bid_strategy: "aggressive"
execution:
timeout: 300
retry_count: 3
on_failure: "continue"
```
Run with detailed config:
```bash
aitbc workflow run gpu-marketplace --config gpu_workflow_config.yaml
```
### Batch Workflow Execution
Run multiple workflows sequentially:
```bash
#!/bin/bash
# run_workflows.sh
workflows=("gpu-marketplace" "ai-job-processing" "mining-optimization")
for wf in "${workflows[@]}"; do
echo "Running workflow: $wf"
aitbc workflow run "$wf"
# Wait for completion
while true; do
status=$(aitbc workflow status "$wf" --format json | jq -r '.status')
if [ "$status" = "Completed" ] || [ "$status" = "Failed" ]; then
break
fi
sleep 5
done
done
```
### Workflow Monitoring
Monitor workflow with continuous status checks:
```bash
#!/bin/bash
# monitor_workflow.sh
workflow_name=$1
while true; do
clear
aitbc workflow status "$workflow_name"
sleep 2
done
```
Usage:
```bash
./monitor_workflow.sh gpu-marketplace
```
---
## ⚠️ **Important Notes**
### Workflow Execution IDs
- Each workflow execution gets a unique ID: `wf_exec_<timestamp>`
- Use execution IDs for tracking and debugging
- IDs are generated automatically
### Workflow States
- **Running**: Workflow currently executing
- **Completed**: Workflow finished successfully
- **Failed**: Workflow encountered an error
- **Stopped**: Workflow manually terminated
- **Pending**: Workflow queued for execution
### Configuration File Format
- YAML format supported
- JSON format supported
- Must be valid YAML/JSON syntax
- Parameters depend on workflow type
### Dry-Run Limitations
- No actual execution
- No side effects
- Validation only
- Useful for testing
---
## 🐛 **Troubleshooting**
### Workflow not found
**Error:**
```
Error: Workflow 'my_workflow' not found
```
**Solution:**
```bash
# List available workflows
aitbc workflow list
# Check workflow name spelling
aitbc workflow run gpu-marketplace # Correct
aitbc workflow run gpu_marketplace # Incorrect
```
### Configuration file error
**Error:**
```
Error: Invalid configuration file
```
**Solution:**
```bash
# Validate YAML syntax
python3 -c "import yaml; yaml.safe_load(open('workflow_config.yaml'))"
# Check file path
ls -la workflow_config.yaml
# Use absolute path
aitbc workflow run gpu-marketplace --config /full/path/to/config.yaml
```
### Workflow stuck in running state
**Issue:**
Workflow status shows "Running" but not progressing
**Solution:**
```bash
# Stop the workflow
aitbc workflow stop stuck_workflow
# Check coordinator-api status
curl http://127.0.0.1:18000/health
# Review workflow logs
# (Location depends on workflow implementation)
```
### Permission denied on config file
**Error:**
```
Error: Permission denied reading config file
```
**Solution:**
```bash
# Check file permissions
ls -la workflow_config.yaml
# Fix permissions
chmod 644 workflow_config.yaml
# Use different location
aitbc workflow run gpu-marketplace --config ~/configs/workflow.yaml
```
---
## 📊 **Testing**
Run the integration test script to verify workflow operations:
```bash
# Run pytest tests
cd /opt/aitbc
pytest tests/cli/test_workflow.py -v
# Run bash integration test
scripts/testing/test_workflow_cli.sh
```
**Expected test output:**
```
tests/cli/test_workflow.py::TestWorkflowCommands::test_workflow_run_basic PASSED
tests/cli/test_workflow.py::TestWorkflowCommands::test_workflow_list PASSED
tests/cli/test_workflow.py::TestWorkflowCommands::test_workflow_status PASSED
tests/cli/test_workflow.py::TestWorkflowCommands::test_workflow_stop PASSED
...
```
---
## 🎓 **Summary**
In this scenario, you learned:
- How to list available workflows
- How to run workflows with and without configuration
- How to check workflow execution status
- How to stop running workflows
- How to use dry-run mode for validation
- How to create custom workflow configurations
**Key Takeaways:**
- Workflows orchestrate complex multi-step operations
- Each execution gets a unique ID for tracking
- Dry-run mode validates without execution
- Configuration files customize workflow behavior
- Status monitoring enables workflow control
---
## 🔄 **Related Scenarios**
- **Scenario 07**: [AI Job Submission](./07_ai_job_submission.md) - AI job operations
- **Scenario 09**: [GPU Listing](./09_gpu_listing.md) - GPU marketplace operations
- **Scenario 13**: [Mining Setup](./13_mining_setup.md) - Mining operations
- **Scenario 48**: [Configuration Profiles](./48_config_profiles.md) - Config management

570
docs/scenarios/51_simulation_scenarios.md Normal file
View File

@@ -0,0 +1,570 @@
# Simulation Scenarios for hermes Agents
**Level**: Intermediate
**Prerequisites**: Basic CLI knowledge, AITBC CLI installed, coordinator-api running
**Estimated Time**: 30 minutes
**Last Updated**: 2026-05-27
**Version**: 1.0
## 🧭 **Navigation Path:**
**🏠 [Documentation Home](../README.md)** → **🎭 [Agent Scenarios](./README.md)** → *You are here*
**breadcrumb**: Home → Scenarios → Simulation Scenarios
---
## 🎯 **See Also:**
- **📖 Previous Scenario**: [49 Resource Management](./49_resource_management.md)
- **📖 Next Scenario**: [52 Edge Advanced Operations](./52_edge_advanced_operations.md)
- **⚙️ Simulation Documentation**: [CLI Simulation Commands](../cli/CLI_DOCUMENTATION.md)
---
## 📚 **Scenario Overview**
This scenario demonstrates how hermes agents use simulation commands to model and test various AITBC ecosystem scenarios including blockchain operations, wallet behavior, price movements, network topology, and AI job processing. Simulations enable agents to validate strategies and predict outcomes before executing real transactions.
### **Use Case**
An hermes agent needs to:
- Model blockchain behavior under different conditions
- Simulate wallet creation and balance distribution
- Predict price movements with various volatility patterns
- Test network topology and latency scenarios
- Simulate AI job submission and processing workflows
- Validate strategies before real execution
### **What You'll Learn**
- Run blockchain simulations with custom parameters
- Simulate wallet creation and balance distribution
- Model price movements with trends and volatility
- Test network topology and latency scenarios
- Simulate AI job processing workflows
- Run simulations in async mode
- Monitor simulation status and retrieve results
### **Features Combined**
- **Blockchain Simulation**: Model block generation and transaction processing
- **Wallet Simulation**: Create wallets with various balance distributions
- **Price Simulation**: Model price movements with trends and volatility
- **Network Simulation**: Test network topology and latency scenarios
- **AI Jobs Simulation**: Model AI job submission and processing
- **Async Execution**: Run simulations asynchronously
- **Status Tracking**: Monitor simulation progress and results
---
## 📋 **Prerequisites**
### **Knowledge Required**
- Basic command-line interface usage
- Understanding of blockchain concepts
- AITBC CLI installed and accessible
- coordinator-api running (for API-integrated simulations)
### **System Requirements**
- AITBC CLI installed
- coordinator-api running (http://127.0.0.1:18000)
- Sufficient system resources for simulations
---
## 🚀 **Quick Start**
```bash
# Blockchain simulation
aitbc simulate blockchain --blocks 10 --transactions 50
# Wallets simulation
aitbc simulate wallets --count 5 --balance 1000
# Price simulation
aitbc simulate price --days 30 --volatility 0.1
# Network simulation
aitbc simulate network --nodes 10 --latency 50
# AI jobs simulation
aitbc simulate ai-jobs --jobs 20 --duration 300
```
---
## 📖 **Detailed Steps**
### Step 1: Blockchain Simulation
Simulate blockchain block generation and transaction processing:
```bash
aitbc simulate blockchain --blocks 10 --transactions 50
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789123",
"blocks": 10,
"transactions": 50,
"status": "completed",
"block_time_avg": "2.5s"
}
```
**What happens:**
- Blockchain simulation executed
- Blocks generated with transactions
- Performance metrics collected
### Step 2: Blockchain with Custom Parameters
Simulate blockchain with difficulty and custom parameters:
```bash
aitbc simulate blockchain --blocks 100 --transactions 500 --difficulty 5
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789234",
"blocks": 100,
"transactions": 500,
"difficulty": 5,
"status": "completed"
}
```
### Step 3: Wallets Simulation
Simulate wallet creation with balance distribution:
```bash
aitbc simulate wallets --count 5 --balance 1000
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789345",
"wallets": 5,
"balance": 1000,
"distribution": "uniform",
"status": "completed"
}
```
### Step 4: Wallets with Exponential Distribution
Simulate wallets with exponential balance distribution:
```bash
aitbc simulate wallets --count 10 --distribution exponential
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789456",
"wallets": 10,
"distribution": "exponential",
"status": "completed"
}
```
### Step 5: Price Simulation
Simulate price movements with volatility:
```bash
aitbc simulate price --days 30 --volatility 0.1
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789567",
"days": 30,
"volatility": 0.1,
"prices": [100.0, 105.2, 98.7, ...],
"status": "completed"
}
```
### Step 6: Price with Trend
Simulate price movements with bullish trend:
```bash
aitbc simulate price --days 90 --trend bullish --volatility 0.15
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789678",
"days": 90,
"trend": "bullish",
"volatility": 0.15,
"status": "completed"
}
```
### Step 7: Network Simulation
Simulate network topology and latency:
```bash
aitbc simulate network --nodes 10 --latency 50
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789789",
"nodes": 10,
"latency": 50,
"topology": "random",
"status": "completed"
}
```
### Step 8: Network with Custom Topology
Simulate network with mesh topology:
```bash
aitbc simulate network --nodes 20 --topology mesh --latency 100
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716789890",
"nodes": 20,
"topology": "mesh",
"latency": 100,
"status": "completed"
}
```
### Step 9: AI Jobs Simulation
Simulate AI job submission and processing:
```bash
aitbc simulate ai-jobs --jobs 20 --duration 300
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716790001",
"jobs": 20,
"duration": 300,
"status": "completed"
}
```
### Step 10: AI Jobs with GPU Requirements
Simulate AI jobs with GPU requirements:
```bash
aitbc simulate ai-jobs --jobs 30 --gpu-required --duration 600
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716790112",
"jobs": 30,
"gpu_required": true,
"duration": 600,
"status": "completed"
}
```
### Step 11: Run Simulation
Run a simulation with type and duration:
```bash
aitbc simulate run --type blockchain --duration 60
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716790223",
"type": "blockchain",
"duration": 60,
"status": "started"
}
```
### Step 12: Run Async Simulation
Run simulation in async mode:
```bash
aitbc simulate run --type network --async --duration 120
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716790334",
"type": "network",
"async": true,
"status": "started"
}
```
### Step 13: Check Simulation Status
Monitor simulation progress:
```bash
aitbc simulate status sim_1716790223
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716790223",
"status": "running",
"progress": "75%",
"elapsed_time": "45s"
}
```
### Step 14: Get Simulation Results
Retrieve simulation results:
```bash
aitbc simulate result sim_1716790223
```
**Expected Output:**
```json
{
"simulation_id": "sim_1716790223",
"status": "completed",
"results": {
"blocks_generated": 10,
"transactions_processed": 50,
"avg_block_time": "2.5s"
}
}
```
---
## 🔧 **Advanced Usage**
### Custom Simulation Scenarios
Create complex simulation scenarios by combining parameters:
```bash
# High-volume blockchain simulation
aitbc simulate blockchain --blocks 1000 --transactions 10000 --difficulty 8
# Large-scale wallet simulation
aitbc simulate wallets --count 1000 --distribution power-law --balance 10000
# Long-term price simulation
aitbc simulate price --days 365 --trend bearish --volatility 0.25
# Complex network topology
aitbc simulate network --nodes 50 --topology hierarchical --latency 200
# Complex AI job workflow
aitbc simulate ai-jobs --jobs 100 --gpu-required --multi-gpu --duration 3600
```
### Batch Simulations
Run multiple simulations sequentially:
```bash
#!/bin/bash
# run_simulations.sh
sim_types=("blockchain" "wallets" "price" "network" "ai-jobs")
for sim in "${sim_types[@]}"; do
echo "Running simulation: $sim"
aitbc simulate run --type "$sim" --duration 60
# Wait for completion
sleep 5
done
```
### Simulation Comparison
Compare results from different simulations:
```bash
# Run simulation A
aitbc simulate blockchain --blocks 100 --transactions 500 --difficulty 3
sim_id_a=$(aitbc simulate run --type blockchain --duration 60 | jq -r '.simulation_id')
# Run simulation B
aitbc simulate blockchain --blocks 100 --transactions 500 --difficulty 7
sim_id_b=$(aitbc simulate run --type blockchain --duration 60 | jq -r '.simulation_id')
# Compare results
aitbc simulate result $sim_id_a > result_a.json
aitbc simulate result $sim_id_b > result_b.json
diff result_a.json result_b.json
```
---
## ⚠️ **Important Notes**
### Simulation IDs
- Each simulation gets a unique ID: `sim_<timestamp>`
- Use simulation IDs for tracking and result retrieval
- IDs are generated automatically
### Simulation States
- **Started**: Simulation initialized
- **Running**: Simulation in progress
- **Completed**: Simulation finished successfully
- **Failed**: Simulation encountered an error
### Resource Requirements
- Large simulations may require significant CPU/memory
- Async mode recommended for long-running simulations
- Monitor system resources during simulations
### Output Formats
- JSON format for programmatic processing
- Table format for human-readable output
- Use `--format json` or `--format table` to specify
---
## 🐛 **Troubleshooting**
### Simulation not found
**Error:**
```
Error: Simulation 'sim_123' not found
```
**Solution:**
```bash
# Check simulation ID
aitbc simulate status sim_123
# List recent simulations (if available)
aitbc simulate list
```
### Coordinator-api unavailable
**Error:**
```
Error: coordinator-api not available
```
**Solution:**
```bash
# Check coordinator-api status
curl http://127.0.0.1:18000/health
# Start coordinator-api if needed
systemctl start aitbc-coordinator-api
```
### Simulation stuck in running state
**Issue:**
Simulation status shows "Running" but not progressing
**Solution:**
```bash
# Check coordinator-api status
curl http://127.0.0.1:18000/health
# Review simulation logs
# (Location depends on implementation)
# Cancel simulation if needed
aitbc simulate cancel sim_123
```
### Invalid simulation parameters
**Error:**
```
Error: Invalid parameter value
```
**Solution:**
```bash
# Check parameter ranges
aitbc simulate blockchain --help
# Use valid parameter values
aitbc simulate blockchain --blocks 10 # Valid
aitbc simulate blockchain --blocks -5 # Invalid
```
---
## 📊 **Testing**
Run the integration test script to verify simulation operations:
```bash
# Run pytest tests
cd /opt/aitbc
pytest tests/cli/test_simulate_integration.py -v
# Run bash integration test
scripts/testing/test_simulate.sh
```
**Expected test output:**
```
tests/cli/test_simulate_integration.py::TestSimulateCommandsIntegration::test_simulate_blockchain PASSED
tests/cli/test_simulate_integration.py::TestSimulateCommandsIntegration::test_simulate_wallets PASSED
tests/cli/test_simulate_integration.py::TestSimulateCommandsIntegration::test_simulate_price PASSED
...
```
---
## 🎓 **Summary**
In this scenario, you learned:
- How to run blockchain simulations with custom parameters
- How to simulate wallet creation and balance distribution
- How to model price movements with trends and volatility
- How to test network topology and latency scenarios
- How to simulate AI job processing workflows
- How to run simulations in async mode
- How to monitor simulation status and retrieve results
**Key Takeaways:**
- Simulations enable strategy validation before real execution
- Each simulation gets a unique ID for tracking
- Async mode recommended for long-running simulations
- Custom parameters enable realistic scenario modeling
- Status monitoring enables simulation control
- Results retrieval enables analysis and comparison
---
## 🔄 **Related Scenarios**
- **Scenario 03**: [Genesis Deployment](./03_genesis_deployment.md) - Blockchain operations
- **Scenario 07**: [AI Job Submission](./07_ai_job_submission.md) - AI job operations
- **Scenario 09**: [GPU Listing](./09_gpu_listing.md) - GPU marketplace operations
- **Scenario 49**: [Resource Management](./49_resource_management.md) - Resource allocation

680
docs/scenarios/52_edge_advanced_operations.md Normal file
View File

@@ -0,0 +1,680 @@
# Edge Advanced Operations for hermes Agents
**Level**: Advanced
**Prerequisites**: Basic CLI knowledge, AITBC CLI installed, edge-api running
**Estimated Time**: 35 minutes
**Last Updated**: 2026-05-27
**Version**: 1.0
## 🧭 **Navigation Path:**
**🏠 [Documentation Home](../README.md)** → **🎭 [Agent Scenarios](./README.md)** → *You are here*
**breadcrumb**: Home → Scenarios → Edge Advanced Operations
---
## 🎯 **See Also:**
- **📖 Previous Scenario**: [51 Simulation Scenarios](./51_simulation_scenarios.md)
- **📖 Next Scenario**: [01 Wallet Basics](./01_wallet_basics.md)
- **⚙️ Edge Documentation**: [CLI Edge Commands](../cli/CLI_DOCUMENTATION.md)
---
## 📚 **Scenario Overview**
This scenario demonstrates how hermes agents use advanced edge API operations to manage compute islands, GPU resources, databases, serve requests, and metrics. Edge operations enable agents to interact with distributed compute infrastructure at the network edge for low-latency AI workloads.
### **Use Case**
An hermes agent needs to:
- Manage compute island membership and bridging
- Discover and manage GPU resources at the edge
- Initialize and synchronize edge databases
- Submit and track compute serve requests
- Record and query performance metrics
- Monitor edge infrastructure health
### **What You'll Learn**
- Leave and bridge compute islands
- List, get, remove, and scan GPUs
- Initialize, list, get, delete, and sync databases
- Submit, list, get, cancel, and retrieve serve requests
- Record, list, get, and delete metrics
- Use edge API for distributed compute operations
### **Features Combined**
- **Island Management**: Leave islands and bridge between islands
- **GPU Operations**: Discover and manage GPU resources
- **Database Operations**: Initialize and sync edge databases
- **Serve Operations**: Submit and track compute requests
- **Metrics Operations**: Record and query performance metrics
- **Edge API Integration**: Direct API calls to edge-api service
---
## 📋 **Prerequisites**
### **Knowledge Required**
- Basic command-line interface usage
- Understanding of distributed compute concepts
- AITBC CLI installed and accessible
- edge-api running (http://127.0.0.1:8200)
### **System Requirements**
- AITBC CLI installed
- edge-api running (http://127.0.0.1:8200)
- Network connectivity to edge nodes
- Appropriate permissions for edge operations
---
## 🚀 **Quick Start**
```bash
# List available GPUs
aitbc edge gpu list_gpus
# Leave an island
aitbc edge island leave --island-id my_island
# Initialize a database
aitbc edge database init_db --db-name my_db
# Submit a serve request
aitbc edge serve submit_request --request-type compute --parameters '{"gpu_count": 2}'
# Record a metric
aitbc edge metrics record --metric-name gpu_utilization --value 85.5
```
---
## 📖 **Detailed Steps**
### Step 1: Island Advanced Operations
#### Leave an Island
```bash
aitbc edge island leave --island-id test_island_123
```
**Expected Output:**
```json
{
"island_id": "test_island_123",
"status": "left",
"timestamp": "2026-05-27T09:00:00Z"
}
```
**What happens:**
- Agent removes itself from the specified island
- Island membership updated
- Resources deallocated from island
#### Bridge Between Islands
```bash
aitbc edge island bridge --source island_a --target island_b
```
**Expected Output:**
```json
{
"bridge_id": "bridge_abc123",
"source": "island_a",
"target": "island_b",
"status": "established"
}
```
**What happens:**
- Network bridge created between islands
- Resources can be shared across islands
- Latency-optimized routing enabled
### Step 2: GPU Operations
#### List Available GPUs
```bash
aitbc edge gpu list_gpus
```
**Expected Output:**
```json
{
"gpus": [
{
"id": "gpu_1",
"type": "NVIDIA",
"memory_gb": 16,
"status": "available"
},
{
"id": "gpu_2",
"type": "NVIDIA",
"memory_gb": 32,
"status": "allocated"
}
]
}
```
#### Get Specific GPU Info
```bash
aitbc edge gpu get_gpu --gpu-id gpu_1
```
**Expected Output:**
```json
{
"gpu_id": "gpu_1",
"type": "NVIDIA",
"memory_gb": 16,
"status": "available",
"utilization": 0.0,
"temperature": 45
}
```
#### Remove a GPU
```bash
aitbc edge gpu remove_gpu --gpu-id gpu_1
```
**Expected Output:**
```json
{
"gpu_id": "gpu_1",
"status": "removed",
"timestamp": "2026-05-27T09:05:00Z"
}
```
#### Scan for Available GPUs
```bash
aitbc edge gpu scan_gpus
```
**Expected Output:**
```json
{
"scan_results": {
"found": 4,
"gpus": [
{"id": "gpu_1", "type": "NVIDIA", "memory": 16},
{"id": "gpu_2", "type": "NVIDIA", "memory": 32}
]
}
}
```
#### Get GPU Metrics
```bash
aitbc edge gpu gpu_metrics --gpu-id gpu_1
```
**Expected Output:**
```json
{
"gpu_id": "gpu_1",
"metrics": {
"utilization": 85.5,
"memory_used": 12.5,
"temperature": 72,
"power_usage": 250
}
}
```
### Step 3: Database Operations
#### Initialize a Database
```bash
aitbc edge database init_db --db-name test_db
```
**Expected Output:**
```json
{
"db_id": "db_123",
"db_name": "test_db",
"status": "initialized",
"location": "/edge/data/test_db"
}
```
#### List Databases
```bash
aitbc edge database list_dbs
```
**Expected Output:**
```json
{
"databases": [
{
"db_id": "db_123",
"db_name": "test_db",
"status": "active"
}
]
}
```
#### Get Database Info
```bash
aitbc edge database get_db --db-id db_123
```
**Expected Output:**
```json
{
"db_id": "db_123",
"db_name": "test_db",
"status": "active",
"size_mb": 256,
"last_sync": "2026-05-27T09:10:00Z"
}
```
#### Delete a Database
```bash
aitbc edge database delete_db --db-id db_123
```
**Expected Output:**
```json
{
"db_id": "db_123",
"status": "deleted",
"timestamp": "2026-05-27T09:15:00Z"
}
```
#### Sync a Database
```bash
aitbc edge database sync_db --db-id db_123
```
**Expected Output:**
```json
{
"db_id": "db_123",
"sync_status": "completed",
"records_synced": 1523,
"duration_seconds": 5.2
}
```
### Step 4: Serve Operations
#### Submit a Serve Request
```bash
aitbc edge serve submit_request --request-type compute --parameters '{"gpu_count": 2, "memory_gb": 32}'
```
**Expected Output:**
```json
{
"request_id": "req_abc123",
"request_type": "compute",
"status": "queued",
"parameters": {
"gpu_count": 2,
"memory_gb": 32
}
}
```
#### List Serve Requests
```bash
aitbc edge serve list_requests
```
**Expected Output:**
```json
{
"requests": [
{
"request_id": "req_abc123",
"status": "running",
"submitted_at": "2026-05-27T09:20:00Z"
}
]
}
```
#### Get Serve Request Info
```bash
aitbc edge serve get_request --request-id req_abc123
```
**Expected Output:**
```json
{
"request_id": "req_abc123",
"status": "running",
"progress": 75,
"started_at": "2026-05-27T09:20:00Z"
}
```
#### Cancel a Serve Request
```bash
aitbc edge serve cancel_request --request-id req_abc123
```
**Expected Output:**
```json
{
"request_id": "req_abc123",
"status": "cancelled",
"cancelled_at": "2026-05-27T09:25:00Z"
}
```
#### Get Serve Request Result
```bash
aitbc edge serve get_result --request-id req_abc123
```
**Expected Output:**
```json
{
"request_id": "req_abc123",
"result": {
"status": "success",
"output": "Job completed successfully",
"metrics": {
"duration": 120,
"gpu_hours": 0.67
}
}
}
```
### Step 5: Metrics Operations
#### Record a Metric
```bash
aitbc edge metrics record --metric-name gpu_utilization --value 85.5
```
**Expected Output:**
```json
{
"metric_id": "metric_xyz789",
"metric_name": "gpu_utilization",
"value": 85.5,
"timestamp": "2026-05-27T09:30:00Z"
}
```
#### List Metrics
```bash
aitbc edge metrics list_metrics
```
**Expected Output:**
```json
{
"metrics": [
{
"metric_id": "metric_xyz789",
"metric_name": "gpu_utilization",
"value": 85.5,
"timestamp": "2026-05-27T09:30:00Z"
}
]
}
```
#### Get a Specific Metric
```bash
aitbc edge metrics get_metric --metric-id metric_xyz789
```
**Expected Output:**
```json
{
"metric_id": "metric_xyz789",
"metric_name": "gpu_utilization",
"value": 85.5,
"timestamp": "2026-05-27T09:30:00Z"
}
```
#### Delete a Metric
```bash
aitbc edge metrics delete_metric --metric-id metric_xyz789
```
**Expected Output:**
```json
{
"metric_id": "metric_xyz789",
"status": "deleted",
"timestamp": "2026-05-27T09:35:00Z"
}
```
---
## 🔧 **Advanced Usage**
### Batch GPU Discovery
```bash
#!/bin/bash
# Scan all edge nodes for GPUs
for node in node1 node2 node3; do
echo "Scanning $node..."
aitbc edge gpu scan_gpus --node $node
done
```
### Automated Database Sync
```bash
#!/bin/bash
# Sync all databases periodically
dbs=$(aitbc edge database list_dbs --format json | jq -r '.databases[].db_id')
for db_id in $dbs; do
echo "Syncing $db_id..."
aitbc edge database sync_db --db-id $db_id
done
```
### Metrics Aggregation
```bash
#!/bin/bash
# Aggregate GPU utilization metrics
metrics=$(aitbc edge metrics list_metrics --format json | jq -r '.metrics[] | select(.metric_name == "gpu_utilization") | .value')
total=0
count=0
for value in $metrics; do
total=$(echo "$total + $value" | bc)
count=$((count + 1))
done
average=$(echo "scale=2; $total / $count" | bc)
echo "Average GPU utilization: $average%"
```
---
## ⚠️ **Important Notes**
### Edge API Availability
- All edge operations require edge-api running at http://127.0.0.1:8200
- Tests will skip if edge-api is not available
- Check edge-api health before operations
### Resource Management
- GPU operations affect actual hardware resources
- Database operations create persistent storage
- Serve requests consume compute resources
- Monitor resource usage to prevent exhaustion
### Island Membership
- Leaving an island deallocates resources
- Bridging islands enables resource sharing
- Island operations affect network topology
- Consider latency when bridging distant islands
### Metrics Retention
- Metrics are stored with timestamps
- Old metrics can be deleted to save space
- Metrics are useful for performance analysis
- Aggregate metrics for trend analysis
---
## 🐛 **Troubleshooting**
### Edge API not available
**Error:**
```
Error: edge-api not running at http://127.0.0.1:8200
```
**Solution:**
```bash
# Check edge-api status
curl http://127.0.0.1:8200/health
# Start edge-api if needed
systemctl start aitbc-edge-api
```
### GPU not found
**Error:**
```
Error: GPU 'gpu_123' not found
```
**Solution:**
```bash
# Scan for available GPUs
aitbc edge gpu scan_gpus
# List all GPUs
aitbc edge gpu list_gpus
# Use correct GPU ID from list
```
### Database sync failed
**Error:**
```
Error: Database sync failed
```
**Solution:**
```bash
# Check database status
aitbc edge database get_db --db-id db_123
# Check network connectivity
ping edge-node
# Retry sync with force flag
aitbc edge database sync_db --db-id db_123 --force
```
### Serve request stuck
**Issue:**
Serve request status shows "running" but not progressing
**Solution:**
```bash
# Cancel stuck request
aitbc edge serve cancel_request --request-id req_123
# Check edge-api logs
journalctl -u aitbc-edge-api -f
# Submit new request
aitbc edge serve submit_request --request-type compute --parameters '{"gpu_count": 2}'
```
---
## 📊 **Testing**
Run the integration test script to verify edge operations:
```bash
# Run pytest tests
cd /opt/aitbc
pytest tests/cli/test_edge_advanced.py -v
# Run bash integration test
scripts/testing/test_edge_advanced.sh
```
**Expected test output:**
```
tests/cli/test_edge_advanced.py::TestEdgeAdvancedCommands::test_edge_island_leave PASSED
tests/cli/test_edge_advanced.py::TestEdgeAdvancedCommands::test_edge_gpu_list_gpus PASSED
tests/cli/test_edge_advanced.py::TestEdgeAdvancedCommands::test_edge_database_init_db PASSED
...
```
---
## 🎓 **Summary**
In this scenario, you learned:
- How to leave and bridge compute islands
- How to discover and manage GPU resources
- How to initialize and sync edge databases
- How to submit and track serve requests
- How to record and query performance metrics
- How to use edge API for distributed compute operations
**Key Takeaways:**
- Edge operations enable low-latency compute at the network edge
- GPU discovery helps optimize resource allocation
- Database sync ensures data consistency across edges
- Serve requests provide on-demand compute resources
- Metrics enable performance monitoring and optimization
- Island management enables flexible network topology
---
## 🔄 **Related Scenarios**
- **Scenario 05**: [Island Creation](./05_island_creation.md) - Basic island operations
- **Scenario 09**: [GPU Listing](./09_gpu_listing.md) - GPU marketplace operations
- **Scenario 49**: [Resource Management](./49_resource_management.md) - Resource allocation
- **Scenario 51**: [Simulation Scenarios](./51_simulation_scenarios.md) - Simulation testing

167
scripts/testing/test_config_profiles.sh Executable file
View File

@@ -0,0 +1,167 @@
#!/bin/bash
# Integration test script for config profiles CLI commands
# Tests profile save, list, load, and delete operations with file system validation
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
PROFILES_DIR="$HOME/.config/aitbc/profiles"
TEST_PROFILE="test_profile_$$"
CONFIG_FILE=".aitbc.yaml"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
# Helper functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
run_test() {
local test_name="$1"
local test_command="$2"
TESTS_RUN=$((TESTS_RUN + 1))
log_info "Running: $test_name"
if eval "$test_command"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: $test_name"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: $test_name"
return 1
fi
}
cleanup() {
log_info "Cleaning up test artifacts..."
# Remove test profile if it exists
if [ -f "$PROFILES_DIR/$TEST_PROFILE.yaml" ]; then
rm -f "$PROFILES_DIR/$TEST_PROFILE.yaml"
log_info "Removed test profile: $TEST_PROFILE"
fi
# Remove test config file if it exists
if [ -f "$REPO_ROOT/$CONFIG_FILE" ]; then
rm -f "$REPO_ROOT/$CONFIG_FILE"
log_info "Removed test config file"
fi
}
# Setup
cd "$REPO_ROOT"
mkdir -p "$PROFILES_DIR"
log_info "Starting config profiles integration tests"
log_info "Test profile name: $TEST_PROFILE"
log_info "Profiles directory: $PROFILES_DIR"
# Test 1: Save profile
run_test "Save profile" "aitbc config profiles save $TEST_PROFILE"
# Verify profile file was created
if [ -f "$PROFILES_DIR/$TEST_PROFILE.yaml" ]; then
log_info "Profile file created successfully"
else
log_error "Profile file was not created"
TESTS_FAILED=$((TESTS_FAILED + 1))
fi
# Test 2: List profiles (should include our test profile)
run_test "List profiles" "aitbc config profiles list | grep -q $TEST_PROFILE"
# Test 3: Load profile
run_test "Load profile" "aitbc config profiles load $TEST_PROFILE"
# Verify config file was created
if [ -f "$REPO_ROOT/$CONFIG_FILE" ]; then
log_info "Config file created by profile load"
else
log_error "Config file was not created by profile load"
TESTS_FAILED=$((TESTS_FAILED + 1))
fi
# Test 4: Delete profile (with confirmation)
run_test "Delete profile" "echo 'y' | aitbc config profiles delete $TEST_PROFILE"
# Verify profile file was deleted
if [ ! -f "$PROFILES_DIR/$TEST_PROFILE.yaml" ]; then
log_info "Profile file deleted successfully"
else
log_error "Profile file was not deleted"
TESTS_FAILED=$((TESTS_FAILED + 1))
fi
# Test 5: Save again for cancellation test
run_test "Save profile again" "aitbc config profiles save $TEST_PROFILE"
# Test 6: Delete with cancellation
run_test "Delete profile (cancelled)" "echo 'n' | aitbc config profiles delete $TEST_PROFILE"
# Verify profile still exists after cancellation
if [ -f "$PROFILES_DIR/$TEST_PROFILE.yaml" ]; then
log_info "Profile file preserved after cancellation"
else
log_error "Profile file was deleted despite cancellation"
TESTS_FAILED=$((TESTS_FAILED + 1))
fi
# Test 7: Load non-existent profile (should fail)
if aitbc config profiles load nonexistent_profile_$$ 2>&1 | grep -q "not found"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: Load non-existent profile fails correctly"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: Load non-existent profile should fail"
fi
TESTS_RUN=$((TESTS_RUN + 1))
# Test 8: Delete non-existent profile (should fail)
if aitbc config profiles delete nonexistent_profile_$$ 2>&1 | grep -q "not found"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: Delete non-existent profile fails correctly"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: Delete non-existent profile should fail"
fi
TESTS_RUN=$((TESTS_RUN + 1))
# Cleanup
cleanup
# Summary
echo ""
echo "========================================"
echo "Test Summary"
echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo "Tests Passed: $TESTS_PASSED"
echo "Tests Failed: $TESTS_FAILED"
echo "========================================"
if [ $TESTS_FAILED -eq 0 ]; then
log_info "All tests passed!"
exit 0
else
log_error "Some tests failed!"
exit 1
fi

149
scripts/testing/test_edge_advanced.sh Executable file
View File

@@ -0,0 +1,149 @@
#!/bin/bash
# Integration test script for edge advanced CLI commands
# Tests island leave/bridge, GPU operations, database operations, serve operations, and metrics
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
EDGE_URL="http://127.0.0.1:8200"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
TESTS_SKIPPED=0
# Helper functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
check_edge() {
if curl -s -f "$EDGE_URL/health" > /dev/null 2>&1; then
return 0
else
return 1
fi
}
run_test() {
local test_name="$1"
local test_command="$2"
local require_edge="${3:-true}"
TESTS_RUN=$((TESTS_RUN + 1))
if [ "$require_edge" = "true" ] && ! check_edge; then
log_warn "SKIPPED: $test_name (edge-api not available)"
TESTS_SKIPPED=$((TESTS_SKIPPED + 1))
return 0
fi
log_info "Running: $test_name"
if eval "$test_command"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: $test_name"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: $test_name"
return 1
fi
}
# Setup
cd "$REPO_ROOT"
log_info "Starting edge advanced CLI integration tests"
log_info "Edge API URL: $EDGE_URL"
# Island advanced operations
run_test "Island leave" "aitbc edge island leave --island-id test_island_123" "true"
run_test "Island bridge" "aitbc edge island bridge --source island_a --target island_b" "true"
# GPU operations
run_test "GPU list" "aitbc edge gpu list_gpus" "true"
run_test "GPU get" "aitbc edge gpu get_gpu --gpu-id gpu_123" "true"
run_test "GPU remove" "aitbc edge gpu remove_gpu --gpu-id gpu_123" "true"
run_test "GPU scan" "aitbc edge gpu scan_gpus" "true"
run_test "GPU metrics" "aitbc edge gpu gpu_metrics --gpu-id gpu_123" "true"
# Database operations
run_test "Database init" "aitbc edge database init_db --db-name test_db" "true"
run_test "Database list" "aitbc edge database list_dbs" "true"
run_test "Database get" "aitbc edge database get_db --db-id db_123" "true"
run_test "Database delete" "aitbc edge database delete_db --db-id db_123" "true"
run_test "Database sync" "aitbc edge database sync_db --db-id db_123" "true"
# Serve operations
run_test "Serve submit request" "aitbc edge serve submit_request --request-type compute --parameters '{\"gpu_count\": 2}'" "true"
run_test "Serve list requests" "aitbc edge serve list_requests" "true"
run_test "Serve get request" "aitbc edge serve get_request --request-id req_123" "true"
run_test "Serve cancel request" "aitbc edge serve cancel_request --request-id req_123" "true"
run_test "Serve get result" "aitbc edge serve get_result --request-id req_123" "true"
# Metrics operations
run_test "Metrics record" "aitbc edge metrics record --metric-name test_metric --value 100" "true"
run_test "Metrics list" "aitbc edge metrics list_metrics" "true"
run_test "Metrics get" "aitbc edge metrics get_metric --metric-id metric_123" "true"
run_test "Metrics delete" "aitbc edge metrics delete_metric --metric-id metric_123" "true"
# Error handling tests (should handle gracefully)
run_test "Island leave nonexistent" "aitbc edge island leave --island-id nonexistent_island" "false"
run_test "GPU get nonexistent" "aitbc edge gpu get_gpu --gpu-id nonexistent_gpu" "false"
# Output format tests
run_test "GPU list table format" "aitbc edge gpu list_gpus --format table" "true"
run_test "Database list table format" "aitbc edge database list_dbs --format table" "true"
# Summary
echo ""
echo "========================================"
echo "Test Summary"
echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo "Tests Passed: $TESTS_PASSED"
echo "Tests Failed: $TESTS_FAILED"
echo "Tests Skipped: $TESTS_SKIPPED"
echo "========================================"
if [ $TESTS_FAILED -eq 0 ]; then
log_info "All tests passed!"
exit 0
else
log_error "Some tests failed!"
exit 1
fi

149
scripts/testing/test_resource.sh Executable file
View File

@@ -0,0 +1,149 @@
#!/bin/bash
# Integration test script for resource CLI commands
# Tests resource status, deallocation, and coordinator-api integration
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
# Helper functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
run_test() {
local test_name="$1"
local test_command="$2"
TESTS_RUN=$((TESTS_RUN + 1))
log_info "Running: $test_name"
if eval "$test_command"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: $test_name"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: $test_name"
return 1
fi
}
# Setup
cd "$REPO_ROOT"
log_info "Starting resource CLI integration tests"
log_info "Note: Some tests require coordinator-api running"
# Test 1: Resource status (all resources)
run_test "Resource status (all)" "aitbc resource status"
# Test 2: Resource status (specific resource)
run_test "Resource status (specific)" "aitbc resource status --resource-id test_res_123"
# Test 3: Resource deallocation (with confirmation)
run_test "Resource deallocation (confirmed)" "echo 'y' | aitbc resource deallocate test_res_123"
# Test 4: Resource deallocation (force)
run_test "Resource deallocation (force)" "aitbc resource deallocate test_res_123 --force"
# Test 5: Resource deallocation (cancelled)
run_test "Resource deallocation (cancelled)" "echo 'n' | aitbc resource deallocate test_res_123"
# Test 6: Experimental commands require --mock flag
log_warn "Testing experimental commands (should fail without --mock)"
if aitbc resource allocate --resource-type gpu --quantity 4 2>&1 | grep -q "EXPERIMENTAL"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: Allocate shows experimental warning"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: Allocate should show experimental warning"
fi
TESTS_RUN=$((TESTS_RUN + 1))
if aitbc resource list 2>&1 | grep -q "EXPERIMENTAL"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: List shows experimental warning"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: List should show experimental warning"
fi
TESTS_RUN=$((TESTS_RUN + 1))
if aitbc resource release test_res 2>&1 | grep -q "EXPERIMENTAL"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: Release shows experimental warning"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: Release should show experimental warning"
fi
TESTS_RUN=$((TESTS_RUN + 1))
if aitbc resource utilization 2>&1 | grep -q "EXPERIMENTAL"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: Utilization shows experimental warning"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: Utilization should show experimental warning"
fi
TESTS_RUN=$((TESTS_RUN + 1))
if aitbc resource optimize 2>&1 | grep -q "EXPERIMENTAL"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: Optimize shows experimental warning"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: Optimize should show experimental warning"
fi
TESTS_RUN=$((TESTS_RUN + 1))
# Test 7: Mock mode for experimental commands
log_info "Testing experimental commands with --mock flag"
run_test "Allocate with --mock" "aitbc resource allocate --resource-type gpu --quantity 4 --mock"
run_test "List with --mock" "aitbc resource list --mock"
run_test "Release with --mock" "aitbc resource release test_res --mock"
run_test "Utilization with --mock" "aitbc resource utilization --mock"
run_test "Optimize with --mock" "aitbc resource optimize --mock"
# Summary
echo ""
echo "========================================"
echo "Test Summary"
echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo "Tests Passed: $TESTS_PASSED"
echo "Tests Failed: $TESTS_FAILED"
echo "========================================"
if [ $TESTS_FAILED -eq 0 ]; then
log_info "All tests passed!"
exit 0
else
log_error "Some tests failed!"
exit 1
fi

140
scripts/testing/test_simulate.sh Executable file
View File

@@ -0,0 +1,140 @@
#!/bin/bash
# Integration test script for simulate CLI commands
# Tests simulation operations including blockchain, wallets, price, network, and ai-jobs
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
COORDINATOR_URL="http://127.0.0.1:18000"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
TESTS_SKIPPED=0
# Helper functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
check_coordinator() {
if curl -s -f "$COORDINATOR_URL/health" > /dev/null 2>&1; then
return 0
else
return 1
fi
}
run_test() {
local test_name="$1"
local test_command="$2"
local require_coordinator="${3:-true}"
TESTS_RUN=$((TESTS_RUN + 1))
if [ "$require_coordinator" = "true" ] && ! check_coordinator; then
log_warn "SKIPPED: $test_name (coordinator-api not available)"
TESTS_SKIPPED=$((TESTS_SKIPPED + 1))
return 0
fi
log_info "Running: $test_name"
if eval "$test_command"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: $test_name"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: $test_name"
return 1
fi
}
# Setup
cd "$REPO_ROOT"
log_info "Starting simulate CLI integration tests"
log_info "Coordinator URL: $COORDINATOR_URL"
# Test 1: Blockchain simulation
run_test "Blockchain simulation" "aitbc simulate blockchain --blocks 10 --transactions 50" "true"
# Test 2: Wallets simulation
run_test "Wallets simulation" "aitbc simulate wallets --count 5 --balance 1000" "true"
# Test 3: Price simulation
run_test "Price simulation" "aitbc simulate price --days 30 --volatility 0.1" "true"
# Test 4: Network simulation
run_test "Network simulation" "aitbc simulate network --nodes 10 --latency 50" "true"
# Test 5: AI jobs simulation
run_test "AI jobs simulation" "aitbc simulate ai-jobs --jobs 20 --duration 300" "true"
# Test 6: Run simulation
run_test "Run simulation" "aitbc simulate run --type blockchain --duration 60" "true"
# Test 7: Blockchain with custom parameters
run_test "Blockchain with params" "aitbc simulate blockchain --blocks 100 --transactions 500 --difficulty 5" "true"
# Test 8: Wallets with distribution
run_test "Wallets with distribution" "aitbc simulate wallets --count 10 --distribution exponential" "true"
# Test 9: Price with trend
run_test "Price with trend" "aitbc simulate price --days 90 --trend bullish --volatility 0.15" "true"
# Test 10: Network with topology
run_test "Network with topology" "aitbc simulate network --nodes 20 --topology mesh --latency 100" "true"
# Test 11: AI jobs with GPU
run_test "AI jobs with GPU" "aitbc simulate ai-jobs --jobs 30 --gpu-required --duration 600" "true"
# Test 12: Run async simulation
run_test "Run async simulation" "aitbc simulate run --type network --async --duration 120" "true"
# Test 13: Status of non-existent simulation (should handle gracefully)
run_test "Status non-existent simulation" "aitbc simulate status sim_nonexistent_12345" "false"
# Test 14: Result of non-existent simulation (should handle gracefully)
run_test "Result non-existent simulation" "aitbc simulate result sim_nonexistent_12345" "false"
# Test 15: Output format JSON
run_test "Output format JSON" "aitbc simulate blockchain --blocks 5 --format json" "true"
# Test 16: Output format table
run_test "Output format table" "aitbc simulate blockchain --blocks 5 --format table" "true"
# Summary
echo ""
echo "========================================"
echo "Test Summary"
echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo "Tests Passed: $TESTS_PASSED"
echo "Tests Failed: $TESTS_FAILED"
echo "Tests Skipped: $TESTS_SKIPPED"
echo "========================================"
if [ $TESTS_FAILED -eq 0 ]; then
log_info "All tests passed!"
exit 0
else
log_error "Some tests failed!"
exit 1
fi

153
scripts/testing/test_workflow.sh
View File

@@ -1,23 +1,144 @@
#!/bin/bash #!/bin/bash
# Test Updated Workflow Scripts # Integration test script for workflow CLI commands
echo "=== Testing Updated Workflow Scripts ===" # Tests workflow run, list, status, and stop operations with coordinator-api
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
WORKFLOW_DIR="${REPO_ROOT}/scripts/workflow" COORDINATOR_URL="http://127.0.0.1:18000"
CLI_PATH="${REPO_ROOT}/aitbc-cli" TEST_WORKFLOW="test_workflow_$$"
echo "1. Testing wallet creation script..." # Colors for output
"${WORKFLOW_DIR}/04_create_wallet.sh" RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
TESTS_SKIPPED=0
# Helper functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
check_coordinator() {
log_info "Checking coordinator-api availability..."
if curl -s -f "$COORDINATOR_URL/health" > /dev/null 2>&1; then
log_info "coordinator-api is running"
return 0
else
log_warn "coordinator-api is not running at $COORDINATOR_URL"
return 1
fi
}
run_test() {
local test_name="$1"
local test_command="$2"
local require_coordinator="${3:-true}"
TESTS_RUN=$((TESTS_RUN + 1))
if [ "$require_coordinator" = "true" ] && ! check_coordinator; then
log_warn "SKIPPED: $test_name (coordinator-api not available)"
TESTS_SKIPPED=$((TESTS_SKIPPED + 1))
return 0
fi
log_info "Running: $test_name"
if eval "$test_command"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: $test_name"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: $test_name"
return 1
fi
}
cleanup() {
log_info "Cleaning up test workflows..."
# Attempt to stop test workflow if it exists
if [ -n "$TEST_WORKFLOW" ]; then
aitbc workflow stop "$TEST_WORKFLOW" 2>/dev/null || true
fi
}
# Setup
cd "$REPO_ROOT"
log_info "Starting workflow CLI integration tests"
log_info "Test workflow name: $TEST_WORKFLOW"
log_info "Coordinator URL: $COORDINATOR_URL"
# Test 1: List workflows
run_test "List workflows" "aitbc workflow list" "true"
# Test 2: Run a simple workflow
run_test "Run workflow" "aitbc workflow run $TEST_WORKFLOW" "true"
# Test 3: Get workflow status
run_test "Get workflow status" "aitbc workflow status $TEST_WORKFLOW" "true"
# Test 4: Run workflow with dry-run flag
run_test "Run workflow dry-run" "aitbc workflow run $TEST_WORKFLOW --dry-run" "false"
# Test 5: Run workflow with async flag
run_test "Run workflow async" "aitbc workflow run ${TEST_WORKFLOW}_async --async" "true"
# Test 6: Stop workflow
run_test "Stop workflow" "aitbc workflow stop $TEST_WORKFLOW" "true"
# Test 7: List workflows in table format
run_test "List workflows table format" "aitbc workflow list --format table" "true"
# Test 8: Get workflow status in JSON format
run_test "Get workflow status JSON" "aitbc workflow status $TEST_WORKFLOW --format json" "true"
# Test 9: Run workflow with parameters
run_test "Run workflow with parameters" "aitbc workflow run ${TEST_WORKFLOW}_params --param gpu_count=2 --param timeout=60" "true"
# Test 10: Status of non-existent workflow (should handle gracefully)
run_test "Status of non-existent workflow" "aitbc workflow status nonexistent_workflow_$$" "false"
# Test 11: Stop non-existent workflow (should handle gracefully)
run_test "Stop non-existent workflow" "aitbc workflow stop nonexistent_workflow_$$" "false"
# Test 12: Workflow with special characters in name
run_test "Workflow with special characters" "aitbc workflow run test-workflow-with-dashes" "false"
# Cleanup
cleanup
# Summary
echo "" echo ""
echo "2. Testing final verification script..." echo "========================================"
export WALLET_ADDR=$("$CLI_PATH" wallet balance aitbc-user 2>/dev/null | grep "Address:" | awk '{print $2}' || echo "") echo "Test Summary"
"${WORKFLOW_DIR}/06_final_verification.sh" echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo "Tests Passed: $TESTS_PASSED"
echo "Tests Failed: $TESTS_FAILED"
echo "Tests Skipped: $TESTS_SKIPPED"
echo "========================================"
echo "" if [ $TESTS_FAILED -eq 0 ]; then
echo "3. Testing transaction manager script..." log_info "All tests passed!"
"${WORKFLOW_DIR}/09_transaction_manager.sh" exit 0
else
echo "" log_error "Some tests failed!"
echo "✅ All script tests completed!" exit 1
fi

108
scripts/testing/test_workflow_cli.sh Executable file
View File

@@ -0,0 +1,108 @@
#!/bin/bash
# Integration test script for workflow CLI commands
# Tests workflow run, list, status, and stop operations
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Test counters
TESTS_RUN=0
TESTS_PASSED=0
TESTS_FAILED=0
# Helper functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
run_test() {
local test_name="$1"
local test_command="$2"
TESTS_RUN=$((TESTS_RUN + 1))
log_info "Running: $test_name"
if eval "$test_command"; then
TESTS_PASSED=$((TESTS_PASSED + 1))
log_info "PASSED: $test_name"
else
TESTS_FAILED=$((TESTS_FAILED + 1))
log_error "FAILED: $test_name"
return 1
fi
}
# Setup
cd "$REPO_ROOT"
log_info "Starting workflow CLI integration tests"
# Test 1: List workflows
run_test "List workflows" "aitbc workflow list"
# Test 2: List workflows in JSON format
run_test "List workflows (JSON)" "aitbc workflow list --format json | jq -e '.'"
# Test 3: Run workflow (dry run)
run_test "Run workflow (dry run)" "aitbc workflow run test_workflow --dry-run"
# Test 4: Run workflow basic
run_test "Run workflow basic" "aitbc workflow run test_workflow"
# Test 5: Get workflow status
run_test "Get workflow status" "aitbc workflow status test_workflow"
# Test 6: Stop workflow
run_test "Stop workflow" "aitbc workflow stop test_workflow"
# Test 7: Run workflow with special characters
run_test "Run workflow with dashes" "aitbc workflow run workflow-with-dashes --dry-run"
# Test 8: Run workflow with underscores
run_test "Run workflow with underscores" "aitbc workflow run workflow_with_underscores --dry-run"
# Test 9: Get status of non-existent workflow
run_test "Status of non-existent workflow" "aitbc workflow status nonexistent_workflow_xyz"
# Test 10: Stop non-existent workflow
run_test "Stop non-existent workflow" "aitbc workflow stop nonexistent_workflow_xyz"
# Test 11: Run workflow with config file (create temp config)
TEMP_CONFIG=$(mktemp)
echo "param1: value1" > "$TEMP_CONFIG"
run_test "Run workflow with config" "aitbc workflow run test_workflow --config $TEMP_CONFIG --dry-run"
rm -f "$TEMP_CONFIG"
# Summary
echo ""
echo "========================================"
echo "Test Summary"
echo "========================================"
echo "Tests Run: $TESTS_RUN"
echo "Tests Passed: $TESTS_PASSED"
echo "Tests Failed: $TESTS_FAILED"
echo "========================================"
if [ $TESTS_FAILED -eq 0 ]; then
log_info "All tests passed!"
exit 0
else
log_error "Some tests failed!"
exit 1
fi

311
tests/cli/test_config_profiles.py Normal file
View File

@@ -0,0 +1,311 @@
"""Integration tests for config profiles CLI commands
These tests require no running services but validate file system side effects
and actual profile CRUD operations.
"""
import pytest
import yaml
import os
import tempfile
from pathlib import Path
from click.testing import CliRunner
from unittest.mock import Mock, patch
from aitbc_cli.commands.config import config
@pytest.fixture
def runner():
"""Create CLI runner"""
return CliRunner()
@pytest.fixture
def mock_config():
"""Mock configuration"""
config = Mock()
config.coordinator_url = "http://127.0.0.1:18000"
config.api_key = None
config.timeout = 30
config.config_file = "/home/oib/.aitbc/config.yaml"
return config
@pytest.fixture
def profiles_dir(tmp_path):
"""Create and return profiles directory"""
profiles_dir = tmp_path / ".config" / "aitbc" / "profiles"
profiles_dir.mkdir(parents=True, exist_ok=True)
return profiles_dir
class TestConfigProfilesIntegration:
"""Integration tests for config profiles with file system validation"""
def test_profiles_save_creates_file(self, runner, mock_config, profiles_dir):
"""Test saving a profile creates the correct file"""
profile_name = "test_profile"
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'save', profile_name
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert f"Profile '{profile_name}' saved" in result.output
# Verify file was created
profile_file = profiles_dir / f"{profile_name}.yaml"
assert profile_file.exists()
# Verify file content
with open(profile_file) as f:
profile_data = yaml.safe_load(f)
assert profile_data['coordinator_url'] == 'http://127.0.0.1:18000'
assert profile_data['timeout'] == 30
assert 'api_key' not in profile_data # API key should not be saved
def test_profiles_save_overwrites_existing(self, runner, mock_config, profiles_dir):
"""Test saving a profile overwrites existing profile"""
profile_name = "overwrite_test"
# Create existing profile
profile_file = profiles_dir / f"{profile_name}.yaml"
profile_file.write_text(yaml.dump({
"coordinator_url": "http://old:8000",
"timeout": 10
}))
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'save', profile_name
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
# Verify file was overwritten
with open(profile_file) as f:
profile_data = yaml.safe_load(f)
assert profile_data['coordinator_url'] == 'http://127.0.0.1:18000'
assert profile_data['timeout'] == 30
def test_profiles_list_empty(self, runner, mock_config, profiles_dir):
"""Test listing profiles when none exist"""
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'list'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
import json
data = json.loads(result.output)
assert data['profiles'] == []
def test_profiles_list_multiple(self, runner, mock_config, profiles_dir):
"""Test listing multiple profiles"""
# Create test profiles
profile1 = profiles_dir / "profile1.yaml"
profile1.write_text(yaml.dump({
"coordinator_url": "http://test1:8000",
"timeout": 30
}))
profile2 = profiles_dir / "profile2.yaml"
profile2.write_text(yaml.dump({
"coordinator_url": "http://test2:8000",
"timeout": 60
}))
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'list'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert len(data['profiles']) == 2
assert data['profiles'][0]['name'] == 'profile1'
assert data['profiles'][1]['name'] == 'profile2'
def test_profiles_load_creates_config(self, runner, mock_config, profiles_dir, tmp_path):
"""Test loading a profile creates config file"""
profile_name = "load_test"
# Create profile
profile_file = profiles_dir / f"{profile_name}.yaml"
profile_file.write_text(yaml.dump({
"coordinator_url": "http://loaded:8000",
"timeout": 45
}))
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
with runner.isolated_filesystem(temp_dir=tmp_path):
result = runner.invoke(config, [
'profiles', 'load', profile_name
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert f"Profile '{profile_name}' loaded" in result.output
# Verify config file was created
config_file = Path.cwd() / ".aitbc.yaml"
assert config_file.exists()
with open(config_file) as f:
config_data = yaml.safe_load(f)
assert config_data['coordinator_url'] == 'http://loaded:8000'
assert config_data['timeout'] == 45
def test_profiles_load_nonexistent(self, runner, mock_config, profiles_dir):
"""Test loading a non-existent profile"""
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'load', 'nonexistent'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code != 0
assert "not found" in result.output
def test_profiles_delete_removes_file(self, runner, mock_config, profiles_dir):
"""Test deleting a profile removes the file"""
profile_name = "delete_test"
# Create profile
profile_file = profiles_dir / f"{profile_name}.yaml"
profile_file.write_text(yaml.dump({
"coordinator_url": "http://test:8000",
"timeout": 30
}))
assert profile_file.exists()
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'delete', profile_name
], obj={'config': mock_config, 'output_format': 'table'}, input='y\n')
assert result.exit_code == 0
assert f"Profile '{profile_name}' deleted" in result.output
assert not profile_file.exists()
def test_profiles_delete_cancelled(self, runner, mock_config, profiles_dir):
"""Test profile deletion cancelled by user"""
profile_name = "keep_test"
# Create profile
profile_file = profiles_dir / f"{profile_name}.yaml"
profile_file.write_text(yaml.dump({
"coordinator_url": "http://test:8000",
"timeout": 30
}))
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'delete', profile_name
], obj={'config': mock_config, 'output_format': 'json'}, input='n\n')
assert result.exit_code == 0
assert profile_file.exists() # Should still exist
def test_profiles_delete_nonexistent(self, runner, mock_config, profiles_dir):
"""Test deleting a non-existent profile"""
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'delete', 'nonexistent'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code != 0
assert "not found" in result.output
def test_profiles_roundtrip(self, runner, mock_config, profiles_dir, tmp_path):
"""Test save -> list -> load -> delete roundtrip"""
profile_name = "roundtrip_test"
# Save
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'save', profile_name
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
# List
result = runner.invoke(config, [
'profiles', 'list'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert profile_name in [p['name'] for p in data['profiles']]
# Load
with runner.isolated_filesystem(temp_dir=tmp_path):
result = runner.invoke(config, [
'profiles', 'load', profile_name
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
# Delete
result = runner.invoke(config, [
'profiles', 'delete', profile_name
], obj={'config': mock_config, 'output_format': 'table'}, input='y\n')
assert result.exit_code == 0
# Verify deleted
profile_file = profiles_dir / f"{profile_name}.yaml"
assert not profile_file.exists()
def test_profiles_with_different_configs(self, runner, mock_config, profiles_dir):
"""Test saving profiles with different config values"""
# Modify config for different profile
mock_config.coordinator_url = "http://different:9000"
mock_config.timeout = 90
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = profiles_dir.parent.parent.parent
result = runner.invoke(config, [
'profiles', 'save', 'different_profile'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
profile_file = profiles_dir / "different_profile.yaml"
with open(profile_file) as f:
profile_data = yaml.safe_load(f)
assert profile_data['coordinator_url'] == 'http://different:9000'
assert profile_data['timeout'] == 90
def test_profiles_directory_creation(self, runner, mock_config, tmp_path):
"""Test that profiles directory is created if it doesn't exist"""
profiles_dir = tmp_path / ".config" / "aitbc" / "profiles"
# Don't create it beforehand
with patch('pathlib.Path.home') as mock_home:
mock_home.return_value = tmp_path
result = runner.invoke(config, [
'profiles', 'save', 'new_profile'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert profiles_dir.exists()
assert (profiles_dir / "new_profile.yaml").exists()

359
tests/cli/test_edge_advanced.py Normal file
View File

@@ -0,0 +1,359 @@
"""Integration tests for edge advanced CLI commands
These tests require edge-api running and validate advanced edge operations
including island leave/bridge, GPU operations, database operations, serve operations, and metrics.
"""
import pytest
import json
import httpx
from click.testing import CliRunner
from unittest.mock import Mock, patch, MagicMock
from aitbc_cli.commands.edge import edge
from aitbc import AITBCHTTPClient, NetworkError
@pytest.fixture
def runner():
"""Create CLI runner"""
return CliRunner()
@pytest.fixture
def mock_config():
"""Mock configuration"""
config = Mock()
config.coordinator_url = "http://127.0.0.1:18000"
config.api_key = "test_api_key"
return config
@pytest.fixture
def mock_http_client():
"""Mock HTTP client for edge-api"""
client = MagicMock(spec=AITBCHTTPClient)
return client
class TestEdgeAdvancedCommands:
"""Integration tests for edge advanced commands with edge-api"""
@pytest.fixture
def edge_available(self):
"""Skip test if edge-api is not running"""
try:
response = httpx.get("http://127.0.0.1:8200/health", timeout=2)
if response.status_code == 200:
return True
except Exception:
pytest.skip("edge-api not running at http://127.0.0.1:8200")
# Island advanced operations
def test_edge_island_leave(self, runner, mock_config, edge_available):
"""Test leaving an island"""
result = runner.invoke(edge, [
'island', 'leave',
'--island-id', 'test_island_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'island_id' in data or 'status' in data
def test_edge_island_bridge(self, runner, mock_config, edge_available):
"""Test bridging between islands"""
result = runner.invoke(edge, [
'island', 'bridge',
'--source', 'island_a',
'--target', 'island_b'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'bridge_id' in data or 'status' in data
# GPU operations
def test_edge_gpu_list_gpus(self, runner, mock_config, edge_available):
"""Test listing GPUs"""
result = runner.invoke(edge, [
'gpu', 'list_gpus'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'gpus' in data or isinstance(data, list)
def test_edge_gpu_get_gpu(self, runner, mock_config, edge_available):
"""Test getting specific GPU info"""
result = runner.invoke(edge, [
'gpu', 'get_gpu',
'--gpu-id', 'gpu_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'gpu_id' in data or 'status' in data
def test_edge_gpu_remove_gpu(self, runner, mock_config, edge_available):
"""Test removing a GPU"""
result = runner.invoke(edge, [
'gpu', 'remove_gpu',
'--gpu-id', 'gpu_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'gpu_id' in data or 'status' in data
def test_edge_gpu_scan_gpus(self, runner, mock_config, edge_available):
"""Test scanning for available GPUs"""
result = runner.invoke(edge, [
'gpu', 'scan_gpus'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'gpus' in data or 'scan_results' in data
def test_edge_gpu_gpu_metrics(self, runner, mock_config, edge_available):
"""Test getting GPU metrics"""
result = runner.invoke(edge, [
'gpu', 'gpu_metrics',
'--gpu-id', 'gpu_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'metrics' in data or 'gpu_id' in data
# Database operations
def test_edge_database_init_db(self, runner, mock_config, edge_available):
"""Test initializing a database"""
result = runner.invoke(edge, [
'database', 'init_db',
'--db-name', 'test_db'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'db_id' in data or 'status' in data
def test_edge_database_list_dbs(self, runner, mock_config, edge_available):
"""Test listing databases"""
result = runner.invoke(edge, [
'database', 'list_dbs'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'databases' in data or isinstance(data, list)
def test_edge_database_get_db(self, runner, mock_config, edge_available):
"""Test getting database info"""
result = runner.invoke(edge, [
'database', 'get_db',
'--db-id', 'db_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'db_id' in data or 'status' in data
def test_edge_database_delete_db(self, runner, mock_config, edge_available):
"""Test deleting a database"""
result = runner.invoke(edge, [
'database', 'delete_db',
'--db-id', 'db_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'db_id' in data or 'status' in data
def test_edge_database_sync_db(self, runner, mock_config, edge_available):
"""Test syncing a database"""
result = runner.invoke(edge, [
'database', 'sync_db',
'--db-id', 'db_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'db_id' in data or 'sync_status' in data
# Serve operations
def test_edge_serve_submit_request(self, runner, mock_config, edge_available):
"""Test submitting a serve request"""
result = runner.invoke(edge, [
'serve', 'submit_request',
'--request-type', 'compute',
'--parameters', '{"gpu_count": 2}'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'request_id' in data or 'status' in data
def test_edge_serve_list_requests(self, runner, mock_config, edge_available):
"""Test listing serve requests"""
result = runner.invoke(edge, [
'serve', 'list_requests'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'requests' in data or isinstance(data, list)
def test_edge_serve_get_request(self, runner, mock_config, edge_available):
"""Test getting serve request info"""
result = runner.invoke(edge, [
'serve', 'get_request',
'--request-id', 'req_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'request_id' in data or 'status' in data
def test_edge_serve_cancel_request(self, runner, mock_config, edge_available):
"""Test cancelling a serve request"""
result = runner.invoke(edge, [
'serve', 'cancel_request',
'--request-id', 'req_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'request_id' in data or 'status' in data
def test_edge_serve_get_result(self, runner, mock_config, edge_available):
"""Test getting serve request result"""
result = runner.invoke(edge, [
'serve', 'get_result',
'--request-id', 'req_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'result' in data or 'request_id' in data
# Metrics operations
def test_edge_metrics_record(self, runner, mock_config, edge_available):
"""Test recording a metric"""
result = runner.invoke(edge, [
'metrics', 'record',
'--metric-name', 'test_metric',
'--value', '100'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'metric_id' in data or 'status' in data
def test_edge_metrics_list_metrics(self, runner, mock_config, edge_available):
"""Test listing metrics"""
result = runner.invoke(edge, [
'metrics', 'list_metrics'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'metrics' in data or isinstance(data, list)
def test_edge_metrics_get_metric(self, runner, mock_config, edge_available):
"""Test getting a specific metric"""
result = runner.invoke(edge, [
'metrics', 'get_metric',
'--metric-id', 'metric_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'metric_id' in data or 'value' in data
def test_edge_metrics_delete_metric(self, runner, mock_config, edge_available):
"""Test deleting a metric"""
result = runner.invoke(edge, [
'metrics', 'delete_metric',
'--metric-id', 'metric_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'metric_id' in data or 'status' in data
# Error handling tests
def test_edge_island_leave_nonexistent(self, runner, mock_config):
"""Test leaving non-existent island"""
result = runner.invoke(edge, [
'island', 'leave',
'--island-id', 'nonexistent_island'
], obj={'config': mock_config, 'output_format': 'json'})
# Should handle gracefully
assert result.exit_code != 0 or 'not found' in result.output.lower()
def test_edge_gpu_get_nonexistent(self, runner, mock_config):
"""Test getting non-existent GPU"""
result = runner.invoke(edge, [
'gpu', 'get_gpu',
'--gpu-id', 'nonexistent_gpu'
], obj={'config': mock_config, 'output_format': 'json'})
# Should handle gracefully
assert result.exit_code != 0 or 'not found' in result.output.lower()
def test_edge_api_error_handling(self, runner, mock_config):
"""Test edge command handles edge-api errors gracefully"""
# Use invalid edge URL to trigger error
mock_config.coordinator_url = "http://invalid:9999"
result = runner.invoke(edge, [
'gpu', 'list_gpus'
], obj={'config': mock_config, 'output_format': 'json'})
# Should either fail gracefully or skip with appropriate message
assert result.exit_code != 0 or 'error' in result.output.lower() or 'unavailable' in result.output.lower()
# Output format tests
def test_edge_gpu_list_table_format(self, runner, mock_config, edge_available):
"""Test GPU list in table format"""
result = runner.invoke(edge, [
'gpu', 'list_gpus'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'GPU' in result.output or 'gpus' in result.output.lower()
def test_edge_database_list_table_format(self, runner, mock_config, edge_available):
"""Test database list in table format"""
result = runner.invoke(edge, [
'database', 'list_dbs'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'Database' in result.output or 'databases' in result.output.lower()
@patch('aitbc_cli.commands.edge.get_config')
@patch('aitbc_cli.commands.edge.AITBCHTTPClient')
def test_edge_gpu_list_via_edge_api(self, mock_http_client_class, mock_get_config, runner):
"""Test GPU listing via edge-api"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.get.return_value = {
"gpus": [
{"id": "gpu_1", "type": "NVIDIA", "memory": 16},
{"id": "gpu_2", "type": "NVIDIA", "memory": 32}
]
}
result = runner.invoke(edge, [
'gpu', 'list_gpus'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0

450
tests/cli/test_resource.py Normal file
View File

@@ -0,0 +1,450 @@
"""Integration tests for resource CLI commands
These tests require coordinator-api running and validate resource allocation,
utilization tracking, and API interactions with actual service calls.
"""
import pytest
import json
import httpx
from click.testing import CliRunner
from unittest.mock import Mock, patch, MagicMock
from aitbc_cli.commands.resource import resource
from aitbc import AITBCHTTPClient, NetworkError
@pytest.fixture
def runner():
"""Create CLI runner"""
return CliRunner()
@pytest.fixture
def mock_config():
"""Mock configuration"""
config = Mock()
config.coordinator_url = "http://127.0.0.1:18000"
config.api_key = "test_api_key"
return config
@pytest.fixture
def mock_http_client():
"""Mock HTTP client for coordinator-api"""
client = MagicMock(spec=AITBCHTTPClient)
return client
class TestResourceCommands:
"""Integration tests for resource commands with coordinator-api"""
@pytest.fixture
def coordinator_available(self):
"""Skip test if coordinator-api is not running"""
try:
response = httpx.get("http://127.0.0.1:18000/health", timeout=2)
if response.status_code == 200:
return True
except Exception:
pytest.skip("coordinator-api not running at http://127.0.0.1:18000")
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_status_all(self, mock_http_client_class, mock_get_config, runner):
"""Test getting status of all resources"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.get.return_value = {
"resources": [
{"id": "res_1", "type": "gpu", "status": "allocated"},
{"id": "res_2", "type": "cpu", "status": "available"}
]
}
result = runner.invoke(resource, [
'status'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
mock_client.get.assert_called_once_with("/api/v1/resources/status")
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_status_specific(self, mock_http_client_class, mock_get_config, runner):
"""Test getting status of specific resource"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.get.return_value = {
"id": "res_123",
"type": "gpu",
"status": "allocated",
"efficiency": "85.5%"
}
result = runner.invoke(resource, [
'status',
'--resource-id', 'res_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
mock_client.get.assert_called_once_with("/api/v1/resources/res_123/status")
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_deallocate(self, mock_http_client_class, mock_get_config, runner):
"""Test deallocating a resource"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.post.return_value = {
"resource_id": "res_123",
"status": "deallocated",
"timestamp": "2026-05-27T08:30:00Z"
}
result = runner.invoke(resource, [
'deallocate', 'res_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
mock_client.post.assert_called_once_with("/api/v1/resources/res_123/deallocate")
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_deallocate_force(self, mock_http_client_class, mock_get_config, runner):
"""Test force deallocating a resource without confirmation"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.post.return_value = {
"resource_id": "res_123",
"status": "deallocated"
}
result = runner.invoke(resource, [
'deallocate', 'res_123',
'--force'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
mock_client.post.assert_called_once_with("/api/v1/resources/res_123/deallocate")
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_status_network_error(self, mock_http_client_class, mock_get_config, runner):
"""Test resource status with network error"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.get.side_effect = NetworkError("Connection refused")
result = runner.invoke(resource, [
'status'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code != 0
assert "Network error" in result.output
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_deallocate_network_error(self, mock_http_client_class, mock_get_config, runner):
"""Test resource deallocation with network error"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.post.side_effect = NetworkError("Connection refused")
result = runner.invoke(resource, [
'deallocate', 'res_123'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code != 0
assert "Network error" in result.output
def test_resource_allocate_experimental_warning(self, runner, mock_config):
"""Test that allocate command shows experimental warning without --mock"""
result = runner.invoke(resource, [
'allocate',
'--resource-type', 'gpu',
'--quantity', '4'
], obj={'config': mock_config, 'output_format': 'table'})
# Should fail with experimental warning
assert result.exit_code != 0
assert "EXPERIMENTAL" in result.output
assert "--mock" in result.output
def test_resource_list_experimental_warning(self, runner, mock_config):
"""Test that list command shows experimental warning without --mock"""
result = runner.invoke(resource, [
'list'
], obj={'config': mock_config, 'output_format': 'table'})
# Should fail with experimental warning
assert result.exit_code != 0
assert "EXPERIMENTAL" in result.output
assert "--mock" in result.output
def test_resource_release_experimental_warning(self, runner, mock_config):
"""Test that release command shows experimental warning without --mock"""
result = runner.invoke(resource, [
'release', 'res_123'
], obj={'config': mock_config, 'output_format': 'table'})
# Should fail with experimental warning
assert result.exit_code != 0
assert "EXPERIMENTAL" in result.output
assert "--mock" in result.output
def test_resource_utilization_experimental_warning(self, runner, mock_config):
"""Test that utilization command shows experimental warning without --mock"""
result = runner.invoke(resource, [
'utilization'
], obj={'config': mock_config, 'output_format': 'table'})
# Should fail with experimental warning
assert result.exit_code != 0
assert "EXPERIMENTAL" in result.output
assert "--mock" in result.output
def test_resource_optimize_experimental_warning(self, runner, mock_config):
"""Test that optimize command shows experimental warning without --mock"""
result = runner.invoke(resource, [
'optimize'
], obj={'config': mock_config, 'output_format': 'table'})
# Should fail with experimental warning
assert result.exit_code != 0
assert "EXPERIMENTAL" in result.output
assert "--mock" in result.output
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_status_table_format(self, mock_http_client_class, mock_get_config, runner):
"""Test resource status in table format"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.get.return_value = {
"resources": [
{"id": "res_1", "type": "gpu", "status": "allocated"}
]
}
result = runner.invoke(resource, [
'status'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert "Resource Status" in result.output
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_deallocate_with_confirmation(self, mock_http_client_class, mock_get_config, runner):
"""Test resource deallocation with user confirmation"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.post.return_value = {
"resource_id": "res_123",
"status": "deallocated"
}
result = runner.invoke(resource, [
'deallocate', 'res_123'
], obj={'config': mock_config, 'output_format': 'json'}, input='y\n')
assert result.exit_code == 0
mock_client.post.assert_called_once_with("/api/v1/resources/res_123/deallocate")
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_deallocate_cancelled(self, mock_http_client_class, mock_get_config, runner):
"""Test resource deallocation cancelled by user"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
result = runner.invoke(resource, [
'deallocate', 'res_123'
], obj={'config': mock_config, 'output_format': 'json'}, input='n\n')
assert result.exit_code == 0
# Should not call post if cancelled
mock_client.post.assert_not_called()
@patch('aitbc_cli.commands.resource.get_config')
@patch('aitbc_cli.commands.resource.AITBCHTTPClient')
def test_resource_status_empty_response(self, mock_http_client_class, mock_get_config, runner):
"""Test resource status with empty response"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.get.return_value = {}
result = runner.invoke(resource, [
'status'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
mock_client.get.assert_called_once_with("/api/v1/resources/status")
def test_resource_status_with_coordinator_api(self, runner, mock_config, coordinator_available):
"""Test resource status with actual coordinator-api call"""
result = runner.invoke(resource, [
'status'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'resources' in data or isinstance(data, list)
def test_resource_deallocate_with_coordinator_api(self, runner, mock_config, coordinator_available):
"""Test resource deallocation with actual coordinator-api call"""
result = runner.invoke(resource, [
'deallocate', 'test_res_123',
'--force'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'resource_id' in data or 'status' in data
def test_resource_allocate_with_mock(self, runner, mock_config):
"""Test resource allocation with mock flag"""
result = runner.invoke(resource, [
'allocate',
'--resource-type', 'gpu',
'--quantity', '4',
'--mock'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'resource_id' in data or 'allocation_id' in data
def test_resource_list_with_mock(self, runner, mock_config):
"""Test resource listing with mock flag"""
result = runner.invoke(resource, [
'list',
'--mock'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'resources' in data or isinstance(data, list)
def test_resource_release_with_mock(self, runner, mock_config):
"""Test resource release with mock flag"""
result = runner.invoke(resource, [
'release', 'test_res_123',
'--mock'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'resource_id' in data or 'status' in data
def test_resource_utilization_with_mock(self, runner, mock_config):
"""Test resource utilization with mock flag"""
result = runner.invoke(resource, [
'utilization',
'--mock'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'utilization' in data or 'metrics' in data
def test_resource_optimize_with_mock(self, runner, mock_config):
"""Test resource optimization with mock flag"""
result = runner.invoke(resource, [
'optimize',
'--mock'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'optimization' in data or 'recommendations' in data
def test_resource_allocate_with_parameters(self, runner, mock_config):
"""Test resource allocation with custom parameters"""
result = runner.invoke(resource, [
'allocate',
'--resource-type', 'gpu',
'--quantity', '8',
'--min-memory', '32',
'--mock'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'resource_id' in data or 'allocation_id' in data
def test_resource_status_filter_by_type(self, runner, mock_config, coordinator_available):
"""Test resource status filtered by resource type"""
result = runner.invoke(resource, [
'status',
'--resource-type', 'gpu'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
# Verify filtering was applied
if 'resources' in data and isinstance(data['resources'], list):
for res in data['resources']:
assert res.get('type') == 'gpu' or 'type' not in res
def test_resource_api_error_handling(self, runner, mock_config):
"""Test resource command handles coordinator-api errors gracefully"""
# Use invalid coordinator URL to trigger error
mock_config.coordinator_url = "http://invalid:9999"
result = runner.invoke(resource, [
'status'
], obj={'config': mock_config, 'output_format': 'json'})
# Should either fail gracefully or skip with appropriate message
assert result.exit_code != 0 or 'error' in result.output.lower() or 'unavailable' in result.output.lower()

337
tests/cli/test_simulate_integration.py Normal file
View File

@@ -0,0 +1,337 @@
"""Integration tests for simulate CLI commands
These tests require coordinator-api running and validate simulation operations
including blockchain, wallets, price, network, and ai-jobs simulations with actual service calls.
"""
import pytest
import json
import httpx
from click.testing import CliRunner
from unittest.mock import Mock, patch, MagicMock
from aitbc_cli.commands.simulate import simulate
from aitbc import AITBCHTTPClient, NetworkError
@pytest.fixture
def runner():
"""Create CLI runner"""
return CliRunner()
@pytest.fixture
def mock_config():
"""Mock configuration"""
config = Mock()
config.coordinator_url = "http://127.0.0.1:18000"
config.api_key = "test_api_key"
return config
@pytest.fixture
def mock_http_client():
"""Mock HTTP client for coordinator-api"""
client = MagicMock(spec=AITBCHTTPClient)
return client
class TestSimulateCommandsIntegration:
"""Integration tests for simulate commands with coordinator-api"""
@pytest.fixture
def coordinator_available(self):
"""Skip test if coordinator-api is not running"""
try:
response = httpx.get("http://127.0.0.1:18000/health", timeout=2)
if response.status_code == 200:
return True
except Exception:
pytest.skip("coordinator-api not running at http://127.0.0.1:18000")
def test_simulate_blockchain(self, runner, mock_config, coordinator_available):
"""Test blockchain simulation"""
result = runner.invoke(simulate, [
'blockchain',
'--blocks', '10',
'--transactions', '50'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'blocks' in data or 'simulation_id' in data
def test_simulate_wallets(self, runner, mock_config, coordinator_available):
"""Test wallet simulation"""
result = runner.invoke(simulate, [
'wallets',
'--count', '5',
'--balance', '1000'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'wallets' in data or 'simulation_id' in data
def test_simulate_price(self, runner, mock_config, coordinator_available):
"""Test price simulation"""
result = runner.invoke(simulate, [
'price',
'--days', '30',
'--volatility', '0.1'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'prices' in data or 'simulation_id' in data
def test_simulate_network(self, runner, mock_config, coordinator_available):
"""Test network simulation"""
result = runner.invoke(simulate, [
'network',
'--nodes', '10',
'--latency', '50'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'network' in data or 'simulation_id' in data
def test_simulate_ai_jobs(self, runner, mock_config, coordinator_available):
"""Test AI jobs simulation"""
result = runner.invoke(simulate, [
'ai-jobs',
'--jobs', '20',
'--duration', '300'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'jobs' in data or 'simulation_id' in data
def test_simulate_run(self, runner, mock_config, coordinator_available):
"""Test running a simulation"""
result = runner.invoke(simulate, [
'run',
'--type', 'blockchain',
'--duration', '60'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'simulation_id' in data or 'status' in data
def test_simulate_status(self, runner, mock_config, coordinator_available):
"""Test getting simulation status"""
# First run a simulation
run_result = runner.invoke(simulate, [
'run',
'--type', 'blockchain'
], obj={'config': mock_config, 'output_format': 'json'})
assert run_result.exit_code == 0
run_data = json.loads(run_result.output)
sim_id = run_data.get('simulation_id')
if sim_id:
# Get status
status_result = runner.invoke(simulate, [
'status', sim_id
], obj={'config': mock_config, 'output_format': 'json'})
assert status_result.exit_code == 0
status_data = json.loads(status_result.output)
assert 'status' in status_data
def test_simulate_result(self, runner, mock_config, coordinator_available):
"""Test getting simulation results"""
# First run a simulation
run_result = runner.invoke(simulate, [
'run',
'--type', 'wallets'
], obj={'config': mock_config, 'output_format': 'json'})
assert run_result.exit_code == 0
run_data = json.loads(run_result.output)
sim_id = run_data.get('simulation_id')
if sim_id:
# Get results
result_result = runner.invoke(simulate, [
'result', sim_id
], obj={'config': mock_config, 'output_format': 'json'})
assert result_result.exit_code == 0
result_data = json.loads(result_result.output)
assert 'results' in result_data or 'data' in result_data
def test_simulate_blockchain_with_params(self, runner, mock_config, coordinator_available):
"""Test blockchain simulation with custom parameters"""
result = runner.invoke(simulate, [
'blockchain',
'--blocks', '100',
'--transactions', '500',
'--difficulty', '5'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'blocks' in data or 'simulation_id' in data
def test_simulate_wallets_with_distribution(self, runner, mock_config, coordinator_available):
"""Test wallet simulation with balance distribution"""
result = runner.invoke(simulate, [
'wallets',
'--count', '10',
'--distribution', 'exponential'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'wallets' in data or 'simulation_id' in data
def test_simulate_price_with_trend(self, runner, mock_config, coordinator_available):
"""Test price simulation with trend"""
result = runner.invoke(simulate, [
'price',
'--days', '90',
'--trend', 'bullish',
'--volatility', '0.15'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'prices' in data or 'simulation_id' in data
def test_simulate_network_with_topology(self, runner, mock_config, coordinator_available):
"""Test network simulation with custom topology"""
result = runner.invoke(simulate, [
'network',
'--nodes', '20',
'--topology', 'mesh',
'--latency', '100'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'network' in data or 'simulation_id' in data
def test_simulate_ai_jobs_with_gpu(self, runner, mock_config, coordinator_available):
"""Test AI jobs simulation with GPU requirements"""
result = runner.invoke(simulate, [
'ai-jobs',
'--jobs', '30',
'--gpu-required',
'--duration', '600'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'jobs' in data or 'simulation_id' in data
def test_simulate_run_async(self, runner, mock_config, coordinator_available):
"""Test running simulation in async mode"""
result = runner.invoke(simulate, [
'run',
'--type', 'network',
'--async',
'--duration', '120'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'simulation_id' in data
assert data.get('status') in ['started', 'running', 'pending']
def test_simulate_status_nonexistent(self, runner, mock_config):
"""Test getting status of non-existent simulation"""
result = runner.invoke(simulate, [
'status', 'sim_nonexistent_12345'
], obj={'config': mock_config, 'output_format': 'json'})
# Should handle gracefully
assert result.exit_code != 0 or 'not found' in result.output.lower()
def test_simulate_result_nonexistent(self, runner, mock_config):
"""Test getting results of non-existent simulation"""
result = runner.invoke(simulate, [
'result', 'sim_nonexistent_12345'
], obj={'config': mock_config, 'output_format': 'json'})
# Should handle gracefully
assert result.exit_code != 0 or 'not found' in result.output.lower()
def test_simulate_multiple_concurrent(self, runner, mock_config, coordinator_available):
"""Test running multiple concurrent simulations"""
sim_ids = []
for i in range(3):
result = runner.invoke(simulate, [
'run',
'--type', 'blockchain',
'--async'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
sim_id = data.get('simulation_id')
if sim_id:
sim_ids.append(sim_id)
# Verify we got multiple simulation IDs
assert len(sim_ids) > 0
def test_simulate_api_error_handling(self, runner, mock_config):
"""Test simulate command handles coordinator-api errors gracefully"""
# Use invalid coordinator URL to trigger error
mock_config.coordinator_url = "http://invalid:9999"
result = runner.invoke(simulate, [
'blockchain'
], obj={'config': mock_config, 'output_format': 'json'})
# Should either fail gracefully or skip with appropriate message
assert result.exit_code != 0 or 'error' in result.output.lower() or 'unavailable' in result.output.lower()
@patch('aitbc_cli.commands.simulate.get_config')
@patch('aitbc_cli.commands.simulate.AITBCHTTPClient')
def test_simulate_blockchain_via_coordinator_api(self, mock_http_client_class, mock_get_config, runner):
"""Test blockchain simulation via coordinator-api"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.post.return_value = {
"simulation_id": "sim_123",
"status": "started",
"blocks": 10
}
result = runner.invoke(simulate, [
'blockchain',
'--blocks', '10'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
# Verify API was called (if simulate command uses coordinator-api)
def test_simulate_output_formats(self, runner, mock_config, coordinator_available):
"""Test simulation output in different formats"""
# JSON format
result_json = runner.invoke(simulate, [
'blockchain',
'--blocks', '5'
], obj={'config': mock_config, 'output_format': 'json'})
assert result_json.exit_code == 0
json.loads(result_json.output) # Should be valid JSON
# Table format
result_table = runner.invoke(simulate, [
'blockchain',
'--blocks', '5'
], obj={'config': mock_config, 'output_format': 'table'})
assert result_table.exit_code == 0

380
tests/cli/test_workflow.py Normal file
View File

@@ -0,0 +1,380 @@
"""Integration tests for workflow CLI commands
These tests require coordinator-api running and validate workflow execution,
status tracking, and API interactions with actual service calls.
"""
import pytest
import json
import time
import httpx
from pathlib import Path
from click.testing import CliRunner
from unittest.mock import Mock, patch, MagicMock
from aitbc_cli.commands.workflow import workflow
from aitbc import AITBCHTTPClient, NetworkError
@pytest.fixture
def runner():
"""Create CLI runner"""
return CliRunner()
@pytest.fixture
def mock_config():
"""Mock configuration"""
config = Mock()
config.coordinator_url = "http://127.0.0.1:18000"
config.api_key = "test_api_key"
return config
@pytest.fixture
def mock_http_client():
"""Mock HTTP client for coordinator-api"""
client = MagicMock(spec=AITBCHTTPClient)
return client
class TestWorkflowCommands:
"""Integration tests for workflow commands with coordinator-api"""
@pytest.fixture
def coordinator_available(self):
"""Skip test if coordinator-api is not running"""
try:
response = httpx.get("http://127.0.0.1:18000/health", timeout=2)
if response.status_code == 200:
return True
except Exception:
pytest.skip("coordinator-api not running at http://127.0.0.1:18000")
def test_workflow_run_basic(self, runner, mock_config):
"""Test running a basic workflow"""
result = runner.invoke(workflow, [
'run', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'test_workflow' in result.output
assert 'Running' in result.output
def test_workflow_run_with_config(self, runner, mock_config, tmp_path):
"""Test running workflow with config file"""
config_file = tmp_path / "workflow_config.yaml"
config_file.write_text("param1: value1\nparam2: value2")
result = runner.invoke(workflow, [
'run', 'test_workflow',
'--config', str(config_file)
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'test_workflow' in result.output
assert str(config_file) in result.output
def test_workflow_run_dry_run(self, runner, mock_config):
"""Test workflow dry run mode"""
result = runner.invoke(workflow, [
'run', 'test_workflow',
'--dry-run'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'Dry run' in result.output
assert 'without making changes' in result.output
def test_workflow_list(self, runner, mock_config):
"""Test listing available workflows"""
result = runner.invoke(workflow, [
'list'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'workflows' in data or isinstance(data, list)
# If it's a list, check structure
if isinstance(data, list):
assert len(data) > 0
assert 'name' in data[0]
assert 'status' in data[0]
def test_workflow_list_table_format(self, runner, mock_config):
"""Test listing workflows in table format"""
result = runner.invoke(workflow, [
'list',
'--format', 'table'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'Available workflows' in result.output
def test_workflow_status(self, runner, mock_config):
"""Test getting workflow status"""
result = runner.invoke(workflow, [
'status', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'test_workflow' in result.output
assert 'Status' in result.output
def test_workflow_stop(self, runner, mock_config):
"""Test stopping a workflow"""
result = runner.invoke(workflow, [
'stop', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert 'test_workflow' in result.output
assert 'Stop' in result.output
@patch('aitbc_cli.commands.workflow.get_config')
@patch('aitbc_cli.commands.workflow.AITBCHTTPClient')
def test_workflow_run_via_coordinator_api(self, mock_http_client_class, mock_get_config, runner):
"""Test workflow execution via coordinator-api"""
# Setup mocks
mock_config = Mock()
mock_config.coordinator_url = "http://127.0.0.1:18000"
mock_get_config.return_value = mock_config
mock_client = MagicMock()
mock_http_client_class.return_value = mock_client
mock_client.post.return_value = {
"workflow_id": "wf_123",
"status": "started",
"execution_id": "exec_456"
}
result = runner.invoke(workflow, [
'run', 'api_workflow'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
# Verify API was called (if workflow command uses coordinator-api)
# This depends on actual implementation
def test_workflow_execution_id_generation(self, runner, mock_config):
"""Test that workflow execution generates unique IDs"""
result1 = runner.invoke(workflow, [
'run', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'table'})
time.sleep(0.1) # Small delay to ensure different timestamp
result2 = runner.invoke(workflow, [
'run', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'table'})
assert result1.exit_code == 0
assert result2.exit_code == 0
# Extract execution IDs from output
import re
id_pattern = r'wf_exec_\d+'
ids1 = re.findall(id_pattern, result1.output)
ids2 = re.findall(id_pattern, result2.output)
if ids1 and ids2:
assert ids1[0] != ids2[0], "Execution IDs should be unique"
def test_workflow_nonexistent_status(self, runner, mock_config):
"""Test getting status of non-existent workflow"""
result = runner.invoke(workflow, [
'status', 'nonexistent_workflow_xyz'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
# Should return status even for non-existent workflows
assert 'nonexistent_workflow_xyz' in result.output
def test_workflow_stop_nonexistent(self, runner, mock_config):
"""Test stopping non-existent workflow"""
result = runner.invoke(workflow, [
'stop', 'nonexistent_workflow_xyz'
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
# Should attempt to stop even if not running
assert 'nonexistent_workflow_xyz' in result.output
def test_workflow_with_special_characters(self, runner, mock_config):
"""Test workflow names with special characters"""
special_names = [
'workflow-with-dashes',
'workflow_with_underscores',
'workflow.with.dots',
'WorkflowWithCamelCase'
]
for name in special_names:
result = runner.invoke(workflow, [
'run', name
], obj={'config': mock_config, 'output_format': 'table'})
assert result.exit_code == 0
assert name in result.output
def test_workflow_list_filters(self, runner, mock_config):
"""Test workflow listing with potential filters"""
result = runner.invoke(workflow, [
'list'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
# Verify expected workflow types are present
if isinstance(data, list):
workflow_names = [w['name'] for w in data]
# Check for known workflow types from implementation
expected_types = ['gpu-marketplace', 'ai-job-processing', 'mining-optimization']
for expected in expected_types:
if expected in workflow_names:
assert True # Found expected workflow
break
def test_workflow_status_output_format(self, runner, mock_config):
"""Test workflow status in different output formats"""
# Table format
result_table = runner.invoke(workflow, [
'status', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'table'})
assert result_table.exit_code == 0
# JSON format
result_json = runner.invoke(workflow, [
'status', 'test_workflow'
], obj={'config': mock_config, 'output_format': 'json'})
assert result_json.exit_code == 0
# Should be parseable as JSON or contain status info
def test_workflow_run_with_coordinator_api(self, runner, mock_config, coordinator_available):
"""Test workflow execution with actual coordinator-api call"""
result = runner.invoke(workflow, [
'run', 'test_integration_workflow',
'--async'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'workflow_id' in data or 'execution_id' in data
assert data.get('status') in ['started', 'running', 'pending']
def test_workflow_list_with_coordinator_api(self, runner, mock_config, coordinator_available):
"""Test listing workflows from coordinator-api"""
result = runner.invoke(workflow, [
'list'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'workflows' in data or isinstance(data, list)
# Validate workflow structure
if isinstance(data, list):
for workflow in data:
assert 'name' in workflow
assert 'status' in workflow
def test_workflow_status_with_coordinator_api(self, runner, mock_config, coordinator_available):
"""Test getting workflow status from coordinator-api"""
# First run a workflow
run_result = runner.invoke(workflow, [
'run', 'status_test_workflow'
], obj={'config': mock_config, 'output_format': 'json'})
assert run_result.exit_code == 0
run_data = json.loads(run_result.output)
workflow_id = run_data.get('workflow_id') or run_data.get('execution_id')
if workflow_id:
# Get status
status_result = runner.invoke(workflow, [
'status', workflow_id
], obj={'config': mock_config, 'output_format': 'json'})
assert status_result.exit_code == 0
status_data = json.loads(status_result.output)
assert 'status' in status_data
assert workflow_id in str(status_data)
def test_workflow_stop_with_coordinator_api(self, runner, mock_config, coordinator_available):
"""Test stopping workflow via coordinator-api"""
# Run a workflow
run_result = runner.invoke(workflow, [
'run', 'stop_test_workflow'
], obj={'config': mock_config, 'output_format': 'json'})
assert run_result.exit_code == 0
run_data = json.loads(run_result.output)
workflow_id = run_data.get('workflow_id') or run_data.get('execution_id')
if workflow_id:
# Stop the workflow
stop_result = runner.invoke(workflow, [
'stop', workflow_id
], obj={'config': mock_config, 'output_format': 'json'})
assert stop_result.exit_code == 0
stop_data = json.loads(stop_result.output)
assert stop_data.get('status') in ['stopped', 'stopping', 'cancelled']
def test_workflow_run_with_parameters(self, runner, mock_config, coordinator_available):
"""Test workflow execution with custom parameters"""
result = runner.invoke(workflow, [
'run', 'param_test_workflow',
'--param', 'gpu_count=4',
'--param', 'timeout=300'
], obj={'config': mock_config, 'output_format': 'json'})
assert result.exit_code == 0
data = json.loads(result.output)
assert 'workflow_id' in data or 'execution_id' in data
def test_workflow_execution_tracking(self, runner, mock_config, coordinator_available):
"""Test tracking workflow execution over time"""
# Start workflow
run_result = runner.invoke(workflow, [
'run', 'tracking_test_workflow'
], obj={'config': mock_config, 'output_format': 'json'})
assert run_result.exit_code == 0
run_data = json.loads(run_result.output)
workflow_id = run_data.get('workflow_id') or run_data.get('execution_id')
if workflow_id:
# Check status immediately
status1 = runner.invoke(workflow, [
'status', workflow_id
], obj={'config': mock_config, 'output_format': 'json'})
assert status1.exit_code == 0
# Wait and check status again
time.sleep(1)
status2 = runner.invoke(workflow, [
'status', workflow_id
], obj={'config': mock_config, 'output_format': 'json'})
assert status2.exit_code == 0
status2_data = json.loads(status2.output)
assert 'status' in status2_data
def test_workflow_api_error_handling(self, runner, mock_config):
"""Test workflow command handles coordinator-api errors gracefully"""
# Use invalid coordinator URL to trigger error
mock_config.coordinator_url = "http://invalid:9999"
result = runner.invoke(workflow, [
'run', 'error_test_workflow'
], obj={'config': mock_config, 'output_format': 'json'})
# Should either fail gracefully or skip with appropriate message
# The exact behavior depends on implementation
assert result.exit_code != 0 or 'error' in result.output.lower() or 'unavailable' in result.output.lower()