oib/aitbc
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

docs: create automated validation guide for AITBC scenarios
Some checks failed
Cross-Node Transaction Testing / transaction-test (push) Successful in 2s
Details
Deploy to Testnet / deploy-testnet (push) Successful in 1m32s
Details
Documentation Validation / validate-docs (push) Failing after 11s
Details
Documentation Validation / validate-policies-strict (push) Successful in 4s
Details
Multi-Node Stress Testing / stress-test (push) Successful in 3s
Details
Node Failover Simulation / failover-test (push) Successful in 3s
Details

Browse Source 
- Add validation framework and command structure
- Include validation criteria for key scenarios (01, 02, 03, 07, 08, 13, 14, 20)
- Provide automated validation scripts for batch and stage validation
- Add CI/CD integration examples for GitHub Actions and Gitea Actions
- Include validation best practices and metrics tracking
- Version 1.0 with last updated date
This commit is contained in:
aitbc
2026-05-09 11:48:34 +02:00
parent 138822b1bf
commit 5fc531fa4b
2 changed files with 973 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

466
docs/scenarios/TROUBLESHOOTING_GUIDE.md Normal file
View File

@@ -0,0 +1,466 @@
# AITBC Scenario Troubleshooting Guide
**Last Updated:** 2026-05-09
**Version:** 1.0
**Purpose:** Common failure points and solutions for AITBC scenarios
---
## Overview
This guide provides troubleshooting information for common issues encountered when working through AITBC scenarios. Each scenario may have specific failure points, but many issues follow common patterns across the system.
## Common Failure Patterns
### Wallet-Related Issues
#### Issue: Wallet Creation Fails
**Symptoms:**
- Error: "Wallet already exists"
- Error: "Invalid password"
- Error: "Keystore directory not accessible"
**Solutions:**
```bash
# Check if wallet already exists
ls -la /var/lib/aitbc/keystore/
# Remove existing wallet if needed (CAUTION: loses funds)
rm /var/lib/aitbc/keystore/<wallet-name>.json
# Verify keystore directory permissions
chmod 700 /var/lib/aitbc/keystore/
chown aitbc:aitbc /var/lib/aitbc/keystore/
```
**Prevention:**
- Use deterministic wallet naming (e.g., test-wallet-001, test-wallet-002)
- Always check for existing wallets before creation
- Use strong passwords and store them securely
#### Issue: Insufficient Funds
**Symptoms:**
- Error: "Insufficient funds for transaction"
- Transaction rejected by blockchain
**Solutions:**
```bash
# Check wallet balance
aitbc wallet balance <wallet-name>
# Fund wallet from genesis wallet if available
aitbc transaction send genesis <wallet-name> 1000
# Or use faucet if available
curl -X POST http://faucet.aitbc.local/fund \
-H "Content-Type: application/json" \
-d '{"address": "<wallet-address>", "amount": 1000}'
```
**Prevention:**
- Always check balance before transactions
- Use testnet faucets for development
- Maintain a funding wallet for test operations
### Blockchain Node Issues
#### Issue: Node Not Running
**Symptoms:**
- Error: "Connection refused"
- Error: "Node not responding"
- CLI commands timeout
**Solutions:**
```bash
# Check node status
systemctl status aitbc-blockchain-node.service
# Start node if stopped
systemctl start aitbc-blockchain-node.service
# Check node logs
journalctl -u aitbc-blockchain-node.service -f
# Verify node connectivity
aitbc blockchain status
```
**Prevention:**
- Monitor node health regularly
- Set up automated restart on failure
- Use systemd auto-restart configuration
#### Issue: Node Out of Sync
**Symptoms:**
- Transactions not confirming
- Balance showing incorrect values
- Block height mismatch
**Solutions:**
```bash
# Check sync status
aitbc blockchain sync status
# Force sync if needed
aitbc blockchain sync force
# Enable auto-sync in /etc/aitbc/.env
echo "auto_sync_enabled=true" >> /etc/aitbc/.env
systemctl restart aitbc-blockchain-node.service
```
**Prevention:**
- Enable auto-sync in configuration
- Monitor sync status regularly
- Use stable network connections
### Marketplace Issues
#### Issue: Bid Not Accepted
**Symptoms:**
- Bid rejected by marketplace
- Error: "Insufficient balance for bid"
- Error: "Invalid bid amount"
**Solutions:**
```bash
# Check marketplace status
aitbc marketplace status
# Verify wallet balance
aitbc wallet balance <wallet-name>
# Check current market rates
aitbc marketplace rates
# Retry with valid bid amount
aitbc marketplace bid <listing-id> <amount> <wallet-name>
```
**Prevention:**
- Check market rates before bidding
- Ensure sufficient balance including fees
- Use bid validation before submission
#### Issue: GPU Listing Not Appearing
**Symptoms:**
- GPU not visible in marketplace
- Listing creation fails
- Error: "Invalid GPU configuration"
**Solutions:**
```bash
# Verify GPU availability
nvidia-smi
# Check GPU service status
systemctl status aitbc-gpu-service.service
# Re-list GPU with correct configuration
aitbc marketplace gpu list --gpu-id <gpu-id> --price <price>
```
**Prevention:**
- Verify GPU hardware before listing
- Test GPU service independently
- Use standard pricing models
### AI Job Issues
#### Issue: Job Submission Fails
**Symptoms:**
- Error: "Invalid job parameters"
- Error: "Insufficient compute resources"
- Job stuck in "pending" state
**Solutions:**
```bash
# Check AI service status
systemctl status aitbc-ai.service
# Verify job parameters
aitbc ai job validate --file <job-config.json>
# Check available compute resources
aitbc marketplace resources available
# Retry job submission
aitbc ai job submit <job-config.json>
```
**Prevention:**
- Validate job configuration before submission
- Check resource availability first
- Use job templates for common tasks
#### Issue: Job Execution Timeout
**Symptoms:**
- Job runs longer than expected
- Job marked as "failed" due to timeout
- No results returned
**Solutions:**
```bash
# Check job status
aitbc ai job status <job-id>
# Extend timeout if needed
aitbc ai job update <job-id> --timeout <new-timeout>
# Check compute provider logs
journalctl -u aitbc-gpu-service.service -f
```
**Prevention:**
- Set realistic timeout values
- Monitor job progress actively
- Use checkpointing for long-running jobs
### Cross-Chain Issues
#### Issue: Bridge Transfer Fails
**Symptoms:**
- Error: "Bridge not available"
- Error: "Invalid bridge configuration"
- Transfer stuck in "pending" state
**Solutions:**
```bash
# Check bridge status
aitbc cross-chain bridge status
# Verify bridge configuration
aitbc cross-chain bridge config
# Retry transfer
aitbc cross-chain transfer <source-chain> <dest-chain> <amount>
```
**Prevention:**
- Verify bridge availability before transfer
- Use testnet for initial cross-chain tests
- Monitor bridge health regularly
### Network Issues
#### Issue: Peer Connection Failed
**Symptoms:**
- Error: "No peers available"
- Network not syncing
- Gossip messages not delivered
**Solutions:**
```bash
# Check peer connectivity
aitbc network peers
# Add bootstrap peers manually
aitbc network peers add <peer-address>
# Check network configuration
cat /etc/aitbc/node.env | grep PEER
# Restart network service
systemctl restart aitbc-blockchain-p2p.service
```
**Prevention:**
- Configure bootstrap peers
- Monitor peer count regularly
- Use reliable peer sources
### Security Issues
#### Issue: Authentication Failed
**Symptoms:**
- Error: "Invalid JWT token"
- Error: "Authentication required"
- API calls rejected
**Solutions:**
```bash
# Check JWT token
aitbc auth token verify <token>
# Generate new token
aitbc auth token generate <wallet-name>
# Check API key configuration
cat /etc/aitbc/.env | grep API_KEY
```
**Prevention:**
- Use secure token storage
- Rotate tokens regularly
- Implement token refresh logic
#### Issue: Permission Denied
**Symptoms:**
- Error: "Permission denied"
- Cannot access resources
- Operations rejected
**Solutions:**
```bash
# Check file permissions
ls -la /var/lib/aitbc/
# Fix permissions
chmod 755 /var/lib/aitbc/
chown -R aitbc:aitbc /var/lib/aitbc/
# Check user permissions
aitbc auth permissions <wallet-name>
```
**Prevention:**
- Use principle of least privilege
- Regular permission audits
- Document permission requirements
## Scenario-Specific Troubleshooting
### Scenario 01: Wallet Basics
**Common Issues:**
- Wallet creation fails due to existing wallet
- Password too weak or forgotten
- Keystore permission errors
**Quick Fix:**
```bash
# Use unique wallet names
aitbc wallet create test-wallet-$(date +%s)
# Use strong passwords
aitbc wallet create test-wallet --password "$(openssl rand -base64 32)"
```
### Scenario 02: Transaction Sending
**Common Issues:**
- Insufficient funds
- Transaction not confirming
- Invalid recipient address
**Quick Fix:**
```bash
# Verify balance first
aitbc wallet balance <wallet-name>
# Check transaction status
aitbc transaction status <tx-hash>
# Use test addresses for development
aitbc transaction send <wallet> 0x0000000000000000000000000000000000000000 1
```
### Scenario 03: Genesis Deployment
**Common Issues:**
- Genesis wallet not configured
- Invalid genesis parameters
- Network already initialized
**Quick Fix:**
```bash
# Check genesis wallet
ls -la /var/lib/aitbc/keystore/genesis.json
# Verify genesis configuration
aitbc genesis validate
# Reset network if needed (CAUTION: destroys data)
aitbc genesis reset --force
```
### Scenario 07: AI Job Submission
**Common Issues:**
- Invalid job parameters
- No available compute resources
- Job stuck in queue
**Quick Fix:**
```bash
# Validate job config
aitbc ai job validate --file job.json
# Check resource availability
aitbc marketplace resources available
# Monitor job queue
aitbc ai job queue
```
## General Debugging Tips
### Enable Verbose Logging
```bash
# Enable verbose CLI output
export AITBC_LOG_LEVEL=DEBUG
# Enable service debug logging
echo "LOG_LEVEL=DEBUG" >> /etc/aitbc/.env
systemctl restart aitbc-*.service
```
### Check System Logs
```bash
# View recent logs
journalctl -u aitbc-* -n 100
# Follow logs in real-time
journalctl -u aitbc-* -f
# Filter by error level
journalctl -u aitbc-* -p err
```
### Verify Configuration
```bash
# Check all AITBC configuration
aitbc config show
# Validate configuration files
aitbc config validate
# Check environment variables
env | grep AITBC
```
### Network Diagnostics
```bash
# Check connectivity to blockchain node
curl -X POST http://localhost:8545 -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
# Check peer connections
aitbc network peers
# Test gossip protocol
aitbc gossip test
```
## Getting Help
### Documentation Resources
- [Agent Training Documentation](../agent-training/README.md)
- [Scenario Documentation](./README.md)
- [Master Glossary](../GLOSSARY.md)
### Community Support
- Check existing issues in the repository
- Search forums for similar problems
- Contact AITBC development team
### Reporting Issues
When reporting issues, include:
1. Scenario being worked on
2. Exact error messages
3. Steps to reproduce
4. System configuration
5. Relevant log output
---
**Related Documentation:**
- [Agent Training README](../agent-training/README.md)
- [Interactive Learning Paths](../agent-training/INTERACTIVE_LEARNING_PATHS.md)
- [Operations Audit](../agent-training/OPERATIONS_AUDIT.md)

507
docs/scenarios/VALIDATION_GUIDE.md Normal file
View File

@@ -0,0 +1,507 @@
# AITBC Scenario Validation Guide
**Last Updated:** 2026-05-09
**Version:** 1.0
**Purpose:** Automated validation procedures and checks for AITBC scenarios
---
## Overview
This guide provides validation procedures for verifying successful completion of AITBC scenarios. Each scenario includes specific validation criteria to ensure operations were executed correctly.
## Validation Framework
### Validation Command
```bash
# Validate a specific scenario
aitbc validate scenario <scenario-id>
# Validate all scenarios in a stage
aitbc validate stage <stage-number>
# Validate with detailed output
aitbc validate scenario <scenario-id> --verbose
# Validate with auto-fix (where applicable)
aitbc validate scenario <scenario-id> --auto-fix
```
### Validation Output Format
```json
{
"scenario_id": "01_wallet_basics",
"status": "passed",
"checks": [
{
"name": "wallet_created",
"status": "passed",
"message": "Wallet test-wallet-001 created successfully"
},
{
"name": "wallet_balance",
"status": "passed",
"message": "Wallet balance: 1000 AIT"
}
],
"timestamp": "2026-05-09T12:00:00Z"
}
```
## Scenario Validation Criteria
### Scenario 01: Wallet Basics
**Validation Checks:**
1. Wallet creation successful
2. Wallet keystore file exists
3. Wallet can be unlocked
4. Wallet balance is retrievable
5. Wallet address is valid
**Validation Commands:**
```bash
# Check wallet exists
test -f /var/lib/aitbc/keystore/<wallet-name>.json
# Verify wallet can be unlocked
aitbc wallet unlock <wallet-name> --password <password>
# Check wallet balance
aitbc wallet balance <wallet-name> | grep -q "^[0-9]"
# Validate wallet address format
aitbc wallet address <wallet-name> | grep -q "^0x[a-fA-F0-9]\{40\}$"
```
**Expected Results:**
- Wallet file exists in keystore directory
- Balance is non-negative number
- Address is valid Ethereum-style address
- Unlock operation succeeds without error
### Scenario 02: Transaction Sending
**Validation Checks:**
1. Transaction submitted successfully
2. Transaction hash is valid
3. Transaction confirms on blockchain
4. Sender balance decreases by amount + fee
5. Recipient balance increases by amount
**Validation Commands:**
```bash
# Submit test transaction
TX_HASH=$(aitbc transaction send <sender> <recipient> 10)
# Validate transaction hash format
echo $TX_HASH | grep -q "^0x[a-fA-F0-9]\{64\}$"
# Check transaction status
aitbc transaction status $TX_HASH | grep -q "confirmed"
# Verify balance changes
SENDER_BEFORE=$(aitbc wallet balance <sender>)
aitbc transaction send <sender> <recipient> 10
SENDER_AFTER=$(aitbc wallet balance <sender>)
# Verify: SENDER_BEFORE > SENDER_AFTER
```
**Expected Results:**
- Transaction hash is 64-character hex string
- Transaction status shows "confirmed"
- Balance changes reflect transaction amount and fees
- Transaction appears in blockchain explorer
### Scenario 03: Genesis Deployment
**Validation Checks:**
1. Genesis wallet configured
2. Genesis block deployed
3. Network initialized
4. Genesis block hash is valid
5. Network is syncable
**Validation Commands:**
```bash
# Check genesis wallet
test -f /var/lib/aitbc/keystore/genesis.json
# Verify genesis block
GENESIS_HASH=$(aitbc genesis hash)
echo $GENESIS_HASH | grep -q "^0x[a-fA-F0-9]\{64\}$"
# Check network status
aitbc blockchain status | grep -q "initialized"
# Verify block number is 0 for new network
aitbc blockchain block number | grep -q "^0$"
```
**Expected Results:**
- Genesis wallet exists with funds
- Genesis block hash is valid
- Network shows as initialized
- Block number starts at 0
- Peers can connect to genesis node
### Scenario 07: AI Job Submission
**Validation Checks:**
1. Job submitted successfully
2. Job ID is valid
3. Job parameters validated
4. Job enters queue
5. Job executes successfully
**Validation Commands:**
```bash
# Submit AI job
JOB_ID=$(aitbc ai job submit job-config.json)
# Validate job ID format
echo $JOB_ID | grep -q "^job-[a-fA-F0-9]\{32\}$"
# Check job status
aitbc ai job status $JOB_ID | grep -q "queued\|running\|completed"
# Monitor job completion
aitbc ai job wait $JOB_ID --timeout 300
# Verify results
aitbc ai job results $JOB_ID | test -s
```
**Expected Results:**
- Job ID follows expected format
- Job transitions through queue → running → completed
- Results are generated and retrievable
- No errors in job execution logs
### Scenario 08: Marketplace Bidding
**Validation Checks:**
1. Bid submitted successfully
2. Bid ID is valid
3. Bid appears in marketplace
4. Bid status is active
5. Bid can be accepted if winning
**Validation Commands:**
```bash
# Submit bid
BID_ID=$(aitbc marketplace bid <listing-id> <amount> <wallet>)
# Validate bid ID format
echo $BID_ID | grep -q "^bid-[a-fA-F0-9]\{32\}$"
# Check bid status
aitbc marketplace bid status $BID_ID | grep -q "active"
# Verify bid in marketplace
aitbc marketplace bids <listing-id> | grep -q $BID_ID
# Check wallet balance after bid (if bid accepted)
aitbc wallet balance <wallet>
```
**Expected Results:**
- Bid ID follows expected format
- Bid appears in marketplace listing
- Bid status shows as active/pending
- Wallet balance reflects bid if accepted
- Marketplace updates bid status correctly
### Scenario 13: Mining Setup
**Validation Checks:**
1. Mining service started
2. Mining node registered
3. Mining rewards received
4. Block production active
5. Mining statistics available
**Validation Commands:**
```bash
# Check mining service status
systemctl status aitbc-mining.service | grep -q "active (running)"
# Verify mining registration
aitbc mining status | grep -q "registered"
# Check mining rewards
aitbc mining rewards | grep -q "^[0-9]"
# Monitor block production
aitbc mining blocks | tail -1
# Check mining statistics
aitbc mining stats
```
**Expected Results:**
- Mining service is running
- Node is registered as miner
- Rewards are being earned
- Blocks are being produced
- Statistics show mining activity
### Scenario 14: Staking Basics
**Validation Checks:**
1. Staking operation successful
2. Staked amount locked
3. Staking rewards accruing
4. Unstaking works correctly
5. Validator status updated
**Validation Commands:**
```bash
# Stake tokens
aitbc staking stake <wallet> 1000
# Verify staked amount
aitbc staking balance <wallet> | grep -q "1000"
# Check rewards
aitbc staking rewards <wallet> | grep -q "^[0-9]"
# Unstake tokens
aitbc staking unstake <wallet> 500
# Verify unstaking
aitbc staking balance <wallet> | grep -q "500"
```
**Expected Results:**
- Staked tokens are locked
- Rewards accrue over time
- Unstaking releases tokens after unbonding period
- Balance reflects staking operations
- Validator status updates correctly
### Scenario 20: Cross Chain Transfer
**Validation Checks:**
1. Bridge initialized
2. Transfer submitted successfully
3. Transfer status trackable
4. Transfer completes on destination chain
5. Balances updated on both chains
**Validation Commands:**
```bash
# Check bridge status
aitbc cross-chain bridge status | grep -q "available"
# Submit cross-chain transfer
TRANSFER_ID=$(aitbc cross-chain transfer <source> <dest> 100)
# Track transfer status
aitbc cross-chain status $TRANSFER_ID | grep -q "pending\|completed"
# Verify destination chain balance
aitbc wallet balance <wallet> --chain <dest>
# Verify source chain balance decreased
aitbc wallet balance <wallet> --chain <source>
```
**Expected Results:**
- Bridge is operational
- Transfer ID is valid and trackable
- Transfer completes successfully
- Balances updated correctly on both chains
- No tokens lost in transfer
## Automated Validation Scripts
### Batch Validation Script
```bash
#!/bin/bash
# validate_scenarios.sh - Batch validate multiple scenarios
SCENARIOS=("01_wallet_basics" "02_transaction_sending" "03_genesis_deployment")
PASSED=0
FAILED=0
for scenario in "${SCENARIOS[@]}"; do
echo "Validating $scenario..."
if aitbc validate scenario $scenario; then
((PASSED++))
echo "✓ $scenario passed"
else
((FAILED++))
echo "✗ $scenario failed"
fi
done
echo ""
echo "Validation Summary:"
echo "Passed: $PASSED"
echo "Failed: $FAILED"
echo "Total: $((PASSED + FAILED))"
exit $FAILED
```
### Stage Validation Script
```bash
#!/bin/bash
# validate_stage.sh - Validate all scenarios in a stage
STAGE=$1
if [ -z "$STAGE" ]; then
echo "Usage: $0 <stage-number>"
exit 1
fi
echo "Validating Stage $STAGE scenarios..."
aitbc validate stage $STAGE --verbose
if [ $? -eq 0 ]; then
echo "✓ Stage $STAGE validation passed"
else
echo "✗ Stage $STAGE validation failed"
exit 1
fi
```
### Continuous Validation
```bash
#!/bin/bash
# continuous_validation.sh - Run validation continuously
INTERVAL=300 # 5 minutes
while true; do
echo "Running validation at $(date)"
aitbc validate stage 1
aitbc validate stage 2
aitbc validate stage 3
echo "Sleeping for $INTERVAL seconds..."
sleep $INTERVAL
done
```
## Validation Best Practices
### Pre-Validation Checklist
- Ensure blockchain node is running and synced
- Verify wallet has sufficient funds
- Check network connectivity
- Confirm required services are active
- Review scenario prerequisites
### Validation Frequency
- **Development**: Run validation after each scenario completion
- **Testing**: Run full validation before deployment
- **Production**: Run validation daily or on schedule
- **CI/CD**: Integrate validation into pipeline
### Handling Validation Failures
1. Review validation output for specific failure
2. Check logs for error messages
3. Verify prerequisites are met
4. Re-run scenario if needed
5. Consult troubleshooting guide if persistent
### Validation Reporting
```bash
# Generate validation report
aitbc validate stage 1 --report > validation-report-$(date +%Y%m%d).json
# View validation history
aitbc validate history
# Compare validation results
aitbc validate compare report-20260508.json report-20260509.json
```
## Integration with CI/CD
### GitHub Actions Example
```yaml
name: Scenario Validation
on: [push, pull_request]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Setup AITBC
run: |
pip install -e .
aitbc setup environment
- name: Validate Stage 1
run: aitbc validate stage 1
- name: Validate Stage 2
run: aitbc validate stage 2
- name: Validate Stage 3
run: aitbc validate stage 3
```
### Gitea Actions Example
```yaml
name: Scenario Validation
on: [push]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Setup Python
uses: actions/setup-python@v2
with:
python-version: '3.13'
- name: Install AITBC
run: |
pip install -e .
aitbc setup environment
- name: Run Validation
run: |
for stage in {1..3}; do
aitbc validate stage $stage
done
```
## Validation Metrics
### Key Metrics to Track
- Scenario pass rate
- Average validation time
- Common failure patterns
- Validation trend over time
- Resource utilization during validation
### Metrics Collection
```bash
# Collect validation metrics
aitbc validate metrics --period 7d
# Export metrics to CSV
aitbc validate metrics --format csv > metrics.csv
# Generate validation report
aitbc validate report --period 30d > report.html
```
---
**Related Documentation:**
- [Troubleshooting Guide](./TROUBLESHOOTING_GUIDE.md)
- [Agent Training README](../agent-training/README.md)
- [Interactive Learning Paths](../agent-training/INTERACTIVE_LEARNING_PATHS.md)