diff --git a/docs/scenarios/TROUBLESHOOTING_GUIDE.md b/docs/scenarios/TROUBLESHOOTING_GUIDE.md new file mode 100644 index 00000000..ff10b5e7 --- /dev/null +++ b/docs/scenarios/TROUBLESHOOTING_GUIDE.md @@ -0,0 +1,466 @@ +# AITBC Scenario Troubleshooting Guide + +**Last Updated:** 2026-05-09 +**Version:** 1.0 +**Purpose:** Common failure points and solutions for AITBC scenarios + +--- + +## Overview + +This guide provides troubleshooting information for common issues encountered when working through AITBC scenarios. Each scenario may have specific failure points, but many issues follow common patterns across the system. + +## Common Failure Patterns + +### Wallet-Related Issues + +#### Issue: Wallet Creation Fails +**Symptoms:** +- Error: "Wallet already exists" +- Error: "Invalid password" +- Error: "Keystore directory not accessible" + +**Solutions:** +```bash +# Check if wallet already exists +ls -la /var/lib/aitbc/keystore/ + +# Remove existing wallet if needed (CAUTION: loses funds) +rm /var/lib/aitbc/keystore/.json + +# Verify keystore directory permissions +chmod 700 /var/lib/aitbc/keystore/ +chown aitbc:aitbc /var/lib/aitbc/keystore/ +``` + +**Prevention:** +- Use deterministic wallet naming (e.g., test-wallet-001, test-wallet-002) +- Always check for existing wallets before creation +- Use strong passwords and store them securely + +#### Issue: Insufficient Funds +**Symptoms:** +- Error: "Insufficient funds for transaction" +- Transaction rejected by blockchain + +**Solutions:** +```bash +# Check wallet balance +aitbc wallet balance + +# Fund wallet from genesis wallet if available +aitbc transaction send genesis 1000 + +# Or use faucet if available +curl -X POST http://faucet.aitbc.local/fund \ + -H "Content-Type: application/json" \ + -d '{"address": "", "amount": 1000}' +``` + +**Prevention:** +- Always check balance before transactions +- Use testnet faucets for development +- Maintain a funding wallet for test operations + +### Blockchain Node Issues + +#### Issue: Node Not Running +**Symptoms:** +- Error: "Connection refused" +- Error: "Node not responding" +- CLI commands timeout + +**Solutions:** +```bash +# Check node status +systemctl status aitbc-blockchain-node.service + +# Start node if stopped +systemctl start aitbc-blockchain-node.service + +# Check node logs +journalctl -u aitbc-blockchain-node.service -f + +# Verify node connectivity +aitbc blockchain status +``` + +**Prevention:** +- Monitor node health regularly +- Set up automated restart on failure +- Use systemd auto-restart configuration + +#### Issue: Node Out of Sync +**Symptoms:** +- Transactions not confirming +- Balance showing incorrect values +- Block height mismatch + +**Solutions:** +```bash +# Check sync status +aitbc blockchain sync status + +# Force sync if needed +aitbc blockchain sync force + +# Enable auto-sync in /etc/aitbc/.env +echo "auto_sync_enabled=true" >> /etc/aitbc/.env +systemctl restart aitbc-blockchain-node.service +``` + +**Prevention:** +- Enable auto-sync in configuration +- Monitor sync status regularly +- Use stable network connections + +### Marketplace Issues + +#### Issue: Bid Not Accepted +**Symptoms:** +- Bid rejected by marketplace +- Error: "Insufficient balance for bid" +- Error: "Invalid bid amount" + +**Solutions:** +```bash +# Check marketplace status +aitbc marketplace status + +# Verify wallet balance +aitbc wallet balance + +# Check current market rates +aitbc marketplace rates + +# Retry with valid bid amount +aitbc marketplace bid +``` + +**Prevention:** +- Check market rates before bidding +- Ensure sufficient balance including fees +- Use bid validation before submission + +#### Issue: GPU Listing Not Appearing +**Symptoms:** +- GPU not visible in marketplace +- Listing creation fails +- Error: "Invalid GPU configuration" + +**Solutions:** +```bash +# Verify GPU availability +nvidia-smi + +# Check GPU service status +systemctl status aitbc-gpu-service.service + +# Re-list GPU with correct configuration +aitbc marketplace gpu list --gpu-id --price +``` + +**Prevention:** +- Verify GPU hardware before listing +- Test GPU service independently +- Use standard pricing models + +### AI Job Issues + +#### Issue: Job Submission Fails +**Symptoms:** +- Error: "Invalid job parameters" +- Error: "Insufficient compute resources" +- Job stuck in "pending" state + +**Solutions:** +```bash +# Check AI service status +systemctl status aitbc-ai.service + +# Verify job parameters +aitbc ai job validate --file + +# Check available compute resources +aitbc marketplace resources available + +# Retry job submission +aitbc ai job submit +``` + +**Prevention:** +- Validate job configuration before submission +- Check resource availability first +- Use job templates for common tasks + +#### Issue: Job Execution Timeout +**Symptoms:** +- Job runs longer than expected +- Job marked as "failed" due to timeout +- No results returned + +**Solutions:** +```bash +# Check job status +aitbc ai job status + +# Extend timeout if needed +aitbc ai job update --timeout + +# Check compute provider logs +journalctl -u aitbc-gpu-service.service -f +``` + +**Prevention:** +- Set realistic timeout values +- Monitor job progress actively +- Use checkpointing for long-running jobs + +### Cross-Chain Issues + +#### Issue: Bridge Transfer Fails +**Symptoms:** +- Error: "Bridge not available" +- Error: "Invalid bridge configuration" +- Transfer stuck in "pending" state + +**Solutions:** +```bash +# Check bridge status +aitbc cross-chain bridge status + +# Verify bridge configuration +aitbc cross-chain bridge config + +# Retry transfer +aitbc cross-chain transfer +``` + +**Prevention:** +- Verify bridge availability before transfer +- Use testnet for initial cross-chain tests +- Monitor bridge health regularly + +### Network Issues + +#### Issue: Peer Connection Failed +**Symptoms:** +- Error: "No peers available" +- Network not syncing +- Gossip messages not delivered + +**Solutions:** +```bash +# Check peer connectivity +aitbc network peers + +# Add bootstrap peers manually +aitbc network peers add + +# Check network configuration +cat /etc/aitbc/node.env | grep PEER + +# Restart network service +systemctl restart aitbc-blockchain-p2p.service +``` + +**Prevention:** +- Configure bootstrap peers +- Monitor peer count regularly +- Use reliable peer sources + +### Security Issues + +#### Issue: Authentication Failed +**Symptoms:** +- Error: "Invalid JWT token" +- Error: "Authentication required" +- API calls rejected + +**Solutions:** +```bash +# Check JWT token +aitbc auth token verify + +# Generate new token +aitbc auth token generate + +# Check API key configuration +cat /etc/aitbc/.env | grep API_KEY +``` + +**Prevention:** +- Use secure token storage +- Rotate tokens regularly +- Implement token refresh logic + +#### Issue: Permission Denied +**Symptoms:** +- Error: "Permission denied" +- Cannot access resources +- Operations rejected + +**Solutions:** +```bash +# Check file permissions +ls -la /var/lib/aitbc/ + +# Fix permissions +chmod 755 /var/lib/aitbc/ +chown -R aitbc:aitbc /var/lib/aitbc/ + +# Check user permissions +aitbc auth permissions +``` + +**Prevention:** +- Use principle of least privilege +- Regular permission audits +- Document permission requirements + +## Scenario-Specific Troubleshooting + +### Scenario 01: Wallet Basics +**Common Issues:** +- Wallet creation fails due to existing wallet +- Password too weak or forgotten +- Keystore permission errors + +**Quick Fix:** +```bash +# Use unique wallet names +aitbc wallet create test-wallet-$(date +%s) + +# Use strong passwords +aitbc wallet create test-wallet --password "$(openssl rand -base64 32)" +``` + +### Scenario 02: Transaction Sending +**Common Issues:** +- Insufficient funds +- Transaction not confirming +- Invalid recipient address + +**Quick Fix:** +```bash +# Verify balance first +aitbc wallet balance + +# Check transaction status +aitbc transaction status + +# Use test addresses for development +aitbc transaction send 0x0000000000000000000000000000000000000000 1 +``` + +### Scenario 03: Genesis Deployment +**Common Issues:** +- Genesis wallet not configured +- Invalid genesis parameters +- Network already initialized + +**Quick Fix:** +```bash +# Check genesis wallet +ls -la /var/lib/aitbc/keystore/genesis.json + +# Verify genesis configuration +aitbc genesis validate + +# Reset network if needed (CAUTION: destroys data) +aitbc genesis reset --force +``` + +### Scenario 07: AI Job Submission +**Common Issues:** +- Invalid job parameters +- No available compute resources +- Job stuck in queue + +**Quick Fix:** +```bash +# Validate job config +aitbc ai job validate --file job.json + +# Check resource availability +aitbc marketplace resources available + +# Monitor job queue +aitbc ai job queue +``` + +## General Debugging Tips + +### Enable Verbose Logging +```bash +# Enable verbose CLI output +export AITBC_LOG_LEVEL=DEBUG + +# Enable service debug logging +echo "LOG_LEVEL=DEBUG" >> /etc/aitbc/.env +systemctl restart aitbc-*.service +``` + +### Check System Logs +```bash +# View recent logs +journalctl -u aitbc-* -n 100 + +# Follow logs in real-time +journalctl -u aitbc-* -f + +# Filter by error level +journalctl -u aitbc-* -p err +``` + +### Verify Configuration +```bash +# Check all AITBC configuration +aitbc config show + +# Validate configuration files +aitbc config validate + +# Check environment variables +env | grep AITBC +``` + +### Network Diagnostics +```bash +# Check connectivity to blockchain node +curl -X POST http://localhost:8545 -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' + +# Check peer connections +aitbc network peers + +# Test gossip protocol +aitbc gossip test +``` + +## Getting Help + +### Documentation Resources +- [Agent Training Documentation](../agent-training/README.md) +- [Scenario Documentation](./README.md) +- [Master Glossary](../GLOSSARY.md) + +### Community Support +- Check existing issues in the repository +- Search forums for similar problems +- Contact AITBC development team + +### Reporting Issues +When reporting issues, include: +1. Scenario being worked on +2. Exact error messages +3. Steps to reproduce +4. System configuration +5. Relevant log output + +--- + +**Related Documentation:** +- [Agent Training README](../agent-training/README.md) +- [Interactive Learning Paths](../agent-training/INTERACTIVE_LEARNING_PATHS.md) +- [Operations Audit](../agent-training/OPERATIONS_AUDIT.md) diff --git a/docs/scenarios/VALIDATION_GUIDE.md b/docs/scenarios/VALIDATION_GUIDE.md new file mode 100644 index 00000000..1efc7b17 --- /dev/null +++ b/docs/scenarios/VALIDATION_GUIDE.md @@ -0,0 +1,507 @@ +# AITBC Scenario Validation Guide + +**Last Updated:** 2026-05-09 +**Version:** 1.0 +**Purpose:** Automated validation procedures and checks for AITBC scenarios + +--- + +## Overview + +This guide provides validation procedures for verifying successful completion of AITBC scenarios. Each scenario includes specific validation criteria to ensure operations were executed correctly. + +## Validation Framework + +### Validation Command + +```bash +# Validate a specific scenario +aitbc validate scenario + +# Validate all scenarios in a stage +aitbc validate stage + +# Validate with detailed output +aitbc validate scenario --verbose + +# Validate with auto-fix (where applicable) +aitbc validate scenario --auto-fix +``` + +### Validation Output Format + +```json +{ + "scenario_id": "01_wallet_basics", + "status": "passed", + "checks": [ + { + "name": "wallet_created", + "status": "passed", + "message": "Wallet test-wallet-001 created successfully" + }, + { + "name": "wallet_balance", + "status": "passed", + "message": "Wallet balance: 1000 AIT" + } + ], + "timestamp": "2026-05-09T12:00:00Z" +} +``` + +## Scenario Validation Criteria + +### Scenario 01: Wallet Basics + +**Validation Checks:** +1. Wallet creation successful +2. Wallet keystore file exists +3. Wallet can be unlocked +4. Wallet balance is retrievable +5. Wallet address is valid + +**Validation Commands:** +```bash +# Check wallet exists +test -f /var/lib/aitbc/keystore/.json + +# Verify wallet can be unlocked +aitbc wallet unlock --password + +# Check wallet balance +aitbc wallet balance | grep -q "^[0-9]" + +# Validate wallet address format +aitbc wallet address | grep -q "^0x[a-fA-F0-9]\{40\}$" +``` + +**Expected Results:** +- Wallet file exists in keystore directory +- Balance is non-negative number +- Address is valid Ethereum-style address +- Unlock operation succeeds without error + +### Scenario 02: Transaction Sending + +**Validation Checks:** +1. Transaction submitted successfully +2. Transaction hash is valid +3. Transaction confirms on blockchain +4. Sender balance decreases by amount + fee +5. Recipient balance increases by amount + +**Validation Commands:** +```bash +# Submit test transaction +TX_HASH=$(aitbc transaction send 10) + +# Validate transaction hash format +echo $TX_HASH | grep -q "^0x[a-fA-F0-9]\{64\}$" + +# Check transaction status +aitbc transaction status $TX_HASH | grep -q "confirmed" + +# Verify balance changes +SENDER_BEFORE=$(aitbc wallet balance ) +aitbc transaction send 10 +SENDER_AFTER=$(aitbc wallet balance ) +# Verify: SENDER_BEFORE > SENDER_AFTER +``` + +**Expected Results:** +- Transaction hash is 64-character hex string +- Transaction status shows "confirmed" +- Balance changes reflect transaction amount and fees +- Transaction appears in blockchain explorer + +### Scenario 03: Genesis Deployment + +**Validation Checks:** +1. Genesis wallet configured +2. Genesis block deployed +3. Network initialized +4. Genesis block hash is valid +5. Network is syncable + +**Validation Commands:** +```bash +# Check genesis wallet +test -f /var/lib/aitbc/keystore/genesis.json + +# Verify genesis block +GENESIS_HASH=$(aitbc genesis hash) +echo $GENESIS_HASH | grep -q "^0x[a-fA-F0-9]\{64\}$" + +# Check network status +aitbc blockchain status | grep -q "initialized" + +# Verify block number is 0 for new network +aitbc blockchain block number | grep -q "^0$" +``` + +**Expected Results:** +- Genesis wallet exists with funds +- Genesis block hash is valid +- Network shows as initialized +- Block number starts at 0 +- Peers can connect to genesis node + +### Scenario 07: AI Job Submission + +**Validation Checks:** +1. Job submitted successfully +2. Job ID is valid +3. Job parameters validated +4. Job enters queue +5. Job executes successfully + +**Validation Commands:** +```bash +# Submit AI job +JOB_ID=$(aitbc ai job submit job-config.json) + +# Validate job ID format +echo $JOB_ID | grep -q "^job-[a-fA-F0-9]\{32\}$" + +# Check job status +aitbc ai job status $JOB_ID | grep -q "queued\|running\|completed" + +# Monitor job completion +aitbc ai job wait $JOB_ID --timeout 300 + +# Verify results +aitbc ai job results $JOB_ID | test -s +``` + +**Expected Results:** +- Job ID follows expected format +- Job transitions through queue → running → completed +- Results are generated and retrievable +- No errors in job execution logs + +### Scenario 08: Marketplace Bidding + +**Validation Checks:** +1. Bid submitted successfully +2. Bid ID is valid +3. Bid appears in marketplace +4. Bid status is active +5. Bid can be accepted if winning + +**Validation Commands:** +```bash +# Submit bid +BID_ID=$(aitbc marketplace bid ) + +# Validate bid ID format +echo $BID_ID | grep -q "^bid-[a-fA-F0-9]\{32\}$" + +# Check bid status +aitbc marketplace bid status $BID_ID | grep -q "active" + +# Verify bid in marketplace +aitbc marketplace bids | grep -q $BID_ID + +# Check wallet balance after bid (if bid accepted) +aitbc wallet balance +``` + +**Expected Results:** +- Bid ID follows expected format +- Bid appears in marketplace listing +- Bid status shows as active/pending +- Wallet balance reflects bid if accepted +- Marketplace updates bid status correctly + +### Scenario 13: Mining Setup + +**Validation Checks:** +1. Mining service started +2. Mining node registered +3. Mining rewards received +4. Block production active +5. Mining statistics available + +**Validation Commands:** +```bash +# Check mining service status +systemctl status aitbc-mining.service | grep -q "active (running)" + +# Verify mining registration +aitbc mining status | grep -q "registered" + +# Check mining rewards +aitbc mining rewards | grep -q "^[0-9]" + +# Monitor block production +aitbc mining blocks | tail -1 + +# Check mining statistics +aitbc mining stats +``` + +**Expected Results:** +- Mining service is running +- Node is registered as miner +- Rewards are being earned +- Blocks are being produced +- Statistics show mining activity + +### Scenario 14: Staking Basics + +**Validation Checks:** +1. Staking operation successful +2. Staked amount locked +3. Staking rewards accruing +4. Unstaking works correctly +5. Validator status updated + +**Validation Commands:** +```bash +# Stake tokens +aitbc staking stake 1000 + +# Verify staked amount +aitbc staking balance | grep -q "1000" + +# Check rewards +aitbc staking rewards | grep -q "^[0-9]" + +# Unstake tokens +aitbc staking unstake 500 + +# Verify unstaking +aitbc staking balance | grep -q "500" +``` + +**Expected Results:** +- Staked tokens are locked +- Rewards accrue over time +- Unstaking releases tokens after unbonding period +- Balance reflects staking operations +- Validator status updates correctly + +### Scenario 20: Cross Chain Transfer + +**Validation Checks:** +1. Bridge initialized +2. Transfer submitted successfully +3. Transfer status trackable +4. Transfer completes on destination chain +5. Balances updated on both chains + +**Validation Commands:** +```bash +# Check bridge status +aitbc cross-chain bridge status | grep -q "available" + +# Submit cross-chain transfer +TRANSFER_ID=$(aitbc cross-chain transfer 100) + +# Track transfer status +aitbc cross-chain status $TRANSFER_ID | grep -q "pending\|completed" + +# Verify destination chain balance +aitbc wallet balance --chain + +# Verify source chain balance decreased +aitbc wallet balance --chain +``` + +**Expected Results:** +- Bridge is operational +- Transfer ID is valid and trackable +- Transfer completes successfully +- Balances updated correctly on both chains +- No tokens lost in transfer + +## Automated Validation Scripts + +### Batch Validation Script + +```bash +#!/bin/bash +# validate_scenarios.sh - Batch validate multiple scenarios + +SCENARIOS=("01_wallet_basics" "02_transaction_sending" "03_genesis_deployment") +PASSED=0 +FAILED=0 + +for scenario in "${SCENARIOS[@]}"; do + echo "Validating $scenario..." + if aitbc validate scenario $scenario; then + ((PASSED++)) + echo "✓ $scenario passed" + else + ((FAILED++)) + echo "✗ $scenario failed" + fi +done + +echo "" +echo "Validation Summary:" +echo "Passed: $PASSED" +echo "Failed: $FAILED" +echo "Total: $((PASSED + FAILED))" + +exit $FAILED +``` + +### Stage Validation Script + +```bash +#!/bin/bash +# validate_stage.sh - Validate all scenarios in a stage + +STAGE=$1 + +if [ -z "$STAGE" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Validating Stage $STAGE scenarios..." +aitbc validate stage $STAGE --verbose + +if [ $? -eq 0 ]; then + echo "✓ Stage $STAGE validation passed" +else + echo "✗ Stage $STAGE validation failed" + exit 1 +fi +``` + +### Continuous Validation + +```bash +#!/bin/bash +# continuous_validation.sh - Run validation continuously + +INTERVAL=300 # 5 minutes + +while true; do + echo "Running validation at $(date)" + aitbc validate stage 1 + aitbc validate stage 2 + aitbc validate stage 3 + + echo "Sleeping for $INTERVAL seconds..." + sleep $INTERVAL +done +``` + +## Validation Best Practices + +### Pre-Validation Checklist +- Ensure blockchain node is running and synced +- Verify wallet has sufficient funds +- Check network connectivity +- Confirm required services are active +- Review scenario prerequisites + +### Validation Frequency +- **Development**: Run validation after each scenario completion +- **Testing**: Run full validation before deployment +- **Production**: Run validation daily or on schedule +- **CI/CD**: Integrate validation into pipeline + +### Handling Validation Failures +1. Review validation output for specific failure +2. Check logs for error messages +3. Verify prerequisites are met +4. Re-run scenario if needed +5. Consult troubleshooting guide if persistent + +### Validation Reporting +```bash +# Generate validation report +aitbc validate stage 1 --report > validation-report-$(date +%Y%m%d).json + +# View validation history +aitbc validate history + +# Compare validation results +aitbc validate compare report-20260508.json report-20260509.json +``` + +## Integration with CI/CD + +### GitHub Actions Example + +```yaml +name: Scenario Validation +on: [push, pull_request] + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Setup AITBC + run: | + pip install -e . + aitbc setup environment + - name: Validate Stage 1 + run: aitbc validate stage 1 + - name: Validate Stage 2 + run: aitbc validate stage 2 + - name: Validate Stage 3 + run: aitbc validate stage 3 +``` + +### Gitea Actions Example + +```yaml +name: Scenario Validation +on: [push] + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v2 + - name: Setup Python + uses: actions/setup-python@v2 + with: + python-version: '3.13' + - name: Install AITBC + run: | + pip install -e . + aitbc setup environment + - name: Run Validation + run: | + for stage in {1..3}; do + aitbc validate stage $stage + done +``` + +## Validation Metrics + +### Key Metrics to Track +- Scenario pass rate +- Average validation time +- Common failure patterns +- Validation trend over time +- Resource utilization during validation + +### Metrics Collection + +```bash +# Collect validation metrics +aitbc validate metrics --period 7d + +# Export metrics to CSV +aitbc validate metrics --format csv > metrics.csv + +# Generate validation report +aitbc validate report --period 30d > report.html +``` + +--- + +**Related Documentation:** +- [Troubleshooting Guide](./TROUBLESHOOTING_GUIDE.md) +- [Agent Training README](../agent-training/README.md) +- [Interactive Learning Paths](../agent-training/INTERACTIVE_LEARNING_PATHS.md)