aitbc/docs/scenarios/TROUBLESHOOTING_GUIDE.md

AITBC Scenario Troubleshooting Guide

Last Updated: 2026-05-09
Version: 1.0
Purpose: Common failure points and solutions for AITBC scenarios

---

Overview

This guide provides troubleshooting information for common issues encountered when working through AITBC scenarios. Each scenario may have specific failure points, but many issues follow common patterns across the system.

Common Failure Patterns

Wallet-Related Issues

Issue: Wallet Creation Fails

Symptoms:

• Error: "Wallet already exists"
• Error: "Invalid password"
• Error: "Keystore directory not accessible"

Solutions:

# Check if wallet already exists
ls -la /var/lib/aitbc/keystore/

# Remove existing wallet if needed (CAUTION: loses funds)
rm /var/lib/aitbc/keystore/<wallet-name>.json

# Verify keystore directory permissions
chmod 700 /var/lib/aitbc/keystore/
chown aitbc:aitbc /var/lib/aitbc/keystore/

Prevention:

• Use deterministic wallet naming (e.g., test-wallet-001, test-wallet-002)
• Always check for existing wallets before creation
• Use strong passwords and store them securely

Issue: Insufficient Funds

Symptoms:

• Error: "Insufficient funds for transaction"
• Transaction rejected by blockchain

Solutions:

# Check wallet balance
aitbc wallet balance <wallet-name>

# Fund wallet from genesis wallet if available
aitbc transaction send genesis <wallet-name> 1000

# Or use faucet if available
curl -X POST http://faucet.aitbc.local/fund \
  -H "Content-Type: application/json" \
  -d '{"address": "<wallet-address>", "amount": 1000}'

Prevention:

• Always check balance before transactions
• Use testnet faucets for development
• Maintain a funding wallet for test operations

Blockchain Node Issues

Issue: Node Not Running

Symptoms:

• Error: "Connection refused"
• Error: "Node not responding"
• CLI commands timeout

Solutions:

# Check node status
systemctl status aitbc-blockchain-node.service

# Start node if stopped
systemctl start aitbc-blockchain-node.service

# Check node logs
journalctl -u aitbc-blockchain-node.service -f

# Verify node connectivity
aitbc blockchain status

Prevention:

• Monitor node health regularly
• Set up automated restart on failure
• Use systemd auto-restart configuration

Issue: Node Out of Sync

Symptoms:

• Transactions not confirming
• Balance showing incorrect values
• Block height mismatch

Solutions:

# Check sync status
aitbc blockchain sync status

# Force sync if needed
aitbc blockchain sync force

# Enable auto-sync in /etc/aitbc/.env
echo "auto_sync_enabled=true" >> /etc/aitbc/.env
systemctl restart aitbc-blockchain-node.service

Prevention:

• Enable auto-sync in configuration
• Monitor sync status regularly
• Use stable network connections

Marketplace Issues

Issue: Bid Not Accepted

Symptoms:

• Bid rejected by marketplace
• Error: "Insufficient balance for bid"
• Error: "Invalid bid amount"

Solutions:

# Check marketplace status
aitbc marketplace status

# Verify wallet balance
aitbc wallet balance <wallet-name>

# Check current market rates
aitbc marketplace rates

# Retry with valid bid amount
aitbc marketplace bid <listing-id> <amount> <wallet-name>

Prevention:

• Check market rates before bidding
• Ensure sufficient balance including fees
• Use bid validation before submission

Issue: GPU Listing Not Appearing

Symptoms:

• GPU not visible in marketplace
• Listing creation fails
• Error: "Invalid GPU configuration"

Solutions:

# Verify GPU availability
nvidia-smi

# Check GPU service status
systemctl status aitbc-gpu-service.service

# Re-list GPU with correct configuration
aitbc marketplace gpu list --gpu-id <gpu-id> --price <price>

Prevention:

• Verify GPU hardware before listing
• Test GPU service independently
• Use standard pricing models

AI Job Issues

Issue: Job Submission Fails

Symptoms:

• Error: "Invalid job parameters"
• Error: "Insufficient compute resources"
• Job stuck in "pending" state

Solutions:

# Check AI service status
systemctl status aitbc-ai.service

# Verify job parameters
aitbc ai job validate --file <job-config.json>

# Check available compute resources
aitbc marketplace resources available

# Retry job submission
aitbc ai job submit <job-config.json>

Prevention:

• Validate job configuration before submission
• Check resource availability first
• Use job templates for common tasks

Issue: Job Execution Timeout

Symptoms:

• Job runs longer than expected
• Job marked as "failed" due to timeout
• No results returned

Solutions:

# Check job status
aitbc ai job status <job-id>

# Extend timeout if needed
aitbc ai job update <job-id> --timeout <new-timeout>

# Check compute provider logs
journalctl -u aitbc-gpu-service.service -f

Prevention:

• Set realistic timeout values
• Monitor job progress actively
• Use checkpointing for long-running jobs

Cross-Chain Issues

Issue: Bridge Transfer Fails

Symptoms:

• Error: "Bridge not available"
• Error: "Invalid bridge configuration"
• Transfer stuck in "pending" state

Solutions:

# Check bridge status
aitbc cross-chain bridge status

# Verify bridge configuration
aitbc cross-chain bridge config

# Retry transfer
aitbc cross-chain transfer <source-chain> <dest-chain> <amount>

Prevention:

• Verify bridge availability before transfer
• Use testnet for initial cross-chain tests
• Monitor bridge health regularly

Network Issues

Issue: Peer Connection Failed

Symptoms:

• Error: "No peers available"
• Network not syncing
• Gossip messages not delivered

Solutions:

# Check peer connectivity
aitbc network peers

# Add bootstrap peers manually
aitbc network peers add <peer-address>

# Check network configuration
cat /etc/aitbc/node.env | grep PEER

# Restart network service
systemctl restart aitbc-blockchain-p2p.service

Prevention:

• Configure bootstrap peers
• Monitor peer count regularly
• Use reliable peer sources

Security Issues

Issue: Authentication Failed

Symptoms:

• Error: "Invalid JWT token"
• Error: "Authentication required"
• API calls rejected

Solutions:

# Check JWT token
aitbc auth token verify <token>

# Generate new token
aitbc auth token generate <wallet-name>

# Check API key configuration
cat /etc/aitbc/.env | grep API_KEY

Prevention:

• Use secure token storage
• Rotate tokens regularly
• Implement token refresh logic

Issue: Permission Denied

Symptoms:

• Error: "Permission denied"
• Cannot access resources
• Operations rejected

Solutions:

# Check file permissions
ls -la /var/lib/aitbc/

# Fix permissions
chmod 755 /var/lib/aitbc/
chown -R aitbc:aitbc /var/lib/aitbc/

# Check user permissions
aitbc auth permissions <wallet-name>

Prevention:

• Use principle of least privilege
• Regular permission audits
• Document permission requirements

Scenario-Specific Troubleshooting

Scenario 01: Wallet Basics

Common Issues:

• Wallet creation fails due to existing wallet
• Password too weak or forgotten
• Keystore permission errors

Quick Fix:

# Use unique wallet names
aitbc wallet create test-wallet-$(date +%s)

# Use strong passwords
aitbc wallet create test-wallet --password "$(openssl rand -base64 32)"

Scenario 02: Transaction Sending

Common Issues:

• Insufficient funds
• Transaction not confirming
• Invalid recipient address

Quick Fix:

# Verify balance first
aitbc wallet balance <wallet-name>

# Check transaction status
aitbc transaction status <tx-hash>

# Use test addresses for development
aitbc transaction send <wallet> 0x0000000000000000000000000000000000000000 1

Scenario 03: Genesis Deployment

Common Issues:

• Genesis wallet not configured
• Invalid genesis parameters
• Network already initialized

Quick Fix:

# Check genesis wallet
ls -la /var/lib/aitbc/keystore/genesis.json

# Verify genesis configuration
aitbc genesis validate

# Reset network if needed (CAUTION: destroys data)
aitbc genesis reset --force

Scenario 07: AI Job Submission

Common Issues:

• Invalid job parameters
• No available compute resources
• Job stuck in queue

Quick Fix:

# Validate job config
aitbc ai job validate --file job.json

# Check resource availability
aitbc marketplace resources available

# Monitor job queue
aitbc ai job queue

General Debugging Tips

Enable Verbose Logging

# Enable verbose CLI output
export AITBC_LOG_LEVEL=DEBUG

# Enable service debug logging
echo "LOG_LEVEL=DEBUG" >> /etc/aitbc/.env
systemctl restart aitbc-*.service

Check System Logs

# View recent logs
journalctl -u aitbc-* -n 100

# Follow logs in real-time
journalctl -u aitbc-* -f

# Filter by error level
journalctl -u aitbc-* -p err

Verify Configuration

# Check all AITBC configuration
aitbc config show

# Validate configuration files
aitbc config validate

# Check environment variables
env | grep AITBC

Network Diagnostics

# Check connectivity to blockchain node
curl -X POST http://localhost:8006 -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Check peer connections
aitbc network peers

# Test gossip protocol
aitbc gossip test

Getting Help

Documentation Resources

• Agent Training Documentation
• Scenario Documentation
• Master Glossary

Community Support

• Check existing issues in the repository
• Search forums for similar problems
• Contact AITBC development team

Reporting Issues

When reporting issues, include:

1. Scenario being worked on
2. Exact error messages
3. Steps to reproduce
4. System configuration
5. Relevant log output

---

Related Documentation:

• Agent Training README
• Interactive Learning Paths
• Operations Audit