docs: enhance Ollama GPU test workflow, reorganize project structure, and fix pytest warnings
- Upgrade ollama-gpu-provider skill to v2.0 with complete test workflow documentation - Add comprehensive troubleshooting, monitoring commands, and CI/CD integration examples - Update client.py default coordinator port from 8000 to 18000 - Clear currentissue.md and add usage guidelines for issue tracking - Create dev-utils/ directory and move aitbc-pythonpath.pth from root - Create docs/guides/ and docs
This commit is contained in:
@@ -1,39 +1,215 @@
|
||||
---
|
||||
name: ollama-gpu-provider
|
||||
description: End-to-end Ollama prompt payment test against the GPU miner provider
|
||||
version: 1.0.0
|
||||
author: Cascade
|
||||
tags: [gpu, miner, ollama, payments, receipts, test]
|
||||
title: Ollama GPU Provider Complete Test Workflow
|
||||
description: Complete end-to-end test workflow for Ollama GPU inference jobs including client submission, miner processing, receipt generation, payment processing, and blockchain recording
|
||||
version: 2.0
|
||||
author: AITBC Team
|
||||
tags: [ollama, gpu, miner, testing, workflow, blockchain, payment]
|
||||
---
|
||||
|
||||
# Ollama GPU Provider Test Skill
|
||||
# Ollama GPU Provider Complete Test Workflow
|
||||
|
||||
This skill runs an end-to-end client → coordinator → GPU miner → receipt flow using an Ollama prompt.
|
||||
This skill provides a comprehensive test workflow for verifying the entire Ollama GPU inference pipeline from client job submission through blockchain transaction recording.
|
||||
|
||||
## Overview
|
||||
|
||||
The test submits a prompt (default: "hello") to the coordinator via the host proxy, waits for completion, and verifies that the job result and signed receipt are returned.
|
||||
The complete flow includes:
|
||||
1. Client submits inference job to coordinator
|
||||
2. GPU miner picks up and processes job via Ollama
|
||||
3. Miner submits result with metrics
|
||||
4. Coordinator generates signed receipt
|
||||
5. Client processes payment to miner
|
||||
6. Transaction recorded on blockchain
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Host GPU miner running and registered (RTX 4060 Ti + Ollama)
|
||||
- Incus proxy forwarding `127.0.0.1:18000` → container `127.0.0.1:8000`
|
||||
- Coordinator running in container (`coordinator-api.service`)
|
||||
- Receipt signing key configured in `/opt/coordinator-api/src/.env`
|
||||
|
||||
## Test Command
|
||||
### Required Services
|
||||
- Coordinator API running on port 18000
|
||||
- GPU miner service running (`aitbc-host-gpu-miner.service`)
|
||||
- Ollama service running on port 11434
|
||||
- Blockchain node accessible (local: 19000 or remote: aitbc.keisanki.net/rpc)
|
||||
- Home directory wallets configured
|
||||
|
||||
### Configuration
|
||||
```bash
|
||||
python3 cli/test_ollama_gpu_provider.py --url http://127.0.0.1:18000 --prompt "hello"
|
||||
# Verify services
|
||||
./scripts/aitbc-cli.sh health
|
||||
curl -s http://localhost:11434/api/tags
|
||||
systemctl status aitbc-host-gpu-miner.service
|
||||
```
|
||||
|
||||
## Expected Outcome
|
||||
## Test Options
|
||||
|
||||
- Job reaches `COMPLETED`
|
||||
- Output returned from Ollama
|
||||
- Receipt present with a `receipt_id`
|
||||
### Option 1: Basic API Test (No Payment)
|
||||
```bash
|
||||
# Simple test without blockchain
|
||||
python3 cli/test_ollama_gpu_provider.py \
|
||||
--url http://127.0.0.1:18000 \
|
||||
--prompt "What is the capital of France?" \
|
||||
--model llama3.2:latest
|
||||
```
|
||||
|
||||
## Notes
|
||||
### Option 2: Complete Workflow with Home Directory Users
|
||||
```bash
|
||||
# Full test with payment and blockchain
|
||||
cd /home/oib/windsurf/aitbc/home
|
||||
python3 test_ollama_blockchain.py
|
||||
```
|
||||
|
||||
- Use `--timeout` to allow longer runs for large models.
|
||||
- If the receipt is missing, verify `receipt_signing_key_hex` is set and restart the coordinator.
|
||||
### Option 3: Manual Step-by-Step
|
||||
```bash
|
||||
# 1. Submit job
|
||||
cd /home/oib/windsurf/aitbc/home
|
||||
job_id=$(../cli/client.py submit inference \
|
||||
--prompt "What is the capital of France?" \
|
||||
--model llama3.2:latest | grep "Job ID" | awk '{print $3}')
|
||||
|
||||
# 2. Monitor progress
|
||||
watch -n 2 "../cli/client.py status $job_id"
|
||||
|
||||
# 3. Get result and receipt
|
||||
curl -H "X-Api-Key: REDACTED_CLIENT_KEY" \
|
||||
"http://127.0.0.1:18000/v1/jobs/$job_id/result" | python3 -m json.tool
|
||||
|
||||
# 4. Process payment (manual)
|
||||
miner_addr=$(cd miner && python3 wallet.py address | grep Address | awk '{print $3}')
|
||||
amount=0.05 # Based on receipt
|
||||
cd client && python3 wallet.py send $amount $miner_addr "Payment for job $job_id"
|
||||
|
||||
# 5. Record earnings
|
||||
cd ../miner && python3 wallet.py earn $amount --job $job_id --desc "Inference job"
|
||||
|
||||
# 6. Check blockchain
|
||||
curl -s "http://aitbc.keisanki.net/rpc/transactions" | \
|
||||
python3 -c "import sys, json; data=json.load(sys.stdin); \
|
||||
[print(f\"TX: {t['tx_hash']} - Block: {t['block_height']}\") \
|
||||
for t in data.get('transactions', []) \
|
||||
if 'receipt_id' in str(t.get('payload', {}))]"
|
||||
```
|
||||
|
||||
## Expected Results
|
||||
|
||||
| Step | Expected Output | Verification |
|
||||
|------|----------------|--------------|
|
||||
| Job Submission | Job ID returned, state = QUEUED | `client.py status` shows job |
|
||||
| Processing | State → RUNNING, miner assigned | Miner logs show job pickup |
|
||||
| Completion | State = COMPLETED, output received | Result endpoint returns data |
|
||||
| Receipt | Generated with price > 0 | Receipt has valid signature |
|
||||
| Payment | Client balance ↓, miner ↑ | Wallet balances update |
|
||||
| Blockchain | Transaction recorded | TX hash searchable |
|
||||
|
||||
## Monitoring Commands
|
||||
|
||||
```bash
|
||||
# Real-time miner logs
|
||||
sudo journalctl -u aitbc-host-gpu-miner.service -f
|
||||
|
||||
# Recent receipts
|
||||
curl -H "X-Api-Key: REDACTED_CLIENT_KEY" \
|
||||
http://127.0.0.1:18000/v1/explorer/receipts?limit=5
|
||||
|
||||
# Wallet balances
|
||||
cd /home/oib/windsurf/aitbc/home && \
|
||||
echo "Client:" && cd client && python3 wallet.py balance && \
|
||||
echo "Miner:" && cd ../miner && python3 wallet.py balance
|
||||
|
||||
# Blockchain transactions
|
||||
curl -s http://aitbc.keisanki.net/rpc/transactions | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Job Stuck in QUEUED
|
||||
```bash
|
||||
# Check miner service
|
||||
systemctl status aitbc-host-gpu-miner.service
|
||||
|
||||
# Restart if needed
|
||||
sudo systemctl restart aitbc-host-gpu-miner.service
|
||||
|
||||
# Check miner registration
|
||||
curl -H "X-Api-Key: REDACTED_ADMIN_KEY" \
|
||||
http://127.0.0.1:18000/v1/admin/miners
|
||||
```
|
||||
|
||||
### No Receipt Generated
|
||||
```bash
|
||||
# Verify receipt signing key
|
||||
grep receipt_signing_key_hex /opt/coordinator-api/src/.env
|
||||
|
||||
# Check job result for receipt
|
||||
curl -H "X-Api-Key: REDACTED_CLIENT_KEY" \
|
||||
http://127.0.0.1:18000/v1/jobs/<job_id>/result | jq .receipt
|
||||
```
|
||||
|
||||
### Payment Issues
|
||||
```bash
|
||||
# Check wallet addresses
|
||||
cd /home/oib/windsurf/aitbc/home/client && python3 wallet.py address
|
||||
cd /home/oib/windsurf/aitbc/home/miner && python3 wallet.py address
|
||||
|
||||
# Verify transaction
|
||||
python3 wallet.py transactions
|
||||
```
|
||||
|
||||
### Blockchain Not Recording
|
||||
```bash
|
||||
# Check node availability
|
||||
curl -s http://aitbc.keisanki.net/rpc/health
|
||||
|
||||
# Search for receipt
|
||||
curl -s "http://aitbc.keisanki.net/rpc/transactions" | \
|
||||
grep <receipt_id>
|
||||
```
|
||||
|
||||
## Test Data Examples
|
||||
|
||||
### Sample Job Result
|
||||
```json
|
||||
{
|
||||
"result": {
|
||||
"output": "The capital of France is Paris.",
|
||||
"model": "llama3.2:latest",
|
||||
"tokens_processed": 8,
|
||||
"execution_time": 0.52,
|
||||
"gpu_used": true
|
||||
},
|
||||
"receipt": {
|
||||
"receipt_id": "8c4db70a1d413188681e003f0de7342f",
|
||||
"units": 2.603,
|
||||
"unit_price": 0.02,
|
||||
"price": 0.05206
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Sample Blockchain Transaction
|
||||
```json
|
||||
{
|
||||
"tx_hash": "0xabc123...",
|
||||
"block_height": 12345,
|
||||
"sender": "aitbc18f75b7eb7e2ecc7567b6",
|
||||
"recipient": "aitbc1721d5bf8c0005ded6704",
|
||||
"amount": 0.05206,
|
||||
"payload": {
|
||||
"receipt_id": "8c4db70a1d413188681e003f0de7342f"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with CI/CD
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Run Ollama GPU Provider Test
|
||||
run: |
|
||||
cd /home/oib/windsurf/aitbc/home
|
||||
python3 test_ollama_blockchain.py --timeout 300
|
||||
```
|
||||
|
||||
## Related Files
|
||||
|
||||
- `/home/oib/windsurf/aitbc/home/test_ollama_blockchain.py` - Complete test script
|
||||
- `/home/oib/windsurf/aitbc/cli/test_ollama_gpu_provider.py` - Basic API test
|
||||
- `/home/oib/windsurf/aitbc/.windsurf/skills/blockchain-operations/` - Blockchain management
|
||||
- `/home/oib/windsurf/aitbc/docs/infrastructure.md` - Infrastructure details
|
||||
|
||||
97
.windsurf/workflows/issue-management.md
Normal file
97
.windsurf/workflows/issue-management.md
Normal file
@@ -0,0 +1,97 @@
|
||||
---
|
||||
description: Workflow for managing and documenting issues from current to resolved
|
||||
---
|
||||
|
||||
# Issue Management Workflow
|
||||
|
||||
This workflow handles the lifecycle of issues from identification to resolution and archival.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
// turbo
|
||||
- Ensure you have write access to the docs directory
|
||||
- Check if the issue is already tracked in docs/currentissue.md
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Identify New Issue
|
||||
```bash
|
||||
# Check if currentissue.md already exists and has content
|
||||
cat /home/oib/windsurf/aitbc/docs/currentissue.md
|
||||
```
|
||||
|
||||
### 2. Document Issue in currentissue.md
|
||||
If tracking a new issue:
|
||||
- Add section with clear, descriptive title
|
||||
- Include date, status, description
|
||||
- List affected components
|
||||
- Document attempted fixes
|
||||
- Update status regularly
|
||||
|
||||
### 3. Monitor Progress
|
||||
- Update the issue status as work progresses
|
||||
- Add resolution details when fixed
|
||||
- Include code changes, configuration updates, etc.
|
||||
|
||||
### 4. When Issue is Resolved
|
||||
```bash
|
||||
# Move to issues folder with machine-readable name
|
||||
mv /home/oib/windsurf/aitbc/docs/currentissue.md \
|
||||
/home/oib/windsurf/aitbc/docs/issues/YYYY-MM-DD_brief-description.md
|
||||
|
||||
# Example:
|
||||
# mv docs/currentissue.md docs/issues/2026-01-29_cross-site-sync-resolved.md
|
||||
```
|
||||
|
||||
### 5. Create New Empty currentissue.md
|
||||
```bash
|
||||
# Create fresh currentissue.md
|
||||
cat > /home/oib/windsurf/aitbc/docs/currentissue.md << 'EOF'
|
||||
# Current Issues
|
||||
|
||||
*No current issues to report.*
|
||||
|
||||
---
|
||||
|
||||
## Usage Guidelines
|
||||
|
||||
When tracking a new issue:
|
||||
1. Add a new section with a descriptive title
|
||||
2. Include the date and current status
|
||||
3. Describe the issue, affected components, and any fixes attempted
|
||||
4. Update status as progress is made
|
||||
5. Once resolved, move this file to `docs/issues/` with a machine-readable name
|
||||
|
||||
## Recent Resolved Issues
|
||||
|
||||
See `docs/issues/` for resolved issues and their solutions.
|
||||
EOF
|
||||
```
|
||||
|
||||
## Naming Convention for Archived Issues
|
||||
|
||||
Use format: `YYYY-MM-DD_brief-description.md`
|
||||
- Date: Year-Month-Day of resolution
|
||||
- Description: Brief, lowercase, hyphen-separated summary
|
||||
- Examples:
|
||||
- `2026-01-29_cross-site-sync-resolved.md`
|
||||
- `2026-01-15_pytest-warnings-fixed.md`
|
||||
- `2026-01-10_database-migration-issue.md`
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **For Complex Issues**: Use `docs/currentissue.md` as the central tracking document
|
||||
2. **Regular Updates**: Update status daily for active issues
|
||||
3. **Detailed Resolution**: Document root cause and solution clearly
|
||||
4. **Cross-References**: Link to related code changes, PRs, or documentation
|
||||
5. **Archive Promptly**: Move resolved issues within 24 hours of resolution
|
||||
|
||||
## Integration with Other Workflows
|
||||
|
||||
- Use with `/docs` workflow to keep documentation current
|
||||
- Reference resolved issues in `docs/done.md`
|
||||
- Link technical details in `docs/reports/` as needed
|
||||
|
||||
## Memory Aid
|
||||
|
||||
Remember: For hard-to-track or complex issues spanning multiple components, always use `docs/currentissue.md` as the single source of truth for current status and resolution progress.
|
||||
116
.windsurf/workflows/ollama-gpu-test.md
Normal file
116
.windsurf/workflows/ollama-gpu-test.md
Normal file
@@ -0,0 +1,116 @@
|
||||
---
|
||||
description: Complete Ollama GPU provider test workflow from client submission to blockchain recording
|
||||
---
|
||||
|
||||
# Ollama GPU Provider Test Workflow
|
||||
|
||||
This workflow executes the complete end-to-end test for Ollama GPU inference jobs, including payment processing and blockchain transaction recording.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
// turbo
|
||||
- Ensure all services are running: coordinator, GPU miner, Ollama, blockchain node
|
||||
- Verify home directory wallets are configured
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Environment Check
|
||||
```bash
|
||||
# Check service health
|
||||
./scripts/aitbc-cli.sh health
|
||||
curl -s http://localhost:11434/api/tags
|
||||
systemctl is-active aitbc-host-gpu-miner.service
|
||||
```
|
||||
|
||||
### 2. Run Complete Test
|
||||
```bash
|
||||
# Execute the full workflow test
|
||||
cd /home/oib/windsurf/aitbc/home
|
||||
python3 test_ollama_blockchain.py
|
||||
```
|
||||
|
||||
### 3. Verify Results
|
||||
The test will display:
|
||||
- Initial wallet balances
|
||||
- Job submission and ID
|
||||
- Real-time job progress
|
||||
- Inference result from Ollama
|
||||
- Receipt details with pricing
|
||||
- Payment confirmation
|
||||
- Final wallet balances
|
||||
- Blockchain transaction status
|
||||
|
||||
### 4. Manual Verification (Optional)
|
||||
```bash
|
||||
# Check recent receipts
|
||||
curl -H "X-Api-Key: REDACTED_CLIENT_KEY" \
|
||||
http://127.0.0.1:18000/v1/explorer/receipts?limit=3
|
||||
|
||||
# Verify blockchain transaction
|
||||
curl -s http://aitbc.keisanki.net/rpc/transactions | \
|
||||
python3 -c "import sys, json; data=json.load(sys.stdin); \
|
||||
[print(f\"TX: {t['tx_hash']} - Block: {t['block_height']}\") \
|
||||
for t in data.get('transactions', [])[-5:]]"
|
||||
```
|
||||
|
||||
## Expected Output
|
||||
|
||||
```
|
||||
🚀 Ollama GPU Provider Test with Home Directory Users
|
||||
============================================================
|
||||
|
||||
💰 Initial Wallet Balances:
|
||||
----------------------------------------
|
||||
Client: 9365.0 AITBC
|
||||
Miner: 1525.0 AITBC
|
||||
|
||||
📤 Submitting Inference Job:
|
||||
----------------------------------------
|
||||
Prompt: What is the capital of France?
|
||||
Model: llama3.2:latest
|
||||
✅ Job submitted: <job_id>
|
||||
|
||||
⏳ Monitoring Job Progress:
|
||||
----------------------------------------
|
||||
State: QUEUED
|
||||
State: RUNNING
|
||||
State: COMPLETED
|
||||
|
||||
📊 Job Result:
|
||||
----------------------------------------
|
||||
Output: The capital of France is Paris.
|
||||
|
||||
🧾 Receipt Information:
|
||||
Receipt ID: <receipt_id>
|
||||
Provider: REDACTED_MINER_KEY
|
||||
Units: <gpu_seconds> gpu_seconds
|
||||
Unit Price: 0.02 AITBC
|
||||
Total Price: <price> AITBC
|
||||
|
||||
⛓️ Checking Blockchain:
|
||||
----------------------------------------
|
||||
✅ Transaction found on blockchain!
|
||||
TX Hash: <tx_hash>
|
||||
Block: <block_height>
|
||||
|
||||
💰 Final Wallet Balances:
|
||||
----------------------------------------
|
||||
Client: <new_balance> AITBC
|
||||
Miner: <new_balance> AITBC
|
||||
|
||||
✅ Test completed successfully!
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If the test fails:
|
||||
1. Check GPU miner service status
|
||||
2. Verify Ollama is running
|
||||
3. Ensure coordinator API is accessible
|
||||
4. Check wallet configurations
|
||||
5. Verify blockchain node connectivity
|
||||
|
||||
## Related Skills
|
||||
|
||||
- ollama-gpu-provider - Detailed test documentation
|
||||
- blockchain-operations - Blockchain node management
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Fix transaction block foreign key
|
||||
|
||||
Revision ID: fix_transaction_block_foreign_key
|
||||
Revises:
|
||||
Create Date: 2026-01-29 12:45:00.000000
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'fix_transaction_block_foreign_key'
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
# Drop existing foreign key constraint (unnamed, SQLite auto-generated)
|
||||
# In SQLite, we need to recreate the table
|
||||
|
||||
# Create new transaction table with correct foreign key
|
||||
op.execute("""
|
||||
CREATE TABLE transaction_new (
|
||||
id INTEGER NOT NULL PRIMARY KEY,
|
||||
tx_hash VARCHAR NOT NULL,
|
||||
block_height INTEGER,
|
||||
sender VARCHAR NOT NULL,
|
||||
recipient VARCHAR NOT NULL,
|
||||
payload JSON NOT NULL,
|
||||
created_at DATETIME NOT NULL,
|
||||
FOREIGN KEY(block_height) REFERENCES block(id)
|
||||
)
|
||||
""")
|
||||
|
||||
# Copy data from old table
|
||||
op.execute("""
|
||||
INSERT INTO transaction_new (id, tx_hash, block_height, sender, recipient, payload, created_at)
|
||||
SELECT id, tx_hash, block_height, sender, recipient, payload, created_at FROM transaction
|
||||
""")
|
||||
|
||||
# Drop old table and rename new one
|
||||
op.execute("DROP TABLE transaction")
|
||||
op.execute("ALTER TABLE transaction_new RENAME TO transaction")
|
||||
|
||||
# Recreate indexes
|
||||
op.execute("CREATE UNIQUE INDEX ix_transaction_tx_hash ON transaction (tx_hash)")
|
||||
op.execute("CREATE INDEX ix_transaction_block_height ON transaction (block_height)")
|
||||
op.execute("CREATE INDEX ix_transaction_created_at ON transaction (created_at)")
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Revert back to referencing block.height
|
||||
|
||||
# Create new transaction table with old foreign key
|
||||
op.execute("""
|
||||
CREATE TABLE transaction_new (
|
||||
id INTEGER NOT NULL PRIMARY KEY,
|
||||
tx_hash VARCHAR NOT NULL,
|
||||
block_height INTEGER,
|
||||
sender VARCHAR NOT NULL,
|
||||
recipient VARCHAR NOT NULL,
|
||||
payload JSON NOT NULL,
|
||||
created_at DATETIME NOT NULL,
|
||||
FOREIGN KEY(block_height) REFERENCES block(height)
|
||||
)
|
||||
""")
|
||||
|
||||
# Copy data from old table
|
||||
op.execute("""
|
||||
INSERT INTO transaction_new (id, tx_hash, block_height, sender, recipient, payload, created_at)
|
||||
SELECT id, tx_hash, block_height, sender, recipient, payload, created_at FROM transaction
|
||||
""")
|
||||
|
||||
# Drop old table and rename new one
|
||||
op.execute("DROP TABLE transaction")
|
||||
op.execute("ALTER TABLE transaction_new RENAME TO transaction")
|
||||
|
||||
# Recreate indexes
|
||||
op.execute("CREATE UNIQUE INDEX ix_transaction_tx_hash ON transaction (tx_hash)")
|
||||
op.execute("CREATE INDEX ix_transaction_block_height ON transaction (block_height)")
|
||||
op.execute("CREATE INDEX ix_transaction_created_at ON transaction (created_at)")
|
||||
@@ -11,7 +11,7 @@ from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
# Configuration
|
||||
DEFAULT_COORDINATOR = "http://127.0.0.1:8000"
|
||||
DEFAULT_COORDINATOR = "http://127.0.0.1:18000"
|
||||
DEFAULT_API_KEY = "REDACTED_CLIENT_KEY"
|
||||
|
||||
class AITBCClient:
|
||||
|
||||
258
cli/test_ollama_blockchain.py
Executable file
258
cli/test_ollama_blockchain.py
Executable file
@@ -0,0 +1,258 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Ollama GPU Provider Test with Blockchain Verification
|
||||
Submits an inference job and verifies the complete flow:
|
||||
- Job submission to coordinator
|
||||
- Processing by GPU miner
|
||||
- Receipt generation
|
||||
- Blockchain transaction recording
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
from typing import Optional
|
||||
import json
|
||||
|
||||
import httpx
|
||||
|
||||
# Configuration
|
||||
DEFAULT_COORDINATOR = "http://127.0.0.1:18000"
|
||||
DEFAULT_BLOCKCHAIN = "http://127.0.0.1:19000"
|
||||
DEFAULT_API_KEY = "REDACTED_CLIENT_KEY"
|
||||
DEFAULT_PROMPT = "What is the capital of France?"
|
||||
DEFAULT_MODEL = "llama3.2:latest"
|
||||
DEFAULT_TIMEOUT = 180
|
||||
POLL_INTERVAL = 3
|
||||
|
||||
|
||||
def submit_job(client: httpx.Client, base_url: str, api_key: str, prompt: str, model: str) -> Optional[str]:
|
||||
"""Submit an inference job to the coordinator"""
|
||||
payload = {
|
||||
"payload": {
|
||||
"type": "inference",
|
||||
"prompt": prompt,
|
||||
"parameters": {
|
||||
"prompt": prompt,
|
||||
"model": model,
|
||||
"stream": False
|
||||
},
|
||||
},
|
||||
"ttl_seconds": 900,
|
||||
}
|
||||
response = client.post(
|
||||
f"{base_url}/v1/jobs",
|
||||
headers={"X-Api-Key": api_key, "Content-Type": "application/json"},
|
||||
json=payload,
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code != 201:
|
||||
print(f"❌ Job submission failed: {response.status_code} {response.text}")
|
||||
return None
|
||||
return response.json().get("job_id")
|
||||
|
||||
|
||||
def fetch_status(client: httpx.Client, base_url: str, api_key: str, job_id: str) -> Optional[dict]:
|
||||
"""Fetch job status from coordinator"""
|
||||
response = client.get(
|
||||
f"{base_url}/v1/jobs/{job_id}",
|
||||
headers={"X-Api-Key": api_key},
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
print(f"❌ Status check failed: {response.status_code} {response.text}")
|
||||
return None
|
||||
return response.json()
|
||||
|
||||
|
||||
def fetch_result(client: httpx.Client, base_url: str, api_key: str, job_id: str) -> Optional[dict]:
|
||||
"""Fetch job result from coordinator"""
|
||||
response = client.get(
|
||||
f"{base_url}/v1/jobs/{job_id}/result",
|
||||
headers={"X-Api-Key": api_key},
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
print(f"❌ Result fetch failed: {response.status_code} {response.text}")
|
||||
return None
|
||||
return response.json()
|
||||
|
||||
|
||||
def fetch_receipt(client: httpx.Client, base_url: str, api_key: str, job_id: str) -> Optional[dict]:
|
||||
"""Fetch job receipt from coordinator"""
|
||||
response = client.get(
|
||||
f"{base_url}/v1/jobs/{job_id}/receipt",
|
||||
headers={"X-Api-Key": api_key},
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
print(f"❌ Receipt fetch failed: {response.status_code} {response.text}")
|
||||
return None
|
||||
return response.json()
|
||||
|
||||
|
||||
def check_blockchain_transaction(client: httpx.Client, blockchain_url: str, receipt_id: str) -> Optional[dict]:
|
||||
"""Check if receipt is recorded on blockchain"""
|
||||
# Search for transaction by receipt ID
|
||||
response = client.get(
|
||||
f"{blockchain_url}/rpc/transactions/search",
|
||||
params={"receipt_id": receipt_id},
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
print(f"⚠️ Blockchain search failed: {response.status_code}")
|
||||
return None
|
||||
|
||||
transactions = response.json().get("transactions", [])
|
||||
if transactions:
|
||||
return transactions[0] # Return the first matching transaction
|
||||
return None
|
||||
|
||||
|
||||
def get_miner_info(client: httpx.Client, base_url: str, api_key: str) -> Optional[dict]:
|
||||
"""Get registered miner information"""
|
||||
response = client.get(
|
||||
f"{base_url}/v1/admin/miners",
|
||||
headers={"X-Api-Key": api_key},
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
print(f"⚠️ Could not fetch miner info: {response.status_code}")
|
||||
return None
|
||||
|
||||
data = response.json()
|
||||
# Handle different response formats
|
||||
if isinstance(data, list):
|
||||
return data[0] if data else None
|
||||
elif isinstance(data, dict):
|
||||
if 'miners' in data:
|
||||
miners = data['miners']
|
||||
return miners[0] if miners else None
|
||||
elif 'items' in data:
|
||||
items = data['items']
|
||||
return items[0] if items else None
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Ollama GPU provider with blockchain verification")
|
||||
parser.add_argument("--coordinator-url", default=DEFAULT_COORDINATOR, help="Coordinator base URL")
|
||||
parser.add_argument("--blockchain-url", default=DEFAULT_BLOCKCHAIN, help="Blockchain node URL")
|
||||
parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="Client API key")
|
||||
parser.add_argument("--prompt", default=DEFAULT_PROMPT, help="Prompt to send")
|
||||
parser.add_argument("--model", default=DEFAULT_MODEL, help="Model to use")
|
||||
parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT, help="Timeout in seconds")
|
||||
args = parser.parse_args()
|
||||
|
||||
print("🚀 Starting Ollama GPU Provider Test with Blockchain Verification")
|
||||
print("=" * 60)
|
||||
|
||||
# Check miner registration
|
||||
print("\n📋 Checking miner registration...")
|
||||
with httpx.Client() as client:
|
||||
miner_info = get_miner_info(client, args.coordinator_url, "REDACTED_ADMIN_KEY")
|
||||
if miner_info:
|
||||
print(f"✅ Found registered miner: {miner_info.get('miner_id')}")
|
||||
print(f" Status: {miner_info.get('status')}")
|
||||
print(f" Last seen: {miner_info.get('last_seen')}")
|
||||
else:
|
||||
print("⚠️ No miners registered. Job may not be processed.")
|
||||
|
||||
# Submit job
|
||||
print(f"\n📤 Submitting inference job...")
|
||||
print(f" Prompt: {args.prompt}")
|
||||
print(f" Model: {args.model}")
|
||||
|
||||
with httpx.Client() as client:
|
||||
job_id = submit_job(client, args.coordinator_url, args.api_key, args.prompt, args.model)
|
||||
if not job_id:
|
||||
return 1
|
||||
print(f"✅ Job submitted successfully: {job_id}")
|
||||
|
||||
# Monitor job progress
|
||||
print(f"\n⏳ Monitoring job progress...")
|
||||
deadline = time.time() + args.timeout
|
||||
status = None
|
||||
|
||||
while time.time() < deadline:
|
||||
status = fetch_status(client, args.coordinator_url, args.api_key, job_id)
|
||||
if not status:
|
||||
return 1
|
||||
|
||||
state = status.get("state")
|
||||
assigned_miner = status.get("assigned_miner_id", "None")
|
||||
|
||||
print(f" State: {state} | Miner: {assigned_miner}")
|
||||
|
||||
if state == "COMPLETED":
|
||||
break
|
||||
if state in {"FAILED", "CANCELED", "EXPIRED"}:
|
||||
print(f"❌ Job ended in state: {state}")
|
||||
if status.get("error"):
|
||||
print(f" Error: {status['error']}")
|
||||
return 1
|
||||
time.sleep(POLL_INTERVAL)
|
||||
|
||||
if not status or status.get("state") != "COMPLETED":
|
||||
print("❌ Job did not complete within timeout")
|
||||
return 1
|
||||
|
||||
# Fetch result and receipt
|
||||
print(f"\n📊 Fetching job results...")
|
||||
result = fetch_result(client, args.coordinator_url, args.api_key, job_id)
|
||||
if result is None:
|
||||
return 1
|
||||
|
||||
receipt = fetch_receipt(client, args.coordinator_url, args.api_key, job_id)
|
||||
if receipt is None:
|
||||
print("⚠️ No receipt found (payment may not be processed)")
|
||||
receipt = {}
|
||||
|
||||
# Display results
|
||||
payload = result.get("result") or {}
|
||||
output = payload.get("output", "No output")
|
||||
|
||||
print(f"\n✅ Job completed successfully!")
|
||||
print(f"📝 Output: {output[:200]}{'...' if len(output) > 200 else ''}")
|
||||
|
||||
if receipt:
|
||||
print(f"\n🧾 Receipt Information:")
|
||||
print(f" Receipt ID: {receipt.get('receipt_id')}")
|
||||
print(f" Provider: {receipt.get('provider')}")
|
||||
print(f" Units: {receipt.get('units')} {receipt.get('unit_type', 'seconds')}")
|
||||
print(f" Unit Price: {receipt.get('unit_price')} AITBC")
|
||||
print(f" Total Price: {receipt.get('price')} AITBC")
|
||||
print(f" Status: {receipt.get('status')}")
|
||||
|
||||
# Check blockchain
|
||||
print(f"\n⛓️ Checking blockchain recording...")
|
||||
receipt_id = receipt.get('receipt_id')
|
||||
|
||||
with httpx.Client() as bc_client:
|
||||
tx = check_blockchain_transaction(bc_client, args.blockchain_url, receipt_id)
|
||||
|
||||
if tx:
|
||||
print(f"✅ Transaction found on blockchain!")
|
||||
print(f" TX Hash: {tx.get('tx_hash')}")
|
||||
print(f" Block: {tx.get('block_height')}")
|
||||
print(f" From: {tx.get('sender')}")
|
||||
print(f" To: {tx.get('recipient')}")
|
||||
print(f" Amount: {tx.get('amount')} AITBC")
|
||||
|
||||
# Show transaction payload
|
||||
payload = tx.get('payload', {})
|
||||
if 'receipt_id' in payload:
|
||||
print(f" Payload Receipt: {payload['receipt_id']}")
|
||||
else:
|
||||
print(f"⚠️ Transaction not yet found on blockchain")
|
||||
print(f" This may take a few moments to be mined...")
|
||||
print(f" Receipt ID: {receipt_id}")
|
||||
else:
|
||||
print(f"\n❌ No receipt generated - payment not processed")
|
||||
|
||||
print(f"\n🎉 Test completed!")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,100 +1,18 @@
|
||||
# Current Issues
|
||||
|
||||
## Cross-Site Synchronization - PARTIALLY IMPLEMENTED
|
||||
*No current issues to report.*
|
||||
|
||||
### Date
|
||||
2026-01-29
|
||||
---
|
||||
|
||||
### Status
|
||||
**PARTIALLY IMPLEMENTED** - Cross-site sync is running on all nodes. Transaction propagation works. Block import endpoint exists but has a database constraint issue with transaction import.
|
||||
## Usage Guidelines
|
||||
|
||||
### Description
|
||||
Cross-site synchronization has been integrated into all blockchain nodes. The sync module detects height differences between nodes and can propagate transactions via RPC.
|
||||
When tracking a new issue:
|
||||
1. Add a new section with a descriptive title
|
||||
2. Include the date and current status
|
||||
3. Describe the issue, affected components, and any fixes attempted
|
||||
4. Update status as progress is made
|
||||
5. Once resolved, move this file to `docs/issues/` with a machine-readable name
|
||||
|
||||
### Components Affected
|
||||
- `/src/aitbc_chain/main.py` - Main blockchain node process
|
||||
- `/src/aitbc_chain/cross_site.py` - Cross-site sync module (implemented but not integrated)
|
||||
- All three blockchain nodes (localhost Node 1 & 2, remote Node 3)
|
||||
## Recent Resolved Issues
|
||||
|
||||
### What Was Fixed
|
||||
1. **main.py integration**: Removed problematic `AbstractAsyncContextManager` type annotation and simplified the code structure
|
||||
2. **Cross-site sync module**: Integrated into all three nodes and now starts automatically
|
||||
3. **Config settings**: Added `cross_site_sync_enabled`, `cross_site_remote_endpoints`, `cross_site_poll_interval` inside the `ChainSettings` class
|
||||
4. **URL paths**: Fixed RPC endpoint paths (e.g., `/head` instead of `/rpc/head` for remote endpoints that already include `/rpc`)
|
||||
|
||||
### Current Status
|
||||
- **All nodes**: Running with cross-site sync enabled
|
||||
- **Transaction sync**: Working - mempool transactions can propagate between sites
|
||||
- **Block sync**: ✅ FULLY IMPLEMENTED - `/blocks/import` endpoint works with transactions
|
||||
- **Height difference**: Nodes maintain independent chains (local: 771153, remote: 40324)
|
||||
- **Status**: Block import with transactions now working after nginx routing fix
|
||||
|
||||
### Resolved Issues
|
||||
Block synchronization transaction import issue has been **FIXED**:
|
||||
- `/blocks/import` POST endpoint is functional and deployed on all nodes
|
||||
- Endpoint validates block hashes, parent blocks, and prevents conflicts
|
||||
- ✅ Can import blocks with and without transactions
|
||||
- ✅ Transaction data properly saved to database
|
||||
- Root cause: nginx was routing to wrong port (8082 instead of 8081)
|
||||
- Fix: Updated nginx config to route to correct blockchain-rpc-2 service
|
||||
|
||||
### Block Sync Implementation Progress
|
||||
|
||||
1. **✅ Block Import Endpoint Created** - `/src/aitbc_chain/rpc/router.py`:
|
||||
- Added `@router.post("/blocks/import")` endpoint
|
||||
- Implemented block validation (hash, parent, existence checks)
|
||||
- Added transaction and receipt import logic
|
||||
- Returns status: "imported", "exists", or error details
|
||||
|
||||
2. **✅ Cross-Site Sync Updated** - `/src/aitbc_chain/sync/cross_site.py`:
|
||||
- Modified `import_block()` to call `/rpc/blocks/import`
|
||||
- Formats block data correctly for import
|
||||
- Handles import success/failure responses
|
||||
|
||||
3. **✅ Runtime Error Fixed**:
|
||||
- Moved inline imports (hashlib, datetime, config) to top of file
|
||||
- Added proper error logging and exception handling
|
||||
- Fixed indentation issues in the function
|
||||
- Endpoint now returns proper validation responses
|
||||
|
||||
4. **✅ Transaction Import Fixed**:
|
||||
- Root cause was nginx routing to wrong port (8082 instead of 8081)
|
||||
- Updated transaction creation to use constructor with all fields
|
||||
- Server rebooted to clear all caches
|
||||
- Nginx config fixed to route to blockchain-rpc-2 on port 8081
|
||||
- Verified transaction is saved correctly with all fields
|
||||
|
||||
5. **⏳ Future Enhancements**:
|
||||
- Add proposer signature validation
|
||||
- Implement fork resolution for conflicting chains
|
||||
- Add authorized node list configuration
|
||||
|
||||
### What Works Now
|
||||
- Cross-site sync loop runs every 10 seconds
|
||||
- Remote endpoint polling detects height differences
|
||||
- Transaction propagation between sites via mempool sync
|
||||
- ✅ Block import endpoint functional with validation
|
||||
- ✅ Blocks with and without transactions can be imported between sites via RPC
|
||||
- ✅ Transaction data properly saved to database
|
||||
- Logging shows sync activity in journalctl
|
||||
|
||||
### Files Modified
|
||||
- `/src/aitbc_chain/main.py` - Added cross-site sync integration
|
||||
- `/src/aitbc_chain/cross_site.py` - Fixed URL paths, updated to use /blocks/import endpoint
|
||||
- `/src/aitbc_chain/config.py` - Added sync settings inside ChainSettings class (all nodes)
|
||||
- `/src/aitbc_chain/rpc/router.py` - Added /blocks/import POST endpoint with validation
|
||||
|
||||
### Next Steps
|
||||
1. **Monitor Block Synchronization**:
|
||||
- Watch logs for successful block imports with transactions
|
||||
- Verify cross-site sync is actively syncing block heights
|
||||
- Monitor for any validation errors or conflicts
|
||||
|
||||
2. **Future Enhancements**:
|
||||
- Add proposer signature validation for security
|
||||
- Implement fork resolution for conflicting chains
|
||||
- Add sync metrics and monitoring dashboard
|
||||
|
||||
**Status**: ✅ COMPLETE - Block import with transactions working
|
||||
**Impact**: Full cross-site block synchronization now available
|
||||
**Resolution**: Server rebooted, nginx routing fixed to port 8081
|
||||
See `docs/issues/` for resolved issues and their solutions.
|
||||
|
||||
@@ -32,7 +32,8 @@ Welcome to the AITBC developer documentation. This section contains resources fo
|
||||
|
||||
## Testing
|
||||
|
||||
- [Testing Guide](testing.md) - How to test your AITBC applications
|
||||
- [Testing Guide](../guides/WINDSURF_TESTING_GUIDE.md) - Comprehensive testing with Windsurf
|
||||
- [Test Setup](../guides/WINDSURF_TEST_SETUP.md) - Quick testing setup
|
||||
- [Test Examples](../examples/) - Test code examples
|
||||
|
||||
## Deployment
|
||||
|
||||
33
docs/done.md
33
docs/done.md
@@ -406,3 +406,36 @@ This document tracks components that have been successfully deployed and are ope
|
||||
- Updated `publicSignals` to `uint[1]` (1 public signal: receiptHash)
|
||||
- Fixed authorization checks: `require(authorizedVerifiers[msg.sender])`
|
||||
- Created `contracts/docs/ZK-VERIFICATION.md` with integration guide
|
||||
|
||||
### Recent Updates (2026-01-29)
|
||||
|
||||
- ✅ **Cross-Site Synchronization Issue Resolved**
|
||||
- Fixed database foreign key constraint in transaction/receipt tables
|
||||
- Updated import code to use block.id instead of block.height
|
||||
- Applied database migration to all nodes
|
||||
- Full details in: `docs/issues/2026-01-29_cross-site-sync-resolved.md`
|
||||
|
||||
- ✅ **Ollama GPU Provider Test Workflow**
|
||||
- Complete end-to-end test from client submission to blockchain recording
|
||||
- Created `/home/oib/windsurf/aitbc/home/test_ollama_blockchain.py`
|
||||
- Updated skill: `.windsurf/skills/ollama-gpu-provider/SKILL.md` (v2.0)
|
||||
- Created workflow: `.windsurf/workflows/ollama-gpu-test.md`
|
||||
- Verified payment flow: Client → Miner (0.05206 AITBC for inference)
|
||||
|
||||
- ✅ **Issue Management Workflow**
|
||||
- Created `.windsurf/workflows/issue-management.md`
|
||||
- Established process for tracking and archiving resolved issues
|
||||
- Moved resolved cross-site sync issue to `docs/issues/`
|
||||
|
||||
- ✅ **Pytest Warning Fixes**
|
||||
- Fixed `PytestReturnNotNoneWarning` in `test_blockchain_nodes.py`
|
||||
- Fixed `PydanticDeprecatedSince20` by migrating to V2 style validators
|
||||
- Fixed `PytestUnknownMarkWarning` by moving `pytest.ini` to project root
|
||||
|
||||
- ✅ **Directory Organization**
|
||||
- Created `docs/guides/` and moved 2 guide files from root
|
||||
- Created `docs/reports/` and moved 10 report files from root
|
||||
- Created `scripts/testing/` and moved 13 test scripts from root
|
||||
- Created `dev-utils/` and moved `aitbc-pythonpath.pth`
|
||||
- Updated `docs/files.md` with new structure
|
||||
- Fixed systemd service path for GPU miner
|
||||
|
||||
@@ -5,7 +5,7 @@ This document categorizes all files and folders in the repository by their statu
|
||||
- **Greylist (⚠️)**: Uncertain status, may need review
|
||||
- **Blacklist (❌)**: Legacy, unused, outdated, candidates for removal
|
||||
|
||||
Last updated: 2026-01-26
|
||||
Last updated: 2026-01-29
|
||||
|
||||
---
|
||||
|
||||
@@ -32,6 +32,7 @@ Last updated: 2026-01-26
|
||||
| `scripts/deploy/` | ✅ Active | Deployment scripts |
|
||||
| `scripts/service/` | ✅ Active | Service management |
|
||||
| `scripts/dev_services.sh` | ✅ Active | Local development |
|
||||
| `scripts/testing/` | ✅ Active | Test scripts (moved from root, 13 files) |
|
||||
|
||||
### Infrastructure (`infra/`, `systemd/`)
|
||||
|
||||
@@ -61,6 +62,8 @@ Last updated: 2026-01-26
|
||||
| `docs/reference/components/miner_node.md` | ✅ Active | Miner documentation |
|
||||
| `docs/reference/components/coordinator_api.md` | ✅ Active | API documentation |
|
||||
| `docs/developer/integration/skills-framework.md` | ✅ Active | Skills documentation |
|
||||
| `docs/guides/` | ✅ Active | Development guides (moved from root) |
|
||||
| `docs/reports/` | ✅ Active | Generated reports (moved from root) |
|
||||
|
||||
### Cascade Skills (`.windsurf/`)
|
||||
|
||||
@@ -94,19 +97,32 @@ Last updated: 2026-01-26
|
||||
|------|--------|-------|
|
||||
| `plugins/ollama/` | ✅ Active | Ollama integration |
|
||||
|
||||
### Development Utilities (`dev-utils/`)
|
||||
|
||||
| Path | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| `dev-utils/` | ✅ Active | Development utilities (newly created) |
|
||||
| `dev-utils/aitbc-pythonpath.pth` | ✅ Active | Python path configuration |
|
||||
|
||||
### Data Directory (`data/`)
|
||||
|
||||
| Path | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| `data/` | ✅ Active | Runtime data directory (gitignored) |
|
||||
| `data/coordinator.db` | ⚠️ Runtime | SQLite database, moved from root |
|
||||
|
||||
### Root Files
|
||||
|
||||
| Path | Status | Notes |
|
||||
|------|--------|-------|
|
||||
| `README.md` | ✅ Active | Project readme |
|
||||
| `README.md` | ✅ Active | Project readme, updated with new structure |
|
||||
| `LICENSE` | ✅ Active | License file |
|
||||
| `.gitignore` | ✅ Active | Recently updated (145 lines) |
|
||||
| `pyproject.toml` | ✅ Active | Python project config |
|
||||
| `.editorconfig` | ✅ Active | Editor config |
|
||||
| `INTEGRATION_TEST_FIXES.md` | ✅ Active | Integration test fixes documentation |
|
||||
| `INTEGRATION_TEST_UPDATES.md` | ✅ Active | Integration test real features implementation |
|
||||
| `SKIPPED_TESTS_ROADMAP.md` | ✅ Active | Skipped tests roadmap status |
|
||||
| `TEST_FIXES_COMPLETE.md` | ✅ Active | Complete test fixes summary |
|
||||
| `pytest.ini` | ✅ Active | Pytest configuration with custom markers |
|
||||
| `CLEANUP_SUMMARY.md` | ✅ Active | Documentation of directory cleanup |
|
||||
| `test_block_import.py` | ⚠️ Duplicate | Recreated in root (exists in scripts/testing/) |
|
||||
|
||||
---
|
||||
|
||||
@@ -283,6 +299,15 @@ These empty folders are intentional scaffolding for planned future work per the
|
||||
- Q2 2026: Infrastructure (`infra/terraform/`, `infra/helm/`)
|
||||
- Q2 2026: Pool Hub components
|
||||
|
||||
5. **Directory Organization (2026-01-29)**:
|
||||
- ✅ Created `docs/guides/` and moved 2 guide files from root
|
||||
- ✅ Created `docs/reports/` and moved 10 report files from root
|
||||
- ✅ Created `scripts/testing/` and moved 13 test scripts from root
|
||||
- ✅ Created `dev-utils/` and moved `aitbc-pythonpath.pth`
|
||||
- ✅ Moved `coordinator.db` to `data/` directory
|
||||
- ✅ Updated README.md with new structure
|
||||
- ✅ Created index README files for new directories
|
||||
|
||||
---
|
||||
|
||||
## Folder Structure Recommendation
|
||||
@@ -297,13 +322,18 @@ aitbc/
|
||||
│ └── zk-circuits/ # ✅ Keep
|
||||
├── cli/ # ✅ CLI tools
|
||||
├── docs/ # ✅ Markdown documentation
|
||||
│ ├── guides/ # Development guides
|
||||
│ └── reports/ # Generated reports
|
||||
├── infra/ # ✅ Infrastructure configs
|
||||
├── packages/ # ✅ Keep (aitbc-crypto, aitbc-sdk, aitbc-token)
|
||||
├── plugins/ # ✅ Keep (ollama)
|
||||
├── scripts/ # ✅ Keep - organized
|
||||
│ └── testing/ # Test scripts
|
||||
├── systemd/ # ✅ Keep
|
||||
├── tests/ # ✅ Keep (e2e, integration, unit, security, load)
|
||||
├── website/ # ✅ Keep
|
||||
├── dev-utils/ # ✅ Development utilities
|
||||
├── data/ # ✅ Runtime data (gitignored)
|
||||
└── .windsurf/ # ✅ Keep
|
||||
```
|
||||
|
||||
|
||||
@@ -26,9 +26,14 @@ This guide explains how to use Windsurf's integrated testing features with the A
|
||||
|
||||
### 4. Pytest Configuration
|
||||
- ✅ `pyproject.toml` - Main configuration with markers
|
||||
- ✅ `tests/pytest.ini` - Simplified for discovery
|
||||
- ✅ `pytest.ini` - Moved to project root with custom markers
|
||||
- ✅ `tests/conftest.py` - Fixtures with fallback mocks
|
||||
|
||||
### 5. Test Scripts (2026-01-29)
|
||||
- ✅ `scripts/testing/` - All test scripts moved here
|
||||
- ✅ `test_ollama_blockchain.py` - Complete GPU provider test
|
||||
- ✅ `test_block_import.py` - Blockchain block import testing
|
||||
|
||||
## 🚀 How to Use
|
||||
|
||||
### Test Discovery
|
||||
|
||||
108
docs/issues/2026-01-29_cross-site-sync-resolved.md
Normal file
108
docs/issues/2026-01-29_cross-site-sync-resolved.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Current Issues
|
||||
|
||||
## Cross-Site Synchronization - ✅ RESOLVED
|
||||
|
||||
### Date
|
||||
2026-01-29
|
||||
|
||||
### Status
|
||||
**FULLY IMPLEMENTED** - Cross-site sync is running on all nodes. Transaction propagation works. Block import endpoint works with transactions after database foreign key fix.
|
||||
|
||||
### Description
|
||||
Cross-site synchronization has been integrated into all blockchain nodes. The sync module detects height differences between nodes and can propagate transactions via RPC.
|
||||
|
||||
### Components Affected
|
||||
- `/src/aitbc_chain/main.py` - Main blockchain node process
|
||||
- `/src/aitbc_chain/cross_site.py` - Cross-site sync module (implemented but not integrated)
|
||||
- All three blockchain nodes (localhost Node 1 & 2, remote Node 3)
|
||||
|
||||
### What Was Fixed
|
||||
1. **main.py integration**: Removed problematic `AbstractAsyncContextManager` type annotation and simplified the code structure
|
||||
2. **Cross-site sync module**: Integrated into all three nodes and now starts automatically
|
||||
3. **Config settings**: Added `cross_site_sync_enabled`, `cross_site_remote_endpoints`, `cross_site_poll_interval` inside the `ChainSettings` class
|
||||
4. **URL paths**: Fixed RPC endpoint paths (e.g., `/head` instead of `/rpc/head` for remote endpoints that already include `/rpc`)
|
||||
|
||||
### Current Status
|
||||
- **All nodes**: Running with cross-site sync enabled
|
||||
- **Transaction sync**: Working - mempool transactions can propagate between sites
|
||||
- **Block sync**: ✅ FULLY IMPLEMENTED - `/blocks/import` endpoint works with transactions
|
||||
- **Height difference**: Nodes maintain independent chains (local: 771153, remote: 40324)
|
||||
- **Status**: ✅ RESOLVED - Fixed database foreign key constraint issue (2026-01-29)
|
||||
|
||||
### Database Fix Applied (2026-01-29)
|
||||
- **Issue**: Transaction and receipt tables had foreign key to `block.height` instead of `block.id`
|
||||
- **Solution**:
|
||||
1. Updated database schema to reference `block.id`
|
||||
2. Fixed import code in `/src/aitbc_chain/rpc/router.py` to use `block.id`
|
||||
3. Applied migration to existing databases
|
||||
- **Result**: Block import with transactions now works correctly
|
||||
|
||||
### Resolved Issues
|
||||
Block synchronization transaction import issue has been **FIXED**:
|
||||
- `/blocks/import` POST endpoint is functional and deployed on all nodes
|
||||
- Endpoint validates block hashes, parent blocks, and prevents conflicts
|
||||
- ✅ Can import blocks with and without transactions
|
||||
- ✅ Transaction data properly saved to database
|
||||
- Root cause: nginx was routing to wrong port (8082 instead of 8081)
|
||||
- Fix: Updated nginx config to route to correct blockchain-rpc-2 service
|
||||
|
||||
### Block Sync Implementation Progress
|
||||
|
||||
1. **✅ Block Import Endpoint Created** - `/src/aitbc_chain/rpc/router.py`:
|
||||
- Added `@router.post("/blocks/import")` endpoint
|
||||
- Implemented block validation (hash, parent, existence checks)
|
||||
- Added transaction and receipt import logic
|
||||
- Returns status: "imported", "exists", or error details
|
||||
|
||||
2. **✅ Cross-Site Sync Updated** - `/src/aitbc_chain/sync/cross_site.py`:
|
||||
- Modified `import_block()` to call `/rpc/blocks/import`
|
||||
- Formats block data correctly for import
|
||||
- Handles import success/failure responses
|
||||
|
||||
3. **✅ Runtime Error Fixed**:
|
||||
- Moved inline imports (hashlib, datetime, config) to top of file
|
||||
- Added proper error logging and exception handling
|
||||
- Fixed indentation issues in the function
|
||||
- Endpoint now returns proper validation responses
|
||||
|
||||
4. **✅ Transaction Import Fixed**:
|
||||
- Root cause was nginx routing to wrong port (8082 instead of 8081)
|
||||
- Updated transaction creation to use constructor with all fields
|
||||
- Server rebooted to clear all caches
|
||||
- Nginx config fixed to route to blockchain-rpc-2 on port 8081
|
||||
- Verified transaction is saved correctly with all fields
|
||||
|
||||
5. **⏳ Future Enhancements**:
|
||||
- Add proposer signature validation
|
||||
- Implement fork resolution for conflicting chains
|
||||
- Add authorized node list configuration
|
||||
|
||||
### What Works Now
|
||||
- Cross-site sync loop runs every 10 seconds
|
||||
- Remote endpoint polling detects height differences
|
||||
- Transaction propagation between sites via mempool sync
|
||||
- ✅ Block import endpoint functional with validation
|
||||
- ✅ Blocks with and without transactions can be imported between sites via RPC
|
||||
- ✅ Transaction data properly saved to database
|
||||
- Logging shows sync activity in journalctl
|
||||
|
||||
### Files Modified
|
||||
- `/src/aitbc_chain/main.py` - Added cross-site sync integration
|
||||
- `/src/aitbc_chain/cross_site.py` - Fixed URL paths, updated to use /blocks/import endpoint
|
||||
- `/src/aitbc_chain/config.py` - Added sync settings inside ChainSettings class (all nodes)
|
||||
- `/src/aitbc_chain/rpc/router.py` - Added /blocks/import POST endpoint with validation
|
||||
|
||||
### Next Steps
|
||||
1. **Monitor Block Synchronization**:
|
||||
- Watch logs for successful block imports with transactions
|
||||
- Verify cross-site sync is actively syncing block heights
|
||||
- Monitor for any validation errors or conflicts
|
||||
|
||||
2. **Future Enhancements**:
|
||||
- Add proposer signature validation for security
|
||||
- Implement fork resolution for conflicting chains
|
||||
- Add sync metrics and monitoring dashboard
|
||||
|
||||
**Status**: ✅ COMPLETE - Block import with transactions working
|
||||
**Impact**: Full cross-site block synchronization now available
|
||||
**Resolution**: Server rebooted, nginx routing fixed to port 8081
|
||||
@@ -31,8 +31,8 @@ Welcome to the AITBC reference documentation. This section contains technical sp
|
||||
|
||||
## Project Documentation
|
||||
|
||||
- [Roadmap](roadmap.md) - Development roadmap
|
||||
- [Completed Tasks](done.md) - List of completed features
|
||||
- [Roadmap](../roadmap.md) - Development roadmap
|
||||
- [Completed Tasks](../done.md) - List of completed features
|
||||
- [Beta Release Plan](beta-release-plan.md) - Beta release planning
|
||||
|
||||
## Historical
|
||||
|
||||
@@ -731,5 +731,26 @@ Current Status: Canonical receipt schema specification moved from `protocols/rec
|
||||
| `docs/reference/specs/receipt-spec.md` finalize | Low | Q2 2026 | 🔄 Pending extensions |
|
||||
| Cross-site synchronization | High | Q1 2026 | ✅ Complete (2026-01-29) |
|
||||
|
||||
## Recent Progress (2026-01-29)
|
||||
|
||||
### Testing Infrastructure
|
||||
- **Ollama GPU Provider Test Workflow** ✅ COMPLETE
|
||||
- End-to-end test from client submission to blockchain recording
|
||||
- Payment processing verified (0.05206 AITBC for inference job)
|
||||
- Created comprehensive test script and workflow documentation
|
||||
|
||||
### Code Quality
|
||||
- **Pytest Warning Fixes** ✅ COMPLETE
|
||||
- Fixed all pytest warnings (`PytestReturnNotNoneWarning`, `PydanticDeprecatedSince20`, `PytestUnknownMarkWarning`)
|
||||
- Migrated Pydantic validators to V2 style
|
||||
- Moved `pytest.ini` to project root with proper marker configuration
|
||||
|
||||
### Project Organization
|
||||
- **Directory Cleanup** ✅ COMPLETE
|
||||
- Reorganized root files into logical directories
|
||||
- Created `docs/guides/`, `docs/reports/`, `scripts/testing/`, `dev-utils/`
|
||||
- Updated documentation to reflect new structure
|
||||
- Fixed GPU miner systemd service path
|
||||
|
||||
the canonical checklist during implementation. Mark completed tasks with ✅ and add dates or links to relevant PRs as development progresses.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"address": "aitbc18f75b7eb7e2ecc7567b6",
|
||||
"balance": 9365.0,
|
||||
"balance": 9364.94794,
|
||||
"transactions": [
|
||||
{
|
||||
"type": "earn",
|
||||
@@ -32,6 +32,12 @@
|
||||
"amount": -25.0,
|
||||
"description": "Payment for job e3b3fd7ddd684270932cfd8107771e81",
|
||||
"timestamp": "2026-01-23T15:09:41.183693"
|
||||
},
|
||||
{
|
||||
"type": "spend",
|
||||
"amount": -0.05206,
|
||||
"description": "Payment for job 44c27e1a02e74a5584691eacf9e06e29",
|
||||
"timestamp": "2026-01-29T12:56:26.896360"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-01-23T14:55:27.329386"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"address": "aitbc1721d5bf8c0005ded6704",
|
||||
"balance": 1525.0,
|
||||
"balance": 1525.05206,
|
||||
"transactions": [
|
||||
{
|
||||
"type": "earn",
|
||||
@@ -22,6 +22,13 @@
|
||||
"job_id": "e3b3fd7ddd684270932cfd8107771e81",
|
||||
"description": "Processed hello job",
|
||||
"timestamp": "2026-01-23T15:09:41.109718"
|
||||
},
|
||||
{
|
||||
"type": "earn",
|
||||
"amount": 0.05206,
|
||||
"job_id": "44c27e1a02e74a5584691eacf9e06e29",
|
||||
"description": "Inference: capital of France",
|
||||
"timestamp": "2026-01-29T12:56:42.732228"
|
||||
}
|
||||
],
|
||||
"created_at": "2026-01-23T14:55:27.329401"
|
||||
|
||||
259
home/test_ollama_blockchain.py
Executable file
259
home/test_ollama_blockchain.py
Executable file
@@ -0,0 +1,259 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Ollama GPU Provider Test with Blockchain Verification using Home Directory Users
|
||||
Tests the complete flow: Client -> Coordinator -> GPU Miner -> Receipt -> Blockchain
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import time
|
||||
import json
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
|
||||
# Add parent directories to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'cli'))
|
||||
|
||||
# Configuration
|
||||
DEFAULT_COORDINATOR = "http://127.0.0.1:18000"
|
||||
DEFAULT_BLOCKCHAIN = "http://aitbc.keisanki.net/rpc"
|
||||
DEFAULT_PROMPT = "What is the capital of France?"
|
||||
DEFAULT_MODEL = "llama3.2:latest"
|
||||
DEFAULT_TIMEOUT = 180
|
||||
POLL_INTERVAL = 3
|
||||
|
||||
|
||||
def get_wallet_balance(wallet_dir: str) -> float:
|
||||
"""Get wallet balance from home directory wallet"""
|
||||
result = subprocess.run(
|
||||
f'cd {wallet_dir} && python3 wallet.py balance',
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
if result.returncode == 0:
|
||||
for line in result.stdout.split('\n'):
|
||||
if 'Balance:' in line:
|
||||
# Extract the value after "Balance:"
|
||||
balance_part = line.split('Balance:')[1].strip()
|
||||
balance_value = balance_part.split()[0] # Get the numeric part before "AITBC"
|
||||
try:
|
||||
return float(balance_value)
|
||||
except ValueError:
|
||||
continue
|
||||
return 0.0
|
||||
|
||||
|
||||
def get_wallet_address(wallet_dir: str) -> Optional[str]:
|
||||
"""Get wallet address from home directory wallet"""
|
||||
result = subprocess.run(
|
||||
f'cd {wallet_dir} && python3 wallet.py address',
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
if result.returncode == 0:
|
||||
for line in result.stdout.split('\n'):
|
||||
if 'Address:' in line:
|
||||
return line.split()[-1]
|
||||
return None
|
||||
|
||||
|
||||
def submit_job_via_client(prompt: str, model: str) -> Optional[str]:
|
||||
"""Submit job using the CLI client"""
|
||||
cmd = f'cd ../cli && python3 client.py submit inference --prompt "{prompt}" --model {model}'
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f"❌ Job submission failed: {result.stderr}")
|
||||
return None
|
||||
|
||||
# Extract job ID
|
||||
for line in result.stdout.split('\n'):
|
||||
if "Job ID:" in line:
|
||||
return line.split()[-1]
|
||||
return None
|
||||
|
||||
|
||||
def get_job_status(job_id: str) -> Optional[str]:
|
||||
"""Get job status using CLI client"""
|
||||
result = subprocess.run(
|
||||
f'cd ../cli && python3 client.py status {job_id}',
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
return None
|
||||
|
||||
# Extract state
|
||||
for line in result.stdout.split('\n'):
|
||||
if "State:" in line:
|
||||
return line.split()[-1]
|
||||
return None
|
||||
|
||||
|
||||
def get_job_result(job_id: str) -> Optional[dict]:
|
||||
"""Get job result via API"""
|
||||
with httpx.Client() as client:
|
||||
response = client.get(
|
||||
f"{DEFAULT_COORDINATOR}/v1/jobs/{job_id}/result",
|
||||
headers={"X-Api-Key": "REDACTED_CLIENT_KEY"},
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code == 200:
|
||||
return response.json()
|
||||
return None
|
||||
|
||||
|
||||
def check_blockchain_transaction(receipt_id: str) -> Optional[dict]:
|
||||
"""Check if receipt is recorded on blockchain"""
|
||||
try:
|
||||
with httpx.Client() as client:
|
||||
# Try to get recent transactions
|
||||
response = client.get(
|
||||
f"{DEFAULT_BLOCKCHAIN}/transactions",
|
||||
timeout=10,
|
||||
)
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
transactions = data.get("transactions", data.get("items", []))
|
||||
# Look for matching receipt
|
||||
for tx in transactions:
|
||||
payload = tx.get("payload", {})
|
||||
if payload.get("receipt_id") == receipt_id:
|
||||
return tx
|
||||
return None
|
||||
except httpx.ConnectError:
|
||||
print(f"⚠️ Blockchain node not available at {DEFAULT_BLOCKCHAIN}")
|
||||
return None
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error checking blockchain: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
print("🚀 Ollama GPU Provider Test with Home Directory Users")
|
||||
print("=" * 60)
|
||||
|
||||
# Get initial balances
|
||||
print("\n💰 Initial Wallet Balances:")
|
||||
print("-" * 40)
|
||||
|
||||
client_balance = get_wallet_balance("client")
|
||||
miner_balance = get_wallet_balance("miner")
|
||||
|
||||
print(f" Client: {client_balance} AITBC")
|
||||
print(f" Miner: {miner_balance} AITBC")
|
||||
|
||||
# Submit job
|
||||
print(f"\n📤 Submitting Inference Job:")
|
||||
print("-" * 40)
|
||||
print(f" Prompt: {DEFAULT_PROMPT}")
|
||||
print(f" Model: {DEFAULT_MODEL}")
|
||||
|
||||
job_id = submit_job_via_client(DEFAULT_PROMPT, DEFAULT_MODEL)
|
||||
if not job_id:
|
||||
print("❌ Failed to submit job")
|
||||
return 1
|
||||
|
||||
print(f"✅ Job submitted: {job_id}")
|
||||
|
||||
# Monitor job progress
|
||||
print(f"\n⏳ Monitoring Job Progress:")
|
||||
print("-" * 40)
|
||||
|
||||
deadline = time.time() + DEFAULT_TIMEOUT
|
||||
|
||||
while time.time() < deadline:
|
||||
state = get_job_status(job_id)
|
||||
if not state:
|
||||
print(" ⚠️ Could not fetch status")
|
||||
time.sleep(POLL_INTERVAL)
|
||||
continue
|
||||
|
||||
print(f" State: {state}")
|
||||
|
||||
if state == "COMPLETED":
|
||||
break
|
||||
elif state in {"FAILED", "CANCELED", "EXPIRED"}:
|
||||
print(f"❌ Job ended in state: {state}")
|
||||
return 1
|
||||
|
||||
time.sleep(POLL_INTERVAL)
|
||||
|
||||
if state != "COMPLETED":
|
||||
print("❌ Job did not complete within timeout")
|
||||
return 1
|
||||
|
||||
# Get job result
|
||||
print(f"\n📊 Job Result:")
|
||||
print("-" * 40)
|
||||
|
||||
result = get_job_result(job_id)
|
||||
if result:
|
||||
output = result.get("result", {}).get("output", "No output")
|
||||
receipt = result.get("receipt")
|
||||
|
||||
print(f" Output: {output[:200]}{'...' if len(output) > 200 else ''}")
|
||||
|
||||
if receipt:
|
||||
print(f"\n🧾 Receipt Information:")
|
||||
print(f" Receipt ID: {receipt.get('receipt_id')}")
|
||||
print(f" Provider: {receipt.get('provider')}")
|
||||
print(f" Units: {receipt.get('units')} {receipt.get('unit_type', 'seconds')}")
|
||||
print(f" Unit Price: {receipt.get('unit_price')} AITBC")
|
||||
print(f" Total Price: {receipt.get('price')} AITBC")
|
||||
|
||||
# Check blockchain
|
||||
print(f"\n⛓️ Checking Blockchain:")
|
||||
print("-" * 40)
|
||||
|
||||
tx = check_blockchain_transaction(receipt.get('receipt_id'))
|
||||
if tx:
|
||||
print(f"✅ Transaction found on blockchain!")
|
||||
print(f" TX Hash: {tx.get('tx_hash')}")
|
||||
print(f" Block: {tx.get('block_height')}")
|
||||
print(f" From: {tx.get('sender')}")
|
||||
print(f" To: {tx.get('recipient')}")
|
||||
print(f" Amount: {tx.get('amount')} AITBC")
|
||||
else:
|
||||
print(f"⚠️ Transaction not yet found on blockchain")
|
||||
print(f" (May take a few moments to be mined)")
|
||||
else:
|
||||
print(f"⚠️ No receipt generated")
|
||||
else:
|
||||
print(" Could not fetch result")
|
||||
|
||||
# Show final balances
|
||||
print(f"\n💰 Final Wallet Balances:")
|
||||
print("-" * 40)
|
||||
|
||||
client_balance = get_wallet_balance("client")
|
||||
miner_balance = get_wallet_balance("miner")
|
||||
|
||||
print(f" Client: {client_balance} AITBC")
|
||||
print(f" Miner: {miner_balance} AITBC")
|
||||
|
||||
# Calculate difference
|
||||
client_diff = client_balance - get_wallet_balance("client")
|
||||
print(f"\n📈 Transaction Summary:")
|
||||
print("-" * 40)
|
||||
print(f" Client spent: {abs(client_diff):.4f} AITBC")
|
||||
print(f" Miner earned: {abs(client_diff):.4f} AITBC")
|
||||
|
||||
print(f"\n✅ Test completed successfully!")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -397,7 +397,7 @@ async def import_block(request: BlockImportRequest) -> Dict[str, Any]:
|
||||
# Create transaction using constructor with all fields
|
||||
tx = Transaction(
|
||||
tx_hash=str(tx_data.tx_hash),
|
||||
block_height=block.height,
|
||||
block_height=block.id, # Use block.id instead of block.height for foreign key
|
||||
sender=str(tx_data.sender),
|
||||
recipient=str(tx_data.recipient),
|
||||
payload=tx_data.payload if tx_data.payload else {},
|
||||
@@ -409,7 +409,7 @@ async def import_block(request: BlockImportRequest) -> Dict[str, Any]:
|
||||
# Add receipts if provided
|
||||
for receipt_data in request.receipts:
|
||||
receipt = Receipt(
|
||||
block_height=block.height,
|
||||
block_height=block.id, # Use block.id instead of block.height for foreign key
|
||||
receipt_id=receipt_data.receipt_id,
|
||||
job_id=receipt_data.job_id,
|
||||
payload=receipt_data.payload,
|
||||
|
||||
Reference in New Issue
Block a user