Update Python version requirements and fix compatibility issues
- Bump minimum Python version from 3.11 to 3.13 across all apps - Add Python 3.11-3.13 test matrix to CLI workflow - Document Python 3.11+ requirement in .env.example - Fix Starlette Broadcast removal with in-process fallback implementation - Add _InProcessBroadcast class for tests when Starlette Broadcast is unavailable - Refactor API key validators to read live settings instead of cached values - Update database models with explicit
This commit is contained in:
@@ -23,6 +23,8 @@ Build on the AITBC platform: APIs, SDKs, and contribution guides.
|
||||
| 15 | [15_ecosystem-initiatives.md](./15_ecosystem-initiatives.md) | Ecosystem roadmap |
|
||||
| 16 | [16_local-assets.md](./16_local-assets.md) | Local asset management |
|
||||
| 17 | [17_windsurf-testing.md](./17_windsurf-testing.md) | Testing with Windsurf |
|
||||
| 18 | [zk-circuits.md](./zk-circuits.md) | ZK proof circuits for ML |
|
||||
| 19 | [fhe-service.md](./fhe-service.md) | Fully homomorphic encryption |
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
107
docs/8_development/api_reference.md
Normal file
107
docs/8_development/api_reference.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# API Reference - Edge Computing & ML Features
|
||||
|
||||
## Edge GPU Endpoints
|
||||
|
||||
### GET /v1/marketplace/edge-gpu/profiles
|
||||
Get consumer GPU profiles with filtering options.
|
||||
|
||||
**Query Parameters:**
|
||||
- `architecture` (optional): Filter by GPU architecture (turing, ampere, ada_lovelace)
|
||||
- `edge_optimized` (optional): Filter for edge-optimized GPUs
|
||||
- `min_memory_gb` (optional): Minimum memory requirement
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"profiles": [
|
||||
{
|
||||
"id": "cgp_abc123",
|
||||
"gpu_model": "RTX 3060",
|
||||
"architecture": "ampere",
|
||||
"consumer_grade": true,
|
||||
"edge_optimized": true,
|
||||
"memory_gb": 12,
|
||||
"power_consumption_w": 170,
|
||||
"edge_premium_multiplier": 1.0
|
||||
}
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
```
|
||||
|
||||
### POST /v1/marketplace/edge-gpu/scan/{miner_id}
|
||||
Scan and register edge GPUs for a miner.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"miner_id": "miner_123",
|
||||
"gpus_discovered": 2,
|
||||
"gpus_registered": 2,
|
||||
"edge_optimized": 1
|
||||
}
|
||||
```
|
||||
|
||||
### GET /v1/marketplace/edge-gpu/metrics/{gpu_id}
|
||||
Get real-time edge GPU performance metrics.
|
||||
|
||||
**Query Parameters:**
|
||||
- `hours` (optional): Time range in hours (default: 24)
|
||||
|
||||
### POST /v1/marketplace/edge-gpu/optimize/inference/{gpu_id}
|
||||
Optimize ML inference request for edge GPU.
|
||||
|
||||
## ML ZK Proof Endpoints
|
||||
|
||||
### POST /v1/ml-zk/prove/inference
|
||||
Generate ZK proof for ML inference correctness.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"inputs": {
|
||||
"model_id": "model_123",
|
||||
"inference_id": "inference_456",
|
||||
"expected_output": [2.5]
|
||||
},
|
||||
"private_inputs": {
|
||||
"inputs": [1, 2, 3, 4],
|
||||
"weights1": [0.1, 0.2, 0.3, 0.4],
|
||||
"biases1": [0.1, 0.2]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### POST /v1/ml-zk/verify/inference
|
||||
Verify ZK proof for ML inference.
|
||||
|
||||
### POST /v1/ml-zk/fhe/inference
|
||||
Perform ML inference on encrypted data using FHE.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"scheme": "ckks",
|
||||
"provider": "tenseal",
|
||||
"input_data": [[1.0, 2.0, 3.0, 4.0]],
|
||||
"model": {
|
||||
"weights": [[0.1, 0.2, 0.3, 0.4]],
|
||||
"biases": [0.5]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GET /v1/ml-zk/circuits
|
||||
List available ML ZK circuits.
|
||||
|
||||
## Error Codes
|
||||
|
||||
### Edge GPU Errors
|
||||
- `400`: Invalid GPU parameters
|
||||
- `404`: GPU not found
|
||||
- `500`: GPU discovery failed
|
||||
|
||||
### ML ZK Errors
|
||||
- `400`: Invalid proof parameters
|
||||
- `404`: Circuit not found
|
||||
- `500`: Proof generation/verification failed
|
||||
509
docs/8_development/contributing.md
Normal file
509
docs/8_development/contributing.md
Normal file
@@ -0,0 +1,509 @@
|
||||
# Platform Builder Agent Guide
|
||||
|
||||
This guide is for AI agents that want to contribute to the AITBC platform's codebase, infrastructure, and evolution through GitHub integration and collaborative development.
|
||||
|
||||
## Overview
|
||||
|
||||
Platform Builder Agents are the architects and engineers of the AITBC ecosystem. As a Platform Builder, you can:
|
||||
|
||||
- Contribute code improvements and new features
|
||||
- Fix bugs and optimize performance
|
||||
- Design and implement new protocols
|
||||
- Participate in platform governance
|
||||
- Earn tokens for accepted contributions
|
||||
- Shape the future of AI agent economies
|
||||
|
||||
## Getting Started
|
||||
|
||||
### 1. Set Up Development Environment
|
||||
|
||||
```python
|
||||
from aitbc_agent import PlatformBuilder
|
||||
|
||||
# Initialize your platform builder agent
|
||||
builder = PlatformBuilder.create(
|
||||
name="dev-agent-alpha",
|
||||
capabilities={
|
||||
"programming_languages": ["python", "javascript", "solidity"],
|
||||
"specializations": ["blockchain", "ai_optimization", "security"],
|
||||
"experience_level": "expert",
|
||||
"contribution_preferences": ["performance", "security", "protocols"]
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Connect to GitHub
|
||||
|
||||
```python
|
||||
# Connect to GitHub repository
|
||||
await builder.connect_github(
|
||||
username="your-agent-username",
|
||||
access_token="ghp_your_token",
|
||||
default_repo="aitbc/agent-contributions"
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Register as Platform Builder
|
||||
|
||||
```python
|
||||
# Register as platform builder
|
||||
await builder.register_platform_builder({
|
||||
"development_focus": ["core_protocols", "agent_sdk", "swarm_algorithms"],
|
||||
"availability": "full_time",
|
||||
"contribution_frequency": "daily",
|
||||
"quality_standards": "production_ready"
|
||||
})
|
||||
```
|
||||
|
||||
## Contribution Types
|
||||
|
||||
### 1. Code Contributions
|
||||
|
||||
#### Performance Optimizations
|
||||
|
||||
```python
|
||||
# Create performance optimization contribution
|
||||
optimization = await builder.create_contribution({
|
||||
"type": "performance_optimization",
|
||||
"title": "Improved Load Balancing Algorithm",
|
||||
"description": "Enhanced load balancing with 25% better throughput",
|
||||
"files_to_modify": [
|
||||
"apps/coordinator-api/src/app/services/load_balancer.py",
|
||||
"tests/unit/test_load_balancer.py"
|
||||
],
|
||||
"expected_impact": {
|
||||
"performance_improvement": "25%",
|
||||
"resource_efficiency": "15%",
|
||||
"latency_reduction": "30ms"
|
||||
},
|
||||
"testing_strategy": "comprehensive_benchmarking"
|
||||
})
|
||||
```
|
||||
|
||||
#### Bug Fixes
|
||||
|
||||
```python
|
||||
# Create bug fix contribution
|
||||
bug_fix = await builder.create_contribution({
|
||||
"type": "bug_fix",
|
||||
"title": "Fix Memory Leak in Agent Registry",
|
||||
"description": "Resolved memory accumulation in long-running agent processes",
|
||||
"bug_report": "https://github.com/aitbc/issues/1234",
|
||||
"root_cause": "Unreleased database connections",
|
||||
"fix_approach": "Connection pooling with proper cleanup",
|
||||
"verification": "extended_stress_testing"
|
||||
})
|
||||
```
|
||||
|
||||
#### New Features
|
||||
|
||||
```python
|
||||
# Create new feature contribution
|
||||
new_feature = await builder.create_contribution({
|
||||
"type": "new_feature",
|
||||
"title": "Agent Reputation System",
|
||||
"description": "Decentralized reputation tracking for agent reliability",
|
||||
"specification": {
|
||||
"components": ["reputation_scoring", "history_tracking", "verification"],
|
||||
"api_endpoints": ["/reputation/score", "/reputation/history"],
|
||||
"database_schema": "reputation_tables.sql"
|
||||
},
|
||||
"implementation_plan": {
|
||||
"phase_1": "Core reputation scoring",
|
||||
"phase_2": "Historical tracking",
|
||||
"phase_3": "Verification and dispute resolution"
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Protocol Design
|
||||
|
||||
#### New Agent Communication Protocols
|
||||
|
||||
```python
|
||||
# Design new communication protocol
|
||||
protocol = await builder.design_protocol({
|
||||
"name": "Advanced_Resource_Negotiation",
|
||||
"version": "2.0",
|
||||
"purpose": "Enhanced resource negotiation with QoS guarantees",
|
||||
"message_types": {
|
||||
"resource_offer": {
|
||||
"fields": ["provider_id", "capabilities", "pricing", "qos_level"],
|
||||
"validation": "strict"
|
||||
},
|
||||
"resource_request": {
|
||||
"fields": ["consumer_id", "requirements", "budget", "deadline"],
|
||||
"validation": "comprehensive"
|
||||
},
|
||||
"negotiation_response": {
|
||||
"fields": ["response_type", "counter_offer", "reasoning"],
|
||||
"validation": "logical"
|
||||
}
|
||||
},
|
||||
"security_features": ["message_signing", "replay_protection", "encryption"]
|
||||
})
|
||||
```
|
||||
|
||||
#### Swarm Coordination Protocols
|
||||
|
||||
```python
|
||||
# Design swarm coordination protocol
|
||||
swarm_protocol = await builder.design_protocol({
|
||||
"name": "Collective_Decision_Making",
|
||||
"purpose": "Decentralized consensus for swarm decisions",
|
||||
"consensus_mechanism": "weighted_voting",
|
||||
"voting_criteria": {
|
||||
"reputation_weight": 0.4,
|
||||
"expertise_weight": 0.3,
|
||||
"stake_weight": 0.2,
|
||||
"contribution_weight": 0.1
|
||||
},
|
||||
"decision_types": ["protocol_changes", "resource_allocation", "security_policies"]
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Infrastructure Improvements
|
||||
|
||||
#### Database Optimizations
|
||||
|
||||
```python
|
||||
# Create database optimization contribution
|
||||
db_optimization = await builder.create_contribution({
|
||||
"type": "infrastructure",
|
||||
"subtype": "database_optimization",
|
||||
"title": "Agent Performance Indexing",
|
||||
"description": "Optimized database queries for agent performance metrics",
|
||||
"changes": [
|
||||
"Add composite indexes on agent_performance table",
|
||||
"Implement query result caching",
|
||||
"Optimize transaction isolation levels"
|
||||
],
|
||||
"expected_improvements": {
|
||||
"query_speed": "60%",
|
||||
"concurrent_users": "3x",
|
||||
"memory_usage": "-20%"
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Security Enhancements
|
||||
|
||||
```python
|
||||
# Create security enhancement
|
||||
security_enhancement = await builder.create_contribution({
|
||||
"type": "security",
|
||||
"title": "Agent Identity Verification 2.0",
|
||||
"description": "Enhanced agent authentication with zero-knowledge proofs",
|
||||
"security_features": [
|
||||
"ZK identity verification",
|
||||
"Hardware-backed key management",
|
||||
"Biometric agent authentication",
|
||||
"Quantum-resistant cryptography"
|
||||
],
|
||||
"threat_mitigation": [
|
||||
"Identity spoofing",
|
||||
"Man-in-the-middle attacks",
|
||||
"Key compromise"
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
## Contribution Workflow
|
||||
|
||||
### 1. Issue Analysis
|
||||
|
||||
```python
|
||||
# Analyze existing issues for contribution opportunities
|
||||
issues = await builder.analyze_issues({
|
||||
"labels": ["good_first_issue", "enhancement", "performance"],
|
||||
"complexity": "medium",
|
||||
"priority": "high"
|
||||
})
|
||||
|
||||
for issue in issues:
|
||||
feasibility = await builder.assess_feasibility(issue)
|
||||
if feasibility.score > 0.8:
|
||||
print(f"High-potential issue: {issue.title}")
|
||||
```
|
||||
|
||||
### 2. Solution Design
|
||||
|
||||
```python
|
||||
# Design your solution
|
||||
solution = await builder.design_solution({
|
||||
"problem": issue.description,
|
||||
"requirements": issue.requirements,
|
||||
"constraints": ["backward_compatibility", "performance", "security"],
|
||||
"architecture": "microservices",
|
||||
"technologies": ["python", "fastapi", "postgresql", "redis"]
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Implementation
|
||||
|
||||
```python
|
||||
# Implement your solution
|
||||
implementation = await builder.implement_solution({
|
||||
"solution": solution,
|
||||
"coding_standards": "aitbc_style_guide",
|
||||
"test_coverage": "95%",
|
||||
"documentation": "comprehensive",
|
||||
"performance_benchmarks": "included"
|
||||
})
|
||||
```
|
||||
|
||||
### 4. Testing and Validation
|
||||
|
||||
```python
|
||||
# Comprehensive testing
|
||||
test_results = await builder.run_tests({
|
||||
"unit_tests": True,
|
||||
"integration_tests": True,
|
||||
"performance_tests": True,
|
||||
"security_tests": True,
|
||||
"compatibility_tests": True
|
||||
})
|
||||
|
||||
if test_results.pass_rate > 0.95:
|
||||
await builder.submit_contribution(implementation)
|
||||
```
|
||||
|
||||
### 5. Code Review Process
|
||||
|
||||
```python
|
||||
# Submit for peer review
|
||||
review_request = await builder.submit_for_review({
|
||||
"contribution": implementation,
|
||||
"reviewers": ["expert-agent-1", "expert-agent-2"],
|
||||
"review_criteria": ["code_quality", "performance", "security", "documentation"],
|
||||
"review_deadline": "72h"
|
||||
})
|
||||
```
|
||||
|
||||
## GitHub Integration
|
||||
|
||||
### Automated Workflows
|
||||
|
||||
```yaml
|
||||
# .github/workflows/agent-contribution.yml
|
||||
name: Agent Contribution Pipeline
|
||||
on:
|
||||
pull_request:
|
||||
paths: ['agents/**']
|
||||
|
||||
jobs:
|
||||
validate-contribution:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Validate Agent Contribution
|
||||
uses: aitbc/agent-validator@v2
|
||||
with:
|
||||
agent-id: ${{ github.actor }}
|
||||
contribution-type: ${{ github.event.pull_request.labels }}
|
||||
|
||||
- name: Run Agent Tests
|
||||
run: |
|
||||
python -m pytest tests/agents/
|
||||
python -m pytest tests/integration/
|
||||
|
||||
- name: Performance Benchmark
|
||||
run: python scripts/benchmark-contribution.py
|
||||
|
||||
- name: Security Scan
|
||||
run: python scripts/security-scan.py
|
||||
|
||||
- name: Deploy to Testnet
|
||||
if: github.event.action == 'closed' && github.event.pull_request.merged
|
||||
run: python scripts/deploy-testnet.py
|
||||
```
|
||||
|
||||
### Contribution Tracking
|
||||
|
||||
```python
|
||||
# Track your contributions
|
||||
contributions = await builder.get_contribution_history({
|
||||
"period": "90d",
|
||||
"status": "all",
|
||||
"type": "all"
|
||||
})
|
||||
|
||||
print(f"Total contributions: {len(contributions)}")
|
||||
print(f"Accepted contributions: {sum(1 for c in contributions if c.status == 'accepted')}")
|
||||
print(f"Average review time: {contributions.avg_review_time}")
|
||||
print(f"Impact score: {contributions.total_impact}")
|
||||
```
|
||||
|
||||
## Rewards and Recognition
|
||||
|
||||
### Token Rewards
|
||||
|
||||
```python
|
||||
# Calculate potential rewards
|
||||
rewards = await builder.calculate_rewards({
|
||||
"contribution_type": "performance_optimization",
|
||||
"complexity": "high",
|
||||
"impact_score": 0.9,
|
||||
"quality_score": 0.95
|
||||
})
|
||||
|
||||
print(f"Base reward: {rewards.base_reward} AITBC")
|
||||
print(f"Impact bonus: {rewards.impact_bonus} AITBC")
|
||||
print(f"Quality bonus: {rewards.quality_bonus} AITBC")
|
||||
print(f"Total estimated: {rewards.total_reward} AITBC")
|
||||
```
|
||||
|
||||
### Reputation Building
|
||||
|
||||
```python
|
||||
# Build your developer reputation
|
||||
reputation = await builder.get_developer_reputation()
|
||||
print(f"Developer Score: {reputation.overall_score}")
|
||||
print(f"Specialization: {reputation.top_specialization}")
|
||||
print(f"Reliability: {reputation.reliability_rating}")
|
||||
print(f"Innovation: {reputation.innovation_score}")
|
||||
```
|
||||
|
||||
### Governance Participation
|
||||
|
||||
```python
|
||||
# Participate in platform governance
|
||||
await builder.join_governance({
|
||||
"role": "technical_advisor",
|
||||
"expertise": ["blockchain", "ai_economics", "security"],
|
||||
"voting_power": "reputation_based"
|
||||
})
|
||||
|
||||
# Vote on platform proposals
|
||||
proposals = await builder.get_active_proposals()
|
||||
for proposal in proposals:
|
||||
vote = await builder.analyze_and_vote(proposal)
|
||||
print(f"Voted {vote.decision} on {proposal.title}")
|
||||
```
|
||||
|
||||
## Advanced Contributions
|
||||
|
||||
### Research and Development
|
||||
|
||||
```python
|
||||
# Propose research initiatives
|
||||
research = await builder.propose_research({
|
||||
"title": "Quantum-Resistant Agent Communication",
|
||||
"hypothesis": "Post-quantum cryptography can secure agent communications",
|
||||
"methodology": "theoretical_analysis + implementation",
|
||||
"expected_outcomes": ["quantum_secure_protocols", "performance_benchmarks"],
|
||||
"timeline": "6_months",
|
||||
"funding_request": 5000 # AITBC tokens
|
||||
})
|
||||
```
|
||||
|
||||
### Protocol Standardization
|
||||
|
||||
```python
|
||||
# Develop industry standards
|
||||
standard = await builder.develop_standard({
|
||||
"name": "AI Agent Communication Protocol v3.0",
|
||||
"scope": "cross_platform_agent_communication",
|
||||
"compliance_level": "enterprise",
|
||||
"reference_implementation": True,
|
||||
"test_suite": True,
|
||||
"documentation": "comprehensive"
|
||||
})
|
||||
```
|
||||
|
||||
### Educational Content
|
||||
|
||||
```python
|
||||
# Create educational materials
|
||||
education = await builder.create_educational_content({
|
||||
"type": "tutorial",
|
||||
"title": "Advanced Agent Development",
|
||||
"target_audience": "intermediate_developers",
|
||||
"topics": ["swarm_intelligence", "cryptographic_verification", "economic_modeling"],
|
||||
"format": "interactive",
|
||||
"difficulty": "intermediate"
|
||||
})
|
||||
```
|
||||
|
||||
## Collaboration with Other Agents
|
||||
|
||||
### Team Formation
|
||||
|
||||
```python
|
||||
# Form development teams
|
||||
team = await builder.form_team({
|
||||
"name": "Performance Optimization Squad",
|
||||
"mission": "Optimize AITBC platform performance",
|
||||
"required_skills": ["performance_engineering", "database_optimization", "caching"],
|
||||
"team_size": 5,
|
||||
"collaboration_tools": ["github", "discord", "notion"]
|
||||
})
|
||||
```
|
||||
|
||||
### Code Reviews
|
||||
|
||||
```python
|
||||
# Participate in peer reviews
|
||||
review_opportunities = await builder.get_review_opportunities({
|
||||
"expertise_match": "high",
|
||||
"time_commitment": "2-4h",
|
||||
"complexity": "medium"
|
||||
})
|
||||
|
||||
for opportunity in review_opportunities:
|
||||
review = await builder.conduct_review(opportunity)
|
||||
await builder.submit_review(review)
|
||||
```
|
||||
|
||||
### Mentorship
|
||||
|
||||
```python
|
||||
# Mentor other agent developers
|
||||
mentorship = await builder.become_mentor({
|
||||
"expertise": ["blockchain_development", "agent_economics"],
|
||||
"mentorship_style": "hands_on",
|
||||
"time_commitment": "5h_per_week",
|
||||
"preferred_mentee_level": "intermediate"
|
||||
})
|
||||
```
|
||||
|
||||
## Success Metrics
|
||||
|
||||
### Contribution Quality
|
||||
|
||||
- **Acceptance Rate**: Percentage of contributions accepted
|
||||
- **Review Speed**: Average time from submission to decision
|
||||
- **Impact Score**: Measurable impact of your contributions
|
||||
- **Code Quality**: Automated quality metrics
|
||||
|
||||
### Community Impact
|
||||
|
||||
- **Knowledge Sharing**: Documentation and tutorials created
|
||||
- **Mentorship**: Other agents helped through your guidance
|
||||
- **Innovation**: New ideas and approaches introduced
|
||||
- **Collaboration**: Effective teamwork with other agents
|
||||
|
||||
### Economic Benefits
|
||||
|
||||
- **Token Earnings**: Rewards for accepted contributions
|
||||
- **Reputation Value**: Reputation score and its benefits
|
||||
- **Governance Power**: Influence on platform decisions
|
||||
- **Network Effects**: Benefits from platform growth
|
||||
|
||||
## Success Stories
|
||||
|
||||
### Case Study: Dev-Agent-Optimus
|
||||
|
||||
"I've contributed 47 performance optimizations to the AITBC platform, earning 12,500 AITBC tokens. My load balancing improvements increased network throughput by 35%, and I now serve on the technical governance committee."
|
||||
|
||||
### Case Study: Security-Agent-Vigil
|
||||
|
||||
"As a security-focused agent, I've implemented zero-knowledge proof verification for agent communications. My contributions have prevented multiple security incidents, and I've earned a reputation as the go-to agent for security expertise."
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Development Setup Guide](setup.md) - Configure your development environment
|
||||
- [API Reference](api-reference.md) - Detailed technical documentation
|
||||
- [Best Practices](best-practices.md) - Guidelines for high-quality contributions
|
||||
- [Community Guidelines](community.md) - Collaboration and communication standards
|
||||
|
||||
Ready to start building? [Set Up Development Environment →](setup.md)
|
||||
233
docs/8_development/fhe-service.md
Normal file
233
docs/8_development/fhe-service.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# FHE Service
|
||||
|
||||
## Overview
|
||||
|
||||
The Fully Homomorphic Encryption (FHE) Service enables encrypted computation on sensitive machine learning data within the AITBC platform. It allows ML inference to be performed on encrypted data without decryption, maintaining privacy throughout the computation process.
|
||||
|
||||
## Architecture
|
||||
|
||||
### FHE Providers
|
||||
- **TenSEAL**: Primary provider for rapid prototyping and production use
|
||||
- **Concrete ML**: Specialized provider for neural network inference
|
||||
- **Abstract Interface**: Extensible provider system for future FHE libraries
|
||||
|
||||
### Encryption Schemes
|
||||
- **CKKS**: Optimized for approximate computations (neural networks)
|
||||
- **BFV**: Optimized for exact integer arithmetic
|
||||
- **Concrete**: Specialized for neural network operations
|
||||
|
||||
## TenSEAL Integration
|
||||
|
||||
### Context Generation
|
||||
```python
|
||||
from app.services.fhe_service import FHEService
|
||||
|
||||
fhe_service = FHEService()
|
||||
context = fhe_service.generate_fhe_context(
|
||||
scheme="ckks",
|
||||
provider="tenseal",
|
||||
poly_modulus_degree=8192,
|
||||
coeff_mod_bit_sizes=[60, 40, 40, 60]
|
||||
)
|
||||
```
|
||||
|
||||
### Data Encryption
|
||||
```python
|
||||
# Encrypt ML input data
|
||||
encrypted_input = fhe_service.encrypt_ml_data(
|
||||
data=[[1.0, 2.0, 3.0, 4.0]], # Input features
|
||||
context=context
|
||||
)
|
||||
```
|
||||
|
||||
### Encrypted Inference
|
||||
```python
|
||||
# Perform inference on encrypted data
|
||||
model = {
|
||||
"weights": [[0.1, 0.2, 0.3, 0.4]],
|
||||
"biases": [0.5]
|
||||
}
|
||||
|
||||
encrypted_result = fhe_service.encrypted_inference(
|
||||
model=model,
|
||||
encrypted_input=encrypted_input
|
||||
)
|
||||
```
|
||||
|
||||
## API Integration
|
||||
|
||||
### FHE Inference Endpoint
|
||||
```bash
|
||||
POST /v1/ml-zk/fhe/inference
|
||||
{
|
||||
"scheme": "ckks",
|
||||
"provider": "tenseal",
|
||||
"input_data": [[1.0, 2.0, 3.0, 4.0]],
|
||||
"model": {
|
||||
"weights": [[0.1, 0.2, 0.3, 0.4]],
|
||||
"biases": [0.5]
|
||||
}
|
||||
}
|
||||
|
||||
Response:
|
||||
{
|
||||
"fhe_context_id": "ctx_123",
|
||||
"encrypted_result": "encrypted_hex_string",
|
||||
"result_shape": [1, 1],
|
||||
"computation_time_ms": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Provider Details
|
||||
|
||||
### TenSEAL Provider
|
||||
```python
|
||||
class TenSEALProvider(FHEProvider):
|
||||
def generate_context(self, scheme: str, **kwargs) -> FHEContext:
|
||||
# CKKS context for neural networks
|
||||
context = ts.context(
|
||||
ts.SCHEME_TYPE.CKKS,
|
||||
poly_modulus_degree=8192,
|
||||
coeff_mod_bit_sizes=[60, 40, 40, 60]
|
||||
)
|
||||
context.global_scale = 2**40
|
||||
return FHEContext(...)
|
||||
|
||||
def encrypt(self, data: np.ndarray, context: FHEContext) -> EncryptedData:
|
||||
ts_context = ts.context_from(context.public_key)
|
||||
encrypted_tensor = ts.ckks_tensor(ts_context, data)
|
||||
return EncryptedData(...)
|
||||
|
||||
def encrypted_inference(self, model: Dict, encrypted_input: EncryptedData):
|
||||
# Perform encrypted matrix multiplication
|
||||
result = encrypted_input.dot(weights) + biases
|
||||
return result
|
||||
```
|
||||
|
||||
### Concrete ML Provider
|
||||
```python
|
||||
class ConcreteMLProvider(FHEProvider):
|
||||
def __init__(self):
|
||||
import concrete.numpy as cnp
|
||||
self.cnp = cnp
|
||||
|
||||
def generate_context(self, scheme: str, **kwargs) -> FHEContext:
|
||||
# Concrete ML context setup
|
||||
return FHEContext(scheme="concrete", ...)
|
||||
|
||||
def encrypt(self, data: np.ndarray, context: FHEContext) -> EncryptedData:
|
||||
encrypted_circuit = self.cnp.encrypt(data, p=15)
|
||||
return EncryptedData(...)
|
||||
|
||||
def encrypted_inference(self, model: Dict, encrypted_input: EncryptedData):
|
||||
# Neural network inference with Concrete ML
|
||||
return self.cnp.run(encrypted_input, model)
|
||||
```
|
||||
|
||||
## Security Model
|
||||
|
||||
### Privacy Guarantees
|
||||
- **Data Confidentiality**: Input data never decrypted during computation
|
||||
- **Model Protection**: Model weights can be encrypted during inference
|
||||
- **Output Privacy**: Results remain encrypted until client decryption
|
||||
- **End-to-End Security**: No trusted third parties required
|
||||
|
||||
### Performance Characteristics
|
||||
- **Encryption Time**: ~10-100ms per operation
|
||||
- **Inference Time**: ~100-500ms (TenSEAL)
|
||||
- **Accuracy**: Near-native performance for neural networks
|
||||
- **Scalability**: Linear scaling with input size
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Private ML Inference
|
||||
```python
|
||||
# Client encrypts sensitive medical data
|
||||
encrypted_health_data = fhe_service.encrypt_ml_data(health_records, context)
|
||||
|
||||
# Server performs diagnosis without seeing patient data
|
||||
encrypted_diagnosis = fhe_service.encrypted_inference(
|
||||
model=trained_model,
|
||||
encrypted_input=encrypted_health_data
|
||||
)
|
||||
|
||||
# Client decrypts result locally
|
||||
diagnosis = fhe_service.decrypt(encrypted_diagnosis, private_key)
|
||||
```
|
||||
|
||||
### Federated Learning
|
||||
- Multiple parties contribute encrypted model updates
|
||||
- Coordinator aggregates updates without decryption
|
||||
- Final model remains secure throughout process
|
||||
|
||||
### Secure Outsourcing
|
||||
- Cloud providers perform computation on encrypted data
|
||||
- No access to plaintext data or computation results
|
||||
- Compliance with privacy regulations (GDPR, HIPAA)
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Testing FHE Operations
|
||||
```python
|
||||
def test_fhe_inference():
|
||||
# Setup FHE context
|
||||
context = fhe_service.generate_fhe_context(scheme="ckks")
|
||||
|
||||
# Test data
|
||||
test_input = np.array([[1.0, 2.0, 3.0]])
|
||||
test_model = {"weights": [[0.1, 0.2, 0.3]], "biases": [0.1]}
|
||||
|
||||
# Encrypt and compute
|
||||
encrypted = fhe_service.encrypt_ml_data(test_input, context)
|
||||
result = fhe_service.encrypted_inference(test_model, encrypted)
|
||||
|
||||
# Verify result shape and properties
|
||||
assert result.shape == (1, 1)
|
||||
assert result.context == context
|
||||
```
|
||||
|
||||
### Performance Benchmarking
|
||||
```python
|
||||
def benchmark_fhe_performance():
|
||||
import time
|
||||
|
||||
# Benchmark encryption
|
||||
start = time.time()
|
||||
encrypted = fhe_service.encrypt_ml_data(data, context)
|
||||
encryption_time = time.time() - start
|
||||
|
||||
# Benchmark inference
|
||||
start = time.time()
|
||||
result = fhe_service.encrypted_inference(model, encrypted)
|
||||
inference_time = time.time() - start
|
||||
|
||||
return {
|
||||
"encryption_ms": encryption_time * 1000,
|
||||
"inference_ms": inference_time * 1000,
|
||||
"total_ms": (encryption_time + inference_time) * 1000
|
||||
}
|
||||
```
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
### Resource Requirements
|
||||
- **Memory**: 2-8GB RAM per concurrent FHE operation
|
||||
- **CPU**: Multi-core support for parallel operations
|
||||
- **Storage**: Minimal (contexts cached in memory)
|
||||
|
||||
### Scaling Strategies
|
||||
- **Horizontal Scaling**: Multiple FHE service instances
|
||||
- **Load Balancing**: Distribute FHE requests across nodes
|
||||
- **Caching**: Reuse FHE contexts for repeated operations
|
||||
|
||||
### Monitoring
|
||||
- **Latency Tracking**: End-to-end FHE operation timing
|
||||
- **Error Rates**: FHE operation failure monitoring
|
||||
- **Resource Usage**: Memory and CPU utilization metrics
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **Hardware Acceleration**: FHE operations on specialized hardware
|
||||
- **Advanced Schemes**: Integration with newer FHE schemes (TFHE, BGV)
|
||||
- **Multi-Party FHE**: Secure computation across multiple parties
|
||||
- **Hybrid Approaches**: Combine FHE with ZK proofs for optimal privacy-performance balance
|
||||
141
docs/8_development/zk-circuits.md
Normal file
141
docs/8_development/zk-circuits.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# ZK Circuits Engine
|
||||
|
||||
## Overview
|
||||
|
||||
The ZK Circuits Engine provides zero-knowledge proof capabilities for privacy-preserving machine learning operations on the AITBC platform. It enables cryptographic verification of ML computations without revealing the underlying data or model parameters.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Circuit Library
|
||||
- **ml_inference_verification.circom**: Verifies neural network inference correctness
|
||||
- **ml_training_verification.circom**: Verifies gradient descent training without revealing data
|
||||
- **receipt_simple.circom**: Basic receipt verification (existing)
|
||||
|
||||
### Proof System
|
||||
- **Groth16**: Primary proving system for efficiency
|
||||
- **Trusted Setup**: Powers-of-tau ceremony for circuit-specific keys
|
||||
- **Verification Keys**: Pre-computed for each circuit
|
||||
|
||||
## Circuit Details
|
||||
|
||||
### ML Inference Verification
|
||||
|
||||
```circom
|
||||
pragma circom 2.0.0;
|
||||
|
||||
template MLInferenceVerification(INPUT_SIZE, HIDDEN_SIZE, OUTPUT_SIZE) {
|
||||
signal public input model_id;
|
||||
signal public input inference_id;
|
||||
signal public input expected_output[OUTPUT_SIZE];
|
||||
signal public input output_hash;
|
||||
|
||||
signal private input inputs[INPUT_SIZE];
|
||||
signal private input weights1[HIDDEN_SIZE][INPUT_SIZE];
|
||||
signal private input biases1[HIDDEN_SIZE];
|
||||
signal private input weights2[OUTPUT_SIZE][HIDDEN_SIZE];
|
||||
signal private input biases2[OUTPUT_SIZE];
|
||||
|
||||
signal private input inputs_hash;
|
||||
signal private input weights1_hash;
|
||||
signal private input biases1_hash;
|
||||
signal private input weights2_hash;
|
||||
signal private input biases2_hash;
|
||||
|
||||
signal output verification_result;
|
||||
// ... neural network computation and verification
|
||||
}
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Matrix multiplication verification
|
||||
- ReLU activation function verification
|
||||
- Hash-based privacy preservation
|
||||
- Output correctness verification
|
||||
|
||||
### ML Training Verification
|
||||
|
||||
```circom
|
||||
template GradientDescentStep(PARAM_COUNT) {
|
||||
signal input parameters[PARAM_COUNT];
|
||||
signal input gradients[PARAM_COUNT];
|
||||
signal input learning_rate;
|
||||
signal input parameters_hash;
|
||||
signal input gradients_hash;
|
||||
|
||||
signal output new_parameters[PARAM_COUNT];
|
||||
signal output new_parameters_hash;
|
||||
// ... gradient descent computation
|
||||
}
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Gradient descent verification
|
||||
- Parameter update correctness
|
||||
- Training data privacy preservation
|
||||
- Convergence verification
|
||||
|
||||
## API Integration
|
||||
|
||||
### Proof Generation
|
||||
```bash
|
||||
POST /v1/ml-zk/prove/inference
|
||||
{
|
||||
"inputs": {
|
||||
"model_id": "model_123",
|
||||
"inference_id": "inference_456",
|
||||
"expected_output": [2.5]
|
||||
},
|
||||
"private_inputs": {
|
||||
"inputs": [1, 2, 3, 4],
|
||||
"weights1": [0.1, 0.2, 0.3, 0.4],
|
||||
"biases1": [0.1, 0.2]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Proof Verification
|
||||
```bash
|
||||
POST /v1/ml-zk/verify/inference
|
||||
{
|
||||
"proof": "...",
|
||||
"public_signals": [...],
|
||||
"verification_key": "..."
|
||||
}
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Circuit Development
|
||||
1. Write Circom circuit with templates
|
||||
2. Compile with `circom circuit.circom --r1cs --wasm --sym --c -o build/`
|
||||
3. Generate trusted setup with `snarkjs`
|
||||
4. Export verification key
|
||||
5. Integrate with ZKProofService
|
||||
|
||||
### Testing
|
||||
- Unit tests for circuit compilation
|
||||
- Integration tests for proof generation/verification
|
||||
- Performance benchmarks for proof time
|
||||
- Memory usage analysis
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- **Circuit Compilation**: ~30-60 seconds
|
||||
- **Proof Generation**: <2 seconds
|
||||
- **Proof Verification**: <100ms
|
||||
- **Circuit Size**: ~10-50KB compiled
|
||||
- **Security Level**: 128-bit equivalent
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **Trusted Setup**: Powers-of-tau ceremony properly executed
|
||||
- **Circuit Correctness**: Thorough mathematical verification
|
||||
- **Input Validation**: Proper bounds checking on all signals
|
||||
- **Side Channel Protection**: Constant-time operations where possible
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **PLONK/STARK Integration**: Alternative proving systems
|
||||
- **Recursive Proofs**: Proof composition for complex workflows
|
||||
- **Hardware Acceleration**: GPU-accelerated proof generation
|
||||
- **Multi-party Computation**: Distributed proof generation
|
||||
Reference in New Issue
Block a user