oib/aitbc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add marketplace metrics, privacy features, and service registry endpoints

Browse Source 
- Add Prometheus metrics for marketplace API throughput and error rates with new dashboard panels
- Implement confidential transaction models with encryption support and access control
- Add key management system with registration, rotation, and audit logging
- Create services and registry routers for service discovery and management
- Integrate ZK proof generation for privacy-preserving receipts
- Add metrics instru
This commit is contained in:
oib
2025-12-22 10:33:23 +01:00
parent d98b2c7772
commit c8be9d7414
260 changed files with 59033 additions and 351 deletions
Download Patch File Download Diff File Expand all files Collapse all files

354
docs/reference/confidential-transactions.md Normal file
View File

@ -0,0 +1,354 @@
# Confidential Transactions Architecture
## Overview
Design for opt-in confidential transaction support in AITBC, enabling participants to encrypt sensitive transaction data while maintaining selective disclosure and audit capabilities.
## Architecture
### Encryption Model
**Hybrid Encryption with Envelope Pattern**:
1. **Data Encryption**: AES-256-GCM for transaction data
2. **Key Exchange**: X25519 ECDH for per-recipient key distribution
3. **Envelope Pattern**: Random DEK per transaction, encrypted for each authorized party
### Key Components
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Transaction │───▶│ Encryption │───▶│ Storage │
│ Service │ │ Service │ │ Layer │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Key Manager │ │ Access Control │ │ Audit Log │
└─────────────────┘ └──────────────────┘ └─────────────────┘
```
## Data Flow
### 1. Transaction Creation (Opt-in)
```python
# Client requests confidential transaction
transaction = {
"job_id": "job-123",
"amount": "1000",
"confidential": True,
"participants": ["client-456", "miner-789", "auditor-001"]
}
# Coordinator encrypts sensitive fields
encrypted = encryption_service.encrypt(
data={"amount": "1000", "pricing": "details"},
participants=transaction["participants"]
)
# Store with encrypted payload
stored_transaction = {
"job_id": "job-123",
"public_data": {"job_id": "job-123"},
"encrypted_data": encrypted.ciphertext,
"encrypted_keys": encrypted.encrypted_keys,
"confidential": True
}
```
### 2. Data Access (Authorized Party)
```python
# Miner requests access to transaction data
access_request = {
"transaction_id": "tx-456",
"requester": "miner-789",
"purpose": "settlement"
}
# Verify access rights
if access_control.verify(access_request):
# Decrypt using recipient's private key
decrypted = encryption_service.decrypt(
ciphertext=stored_transaction.encrypted_data,
encrypted_key=stored_transaction.encrypted_keys["miner-789"],
private_key=miner_private_key
)
```
### 3. Audit Access (Regulatory)
```python
# Auditor with court order requests access
audit_request = {
"transaction_id": "tx-456",
"requester": "auditor-001",
"authorization": "court-order-123"
}
# Special audit key escrow
audit_key = key_manager.get_audit_key(audit_request.authorization)
decrypted = encryption_service.audit_decrypt(
ciphertext=stored_transaction.encrypted_data,
audit_key=audit_key
)
```
## Implementation Details
### Encryption Service
```python
class ConfidentialTransactionService:
"""Service for handling confidential transactions"""
def __init__(self, key_manager: KeyManager):
self.key_manager = key_manager
self.cipher = AES256GCM()
def encrypt(self, data: Dict, participants: List[str]) -> EncryptedData:
"""Encrypt data for multiple participants"""
# Generate random DEK
dek = os.urandom(32)
# Encrypt data with DEK
ciphertext = self.cipher.encrypt(dek, json.dumps(data))
# Encrypt DEK for each participant
encrypted_keys = {}
for participant in participants:
public_key = self.key_manager.get_public_key(participant)
encrypted_keys[participant] = self._encrypt_dek(dek, public_key)
# Add audit escrow
audit_public_key = self.key_manager.get_audit_key()
encrypted_keys["audit"] = self._encrypt_dek(dek, audit_public_key)
return EncryptedData(
ciphertext=ciphertext,
encrypted_keys=encrypted_keys,
algorithm="AES-256-GCM+X25519"
)
def decrypt(self, ciphertext: bytes, encrypted_key: bytes,
private_key: bytes) -> Dict:
"""Decrypt data for specific participant"""
# Decrypt DEK
dek = self._decrypt_dek(encrypted_key, private_key)
# Decrypt data
plaintext = self.cipher.decrypt(dek, ciphertext)
return json.loads(plaintext)
```
### Key Management
```python
class KeyManager:
"""Manages encryption keys for participants"""
def __init__(self, storage: KeyStorage):
self.storage = storage
self.key_pairs = {}
def generate_key_pair(self, participant_id: str) -> KeyPair:
"""Generate X25519 key pair for participant"""
private_key = X25519.generate_private_key()
public_key = private_key.public_key()
key_pair = KeyPair(
participant_id=participant_id,
private_key=private_key,
public_key=public_key
)
self.storage.store(key_pair)
return key_pair
def rotate_keys(self, participant_id: str):
"""Rotate encryption keys"""
# Generate new key pair
new_key_pair = self.generate_key_pair(participant_id)
# Re-encrypt active transactions
self._reencrypt_transactions(participant_id, new_key_pair)
```
### Access Control
```python
class AccessController:
"""Controls access to confidential transaction data"""
def __init__(self, policy_store: PolicyStore):
self.policy_store = policy_store
def verify_access(self, request: AccessRequest) -> bool:
"""Verify if requester has access rights"""
# Check participant status
if not self._is_authorized_participant(request.requester):
return False
# Check purpose-based access
if not self._check_purpose(request.purpose, request.requester):
return False
# Check time-based restrictions
if not self._check_time_restrictions(request):
return False
return True
def _is_authorized_participant(self, participant_id: str) -> bool:
"""Check if participant is authorized for confidential transactions"""
# Verify KYC/KYB status
# Check compliance flags
# Validate regulatory approval
return True
```
## Data Models
### Confidential Transaction
```python
class ConfidentialTransaction(BaseModel):
"""Transaction with optional confidential fields"""
# Public fields (always visible)
transaction_id: str
job_id: str
timestamp: datetime
status: str
# Confidential fields (encrypted when opt-in)
amount: Optional[str] = None
pricing: Optional[Dict] = None
settlement_details: Optional[Dict] = None
# Encryption metadata
confidential: bool = False
encrypted_data: Optional[bytes] = None
encrypted_keys: Optional[Dict[str, bytes]] = None
algorithm: Optional[str] = None
# Access control
participants: List[str] = []
access_policies: Dict[str, Any] = {}
```
### Access Log
```python
class ConfidentialAccessLog(BaseModel):
"""Audit log for confidential data access"""
transaction_id: str
requester: str
purpose: str
timestamp: datetime
authorized_by: str
data_accessed: List[str]
ip_address: str
user_agent: str
```
## Security Considerations
### 1. Key Security
- Private keys stored in HSM or secure enclave
- Key rotation every 90 days
- Zero-knowledge proof of key possession
### 2. Data Protection
- AES-256-GCM provides confidentiality + integrity
- Random IV per encryption
- Forward secrecy with per-transaction DEKs
### 3. Access Control
- Multi-factor authentication for decryption
- Role-based access control
- Time-bound access permissions
### 4. Audit Compliance
- Immutable audit logs
- Regulatory access with court orders
- Privacy-preserving audit proofs
## Performance Optimization
### 1. Lazy Encryption
- Only encrypt fields marked as confidential
- Cache encrypted data for frequent access
- Batch encryption for bulk operations
### 2. Key Management
- Pre-compute shared secrets for regular participants
- Use key derivation for multiple access levels
- Implement key caching with secure eviction
### 3. Storage Optimization
- Compress encrypted data
- Deduplicate common encrypted patterns
- Use column-level encryption for databases
## Migration Strategy
### Phase 1: Opt-in Support
- Add confidential flags to existing models
- Deploy encryption service
- Update transaction endpoints
### Phase 2: Participant Onboarding
- Generate key pairs for all participants
- Implement key distribution
- Train users on privacy features
### Phase 3: Full Rollout
- Enable confidential transactions by default for sensitive data
- Implement advanced access controls
- Add privacy analytics and reporting
## Testing Strategy
### 1. Unit Tests
- Encryption/decryption correctness
- Key management operations
- Access control logic
### 2. Integration Tests
- End-to-end confidential transaction flow
- Cross-system key exchange
- Audit trail verification
### 3. Security Tests
- Penetration testing
- Cryptographic validation
- Side-channel resistance
## Compliance
### 1. GDPR
- Right to encryption
- Data minimization
- Privacy by design
### 2. Financial Regulations
- SEC Rule 17a-4
- MiFID II transaction reporting
- AML/KYC requirements
### 3. Industry Standards
- ISO 27001
- NIST Cybersecurity Framework
- PCI DSS for payment data
## Next Steps
1. Implement core encryption service
2. Create key management infrastructure
3. Update transaction models and APIs
4. Deploy access control system
5. Implement audit logging
6. Conduct security testing
7. Gradual rollout with monitoring