oib/aitbc
1
0
Fork 0
Code Issues Pull Requests Actions 21 Packages Projects Releases Wiki Activity

Remove outdated Hermes analysis and decomposition plan documents
Some checks failed
Cross-Node Transaction Testing / transaction-test (push) Has been cancelled
Details
Deploy to Testnet / deploy-testnet (push) Has been cancelled
Details
Multi-Node Stress Testing / stress-test (push) Has been cancelled
Details
Documentation Validation / validate-docs (push) Has been cancelled
Details
Documentation Validation / validate-policies-strict (push) Has been cancelled
Details

Browse Source
This commit is contained in:
aitbc
2026-05-27 13:33:16 +02:00
parent 29b066a5dd
commit d78cb85fd7
13 changed files with 0 additions and 965 deletions
Download Patch File Download Diff File Expand all files Collapse all files

411
.hermes/plans/2026-05-12_102100-aitbc-codebase-analysis.md
View File

@@ -1,411 +0,0 @@
# AITBC Codebase Analysis & Recommendations Plan
**Date:** 2026-05-12
**Scope:** Full analysis of /opt/aitbc/ -- architecture, performance, security, maintainability
**Codebase Version:** 0.6.0
**Python:** 3.13.5
---
## Executive Summary
AITBC is a large monorepo (~212K LOC Python, 75K LOC Solidity, 75K LOC CLI) implementing a blockchain-based AI agent compute network. The system has strong foundational infrastructure (28 systemd services, PoA consensus, multi-chain support, 20 Solidity contracts) but carries significant technical debt: a bloated coordinator-api (117K LOC, 91 files >500 lines), 925 print() statements in production code, 50 bare except clauses, 41 potentially hardcoded secrets, duplicated config/database modules across apps, and 14.3K LOC of stubs that may be dead code. The core blockchain-node (27K LOC) is well-structured, but the coordinator-api has grown into a monolith that needs decomposition.
---
## 1. Codebase Scale Metrics
| Component | Files | LOC | Notes |
|-----------|-------|-----|-------|
| **apps/** | 840 | 211,844 | Main application code |
| **cli/** | 232 | 74,805 | Unified CLI tool |
| **aitbc/** (core pkg) | 52 | 11,190 | Shared utilities |
| **packages/py/** | 42 | 5,458 | SDK packages |
| **contracts/** | 490 | 74,589 | Solidity contracts |
| **tests/** | 56 files | ~789 test functions | Good test count |
| **stubs/** | 65 | 14,300 | Placeholder services |
| **systemd/** | 28 | - | Service definitions |
**App breakdown (largest):**
- `coordinator-api`: 338 files, 116,568 LOC (55% of all app code)
- `blockchain-node`: 127 files, 26,895 LOC
- `zk-circuits`: 52 files, 15,160 LOC
- `stubs`: 65 files, 14,300 LOC
- `pool-hub`: 48 files, 5,461 LOC
- `agent-coordinator`: 45 files, 9,983 LOC
- `exchange`: 19 files, 5,144 LOC
- `wallet`: 27 files, 3,482 LOC
---
## 2. Architecture Analysis
### 2.1 Current Architecture Pattern
The codebase follows a **monorepo with per-app packaging** pattern:
- Root `pyproject.toml` at `/opt/aitbc/pyproject.toml` with shared deps
- 15 per-app `pyproject.toml` files for independent packaging
- 28 systemd services for process management
- Core shared library at `/opt/aitbc/aitbc/` (52 files)
- 3 Python packages in `/opt/aitbc/packages/py/` (crypto, sdk, agent-sdk)
### 2.2 Strengths
1. **Well-structured blockchain-node**: Clear separation of concerns with `consensus/`, `contracts/`, `network/`, `economics/`, `state/` subdirectories. Largest file is 1,278 lines (RPC router) which is reasonable.
2. **Solid middleware layer**: Core package provides `error_handler`, `performance`, `request_id`, `validation` middleware -- all properly implemented.
3. **Database migration support**: Alembic configs for blockchain-node (7 migrations), coordinator-api (2), pool-hub (5).
4. **Comprehensive infrastructure**: Terraform (15 files), Kubernetes (6 manifests), nginx config, Prometheus dashboards, Gitea CI/CD.
5. **Circuit breaker pattern**: Implemented in PoA consensus for resilience.
6. **Good test infrastructure**: 789 test functions across 56 files, 41 conftest.py files, pytest markers for unit/integration/e2e/slow.
### 2.3 Architecture Issues
#### CRITICAL: Coordinator-API Monolith (117K LOC)
The coordinator-api at 116,568 LOC across 338 files is the single biggest architectural risk:
- **91 files over 500 lines** (27% of all files)
- Largest file: `advanced_reinforcement_learning.py` at 2,000 lines
- Service layer has 99 files + 24 sub-module files = 123 service files
- Router layer has 61 files
- Domain layer has 32 files
- This service alone is 55% of all application code
**Root cause**: The coordinator-api has accumulated features without decomposition. It handles RL agents, marketplace, trading, certification, compliance, analytics, payments, staking, governance, AMM, AI surveillance, multi-language support, ZK proofs, and more -- all in one deployable unit.
#### HIGH: Duplicated Cross-App Modules
Each app independently implements:
- `config.py` in 6 different apps (blockchain-node, coordinator-api, agent-coordinator, blockchain-event-bridge, pool-hub)
- `database.py` in 4 apps (blockchain-node, coordinator-api, exchange, pool-hub)
- No shared database connection pooling across apps
#### HIGH: Stubs Directory (14.3K LOC of Dead Code?)
The `apps/stubs/` directory contains 65 files across 68 subdirectories totaling 14,300 LOC. These appear to be placeholder implementations. Including `compliance-service`, `exchange-integration`, `global-ai-agents`, `global-infrastructure`, `multi-region-load-balancer`, `plugin-analytics`, `plugin-marketplace`, `plugin-registry`, `plugin-security`, `simple-explorer`, `trading-engine`. Each has its own tests, suggesting they were meant to be real services but remain as stubs.
#### MEDIUM: Missing Package Structure
Only 128 of 1,026 app directories have `__init__.py` files (12.5%). This means most directories are not proper Python packages, which breaks relative imports and makes the codebase harder to navigate programmatically.
#### MEDIUM: Overlapping Marketplace Services
Three separate marketplace implementations exist:
- `apps/marketplace` (9 files, 1,762 LOC) -- appears to be the primary marketplace
- `apps/marketplace-service` (10 files, 875 LOC) -- unclear differentiation
- `apps/marketplace-service-debug` (8 files, 821 LOC) -- debug variant
The relationship and boundary between these three is unclear.
#### LOW: Coordinator-API Entry Point is a Stub
`/opt/aitbc/apps/coordinator-api/src/app.py` is a single line: `# Import the FastAPI app from main.py for compatibility`. There is no `main.py` in the same directory -- the actual entry point appears to be elsewhere.
---
## 3. Performance Analysis
### 3.1 Strengths
1. **Async-first architecture**: 362 files use `async def`, consistent with FastAPI best practices
2. **FastAPI Depends for DI**: 94 files use FastAPI's dependency injection system
3. **LRU caching implementation**: Core package provides `LRU cache` with hit/miss tracking
4. **Performance middleware**: `PerformanceLoggingMiddleware` tracks request duration and adds `X-Process-Time` headers
5. **Adaptive sync**: Blockchain sync uses tiered batch sizing (10K+ blocks: 500-1000 batch)
6. **Circuit breaker**: PoA consensus implements circuit breaker for fault tolerance
### 3.2 Performance Issues
#### HIGH: No Distributed Caching Layer
Despite Redis being in `pyproject.toml` dependencies, grep found only 21 files mentioning Redis (mostly multi-language services and docs). There is no Redis-based caching layer for:
- API response caching
- Session storage
- Rate limiting state
- Blockchain state caching
This means every request hits the database directly.
#### HIGH: No Connection Pooling
Despite `asyncpg` and `sqlalchemy` with async support being dependencies, there is no visible connection pool configuration across apps. Each app creates its own database connections independently.
#### HIGH: No Background Task System
No Celery, RQ, or FastAPI BackgroundTasks usage found outside of 3 files. Long-running operations (blockchain sync, analytics processing, RL training) appear to be handled inline or via systemd timers rather than a proper task queue.
#### MEDIUM: No API Rate Limiting
`slowapi` and `limits` are in production dependencies but only found in 3 files in coordinator-api routers. Most endpoints have no rate limiting.
#### MEDIUM: No Request Batching
The coordinator-api receives requests for 61 different router domains but implements no request batching for high-throughput operations like marketplace queries or analytics.
#### LOW: Logging Performance
The `PerformanceLoggingMiddleware` logs every request synchronously with `logger.info()`. At high throughput, this creates I/O bottlenecks. Should use `logger.debug()` or sampling.
---
## 4. Security Analysis
### 4.1 Strengths
1. **Security hardening module**: `aitbc/security_hardening.py` (396 lines) provides input validation, sanitization, ethereum address validation, URL validation, email validation
2. **JWT infrastructure**: 30 files reference JWT authentication
3. **CORS configuration**: 23 files configure CORS policies
4. **HTTPS enforcement**: Security headers module (`aitbc/security_headers.py`) present
5. **Encryption at rest**: `database_encryption.py` in blockchain-node provides SQLCipher support
6. **Multi-sig wallets**: Mentioned in README as implemented
7. **Audit logging**: `audit_logging.py` service in coordinator-api (584 lines)
8. **Dev dependencies**: `bandit` (security linter) and `safety` (dependency vulnerability scanner) configured
9. **Pre-commit hooks**: `.pre-commit-config.yaml` configured
10. **Error handler middleware**: Properly returns generic 500 messages without leaking internals
### 4.2 Security Issues
#### CRITICAL: Potentially Hardcoded Secrets (41 instances)
41 files contain patterns matching potential hardcoded secrets, API keys, or tokens:
- `coordinator-api/.../cross_chain_integration.py:191`: `private_key = "mock_private_key"`
- `coordinator-api/.../wallet_service.py:72`: `encrypted_private_key = "[ENCRYPTED_MOCK]"`
- Multiple test files with `api_key="test_key"` patterns
While some are test files, production code also contains these patterns. Need audit to distinguish real secrets from test fixtures.
#### HIGH: 50 Bare Except Clauses
50 instances of bare `except:` in production code across:
- coordinator-api: 44 instances
- exchange: 1 instance
- Others: 5 instances
Bare excepts catch `SystemExit`, `KeyboardInterrupt`, and `GeneratorExit`, making the application unkillable in some error scenarios and hiding real errors.
#### HIGH: SQL Injection Risks (21 instances)
21 files use f-strings in SQL execute statements:
- `multichain_ledger.py:98,110,123`: `cursor.execute(f"""...`) with table name interpolation
- Various migration scripts using `f"SELECT * FROM {table_name}"`
While some are internal migration scripts, the `multichain_ledger.py` pattern is dangerous if table names come from user input.
#### HIGH: Insufficient Rate Limiting
Only 3 coordinator-api routers implement rate limiting. The remaining 58+ routers have no protection against:
- Brute force authentication attempts
- API abuse
- DDoS at the application layer
#### MEDIUM: Feature Flags Not Utilized
`/opt/aitbc/feature_flags.json` is an empty object `{}`. Despite having `aitbc/feature_flags.py` (a feature flag system), no features are actually flagged. This means:
- No gradual rollout capability
- No kill switches for problematic features
- No A/B testing infrastructure
#### MEDIUM: CORS Configuration Cents
23 files configure CORS but the configuration is scattered and inconsistent. No centralized CORS policy management.
#### MEDIUM: Dependency Vulnerability Scanning Not Enforced
While `safety` and `bandit` are in dev dependencies and pre-commit config, there is no evidence of CI/CD pipeline enforcement of security scanning. The Gitea workflows reference "security scanning optimized for changed files" but this should be comprehensive.
#### LOW: Debug Mode References
Several debug patterns found:
- `marketplace-service-debug` app (8 files) appears to be a debug variant
- `mock_coordinator.py` in blockchain-node observability
---
## 5. Maintainability Analysis
### 5.1 Strengths
1. **Comprehensive documentation**: Full docs directory with architecture, deployment, security, and operations guides
2. **Standardized quality tools**: ruff (line length 127), mypy (relaxed mode for gradual adoption), black, isort, pre-commit
3. **Good test coverage infrastructure**: 789 test functions, pytest with coverage (50% minimum threshold), markers for test types
4. **API versioning**: `aitbc/api_versioning.py` provides API versioning support
5. **Distributed tracing**: `aitbc/distributed_tracing.py` exists for observability
6. **Blue-green deployment**: `aitbc/blue_green_deployment.py` for zero-downtime deploys
7. **Domain-driven design**: coordinator-api uses `domain/` subdirectory with bounded contexts
8. **Repository pattern**: coordinator-api has `repositories/` directory
9. **Skills documentation**: 8 operational skills docs for agents
### 5.2 Maintainability Issues
#### CRITICAL: 925 print() Statements in Production Code
925 instances of `print()` in `/opt/aitbc/apps/` and `/opt/aitbc/packages/`. This is the most impactful maintainability issue:
- Prints bypass structured logging, making log aggregation impossible
- Prints cannot be filtered by severity level
- Prints go to stdout, not to configured log handlers
- In async code, prints can interleave unpredictably
Most concentrated in:
- `agent-coordinator/src/coordinator.py` (many debug prints)
- `agent-compliance/src/compliance_agent.py`
- `blockchain-node/scripts/` (expected for scripts, but still 925 total)
#### HIGH: 65 Stub Services (14.3K LOC)
The `apps/stubs/` directory is a significant maintenance burden:
- Each stub has its own tests (increasing test suite time)
- Each stub has its own `pyproject.toml` (15 total per-app pyproject files)
- Creates confusion about which services are real vs. placeholders
- Must be maintained/skipped during CI/CD runs
#### HIGH: Massive Service File Sizes
The coordinator-api service layer has extreme file sizes:
- 2,000 lines: `advanced_reinforcement_learning.py`
- 1,368 lines: `certification_service.py`
- 1,324 lines: `multi_modal_fusion.py`
- 1,159 lines: `agent_integration.py`
- 1,127 lines: `enterprise_integration.py`
- 10 more files over 800 lines
Single-responsibility principle is clearly violated.
#### MEDIUM: Type Coverage Gaps
While mypy is configured, it runs in very permissive mode:
- `check_untyped_defs = false`
- `disallow_intyped_defs = false`
- `disallow_untyped_defs = false`
- `no_implicit_optional = false`
9 generator overrides exist that completely ignore errors in coordinator-api modules:
```python
[[tool.mypy.overrides]]
module = ["apps.coordinator-api.src.app.routers.*", ...]
ignore_errors = true
```
#### MEDIUM: Cross-App Module Duplication
The same concerns are implemented independently across apps:
- **Config management**: 6 separate `config.py` files with no shared base
- **Database**: 4 separate `database.py` files
- **Validation**: Separate validation in each app
The core `aitbc/` package provides utilities but apps don't consistently use them.
#### MEDIUM: Per-App pyproject.toml Proliferation
15 per-app `pyproject.toml` files create:
- Dependency version drift across apps
- Inconsistent linting/formatting rules
- Complex CI/CD matrix
- Harder to maintain security patches across all packages
#### LOW: 25 Uncommitted Changes
Repository has 25 uncommitted changes, indicating active development without proper commit hygiene. This makes it harder to track what changes are in production.
#### LOW: TODO/FIXME/HACK Count is Zero
While the analysis found 0 TODO/FIXME/HACK/XXX/BUG comments, this likely indicates these markers are not used rather than the absence of known issues. This is a maintainability concern because there's no in-code tracking of technical debt.
---
## 6. Recommendations by Time Horizon
### SHORT-TERM (0-2 weeks) -- Quick Wins
| # | Recommendation | Effort | Impact | File(s) |
|---|---------------|--------|--------|---------|
| S1 | **Replace all print() with logger**: Run automated refactoring to replace `print()` with `logger.debug/info` across all production code. Use `sed`/`ruff` automated fix. | Low | Critical | All 925 instances |
| S2 | **Fix bare except clauses**: Replace `except:` with `except Exception:` at minimum. Prefer specific exceptions. | Low | High | 50 locations |
| S3 | **Remove or isolate stubs**: Move `apps/stubs/` to a separate repo or mark clearly as `examples/`. Remove from main test suite and CI/CD. | Low | Medium | `/opt/aitbc/apps/stubs/` |
| S4 | **Add `.coveragerc` or omit stubs from coverage**: Prevent stub code from diluting coverage metrics. | Low | Low | `pyproject.toml` |
| S5 | **Unify CORS configuration**: Create a single CORS policy in core `aitbc/` package, reference from all apps. | Low | Medium | 23 files |
| S6 | **Audit hardcoded secrets**: Review all 41 instances, move real secrets to environment variables/vault. | Medium | Critical | 41 locations |
| S7 | **Fix SQL injection risks**: Use parameterized queries in `multichain_ledger.py` and migration scripts. | Medium | High | 21 locations |
### MEDIUM-TERM (2-6 weeks) -- Structural Improvements
| # | Recommendation | Effort | Impact | File(s) |
|---|---------------|--------|--------|---------|
| M1 | **Decompose coordinator-api**: Split into bounded-context services: `rl-service`, `marketplace-router`, `analytics-service`, `certification-service`, `compliance-service`. Each gets its own deployable unit. | High | Critical | 338 files |
| M2 | **Implement shared config base class**: Create `aitbc.config.BaseSettings` that all apps inherit from, with common patterns for env var loading, secrets management, and validation. | Medium | High | 6 config files |
| M3 | **Add connection pool management**: Implement shared `aitbc.database.ConnectionPool` using `asyncpg` or SQLAlchemy async pool. All apps use the pool instead of creating independent connections. | Medium | High | 4 database files |
| M4 | **Implement distributed caching**: Deploy Redis cluster, add caching layer to coordinator-api for: API responses, session state, rate limiting counters, blockchain state reads. | Medium | High | `aitbc/caching.py` expansion |
| M5 | **Add rate limiting to all routers**: Apply `slowapi` limits to all 61 coordinator-api routers with tiered limits by endpoint sensitivity. | Medium | High | 61 router files |
| M6 | **Tighten mypy configuration**: Enable `check_untyped_defs`, `disallow_untyped_defs` incrementally. Remove `ignore_errors = true` overrides. Start with blockchain-node (best structured). | Medium | Medium | `pyproject.toml` |
| M7 | **Consolidate marketplace services**: Clarify boundaries between `marketplace`, `marketplace-service`, and `marketplace-service-debug`. Merge or clearly document the purpose of each. | Medium | Medium | 3 marketplace dirs |
| M8 | **Add background task system**: Implement Celery or ARQ for long-running tasks: RL training, analytics aggregation, blockchain event processing. | Medium | High | New infrastructure |
### LONG-TERM (1-3 months) -- Strategic Improvements
| # | Recommendation | Effort | Impact | File(s) |
|---|---------------|--------|--------|---------|
| L1 | **Implement API gateway pattern**: The `apps/api-gateway` (4 files, 425 LOC) should become the single entry point handling auth, rate limiting, routing, and load balancing for all backend services. | High | Critical | `apps/api-gateway/` |
| L2 | **Adopt event-driven architecture**: Implement message bus (Redis Streams, RabbitMQ, or NATS) for inter-service communication. Replace direct service-to-service calls with events. | High | High | Cross-cutting |
| L3 | **Implement feature flag system**: Populate `feature_flags.json` with actual flags. Use `aitbc/feature_flags.py` for gradual rollouts of: new RL algorithms, marketplace features, pricing changes. | Medium | Medium | `feature_flags.json` |
| L4 | **Add comprehensive observability**: Implement distributed tracing (OpenTelemetry), structured logging (structlog is already a dep), and metrics (Prometheus client is already a dep) across all services. | High | High | Cross-cutting |
| L5 | **Create shared test fixtures**: Extract common test setup into `packages/py/aitbc-test-utils/` with shared fixtures, factories, and mocks. Reduce duplication across 41 conftest.py files. | Medium | Medium | `tests/` |
| L6 | **Implement contract upgrade pattern**: Review Solidity contracts for proxy pattern usage. 20 contracts with no visible upgrade mechanism is a long-term risk. | High | Medium | `contracts/` |
| L7 | **Documentation-as-code**: Move operational runbooks from docs/ into executable skills. The 8 existing skills docs are a good start but should be executable. | Medium | Low | `docs/skills/` |
---
## 7. Risk Matrix
| Issue | Severity | Likelihood | Risk Score | Time to Fix |
|-------|----------|------------|------------|-------------|
| Coordinator-api monolith | High | High (growing) | **Critical** | 4-8 weeks |
| Hardcoded secrets | Critical | Medium | **Critical** | 1-2 weeks |
| print() in production | Medium | High (growing) | **High** | 1 week |
| Bare except clauses | High | Medium | **High** | 1 week |
| No rate limiting | High | Medium | **High** | 2-3 weeks |
| SQL injection risks | Critical | Low (internal) | **High** | 1-2 weeks |
| No connection pooling | Medium | High | **Medium** | 2-3 weeks |
| No distributed caching | Medium | High | **Medium** | 3-4 weeks |
| Stub code in main repo | Low | Low | **Low** | 1 week |
| Type coverage gaps | Medium | Medium | **Medium** | 4-6 weeks |
---
## 8. Files Most Likely to Change
Based on the analysis, these files/directories will be most impacted by the recommendations:
1. `/opt/aitbc/apps/coordinator-api/src/app/services/` -- Decomposition target (123 files)
2. `/opt/aitbc/apps/coordinator-api/src/app/routers/` -- Rate limiting, CORS (61 files)
3. `/opt/aitbc/apps/coordinator-api/src/app/config.py` -- Shared config base
4. `/opt/aitbc/apps/coordinator-api/src/app/database.py` -- Connection pooling
5. `/opt/aitbc/aitbc/caching.py` -- Distributed caching expansion
6. `/opt/aitbc/aitbc/middleware/` -- CORS unification
7. `/opt/aitbc/apps/stubs/` -- Removal/isolation
8. `/opt/aitbc/pyproject.toml` -- mypy tightening, coverage config
9. `/opt/aitbc/apps/wallet/src/app/chain/multichain_ledger.py` -- SQL injection fix
10. `/opt/aitbc/apps/agent-coordinator/src/coordinator.py` -- print() removal
---
## 9. Verification Steps
After implementing recommendations, verify with:
1. **print() elimination**: `grep -rn "print(" apps/ packages/ | grep -v "node_modules" | grep -v "__pycache__" | wc -l` should return 0
2. **Bare except elimination**: `grep -rn "^\s*except\s*:" apps/ packages/ | wc -l` should return 0
3. **Test suite**: `pytest --cov=apps --cov=packages --cov=cli --cov-fail-under=50` should pass
4. **Security scan**: `bandit -r apps/ packages/ cli/` should return 0 issues
5. **Type check**: `mypy apps/ packages/ cli/` should pass with tightened config
6. **Rate limiting**: `curl -s -o /dev/null -w "%{http_code}" http://localhost:9001/api/...` repeated rapidly should return 429
7. **Health checks**: All 28 systemd services should report `active (running)` and `/health` endpoints should return 200
---
## 10. Open Questions
1. **Stubs purpose**: Are the 65 stub files in `apps/stubs/` planned for implementation, or should they be archived? Who owns the decision?
2. **Coordinator-api decomposition**: What is the preferred decomposition strategy -- by domain (RL, marketplace, analytics) or by function (read vs write)?
3. **Redis deployment**: Is there an existing Redis cluster, or does one need to be provisioned?
4. **Background task system**: Is there a preference between Celery, ARQ, or FastAPI BackgroundTasks for the task queue?
5. **Contract upgrade strategy**: Are the Solidity contracts expected to be upgradeable, or is immutability intentional?
6. **Marketplace service boundary**: What is the intended relationship between `marketplace`, `marketplace-service`, and `marketplace-service-debug`?
7. **Feature flag ownership**: Who manages the feature flag lifecycle -- product or engineering?

97
.hermes/plans/2026-05-12_104500-coordinator-decomposition.md
View File

@@ -1,97 +0,0 @@
# Coordinator-API Decomposition Plan
## Current State
- **1 monolith**: apps/coordinator-api/src/app/
- 89 service files, 46,594 LOC
- 53 routers
- 51 files over 500 LOC
- Largest: agent_integration.py (1,159 LOC)
## Decomposition Strategy: Bounded Contexts
Based on domain analysis, split into 7 microservices:
1. **agent-management** (agent lifecycle, performance, communication)
2. **blockchain** (chain operations, transactions, smart contracts)
3. **computing** (GPU, resources, marketplace for compute)
4. **enterprise** (integration, scalability, compliance)
5. **identity** (authentication, authorization, agents identity)
6. **payment** (billing, transactions, financial operations)
7. **ai-models** (AI services, RL, multi-modal fusion)
Each will be a separate FastAPI app with:
- Its own routers/, services/, models/
- Shared libraries: app.core.config, app.core.logging, app.core.database
- Independent systemd service
- Clear API boundaries
## Implementation Phases
### Phase 1: Infrastructure Setup (Week 1-2)
- Create apps/ directory structure: agent-management/, blockchain/, etc.
- Create shared core library: apps/coordinator-api/src/app/core/
- Extract common config, logging, DB session, exceptions
- Update pyproject.toml to support multiple packages
### Phase 2: Extract Agent Management (Week 2-3)
- Move agent_*.py, agent_service_marketplace.py -> agent-management
- Move agent_communication.py, agent_performance_service.py -> agent-management
- Create new systemd service for agent-management
- Update reverse proxy (nginx) routes
### Phase 3: Extract Blockchain (Week 3-4)
- Move blockchain_context.py, contract_service.py, transaction_service.py -> blockchain
- Move escrow.py, persistent_spending_tracker.py, etc.
- Create blockchain systemd service
### Phase 4: Extract Enterprise (Week 4-5)
- Move enterprise_integration.py, compliance_engine.py, certification related -> enterprise
- Create enterprise systemd service
### Phase 5: Extract Identity (Week 5-6)
- Move auth/identity service files -> identity
- Create identity systemd service
### Phase 6: Extract AI Models (Week 6-7)
- Move advanced_*.py, multi_modal_fusion, ai verification -> ai-models
- Create ai-models systemd service
### Phase 7: Extract Computing & Payment (Week 7-8)
- Move gpu, resource, payment services to their own packages
### Phase 8: Final Integration (Week 8-9)
- Update all clients to use new service endpoints
- Test inter-service communication
- Update documentation
- Deprecate old monolith
## Files to Create/Modify
### New shared core (apps/coordinator-api/src/app/core/)
- config.py (extracted from existing config.py)
- logging.py (centralized logger setup)
- database.py (SQLAlchemy session, Base)
- exceptions.py (common exceptions)
- security.py (auth dependencies)
### New service apps (47 directories total)
Each: apps/<service>/src/app/{routers,services,models,main.py}
### Modified files
- Root pyproject.toml: add service packages
- Systemd: add 7 new .service files
- Nginx config: new upstream blocks
- Docker compose: add 7 new containers
- Monitoring: new service endpoints for health
## Rollback Plan
- Keep original monolith running alongside new services during transition
- Use feature flags to route traffic
- Comprehensive integration tests before cutover
## Success Criteria
- Each service < 3,000 LOC (target 1,500)
- Each service independently deployable
- API contracts stable and documented
- CI/CD per service

239
.hermes/plans/2026-05-12_142930-agent-management-extraction.md
View File

@@ -1,239 +0,0 @@
# Agent-Management Service Extraction Plan
## Overview
Extract the agent-related functionality from the coordinator-api monolith into a standalone microservice while maintaining operational continuity.
## Current State
**Monolith:** `apps/coordinator-api/src/app/`
- Services: 46,594 LOC across 89 files
- Domain layer: `domain/` contains all business entities (Agent, AgentExecution, AgentStatus, etc.)
- Target agent files to extract: **18 files** (6 routers, 12 services)
- Largest files: agent_service.py (1,159 LOC), agent_integration.py (1,117 LOC), agent_communication.py (988 LOC)
## Bounded Context: Agent-Management
**Responsibility:** AI agent lifecycle, orchestration, performance tracking, security, and marketplace registry.
**In-Scope Files:**
### Services (12)
```
services/agent_service.py (1,159 LOC)
services/agent_integration.py (1,117 LOC)
services/agent_communication.py (988 LOC)
services/agent_orchestrator.py
services/agent_performance_service.py
services/agent_security.py
services/agent_portfolio_manager.py
services/agent_service_marketplace.py
services/advanced_rl/agents.py (+ sub-agents: ppo_agent.py, rainbow_dqn_agent.py, sac_agent.py)
```
### Routers (6)
```
routers/agent_router.py
routers/agent_integration_router.py
routers/agent_performance.py
routers/agent_creativity.py
routers/agent_security_router.py
routers/services.py (agent services listing endpoint)
```
## Critical Dependencies
1. **Domain Layer** (`app.domain`)
- All agent services import from `..domain.agent` (AgentExecution, AgentStatus, AIAgentWorkflow, etc.)
- Solution: Keep domain/ in monolith for now; new service imports via a **shared-domain package** to be created
- Create `apps/shared-domain/src/app/domain/` as a symlink or copy that both services can import
- Long-term: Extract entire domain layer to shared-domain package
2. **aitbc package**
- Already available as root package. Use directly.
3. **SQLModel/SQLAlchemy**
- Already in dependencies via root pyproject.toml
4. **Other monolith services**
- Some routers may call agent endpoints. These will need to be updated to use HTTP client to new service (Phase 3 internal routing via nginx)
## Implementation Steps
### Step 0: Prepare Shared Domain Package (Prerequisite)
- Create `apps/shared-domain/src/app/domain/`
- Copy all files from coordinator-api's `domain/` EXCEPT non-agent ones if desired
- Or simpler: symlink entire domain directory: `ln -s ../../coordinator-api/src/app/domain apps/shared-domain/src/app/`
- Update imports in new service to use `from shared-domain.app.domain.agent import ...`
- Add `shared-domain` to pyproject.toml dependencies in consuming services
**Recommendation:** Use symlink for rapid iteration, then formalize package later.
### Step 1: Create agent-management Service Skeleton
```
apps/agent-management/
├── pyproject.toml
├── README.md
└── src/
└── app/
├── __init__.py
├── main.py
├── core/
│ ├── __init__.py
│ ├── config.py (import from shared-core)
│ ├── logging.py (import from shared-core)
│ └── database.py (import from shared-core)
├── domain/ → symlink to ../../shared-domain/src/app/domain
├── routers/
│ ├── __init__.py
│ ├── agent_router.py (copied & adapted)
│ ├── agent_integration_router.py
│ ├── agent_performance.py
│ ├── agent_creativity.py
│ ├── agent_security_router.py
│ └── services.py
└── services/
├── __init__.py
├── agent_service.py
├── agent_orchestrator.py
├── agent_communication.py
├── agent_performance_service.py
├── agent_security.py
├── agent_integration.py
├── agent_portfolio_manager.py
├── agent_service_marketplace.py
└── advanced_rl/
├── __init__.py
├── agents.py
└── ppo_agent.py, rainbow_dqn_agent.py, sac_agent.py
```
### Step 2: Adapt Code for Service Boundaries
**Changes needed per file:**
- Update all `from ..domain.agent import X` to `from shared-domain.app.domain.agent import X`
- Remove any imports from other monolith services (e.g., `from ..services.other_service import X`)
- Replace internal service calls with HTTP client calls or event bus (defer to later phase)
- Update `ServiceSettings` to use agent-management specific defaults (port 8012)
- Add health check endpoint (already in template)
- Verify database setup: AgentExecution etc use shared Base. Need to call `Base.metadata.create_all(bind=engine)` on startup
**Special Case: advanced_rl/**
- These are AI model inference services. Consider moving to `ai-models` service instead.
- For now, keep in agent-management to maintain functionality.
### Step 3: Update Monolith to Proxy Requests (During Transition)
**Option A: Nginx Routing**
- Add nginx upstream for agent-management on port 8012
- Change coordinator-api routes for `/api/v1/agent/*` to proxy to agent-management
- Monolith no longer handles agent endpoints
**Option B: In-app Redirection**
- Keep routers in monolith but replace handlers with `HTTPClient` calls to new service
- More gradual migration but adds latency
**Recommendation:** Option A - cleaner separation, easier to rollback.
### Step 4: Create Systemd Service
```
/etc/systemd/system/aitbc-agent-management.service
[Unit]
Description=AITBC Agent Management Service
After=network.target
[Service]
Type=simple
User=aitbc
WorkingDirectory=/opt/aitbc/apps/agent-management
Environment=PATH=/opt/aitbc/venv/bin
Environment=PYTHONPATH=/opt/aitbc
ExecStart=/opt/aitbc/venv/bin/uvicorn app.main:app --host 127.0.0.1 --port 8012
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
```
### Step 5: Database Migration
- Agent domain models likely already have tables defined via SQLModel
- In `main.py` startup event, call `Base.metadata.create_all(bind=engine)` to ensure tables exist
- Ensure the new service uses same database as monolith (coordinator.db) initially
- Later: separate database (Phase 8)
### Step 6: Integration Testing
1. Start agent-management service
2. Verify health endpoint: `curl http://localhost:8012/health`
3. Test agent creation via API
4. Verify coordinator-api can still access agent data (through new service or direct DB if keeping shared DB)
5. Run existing integration tests against new service
### Step 7: Update Coordinator-API
- Remove the 18 extracted files from monolith
- Remove domain/agent related imports from remaining monolith services if they now use agent-management API
- Update any remaining references to agent endpoints to use HTTP client or nginx proxy
### Step 8: Documentation & Monitoring
- Update README with agent-management API docs
- Add metrics endpoint if enabled
- Update deployment scripts
## Rollback Plan
1. Keep monolith files in git history (do not delete, just move)
2. Keep nginx config either/or - can revert upstream routing
3. Database shared initially, so data is accessible to both
4. Systemd service can be disabled; monolith still runs
## Success Criteria
- [ ] Agent-management service starts and health check passes on port 8012
- [ ] Can create/query agents via API
- [ ] Existing coordinator-api functionality that depends on agents still works
- [ ] No errors in logs during integration test
- [ ] Systemd service auto-restarts on failure
## Open Questions
1. **RL Agents**: Should advanced_rl be part of agent-management or ai-models?
- Recommendation: Keep in agent-management for now (AI agent inference is part of agent runtime). Can split later if ai-models becomes a separate inference service.
2. **Database**: Separate or shared?
- Phase 1: Shared (same coordinator.db) for simplicity
- Phase 8: Split to dedicated agent-management database
3. **Cross-service calls**: Currently agent integration uses other services directly (imports). Need to replace with HTTP or event bus.
- Defer until Phase 8 (Final Integration) to avoid breaking existing flow
4. **Domain extraction**: The domain models are currently in monolith. Should we extract entire domain to a package?
- Immediate need: Create shared-domain package (symlink) to break import cycle
- Future: Extract domain to true package with independent version
## Timeline Estimate
- Step 0 (shared-domain): 2h
- Step 1 (skeleton): 4h
- Step 2 (adaptation): 8h (bulk of work - fixing imports, resolving dependencies)
- Step 3 (nginx routing): 2h
- Step 4 (systemd): 1h
- Step 5 (DB): 1h
- Step 6 (testing): 4h
- Step 7 (monolith cleanup): 4h
- Step 8 (docs): 2h
**Total: ~28 hours (3-4 days)**
## Risks
- Hidden dependencies on other monolith services may cause runtime import errors
- Domain models may have cross-references that require co-migration
- Database migrations may be needed if agent tables don't exist yet
- Existing integration tests may fail and need updating
- Breaking changes if API contracts differ from original

218
.hermes/plans/2026-05-12_150000-tighten-mypy-config.md
View File

@@ -1,218 +0,0 @@
# Tighten Mypy Configuration Plan
## Current State
**Root pyproject.toml [tool.mypy] settings:**
```toml
warn_return_any = true
warn_unused_configs = true
check_untyped_defs = false
disallow_incomplete_defs = false
disallow_untyped_defs = false
disallow_untyped_decorators = false
no_implicit_optional = false
warn_redundant_casts = false
warn_unused_ignores = false
warn_no_return = true
warn_unreachable = false
strict_equality = false
```
**Overrides:**
- Heavy libraries (torch, cv2, pandas, numpy, web3, etc.) are `ignore_missing_imports = true`
- Coordiator-api modules are `ignore_errors = true` (catch-all)
This is **extremely permissive** - essentially just warns on return_any and missing configs. It does not enforce:
- Function argument/return type completeness
- Avoiding implicit `Any`
- Avoiding unnecessary type: ignore comments
- Detecting unreachable code
- Strict equality checks (None vs False)
## Proposed Tightening Phases
### Phase 1: Enable Foundational Checks (Low Effort, High Value)
Target: enable 4 key options that catch real bugs with minimal friction
```toml
disallow_untyped_defs = true
disallow_incomplete_defs = true
warn_redundant_casts = true
warn_unused_ignores = true
```
**Impact:**
- Functions must have complete type signatures (all args+returns typed)
- Redundant cast() calls will be flagged
- Unused `# type: ignore` comments will be flagged
- Minimal code changes required (most functions already typed)
**Estimated effort:**
- 1 hour to update config
- 2-4 hours to fix violations in production code
- Total: ~1 day
**Validation:**
- Run `mypy apps` and ensure 0 errors
- Keep existing overrides for external libraries and coordinator-api
### Phase 2: Stricter Optional Handling (Medium Effort)
Enable:
```toml
no_implicit_optional = true
warn_unreachable = true
strict_equality = true
```
**Impact:**
- Variables defaulting to `None` must be explicitly `Optional[...]`
- Unreachable code will be flagged (dead code detection)
- Equality comparisons with None must use `is` not `==`
**Estimated effort:** 2-3 days to fix violations across codebase
### Phase 3: Gradual Per-Module Strictness (Long-term)
- Move coordinator-api out of catch-all `ignore_errors`
- Add per-module overrides as we achieve correctness
- Eventually remove `ignore_errors` blanket
**Estimated effort:** Ongoing as part of decomposition
## Implementation Steps
### Step 1: Backup Current Config
```bash
cp pyproject.toml pyproject.toml.backup
```
### Step 2: Update Root Configuration
Modify `/opt/aitbc/pyproject.toml` [tool.mypy] section:
```diff
[tool.mypy]
python_version = "3.13"
warn_return_any = true
warn_unused_configs = true
check_untyped_defs = false
-disallow_incomplete_defs = false
-disallow_untyped_defs = false
+disallow_incomplete_defs = true
+disallow_untyped_defs = true
disallow_untyped_decorators = false
no_implicit_optional = false
warn_redundant_casts = false
warn_unused_ignores = false
warn_no_return = true
warn_unreachable = false
strict_equality = false
```
### Step 3: Run Mypy and Collect Errors
```bash
cd /opt/aitbc
venv/bin/mypy apps --show-error-codes --no-color-output > mypy_errors.txt 2>&1
```
### Step 4: Categorize Errors
Typical violations we'll see:
- `Function is missing a return type annotation` (from disallow_untyped_defs)
- `Function is missing a type annotation for one or more arguments` (from disallow_untyped_defs)
- `Class is missing type parameters for generic type` (rare)
- `dict, list, etc. used without type parameters` (from disallow_incomplete_defs)
- `Redundant cast to X` (from warn_redundant_casts)
- `Unused "type: ignore" comment` (from warn_unused_ignores)
### Step 5: Fix in Order of Impact
**A. Add missing type annotations to functions**
- Priority: functions in shared-core, services, routers
- Use explicit return types; if truly dynamic, use `-> Any` (but rarely needed)
- Example:
```python
def get_engine(settings): # BEFORE
def get_engine(settings: ServiceSettings) -> Engine: # AFTER
```
**B. Add generic type parameters**
- `list` -> `List[str]` or `list[int]`
- `dict` -> `Dict[str, Any]`
- Use `from typing import List, Dict`
**C. Remove redundant casts**
- Delete `cast(Type, value)` if type is already clear to mypy
- Use `reveal_type(value)` to check actual inferred type before removing
**D. Remove unused type: ignore**
- Some `# type: ignore` comments are legacy and no longer needed
- Delete them; if mypy still fails, leave or fix underlying issue
### Step 6: Iterate and Validate
After fixing categories, re-run mypy. Continue until `mypy apps` exits with code 0.
**Note:** We preserve `ignore_missing_imports` for heavy libraries, and `ignore_errors` for coordinator-api (since we're deferring decomposition).
### Step 7: Add CI Enforcement
Update pre-commit hooks or CI to run mypy on PRs:
```yaml
# .pre-commit-config.yaml or GitHub Actions
- repo: local
hooks:
- id: mypy
name: mypy
entry: mypy apps
language: system
pass_filenames: false
```
## Rollback Plan
If the effort becomes too large:
1. Revert pyproject.toml from backup
2. Keep per-module `# mypy: ignore-errors` as needed
3. Approach incrementally: enable one flag at a time
## Success Criteria
- `mypy apps` completes with 0 errors
- No new type: ignore comments added without explanation
- Production code has complete type signatures
- CI pipeline includes mypy check
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Overwhelming number of errors | Enable flags incrementally (2 at a time), fix in batches by module |
| Breaking existing functionality by incorrect type fixes | Run test suite after each batch; use `reveal_type` to debug |
| Third-party library types incompatible | Keep `ignore_missing_imports` for those packages |
| Coordinator-api too messy to fix now | Keep `ignore_errors` override; revisit after decomposition |
## Related Tasks
- **Decompose coordinator-api** - Once strict mypy is in place, easier to validate new services
- **Shared-core library** - Strict typing ensures compatibility across services
- **Connection pooling** - Use proper typed database sessions
## Open Questions
1. Should we also enable `strict` mode for new services? (Probably yes)
2. Should we add type-checking to pre-commit hook for changed files only? (Yes, use `mypy --files <changed>`)
3. How to handle legacy coordinator-api code? (Keep ignore_errors for now)
## Estimated Timeline
- **0-2 days:** Implement Phase 1, fix immediate violations
- **3-7 days:** Address accumulated type errors, reach clean mypy
- **Week 2:** Add CI enforcement, document guidelines
- **Ongoing:** Maintain strict typing in new code
## References
- Mypy configuration: https://mypy.readthedocs.io/en/stable/config_file.html
- Strict mode: https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-strict