diff --git a/.hermes/plans/2026-05-12_102100-aitbc-codebase-analysis.md b/.hermes/plans/2026-05-12_102100-aitbc-codebase-analysis.md deleted file mode 100644 index be810eff..00000000 --- a/.hermes/plans/2026-05-12_102100-aitbc-codebase-analysis.md +++ /dev/null @@ -1,411 +0,0 @@ -# AITBC Codebase Analysis & Recommendations Plan - -**Date:** 2026-05-12 -**Scope:** Full analysis of /opt/aitbc/ -- architecture, performance, security, maintainability -**Codebase Version:** 0.6.0 -**Python:** 3.13.5 - ---- - -## Executive Summary - -AITBC is a large monorepo (~212K LOC Python, 75K LOC Solidity, 75K LOC CLI) implementing a blockchain-based AI agent compute network. The system has strong foundational infrastructure (28 systemd services, PoA consensus, multi-chain support, 20 Solidity contracts) but carries significant technical debt: a bloated coordinator-api (117K LOC, 91 files >500 lines), 925 print() statements in production code, 50 bare except clauses, 41 potentially hardcoded secrets, duplicated config/database modules across apps, and 14.3K LOC of stubs that may be dead code. The core blockchain-node (27K LOC) is well-structured, but the coordinator-api has grown into a monolith that needs decomposition. - ---- - -## 1. Codebase Scale Metrics - -| Component | Files | LOC | Notes | -|-----------|-------|-----|-------| -| **apps/** | 840 | 211,844 | Main application code | -| **cli/** | 232 | 74,805 | Unified CLI tool | -| **aitbc/** (core pkg) | 52 | 11,190 | Shared utilities | -| **packages/py/** | 42 | 5,458 | SDK packages | -| **contracts/** | 490 | 74,589 | Solidity contracts | -| **tests/** | 56 files | ~789 test functions | Good test count | -| **stubs/** | 65 | 14,300 | Placeholder services | -| **systemd/** | 28 | - | Service definitions | - -**App breakdown (largest):** -- `coordinator-api`: 338 files, 116,568 LOC (55% of all app code) -- `blockchain-node`: 127 files, 26,895 LOC -- `zk-circuits`: 52 files, 15,160 LOC -- `stubs`: 65 files, 14,300 LOC -- `pool-hub`: 48 files, 5,461 LOC -- `agent-coordinator`: 45 files, 9,983 LOC -- `exchange`: 19 files, 5,144 LOC -- `wallet`: 27 files, 3,482 LOC - ---- - -## 2. Architecture Analysis - -### 2.1 Current Architecture Pattern - -The codebase follows a **monorepo with per-app packaging** pattern: -- Root `pyproject.toml` at `/opt/aitbc/pyproject.toml` with shared deps -- 15 per-app `pyproject.toml` files for independent packaging -- 28 systemd services for process management -- Core shared library at `/opt/aitbc/aitbc/` (52 files) -- 3 Python packages in `/opt/aitbc/packages/py/` (crypto, sdk, agent-sdk) - -### 2.2 Strengths - -1. **Well-structured blockchain-node**: Clear separation of concerns with `consensus/`, `contracts/`, `network/`, `economics/`, `state/` subdirectories. Largest file is 1,278 lines (RPC router) which is reasonable. -2. **Solid middleware layer**: Core package provides `error_handler`, `performance`, `request_id`, `validation` middleware -- all properly implemented. -3. **Database migration support**: Alembic configs for blockchain-node (7 migrations), coordinator-api (2), pool-hub (5). -4. **Comprehensive infrastructure**: Terraform (15 files), Kubernetes (6 manifests), nginx config, Prometheus dashboards, Gitea CI/CD. -5. **Circuit breaker pattern**: Implemented in PoA consensus for resilience. -6. **Good test infrastructure**: 789 test functions across 56 files, 41 conftest.py files, pytest markers for unit/integration/e2e/slow. - -### 2.3 Architecture Issues - -#### CRITICAL: Coordinator-API Monolith (117K LOC) - -The coordinator-api at 116,568 LOC across 338 files is the single biggest architectural risk: -- **91 files over 500 lines** (27% of all files) -- Largest file: `advanced_reinforcement_learning.py` at 2,000 lines -- Service layer has 99 files + 24 sub-module files = 123 service files -- Router layer has 61 files -- Domain layer has 32 files -- This service alone is 55% of all application code - -**Root cause**: The coordinator-api has accumulated features without decomposition. It handles RL agents, marketplace, trading, certification, compliance, analytics, payments, staking, governance, AMM, AI surveillance, multi-language support, ZK proofs, and more -- all in one deployable unit. - -#### HIGH: Duplicated Cross-App Modules - -Each app independently implements: -- `config.py` in 6 different apps (blockchain-node, coordinator-api, agent-coordinator, blockchain-event-bridge, pool-hub) -- `database.py` in 4 apps (blockchain-node, coordinator-api, exchange, pool-hub) -- No shared database connection pooling across apps - -#### HIGH: Stubs Directory (14.3K LOC of Dead Code?) - -The `apps/stubs/` directory contains 65 files across 68 subdirectories totaling 14,300 LOC. These appear to be placeholder implementations. Including `compliance-service`, `exchange-integration`, `global-ai-agents`, `global-infrastructure`, `multi-region-load-balancer`, `plugin-analytics`, `plugin-marketplace`, `plugin-registry`, `plugin-security`, `simple-explorer`, `trading-engine`. Each has its own tests, suggesting they were meant to be real services but remain as stubs. - -#### MEDIUM: Missing Package Structure - -Only 128 of 1,026 app directories have `__init__.py` files (12.5%). This means most directories are not proper Python packages, which breaks relative imports and makes the codebase harder to navigate programmatically. - -#### MEDIUM: Overlapping Marketplace Services - -Three separate marketplace implementations exist: -- `apps/marketplace` (9 files, 1,762 LOC) -- appears to be the primary marketplace -- `apps/marketplace-service` (10 files, 875 LOC) -- unclear differentiation -- `apps/marketplace-service-debug` (8 files, 821 LOC) -- debug variant - -The relationship and boundary between these three is unclear. - -#### LOW: Coordinator-API Entry Point is a Stub - -`/opt/aitbc/apps/coordinator-api/src/app.py` is a single line: `# Import the FastAPI app from main.py for compatibility`. There is no `main.py` in the same directory -- the actual entry point appears to be elsewhere. - ---- - -## 3. Performance Analysis - -### 3.1 Strengths - -1. **Async-first architecture**: 362 files use `async def`, consistent with FastAPI best practices -2. **FastAPI Depends for DI**: 94 files use FastAPI's dependency injection system -3. **LRU caching implementation**: Core package provides `LRU cache` with hit/miss tracking -4. **Performance middleware**: `PerformanceLoggingMiddleware` tracks request duration and adds `X-Process-Time` headers -5. **Adaptive sync**: Blockchain sync uses tiered batch sizing (10K+ blocks: 500-1000 batch) -6. **Circuit breaker**: PoA consensus implements circuit breaker for fault tolerance - -### 3.2 Performance Issues - -#### HIGH: No Distributed Caching Layer - -Despite Redis being in `pyproject.toml` dependencies, grep found only 21 files mentioning Redis (mostly multi-language services and docs). There is no Redis-based caching layer for: -- API response caching -- Session storage -- Rate limiting state -- Blockchain state caching - -This means every request hits the database directly. - -#### HIGH: No Connection Pooling - -Despite `asyncpg` and `sqlalchemy` with async support being dependencies, there is no visible connection pool configuration across apps. Each app creates its own database connections independently. - -#### HIGH: No Background Task System - -No Celery, RQ, or FastAPI BackgroundTasks usage found outside of 3 files. Long-running operations (blockchain sync, analytics processing, RL training) appear to be handled inline or via systemd timers rather than a proper task queue. - -#### MEDIUM: No API Rate Limiting - -`slowapi` and `limits` are in production dependencies but only found in 3 files in coordinator-api routers. Most endpoints have no rate limiting. - -#### MEDIUM: No Request Batching - -The coordinator-api receives requests for 61 different router domains but implements no request batching for high-throughput operations like marketplace queries or analytics. - -#### LOW: Logging Performance - -The `PerformanceLoggingMiddleware` logs every request synchronously with `logger.info()`. At high throughput, this creates I/O bottlenecks. Should use `logger.debug()` or sampling. - ---- - -## 4. Security Analysis - -### 4.1 Strengths - -1. **Security hardening module**: `aitbc/security_hardening.py` (396 lines) provides input validation, sanitization, ethereum address validation, URL validation, email validation -2. **JWT infrastructure**: 30 files reference JWT authentication -3. **CORS configuration**: 23 files configure CORS policies -4. **HTTPS enforcement**: Security headers module (`aitbc/security_headers.py`) present -5. **Encryption at rest**: `database_encryption.py` in blockchain-node provides SQLCipher support -6. **Multi-sig wallets**: Mentioned in README as implemented -7. **Audit logging**: `audit_logging.py` service in coordinator-api (584 lines) -8. **Dev dependencies**: `bandit` (security linter) and `safety` (dependency vulnerability scanner) configured -9. **Pre-commit hooks**: `.pre-commit-config.yaml` configured -10. **Error handler middleware**: Properly returns generic 500 messages without leaking internals - -### 4.2 Security Issues - -#### CRITICAL: Potentially Hardcoded Secrets (41 instances) - -41 files contain patterns matching potential hardcoded secrets, API keys, or tokens: -- `coordinator-api/.../cross_chain_integration.py:191`: `private_key = "mock_private_key"` -- `coordinator-api/.../wallet_service.py:72`: `encrypted_private_key = "[ENCRYPTED_MOCK]"` -- Multiple test files with `api_key="test_key"` patterns - -While some are test files, production code also contains these patterns. Need audit to distinguish real secrets from test fixtures. - -#### HIGH: 50 Bare Except Clauses - -50 instances of bare `except:` in production code across: -- coordinator-api: 44 instances -- exchange: 1 instance -- Others: 5 instances - -Bare excepts catch `SystemExit`, `KeyboardInterrupt`, and `GeneratorExit`, making the application unkillable in some error scenarios and hiding real errors. - -#### HIGH: SQL Injection Risks (21 instances) - -21 files use f-strings in SQL execute statements: -- `multichain_ledger.py:98,110,123`: `cursor.execute(f"""...`) with table name interpolation -- Various migration scripts using `f"SELECT * FROM {table_name}"` - -While some are internal migration scripts, the `multichain_ledger.py` pattern is dangerous if table names come from user input. - -#### HIGH: Insufficient Rate Limiting - -Only 3 coordinator-api routers implement rate limiting. The remaining 58+ routers have no protection against: -- Brute force authentication attempts -- API abuse -- DDoS at the application layer - -#### MEDIUM: Feature Flags Not Utilized - -`/opt/aitbc/feature_flags.json` is an empty object `{}`. Despite having `aitbc/feature_flags.py` (a feature flag system), no features are actually flagged. This means: -- No gradual rollout capability -- No kill switches for problematic features -- No A/B testing infrastructure - -#### MEDIUM: CORS Configuration Cents - -23 files configure CORS but the configuration is scattered and inconsistent. No centralized CORS policy management. - -#### MEDIUM: Dependency Vulnerability Scanning Not Enforced - -While `safety` and `bandit` are in dev dependencies and pre-commit config, there is no evidence of CI/CD pipeline enforcement of security scanning. The Gitea workflows reference "security scanning optimized for changed files" but this should be comprehensive. - -#### LOW: Debug Mode References - -Several debug patterns found: -- `marketplace-service-debug` app (8 files) appears to be a debug variant -- `mock_coordinator.py` in blockchain-node observability - ---- - -## 5. Maintainability Analysis - -### 5.1 Strengths - -1. **Comprehensive documentation**: Full docs directory with architecture, deployment, security, and operations guides -2. **Standardized quality tools**: ruff (line length 127), mypy (relaxed mode for gradual adoption), black, isort, pre-commit -3. **Good test coverage infrastructure**: 789 test functions, pytest with coverage (50% minimum threshold), markers for test types -4. **API versioning**: `aitbc/api_versioning.py` provides API versioning support -5. **Distributed tracing**: `aitbc/distributed_tracing.py` exists for observability -6. **Blue-green deployment**: `aitbc/blue_green_deployment.py` for zero-downtime deploys -7. **Domain-driven design**: coordinator-api uses `domain/` subdirectory with bounded contexts -8. **Repository pattern**: coordinator-api has `repositories/` directory -9. **Skills documentation**: 8 operational skills docs for agents - -### 5.2 Maintainability Issues - -#### CRITICAL: 925 print() Statements in Production Code - -925 instances of `print()` in `/opt/aitbc/apps/` and `/opt/aitbc/packages/`. This is the most impactful maintainability issue: -- Prints bypass structured logging, making log aggregation impossible -- Prints cannot be filtered by severity level -- Prints go to stdout, not to configured log handlers -- In async code, prints can interleave unpredictably - -Most concentrated in: -- `agent-coordinator/src/coordinator.py` (many debug prints) -- `agent-compliance/src/compliance_agent.py` -- `blockchain-node/scripts/` (expected for scripts, but still 925 total) - -#### HIGH: 65 Stub Services (14.3K LOC) - -The `apps/stubs/` directory is a significant maintenance burden: -- Each stub has its own tests (increasing test suite time) -- Each stub has its own `pyproject.toml` (15 total per-app pyproject files) -- Creates confusion about which services are real vs. placeholders -- Must be maintained/skipped during CI/CD runs - -#### HIGH: Massive Service File Sizes - -The coordinator-api service layer has extreme file sizes: -- 2,000 lines: `advanced_reinforcement_learning.py` -- 1,368 lines: `certification_service.py` -- 1,324 lines: `multi_modal_fusion.py` -- 1,159 lines: `agent_integration.py` -- 1,127 lines: `enterprise_integration.py` -- 10 more files over 800 lines - -Single-responsibility principle is clearly violated. - -#### MEDIUM: Type Coverage Gaps - -While mypy is configured, it runs in very permissive mode: -- `check_untyped_defs = false` -- `disallow_intyped_defs = false` -- `disallow_untyped_defs = false` -- `no_implicit_optional = false` - -9 generator overrides exist that completely ignore errors in coordinator-api modules: -```python -[[tool.mypy.overrides]] -module = ["apps.coordinator-api.src.app.routers.*", ...] -ignore_errors = true -``` - -#### MEDIUM: Cross-App Module Duplication - -The same concerns are implemented independently across apps: -- **Config management**: 6 separate `config.py` files with no shared base -- **Database**: 4 separate `database.py` files -- **Validation**: Separate validation in each app - -The core `aitbc/` package provides utilities but apps don't consistently use them. - -#### MEDIUM: Per-App pyproject.toml Proliferation - -15 per-app `pyproject.toml` files create: -- Dependency version drift across apps -- Inconsistent linting/formatting rules -- Complex CI/CD matrix -- Harder to maintain security patches across all packages - -#### LOW: 25 Uncommitted Changes - -Repository has 25 uncommitted changes, indicating active development without proper commit hygiene. This makes it harder to track what changes are in production. - -#### LOW: TODO/FIXME/HACK Count is Zero - -While the analysis found 0 TODO/FIXME/HACK/XXX/BUG comments, this likely indicates these markers are not used rather than the absence of known issues. This is a maintainability concern because there's no in-code tracking of technical debt. - ---- - -## 6. Recommendations by Time Horizon - -### SHORT-TERM (0-2 weeks) -- Quick Wins - -| # | Recommendation | Effort | Impact | File(s) | -|---|---------------|--------|--------|---------| -| S1 | **Replace all print() with logger**: Run automated refactoring to replace `print()` with `logger.debug/info` across all production code. Use `sed`/`ruff` automated fix. | Low | Critical | All 925 instances | -| S2 | **Fix bare except clauses**: Replace `except:` with `except Exception:` at minimum. Prefer specific exceptions. | Low | High | 50 locations | -| S3 | **Remove or isolate stubs**: Move `apps/stubs/` to a separate repo or mark clearly as `examples/`. Remove from main test suite and CI/CD. | Low | Medium | `/opt/aitbc/apps/stubs/` | -| S4 | **Add `.coveragerc` or omit stubs from coverage**: Prevent stub code from diluting coverage metrics. | Low | Low | `pyproject.toml` | -| S5 | **Unify CORS configuration**: Create a single CORS policy in core `aitbc/` package, reference from all apps. | Low | Medium | 23 files | -| S6 | **Audit hardcoded secrets**: Review all 41 instances, move real secrets to environment variables/vault. | Medium | Critical | 41 locations | -| S7 | **Fix SQL injection risks**: Use parameterized queries in `multichain_ledger.py` and migration scripts. | Medium | High | 21 locations | - -### MEDIUM-TERM (2-6 weeks) -- Structural Improvements - -| # | Recommendation | Effort | Impact | File(s) | -|---|---------------|--------|--------|---------| -| M1 | **Decompose coordinator-api**: Split into bounded-context services: `rl-service`, `marketplace-router`, `analytics-service`, `certification-service`, `compliance-service`. Each gets its own deployable unit. | High | Critical | 338 files | -| M2 | **Implement shared config base class**: Create `aitbc.config.BaseSettings` that all apps inherit from, with common patterns for env var loading, secrets management, and validation. | Medium | High | 6 config files | -| M3 | **Add connection pool management**: Implement shared `aitbc.database.ConnectionPool` using `asyncpg` or SQLAlchemy async pool. All apps use the pool instead of creating independent connections. | Medium | High | 4 database files | -| M4 | **Implement distributed caching**: Deploy Redis cluster, add caching layer to coordinator-api for: API responses, session state, rate limiting counters, blockchain state reads. | Medium | High | `aitbc/caching.py` expansion | -| M5 | **Add rate limiting to all routers**: Apply `slowapi` limits to all 61 coordinator-api routers with tiered limits by endpoint sensitivity. | Medium | High | 61 router files | -| M6 | **Tighten mypy configuration**: Enable `check_untyped_defs`, `disallow_untyped_defs` incrementally. Remove `ignore_errors = true` overrides. Start with blockchain-node (best structured). | Medium | Medium | `pyproject.toml` | -| M7 | **Consolidate marketplace services**: Clarify boundaries between `marketplace`, `marketplace-service`, and `marketplace-service-debug`. Merge or clearly document the purpose of each. | Medium | Medium | 3 marketplace dirs | -| M8 | **Add background task system**: Implement Celery or ARQ for long-running tasks: RL training, analytics aggregation, blockchain event processing. | Medium | High | New infrastructure | - -### LONG-TERM (1-3 months) -- Strategic Improvements - -| # | Recommendation | Effort | Impact | File(s) | -|---|---------------|--------|--------|---------| -| L1 | **Implement API gateway pattern**: The `apps/api-gateway` (4 files, 425 LOC) should become the single entry point handling auth, rate limiting, routing, and load balancing for all backend services. | High | Critical | `apps/api-gateway/` | -| L2 | **Adopt event-driven architecture**: Implement message bus (Redis Streams, RabbitMQ, or NATS) for inter-service communication. Replace direct service-to-service calls with events. | High | High | Cross-cutting | -| L3 | **Implement feature flag system**: Populate `feature_flags.json` with actual flags. Use `aitbc/feature_flags.py` for gradual rollouts of: new RL algorithms, marketplace features, pricing changes. | Medium | Medium | `feature_flags.json` | -| L4 | **Add comprehensive observability**: Implement distributed tracing (OpenTelemetry), structured logging (structlog is already a dep), and metrics (Prometheus client is already a dep) across all services. | High | High | Cross-cutting | -| L5 | **Create shared test fixtures**: Extract common test setup into `packages/py/aitbc-test-utils/` with shared fixtures, factories, and mocks. Reduce duplication across 41 conftest.py files. | Medium | Medium | `tests/` | -| L6 | **Implement contract upgrade pattern**: Review Solidity contracts for proxy pattern usage. 20 contracts with no visible upgrade mechanism is a long-term risk. | High | Medium | `contracts/` | -| L7 | **Documentation-as-code**: Move operational runbooks from docs/ into executable skills. The 8 existing skills docs are a good start but should be executable. | Medium | Low | `docs/skills/` | - ---- - -## 7. Risk Matrix - -| Issue | Severity | Likelihood | Risk Score | Time to Fix | -|-------|----------|------------|------------|-------------| -| Coordinator-api monolith | High | High (growing) | **Critical** | 4-8 weeks | -| Hardcoded secrets | Critical | Medium | **Critical** | 1-2 weeks | -| print() in production | Medium | High (growing) | **High** | 1 week | -| Bare except clauses | High | Medium | **High** | 1 week | -| No rate limiting | High | Medium | **High** | 2-3 weeks | -| SQL injection risks | Critical | Low (internal) | **High** | 1-2 weeks | -| No connection pooling | Medium | High | **Medium** | 2-3 weeks | -| No distributed caching | Medium | High | **Medium** | 3-4 weeks | -| Stub code in main repo | Low | Low | **Low** | 1 week | -| Type coverage gaps | Medium | Medium | **Medium** | 4-6 weeks | - ---- - -## 8. Files Most Likely to Change - -Based on the analysis, these files/directories will be most impacted by the recommendations: - -1. `/opt/aitbc/apps/coordinator-api/src/app/services/` -- Decomposition target (123 files) -2. `/opt/aitbc/apps/coordinator-api/src/app/routers/` -- Rate limiting, CORS (61 files) -3. `/opt/aitbc/apps/coordinator-api/src/app/config.py` -- Shared config base -4. `/opt/aitbc/apps/coordinator-api/src/app/database.py` -- Connection pooling -5. `/opt/aitbc/aitbc/caching.py` -- Distributed caching expansion -6. `/opt/aitbc/aitbc/middleware/` -- CORS unification -7. `/opt/aitbc/apps/stubs/` -- Removal/isolation -8. `/opt/aitbc/pyproject.toml` -- mypy tightening, coverage config -9. `/opt/aitbc/apps/wallet/src/app/chain/multichain_ledger.py` -- SQL injection fix -10. `/opt/aitbc/apps/agent-coordinator/src/coordinator.py` -- print() removal - ---- - -## 9. Verification Steps - -After implementing recommendations, verify with: - -1. **print() elimination**: `grep -rn "print(" apps/ packages/ | grep -v "node_modules" | grep -v "__pycache__" | wc -l` should return 0 -2. **Bare except elimination**: `grep -rn "^\s*except\s*:" apps/ packages/ | wc -l` should return 0 -3. **Test suite**: `pytest --cov=apps --cov=packages --cov=cli --cov-fail-under=50` should pass -4. **Security scan**: `bandit -r apps/ packages/ cli/` should return 0 issues -5. **Type check**: `mypy apps/ packages/ cli/` should pass with tightened config -6. **Rate limiting**: `curl -s -o /dev/null -w "%{http_code}" http://localhost:9001/api/...` repeated rapidly should return 429 -7. **Health checks**: All 28 systemd services should report `active (running)` and `/health` endpoints should return 200 - ---- - -## 10. Open Questions - -1. **Stubs purpose**: Are the 65 stub files in `apps/stubs/` planned for implementation, or should they be archived? Who owns the decision? -2. **Coordinator-api decomposition**: What is the preferred decomposition strategy -- by domain (RL, marketplace, analytics) or by function (read vs write)? -3. **Redis deployment**: Is there an existing Redis cluster, or does one need to be provisioned? -4. **Background task system**: Is there a preference between Celery, ARQ, or FastAPI BackgroundTasks for the task queue? -5. **Contract upgrade strategy**: Are the Solidity contracts expected to be upgradeable, or is immutability intentional? -6. **Marketplace service boundary**: What is the intended relationship between `marketplace`, `marketplace-service`, and `marketplace-service-debug`? -7. **Feature flag ownership**: Who manages the feature flag lifecycle -- product or engineering? diff --git a/.hermes/plans/2026-05-12_104500-coordinator-decomposition.md b/.hermes/plans/2026-05-12_104500-coordinator-decomposition.md deleted file mode 100644 index e67ff5c1..00000000 --- a/.hermes/plans/2026-05-12_104500-coordinator-decomposition.md +++ /dev/null @@ -1,97 +0,0 @@ - -# Coordinator-API Decomposition Plan - -## Current State -- **1 monolith**: apps/coordinator-api/src/app/ - - 89 service files, 46,594 LOC - - 53 routers - - 51 files over 500 LOC - - Largest: agent_integration.py (1,159 LOC) - -## Decomposition Strategy: Bounded Contexts - -Based on domain analysis, split into 7 microservices: - -1. **agent-management** (agent lifecycle, performance, communication) -2. **blockchain** (chain operations, transactions, smart contracts) -3. **computing** (GPU, resources, marketplace for compute) -4. **enterprise** (integration, scalability, compliance) -5. **identity** (authentication, authorization, agents identity) -6. **payment** (billing, transactions, financial operations) -7. **ai-models** (AI services, RL, multi-modal fusion) - -Each will be a separate FastAPI app with: -- Its own routers/, services/, models/ -- Shared libraries: app.core.config, app.core.logging, app.core.database -- Independent systemd service -- Clear API boundaries - -## Implementation Phases - -### Phase 1: Infrastructure Setup (Week 1-2) -- Create apps/ directory structure: agent-management/, blockchain/, etc. -- Create shared core library: apps/coordinator-api/src/app/core/ -- Extract common config, logging, DB session, exceptions -- Update pyproject.toml to support multiple packages - -### Phase 2: Extract Agent Management (Week 2-3) -- Move agent_*.py, agent_service_marketplace.py -> agent-management -- Move agent_communication.py, agent_performance_service.py -> agent-management -- Create new systemd service for agent-management -- Update reverse proxy (nginx) routes - -### Phase 3: Extract Blockchain (Week 3-4) -- Move blockchain_context.py, contract_service.py, transaction_service.py -> blockchain -- Move escrow.py, persistent_spending_tracker.py, etc. -- Create blockchain systemd service - -### Phase 4: Extract Enterprise (Week 4-5) -- Move enterprise_integration.py, compliance_engine.py, certification related -> enterprise -- Create enterprise systemd service - -### Phase 5: Extract Identity (Week 5-6) -- Move auth/identity service files -> identity -- Create identity systemd service - -### Phase 6: Extract AI Models (Week 6-7) -- Move advanced_*.py, multi_modal_fusion, ai verification -> ai-models -- Create ai-models systemd service - -### Phase 7: Extract Computing & Payment (Week 7-8) -- Move gpu, resource, payment services to their own packages - -### Phase 8: Final Integration (Week 8-9) -- Update all clients to use new service endpoints -- Test inter-service communication -- Update documentation -- Deprecate old monolith - -## Files to Create/Modify - -### New shared core (apps/coordinator-api/src/app/core/) -- config.py (extracted from existing config.py) -- logging.py (centralized logger setup) -- database.py (SQLAlchemy session, Base) -- exceptions.py (common exceptions) -- security.py (auth dependencies) - -### New service apps (47 directories total) -Each: apps//src/app/{routers,services,models,main.py} - -### Modified files -- Root pyproject.toml: add service packages -- Systemd: add 7 new .service files -- Nginx config: new upstream blocks -- Docker compose: add 7 new containers -- Monitoring: new service endpoints for health - -## Rollback Plan -- Keep original monolith running alongside new services during transition -- Use feature flags to route traffic -- Comprehensive integration tests before cutover - -## Success Criteria -- Each service < 3,000 LOC (target 1,500) -- Each service independently deployable -- API contracts stable and documented -- CI/CD per service diff --git a/.hermes/plans/2026-05-12_142930-agent-management-extraction.md b/.hermes/plans/2026-05-12_142930-agent-management-extraction.md deleted file mode 100644 index f5a84815..00000000 --- a/.hermes/plans/2026-05-12_142930-agent-management-extraction.md +++ /dev/null @@ -1,239 +0,0 @@ -# Agent-Management Service Extraction Plan - -## Overview - -Extract the agent-related functionality from the coordinator-api monolith into a standalone microservice while maintaining operational continuity. - -## Current State - -**Monolith:** `apps/coordinator-api/src/app/` -- Services: 46,594 LOC across 89 files -- Domain layer: `domain/` contains all business entities (Agent, AgentExecution, AgentStatus, etc.) -- Target agent files to extract: **18 files** (6 routers, 12 services) -- Largest files: agent_service.py (1,159 LOC), agent_integration.py (1,117 LOC), agent_communication.py (988 LOC) - -## Bounded Context: Agent-Management - -**Responsibility:** AI agent lifecycle, orchestration, performance tracking, security, and marketplace registry. - -**In-Scope Files:** - -### Services (12) -``` -services/agent_service.py (1,159 LOC) -services/agent_integration.py (1,117 LOC) -services/agent_communication.py (988 LOC) -services/agent_orchestrator.py -services/agent_performance_service.py -services/agent_security.py -services/agent_portfolio_manager.py -services/agent_service_marketplace.py -services/advanced_rl/agents.py (+ sub-agents: ppo_agent.py, rainbow_dqn_agent.py, sac_agent.py) -``` - -### Routers (6) -``` -routers/agent_router.py -routers/agent_integration_router.py -routers/agent_performance.py -routers/agent_creativity.py -routers/agent_security_router.py -routers/services.py (agent services listing endpoint) -``` - -## Critical Dependencies - -1. **Domain Layer** (`app.domain`) - - All agent services import from `..domain.agent` (AgentExecution, AgentStatus, AIAgentWorkflow, etc.) - - Solution: Keep domain/ in monolith for now; new service imports via a **shared-domain package** to be created - - Create `apps/shared-domain/src/app/domain/` as a symlink or copy that both services can import - - Long-term: Extract entire domain layer to shared-domain package - -2. **aitbc package** - - Already available as root package. Use directly. - -3. **SQLModel/SQLAlchemy** - - Already in dependencies via root pyproject.toml - -4. **Other monolith services** - - Some routers may call agent endpoints. These will need to be updated to use HTTP client to new service (Phase 3 internal routing via nginx) - -## Implementation Steps - -### Step 0: Prepare Shared Domain Package (Prerequisite) -- Create `apps/shared-domain/src/app/domain/` -- Copy all files from coordinator-api's `domain/` EXCEPT non-agent ones if desired -- Or simpler: symlink entire domain directory: `ln -s ../../coordinator-api/src/app/domain apps/shared-domain/src/app/` -- Update imports in new service to use `from shared-domain.app.domain.agent import ...` -- Add `shared-domain` to pyproject.toml dependencies in consuming services - -**Recommendation:** Use symlink for rapid iteration, then formalize package later. - -### Step 1: Create agent-management Service Skeleton -``` -apps/agent-management/ -├── pyproject.toml -├── README.md -└── src/ - └── app/ - ├── __init__.py - ├── main.py - ├── core/ - │ ├── __init__.py - │ ├── config.py (import from shared-core) - │ ├── logging.py (import from shared-core) - │ └── database.py (import from shared-core) - ├── domain/ → symlink to ../../shared-domain/src/app/domain - ├── routers/ - │ ├── __init__.py - │ ├── agent_router.py (copied & adapted) - │ ├── agent_integration_router.py - │ ├── agent_performance.py - │ ├── agent_creativity.py - │ ├── agent_security_router.py - │ └── services.py - └── services/ - ├── __init__.py - ├── agent_service.py - ├── agent_orchestrator.py - ├── agent_communication.py - ├── agent_performance_service.py - ├── agent_security.py - ├── agent_integration.py - ├── agent_portfolio_manager.py - ├── agent_service_marketplace.py - └── advanced_rl/ - ├── __init__.py - ├── agents.py - └── ppo_agent.py, rainbow_dqn_agent.py, sac_agent.py -``` - -### Step 2: Adapt Code for Service Boundaries - -**Changes needed per file:** - -- Update all `from ..domain.agent import X` to `from shared-domain.app.domain.agent import X` -- Remove any imports from other monolith services (e.g., `from ..services.other_service import X`) -- Replace internal service calls with HTTP client calls or event bus (defer to later phase) -- Update `ServiceSettings` to use agent-management specific defaults (port 8012) -- Add health check endpoint (already in template) -- Verify database setup: AgentExecution etc use shared Base. Need to call `Base.metadata.create_all(bind=engine)` on startup - -**Special Case: advanced_rl/** -- These are AI model inference services. Consider moving to `ai-models` service instead. -- For now, keep in agent-management to maintain functionality. - -### Step 3: Update Monolith to Proxy Requests (During Transition) - -**Option A: Nginx Routing** -- Add nginx upstream for agent-management on port 8012 -- Change coordinator-api routes for `/api/v1/agent/*` to proxy to agent-management -- Monolith no longer handles agent endpoints - -**Option B: In-app Redirection** -- Keep routers in monolith but replace handlers with `HTTPClient` calls to new service -- More gradual migration but adds latency - -**Recommendation:** Option A - cleaner separation, easier to rollback. - -### Step 4: Create Systemd Service - -``` -/etc/systemd/system/aitbc-agent-management.service -[Unit] -Description=AITBC Agent Management Service -After=network.target - -[Service] -Type=simple -User=aitbc -WorkingDirectory=/opt/aitbc/apps/agent-management -Environment=PATH=/opt/aitbc/venv/bin -Environment=PYTHONPATH=/opt/aitbc -ExecStart=/opt/aitbc/venv/bin/uvicorn app.main:app --host 127.0.0.1 --port 8012 -Restart=on-failure -RestartSec=10 - -[Install] -WantedBy=multi-user.target -``` - -### Step 5: Database Migration - -- Agent domain models likely already have tables defined via SQLModel -- In `main.py` startup event, call `Base.metadata.create_all(bind=engine)` to ensure tables exist -- Ensure the new service uses same database as monolith (coordinator.db) initially -- Later: separate database (Phase 8) - -### Step 6: Integration Testing - -1. Start agent-management service -2. Verify health endpoint: `curl http://localhost:8012/health` -3. Test agent creation via API -4. Verify coordinator-api can still access agent data (through new service or direct DB if keeping shared DB) -5. Run existing integration tests against new service - -### Step 7: Update Coordinator-API - -- Remove the 18 extracted files from monolith -- Remove domain/agent related imports from remaining monolith services if they now use agent-management API -- Update any remaining references to agent endpoints to use HTTP client or nginx proxy - -### Step 8: Documentation & Monitoring - -- Update README with agent-management API docs -- Add metrics endpoint if enabled -- Update deployment scripts - -## Rollback Plan - -1. Keep monolith files in git history (do not delete, just move) -2. Keep nginx config either/or - can revert upstream routing -3. Database shared initially, so data is accessible to both -4. Systemd service can be disabled; monolith still runs - -## Success Criteria - -- [ ] Agent-management service starts and health check passes on port 8012 -- [ ] Can create/query agents via API -- [ ] Existing coordinator-api functionality that depends on agents still works -- [ ] No errors in logs during integration test -- [ ] Systemd service auto-restarts on failure - -## Open Questions - -1. **RL Agents**: Should advanced_rl be part of agent-management or ai-models? - - Recommendation: Keep in agent-management for now (AI agent inference is part of agent runtime). Can split later if ai-models becomes a separate inference service. - -2. **Database**: Separate or shared? - - Phase 1: Shared (same coordinator.db) for simplicity - - Phase 8: Split to dedicated agent-management database - -3. **Cross-service calls**: Currently agent integration uses other services directly (imports). Need to replace with HTTP or event bus. - - Defer until Phase 8 (Final Integration) to avoid breaking existing flow - -4. **Domain extraction**: The domain models are currently in monolith. Should we extract entire domain to a package? - - Immediate need: Create shared-domain package (symlink) to break import cycle - - Future: Extract domain to true package with independent version - -## Timeline Estimate - -- Step 0 (shared-domain): 2h -- Step 1 (skeleton): 4h -- Step 2 (adaptation): 8h (bulk of work - fixing imports, resolving dependencies) -- Step 3 (nginx routing): 2h -- Step 4 (systemd): 1h -- Step 5 (DB): 1h -- Step 6 (testing): 4h -- Step 7 (monolith cleanup): 4h -- Step 8 (docs): 2h - -**Total: ~28 hours (3-4 days)** - -## Risks - -- Hidden dependencies on other monolith services may cause runtime import errors -- Domain models may have cross-references that require co-migration -- Database migrations may be needed if agent tables don't exist yet -- Existing integration tests may fail and need updating -- Breaking changes if API contracts differ from original diff --git a/.hermes/plans/2026-05-12_150000-tighten-mypy-config.md b/.hermes/plans/2026-05-12_150000-tighten-mypy-config.md deleted file mode 100644 index 78ebca56..00000000 --- a/.hermes/plans/2026-05-12_150000-tighten-mypy-config.md +++ /dev/null @@ -1,218 +0,0 @@ - -# Tighten Mypy Configuration Plan - -## Current State - -**Root pyproject.toml [tool.mypy] settings:** -```toml -warn_return_any = true -warn_unused_configs = true -check_untyped_defs = false -disallow_incomplete_defs = false -disallow_untyped_defs = false -disallow_untyped_decorators = false -no_implicit_optional = false -warn_redundant_casts = false -warn_unused_ignores = false -warn_no_return = true -warn_unreachable = false -strict_equality = false -``` - -**Overrides:** -- Heavy libraries (torch, cv2, pandas, numpy, web3, etc.) are `ignore_missing_imports = true` -- Coordiator-api modules are `ignore_errors = true` (catch-all) - -This is **extremely permissive** - essentially just warns on return_any and missing configs. It does not enforce: -- Function argument/return type completeness -- Avoiding implicit `Any` -- Avoiding unnecessary type: ignore comments -- Detecting unreachable code -- Strict equality checks (None vs False) - -## Proposed Tightening Phases - -### Phase 1: Enable Foundational Checks (Low Effort, High Value) -Target: enable 4 key options that catch real bugs with minimal friction - -```toml -disallow_untyped_defs = true -disallow_incomplete_defs = true -warn_redundant_casts = true -warn_unused_ignores = true -``` - -**Impact:** -- Functions must have complete type signatures (all args+returns typed) -- Redundant cast() calls will be flagged -- Unused `# type: ignore` comments will be flagged -- Minimal code changes required (most functions already typed) - -**Estimated effort:** -- 1 hour to update config -- 2-4 hours to fix violations in production code -- Total: ~1 day - -**Validation:** -- Run `mypy apps` and ensure 0 errors -- Keep existing overrides for external libraries and coordinator-api - -### Phase 2: Stricter Optional Handling (Medium Effort) -Enable: -```toml -no_implicit_optional = true -warn_unreachable = true -strict_equality = true -``` - -**Impact:** -- Variables defaulting to `None` must be explicitly `Optional[...]` -- Unreachable code will be flagged (dead code detection) -- Equality comparisons with None must use `is` not `==` - -**Estimated effort:** 2-3 days to fix violations across codebase - -### Phase 3: Gradual Per-Module Strictness (Long-term) -- Move coordinator-api out of catch-all `ignore_errors` -- Add per-module overrides as we achieve correctness -- Eventually remove `ignore_errors` blanket - -**Estimated effort:** Ongoing as part of decomposition - -## Implementation Steps - -### Step 1: Backup Current Config -```bash -cp pyproject.toml pyproject.toml.backup -``` - -### Step 2: Update Root Configuration - -Modify `/opt/aitbc/pyproject.toml` [tool.mypy] section: - -```diff - [tool.mypy] - python_version = "3.13" - warn_return_any = true - warn_unused_configs = true - check_untyped_defs = false --disallow_incomplete_defs = false --disallow_untyped_defs = false -+disallow_incomplete_defs = true -+disallow_untyped_defs = true - disallow_untyped_decorators = false - no_implicit_optional = false - warn_redundant_casts = false - warn_unused_ignores = false - warn_no_return = true - warn_unreachable = false - strict_equality = false -``` - -### Step 3: Run Mypy and Collect Errors - -```bash -cd /opt/aitbc -venv/bin/mypy apps --show-error-codes --no-color-output > mypy_errors.txt 2>&1 -``` - -### Step 4: Categorize Errors - -Typical violations we'll see: -- `Function is missing a return type annotation` (from disallow_untyped_defs) -- `Function is missing a type annotation for one or more arguments` (from disallow_untyped_defs) -- `Class is missing type parameters for generic type` (rare) -- `dict, list, etc. used without type parameters` (from disallow_incomplete_defs) -- `Redundant cast to X` (from warn_redundant_casts) -- `Unused "type: ignore" comment` (from warn_unused_ignores) - -### Step 5: Fix in Order of Impact - -**A. Add missing type annotations to functions** -- Priority: functions in shared-core, services, routers -- Use explicit return types; if truly dynamic, use `-> Any` (but rarely needed) -- Example: - ```python - def get_engine(settings): # BEFORE - def get_engine(settings: ServiceSettings) -> Engine: # AFTER - ``` - -**B. Add generic type parameters** -- `list` -> `List[str]` or `list[int]` -- `dict` -> `Dict[str, Any]` -- Use `from typing import List, Dict` - -**C. Remove redundant casts** -- Delete `cast(Type, value)` if type is already clear to mypy -- Use `reveal_type(value)` to check actual inferred type before removing - -**D. Remove unused type: ignore** -- Some `# type: ignore` comments are legacy and no longer needed -- Delete them; if mypy still fails, leave or fix underlying issue - -### Step 6: Iterate and Validate - -After fixing categories, re-run mypy. Continue until `mypy apps` exits with code 0. - -**Note:** We preserve `ignore_missing_imports` for heavy libraries, and `ignore_errors` for coordinator-api (since we're deferring decomposition). - -### Step 7: Add CI Enforcement - -Update pre-commit hooks or CI to run mypy on PRs: -```yaml -# .pre-commit-config.yaml or GitHub Actions -- repo: local - hooks: - - id: mypy - name: mypy - entry: mypy apps - language: system - pass_filenames: false -``` - -## Rollback Plan - -If the effort becomes too large: -1. Revert pyproject.toml from backup -2. Keep per-module `# mypy: ignore-errors` as needed -3. Approach incrementally: enable one flag at a time - -## Success Criteria - -- `mypy apps` completes with 0 errors -- No new type: ignore comments added without explanation -- Production code has complete type signatures -- CI pipeline includes mypy check - -## Risks & Mitigations - -| Risk | Mitigation | -|------|------------| -| Overwhelming number of errors | Enable flags incrementally (2 at a time), fix in batches by module | -| Breaking existing functionality by incorrect type fixes | Run test suite after each batch; use `reveal_type` to debug | -| Third-party library types incompatible | Keep `ignore_missing_imports` for those packages | -| Coordinator-api too messy to fix now | Keep `ignore_errors` override; revisit after decomposition | - -## Related Tasks - -- **Decompose coordinator-api** - Once strict mypy is in place, easier to validate new services -- **Shared-core library** - Strict typing ensures compatibility across services -- **Connection pooling** - Use proper typed database sessions - -## Open Questions - -1. Should we also enable `strict` mode for new services? (Probably yes) -2. Should we add type-checking to pre-commit hook for changed files only? (Yes, use `mypy --files `) -3. How to handle legacy coordinator-api code? (Keep ignore_errors for now) - -## Estimated Timeline - -- **0-2 days:** Implement Phase 1, fix immediate violations -- **3-7 days:** Address accumulated type errors, reach clean mypy -- **Week 2:** Add CI enforcement, document guidelines -- **Ongoing:** Maintain strict typing in new code - -## References - -- Mypy configuration: https://mypy.readthedocs.io/en/stable/config_file.html -- Strict mode: https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-strict diff --git a/docs/skills/aitbc-ai-operations.md b/docs/hermes/skills/aitbc-ai-operations.md similarity index 100% rename from docs/skills/aitbc-ai-operations.md rename to docs/hermes/skills/aitbc-ai-operations.md diff --git a/docs/skills/aitbc-basic-operations.md b/docs/hermes/skills/aitbc-basic-operations.md similarity index 100% rename from docs/skills/aitbc-basic-operations.md rename to docs/hermes/skills/aitbc-basic-operations.md diff --git a/docs/skills/aitbc-blockchain-troubleshooting.md b/docs/hermes/skills/aitbc-blockchain-troubleshooting.md similarity index 100% rename from docs/skills/aitbc-blockchain-troubleshooting.md rename to docs/hermes/skills/aitbc-blockchain-troubleshooting.md diff --git a/docs/skills/aitbc-cli.md b/docs/hermes/skills/aitbc-cli.md similarity index 100% rename from docs/skills/aitbc-cli.md rename to docs/hermes/skills/aitbc-cli.md diff --git a/docs/skills/aitbc-marketplace.md b/docs/hermes/skills/aitbc-marketplace.md similarity index 100% rename from docs/skills/aitbc-marketplace.md rename to docs/hermes/skills/aitbc-marketplace.md diff --git a/docs/skills/aitbc-multi-node-operations.md b/docs/hermes/skills/aitbc-multi-node-operations.md similarity index 100% rename from docs/skills/aitbc-multi-node-operations.md rename to docs/hermes/skills/aitbc-multi-node-operations.md diff --git a/docs/skills/aitbc-node-coordination.md b/docs/hermes/skills/aitbc-node-coordination.md similarity index 100% rename from docs/skills/aitbc-node-coordination.md rename to docs/hermes/skills/aitbc-node-coordination.md diff --git a/docs/skills/aitbc-wallet-management.md b/docs/hermes/skills/aitbc-wallet-management.md similarity index 100% rename from docs/skills/aitbc-wallet-management.md rename to docs/hermes/skills/aitbc-wallet-management.md diff --git a/docs/skills/aitbc.md b/docs/hermes/skills/aitbc.md similarity index 100% rename from docs/skills/aitbc.md rename to docs/hermes/skills/aitbc.md