oib/aitbc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Based on the repository's commit message style and the changes in the diff, here's an appropriate commit message:

Browse Source 
```
feat: add websocket tests, PoA metrics, marketplace endpoints, and enhanced observability

- Add comprehensive websocket tests for blocks and transactions streams including multi-subscriber and high-volume scenarios
- Extend PoA consensus with per-proposer block metrics and rotation tracking
- Add latest block interval gauge and RPC error spike alerting
- Enhance mock coordinator
This commit is contained in:
oib
2025-12-22 07:55:09 +01:00
parent fb60505cdf
commit d98b2c7772
70 changed files with 3472 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/done.md
View File

@ -26,6 +26,10 @@
- Implemented CLI/Python runners and execution pipeline with result reporting.
- Added starter tests for runners in `apps/miner-node/tests/test_runners.py`.
## Blockchain Node
- Added websocket fan-out, disconnect cleanup, and load-test coverage in `apps/blockchain-node/tests/test_websocket.py`, ensuring gossip topics deliver reliably to multiple subscribers.
## Directory Preparation
- Established scaffolds for Python and JavaScript packages in `packages/py/` and `packages/js/`.
@ -42,6 +46,7 @@
- Added `apps/wallet-daemon/src/app/receipts/service.py` providing `ReceiptVerifierService` that fetches and validates receipts via `aitbc_sdk`.
- Created unit tests under `apps/wallet-daemon/tests/test_receipts.py` verifying service behavior.
- Implemented wallet SDK receipt ingestion + attestation surfacing in `packages/py/aitbc-sdk/src/receipts.py`, including pagination client, signature verification, and failure diagnostics with full pytest coverage.
- Hardened REST API by wiring dependency overrides in `apps/wallet-daemon/tests/test_wallet_api.py`, expanding workflow coverage (create/list/unlock/sign) and enforcing structured password policy errors consumed in CI.
## Explorer Web

26
docs/ports.md Normal file
View File

@ -0,0 +1,26 @@
# Port Allocation Plan
This document tracks current and planned TCP port assignments across the AITBC devnet stack. Update it whenever new services are introduced or defaults change.
## Current Usage
| Port | Service | Location | Notes |
| --- | --- | --- | --- |
| 8080 | Blockchain RPC API (FastAPI) | `apps/blockchain-node/scripts/devnet_up.sh` → `python -m uvicorn aitbc_chain.app:app` | Exposes REST/WebSocket RPC endpoints for blocks, transactions, receipts. |
| 8090 | Mock Coordinator API | `apps/blockchain-node/scripts/devnet_up.sh` → `uvicorn mock_coordinator:app` | Generates synthetic coordinator/miner telemetry consumed by Grafana dashboards. |
| 9090 | Prometheus (planned default) | `apps/blockchain-node/observability/` (targets to be wired) | Scrapes blockchain node + mock coordinator metrics. Ensure firewall allows local-only access. |
| 3000 | Grafana (planned default) | `apps/blockchain-node/observability/grafana-dashboard.json` | Visualizes metrics dashboards; behind devnet Docker compose or local binary. |
## Reserved / Planned Ports
- **Coordinator API (production)** – TBD (`8000` suggested). Align with `apps/coordinator-api/README.md` once the service runs outside mock mode.
- **Marketplace Web** – Vite dev server defaults to `5173`; document overrides when deploying behind nginx.
- **Explorer Web** – Vite dev server defaults to `4173`; ensure it does not collide with other tooling on developer machines.
- **Pool Hub API** – Reserve `8100` for the FastAPI service when devnet integration begins.
## Guidance
- Avoid reusing the same port across services in devnet scripts to prevent binding conflicts (recent issues occurred when `8080`/`8090` were already in use).
- For production-grade environments, place HTTP services behind a reverse proxy (nginx/Traefik) and update this table with the external vs. internal port mapping.
- When adding new dashboards or exporters, note both the scrape port (Prometheus) and any UI port (Grafana/others).
- If a port is deprecated, strike it through in this table and add a note describing the migration path.

202
docs/roadmap.md
View File

@ -2,7 +2,27 @@
This roadmap aggregates high-priority tasks derived from the bootstrap specifications in `docs/bootstrap/` and tracks progress across the monorepo. Update this document as milestones evolve.
## Stage 1 — Core Services (MVP)
## Stage 1 — Upcoming Focus Areas
- **Blockchain Node Foundations**
- ✅ Bootstrap module layout in `apps/blockchain-node/src/`.
- ✅ Implement SQLModel schemas and RPC stubs aligned with historical/attested receipts.
- **Explorer Web Enablement**
- ✅ Finish mock integration across all pages and polish styling + mock/live toggle.
- ✅ Begin wiring coordinator endpoints (e.g., `/v1/jobs/{job_id}/receipts`).
- **Marketplace Web Scaffolding**
- ✅ Scaffold Vite/vanilla frontends consuming coordinator receipt history endpoints and SDK examples.
- **Pool Hub Services**
- ✅ Initialize FastAPI project, scoring registry, and telemetry ingestion hooks leveraging coordinator/miner metrics.
- **CI Enhancements**
- ✅ Add blockchain-node tests once available and frontend build/lint checks to `.github/workflows/python-tests.yml` or follow-on workflows.
- ✅ Provide systemd unit + installer scripts under `scripts/` for streamlined deployment.
## Stage 2 — Core Services (MVP)
- **Coordinator API**
- ✅ Scaffold FastAPI project (`apps/coordinator-api/src/app/`).
@ -23,10 +43,10 @@ This roadmap aggregates high-priority tasks derived from the bootstrap specifica
- Introduced hex/enum validation hooks via Pydantic validators to ensure hash integrity and safe persistence.
- ✅ Implement PoA proposer loop with block assembly (`apps/blockchain-node/src/aitbc_chain/consensus/poa.py`).
- ✅ Expose REST RPC endpoints for tx submission, balances, receipts (`apps/blockchain-node/src/aitbc_chain/rpc/router.py`).
- ⏳ Deliver WebSocket RPC + P2P gossip layer:
- Stand up WebSocket subscription endpoints (`apps/blockchain-node/src/aitbc_chain/rpc/websocket.py`) mirroring REST payloads.
- Implement pub/sub transport for block + transaction gossip backed by an in-memory broker (Starlette `Broadcast` or Redis) with configurable fan-out.
- Add integration tests and load-test harness ensuring gossip convergence and back-pressure handling.
- ✅ Deliver WebSocket RPC + P2P gossip layer:
- ✅ Stand up WebSocket subscription endpoints (`apps/blockchain-node/src/aitbc_chain/rpc/websocket.py`) mirroring REST payloads.
- ✅ Implement pub/sub transport for block + transaction gossip backed by an in-memory broker (Starlette `Broadcast` or Redis) with configurable fan-out.
- ✅ Add integration tests and load-test harness ensuring gossip convergence and back-pressure handling.
- ✅ Ship devnet scripts (`apps/blockchain-node/scripts/`).
- ✅ Add observability hooks (JSON logging, Prometheus metrics) and integrate coordinator mock into devnet tooling.
- ⏳ Expand observability dashboards + miner mock integration:
@ -46,21 +66,23 @@ This roadmap aggregates high-priority tasks derived from the bootstrap specifica
- ✅ Provide REST and JSON-RPC endpoints for wallet management and signing (`api_rest.py`, `api_jsonrpc.py`).
- ✅ Add mock ledger adapter with SQLite backend powering event history (`ledger_mock/`).
- ✅ Integrate Python receipt verification helpers (`aitbc_sdk`) and expose API/service utilities validating miner + coordinator signatures.
- ✅ Harden REST API workflows (create/list/unlock/sign) with structured password policy enforcement and deterministic pytest coverage in `apps/wallet-daemon/tests/test_wallet_api.py`.
- ✅ Implement Wallet SDK receipt ingestion + attestation surfacing:
- Added `/v1/jobs/{job_id}/receipts` client helpers with cursor pagination, retry/backoff, and summary reporting (`packages/py/aitbc-sdk/src/receipts.py`).
- Reused crypto helpers to validate miner and coordinator signatures, capturing per-key failure reasons for downstream UX.
- Surfaced aggregated attestation status (`ReceiptStatus`) and failure diagnostics for SDK + UI consumers; JS helper parity still planned.
## Stage 2 — Pool Hub & Marketplace
## Stage 3 — Pool Hub & Marketplace
- **Pool Hub**
- ✅ Implement miner registry, scoring engine, and `/v1/match` API with Redis/PostgreSQL backing stores.
- ✅ Add observability endpoints (`/v1/health`, `/v1/metrics`) plus Prometheus instrumentation and integration tests.
- **Marketplace Web**
- Initialize Vite project with vanilla TypeScript.
- Build offer list, bid form, stats views sourcing mock JSON.
- Provide API abstraction to switch between mock and real backends.
- ✅ Initialize Vite project with vanilla TypeScript (`apps/marketplace-web/`).
- ✅ Build offer list, bid form, and stats cards powered by mock data fixtures (`public/mock/`).
- ✅ Provide API abstraction toggling mock/live mode (`src/lib/api.ts`) and wire coordinator endpoints.
- ⏳ Validate live mode against coordinator `/v1/marketplace/*` responses and add auth feature flags for rollout.
- **Explorer Web**
- ✅ Initialize Vite + TypeScript project scaffold (`apps/explorer-web/`).
@ -74,42 +96,130 @@ This roadmap aggregates high-priority tasks derived from the bootstrap specifica
- Add fallbacks + error surfacing for partial/failed live responses (toast + console diagnostics).
- Audit responsive breakpoints (`public/css/layout.css`) and adjust grid/typography for tablet + mobile; add regression checks in Percy/Playwright snapshots.
## Stage 4 — Observability & Production Polish
- **Observability & Telemetry**
- ⏳ Build Grafana dashboards for PoA consensus health (block intervals, proposer rotation cadence) leveraging `poa_last_block_interval_seconds`, `poa_proposer_rotations_total`, and per-proposer counters.
- ⏳ Surface RPC latency histograms/summaries for critical endpoints (`rpc_get_head`, `rpc_send_tx`, `rpc_submit_receipt`) and add Grafana panels with SLO thresholds.
- ⏳ Ingest miner mock telemetry (job throughput, failure rate) into the shared Prometheus registry and wire panels/alerts that correlate miner health with consensus metrics.
- **Explorer Web (Live Mode)**
- ⏳ Finalize live `getDataMode() === "live"` workflow: align API payload contracts, render loading/error states, and persist mock/live toggle preference.
- ⏳ Expand responsive testing (tablet/mobile) and add automated visual regression snapshots prior to launch.
- ⏳ Integrate Playwright smoke tests covering overview, blocks, and transactions pages in live mode.
- **Marketplace Web (Launch Readiness)**
- ✅ Connect mock listings/bids to coordinator data sources and provide feature flags for live mode rollout.
- ✅ Implement auth/session scaffolding for marketplace actions and document API assumptions in `apps/marketplace-web/README.md`.
- ⏳ Add Grafana panels monitoring marketplace API throughput and error rates once endpoints are live.
- **Operational Hardening**
- ⏳ Extend Alertmanager rules to cover RPC error spikes, proposer stalls, and miner disconnects using the new metrics.
- ⏳ Document dashboard import + alert deployment steps in `docs/run.md` for operators.
- ⏳ Prepare Stage 3 release checklist linking dashboards, alerts, and smoke tests prior to production cutover.
## Stage 5 — Scaling & Release Readiness
- **Infrastructure Scaling**
- ⏳ Benchmark blockchain node throughput under sustained load; capture CPU/memory targets and suggest horizontal scaling thresholds.
- ⏳ Build Terraform/Helm templates for dev/staging/prod environments, including Prometheus/Grafana bundles.
- ⏳ Implement autoscaling policies for coordinator, miners, and marketplace services with synthetic traffic tests.
- **Reliability & Compliance**
- ⏳ Formalize backup/restore procedures for PostgreSQL, Redis, and ledger storage with scheduled jobs.
- ⏳ Complete security hardening review (TLS termination, API auth, secrets management) and document mitigations in `docs/security.md`.
- ⏳ Add chaos testing scripts (network partition, coordinator outage) and track mean-time-to-recovery metrics.
- **Product Launch Checklist**
- ⏳ Finalize public documentation (API references, onboarding guides) and publish to the docs portal.
- ⏳ Coordinate beta release timeline, including user acceptance testing of explorer/marketplace live modes.
- ⏳ Establish post-launch monitoring playbooks and on-call rotations.
## Stage 6 — Ecosystem Expansion
- **Cross-Chain & Interop**
- ⏳ Prototype cross-chain settlement hooks leveraging external bridges; document integration patterns.
- ⏳ Extend SDKs (Python/JS) with pluggable transport abstractions for multi-network support.
- ⏳ Evaluate third-party explorer/analytics integrations and publish partner onboarding guides.
- **Marketplace Growth**
- ⏳ Launch incentive programs (staking, liquidity mining) and expose telemetry dashboards tracking campaign performance.
- ⏳ Implement governance module (proposal voting, parameter changes) and add API/UX flows to explorer/marketplace.
- ⏳ Provide SLA-backed coordinator/pool hubs with capacity planning and billing instrumentation.
- **Developer Experience**
- ⏳ Publish advanced tutorials (custom proposers, marketplace extensions) and maintain versioned API docs.
- ⏳ Integrate CI/CD pipelines with canary deployments and blue/green release automation.
- ⏳ Host quarterly architecture reviews capturing lessons learned and feeding into roadmap revisions.
## Stage 7 — Innovation & Ecosystem Services
- **Advanced Cryptography & Privacy**
- ⏳ Research zk-proof-based receipt attestation and prototype a privacy-preserving settlement flow.
- ⏳ Add confidential transaction support in coordinator/miner stack with opt-in ciphertext storage.
- ⏳ Publish threat modeling updates and share mitigations with ecosystem partners.
- **Enterprise Integrations**
- ⏳ Deliver reference connectors for ERP/payment systems and document SLA expectations.
- ⏳ Stand up multi-tenant coordinator infrastructure with per-tenant isolation and billing metrics.
- ⏳ Launch ecosystem certification program (SDK conformance, security best practices) with public registry.
- **Community & Governance**
- ⏳ Establish open RFC process, publish governance website, and schedule regular community calls.
- ⏳ Sponsor hackathons/accelerators and provide grants for marketplace extensions and analytics tooling.
- ⏳ Track ecosystem KPIs (active marketplaces, cross-chain volume) and feed them into quarterly strategy reviews.
## Stage 8 — Frontier R&D & Global Expansion
- **Protocol Evolution**
- ⏳ Launch research consortium exploring next-gen consensus (hybrid PoA/PoS) and finalize whitepapers.
- ⏳ Prototype sharding or rollup architectures to scale throughput beyond current limits.
- ⏳ Standardize interoperability specs with industry bodies and submit proposals for adoption.
- **Global Rollout**
- ⏳ Establish regional infrastructure hubs (multi-cloud) with localized compliance and data residency guarantees.
- ⏳ Partner with regulators/enterprises to pilot regulated marketplaces and publish compliance playbooks.
- ⏳ Expand localization (UI, documentation, support) covering top target markets.
- **Long-Term Sustainability**
- ⏳ Create sustainability fund for ecosystem maintenance, bug bounties, and community stewardship.
- ⏳ Define succession planning for core teams, including training programs and contributor pathways.
- ⏳ Publish bi-annual roadmap retrospectives assessing KPI alignment and revising long-term goals.
## Stage 9 — Moonshot Initiatives
- **Decentralized Infrastructure**
- ⏳ Transition coordinator/miner roles toward community-governed validator sets with incentive alignment.
- ⏳ Explore decentralized storage/backbone options (IPFS/Filecoin) for ledger and marketplace artifacts.
- ⏳ Prototype fully trustless marketplace settlement leveraging zero-knowledge rollups.
- **AI & Automation**
- ⏳ Integrate AI-driven monitoring/anomaly detection for proposer health, market liquidity, and fraud detection.
- ⏳ Automate incident response playbooks with ChatOps and policy engines.
- ⏳ Launch research into autonomous agent participation (AI agents bidding/offering in the marketplace) and governance implications.
- **Global Standards Leadership**
- ⏳ chair industry working groups defining receipt/marketplace interoperability standards.
- ⏳ Publish annual transparency reports and sustainability metrics for stakeholders.
- ⏳ Engage with academia and open-source foundations to steward long-term protocol evolution.
### Stage 10 — Stewardship & Legacy Planning
- **Open Governance Maturity**
- ⏳ Transition roadmap ownership to community-elected councils with transparent voting and treasury controls.
- ⏳ Codify constitutional documents (mission, values, conflict resolution) and publish public charters.
- ⏳ Implement on-chain governance modules for protocol upgrades and ecosystem-wide decisions.
- **Educational & Outreach Programs**
- ⏳ Fund university partnerships, research chairs, and developer fellowships focused on decentralized marketplace tech.
- ⏳ Create certification tracks and mentorship programs for new validator/operators.
- ⏳ Launch annual global summit and publish proceedings to share best practices across partners.
- **Long-Term Preservation**
- ⏳ Archive protocol specs, governance records, and cultural artifacts in decentralized storage with redundancy.
- ⏳ Establish legal/organizational frameworks to ensure continuity across jurisdictions.
- ⏳ Develop end-of-life/transition plans for legacy components, documenting deprecation strategies and migration tooling.
## Shared Libraries & Examples
the canonical checklist during implementation. Mark completed tasks with ✅ and add dates or links to relevant PRs as development progresses.
- **Python SDK (`packages/py/aitbc-sdk`)**
- ✅ Implement coordinator receipt client + verification helpers (miner + coordinator attestation support).
- ⏳ Extend helpers to pool hub + wallet APIs and typed models:
- Add REST clients for upcoming Pool Hub endpoints (`/v1/match`, `/v1/miners`) and wallet daemon routes (`/v1/wallets`, `/v1/sign`) with retry/backoff helpers.
- Introduce pydantic/SQLModel-derived typed models mirroring `protocols/api/` and `protocols/receipts/` schemas.
- Provide end-to-end tests + examples validating Pool Hub + wallet flows leveraging the coordinator receipt verification primitives.
- **JavaScript SDK (`packages/js/aitbc-sdk`)**
- ✅ Provide fetch-based wrapper for web clients with TypeScript definitions and basic auth helpers.
- **Examples**
- Populate quickstart clients (Python/JS) with working code.
- Add receipt sign/verify samples using finalized schema.
## Tooling & Operations
- **Scripts & CI**
- ✅ Populate `scripts/ci/run_python_tests.sh` to run coordinator, SDK, wallet-daemon pytest suites with shared `PYTHONPATH` scaffolding.
- ✅ Add GitHub Actions workflow `.github/workflows/python-tests.yml` invoking the shared script on pushes/PRs targeting `main`.
- **Configs**
- Author systemd unit files in `configs/systemd/` for each service.
- Provide Nginx snippets in `configs/nginx/` for reverse proxies.
## Tracking
Use this roadmap as the canonical checklist during implementation. Mark completed tasks with ✅ and add dates or links to relevant PRs as development progresses.
## Upcoming Focus Areas
- **Blockchain Node**: bootstrap module layout (`apps/blockchain-node/src/`), implement SQLModel schemas and RPC stubs aligned with historical/attested receipts.
- **Explorer Web**: finish mock integration across all pages, add styling + mock/live toggle, and begin wiring coordinator endpoints (e.g., `/v1/jobs/{job_id}/receipts`).
- Current focus: reuse new overview metrics scaffolding for blocks/transactions detail views and expand coverage to live data mode.
- **Marketplace Web**: scaffold Vite/vanilla frontends with mock integrations consuming the coordinator receipt history endpoints and SDK examples.
- **Pool Hub**: initialize FastAPI project, scoring registry, and telemetry ingestion hooks leveraging coordinator/miner metrics.
- **CI Enhancements**: add blockchain-node tests once available and frontend build/lint checks to `.github/workflows/python-tests.yml` or follow-on workflows.
- ⏳ Add systemd unit and installer scripts under `scripts/`.

38
docs/run.md
View File

@ -233,6 +233,44 @@ These instructions cover the newly scaffolded services. Install dependencies usi
```
(RPC, consensus, and P2P logic still to be implemented.)
### Observability Dashboards & Alerts
1. Generate the starter Grafana dashboards (if not already present):
```bash
cd apps/blockchain-node
PYTHONPATH=src python - <<'PY'
from pathlib import Path
from aitbc_chain.observability.dashboards import generate_default_dashboards
output_dir = Path("observability/generated_dashboards")
output_dir.mkdir(parents=True, exist_ok=True)
generate_default_dashboards(output_dir)
print("Dashboards written to", output_dir)
PY
```
2. Import each JSON file into Grafana (**Dashboards → Import**):
- `apps/blockchain-node/observability/generated_dashboards/coordinator-overview.json`
- `apps/blockchain-node/observability/generated_dashboards/blockchain-node-overview.json`
Select your Prometheus datasource (pointing at `127.0.0.1:8080` and `127.0.0.1:8090`) during import.
3. Ensure Prometheus scrapes both services. Example snippet from `apps/blockchain-node/observability/prometheus.yml`:
```yaml
scrape_configs:
- job_name: "blockchain-node"
static_configs:
- targets: ["127.0.0.1:8080"]
- job_name: "mock-coordinator"
static_configs:
- targets: ["127.0.0.1:8090"]
```
4. Deploy the Alertmanager rules in `apps/blockchain-node/observability/alerts.yml` (proposer stalls, miner errors, receipt drop-offs, RPC error spikes). After modifying rule files, reload Prometheus/Alertmanager:
```bash
systemctl restart prometheus
systemctl restart alertmanager
```
5. Validate by briefly stopping `aitbc-coordinator.service`, confirming Grafana panels pause and the new alerts fire, then restart the service.
## Next Steps
- Flesh out remaining logic per task breakdowns in `docs/*.md` (e.g., capability-aware scheduling, artifact uploads).