# miner-node/ — Worker Node Daemon for GPU/CPU Tasks

> **Goal:** Implement a Docker‑free worker daemon that connects to the Coordinator API, advertises capabilities (CPU/GPU), fetches jobs, executes them in a sandboxed workspace, and streams results/metrics back.

---

## 1) Scope & MVP

**MVP Features**
- Node registration with Coordinator (auth token + capability descriptor).
- Heartbeat & liveness (interval ± jitter, backoff on failure).
- Job fetch → ack → execute → upload result → finalize.
- Two runner types:
  - **CLI runner**: executes a provided command with arguments (whitelist‑based).
  - **Python runner**: executes a trusted task module with parameters.
- CPU/GPU capability detection (CUDA, VRAM, driver info) without Docker.
- Sandboxed working dir per job under `/var/lib/aitbc/miner/jobs/<job-id>`.
- Resource controls (nice/ionice/ulimit; optional cgroup v2 if present).
- Structured JSON logging and minimal metrics.

**Post‑MVP**
- Chunked artifact upload; resumable transfers.
- Prometheus `/metrics` endpoint (pull).
- GPU multi‑card scheduling & fractional allocation policy.
- On‑node model cache management (size, eviction, pinning).
- Signed task manifests & attestation of execution.
- Secure TMPFS for secrets; hardware key support (YubiKey).

---

## 2) High‑Level Architecture

```
client → coordinator-api → miner-node(s) → results store → coordinator-api → client
```

Miner components:
- **Agent** (control loop): registration, heartbeat, fetch/dispatch, result reporting.
- **Capability Probe**: CPU/GPU inventory (CUDA, VRAM), free RAM/disk, load.
- **Schedulers**: simple FIFO for MVP; one job per GPU or CPU slot.
- **Runners**: CLI runner & Python runner.
- **Sandbox**: working dirs, resource limits, network I/O gating (optional), file allowlist.
- **Telemetry**: JSON logs, minimal metrics; per‑job timeline.

---

## 3) Directory Layout (on node)

```
/var/lib/aitbc/miner/
  ├─ jobs/
  │   ├─ <job-id>/
  │   │   ├─ input/
  │   │   ├─ work/
  │   │   ├─ output/
  │   │   └─ logs/
  ├─ cache/            # model/assets cache (optional)
  └─ tmp/
/etc/aitbc/miner/
  ├─ config.yaml
  └─ allowlist.d/      # allowed CLI programs & argument schema snippets
/var/log/aitbc/miner/
/usr/local/lib/aitbc/miner/  # python package venv install target
```

---

## 4) Config (YAML)

```yaml
node_id: "node-<shortid>"
coordinator:
  base_url: "https://coordinator.local/api/v1"
  auth_token: "env:MINER_AUTH"        # read from env at runtime
  tls_verify: true
  timeout_s: 20

heartbeat:
  interval_s: 15
  jitter_pct: 10
  backoff:
    min_s: 5
    max_s: 120

runners:
  cli:
    enable: true
    allowlist_files:
      - "/etc/aitbc/miner/allowlist.d/ffmpeg.yaml"
      - "/etc/aitbc/miner/allowlist.d/whisper.yaml"
  python:
    enable: true
    task_paths:
      - "/usr/local/lib/aitbc/miner/tasks"
    venv: "/usr/local/lib/aitbc/miner/.venv"

resources:
  max_concurrent_cpu: 2
  max_concurrent_gpu: 1
  cpu_nice: 10
  io_class: "best-effort"
  io_level: 6
  mem_soft_mb: 16384

workspace:
  root: "/var/lib/aitbc/miner/jobs"
  keep_success: 24h
  keep_failed: 7d

logging:
  level: "info"
  json: true
  path: "/var/log/aitbc/miner/miner.jsonl"
```

---

## 5) Environment & Dependencies

- **OS:** Debian 12/13 (systemd).
- **Python:** 3.11+ in venv under `/usr/local/lib/aitbc/miner/.venv`.
- **Libraries:** `httpx`, `pydantic`, `uvloop` (optional), `pyyaml`, `psutil`.
- **GPU (optional):** NVIDIA driver installed; `nvidia-smi` available; CUDA 12.x runtime on path for GPU tasks.

**Install skeleton**
```
python3 -m venv /usr/local/lib/aitbc/miner/.venv
/usr/local/lib/aitbc/miner/.venv/bin/pip install --upgrade pip
/usr/local/lib/aitbc/miner/.venv/bin/pip install httpx pydantic pyyaml psutil uvloop
install -d /etc/aitbc/miner /var/lib/aitbc/miner/{jobs,cache,tmp} /var/log/aitbc/miner
```

---

## 6) Systemd Service

**/etc/systemd/system/aitbc-miner.service**
```
[Unit]
Description=AITBC Miner Node
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=MINER_AUTH=***REDACTED***
ExecStart=/usr/local/lib/aitbc/miner/.venv/bin/python -m aitbc_miner --config /etc/aitbc/miner/config.yaml
User=games
Group=games
# Lower CPU/IO priority by default
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=6
Restart=always
RestartSec=5
# Hardening
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/aitbc/miner /var/log/aitbc/miner

[Install]
WantedBy=multi-user.target
```

---

## 7) Capability Probe (sent to Coordinator)

Example payload:
```json
{
  "node_id": "node-abc123",
  "version": "0.1.0",
  "cpu": {"cores": 16, "arch": "x86_64"},
  "memory_mb": 64000,
  "disk_free_mb": 250000,
  "gpu": [
    {
      "vendor": "nvidia", "name": "RTX 4060 Ti 16GB",
      "vram_mb": 16384,
      "cuda": {"version": "12.3", "driver": "545.23.06"}
    }
  ],
  "runners": ["cli", "python"],
  "tags": ["debian", "cuda", "cpu"]
}
```

---

## 8) Coordinator API Contract (MVP)

**Endpoints (HTTPS, JSON):**
- `POST /nodes/register` → returns signed `node_token` (or 401)
- `POST /nodes/heartbeat` → `{node_id, load, free_mb, gpu_free}` → 200
- `POST /jobs/pull` → `{node_id, filters}` → `{job|none}`
- `POST /jobs/ack` → `{job_id, node_id}` → 200
- `POST /jobs/progress` → `{job_id, pct, note}` → 200
- `POST /jobs/result` → multipart (metadata.json + artifacts/*) → 200
- `POST /jobs/fail` → `{job_id, error_code, error_msg, logs_ref}` → 200

**Auth**
- Bearer token in header (Node → Coordinator): `Authorization: Bearer <node_token>`
- Coordinator signs `job.manifest` with HMAC(sha256) or Ed25519 (post‑MVP).

**Job manifest (subset)**
```json
{
  "job_id": "j-20250926-001",
  "runner": "cli",
  "requirements": {"gpu": true, "vram_mb": 12000, "cpu_threads": 4},
  "timeout_s": 3600,
  "input": {"urls": ["https://.../input1"], "inline": {"text": "..."}},
  "command": "ffmpeg",
  "args": ["-y", "-i", "input1.mp4", "-c:v", "libx264", "output.mp4"],
  "artifacts": [{"path": "output.mp4", "type": "video/mp4", "max_mb": 5000}]
}
```

---

## 9) Runner Design

### CLI Runner
- Validate `command` against allowlist (`/etc/aitbc/miner/allowlist.d/*.yaml`).
- Validate `args` against per‑tool schema (regex & size caps).
- Materialize inputs in job workspace; set `PATH`, `CUDA_VISIBLE_DEVICES`.
- Launch via `subprocess.Popen` with `preexec_fn` applying `nice`, `ionice`, `setrlimit`.
- Live‑tail stdout/stderr to `logs/exec.log`; throttle progress pings.

### Python Runner
- Import trusted module `tasks.<name>:run(**params)` from configured paths.
- Run in same venv; optional `venv per task` later.
- Enforce timeouts; capture logs; write artifacts to `output/`.

---

## 10) Resource Controls (No Docker)

- **CPU:** `nice(10)`; optional cgroups v2 CPU.max if available.
- **IO:** `ionice -c 2 -n 6` (best‑effort) for heavy disk ops.
- **Memory:** `setrlimit(RLIMIT_AS)` soft cap; kill on OOM.
- **GPU:** select by policy (least used VRAM). No hard memory partitioning in MVP.
- **Network:** allowlist outbound hosts; deny by default (optional phase‑2).

---

## 11) Job Lifecycle (State Machine)

`IDLE → PULLING → ACKED → PREP → RUNNING → UPLOADING → DONE | FAILED | RETRY_WAIT`

- Retries: exponential backoff, max N; idempotent uploads.
- On crash: on‑start recovery scans `jobs/*/state.json` and reconciles with Coordinator.

---

## 12) Logging & Metrics

- JSON lines in `/var/log/aitbc/miner/miner.jsonl` with fields: `ts, level, node_id, job_id, event, attrs{}`.
- Optional `/healthz` (HTTP) returning 200 + brief status.
- Future: Prometheus `/metrics` with gauges (queue, running, VRAM free, CPU load).

---

## 13) Security Model

- TLS required; pin CA or enable cert validation per env.
- Node bootstrap token (`MINER_AUTH`) exchanged for `node_token` at registration.
- Strict allowlist for CLI tools + args; size/time caps.
- Secrets never written to disk unencrypted; pass via env vars or in‑memory.
- Wipe workdirs on success (per policy); keep failed for triage.

---

## 14) Windsurf Implementation Plan

**Milestone 1 — Skeleton**
1. `aitbc_miner/` package: `main.py`, `config.py`, `agent.py`, `probe.py`, `runners/{cli.py, python.py}`, `util/{limits.py, fs.py, log.py}`.
2. Load YAML config, bootstrap logs, print probe JSON.
3. Implement `/healthz` (optional FastAPI or bare aiohttp) for local checks.

**Milestone 2 — Control Loop**
1. Register → store `node_token` (in memory only).
2. Heartbeat task (async), backoff on network errors.
3. Pull/ack & single‑slot executor; write state.json.

**Milestone 3 — Runners**
1. CLI allowlist loader + validator; subprocess with limits.
2. Python runner calling `tasks.example:run`.
3. Upload artifacts via multipart; handle large files with chunking stub.

**Milestone 4 — Hardening & Ops**
1. Crash recovery; cleanup policy; TTL sweeper.
2. Metrics counters; structured logging fields.
3. Systemd unit; install scripts; doc.

---

## 15) Minimal Allowlist Example (ffmpeg)

```yaml
# /etc/aitbc/miner/allowlist.d/ffmpeg.yaml
command:
  path: "/usr/bin/ffmpeg"
args:
  - ["-y"]
  - ["-i", ".+\\.(mp4|wav|mkv)$"]
  - ["-c:v", "(libx264|copy)"]
  - ["-c:a", "(aac|copy)"]
  - ["-b:v", "[1-9][0-9]{2,5}k"]
  - ["output\\.(mp4|mkv)"]
max_total_args_len: 4096
max_runtime_s: 7200
max_output_mb: 5000
```

---

## 16) Mock Coordinator (for local testing)

> Run a tiny dev server to hand out a single job and accept results.

```python
# mock_coordinator.py (FastAPI)
from fastapi import FastAPI, UploadFile, File, Form
from pydantic import BaseModel

app = FastAPI()
JOB = {
  "job_id": "j-local-1",
  "runner": "cli",
  "requirements": {"gpu": False},
  "timeout_s": 120,
  "command": "echo",
  "args": ["hello", "world"],
  "artifacts": [{"path": "output.txt", "type": "text/plain", "max_mb": 1}]
}

class PullReq(BaseModel):
    node_id: str
    filters: dict | None = None

@app.post("/api/v1/jobs/pull")
def pull(req: PullReq):
    return {"job": JOB}

@app.post("/api/v1/jobs/ack")
def ack(job_id: str, node_id: str):
    return {"ok": True}

@app.post("/api/v1/jobs/result")
def result(job_id: str = Form(...), metadata: str = Form(...), artifact: UploadFile = File(...)):
    return {"ok": True}
```

---

## 17) Developer UX (Make Targets)

```
make venv        # create venv + install deps
make run         # run miner with local config
make fmt         # ruff/black (optional)
make test        # unit tests
```

---

## 18) Operational Runbook

- **Start/Stop**: `systemctl enable --now aitbc-miner`
- **Logs**: `journalctl -u aitbc-miner -f` and `/var/log/aitbc/miner/miner.jsonl`
- **Rotate**: logrotate config (size 50M, keep 7)
- **Upgrade**: drain → stop → replace venv → start → verify heartbeat
- **Health**: `/healthz` 200 + JSON `{running, queued, cpu_load, vram_free}`

---

## 19) Failure Modes & Recovery

- **Network errors**: exponential backoff; keep heartbeat local status.
- **Job invalid**: fail fast with reason; do not retry.
- **Runner denied**: allowlist miss → fail with `E_DENY`.
- **OOM**: kill process group; mark `E_OOM`.
- **GPU unavailable**: requeue with reason `E_NOGPU`.

---

## 20) Roadmap Notes

- Binary task bundles with signed SBOM.
- Remote cache warming via Coordinator hints.
- Multi‑Queue scheduling (latency vs throughput).
- MIG/compute‑instance support if hardware allows.

---

## 21) Checklist for Windsurf

1. Create `aitbc_miner/` package skeleton with modules listed in §14.
2. Implement config loader + capability probe output.
3. Implement async agent loop: register → heartbeat → pull/ack.
4. Implement CLI runner with allowlist (§15) and exec log.
5. Implement Python runner stub (`tasks/example.py`).
6. Write result uploader (multipart) and finalize call.
7. Add systemd unit (§6) and basic install script.
8. Test end‑to‑end against `mock_coordinator.py` (§16).
9. Document log fields + troubleshooting card.
10. Add optional `/healthz` endpoint.