docs: document secrets management hardening implementation and update setup script

Add comprehensive Phase 5 documentation covering credential system implementation with /etc/aitbc/credentials/ directory (700 permissions), runtime secret loading via load-keystore-secrets.sh and systemd ExecStartPre hooks, removal of insecure API_KEY_HASH_SECRET defaults from tenant_context.py and tenant_management.py, keystore permission fixes (all files now 600), and setup_production.py migration from clear text
2026-04-28 07:35:25 +02:00
parent 04852fc480
commit 2d9ae644ff
2 changed files with 185 additions and 35 deletions
--- a/docs/security/SECURITY_FIXES_SUMMARY.md
+++ b/docs/security/SECURITY_FIXES_SUMMARY.md
@@ -68,6 +68,63 @@
 - Vulnerabilities found but ignored due to unpinned requirements (>= version ranges)
 - This is expected behavior for development dependencies

+### Phase 5: Secrets Management Hardening (April 28, 2026)
+
+#### Credential System Implementation
+- **Created credential directory**: `/etc/aitbc/credentials/` with 700 permissions
+- **Generated secure secrets**:
+  - API_KEY_HASH_SECRET (64-byte hex)
+  - keystore_password (64-byte hex)
+  - proposer_id (copied from .env)
+- **All credential files**: 600 permissions (root read/write only)
+
+#### Runtime Secret Loading
+- **Created load-keystore-secrets.sh**: Loads secrets at service startup
+- **Runtime directory**: `/run/aitbc/secrets/` (tmpfs, cleared on reboot)
+- **Systemd integration**: Services use ExecStartPre to load secrets
+- **Services updated**:
+  - aitbc-blockchain-node.service
+  - aitbc-blockchain-rpc.service
+  - aitbc-wallet.service
+  - aitbc-coordinator-api.service
+
+#### Insecure Default Removal
+- **Removed API_KEY_HASH_SECRET default** from:
+  - tenant_context.py (line 155)
+  - tenant_management.py (line 366)
+- **Now required**: Services fail if API_KEY_HASH_SECRET not set
+- **Error handling**: HTTP 500 error with clear message
+
+#### Keystore Permission Fixes
+- **Fixed permissions**: All files in /var/lib/aitbc/keystore/ now 600
+- **Directory permissions**: 700 on keystore and subdirectories
+- **Files fixed**:
+  - .agent_daemon_password (was 644)
+  - genesis.json.backup (was 644)
+  - .password (was 640)
+  - All *.json files (some were 644)
+
+#### Setup Script Updates
+- **Updated /opt/aitbc/scripts/setup.sh**:
+  - Added credential directory creation
+  - Added setup_credentials() function
+  - Generates secure secrets during installation
+  - Uses link-systemd.sh for service installation
+- **Updated /opt/aitbc/scripts/utils/setup_production.py**:
+  - Removed clear text password storage
+  - Uses credential system for keystore password
+  - Password stored in /etc/aitbc/credentials/keystore_password
+
+#### Documentation Updates
+- **Updated /var/lib/aitbc/keystore/README.md**:
+  - Documented credential system
+  - Added security notes
+  - Added script references
+- **Updated setup script output**:
+  - Added credential directory information
+  - Added security notes
+  - Added load-secrets command
+
 ## Security Best Practices Implemented

 ### Logging Security
@@ -142,11 +199,24 @@
 ### Scripts
 - scripts/utils/generate-api-keys.py
 - scripts/security/security_audit.py
+- scripts/utils/load-keystore-secrets.sh (new)
+- scripts/utils/setup-credentials.py (new)
+- scripts/utils/setup_production.py (updated)
+- scripts/setup.sh (updated)
+
+### Systemd Services
+- systemd/aitbc-blockchain-node.service (updated)
+- systemd/aitbc-blockchain-rpc.service (updated)
+- systemd/aitbc-wallet.service (updated)
+- systemd/aitbc-coordinator-api.service (updated)

 ### Infrastructure
 - .github/workflows/codeql.yml
 - .github/codeql/suppressions.yml
 - .gitignore
+- /etc/aitbc/.env (updated)
+- /etc/aitbc/credentials/ (new directory)
+- /var/lib/aitbc/keystore/README.md (updated)

 ## Security Metrics

@@ -166,6 +236,14 @@
 - Print statements: 0 remaining (replaced with logger)
 - Log injection: 9 key instances fixed, remaining 289 are low-risk

+### Phase 5: Secrets Management Hardening (April 28, 2026)
+- Credential system: Implemented with 600/700 permissions
+- Insecure defaults: Removed (API_KEY_HASH_SECRET now required)
+- Keystore permissions: All files now 600 (was mixed 644/640)
+- Clear text passwords: Removed from setup_production.py
+- Runtime secret loading: Implemented via systemd ExecStartPre
+- Setup script: Updated to generate secure credentials automatically
+
 ### Reduction
 - Exploitable vulnerabilities: 100% reduction
 - High-priority security issues: 100% reduction
@@ -191,8 +269,11 @@
 5. Use environment variables for configuration
 6. Use logger instead of print statements
 7. Run CodeQL before committing security-sensitive changes
+8. Use credential system for secrets (600 permissions)
+9. Never use insecure default values for secrets
+10. Load secrets at runtime via systemd ExecStartPre

 ---
 **Status**: Comprehensive security remediation completed ✅
-**Date**: April 24, 2026
+**Date**: April 28, 2026 (Phase 5: Secrets Management Hardening)
 **Next Review**: May 2026 (monthly dependency updates recommended)
--- a/scripts/setup.sh
+++ b/scripts/setup.sh
@@ -90,9 +90,13 @@ setup_runtime_directories() {
    directories=(
        "/var/lib/aitbc"
        "/var/lib/aitbc/keystore"
+        "/var/lib/aitbc/keystore/config"
+        "/var/lib/aitbc/keystore/passwords"
        "/var/lib/aitbc/data"
        "/var/log/aitbc"
        "/etc/aitbc"
+        "/etc/aitbc/credentials"
+        "/run/aitbc/secrets"
    )
    
    for dir in "${directories[@]}"; do
@@ -103,16 +107,24 @@ setup_runtime_directories() {
    # Set permissions
    chmod 755 /var/lib/aitbc
    chmod 700 /var/lib/aitbc/keystore  # Secure keystore
+    chmod 700 /var/lib/aitbc/keystore/config
+    chmod 700 /var/lib/aitbc/keystore/passwords
    chmod 755 /var/lib/aitbc/data
    chmod 755 /var/log/aitbc
    chmod 755 /etc/aitbc
+    chmod 700 /etc/aitbc/credentials  # Secure credentials
+    chmod 700 /run/aitbc/secrets  # Runtime secrets (tmpfs)
    
    # Set ownership
    chown root:root /var/lib/aitbc
    chown root:root /var/lib/aitbc/keystore
+    chown root:root /var/lib/aitbc/keystore/config
+    chown root:root /var/lib/aitbc/keystore/passwords
    chown root:root /var/lib/aitbc/data
    chown root:root /var/log/aitbc
    chown root:root /etc/aitbc
+    chown root:root /etc/aitbc/credentials
+    chown root:root /run/aitbc/secrets
    
    # Disable Btrfs CoW on data directory to prevent SQLite corruption
    # SQLite expects overwrite-in-place behavior, which conflicts with CoW
@@ -124,9 +136,13 @@ setup_runtime_directories() {
    # Create README files
    echo "# AITBC Runtime Data Directory" > /var/lib/aitbc/README.md
    echo "# Keystore for blockchain keys (SECURE)" > /var/lib/aitbc/keystore/README.md
+    echo "# Encrypted configuration secrets" > /var/lib/aitbc/keystore/config/README.md
+    echo "# Encrypted password storage" > /var/lib/aitbc/keystore/passwords/README.md
    echo "# Application databases" > /var/lib/aitbc/data/README.md
    echo "# Application logs" > /var/log/aitbc/README.md
    echo "# AITBC Configuration Files" > /etc/aitbc/README.md
+    echo "# Secure credential storage (600 permissions)" > /etc/aitbc/credentials/README.md
+    echo "# Runtime secrets (tmpfs, cleared on reboot)" > /run/aitbc/secrets/README.md
    
    success "Runtime directories setup completed"
 }
@@ -200,6 +216,43 @@ EOF
    success "Node identities setup completed"
 }

+# Setup secure credentials
+setup_credentials() {
+    log "Setting up secure credentials..."
+
+    # Generate secure secrets if they don't exist
+    if [ ! -f "/etc/aitbc/credentials/api_hash_secret" ]; then
+        python3 -c "import secrets; print(secrets.token_hex(32))" > /etc/aitbc/credentials/api_hash_secret
+        chmod 600 /etc/aitbc/credentials/api_hash_secret
+        log "Generated API_KEY_HASH_SECRET"
+    else
+        log "API_KEY_HASH_SECRET already exists"
+    fi
+
+    if [ ! -f "/etc/aitbc/credentials/keystore_password" ]; then
+        python3 -c "import secrets; print(secrets.token_hex(32))" > /etc/aitbc/credentials/keystore_password
+        chmod 600 /etc/aitbc/credentials/keystore_password
+        log "Generated keystore password"
+    else
+        log "Keystore password already exists"
+    fi
+
+    # Copy proposer_id from .env to credentials
+    if [ -f "/etc/aitbc/.env" ] && grep -q "^proposer_id=" /etc/aitbc/.env; then
+        grep "^proposer_id=" /etc/aitbc/.env | cut -d'=' -f2 > /etc/aitbc/credentials/proposer_id
+        chmod 600 /etc/aitbc/credentials/proposer_id
+        log "Copied proposer_id to credentials"
+    fi
+
+    # Add API_KEY_HASH_SECRET to .env if not present
+    if [ -f "/etc/aitbc/.env" ] && ! grep -q "^API_KEY_HASH_SECRET=" /etc/aitbc/.env; then
+        echo "API_KEY_HASH_SECRET=$(cat /etc/aitbc/credentials/api_hash_secret)" >> /etc/aitbc/.env
+        log "Added API_KEY_HASH_SECRET to .env"
+    fi
+
+    success "Secure credentials setup completed"
+}
+
 # Setup Python virtual environments
 setup_venvs() {
    log "Setting up Python virtual environments..."
@@ -232,38 +285,45 @@ setup_venvs() {
 # Install systemd services
 install_services() {
    log "Installing systemd services..."
-    
-    # Install core services
-    services=(
-        "aitbc-wallet.service"
-        "aitbc-coordinator-api.service"
-        "aitbc-exchange-api.service"
-        "aitbc-blockchain-node.service"
-        "aitbc-blockchain-rpc.service"
-        "aitbc-marketplace.service"
-        "aitbc-openclaw.service"
-        "aitbc-ai.service"
-        "aitbc-learning.service"
-        "aitbc-explorer.service"
-        "aitbc-web-ui.service"
-        "aitbc-agent-coordinator.service"
-        "aitbc-agent-registry.service"
-        "aitbc-multimodal.service"
-        "aitbc-modality-optimization.service"
-    )
-    
-    for service in "${services[@]}"; do
-        if [ -f "/opt/aitbc/systemd/$service" ]; then
-            log "Installing $service..."
-            ln -sf "/opt/aitbc/systemd/$service" /etc/systemd/system/
-        else
-            warning "Service file not found: $service"
-        fi
-    done
-    
-    # Reload systemd
-    systemctl daemon-reload
-    
+
+    # Use link-systemd.sh script for consistent service linking
+    if [ -f "/opt/aitbc/scripts/utils/link-systemd.sh" ]; then
+        log "Using link-systemd.sh to install services..."
+        /opt/aitbc/scripts/utils/link-systemd.sh
+    else
+        warning "link-systemd.sh not found, using manual installation..."
+        # Install core services
+        services=(
+            "aitbc-wallet.service"
+            "aitbc-coordinator-api.service"
+            "aitbc-exchange-api.service"
+            "aitbc-blockchain-node.service"
+            "aitbc-blockchain-rpc.service"
+            "aitbc-marketplace.service"
+            "aitbc-openclaw.service"
+            "aitbc-ai.service"
+            "aitbc-learning.service"
+            "aitbc-explorer.service"
+            "aitbc-web-ui.service"
+            "aitbc-agent-coordinator.service"
+            "aitbc-agent-registry.service"
+            "aitbc-multimodal.service"
+            "aitbc-modality-optimization.service"
+        )
+
+        for service in "${services[@]}"; do
+            if [ -f "/opt/aitbc/systemd/$service" ]; then
+                log "Installing $service..."
+                ln -sf "/opt/aitbc/systemd/$service" /etc/systemd/system/
+            else
+                warning "Service file not found: $service"
+            fi
+        done
+
+        # Reload systemd
+        systemctl daemon-reload
+    fi
+
    success "Systemd services installed"
 }

@@ -406,18 +466,19 @@ setup_autostart() {
 # Main function
 main() {
    log "Starting AITBC setup..."
-    
+
    check_root
    check_prerequisites
    clone_repo
    setup_runtime_directories
    setup_node_identities
+    setup_credentials
    setup_venvs
    install_services
    create_health_check
    start_services
    setup_autostart
-    
+
    success "AITBC setup completed!"
    echo ""
    echo "Service Information:"
@@ -430,11 +491,19 @@ main() {
    echo "  Data: /var/lib/aitbc/data/"
    echo "  Logs: /var/lib/aitbc/logs/"
    echo "  Config: /etc/aitbc/"
+    echo "  Credentials: /etc/aitbc/credentials/ (600 permissions)"
+    echo "  Runtime secrets: /run/aitbc/secrets/ (tmpfs)"
    echo ""
    echo "Management Commands:"
    echo "  Health check: /opt/aitbc/health-check.sh"
+    echo "  Load secrets: /opt/aitbc/scripts/utils/load-keystore-secrets.sh"
    echo "  Restart services: systemctl restart aitbc-wallet aitbc-coordinator-api aitbc-exchange-api"
    echo "  View logs: journalctl -u aitbc-wallet -f"
+    echo ""
+    echo "Security Notes:"
+    echo "  - Secrets stored in /etc/aitbc/credentials/ with 600 permissions"
+    echo "  - Services load secrets at runtime via systemd ExecStartPre"
+    echo "  - API_KEY_HASH_SECRET required (no insecure default)"
 }

 # Run main function