From dc1c563f6ef0c5824c27e6cba2cea929d5c9721f Mon Sep 17 00:00:00 2001
From: aitbc <aitbc@keisanki.net>
Date: Mon, 11 May 2026 16:03:53 +0200
Subject: [PATCH] docs: add SQL database infrastructure documentation

- Added PostgreSQL and SQLite database architecture documentation
- Documented mixed database strategy (PostgreSQL for services, SQLite for blockchain)
- Added PostgreSQL setup with database and user creation procedures
- Added systemd drop-in configuration for mempool, exchange, and coordinator
- Added SQLite setup with Btrfs CoW disablement (critical for preventing corruption)
- Added chain-specific database setup for multi-chain support
- Added migration
---
 .../skills/sql-database-infrastructure.md     | 372 ++++++++++++++++++
 1 file changed, 372 insertions(+)
 create mode 100644 .windsurf/skills/sql-database-infrastructure.md

diff --git a/.windsurf/skills/sql-database-infrastructure.md b/.windsurf/skills/sql-database-infrastructure.md
new file mode 100644
index 00000000..0410e171
--- /dev/null
+++ b/.windsurf/skills/sql-database-infrastructure.md
@@ -0,0 +1,372 @@
+# SQL Database Infrastructure for AITBC
+
+## Purpose
+Document PostgreSQL and SQLite database infrastructure configuration for AITBC nodes, including setup, migration, and troubleshooting procedures.
+
+## Architecture Overview
+
+### Mixed Database Strategy
+AITBC uses a mixed database architecture:
+- **PostgreSQL**: Application services (coordinator, exchange, marketplace, wallet)
+- **SQLite**: Blockchain node chain data (chain-specific databases)
+
+### PostgreSQL Databases
+- **aitbc_coordinator**: Coordinator API relational data
+- **aitbc_exchange**: Exchange API trading data
+- **aitbc_marketplace**: Marketplace service listings and bids
+- **aitbc_mempool**: Mempool persistence
+- **aitbc_wallet**: Wallet service account data
+
+### SQLite Databases
+- **Chain-specific**: `/var/lib/aitbc/data/{chain_id}/chain.db`
+  - `/var/lib/aitbc/data/ait-mainnet/chain.db` - Mainnet blockchain data
+  - `/var/lib/aitbc/data/ait-testnet/chain.db` - Testnet blockchain data
+
+## PostgreSQL Setup
+
+### Installation
+```bash
+# Install PostgreSQL on Debian/Ubuntu
+sudo apt update
+sudo apt install postgresql postgresql-contrib
+
+# Start PostgreSQL service
+sudo systemctl start postgresql
+sudo systemctl enable postgresql
+```
+
+### Database and User Creation
+```bash
+# Switch to postgres user
+sudo su postgres
+
+# Create databases
+createdb aitbc_coordinator
+createdb aitbc_exchange
+createdb aitbc_marketplace
+createdb aitbc_mempool
+createdb aitbc_wallet
+
+# Create users with passwords
+psql -c "CREATE USER aitbc_coordinator WITH PASSWORD 'secure_password';"
+psql -c "CREATE USER aitbc_exchange WITH PASSWORD 'secure_password';"
+psql -c "CREATE USER aitbc_marketplace WITH PASSWORD 'secure_password';"
+psql -c "CREATE USER aitbc_mempool WITH PASSWORD 'secure_password';"
+psql -c "CREATE USER aitbc_wallet WITH PASSWORD 'secure_password';"
+
+# Grant privileges
+psql -d aitbc_coordinator -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_coordinator TO aitbc_coordinator;"
+psql -d aitbc_exchange -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_exchange TO aitbc_exchange;"
+psql -d aitbc_marketplace -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_marketplace TO aitbc_marketplace;"
+psql -d aitbc_mempool -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_mempool TO aitbc_mempool;"
+psql -d aitbc_wallet -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_wallet TO aitbc_wallet;"
+
+exit
+```
+
+### Systemd Drop-in Configuration
+
+**Mempool** (`/etc/systemd/system/aitbc-blockchain-node.service.d/mempool-postgres.conf`):
+```ini
+[Service]
+Environment="MEMPOOL_DB_URL=postgresql+psycopg://aitbc_mempool:password@localhost:5432/aitbc_mempool"
+```
+
+**Exchange** (`/etc/systemd/system/aitbc-exchange-api.service.d/exchange-postgres.conf`):
+```ini
+[Service]
+Environment="EXCHANGE_DATABASE_URL=postgresql+psycopg://aitbc_exchange:password@localhost:5432/aitbc_exchange"
+```
+
+**Coordinator** (`/etc/systemd/system/aitbc-agent-coordinator.service.d/coordinator-postgres.conf`):
+```ini
+[Service]
+Environment="DATABASE_ADAPTER=postgresql"
+Environment="DATABASE_URL=postgresql+psycopg://aitbc_coordinator:password@localhost:5432/aitbc_coordinator"
+```
+
+### Apply Configuration
+```bash
+# Reload systemd to pick up drop-in files
+sudo systemctl daemon-reload
+
+# Restart services
+sudo systemctl restart aitbc-blockchain-node.service
+sudo systemctl restart aitbc-exchange-api.service
+sudo systemctl restart aitbc-agent-coordinator.service
+```
+
+## SQLite Setup
+
+### Btrfs CoW Disablement (Critical)
+SQLite corruption on Btrfs filesystems is caused by CoW behavior. Disable CoW on data directory:
+
+```bash
+# Disable CoW on AITBC data directory
+sudo chattr +C /var/lib/aitbc
+
+# Verify CoW is disabled
+lsattr -d /var/lib/aitbc
+# Should show: ----C--------- /var/lib/aitbc
+```
+
+### Chain-Specific Database Setup
+```bash
+# Create chain-specific directories
+sudo mkdir -p /var/lib/aitbc/data/ait-mainnet
+sudo mkdir -p /var/lib/aitbc/data/ait-testnet
+
+# Set permissions
+sudo chown -R aitbc:aitbc /var/lib/aitbc/data
+sudo chmod -R 755 /var/lib/aitbc/data
+```
+
+### WAL Mode Configuration
+```bash
+# Enable WAL mode for better performance and concurrency
+sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "PRAGMA journal_mode=WAL;"
+sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "PRAGMA journal_mode=WAL;"
+```
+
+**Note**: WAL mode set via sqlite3 command doesn't persist after service restart. Enable at application level via database connection string or configuration.
+
+### Blockchain Node Configuration
+Edit `/etc/aitbc/blockchain.env`:
+```bash
+# For SQLite (default, recommended for blockchain)
+# DATABASE_URL=postgresql://aitbc:password@localhost:5432/aitbc_prod
+DATA_DIR=/var/lib/aitbc
+CHAIN_ID=ait-mainnet
+```
+
+## Configuration Files
+
+### blockchain.env
+Location: `/etc/aitbc/blockchain.env`
+
+Key settings:
+```bash
+# Database URL (comment out to use SQLite)
+# DATABASE_URL=postgresql://aitbc:secure_password@localhost:5432/aitbc_prod
+
+# Data directory for chain-specific SQLite databases
+DATA_DIR=/var/lib/aitbc
+
+# Chain ID for multi-chain support
+CHAIN_ID=ait-mainnet
+
+# Redis URL for gossip protocol
+REDIS_URL=redis://10.1.223.93:6379
+```
+
+### Database Connection Strings
+```bash
+# PostgreSQL connection string format
+postgresql+psycopg://username:password@localhost:5432/database_name
+
+# SQLite connection string format
+sqlite:////var/lib/aitbc/data/{chain_id}/chain.db
+```
+
+## Migration: SQLite to PostgreSQL
+
+### Pre-Migration Checks
+```bash
+# Check PostgreSQL is running
+sudo systemctl status postgresql
+
+# Check PostgreSQL users and databases
+sudo su postgres -c "psql -l"
+
+# Backup existing SQLite databases
+sudo cp /var/lib/aitbc/data/ait-mainnet/chain.db /var/lib/aitbc/data/ait-mainnet/chain.db.backup
+```
+
+### Migration Steps
+1. Create PostgreSQL databases and users (see PostgreSQL Setup)
+2. Create systemd drop-in configuration files
+3. Update application code to use PostgreSQL
+4. Reload systemd and restart services
+5. Verify database connectivity
+
+### Rollback Procedure
+```bash
+# Comment out DATABASE_URL in blockchain.env
+sudo sed -i 's/^DATABASE_URL/#DATABASE_URL/' /etc/aitbc/blockchain.env
+
+# Remove systemd drop-in files
+sudo rm /etc/systemd/system/aitbc-blockchain-node.service.d/mempool-postgres.conf
+sudo rm /etc/systemd/system/aitbc-exchange-api.service.d/exchange-postgres.conf
+sudo rm /etc/systemd/system/aitbc-agent-coordinator.service.d/coordinator-postgres.conf
+
+# Reload systemd and restart services
+sudo systemctl daemon-reload
+sudo systemctl restart aitbc-blockchain-node.service
+sudo systemctl restart aitbc-exchange-api.service
+sudo systemctl restart aitbc-agent-coordinator.service
+```
+
+## Troubleshooting
+
+### Database Connection Issues
+```bash
+# Check PostgreSQL is running
+sudo systemctl status postgresql
+
+# Check PostgreSQL logs
+sudo tail -50 /var/log/postgresql/postgresql-*.log
+
+# Test PostgreSQL connection
+sudo su postgres -c "psql -d aitbc_coordinator -c 'SELECT 1;'"
+
+# Check service logs
+sudo journalctl -u aitbc-blockchain-node --no-pager -n 50
+sudo journalctl -u aitbc-exchange-api --no-pager -n 50
+sudo journalctl -u aitbc-agent-coordinator --no-pager -n 50
+```
+
+### SQLite Corruption
+```bash
+# Check SQLite database integrity
+sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "PRAGMA integrity_check;"
+
+# If corruption detected, restore from backup
+sudo cp /var/lib/aitbc/data/ait-mainnet/chain.db.backup /var/lib/aitbc/data/ait-mainnet/chain.db
+
+# Ensure CoW is disabled
+sudo chattr +C /var/lib/aitbc
+lsattr -d /var/lib/aitbc
+```
+
+### Missing PostgreSQL Database
+```bash
+# Check if database exists
+sudo su postgres -c "psql -l | grep aitbc_prod"
+
+# If missing, create it
+sudo su postgres -c "createdb aitbc_prod"
+sudo su postgres -c "psql -c \"CREATE USER aitbc WITH PASSWORD 'secure_password';\""
+sudo su postgres -c "psql -d aitbc_prod -c \"GRANT ALL PRIVILEGES ON DATABASE aitbc_prod TO aitbc;\""
+```
+
+### RPC Returns Wrong Block Height
+
+**Root Cause**: The RPC service (`aitbc-blockchain-rpc.service`) runs as a separate systemd service from the blockchain node service (`aitbc-blockchain-node.service`). It maintains its own in-memory SQLAlchemy engine cache in the `_engines` dict. When only the blockchain node service is restarted, the RPC service's cached engines are not cleared, causing it to return stale data from a previous database state.
+
+**Database Engine Architecture** (in `database.py`):
+- **Default `_engine`** (non-chain-specific): `DATA_DIR / "data" / "chain.db"`
+- **Chain-specific engines** (in `_engines` dict): `DATA_DIR / "data" / {chain_id} / "chain.db"`
+- RPC uses chain-specific engines via `session_scope(chain_id)` → `get_engine(chain_id)`
+
+**Resolution Steps**:
+```bash
+# Check which database RPC is reading from
+sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
+sqlite3 /var/lib/aitbc/data/chain.db "SELECT COUNT(*), MAX(height) FROM block WHERE chain_id='ait-mainnet';"
+
+# Check RPC output
+curl -s http://localhost:8006/rpc/head?chain_id=ait-mainnet | jq '.'
+
+# If RPC returns stale data, restart both services
+sudo systemctl restart aitbc-blockchain-node.service
+sudo systemctl restart aitbc-blockchain-rpc.service
+
+# If default chain.db exists and is causing conflicts, move it
+sudo mv /var/lib/aitbc/data/chain.db /var/lib/aitbc/data/chain.db.bak
+sudo systemctl restart aitbc-blockchain-node.service
+sudo systemctl restart aitbc-blockchain-rpc.service
+```
+
+**Important**: When making database configuration changes (e.g., removing `DATABASE_URL`, deleting conflicting database files), always restart BOTH `aitbc-blockchain-node.service` and `aitbc-blockchain-rpc.service` to ensure all in-memory caches are cleared.
+
+## Verification
+
+### PostgreSQL Verification
+```bash
+# Check all PostgreSQL databases
+sudo su postgres -c "psql -l"
+
+# Check database tables
+sudo su postgres -c "psql -d aitbc_coordinator -c '\dt'"
+sudo su postgres -c "psql -d aitbc_exchange -c '\dt'"
+sudo su postgres -c "psql -d aitbc_mempool -c '\dt'"
+
+# Test service connectivity
+curl -s http://localhost:8000/v1/health
+curl -s http://localhost:8001/health
+```
+
+### SQLite Verification
+```bash
+# Check chain-specific databases
+sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
+sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
+
+# Check RPC reflects correct database
+curl -s http://localhost:8006/rpc/head?chain_id=ait-mainnet | jq '.height'
+```
+
+## Important Notes
+
+- **Never use PostgreSQL for blockchain chain data**: Use SQLite for chain-specific databases
+- **Always disable CoW on Btrfs**: SQLite corruption occurs on Btrfs without CoW disablement
+- **Use systemd drop-ins for service configuration**: Don't modify main service files
+- **Backup before migration**: Always backup SQLite databases before PostgreSQL migration
+- **Test connection strings**: Verify database connectivity before restarting services
+- **Chain-specific databases**: Blockchain node uses chain-specific SQLite databases for multi-chain support
+
+## Common Operations
+
+### Check PostgreSQL database size
+```bash
+sudo su postgres -c "psql -c \"SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) FROM pg_database;\""
+```
+
+### Backup PostgreSQL database
+```bash
+sudo su postgres -c "pg_dump aitbc_coordinator > /var/backups/aitbc_coordinator.sql"
+```
+
+### Restore PostgreSQL database
+```bash
+sudo su postgres -c "psql aitbc_coordinator < /var/backups/aitbc_coordinator.sql"
+```
+
+### Check SQLite database size
+```bash
+ls -lh /var/lib/aitbc/data/ait-mainnet/chain.db
+ls -lh /var/lib/aitbc/data/ait-testnet/chain.db
+```
+
+### Vacuum SQLite database
+```bash
+sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "VACUUM;"
+```
+
+## Multi-Node Operations
+
+### Apply PostgreSQL setup to all nodes
+```bash
+for node in aitbc1 gitea-runner; do
+  ssh "$node" "sudo apt install postgresql postgresql-contrib -y"
+  ssh "$node" "sudo systemctl start postgresql"
+  ssh "$node" "sudo systemctl enable postgresql"
+done
+```
+
+### Check database status across all nodes
+```bash
+for node in aitbc1 gitea-runner; do
+  echo "=== $node ==="
+  ssh "$node" "sudo systemctl status postgresql --no-pager"
+  ssh "$node" "sudo su postgres -c 'psql -l'"
+done
+```
+
+### Disable CoW on all nodes
+```bash
+for node in aitbc1 gitea-runner; do
+  ssh "$node" "sudo chattr +C /var/lib/aitbc"
+  ssh "$node" "lsattr -d /var/lib/aitbc"
+done
+```