aitbc/.windsurf/skills/sql-database-infrastructure.md

SQL Database Infrastructure for AITBC

Purpose

Document PostgreSQL and SQLite database infrastructure configuration for AITBC nodes, including setup, migration, and troubleshooting procedures.

Architecture Overview

Mixed Database Strategy

AITBC uses a mixed database architecture:

• PostgreSQL: Application services (coordinator, exchange, marketplace, wallet)
• SQLite: Blockchain node chain data (chain-specific databases)

PostgreSQL Databases

• aitbc_coordinator: Coordinator API relational data
• aitbc_exchange: Exchange API trading data
• aitbc_marketplace: Marketplace service listings and bids
• aitbc_mempool: Mempool persistence
• aitbc_wallet: Wallet service account data

SQLite Databases

• Chain-specific: /var/lib/aitbc/data/{chain_id}/chain.db
  • /var/lib/aitbc/data/ait-mainnet/chain.db - Mainnet blockchain data
  • /var/lib/aitbc/data/ait-testnet/chain.db - Testnet blockchain data

PostgreSQL Setup

Installation

# Install PostgreSQL on Debian/Ubuntu
sudo apt update
sudo apt install postgresql postgresql-contrib

# Start PostgreSQL service
sudo systemctl start postgresql
sudo systemctl enable postgresql

Database and User Creation

# Switch to postgres user
sudo su postgres

# Create databases
createdb aitbc_coordinator
createdb aitbc_exchange
createdb aitbc_marketplace
createdb aitbc_mempool
createdb aitbc_wallet

# Create users with passwords
psql -c "CREATE USER aitbc_coordinator WITH PASSWORD 'secure_password';"
psql -c "CREATE USER aitbc_exchange WITH PASSWORD 'secure_password';"
psql -c "CREATE USER aitbc_marketplace WITH PASSWORD 'secure_password';"
psql -c "CREATE USER aitbc_mempool WITH PASSWORD 'secure_password';"
psql -c "CREATE USER aitbc_wallet WITH PASSWORD 'secure_password';"

# Grant privileges
psql -d aitbc_coordinator -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_coordinator TO aitbc_coordinator;"
psql -d aitbc_exchange -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_exchange TO aitbc_exchange;"
psql -d aitbc_marketplace -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_marketplace TO aitbc_marketplace;"
psql -d aitbc_mempool -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_mempool TO aitbc_mempool;"
psql -d aitbc_wallet -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_wallet TO aitbc_wallet;"

exit

Systemd Drop-in Configuration

Mempool (/etc/systemd/system/aitbc-blockchain-node.service.d/mempool-postgres.conf):

[Service]
Environment="MEMPOOL_DB_URL=postgresql+psycopg://aitbc_mempool:password@localhost:5432/aitbc_mempool"

Exchange (/etc/systemd/system/aitbc-exchange-api.service.d/exchange-postgres.conf):

[Service]
Environment="EXCHANGE_DATABASE_URL=postgresql+psycopg://aitbc_exchange:password@localhost:5432/aitbc_exchange"

Coordinator (/etc/systemd/system/aitbc-agent-coordinator.service.d/coordinator-postgres.conf):

[Service]
Environment="DATABASE_ADAPTER=postgresql"
Environment="DATABASE_URL=postgresql+psycopg://aitbc_coordinator:password@localhost:5432/aitbc_coordinator"

Apply Configuration

# Reload systemd to pick up drop-in files
sudo systemctl daemon-reload

# Restart services
sudo systemctl restart aitbc-blockchain-node.service
sudo systemctl restart aitbc-exchange-api.service
sudo systemctl restart aitbc-agent-coordinator.service

SQLite Setup

Btrfs CoW Disablement (Critical)

SQLite corruption on Btrfs filesystems is caused by CoW behavior. Disable CoW on data directory:

# Disable CoW on AITBC data directory
sudo chattr +C /var/lib/aitbc

# Verify CoW is disabled
lsattr -d /var/lib/aitbc
# Should show: ----C--------- /var/lib/aitbc

Chain-Specific Database Setup

# Create chain-specific directories
sudo mkdir -p /var/lib/aitbc/data/ait-mainnet
sudo mkdir -p /var/lib/aitbc/data/ait-testnet

# Set permissions
sudo chown -R aitbc:aitbc /var/lib/aitbc/data
sudo chmod -R 755 /var/lib/aitbc/data

WAL Mode Configuration

# Enable WAL mode for better performance and concurrency
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "PRAGMA journal_mode=WAL;"
sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "PRAGMA journal_mode=WAL;"

Note: WAL mode set via sqlite3 command doesn't persist after service restart. Enable at application level via database connection string or configuration.

Blockchain Node Configuration

Edit /etc/aitbc/blockchain.env:

# For SQLite (default, recommended for blockchain)
# DATABASE_URL=postgresql://aitbc:password@localhost:5432/aitbc_prod
DATA_DIR=/var/lib/aitbc
CHAIN_ID=ait-mainnet

Configuration Files

blockchain.env

Location: /etc/aitbc/blockchain.env

Key settings:

# Database URL (comment out to use SQLite)
# DATABASE_URL=postgresql://aitbc:secure_password@localhost:5432/aitbc_prod

# Data directory for chain-specific SQLite databases
DATA_DIR=/var/lib/aitbc

# Chain ID for multi-chain support
CHAIN_ID=ait-mainnet

# Redis URL for gossip protocol
REDIS_URL=redis://10.1.223.93:6379

Database Connection Strings

# PostgreSQL connection string format
postgresql+psycopg://username:password@localhost:5432/database_name

# SQLite connection string format
sqlite:////var/lib/aitbc/data/{chain_id}/chain.db

Migration: SQLite to PostgreSQL

Pre-Migration Checks

# Check PostgreSQL is running
sudo systemctl status postgresql

# Check PostgreSQL users and databases
sudo su postgres -c "psql -l"

# Backup existing SQLite databases
sudo cp /var/lib/aitbc/data/ait-mainnet/chain.db /var/lib/aitbc/data/ait-mainnet/chain.db.backup

Migration Steps

1. Create PostgreSQL databases and users (see PostgreSQL Setup)
2. Create systemd drop-in configuration files
3. Update application code to use PostgreSQL
4. Reload systemd and restart services
5. Verify database connectivity

Rollback Procedure

# Comment out DATABASE_URL in blockchain.env
sudo sed -i 's/^DATABASE_URL/#DATABASE_URL/' /etc/aitbc/blockchain.env

# Remove systemd drop-in files
sudo rm /etc/systemd/system/aitbc-blockchain-node.service.d/mempool-postgres.conf
sudo rm /etc/systemd/system/aitbc-exchange-api.service.d/exchange-postgres.conf
sudo rm /etc/systemd/system/aitbc-agent-coordinator.service.d/coordinator-postgres.conf

# Reload systemd and restart services
sudo systemctl daemon-reload
sudo systemctl restart aitbc-blockchain-node.service
sudo systemctl restart aitbc-exchange-api.service
sudo systemctl restart aitbc-agent-coordinator.service

Troubleshooting

Database Connection Issues

# Check PostgreSQL is running
sudo systemctl status postgresql

# Check PostgreSQL logs
sudo tail -50 /var/log/postgresql/postgresql-*.log

# Test PostgreSQL connection
sudo su postgres -c "psql -d aitbc_coordinator -c 'SELECT 1;'"

# Check service logs
sudo journalctl -u aitbc-blockchain-node --no-pager -n 50
sudo journalctl -u aitbc-exchange-api --no-pager -n 50
sudo journalctl -u aitbc-agent-coordinator --no-pager -n 50

SQLite Corruption

# Check SQLite database integrity
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "PRAGMA integrity_check;"

# If corruption detected, restore from backup
sudo cp /var/lib/aitbc/data/ait-mainnet/chain.db.backup /var/lib/aitbc/data/ait-mainnet/chain.db

# Ensure CoW is disabled
sudo chattr +C /var/lib/aitbc
lsattr -d /var/lib/aitbc

Missing PostgreSQL Database

# Check if database exists
sudo su postgres -c "psql -l | grep aitbc_prod"

# If missing, create it
sudo su postgres -c "createdb aitbc_prod"
sudo su postgres -c "psql -c \"CREATE USER aitbc WITH PASSWORD 'secure_password';\""
sudo su postgres -c "psql -d aitbc_prod -c \"GRANT ALL PRIVILEGES ON DATABASE aitbc_prod TO aitbc;\""

RPC Returns Wrong Block Height

Root Cause: The RPC service (aitbc-blockchain-rpc.service) runs as a separate systemd service from the blockchain node service (aitbc-blockchain-node.service). It maintains its own in-memory SQLAlchemy engine cache in the _engines dict. When only the blockchain node service is restarted, the RPC service's cached engines are not cleared, causing it to return stale data from a previous database state.

Database Engine Architecture (in database.py):

• Default _engine (non-chain-specific): DATA_DIR / "data" / "chain.db"
• Chain-specific engines (in _engines dict): DATA_DIR / "data" / {chain_id} / "chain.db"
• RPC uses chain-specific engines via session_scope(chain_id) → get_engine(chain_id)

Resolution Steps:

# Check which database RPC is reading from
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
sqlite3 /var/lib/aitbc/data/chain.db "SELECT COUNT(*), MAX(height) FROM block WHERE chain_id='ait-mainnet';"

# Check RPC output
curl -s http://localhost:8006/rpc/head?chain_id=ait-mainnet | jq '.'

# If RPC returns stale data, restart both services
sudo systemctl restart aitbc-blockchain-node.service
sudo systemctl restart aitbc-blockchain-rpc.service

# If default chain.db exists and is causing conflicts, move it
sudo mv /var/lib/aitbc/data/chain.db /var/lib/aitbc/data/chain.db.bak
sudo systemctl restart aitbc-blockchain-node.service
sudo systemctl restart aitbc-blockchain-rpc.service

Important: When making database configuration changes (e.g., removing DATABASE_URL, deleting conflicting database files), always restart BOTH aitbc-blockchain-node.service and aitbc-blockchain-rpc.service to ensure all in-memory caches are cleared.

Verification

PostgreSQL Verification

# Check all PostgreSQL databases
sudo su postgres -c "psql -l"

# Check database tables
sudo su postgres -c "psql -d aitbc_coordinator -c '\dt'"
sudo su postgres -c "psql -d aitbc_exchange -c '\dt'"
sudo su postgres -c "psql -d aitbc_mempool -c '\dt'"

# Test service connectivity
curl -s http://localhost:8000/v1/health
curl -s http://localhost:8001/health

SQLite Verification

# Check chain-specific databases
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"

# Check RPC reflects correct database
curl -s http://localhost:8006/rpc/head?chain_id=ait-mainnet | jq '.height'

Important Notes

• Never use PostgreSQL for blockchain chain data: Use SQLite for chain-specific databases
• Always disable CoW on Btrfs: SQLite corruption occurs on Btrfs without CoW disablement
• Use systemd drop-ins for service configuration: Don't modify main service files
• Backup before migration: Always backup SQLite databases before PostgreSQL migration
• Test connection strings: Verify database connectivity before restarting services
• Chain-specific databases: Blockchain node uses chain-specific SQLite databases for multi-chain support

Common Operations

Check PostgreSQL database size

sudo su postgres -c "psql -c \"SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) FROM pg_database;\""

Backup PostgreSQL database

sudo su postgres -c "pg_dump aitbc_coordinator > /var/backups/aitbc_coordinator.sql"

Restore PostgreSQL database

sudo su postgres -c "psql aitbc_coordinator < /var/backups/aitbc_coordinator.sql"

Check SQLite database size

ls -lh /var/lib/aitbc/data/ait-mainnet/chain.db
ls -lh /var/lib/aitbc/data/ait-testnet/chain.db

Vacuum SQLite database

sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "VACUUM;"

Multi-Node Operations

Apply PostgreSQL setup to all nodes

for node in aitbc1 gitea-runner; do
  ssh "$node" "sudo apt install postgresql postgresql-contrib -y"
  ssh "$node" "sudo systemctl start postgresql"
  ssh "$node" "sudo systemctl enable postgresql"
done

Check database status across all nodes

for node in aitbc1 gitea-runner; do
  echo "=== $node ==="
  ssh "$node" "sudo systemctl status postgresql --no-pager"
  ssh "$node" "sudo su postgres -c 'psql -l'"
done

Disable CoW on all nodes

for node in aitbc1 gitea-runner; do
  ssh "$node" "sudo chattr +C /var/lib/aitbc"
  ssh "$node" "lsattr -d /var/lib/aitbc"
done