docs: add SQL database infrastructure documentation
- Added PostgreSQL and SQLite database architecture documentation - Documented mixed database strategy (PostgreSQL for services, SQLite for blockchain) - Added PostgreSQL setup with database and user creation procedures - Added systemd drop-in configuration for mempool, exchange, and coordinator - Added SQLite setup with Btrfs CoW disablement (critical for preventing corruption) - Added chain-specific database setup for multi-chain support - Added migration
This commit is contained in:
aitbc
372
.windsurf/skills/sql-database-infrastructure.md
Normal file
372
.windsurf/skills/sql-database-infrastructure.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# SQL Database Infrastructure for AITBC
|
||||
|
||||
## Purpose
|
||||
Document PostgreSQL and SQLite database infrastructure configuration for AITBC nodes, including setup, migration, and troubleshooting procedures.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Mixed Database Strategy
|
||||
AITBC uses a mixed database architecture:
|
||||
- **PostgreSQL**: Application services (coordinator, exchange, marketplace, wallet)
|
||||
- **SQLite**: Blockchain node chain data (chain-specific databases)
|
||||
|
||||
### PostgreSQL Databases
|
||||
- **aitbc_coordinator**: Coordinator API relational data
|
||||
- **aitbc_exchange**: Exchange API trading data
|
||||
- **aitbc_marketplace**: Marketplace service listings and bids
|
||||
- **aitbc_mempool**: Mempool persistence
|
||||
- **aitbc_wallet**: Wallet service account data
|
||||
|
||||
### SQLite Databases
|
||||
- **Chain-specific**: `/var/lib/aitbc/data/{chain_id}/chain.db`
|
||||
- `/var/lib/aitbc/data/ait-mainnet/chain.db` - Mainnet blockchain data
|
||||
- `/var/lib/aitbc/data/ait-testnet/chain.db` - Testnet blockchain data
|
||||
|
||||
## PostgreSQL Setup
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
# Install PostgreSQL on Debian/Ubuntu
|
||||
sudo apt update
|
||||
sudo apt install postgresql postgresql-contrib
|
||||
|
||||
# Start PostgreSQL service
|
||||
sudo systemctl start postgresql
|
||||
sudo systemctl enable postgresql
|
||||
```
|
||||
|
||||
### Database and User Creation
|
||||
```bash
|
||||
# Switch to postgres user
|
||||
sudo su postgres
|
||||
|
||||
# Create databases
|
||||
createdb aitbc_coordinator
|
||||
createdb aitbc_exchange
|
||||
createdb aitbc_marketplace
|
||||
createdb aitbc_mempool
|
||||
createdb aitbc_wallet
|
||||
|
||||
# Create users with passwords
|
||||
psql -c "CREATE USER aitbc_coordinator WITH PASSWORD 'secure_password';"
|
||||
psql -c "CREATE USER aitbc_exchange WITH PASSWORD 'secure_password';"
|
||||
psql -c "CREATE USER aitbc_marketplace WITH PASSWORD 'secure_password';"
|
||||
psql -c "CREATE USER aitbc_mempool WITH PASSWORD 'secure_password';"
|
||||
psql -c "CREATE USER aitbc_wallet WITH PASSWORD 'secure_password';"
|
||||
|
||||
# Grant privileges
|
||||
psql -d aitbc_coordinator -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_coordinator TO aitbc_coordinator;"
|
||||
psql -d aitbc_exchange -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_exchange TO aitbc_exchange;"
|
||||
psql -d aitbc_marketplace -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_marketplace TO aitbc_marketplace;"
|
||||
psql -d aitbc_mempool -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_mempool TO aitbc_mempool;"
|
||||
psql -d aitbc_wallet -c "GRANT ALL PRIVILEGES ON DATABASE aitbc_wallet TO aitbc_wallet;"
|
||||
|
||||
exit
|
||||
```
|
||||
|
||||
### Systemd Drop-in Configuration
|
||||
|
||||
**Mempool** (`/etc/systemd/system/aitbc-blockchain-node.service.d/mempool-postgres.conf`):
|
||||
```ini
|
||||
[Service]
|
||||
Environment="MEMPOOL_DB_URL=postgresql+psycopg://aitbc_mempool:password@localhost:5432/aitbc_mempool"
|
||||
```
|
||||
|
||||
**Exchange** (`/etc/systemd/system/aitbc-exchange-api.service.d/exchange-postgres.conf`):
|
||||
```ini
|
||||
[Service]
|
||||
Environment="EXCHANGE_DATABASE_URL=postgresql+psycopg://aitbc_exchange:password@localhost:5432/aitbc_exchange"
|
||||
```
|
||||
|
||||
**Coordinator** (`/etc/systemd/system/aitbc-agent-coordinator.service.d/coordinator-postgres.conf`):
|
||||
```ini
|
||||
[Service]
|
||||
Environment="DATABASE_ADAPTER=postgresql"
|
||||
Environment="DATABASE_URL=postgresql+psycopg://aitbc_coordinator:password@localhost:5432/aitbc_coordinator"
|
||||
```
|
||||
|
||||
### Apply Configuration
|
||||
```bash
|
||||
# Reload systemd to pick up drop-in files
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Restart services
|
||||
sudo systemctl restart aitbc-blockchain-node.service
|
||||
sudo systemctl restart aitbc-exchange-api.service
|
||||
sudo systemctl restart aitbc-agent-coordinator.service
|
||||
```
|
||||
|
||||
## SQLite Setup
|
||||
|
||||
### Btrfs CoW Disablement (Critical)
|
||||
SQLite corruption on Btrfs filesystems is caused by CoW behavior. Disable CoW on data directory:
|
||||
|
||||
```bash
|
||||
# Disable CoW on AITBC data directory
|
||||
sudo chattr +C /var/lib/aitbc
|
||||
|
||||
# Verify CoW is disabled
|
||||
lsattr -d /var/lib/aitbc
|
||||
# Should show: ----C--------- /var/lib/aitbc
|
||||
```
|
||||
|
||||
### Chain-Specific Database Setup
|
||||
```bash
|
||||
# Create chain-specific directories
|
||||
sudo mkdir -p /var/lib/aitbc/data/ait-mainnet
|
||||
sudo mkdir -p /var/lib/aitbc/data/ait-testnet
|
||||
|
||||
# Set permissions
|
||||
sudo chown -R aitbc:aitbc /var/lib/aitbc/data
|
||||
sudo chmod -R 755 /var/lib/aitbc/data
|
||||
```
|
||||
|
||||
### WAL Mode Configuration
|
||||
```bash
|
||||
# Enable WAL mode for better performance and concurrency
|
||||
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "PRAGMA journal_mode=WAL;"
|
||||
sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "PRAGMA journal_mode=WAL;"
|
||||
```
|
||||
|
||||
**Note**: WAL mode set via sqlite3 command doesn't persist after service restart. Enable at application level via database connection string or configuration.
|
||||
|
||||
### Blockchain Node Configuration
|
||||
Edit `/etc/aitbc/blockchain.env`:
|
||||
```bash
|
||||
# For SQLite (default, recommended for blockchain)
|
||||
# DATABASE_URL=postgresql://aitbc:password@localhost:5432/aitbc_prod
|
||||
DATA_DIR=/var/lib/aitbc
|
||||
CHAIN_ID=ait-mainnet
|
||||
```
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### blockchain.env
|
||||
Location: `/etc/aitbc/blockchain.env`
|
||||
|
||||
Key settings:
|
||||
```bash
|
||||
# Database URL (comment out to use SQLite)
|
||||
# DATABASE_URL=postgresql://aitbc:secure_password@localhost:5432/aitbc_prod
|
||||
|
||||
# Data directory for chain-specific SQLite databases
|
||||
DATA_DIR=/var/lib/aitbc
|
||||
|
||||
# Chain ID for multi-chain support
|
||||
CHAIN_ID=ait-mainnet
|
||||
|
||||
# Redis URL for gossip protocol
|
||||
REDIS_URL=redis://10.1.223.93:6379
|
||||
```
|
||||
|
||||
### Database Connection Strings
|
||||
```bash
|
||||
# PostgreSQL connection string format
|
||||
postgresql+psycopg://username:password@localhost:5432/database_name
|
||||
|
||||
# SQLite connection string format
|
||||
sqlite:////var/lib/aitbc/data/{chain_id}/chain.db
|
||||
```
|
||||
|
||||
## Migration: SQLite to PostgreSQL
|
||||
|
||||
### Pre-Migration Checks
|
||||
```bash
|
||||
# Check PostgreSQL is running
|
||||
sudo systemctl status postgresql
|
||||
|
||||
# Check PostgreSQL users and databases
|
||||
sudo su postgres -c "psql -l"
|
||||
|
||||
# Backup existing SQLite databases
|
||||
sudo cp /var/lib/aitbc/data/ait-mainnet/chain.db /var/lib/aitbc/data/ait-mainnet/chain.db.backup
|
||||
```
|
||||
|
||||
### Migration Steps
|
||||
1. Create PostgreSQL databases and users (see PostgreSQL Setup)
|
||||
2. Create systemd drop-in configuration files
|
||||
3. Update application code to use PostgreSQL
|
||||
4. Reload systemd and restart services
|
||||
5. Verify database connectivity
|
||||
|
||||
### Rollback Procedure
|
||||
```bash
|
||||
# Comment out DATABASE_URL in blockchain.env
|
||||
sudo sed -i 's/^DATABASE_URL/#DATABASE_URL/' /etc/aitbc/blockchain.env
|
||||
|
||||
# Remove systemd drop-in files
|
||||
sudo rm /etc/systemd/system/aitbc-blockchain-node.service.d/mempool-postgres.conf
|
||||
sudo rm /etc/systemd/system/aitbc-exchange-api.service.d/exchange-postgres.conf
|
||||
sudo rm /etc/systemd/system/aitbc-agent-coordinator.service.d/coordinator-postgres.conf
|
||||
|
||||
# Reload systemd and restart services
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart aitbc-blockchain-node.service
|
||||
sudo systemctl restart aitbc-exchange-api.service
|
||||
sudo systemctl restart aitbc-agent-coordinator.service
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Database Connection Issues
|
||||
```bash
|
||||
# Check PostgreSQL is running
|
||||
sudo systemctl status postgresql
|
||||
|
||||
# Check PostgreSQL logs
|
||||
sudo tail -50 /var/log/postgresql/postgresql-*.log
|
||||
|
||||
# Test PostgreSQL connection
|
||||
sudo su postgres -c "psql -d aitbc_coordinator -c 'SELECT 1;'"
|
||||
|
||||
# Check service logs
|
||||
sudo journalctl -u aitbc-blockchain-node --no-pager -n 50
|
||||
sudo journalctl -u aitbc-exchange-api --no-pager -n 50
|
||||
sudo journalctl -u aitbc-agent-coordinator --no-pager -n 50
|
||||
```
|
||||
|
||||
### SQLite Corruption
|
||||
```bash
|
||||
# Check SQLite database integrity
|
||||
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "PRAGMA integrity_check;"
|
||||
|
||||
# If corruption detected, restore from backup
|
||||
sudo cp /var/lib/aitbc/data/ait-mainnet/chain.db.backup /var/lib/aitbc/data/ait-mainnet/chain.db
|
||||
|
||||
# Ensure CoW is disabled
|
||||
sudo chattr +C /var/lib/aitbc
|
||||
lsattr -d /var/lib/aitbc
|
||||
```
|
||||
|
||||
### Missing PostgreSQL Database
|
||||
```bash
|
||||
# Check if database exists
|
||||
sudo su postgres -c "psql -l | grep aitbc_prod"
|
||||
|
||||
# If missing, create it
|
||||
sudo su postgres -c "createdb aitbc_prod"
|
||||
sudo su postgres -c "psql -c \"CREATE USER aitbc WITH PASSWORD 'secure_password';\""
|
||||
sudo su postgres -c "psql -d aitbc_prod -c \"GRANT ALL PRIVILEGES ON DATABASE aitbc_prod TO aitbc;\""
|
||||
```
|
||||
|
||||
### RPC Returns Wrong Block Height
|
||||
|
||||
**Root Cause**: The RPC service (`aitbc-blockchain-rpc.service`) runs as a separate systemd service from the blockchain node service (`aitbc-blockchain-node.service`). It maintains its own in-memory SQLAlchemy engine cache in the `_engines` dict. When only the blockchain node service is restarted, the RPC service's cached engines are not cleared, causing it to return stale data from a previous database state.
|
||||
|
||||
**Database Engine Architecture** (in `database.py`):
|
||||
- **Default `_engine`** (non-chain-specific): `DATA_DIR / "data" / "chain.db"`
|
||||
- **Chain-specific engines** (in `_engines` dict): `DATA_DIR / "data" / {chain_id} / "chain.db"`
|
||||
- RPC uses chain-specific engines via `session_scope(chain_id)` → `get_engine(chain_id)`
|
||||
|
||||
**Resolution Steps**:
|
||||
```bash
|
||||
# Check which database RPC is reading from
|
||||
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
|
||||
sqlite3 /var/lib/aitbc/data/chain.db "SELECT COUNT(*), MAX(height) FROM block WHERE chain_id='ait-mainnet';"
|
||||
|
||||
# Check RPC output
|
||||
curl -s http://localhost:8006/rpc/head?chain_id=ait-mainnet | jq '.'
|
||||
|
||||
# If RPC returns stale data, restart both services
|
||||
sudo systemctl restart aitbc-blockchain-node.service
|
||||
sudo systemctl restart aitbc-blockchain-rpc.service
|
||||
|
||||
# If default chain.db exists and is causing conflicts, move it
|
||||
sudo mv /var/lib/aitbc/data/chain.db /var/lib/aitbc/data/chain.db.bak
|
||||
sudo systemctl restart aitbc-blockchain-node.service
|
||||
sudo systemctl restart aitbc-blockchain-rpc.service
|
||||
```
|
||||
|
||||
**Important**: When making database configuration changes (e.g., removing `DATABASE_URL`, deleting conflicting database files), always restart BOTH `aitbc-blockchain-node.service` and `aitbc-blockchain-rpc.service` to ensure all in-memory caches are cleared.
|
||||
|
||||
## Verification
|
||||
|
||||
### PostgreSQL Verification
|
||||
```bash
|
||||
# Check all PostgreSQL databases
|
||||
sudo su postgres -c "psql -l"
|
||||
|
||||
# Check database tables
|
||||
sudo su postgres -c "psql -d aitbc_coordinator -c '\dt'"
|
||||
sudo su postgres -c "psql -d aitbc_exchange -c '\dt'"
|
||||
sudo su postgres -c "psql -d aitbc_mempool -c '\dt'"
|
||||
|
||||
# Test service connectivity
|
||||
curl -s http://localhost:8000/v1/health
|
||||
curl -s http://localhost:8001/health
|
||||
```
|
||||
|
||||
### SQLite Verification
|
||||
```bash
|
||||
# Check chain-specific databases
|
||||
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
|
||||
sqlite3 /var/lib/aitbc/data/ait-testnet/chain.db "SELECT COUNT(*), MAX(height) FROM block;"
|
||||
|
||||
# Check RPC reflects correct database
|
||||
curl -s http://localhost:8006/rpc/head?chain_id=ait-mainnet | jq '.height'
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
- **Never use PostgreSQL for blockchain chain data**: Use SQLite for chain-specific databases
|
||||
- **Always disable CoW on Btrfs**: SQLite corruption occurs on Btrfs without CoW disablement
|
||||
- **Use systemd drop-ins for service configuration**: Don't modify main service files
|
||||
- **Backup before migration**: Always backup SQLite databases before PostgreSQL migration
|
||||
- **Test connection strings**: Verify database connectivity before restarting services
|
||||
- **Chain-specific databases**: Blockchain node uses chain-specific SQLite databases for multi-chain support
|
||||
|
||||
## Common Operations
|
||||
|
||||
### Check PostgreSQL database size
|
||||
```bash
|
||||
sudo su postgres -c "psql -c \"SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) FROM pg_database;\""
|
||||
```
|
||||
|
||||
### Backup PostgreSQL database
|
||||
```bash
|
||||
sudo su postgres -c "pg_dump aitbc_coordinator > /var/backups/aitbc_coordinator.sql"
|
||||
```
|
||||
|
||||
### Restore PostgreSQL database
|
||||
```bash
|
||||
sudo su postgres -c "psql aitbc_coordinator < /var/backups/aitbc_coordinator.sql"
|
||||
```
|
||||
|
||||
### Check SQLite database size
|
||||
```bash
|
||||
ls -lh /var/lib/aitbc/data/ait-mainnet/chain.db
|
||||
ls -lh /var/lib/aitbc/data/ait-testnet/chain.db
|
||||
```
|
||||
|
||||
### Vacuum SQLite database
|
||||
```bash
|
||||
sqlite3 /var/lib/aitbc/data/ait-mainnet/chain.db "VACUUM;"
|
||||
```
|
||||
|
||||
## Multi-Node Operations
|
||||
|
||||
### Apply PostgreSQL setup to all nodes
|
||||
```bash
|
||||
for node in aitbc1 gitea-runner; do
|
||||
ssh "$node" "sudo apt install postgresql postgresql-contrib -y"
|
||||
ssh "$node" "sudo systemctl start postgresql"
|
||||
ssh "$node" "sudo systemctl enable postgresql"
|
||||
done
|
||||
```
|
||||
|
||||
### Check database status across all nodes
|
||||
```bash
|
||||
for node in aitbc1 gitea-runner; do
|
||||
echo "=== $node ==="
|
||||
ssh "$node" "sudo systemctl status postgresql --no-pager"
|
||||
ssh "$node" "sudo su postgres -c 'psql -l'"
|
||||
done
|
||||
```
|
||||
|
||||
### Disable CoW on all nodes
|
||||
```bash
|
||||
for node in aitbc1 gitea-runner; do
|
||||
ssh "$node" "sudo chattr +C /var/lib/aitbc"
|
||||
ssh "$node" "lsattr -d /var/lib/aitbc"
|
||||
done
|
||||
```
|
||||
Reference in New Issue
Block a user