516 lines
12 KiB
Markdown
516 lines
12 KiB
Markdown
---
|
|
description: Comprehensive code quality workflow with pre-commit hooks, formatting, linting, type checking, and security scanning
|
|
---
|
|
|
|
# Code Quality Workflow
|
|
|
|
## 🎯 **Overview**
|
|
Comprehensive code quality assurance workflow that ensures high standards across the AITBC codebase through automated pre-commit hooks, formatting, linting, type checking, and security scanning.
|
|
|
|
---
|
|
|
|
## 📋 **Workflow Steps**
|
|
|
|
### **Step 1: Setup Pre-commit Environment**
|
|
```bash
|
|
# Install pre-commit hooks
|
|
./venv/bin/pre-commit install
|
|
|
|
# Verify installation
|
|
./venv/bin/pre-commit --version
|
|
```
|
|
|
|
### **Step 2: Run All Quality Checks**
|
|
```bash
|
|
# Run all hooks on all files
|
|
./venv/bin/pre-commit run --all-files
|
|
|
|
# Run on staged files (git commit)
|
|
./venv/bin/pre-commit run
|
|
```
|
|
|
|
### **Step 3: Individual Quality Categories**
|
|
|
|
#### **🧹 Code Formatting**
|
|
```bash
|
|
# Black code formatting
|
|
./venv/bin/black --line-length=127 --check .
|
|
|
|
# Auto-fix formatting issues
|
|
./venv/bin/black --line-length=127 .
|
|
|
|
# Import sorting with isort
|
|
./venv/bin/isort --profile=black --line-length=127 .
|
|
```
|
|
|
|
#### **🔍 Linting & Code Analysis**
|
|
```bash
|
|
# Flake8 linting
|
|
./venv/bin/flake8 --max-line-length=127 --extend-ignore=E203,W503 .
|
|
|
|
# Pydocstyle documentation checking
|
|
./venv/bin/pydocstyle --convention=google .
|
|
|
|
# Python version upgrade checking
|
|
./venv/bin/pyupgrade --py311-plus .
|
|
```
|
|
|
|
#### **🔍 Type Checking**
|
|
```bash
|
|
# Core domain models type checking
|
|
./venv/bin/mypy --ignore-missing-imports --show-error-codes apps/coordinator-api/src/app/domain/job.py apps/coordinator-api/src/app/domain/miner.py apps/coordinator-api/src/app/domain/agent_portfolio.py
|
|
|
|
# Type checking coverage analysis
|
|
./scripts/type-checking/check-coverage.sh
|
|
|
|
# Full mypy checking
|
|
./venv/bin/mypy --ignore-missing-imports apps/coordinator-api/src/app/
|
|
```
|
|
|
|
#### **🛡️ Security Scanning**
|
|
```bash
|
|
# Bandit security scanning
|
|
./venv/bin/bandit -r . -f json -o bandit-report.json
|
|
|
|
# Safety dependency vulnerability check
|
|
./venv/bin/safety check --json --output safety-report.json
|
|
|
|
# Safety dependency check for requirements files
|
|
./venv/bin/safety check requirements.txt
|
|
```
|
|
|
|
#### **🧪 Testing**
|
|
```bash
|
|
# Unit tests
|
|
pytest tests/unit/ --tb=short -q
|
|
|
|
# Security tests
|
|
pytest tests/security/ --tb=short -q
|
|
|
|
# Performance tests
|
|
pytest tests/performance/test_performance_lightweight.py::TestPerformance::test_cli_performance --tb=short -q
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 **Pre-commit Configuration**
|
|
|
|
### **Repository Structure**
|
|
```yaml
|
|
repos:
|
|
# Basic file checks
|
|
- repo: https://github.com/pre-commit/pre-commit-hooks
|
|
rev: v5.0.0
|
|
hooks:
|
|
- id: trailing-whitespace
|
|
- id: end-of-file-fixer
|
|
- id: check-yaml
|
|
- id: check-added-large-files
|
|
- id: check-json
|
|
- id: check-merge-conflict
|
|
- id: debug-statements
|
|
- id: check-docstring-first
|
|
- id: check-executables-have-shebangs
|
|
- id: check-toml
|
|
- id: check-xml
|
|
- id: check-case-conflict
|
|
- id: check-ast
|
|
|
|
# Code formatting
|
|
- repo: https://github.com/psf/black
|
|
rev: 26.3.1
|
|
hooks:
|
|
- id: black
|
|
language_version: python3
|
|
args: [--line-length=127]
|
|
|
|
# Import sorting
|
|
- repo: https://github.com/pycqa/isort
|
|
rev: 8.0.1
|
|
hooks:
|
|
- id: isort
|
|
args: [--profile=black, --line-length=127]
|
|
|
|
# Linting
|
|
- repo: https://github.com/pycqa/flake8
|
|
rev: 7.3.0
|
|
hooks:
|
|
- id: flake8
|
|
args: [--max-line-length=127, --extend-ignore=E203,W503]
|
|
|
|
# Type checking
|
|
- repo: https://github.com/pre-commit/mirrors-mypy
|
|
rev: v1.19.1
|
|
hooks:
|
|
- id: mypy
|
|
additional_dependencies: [types-requests, types-python-dateutil]
|
|
args: [--ignore-missing-imports]
|
|
|
|
# Security scanning
|
|
- repo: https://github.com/PyCQA/bandit
|
|
rev: 1.9.4
|
|
hooks:
|
|
- id: bandit
|
|
args: [-r, ., -f, json, -o, bandit-report.json]
|
|
pass_filenames: false
|
|
|
|
# Documentation checking
|
|
- repo: https://github.com/pycqa/pydocstyle
|
|
rev: 6.3.0
|
|
hooks:
|
|
- id: pydocstyle
|
|
args: [--convention=google]
|
|
|
|
# Python version upgrade
|
|
- repo: https://github.com/asottile/pyupgrade
|
|
rev: v3.21.2
|
|
hooks:
|
|
- id: pyupgrade
|
|
args: [--py311-plus]
|
|
|
|
# Dependency security
|
|
- repo: https://github.com/Lucas-C/pre-commit-hooks-safety
|
|
rev: v1.4.2
|
|
hooks:
|
|
- id: python-safety-dependencies-check
|
|
files: requirements.*\.txt$
|
|
|
|
- repo: https://github.com/Lucas-C/pre-commit-hooks-safety
|
|
rev: v1.3.2
|
|
hooks:
|
|
- id: python-safety-check
|
|
args: [--json, --output, safety-report.json]
|
|
|
|
# Local hooks
|
|
- repo: local
|
|
hooks:
|
|
- id: pytest-check
|
|
name: pytest-check
|
|
entry: pytest
|
|
language: system
|
|
args: [tests/unit/, --tb=short, -q]
|
|
pass_filenames: false
|
|
always_run: true
|
|
|
|
- id: security-check
|
|
name: security-check
|
|
entry: pytest
|
|
language: system
|
|
args: [tests/security/, --tb=short, -q]
|
|
pass_filenames: false
|
|
always_run: true
|
|
|
|
- id: performance-check
|
|
name: performance-check
|
|
entry: pytest
|
|
language: system
|
|
args: [tests/performance/test_performance_lightweight.py::TestPerformance::test_cli_performance, --tb=short, -q]
|
|
pass_filenames: false
|
|
always_run: true
|
|
|
|
- id: mypy-domain-core
|
|
name: mypy-domain-core
|
|
entry: ./venv/bin/mypy
|
|
language: system
|
|
args: [--ignore-missing-imports, --show-error-codes]
|
|
files: ^apps/coordinator-api/src/app/domain/(job|miner|agent_portfolio)\.py$
|
|
pass_filenames: false
|
|
|
|
- id: type-check-coverage
|
|
name: type-check-coverage
|
|
entry: ./scripts/type-checking/check-coverage.sh
|
|
language: script
|
|
files: ^apps/coordinator-api/src/app/
|
|
pass_filenames: false
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 **Quality Metrics & Reporting**
|
|
|
|
### **Coverage Reports**
|
|
```bash
|
|
# Type checking coverage
|
|
./scripts/type-checking/check-coverage.sh
|
|
|
|
# Security scan reports
|
|
cat bandit-report.json | jq '.results | length'
|
|
cat safety-report.json | jq '.vulnerabilities | length'
|
|
|
|
# Test coverage
|
|
pytest --cov=apps --cov-report=html tests/
|
|
```
|
|
|
|
### **Quality Score Calculation**
|
|
```python
|
|
# Quality score components:
|
|
# - Code formatting: 20%
|
|
# - Linting compliance: 20%
|
|
# - Type coverage: 25%
|
|
# - Test coverage: 20%
|
|
# - Security compliance: 15%
|
|
|
|
# Overall quality score >= 80% required
|
|
```
|
|
|
|
### **Automated Reporting**
|
|
```bash
|
|
# Generate comprehensive quality report
|
|
./scripts/quality/generate-quality-report.sh
|
|
|
|
# Quality dashboard metrics
|
|
curl http://localhost:8000/metrics/quality
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 **Integration with Development Workflow**
|
|
|
|
### **Before Commit**
|
|
```bash
|
|
# 1. Stage your changes
|
|
git add .
|
|
|
|
# 2. Pre-commit hooks run automatically
|
|
git commit -m "Your commit message"
|
|
|
|
# 3. If any hook fails, fix the issues and try again
|
|
```
|
|
|
|
### **Manual Quality Checks**
|
|
```bash
|
|
# Run all quality checks manually
|
|
./venv/bin/pre-commit run --all-files
|
|
|
|
# Check specific category
|
|
./venv/bin/black --check .
|
|
./venv/bin/flake8 .
|
|
./venv/bin/mypy apps/coordinator-api/src/app/
|
|
```
|
|
|
|
### **CI/CD Integration**
|
|
```yaml
|
|
# GitHub Actions workflow
|
|
name: Code Quality
|
|
on: [push, pull_request]
|
|
jobs:
|
|
quality:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Setup Python
|
|
uses: actions/setup-python@v4
|
|
with:
|
|
python-version: '3.13'
|
|
- name: Install dependencies
|
|
run: pip install -r requirements.txt
|
|
- name: Run pre-commit
|
|
run: ./venv/bin/pre-commit run --all-files
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 **Quality Standards**
|
|
|
|
### **Code Formatting Standards**
|
|
- **Black**: Line length 127 characters
|
|
- **isort**: Black profile compatibility
|
|
- **Python 3.13+**: Modern Python syntax
|
|
|
|
### **Linting Standards**
|
|
- **Flake8**: Line length 127, ignore E203, W503
|
|
- **Pydocstyle**: Google convention
|
|
- **No debug statements**: Production code only
|
|
|
|
### **Type Safety Standards**
|
|
- **MyPy**: Strict mode for new code
|
|
- **Coverage**: 90% minimum for core domain
|
|
- **Error handling**: Proper exception types
|
|
|
|
### **Security Standards**
|
|
- **Bandit**: Zero high-severity issues
|
|
- **Safety**: No known vulnerabilities
|
|
- **Dependencies**: Regular security updates
|
|
|
|
### **Testing Standards**
|
|
- **Coverage**: 80% minimum test coverage
|
|
- **Unit tests**: All business logic tested
|
|
- **Security tests**: Authentication and authorization
|
|
- **Performance tests**: Critical paths validated
|
|
|
|
---
|
|
|
|
## 📈 **Quality Improvement Workflow**
|
|
|
|
### **1. Initial Setup**
|
|
```bash
|
|
# Install pre-commit hooks
|
|
./venv/bin/pre-commit install
|
|
|
|
# Run initial quality check
|
|
./venv/bin/pre-commit run --all-files
|
|
|
|
# Fix any issues found
|
|
./venv/bin/black .
|
|
./venv/bin/isort .
|
|
# Fix other issues manually
|
|
```
|
|
|
|
### **2. Daily Development**
|
|
```bash
|
|
# Make changes
|
|
vim your_file.py
|
|
|
|
# Stage and commit (pre-commit runs automatically)
|
|
git add your_file.py
|
|
git commit -m "Add new feature"
|
|
|
|
# If pre-commit fails, fix issues and retry
|
|
git commit -m "Add new feature"
|
|
```
|
|
|
|
### **3. Quality Monitoring**
|
|
```bash
|
|
# Check quality metrics
|
|
./scripts/quality/check-quality-metrics.sh
|
|
|
|
# Generate quality report
|
|
./scripts/quality/generate-quality-report.sh
|
|
|
|
# Review quality trends
|
|
./scripts/quality/quality-trends.sh
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 **Troubleshooting**
|
|
|
|
### **Common Issues**
|
|
|
|
#### **Black Formatting Issues**
|
|
```bash
|
|
# Check formatting issues
|
|
./venv/bin/black --check .
|
|
|
|
# Auto-fix formatting
|
|
./venv/bin/black .
|
|
|
|
# Specific file
|
|
./venv/bin/black --check path/to/file.py
|
|
```
|
|
|
|
#### **Import Sorting Issues**
|
|
```bash
|
|
# Check import sorting
|
|
./venv/bin/isort --check-only .
|
|
|
|
# Auto-fix imports
|
|
./venv/bin/isort .
|
|
|
|
# Specific file
|
|
./venv/bin/isort path/to/file.py
|
|
```
|
|
|
|
#### **Type Checking Issues**
|
|
```bash
|
|
# Check type errors
|
|
./venv/bin/mypy apps/coordinator-api/src/app/
|
|
|
|
# Ignore specific errors
|
|
./venv/bin/mypy --ignore-missing-imports apps/coordinator-api/src/app/
|
|
|
|
# Show error codes
|
|
./venv/bin/mypy --show-error-codes apps/coordinator-api/src/app/
|
|
```
|
|
|
|
#### **Security Issues**
|
|
```bash
|
|
# Check security issues
|
|
./venv/bin/bandit -r .
|
|
|
|
# Generate security report
|
|
./venv/bin/bandit -r . -f json -o security-report.json
|
|
|
|
# Check dependencies
|
|
./venv/bin/safety check
|
|
```
|
|
|
|
### **Performance Optimization**
|
|
|
|
#### **Pre-commit Performance**
|
|
```bash
|
|
# Run hooks in parallel
|
|
./venv/bin/pre-commit run --all-files --parallel
|
|
|
|
# Skip slow hooks during development
|
|
./venv/bin/pre-commit run --all-files --hook-stage manual
|
|
|
|
# Cache dependencies
|
|
./venv/bin/pre-commit run --all-files --cache
|
|
```
|
|
|
|
#### **Selective Hook Running**
|
|
```bash
|
|
# Run specific hooks
|
|
./venv/bin/pre-commit run black flake8 mypy
|
|
|
|
# Run on specific files
|
|
./venv/bin/pre-commit run --files apps/coordinator-api/src/app/
|
|
|
|
# Skip hooks
|
|
./venv/bin/pre-commit run --all-files --skip mypy
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 **Quality Checklist**
|
|
|
|
### **Before Commit**
|
|
- [ ] Code formatted with Black
|
|
- [ ] Imports sorted with isort
|
|
- [ ] Linting passes with Flake8
|
|
- [ ] Type checking passes with MyPy
|
|
- [ ] Documentation follows Pydocstyle
|
|
- [ ] No security vulnerabilities
|
|
- [ ] All tests pass
|
|
- [ ] Performance tests pass
|
|
|
|
### **Before Merge**
|
|
- [ ] Code review completed
|
|
- [ ] Quality score >= 80%
|
|
- [ ] Test coverage >= 80%
|
|
- [ ] Type coverage >= 90% (core domain)
|
|
- [ ] Security scan clean
|
|
- [ ] Documentation updated
|
|
- [ ] Performance benchmarks met
|
|
|
|
### **Before Release**
|
|
- [ ] Full quality suite passes
|
|
- [ ] Integration tests pass
|
|
- [ ] Security audit complete
|
|
- [ ] Performance validation
|
|
- [ ] Documentation complete
|
|
- [ ] Release notes prepared
|
|
|
|
---
|
|
|
|
## 🎉 **Benefits**
|
|
|
|
### **Immediate Benefits**
|
|
- **Consistent Code**: Uniform formatting and style
|
|
- **Bug Prevention**: Type checking and linting catch issues early
|
|
- **Security**: Automated vulnerability scanning
|
|
- **Quality Assurance**: Comprehensive test coverage
|
|
|
|
### **Long-term Benefits**
|
|
- **Maintainability**: Clean, well-documented code
|
|
- **Developer Experience**: Automated quality gates
|
|
- **Team Consistency**: Shared quality standards
|
|
- **Production Readiness**: Enterprise-grade code quality
|
|
|
|
---
|
|
|
|
**Last Updated**: March 31, 2026
|
|
**Workflow Version**: 1.0
|
|
**Next Review**: April 30, 2026
|