- Add sqlalchemy to --extra-packages in integration-tests.yml
- Fixes ModuleNotFoundError in test_staking_lifecycle.py
- Test imports sqlalchemy at line 18 but package was missing from cached environment
- Replace bc with integer comparison for wallet balance check
- Replace bc with integer calculation for TPS monitoring
- Replace bc with integer calculation for error rate
- Fixes 'bc: command not found' error in stress test workflow
- Remove JSON parsing from get_wallet_address function
- Remove JSON parsing from get_wallet_balance function
- Use direct CLI output instead of --output json parsing
- Fixes wallet address retrieval failure in cross-node transaction test
- Add cross-node transaction testing workflow (manual dispatch)
- Add node failover simulation workflow (manual dispatch, check logic only)
- Add multi-node stress testing workflow (manual dispatch)
- All workflows use only RPC endpoints (no SSH access)
- All workflows run on manual dispatch only
- No remediation steps (monitoring/testing only)
- Cross-node transaction testing uses real transactions from test wallet
- Failover simulation uses check logic only (no actual shutdown)
- Stress testing generates real transactions with configurable count/rate
- Comprehensive logging to /var/log/aitbc/
- Proper wallet creation and cleanup
- Fix regex to extract actual chain ID value from supported_chains array
- Previously extracted 'supported_chains' field name instead of value
- Now correctly extracts chain ID from ["ait-devnet"] format
- Remove SSH-based P2P peer checks and connectivity tests
- Remove SSH-based P2P log checks and remediation
- Remove SSH-based force sync remediation from sync verification
- P2P verification now only checks Redis gossip backend
- Sync verification skips remediation (requires SSH for chain.db copy)
- All scripts now use only RPC endpoints, no SSH access needed
- Remove SSH-based service status checks (use RPC health instead)
- Remove SSH-based resource usage checks
- Remove SSH-based remediation functions
- Remove ssh_exec function entirely
- Script now uses only RPC endpoints for health checks
- gitea-runner no longer needs SSH access to other nodes
- Compare node IP against local IP instead of hostname
- Fixes SSH permission issues when running on gitea-runner
- gitea-runner (10.1.223.98) should execute commands directly for aitbc2
- Detect when node is localhost or current hostname
- Execute commands directly on local node instead of using SSH
- Fixes SSH permission denied errors when checking services on local node
- gitea-runner runs the workflow, so it should check aitbc services directly
- Use /rpc/head endpoint for block height instead of /blockchain/height
- Use /health endpoint for chain ID instead of /blockchain/chain-id
- Use /rpc/blocks/{height} endpoint for block hash instead of /blockchain/block/{height}/hash
- Fixes workflow failure due to incorrect RPC endpoint paths
- Create multi-node blockchain health monitoring workflow
- Create P2P network verification workflow for all 3 nodes
- Create blockchain synchronization verification workflow
- Update blockchain-communication-test.sh to include aitbc2 (gitea-runner)
- Add shared scripts directory with health check, P2P verification, and sync verification scripts
- All workflows trigger on git push to main/develop branches
- Workflows run on gitea-runner (has SSH access to all nodes)
- Include automatic remediation for failed services and sync issues
- Sync threshold set to 10 blocks
- Logging to /var/log/aitbc/ and alerts in Gitea UI
- Documented Gitea remote setup with HTTP token authentication
- Added credential storage and authentication method documentation
- Used placeholder variables instead of actual tokens
- Added security notes about token management
- Never commit actual tokens to version control
- Updated Gitea domain from gitea.keisanki.net to gitea.bubuit.net
- Fixed GitHub remote to remove exposed token from URL
- Updated Gitea remote to use SSH instead of HTTP with token
- Updated all web interface URLs to use correct domain
- Clarified authentication methods for both remotes
- Added ripgrep (rg) usage notes and preference over grep for targeted searches
- Updated log discovery examples to use rg with --fixed-strings for workflow/job name searches
- Added failure-marker search pattern (❌|Traceback|FAILED|etc.) for quick issue identification
- Replaced grep with rg in runner health checks (dmesg, journalctl)
- Added failure marker search to quick-start one-liner
- Preserved awk usage for tab
Added setup-job-logging.sh initialization step to api-endpoint-tests.yml, cli-level1-tests.yml, docs-validation.yml (both jobs), integration-tests.yml, js-sdk-tests.yml, package-tests.yml (both jobs), python-tests.yml, rust-zk-tests.yml, security-scanning.yml, and smart-contract-tests.yml (both jobs) to ensure consistent logging across all CI workflows.
- Added python-dotenv to --extra-packages in production-tests.yml venv setup
- Added PYTHONUNBUFFERED=1 to agent coordinator uvicorn startup for immediate log output
- Added startup validation check after 2-second sleep to detect early coordinator exits
- Display coordinator logs and fail job if process exits during startup
Added comprehensive documentation for autonomous investigation of failed Gitea Actions runs via SSH access to gitea-runner host. Includes log location mapping, classification heuristics for distinguishing workflow/dependency/application/service/infrastructure failures, and evidence-based debug suggestion templates. Provides read-only investigation sequences with safety constraints to prevent conflating application failures with runner inst
- Added setup-job-logging.sh initialization step to production-tests.yml, all staking-tests.yml jobs, and systemd-sync.yml
- Removed redundant agent-coordinator package installation from production-tests.yml (already included in extra-packages)
- Fixed systemd-sync.yml to use relative paths (scripts/utils/link-systemd.sh) instead of absolute paths (/opt/aitbc/scripts/utils/link-systemd.sh)
Replaced eager imports with lazy loading using __getattr__ to defer service module imports until first access. Added module mapping dictionary and dynamic import logic to reduce initial import overhead while maintaining the same public API.
Added click package to --extra-packages in both staking service tests and integration tests jobs to ensure CLI functionality is available during test execution.
- Changed all Python/pip commands to use `venv/bin/python -m` pattern instead of direct tool invocation or source activation
- package-tests.yml: pip, mypy, black, pytest now use `venv/bin/python -m`
- python-tests.yml: ruff, pip, pytest now use `venv/bin/python -m` or `venv/bin/pytest`
- staking-tests.yml: pytest now uses `venv/bin/pytest` instead of `python3 -m pytest`
- Added missing dependencies to workflow
Reverted to standard requirements.txt installation instead of explicit test package dependencies, aligning with the default behavior of setup-python-venv.sh script.
Added `rm -rf venv` (and `venv-build` where applicable) before venv setup in all CI workflows to ensure clean installations and prevent cache corruption issues. Removed redundant venv corruption detection and rebuild logic from package-tests.yml since explicit cleanup makes it unnecessary.
Added --no-cache-dir to pip install commands for pip/setuptools/wheel, requirements file, and extra packages to reduce disk usage and ensure clean installations in CI environments.
Added validation checks for venv and venv-build directories with automatic rebuild on corruption detection. Checks for executable pip/python binaries and rebuilds the venv if not found.
Changed from `source venv/bin/activate` to direct venv/bin/pip-audit and venv/bin/bandit calls to match the pattern used in other workflows for more reliable venv usage.
- Added explicit service start/stop steps in api-endpoint-tests.yml
- Start coordinator-api, exchange-api, wallet, and blockchain-rpc services before tests
- Stop all started services in cleanup step
- Fixed venv activation in package-tests.yml
- Changed from `source venv/bin/activate` to direct venv/bin/pip and venv/bin/python calls
- Applied same pattern to venv-build for package building
- Fixed venv activation in production-tests.
Replaced manual venv creation and pip install commands with calls to scripts/ci/setup-python-venv.sh across all CI workflows. The script provides consistent venv setup with configurable options for requirements installation, copy mode, and extra packages.
- Changed from manual `python3 -m venv` + `pip install` to setup-python-venv.sh in all workflows
- Added --skip-requirements flag where workflows don't need requirements
- Changed from hardcoded /opt/aitbc paths to Path(__file__).resolve().parents[2] for dynamic repository root resolution
- Reorganized imports to follow standard order (stdlib, third-party, local)
- Removed duplicate sys import in test_staking_service.py
- Extracted repo_root variable in test_staking_lifecycle.py for cleaner path construction
Changed from `git show` to `git diff HEAD^ HEAD` for more reliable detection of changed files in push/PR events. Also increased clone depth from 1 to 2 and added explicit fetch/checkout of the target ref to ensure HEAD^ is available for comparison.
Added conditional logic to security-scanning.yml to scan only changed files
during push/PR events while maintaining full scans for scheduled and manual runs.
- Bandit now scans only modified Python files on push/PR using git diff
- Secret scanning now checks only changed files on push/PR
- Both tools still perform full repository scans on schedule/workflow_dispatch
- Added early exit when no relevant files changed to avoid unnecessary processing
Changed test from 'blockchain info' to 'blockchain block --number 1' and added
verification for "Block #1" in output. Also improved error output to include
stdout when stderr is empty.
- Changed test paths from apps/coordinator-api and apps/blockchain-node to tests/archived_phase_tests and tests/cross_phase
- Removed --ignore flags for tests/production and test_confidential*.py files
- Added asyncio_mode = auto and asyncio_default_fixture_loop_scope = function to pytest.ini for better async test handling
Changed uvicorn application path from src.app.main:app to app.main:app in both
startup and cleanup steps to match the actual module structure where src is
already in PYTHONPATH.
- Added Redis server installation step in production-tests.yml workflow
- Checks if Redis binaries are already available before installing
- Installs redis-server package if needed
- Improved aitbc_crypto mocking in conftest.py
- Try importing real aitbc_crypto module first before mocking
- Only mock functions if they don't already exist
- Prevents overriding real implementations when aitbc_crypto is available
- Simplified npm install commands in CI workflows by removing fallback logic
- Added aitbc-crypto local dependency installation for aitbc-sdk in package-tests.yml
- Removed aitbc-token specific Hardhat dependency workarounds from package-tests.yml
- Fixed bare except clause in agent_daemon.py to catch specific json.JSONDecodeError
- Moved aitbc-crypto from poetry.dependencies to standard dependencies in aitbc-sdk
- Fixed MyPy type errors in receip
