feat: implement automatic bulk sync for blockchain nodes
Some checks failed
Integration Tests / test-service-integration (push) Failing after 8s
Multi-Node Blockchain Health Monitoring / health-check (push) Successful in 2s
P2P Network Verification / p2p-verification (push) Successful in 2s
Python Tests / test-python (push) Successful in 17s
Security Scanning / security-scan (push) Failing after 37s
Blockchain Synchronization Verification / sync-verification (push) Failing after 2s
Some checks failed
Integration Tests / test-service-integration (push) Failing after 8s
Multi-Node Blockchain Health Monitoring / health-check (push) Successful in 2s
P2P Network Verification / p2p-verification (push) Successful in 2s
Python Tests / test-python (push) Successful in 17s
Security Scanning / security-scan (push) Failing after 37s
Blockchain Synchronization Verification / sync-verification (push) Failing after 2s
- Add auto_sync_enabled, auto_sync_threshold, auto_sync_max_retries config - Add min_bulk_sync_interval for rate limiting - Implement gap detection in process_blocks() with automatic bulk sync trigger - Add rate limiting to ChainSync.bulk_import_from() to prevent sync loops - Automatic bulk sync triggers when gap > threshold (default 10 blocks) - Retries block import after bulk sync completes - Logs sync events for monitoring and debugging
This commit is contained in:
@@ -111,6 +111,8 @@ class ChainSync:
|
||||
self._batch_size = batch_size
|
||||
self._poll_interval = poll_interval
|
||||
self._client = httpx.AsyncClient(timeout=10.0)
|
||||
self._last_bulk_sync_time = 0
|
||||
self._min_bulk_sync_interval = getattr(settings, 'min_bulk_sync_interval', 60)
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close HTTP client."""
|
||||
@@ -138,6 +140,13 @@ class ChainSync:
|
||||
if import_url is None:
|
||||
import_url = "http://127.0.0.1:8006" # default local RPC
|
||||
|
||||
# Rate limiting check
|
||||
current_time = time.time()
|
||||
time_since_last_sync = current_time - self._last_bulk_sync_time
|
||||
if time_since_last_sync < self._min_bulk_sync_interval:
|
||||
logger.warning("Bulk sync rate limited", extra={"time_since_last_sync": time_since_last_sync, "min_interval": self._min_bulk_sync_interval})
|
||||
return 0
|
||||
|
||||
# Get local head
|
||||
with self._session_factory() as session:
|
||||
local_head = session.exec(
|
||||
@@ -185,6 +194,10 @@ class ChainSync:
|
||||
await asyncio.sleep(self._poll_interval)
|
||||
|
||||
logger.info("Bulk import completed", extra={"imported": imported, "final_height": remote_height})
|
||||
|
||||
# Update last bulk sync time
|
||||
self._last_bulk_sync_time = current_time
|
||||
|
||||
return imported
|
||||
|
||||
def import_block(self, block_data: Dict[str, Any], transactions: Optional[List[Dict[str, Any]]] = None) -> ImportResult:
|
||||
|
||||
Reference in New Issue
Block a user