b3ddb67ced
run_deep_crawl picks one tube/run and crawls 60 pages under _job_deep_crawl's 3600s hard timeout. A detail-fetch scraper on a slow patch (per-scene page fetch, e.g. via proxy) could exceed it → the run is killed mid-page, the cursor is never saved (orphan thread), and that tube makes zero progress — recurring Sentry GOON-V. Added a 3000s in-run budget that breaks after a completed page, saves the cursor, and returns cleanly; the next run continues. budget_hit surfaced in the summary log to spot the slow tube. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| api | ||
| connectors | ||
| extractors | ||
| models | ||
| normalize | ||
| resolve | ||
| scheduler | ||
| templates | ||
| __init__.py | ||
| auth.py | ||
| config.py | ||
| db.py | ||
| ingest.py | ||
| main.py | ||