b3ddb67ced
run_deep_crawl picks one tube/run and crawls 60 pages under _job_deep_crawl's 3600s hard timeout. A detail-fetch scraper on a slow patch (per-scene page fetch, e.g. via proxy) could exceed it → the run is killed mid-page, the cursor is never saved (orphan thread), and that tube makes zero progress — recurring Sentry GOON-V. Added a 3000s in-run budget that breaks after a completed page, saves the cursor, and returns cleanly; the next run continues. budget_hit surfaced in the summary log to spot the slow tube. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| browse_latest.py | ||
| bulk_dedup.py | ||
| deep_crawl.py | ||
| hetzner_monitor.py | ||
| ingest_watchdog.py | ||
| jobs.py | ||
| performer_driven.py | ||
| source_stats.py | ||
| taxonomy_counts.py | ||
| thumb_dedup.py | ||
| title_duration_dedup.py | ||
| worker.py | ||