Watchdog dzialal poprawnie - vjav i superporn wypisywal co 6h przez caly czas ich
ciszy. Zawiodlo POWIADOMIENIE, w trzech miejscach naraz.
1. Filmy nie byly pokryte w ogole. Watchdog skanowal tylko origins tube:<sitetag> ze
scen, wiec streamporn.vip stal 23 dni i zaden automat nie mial jak tego zauwazyc.
Doszlo 6 konektorow filmowych (35 -> 41 sprawdzanych zrodel), prog 72h. Zweryfikowane
przez celowe obnizenie progu do 1h: wszystkie 6 matchuje realne liczby pozycji, a nie
zwraca cicho zera - bo to dokladnie ta klasa bledu, ktora naprawiamy.
2. Stabilny fingerprint per origin tworzyl JEDNO issue przy pierwszym wystapieniu i
potem tylko licznik. Sentry powiadamia o nowych i regresjach, nie o kolejnych
wystapieniach otwartego issue. Teraz fingerprint zawiera kubelek wieku (2d+/7d+/30d+),
wiec przekroczenie kazdego progu zaklada nowe issue, ale w obrebie kubelka nadal nie
ma spamu co 6h.
3. Poziom byl zawsze warning, a reguly alertow celuja w error. Od 7 dni ciszy idzie
error. Sprawdzone: vjav i superporn eskaluja teraz do error, pornbusy/youperv/porndish
zostaja na warning.
Plus kanal, ktory czlowiek realnie czyta: dobowy digest na Slacka (osobny job, zeby
nie wysylac tej samej listy 4x dziennie). Domyslnie NO-OP - bez GOON_SLACK_BOT_TOKEN
i GOON_SLACK_CHANNEL nic nie probuje wysylac, kanalu nie zgadujemy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kategorie i studia podszywajace sie pod osoby: "Creampie" (269 scen), "Natural tits"
(670), "reverse cowgirl" (315), "69", "Brazzers", "Blacked", "Twistys". Razem 277
rekordow i 33 784 przypisan do scen.
Zrodlo: _search_base filtruje wyniki tube-searcha warunkiem "slug zawiera >=1 token
z zapytania", a resolve_performer tworzy performera z dowolnej nazwy podanej przez
tube. Zadnej bramki po drodze.
KRYTERIUM JEST WASKIE CELOWO. To uzywane w app/api/scenes.py do de-rankingu (slug
pokrywa sie z tagiem I brak refa tpdb/stashdb) daje 3111 trafien, ale probka pokazala
~60% falszywek dla nazw wielowyrazowych: wpadaja w nie Hatano Yui, Davina Raines,
Michelle Ferrari, Clanddi, SolaZola, Kate Kuray. Zalozenie "prawdziwy performer ma ref
kanoniczny" nie dziala dla JAV, amatorek i debiutantek. Do de-rankingu wystarcza, do
kasowania nie.
Rozstrzyga promiskuitycznosc taga: ile ROZNYCH innych performerow pojawia sie na
scenach z tym samym slugiem. Zmierzone: prawdziwe osoby maksimum 20 (Julia Reaves),
wiekszosc 0-4, Hatano Yui 0. Kategorie: Deepthroat 39541, Creampie 33661, riding
16284. Prog 50 lezy ponad 2x powyzej najwyzszej prawdziwej osoby. Odrzucilem tez
sygnal "slug rowny studiu", bo wciagal marki osobiste (Katekuray, Julia Reaves).
Tresc nie ginie: te slugi dalej istnieja jako tagi (creampie przy 176 tys. scen),
przestaja tylko udawac osoby. 16 968 scen zostalo bez obsady, co dokłada 1,8 procent
do istniejacych 943 tys. scen bez obsady.
Job co 24h zamiast bramki w resolve_performer: nie dotyka goracej sciezki ingestu,
a smieci odrastaja wolno. Ta sama logika co przy blackliscie phashy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audyt duplikatow po phashu miniatur wykazal dwa rozne zjawiska, ktore wygladaly
jak jedno.
1. Zdegenerowane phashe. 73 wartosci wystepowaly przy >=10 scenach kazda, lacznie
przy 4375 scenach; rekordzistka byla dzielona przez 892 sceny o 892 roznych
tytulach i 1886 roznych performerach. To zaslepki i czarne klatki, nie odciski
scen. find_by_phash_within bierze najblizsza wartosc z calej tabeli, wiec taka
zaslepka zawsze wygrywala z prawdziwym duplikatem (dist 0). Do tej pory bronila
nas bramka dur_prox i nic sie nie skleilo, ale to zabezpieczenie drugiej linii.
Blacklista jest tabela, nie jednorazowym DELETE, bo sam DELETE nic nie daje:
zaslepka wraca przy kolejnym ingescie. Trzeba pamietac, ze wartosc jest
bezuzyteczna. Job co 24h dopisuje nowe i czysci odciski. Po czyszczeniu zero
grup >=10, najwieksza pozostala ma 9.
2. Realne duplikaty. 4647 scalonych. Przyczyna byla jedna: 97 procent par ma
perverzije po dokladnie jednej stronie, bo pisze tytuly z prefiksem studia i
performera, a reszta tubow daje goly tytul. Wbrew mojej pierwszej diagnozie NIE
trzeba tu ruszac scoringu tytulu (token_set_ratio i tak radzi sobie z prefiksem,
a sciezka phash idzie przed composite): 80 procent par to dlug sprzed 60+ dni, a
biezacy wyciek to okolo 1 dziennie. Nadmiarowe wiersze: 18452 na 9449.
merge_phash_exact_dupes.py dostal wykluczenie blacklisty. Bez tego byl grozny:
sam klaster 892 scen dawal ~397 tys. par do rozwazenia.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Enrich existing movies (from paradisehill/dooplay, which mostly lack cast)
with metadata from TPDB's /movies API: cast, categories (tags), studio,
director + a canonical TPDB UUID for dedup. Chosen over IAFD after a
source-comparison research pass — IAFD has strong cast/studio but ZERO
categories, while TPDB /movies has ~11 tags/movie, cast, studio, director,
a canonical UUID (+ sparse phash), is already an integrated API (no
scraping/anti-bot), and covers ~75-85% of our western-DVD-feature catalog.
Enrichment only ever augments EXISTING movies and never creates new ones
(TPDB has no playback, so a standalone TPDB movie would be unplayable).
Writes to movie_performers / movie_tags / movie.studio_id, which the movies
API + mobile detail already render, so no schema/API/UI change is needed.
- connectors/tpdb.py: search_movies() + fetch_movie() + _parse_movie()
reusing the existing _parse_studio/_parse_performer/_parse_tag.
- enrich/tpdb_movies.py: match our movie to a TPDB /movies result by
token_sort_ratio on normalized titles (sort, not set, to reject the
short-title-subset trap "Fantasies" -> "Tara's Fetish Fantasies") with a
+/-2yr guard; then attach cast/tags/studio/director. Incoming performers
deduped by external_id to avoid the performer_external_refs PK clash.
- resolve/movie_merge.py: merge_movies() mirror of scene_merge; two of our
movies mapping to the same TPDB UUID are the same film -> merge.
- scheduler: _job_tpdb_movie_enrich every 6h, batch 200, prioritizing
playable movies missing cast/studio.
Verified on a 150-movie batch: 119 enriched, 4 deduped, 26 no-match,
0 errors; matched titles/studios spot-checked correct (Big Butts Drive Me
Nuts 4 -> 33 tags, Seinfeld #2 -> 10 cast/17 tags, German BB Video titles
-> categories+studio).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rates each source on three axes the user asked for:
- freshness: how recently/often new content arrives (newest age + 7d volume)
- richness: metadata coverage (thumbnail/tags/performers/description/studio/duration)
- plays: does it actually play — from real playback telemetry when available,
else a proxy from the resolve mechanism. 0★ = offline (gates the overall stars,
so a fresh+rich source that doesn't play still ranks bottom — the hqfap/4k69 case)
Backend:
- playback_events: fire-and-forget telemetry POST from the app per playback attempt
(origin + success/error + time-to-first-frame), append-only, 30d retention
- source_stats: per-origin computed scores, refreshed by a scheduler job (6h);
/sources joins it and sorts by stars
- models + local migration 0025; new GOON_SCHED_SOURCE_STATS_HOURS setting
Mobile:
- Sites rows show ★ rating; tap the stars for a breakdown (axes + metadata %, plus
whether "plays" is measured or estimated)
- PlayerScreen reports playback success/failure per source (native path only —
symmetric, conservative); origin threaded through Scene/Movie play callsites
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Missing-merge duplicates (same performer + identical normalized title + identical duration-to-the-second) that bulk_dedup misses — tube re-scrapes and cross-tube re-ingests like porn00 pulling a video already present from xnxx (reports 28fe8181/32df33b1). Extracted the proven merge_exact_title_duration logic into app/scheduler/title_duration_dedup.py (script now a thin wrapper), wired a 12h scheduler job (playback-only = what users actually see, GOON_SCHED_TITLE_DEDUP_HOURS). Signal is near-certain (two different videos don't share byte-identical title AND exact duration); no shared performer = not merged (over-match guard). Verified: job registers (jobs=14), backlog currently 0 after the one-shot global merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two observability additions to the worker scheduler (intertwined in the same files): (1) ingest-watchdog now also covers performer-driven search scrapers (ALL_DIRECT_SCRAPERS) with a separate 7d threshold, not just browse tubes at 48h — several search tubes (perverzija, fpoxxx, porndish, ...) had frozen silently for weeks. (2) New Hetzner Cloud bandwidth monitor (app/scheduler/hetzner_monitor.py): polls outgoing_traffic vs included_traffic and fires a Sentry message at info/warning/error % thresholds with a per-level fingerprint. The config fields existed for ages but the monitor was never implemented. No-op until HETZNER_API_TOKEN + HETZNER_SERVER_ID are set in .env (verified: returns {enabled: False}, job registers as 'hetzner-monitor every 6h', jobs=13).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global source monitor can't catch a single stalled tube because every tube scraper shares one Source row (tube-scraper), so an aggregate run still reports success while one origin freezes (freshporno browsing the rotating KVS homepage root, report 14f3a655). New watchdog checks max(created_at) per active browse-scraper origin (tube:<sitetag>); if a tube with history hasn't produced a new scene in > max_age_hours it fires a Sentry message with a stable per-origin fingerprint (age in extras, not the title, so it stays one grouped issue). Runs every 6h, 48h threshold, both env-tunable (GOON_SCHED_INGEST_WATCHDOG_HOURS / GOON_INGEST_WATCHDOG_MAX_AGE_HOURS). Verified: 0 stale at 48h post-fix, detects neporn at a strict 12h threshold.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one-off cleanup merged ~13.5k same-video-different-title dupes, but they regrow as
these sibling tubes re-ingest under new titles. Wire the asset-id+duration merge into
the scheduler (every 12h, GOON_SCHED_THUMB_DEDUP_HOURS, 0=off) so it stays clean.
Shared logic lives in app/scheduler/thumb_dedup.py (run_thumb_asset_dedup); the one-shot
script now imports it. Same tight signature as the cleanup: family hosts only + identical
duration (the bare asset-id number is reused across unrelated CDNs, so cross-host/diff-
duration grouping is excluded). Reports 205b17d9 / 5a2944cb.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CORRECTION: trafficdeposit thumbnail tokens are hour-bucketed and valid only ~1h
(verified 2026-06-10: stored ts=11:00 dead at 12:27, current ts=13:00 loads). Earlier
"~weekly rot" read was wrong. Storing/periodically-refreshing sxyprn thumbnail URLs
is futile — they expire within the hour. Default the refresh job OFF (kept in code).
The dead-marking sweep (Post Not Found → dead_at) it performed was still valid. Live
sxyprn thumbnails need on-demand resolution at serve time (future work).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CORRECTION to earlier "unrecoverable" call: the /post/<id> page is alive (200) and
DOES expose the scene's own fresh-signed poster via og:image / <video poster>
(post-id embedded, current timestamp) — only the STORED thumbnail URL had rotted.
Search/listings don't re-surface old posts (0 overlap), but per-post fetch works.
scripts/refresh_sxyprn_thumbs.py: iterate live sxyprn sources, fetch post page,
extract fresh og:image, UPDATE thumbnail_url (verified: refreshed URLs return 200).
_job_refresh_sxyprn_thumbs: every 12h refresh the 1200 least-recently-updated sources
(cycles the ~19k catalog within the expiry window). Pairs with the scene_resolver
overwrite fix so refreshed thumbnails stick.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_job_bulk_dedup_performers called run_bulk_dedup(strategy="performers") without
the cross_source_only guard whose docstring exists precisely to prevent this OOM.
At current catalog scale the unguarded path materializes N²/2 pairs per prolific
performer into a list → worker hit 6GB RSS and was OOM-killed every 12h (05:00/
17:00), taking down concurrent tpdb/stashdb/movie ingests as killed_by_restart
(0 new movies). Verified in prod: 05:00 run now completes (885k pairs scored, no
OOM) and ingests succeed (stashdb +241, tpdb +175).
Also wrap in _run_with_timeout like tpdb/stashdb (job had no hard-timeout).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
We ingested only ~3% of each browse tube's catalog (porndoe >62k scenes; we had 1959)
because tubes were hit only by performer-search + top-N browse. Pilot (porndoe pages
64-110): 1119 new scenes, 100% playable + 100% tagged, 0% canonical overlap (purely
additive — content not in TPDB/StashDB).
- app/scheduler/deep_crawl.py: round-robin over ALL_BROWSE_SCRAPERS, per-tube page cursor
in app/_state/deepcrawl_state.json (no DB migration), deep-paginate from the cursor,
idempotent (resolver skips known by raw_hash), mark 'exhausted' at catalog end then
reset cursors for an incremental re-sweep.
- _job_deep_crawl: hourly, 60 pages/run (~1860 scenes, ~22 min), wrapped in the 1h
hard-timeout; registered in build_scheduler (jobs=10).
- config: sched_deep_crawl_hours=1, deep_crawl_pages_per_run=60, deepcrawl_state_path.
- scripts/pilot_porndoe_deepcrawl.py: one-off pilot used to validate the approach.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
At the shared 05:00 anchor all heavy jobs fire together; tpdb/stashdb/performer-driven
had no timeout, so a hung connector blocked the whole job and — with max_instances=1 —
blocked every future fire of that job until a worker restart (incident 2026-06-02: 6 runs
hung 8.7h, movie mirrors 47h stale, tube ingest stalled).
- _run_with_timeout wraps tpdb/stashdb/performer-driven in a 30-min hard cap (same
ThreadPoolExecutor pattern movie-ingest already uses): on timeout the job returns and
frees the scheduler slot; the orphaned thread lives until restart.
- _job_reap_stuck: hourly reaper of 'running' >2h rows, registered in the scheduler —
the startup-only reaper missed hangs while the worker stayed up for hours.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- paradisehill.fetch_movies compared release_date coerced to midnight against the
`since` timestamp, so the chronological crawl stopped at the first upload dated
the same calendar day as `since` and silently dropped most new movies (0-2 seen
per run; Movies tab stalled). Compare by DATE with a 1-day grace instead; idempotent
external_records upsert dedups the re-fetched recent window.
- scripts/backfill_paradisehill_movies.py: one-off no-delta deep crawl to recover the
backlog missed during the bug (idempotent, resumable).
- docs: correct stale 'raz dziennie/24h' browse-latest comments to 6h (4x/day), the
actual configured cadence (config.py sched_browse_latest_hours=6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Counts for /tags, /performers, /studios and /favorites were computed live
per-request by aggregating scene_tags / scene_performers with an EXISTS to
playback_sources. As the catalog grew to ~1.7M scenes (6.3M scene_tags) this
ran ~4.3s for /tags?order=popular (x2 incl. the total count) and ~950ms for
the default /scenes count, making those screens load in several seconds.
- migration 0019: add scene_count (+ DESC index) to tags/performers/studios
- background job _job_refresh_taxonomy_counts (every 3h) recomputes the counts
in one UPDATE..FROM each (IS DISTINCT FROM to skip unchanged rows)
- /tags, /performers, /studios scenes path now read the column + ORDER BY the
indexed scene_count; for_movies paths keep live aggregation (small tables)
- favorites read denormalized scene_count instead of a grouped EXISTS aggregate
- /scenes default count: 10-min in-process TTL cache (header is approximate)
Measured: /tags?order=popular&per_page=500 ~8s -> 66ms incl. serialization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bug-report 2026-05-30 "ingest znów się zawiesił". streamporn/pandamovies
wieszały się intermittentnie mid-run (zależnie od live-contentu danego dnia),
blokując sekwencyjny _job_movie_ingest → mangoporn (jedyny mirror z realnym
new-content: 72 nowych 05-28) nigdy nie startował. try/except chronił przed
wyjątkiem, NIE przed hangiem.
Fix:
- _job_movie_ingest: każdy connector w ThreadPoolExecutor z future.result
(timeout=360s). Hang jednego źródła → log + shutdown(wait=False) + kolejka
leci dalej. Healthy run ~50s, cap 6min = zapas.
- get_movie_connectors: reorder paradisehill, MANGOPORN, streamporn, pandamovies
— mangoporn zaraz po canonical primary, przed wolniejszymi/wieszającymi się.
Zweryfikowane: pełny _job_movie_ingest przeszedł wszystkie 4 success w nowej
kolejności (mangoporn 2nd, 23s). 33 osierocone "running" rows (worker ubity
mid-run przy deployach) wyczyszczone osobno.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Goon — self-hosted aggregator for adult-content scene metadata.
Indexes scenes from TPDB, StashDB, and 30+ public adult tube sites.
Cross-source deduplication via perceptual hash + Levenshtein distance.
FastAPI backend + APScheduler worker + React Native (Expo) mobile client.
FOSS, ad-free, donation-funded. See README for details.
