Na zyczenie uzytkownika. Alerty zostaja wylacznie w Sentry - z eskalacja poziomu i
kubelkami wieku w fingerprincie z poprzedniego commita, wiec sama poprawa
powiadamiania zostaje nienaruszona.
Usuniete: app/notify/ (caly pakiet), job watchdog-digest, parametr to_slack w
run_ingest_freshness_watchdog oraz klucze konfiguracji sched_watchdog_digest_hours,
slack_bot_token i slack_channel. Zadna wartosc SLACK nigdy nie trafila do .env ani do
compose, wiec nic nie bylo wysylane.
Po usunieciu watchdog dalej sprawdza 41 zrodel (sceny + filmy), scheduler rejestruje
juz tylko ingest-watchdog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Watchdog dzialal poprawnie - vjav i superporn wypisywal co 6h przez caly czas ich
ciszy. Zawiodlo POWIADOMIENIE, w trzech miejscach naraz.
1. Filmy nie byly pokryte w ogole. Watchdog skanowal tylko origins tube:<sitetag> ze
scen, wiec streamporn.vip stal 23 dni i zaden automat nie mial jak tego zauwazyc.
Doszlo 6 konektorow filmowych (35 -> 41 sprawdzanych zrodel), prog 72h. Zweryfikowane
przez celowe obnizenie progu do 1h: wszystkie 6 matchuje realne liczby pozycji, a nie
zwraca cicho zera - bo to dokladnie ta klasa bledu, ktora naprawiamy.
2. Stabilny fingerprint per origin tworzyl JEDNO issue przy pierwszym wystapieniu i
potem tylko licznik. Sentry powiadamia o nowych i regresjach, nie o kolejnych
wystapieniach otwartego issue. Teraz fingerprint zawiera kubelek wieku (2d+/7d+/30d+),
wiec przekroczenie kazdego progu zaklada nowe issue, ale w obrebie kubelka nadal nie
ma spamu co 6h.
3. Poziom byl zawsze warning, a reguly alertow celuja w error. Od 7 dni ciszy idzie
error. Sprawdzone: vjav i superporn eskaluja teraz do error, pornbusy/youperv/porndish
zostaja na warning.
Plus kanal, ktory czlowiek realnie czyta: dobowy digest na Slacka (osobny job, zeby
nie wysylac tej samej listy 4x dziennie). Domyslnie NO-OP - bez GOON_SLACK_BOT_TOKEN
i GOON_SLACK_CHANNEL nic nie probuje wysylac, kanalu nie zgadujemy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ranking z poprzedniego commita ujawnil klase, ktorej zaden wczesniejszy sygnal nie
widzial: wpisy bedace SAMYM NAZWISKIEM albo samym imieniem, powstale z rozbicia peleno
nazwy przez tube-search. "Devine" obok "Shalina Devine", "Kitana" obok "Kitana A",
"Rayne" obok "Tiffany Rayne". Taki wpis lapie potem kazda scene z tym slowem,
nalezaca do zupelnie innej osoby.
Regula jest DOWODLIWIE BEZSTRATNA: kasujemy przypisanie jednotokenowe tylko wtedy, gdy
ta sama scena ma juz przypisana osobe pelnoimienna zawierajaca to slowo. Informacja o
obsadzie zostaje, znika tylko niejednoznaczny duplikat. Kontrola potwierdzila, ze zadna
scena nie zostala przez to bez obsady (0).
Prowenancja z migracji 0026 dala zabezpieczenie: z 62 783 zbednych przypisan 62 120 to
scraper (skasowane), 73 kanon (chronione), 590 nieznane (zostawione). Bez niej nie dalo
by sie tego rozdzielic.
Wynik: 62 120 przypisan zdjetych, 2156 wpisow odchudzonych (Devine 170 na 60), 1740
pustych rekordow usunietych. Regula dopieta do joba junk_performers, drugi przebieg
daje zera.
Zostaje residuum: sceny, na ktorych osoba pelnoimienna NIE byla przypisana, wiec
regula nie miala sie o co oprzec (James, Blue, Moon, Crystal, Cruz, Starr, Knight po
60-160 przypisan). Tam trzeba by DOPISAC wlasciwa osobe, a nie kasowac - to osobne,
bardziej ryzykowne zadanie.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"A", "L", "Ro", "Be", "Js" mialy po 250-290 scen kazdy. To artefakty parsowania
tytulow przez tube-search, ale NIE wszystkie: "Cj", "PD" i "JD" to realne krotkie
pseudonimy z obsada potwierdzona w TPDB i StashDB, z prawdziwymi partnerami
scenicznymi (Cj z Lucy Belle i Denisem Martim, PD z Victoria Love).
Dlatego nie kasujemy tu performera, tylko przypisania, o ktorych WIEMY ze dokleil je
scraper. Reszta rozstrzyga sie sama: komu nic nie zostanie, ten znika. To pierwsze
uzycie provenance z migracji 0026 - bez niej nie dalo sie odroznic "Cj" z obsady
TPDB od "Cj" wylusknietego z tytulu.
Rozklad byl jednoznaczny: z 9404 przypisan 9232 (98,2 proc.) ze scrapera, 141 z
kanonu, 31 nieznanych. Wiersze z NULL-em zostawiamy, nie zgadujemy.
Wynik: 9232 przypisan zdjetych, 20 performerow zniknelo calkiem, 31 zostalo z sama
prawdziwa obsada (Cj 41, PD 38, JD 20, reszta <=10). Dopiete do joba junk_performers,
zeby nie odroslo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Klasa "prawdziwy performer o pospolitej nazwie" okazala sie zawierac 211 rekordow,
ktore w ogole nie sa osobami, tylko kategoriami z refem kanonicznym: "Big Tits"
(931 scen), "Big Ass" (640), "Hardcore" (222), "Blonde" (238), "Masturbation".
TPDB i StashDB SAME zawieraja takie wpisy, wiec ref kanoniczny ich nie oczyszcza.
Kryterium promiskuitycznosci z poprzedniego commita tu NIE dziala i sprawdzenie tego
bylo kluczowe: w tej populacji obcy mierzy popularnosc, nie smieciowosc. Przy
wartosciach 50-59 stoja Adriana Chechik (4201 scen), Lauren Phillips, Jane Wilde i
Jordi El Nino Polla. Moja wczesniejsza walidacja progu 50 byla obciazona, bo grupa
bez refa to sami malo znani.
Rozstrzyga stosunek scen-z-tagiem do scen-performera, i to z uzasadnienia, nie z
probki: nazwisko uzyte jako wlasny tag daje stosunek ~1 Z DEFINICJI, bo tag siedzi
dokladnie na jej scenach. Potwierdzone na markach osobistych: Hotwifevictoria 1,02,
Rossmexicana 0,99, Sexwithmilfstella 0,87. Wielkie gwiazdy jeszcze nizej, maksimum
0,35 wsrod 921 pewnych osob. Kategorie: Hardcore 1720, Masturbation 1570, Blonde 770.
Prog 2 + wymog >=50 scen (ponizej stosunek jest szumem).
Skasowane 211 rekordow / ~32 tys. przypisan w dwoch przebiegach. Job idempotentny.
Ogon klasy jest zdrowy: z 8116 pozostalych 92% miesci sie ponizej 0,35.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Kategorie i studia podszywajace sie pod osoby: "Creampie" (269 scen), "Natural tits"
(670), "reverse cowgirl" (315), "69", "Brazzers", "Blacked", "Twistys". Razem 277
rekordow i 33 784 przypisan do scen.
Zrodlo: _search_base filtruje wyniki tube-searcha warunkiem "slug zawiera >=1 token
z zapytania", a resolve_performer tworzy performera z dowolnej nazwy podanej przez
tube. Zadnej bramki po drodze.
KRYTERIUM JEST WASKIE CELOWO. To uzywane w app/api/scenes.py do de-rankingu (slug
pokrywa sie z tagiem I brak refa tpdb/stashdb) daje 3111 trafien, ale probka pokazala
~60% falszywek dla nazw wielowyrazowych: wpadaja w nie Hatano Yui, Davina Raines,
Michelle Ferrari, Clanddi, SolaZola, Kate Kuray. Zalozenie "prawdziwy performer ma ref
kanoniczny" nie dziala dla JAV, amatorek i debiutantek. Do de-rankingu wystarcza, do
kasowania nie.
Rozstrzyga promiskuitycznosc taga: ile ROZNYCH innych performerow pojawia sie na
scenach z tym samym slugiem. Zmierzone: prawdziwe osoby maksimum 20 (Julia Reaves),
wiekszosc 0-4, Hatano Yui 0. Kategorie: Deepthroat 39541, Creampie 33661, riding
16284. Prog 50 lezy ponad 2x powyzej najwyzszej prawdziwej osoby. Odrzucilem tez
sygnal "slug rowny studiu", bo wciagal marki osobiste (Katekuray, Julia Reaves).
Tresc nie ginie: te slugi dalej istnieja jako tagi (creampie przy 176 tys. scen),
przestaja tylko udawac osoby. 16 968 scen zostalo bez obsady, co dokłada 1,8 procent
do istniejacych 943 tys. scen bez obsady.
Job co 24h zamiast bramki w resolve_performer: nie dotyka goracej sciezki ingestu,
a smieci odrastaja wolno. Ta sama logika co przy blackliscie phashy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audyt duplikatow po phashu miniatur wykazal dwa rozne zjawiska, ktore wygladaly
jak jedno.
1. Zdegenerowane phashe. 73 wartosci wystepowaly przy >=10 scenach kazda, lacznie
przy 4375 scenach; rekordzistka byla dzielona przez 892 sceny o 892 roznych
tytulach i 1886 roznych performerach. To zaslepki i czarne klatki, nie odciski
scen. find_by_phash_within bierze najblizsza wartosc z calej tabeli, wiec taka
zaslepka zawsze wygrywala z prawdziwym duplikatem (dist 0). Do tej pory bronila
nas bramka dur_prox i nic sie nie skleilo, ale to zabezpieczenie drugiej linii.
Blacklista jest tabela, nie jednorazowym DELETE, bo sam DELETE nic nie daje:
zaslepka wraca przy kolejnym ingescie. Trzeba pamietac, ze wartosc jest
bezuzyteczna. Job co 24h dopisuje nowe i czysci odciski. Po czyszczeniu zero
grup >=10, najwieksza pozostala ma 9.
2. Realne duplikaty. 4647 scalonych. Przyczyna byla jedna: 97 procent par ma
perverzije po dokladnie jednej stronie, bo pisze tytuly z prefiksem studia i
performera, a reszta tubow daje goly tytul. Wbrew mojej pierwszej diagnozie NIE
trzeba tu ruszac scoringu tytulu (token_set_ratio i tak radzi sobie z prefiksem,
a sciezka phash idzie przed composite): 80 procent par to dlug sprzed 60+ dni, a
biezacy wyciek to okolo 1 dziennie. Nadmiarowe wiersze: 18452 na 9449.
merge_phash_exact_dupes.py dostal wykluczenie blacklisty. Bez tego byl grozny:
sam klaster 892 scen dawal ~397 tys. par do rozwazenia.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
youperv.com (DataLife Engine) carries paysite rips titled
"Studio - Performer - Title" (71% of a 132-title sample), ~60-70 new
scenes/day. Browse the homepage + /page/N/ (19 scenes/page, no overlap);
scene URL is /<category>/<id>-<slug>.html.
Cast is scoped to the fmeta block (up to the Related section): the whole
page carries 19-26 /xfsearch/pornstar/ links but only 1-2 are the actual
cast, so without scoping this would repeat the page-wide pollution that
got xxxfiles rejected. Studio comes from the title prefix, guarded so a
performer name is never mistaken for a studio. Duration, ISO release date,
per-scene tags and thumbnail all come from the same block.
Playback is a plain <source> mp4 (files.klubnichka-hd.com) with no token
or expiry, but the CDN hotlink-guards on Referer: bare Range gets 403,
Range + Referer + browser UA gets 206 cross-IP from the VPS. So the
extractor returns it with referer + mobile_direct_ok and the phone streams
straight from the CDN, no WebView and no proxy. Path is percent-encoded
because the filenames contain spaces.
Deep-crawl capped at 2000 pages: beyond that (<=09.2023) the catalog turns
into generic amateur uploads with no performers and dead CDN files.
Verified: 19 scenes/page with studio+cast+duration+date, max 2 performers
per scene (pollution guard holds), 0/19 missing duration, playback 206
video/mp4. Pilot ingest 3 pages: 57 seen, 39 attached to existing canonical
scenes, 18 new, 0 errors.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
freshporno gra natywnie (zweryfikowane na realnym telefonie, PlaybackState=
PLAYING) ale udane odtworzenia nie raportują się jako success event → sam błąd
w telemetrii → 0% success → health=0 → fałszywe 'offline' i demote.
_overall_stars: gdy health==0 ale źródło jest ewidentnie żywe (freshness>=4 i
richness>=4), floor 2★ 'flaky' zamiast 0★ 'offline'. Martwe źródła mają niski
freshness więc gate dalej działa.
Enrich existing movies (from paradisehill/dooplay, which mostly lack cast)
with metadata from TPDB's /movies API: cast, categories (tags), studio,
director + a canonical TPDB UUID for dedup. Chosen over IAFD after a
source-comparison research pass — IAFD has strong cast/studio but ZERO
categories, while TPDB /movies has ~11 tags/movie, cast, studio, director,
a canonical UUID (+ sparse phash), is already an integrated API (no
scraping/anti-bot), and covers ~75-85% of our western-DVD-feature catalog.
Enrichment only ever augments EXISTING movies and never creates new ones
(TPDB has no playback, so a standalone TPDB movie would be unplayable).
Writes to movie_performers / movie_tags / movie.studio_id, which the movies
API + mobile detail already render, so no schema/API/UI change is needed.
- connectors/tpdb.py: search_movies() + fetch_movie() + _parse_movie()
reusing the existing _parse_studio/_parse_performer/_parse_tag.
- enrich/tpdb_movies.py: match our movie to a TPDB /movies result by
token_sort_ratio on normalized titles (sort, not set, to reject the
short-title-subset trap "Fantasies" -> "Tara's Fetish Fantasies") with a
+/-2yr guard; then attach cast/tags/studio/director. Incoming performers
deduped by external_id to avoid the performer_external_refs PK clash.
- resolve/movie_merge.py: merge_movies() mirror of scene_merge; two of our
movies mapping to the same TPDB UUID are the same film -> merge.
- scheduler: _job_tpdb_movie_enrich every 6h, batch 200, prioritizing
playable movies missing cast/studio.
Verified on a 150-movie batch: 119 enriched, 4 deduped, 26 no-match,
0 errors; matched titles/studios spot-checked correct (Big Butts Drive Me
Nuts 4 -> 33 tags, Seinfeld #2 -> 10 cast/17 tags, German BB Video titles
-> categories+studio).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up cleanup from the ultra-review (behaviour-preserving, verified equivalent):
- scenes.py: extract live_playback_exists / blacklist_clauses / stub_exclusion_clause;
list_scenes and favorites now share ONE definition of "visible scene" (was a hand-kept
copy in favorites -> the count-vs-list drift class). Verified identical: helper vs old
inline both count 1,853,327 scenes.
- favorites.py: replace the two copy-pasted count blocks with one _new_counts(kind=...)
that counts in SQL (count(*) FILTER over the windowed subquery) instead of streaming up
to N*200 rows to Python; joins the favorite table for per-row last_seen. Deployed
_new_counts verified == hand SQL (studios 364). Dropped now-unused imports.
- mobile: extract lib/newScenes.ts (isNewScene / sortNewFirst); SceneTile + Performer/
StudioScenes use it (was triplicated, already drifted once in the deleted FavoriteSceneRow).
- SceneDetail: fold tube:pornxpph into the phoneResolver map (was a verbatim-duplicated block).
- deep_crawl: persist the cursor after every completed page (was once at run end), so a
mid-page hard-kill past the soft budget can't lose progress (GOON-V hardening).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the ultra-review findings on this branch:
Player (PlayerScreen.tsx): the new recoveryPending mirrored the fallback-chain guards
by hand and could deadlock into a permanent "Reconnecting" spinner with no way to Mark
broken — for gone (410) sources on IP-bound tubes (re-resolve bails before setting
reResolveDone) and for any post-load error on those tubes (re-resolve is initial-load
only). Derive one reResolveApplicable flag (IP-bound AND initial-load AND not-gone) and
use it for both the chain gate and the spinner, so gone/post-load errors fall through to
proxy/WebView or the terminal error card. Seek-recovery now falls through to the chain
when player.replace() throws instead of returning.
Quick-play (SceneDetail): the autoplay route param persisted and autoPlay={i===0} re-fired
when the source list reordered (e.g. after Mark broken drops the dead source), bouncing the
user into the player. Consume it once via onAutoPlayConsumed -> nav.setParams({autoplay:false}).
Backfill semantics: performer-driven direct-scraper "backward fill" now tags scenes
backfill=True (search-by-name pulls the whole old catalog); merge coalesces backfill
(keep AND drop) so a fresh scene merged into a dead dup keeps NEW; deep-crawl only tags
backfill on a tube's FIRST sweep (swept_once) so re-sweep catalog growth stays genuine;
pilot script tags backfill.
Perf/migration: migration 0026 is now idempotent (IF NOT EXISTS; prod got the column via
manual ALTER) and adds ix_scene_performers_performer_id (favorites count filtered
performer_id with no index); index also created on prod.
Cleanup: deleted dead FavoriteSceneRow (unused import in two screens, stale isNew without
the backfill guard); removed em-dashes from all lines this branch added (user CLAUDE.md
rule), including the user-facing changelog / Settings / player-overlay strings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Browse scrapers backfilling old catalogs stamp the tube's import/post date as
release_date, so old content (e.g. 83 MissaX classics via perverzija, ~3600/3 days
across eporner/youporn/etc.) fake-ranked as newest and flooded the favorites "+N".
NULLing the dates was a non-starter — the stub filter would hide 251k performer-less
scenes. Instead: a Scene.backfill flag marks bulk catalog imports; they stay visible but
never count as "new".
- scenes.backfill column (+ index, migration 0026); resolve_scene/_process_scene thread it.
- deep_crawl tags scenes from pages beyond the "latest" threshold (>2) as backfill;
latest pages + TPDB/StashDB delta stay genuine. Cursor reset re-sweeps page 1 so real
new content is always caught fresh.
- favorites +N (performers + studios) excludes backfill within the top-200 window.
- SceneOut exposes `backfill`; mobile NEW badge + NEW-first re-sort skip it (badge==count).
- Retroactive: tagged 439k existing scenes in bulk (>10 non-canonical / studio / day)
clusters. Device check: favorite-studios +N 11626 (naive) -> 236.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run_deep_crawl picks one tube/run and crawls 60 pages under _job_deep_crawl's 3600s
hard timeout. A detail-fetch scraper on a slow patch (per-scene page fetch, e.g. via
proxy) could exceed it → the run is killed mid-page, the cursor is never saved (orphan
thread), and that tube makes zero progress — recurring Sentry GOON-V. Added a 3000s
in-run budget that breaks after a completed page, saves the cursor, and returns cleanly;
the next run continues. budget_hit surfaced in the summary log to spot the slow tube.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
freshporno (5★ fresh+rich, verified working — 206/507MB) was labeled OFFLINE off 10
playback attempts that all failed in one unlucky window (a CDN-node blip; it resolves
fine now). 10 was too thin a sample to zero out a known-good source's stars. Raised
the telemetry-trust threshold 10→25; below it we fall back to the proxy/heuristic
health instead of declaring offline (user-report cb526949).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rates each source on three axes the user asked for:
- freshness: how recently/often new content arrives (newest age + 7d volume)
- richness: metadata coverage (thumbnail/tags/performers/description/studio/duration)
- plays: does it actually play — from real playback telemetry when available,
else a proxy from the resolve mechanism. 0★ = offline (gates the overall stars,
so a fresh+rich source that doesn't play still ranks bottom — the hqfap/4k69 case)
Backend:
- playback_events: fire-and-forget telemetry POST from the app per playback attempt
(origin + success/error + time-to-first-frame), append-only, 30d retention
- source_stats: per-origin computed scores, refreshed by a scheduler job (6h);
/sources joins it and sorts by stars
- models + local migration 0025; new GOON_SCHED_SOURCE_STATS_HOURS setting
Mobile:
- Sites rows show ★ rating; tap the stars for a breakdown (axes + metadata %, plus
whether "plays" is measured or estimated)
- PlayerScreen reports playback success/failure per source (native path only —
symmetric, conservative); origin threaded through Scene/Movie play callsites
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Missing-merge duplicates (same performer + identical normalized title + identical duration-to-the-second) that bulk_dedup misses — tube re-scrapes and cross-tube re-ingests like porn00 pulling a video already present from xnxx (reports 28fe8181/32df33b1). Extracted the proven merge_exact_title_duration logic into app/scheduler/title_duration_dedup.py (script now a thin wrapper), wired a 12h scheduler job (playback-only = what users actually see, GOON_SCHED_TITLE_DEDUP_HOURS). Signal is near-certain (two different videos don't share byte-identical title AND exact duration); no shared performer = not merged (over-match guard). Verified: job registers (jobs=14), backlog currently 0 after the one-shot global merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two observability additions to the worker scheduler (intertwined in the same files): (1) ingest-watchdog now also covers performer-driven search scrapers (ALL_DIRECT_SCRAPERS) with a separate 7d threshold, not just browse tubes at 48h — several search tubes (perverzija, fpoxxx, porndish, ...) had frozen silently for weeks. (2) New Hetzner Cloud bandwidth monitor (app/scheduler/hetzner_monitor.py): polls outgoing_traffic vs included_traffic and fires a Sentry message at info/warning/error % thresholds with a per-level fingerprint. The config fields existed for ages but the monitor was never implemented. No-op until HETZNER_API_TOKEN + HETZNER_SERVER_ID are set in .env (verified: returns {enabled: False}, job registers as 'hetzner-monitor every 6h', jobs=13).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global source monitor can't catch a single stalled tube because every tube scraper shares one Source row (tube-scraper), so an aggregate run still reports success while one origin freezes (freshporno browsing the rotating KVS homepage root, report 14f3a655). New watchdog checks max(created_at) per active browse-scraper origin (tube:<sitetag>); if a tube with history hasn't produced a new scene in > max_age_hours it fires a Sentry message with a stable per-origin fingerprint (age in extras, not the title, so it stays one grouped issue). Runs every 6h, 48h threshold, both env-tunable (GOON_SCHED_INGEST_WATCHDOG_HOURS / GOON_INGEST_WATCHDOG_MAX_AGE_HOURS). Verified: 0 stale at 48h post-fix, detects neporn at a strict 12h threshold.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one-off cleanup merged ~13.5k same-video-different-title dupes, but they regrow as
these sibling tubes re-ingest under new titles. Wire the asset-id+duration merge into
the scheduler (every 12h, GOON_SCHED_THUMB_DEDUP_HOURS, 0=off) so it stays clean.
Shared logic lives in app/scheduler/thumb_dedup.py (run_thumb_asset_dedup); the one-shot
script now imports it. Same tight signature as the cleanup: family hosts only + identical
duration (the bare asset-id number is reused across unrelated CDNs, so cross-host/diff-
duration grouping is excluded). Reports 205b17d9 / 5a2944cb.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CORRECTION: trafficdeposit thumbnail tokens are hour-bucketed and valid only ~1h
(verified 2026-06-10: stored ts=11:00 dead at 12:27, current ts=13:00 loads). Earlier
"~weekly rot" read was wrong. Storing/periodically-refreshing sxyprn thumbnail URLs
is futile — they expire within the hour. Default the refresh job OFF (kept in code).
The dead-marking sweep (Post Not Found → dead_at) it performed was still valid. Live
sxyprn thumbnails need on-demand resolution at serve time (future work).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CORRECTION to earlier "unrecoverable" call: the /post/<id> page is alive (200) and
DOES expose the scene's own fresh-signed poster via og:image / <video poster>
(post-id embedded, current timestamp) — only the STORED thumbnail URL had rotted.
Search/listings don't re-surface old posts (0 overlap), but per-post fetch works.
scripts/refresh_sxyprn_thumbs.py: iterate live sxyprn sources, fetch post page,
extract fresh og:image, UPDATE thumbnail_url (verified: refreshed URLs return 200).
_job_refresh_sxyprn_thumbs: every 12h refresh the 1200 least-recently-updated sources
(cycles the ~19k catalog within the expiry window). Pairs with the scene_resolver
overwrite fix so refreshed thumbnails stick.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_candidate used OR logic (studio OR date±7d OR dur±30s) → 938,950 pairs;
Etap-2 scoring at ~110/s never finished in 1800s → bulk_dedup_performers HUNG
every run, orphan thread leaked until restart. Require AND: same studio plus
(date±2d OR dur±30s). 939k→16k pairs, full run 213s. Real cross-source dup of
one master shares studio + near date/duration; rare studio_id-mismatch pairs
skipped on purpose — a job that COMPLETES beats one that times out merging nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bug-report 2026-06-03 ("ten sam czas, ta sama miniaturka, czemu się nie mergują"):
duplicate scenes not merged at ingest. Exact phash alone is noisy here (95% are
collisions on shared thumbnails/intro frames — different scenes; bulk_dedup scorer
correctly gives 0 auto-merge). The safe subset is exact-phash AND same duration
(±3s) AND shared performer/title — near-certain same scene. Same-duration is key:
it excludes the false-merge pattern (short-clip-vs-full has DIFFERING durations).
- scripts/merge_phash_exact_dupes.py: one-off, dry-run by default, per-pair re-fetch
(handles clusters). Applied: 30 merged.
- bulk_dedup: add `_pairs_exact_phash` (SQL O(N log N), not the O(N²) Hamming scan)
+ strategy "phash_exact" — gated by the normal scorer (surfaces review candidates,
no risky auto-merge), schedulable for ongoing exact-collision review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The umbrella Source.name for all direct tube scrapers (deep-crawl, browse-latest,
performer-driven) was "pornapp" — a misleading leftover from the removed external
porn-app API. It read like a dependency on a third-party "pornapp" service; it is
not — these are our own scrapers hitting 25+ tubes directly (kind=scraper,
origin tube:<sitetag>). Renamed to "tube-scraper" via a single SCRAPER_SOURCE_NAME
constant; DB row renamed in place (UPDATE name, same id) so all ingest_runs +
external_records history stays linked. No behavior change — external_id keying
(sitetag:url) and dedup are unaffected.
NOTE: playback_sources.origin "pornapp:<sitetag>" prefix is a separate legacy
format (resolve_playback parses it) and is intentionally left untouched.
Verified on prod: row renamed (0 stray "pornapp"), new runs land on "tube-scraper".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_job_bulk_dedup_performers called run_bulk_dedup(strategy="performers") without
the cross_source_only guard whose docstring exists precisely to prevent this OOM.
At current catalog scale the unguarded path materializes N²/2 pairs per prolific
performer into a list → worker hit 6GB RSS and was OOM-killed every 12h (05:00/
17:00), taking down concurrent tpdb/stashdb/movie ingests as killed_by_restart
(0 new movies). Verified in prod: 05:00 run now completes (885k pairs scored, no
OOM) and ingests succeed (stashdb +241, tpdb +175).
Also wrap in _run_with_timeout like tpdb/stashdb (job had no hard-timeout).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xvideos SSR's JSON-LD VideoObject (duration/title/uploadDate) + on-page /models/ (perf)
+ /tags/. Sample: median ~10.5min, 93% >=3min. Pilot (2 pages): 29 new, 100% playable +
visible + tagged (performers sparse — xvideos 'new' is amateur-heavy; /models/ tagged
mostly on studio rips).
- XVideosBrowseScraper (JSON-LD + page-parse models/tags), in ALL_BROWSE_SCRAPERS.
- deep_crawl._PAGE_CAP: per-sitetag depth cap; xvideoscom=1800 (~newest 50k). At the cap
the tube is marked exhausted (reset -> incremental re-sweep) so a mega-tube cannot
monopolize the round-robin or balloon the DB.
- ported yesporn.py into the public repo (was prod-only, like hdporngg) ending the
__init__ public/prod divergence.
youporn rejected: JSON-LD lacks actor/keywords, its /pornstar//category/ links are A-Z
nav not scene-specific. xhamster: 429/Cloudflare from the VPS IP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
porntrex/hqporner rejected for deep-crawl: KVS sites with no SSR metadata (77% of
existing porntrex has no duration -> invisible under the app's >=60 filter). eporner
instead exposes a public JSON API (api/v2/video/search) returning title + length_sec
+ keywords + added per video; ~100k videos, ~100/page, no per-scene detail fetch.
- BaseBrowseScraper.crawl_page(page): factored out of latest_scenes; returns None
(transient fail) / [] (catalog end) / [scenes]. API subclasses override it.
- deep_crawl drives via crawl_page (supports HTML-listing AND API sources).
- EpornerApiScraper: crawl_page hits the eporner API -> RawScene with duration+tags+
date+thumb+playback; registered in ALL_BROWSE_SCRAPERS.
- Pilot (2 API pages): 192 new, 100% playable + tagged + visible (>=60); the <180s
trailer filter dropped 6 short clips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
We ingested only ~3% of each browse tube's catalog (porndoe >62k scenes; we had 1959)
because tubes were hit only by performer-search + top-N browse. Pilot (porndoe pages
64-110): 1119 new scenes, 100% playable + 100% tagged, 0% canonical overlap (purely
additive — content not in TPDB/StashDB).
- app/scheduler/deep_crawl.py: round-robin over ALL_BROWSE_SCRAPERS, per-tube page cursor
in app/_state/deepcrawl_state.json (no DB migration), deep-paginate from the cursor,
idempotent (resolver skips known by raw_hash), mark 'exhausted' at catalog end then
reset cursors for an incremental re-sweep.
- _job_deep_crawl: hourly, 60 pages/run (~1860 scenes, ~22 min), wrapped in the 1h
hard-timeout; registered in build_scheduler (jobs=10).
- config: sched_deep_crawl_hours=1, deep_crawl_pages_per_run=60, deepcrawl_state_path.
- scripts/pilot_porndoe_deepcrawl.py: one-off pilot used to validate the approach.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
At the shared 05:00 anchor all heavy jobs fire together; tpdb/stashdb/performer-driven
had no timeout, so a hung connector blocked the whole job and — with max_instances=1 —
blocked every future fire of that job until a worker restart (incident 2026-06-02: 6 runs
hung 8.7h, movie mirrors 47h stale, tube ingest stalled).
- _run_with_timeout wraps tpdb/stashdb/performer-driven in a 30-min hard cap (same
ThreadPoolExecutor pattern movie-ingest already uses): on timeout the job returns and
frees the scheduler slot; the orphaned thread lives until restart.
- _job_reap_stuck: hourly reaper of 'running' >2h rows, registered in the scheduler —
the startup-only reaper missed hangs while the worker stayed up for hours.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scene.duration_sec was NULL for ~74% of playable scenes (tube duration lives on
playback_source, never propagated to Scene), so the mobile min_duration_sec=60 filter
(Scene.duration_sec >= 60; NULL fails) silently hid them — surfaced as '119 in favorites,
14 after entering the performer' (Safira Yakkuza).
- resolver: _effective_duration() falls back to max live playback_source duration when the
connector provides no scene-level duration (forward fix, used in create + update).
- scripts/backfill_scene_duration_from_playback.py: one-off idempotent backfill (recovered
204,014 scenes).
- taxonomy_counts: scene_count now counts playable AND duration_sec >= 60, matching the
always-60s-filtered scene lists, so favorites/performer/studio/tag badges agree with what
the scene screen actually shows (Safira: 39 == 39).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- paradisehill.fetch_movies compared release_date coerced to midnight against the
`since` timestamp, so the chronological crawl stopped at the first upload dated
the same calendar day as `since` and silently dropped most new movies (0-2 seen
per run; Movies tab stalled). Compare by DATE with a 1-day grace instead; idempotent
external_records upsert dedups the re-fetched recent window.
- scripts/backfill_paradisehill_movies.py: one-off no-delta deep crawl to recover the
backlog missed during the bug (idempotent, resumable).
- docs: correct stale 'raz dziennie/24h' browse-latest comments to 6h (4x/day), the
actual configured cadence (config.py sched_browse_latest_hours=6).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolver/perf:
- find_by_phash_within: nearest match via Postgres bit_count over bit(64) XOR
instead of Python scan of all phash fingerprints (~20x faster per scene;
unblocks long delta runs that were killed mid-run before since advanced).
Scheduler/reliability:
- reap ingest_runs stuck in 'running' on worker startup (killed_by_restart).
- smoke_test: per-source ingest health, stuck-run and browse-freshness checks
-> Sentry; exclude killed_by_restart from the failed-run alarm.
Tags (ingest with tags + fill blanks):
- wire infer_tag_slugs into normalize_scene so tube scenes get title-inferred
tags (was dead code); union with connector tags.
- scripts/backfill_inferred_tags.py: keyset/batched/idempotent backfill for
existing tagless scenes (playable tag coverage 16% -> ~52%).
Clip-store:
- skip ManyVids/IWantClips/Clips4Sale/... from canonical sources at ingest
(GOON_SKIP_CLIP_STORE, default on) — permanent orphans, ~56% of canonical
ingest, never have a free-tube playback source.
Browse tubes:
- enable fullmovies + hdporn.gg: studio parsed from title prefix instead of
the /networks/ sidebar (which always yielded the first listed network);
drop phash compute (pilot: 0% canonical hit within Hamming 5 — auto-screenshots),
matching relies on title/performer/duration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Counts for /tags, /performers, /studios and /favorites were computed live
per-request by aggregating scene_tags / scene_performers with an EXISTS to
playback_sources. As the catalog grew to ~1.7M scenes (6.3M scene_tags) this
ran ~4.3s for /tags?order=popular (x2 incl. the total count) and ~950ms for
the default /scenes count, making those screens load in several seconds.
- migration 0019: add scene_count (+ DESC index) to tags/performers/studios
- background job _job_refresh_taxonomy_counts (every 3h) recomputes the counts
in one UPDATE..FROM each (IS DISTINCT FROM to skip unchanged rows)
- /tags, /performers, /studios scenes path now read the column + ORDER BY the
indexed scene_count; for_movies paths keep live aggregation (small tables)
- favorites read denormalized scene_count instead of a grouped EXISTS aggregate
- /scenes default count: 10-min in-process TTL cache (header is approximate)
Measured: /tags?order=popular&per_page=500 ~8s -> 66ms incl. serialization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bug-report 2026-05-30 "ingest znów się zawiesił". streamporn/pandamovies
wieszały się intermittentnie mid-run (zależnie od live-contentu danego dnia),
blokując sekwencyjny _job_movie_ingest → mangoporn (jedyny mirror z realnym
new-content: 72 nowych 05-28) nigdy nie startował. try/except chronił przed
wyjątkiem, NIE przed hangiem.
Fix:
- _job_movie_ingest: każdy connector w ThreadPoolExecutor z future.result
(timeout=360s). Hang jednego źródła → log + shutdown(wait=False) + kolejka
leci dalej. Healthy run ~50s, cap 6min = zapas.
- get_movie_connectors: reorder paradisehill, MANGOPORN, streamporn, pandamovies
— mangoporn zaraz po canonical primary, przed wolniejszymi/wieszającymi się.
Zweryfikowane: pełny _job_movie_ingest przeszedł wszystkie 4 success w nowej
kolejności (mangoporn 2nd, 23s). 33 osierocone "running" rows (worker ubity
mid-run przy deployach) wyczyszczone osobno.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Goon — self-hosted aggregator for adult-content scene metadata.
Indexes scenes from TPDB, StashDB, and 30+ public adult tube sites.
Cross-source deduplication via perceptual hash + Levenshtein distance.
FastAPI backend + APScheduler worker + React Native (Expo) mobile client.
FOSS, ad-free, donation-funded. See README for details.
