Commit graph

30 commits

Author SHA1 Message Date
goon-foss
aa11fb022a fix(stars): fresh+rich source not marked offline on telemetry gap
Some checks are pending
Backend tests / test (push) Waiting to run
freshporno gra natywnie (zweryfikowane na realnym telefonie, PlaybackState=
PLAYING) ale udane odtworzenia nie raportują się jako success event → sam błąd
w telemetrii → 0% success → health=0 → fałszywe 'offline' i demote.

_overall_stars: gdy health==0 ale źródło jest ewidentnie żywe (freshness>=4 i
richness>=4), floor 2★ 'flaky' zamiast 0★ 'offline'. Martwe źródła mają niski
freshness więc gate dalej działa.
2026-07-20 20:58:02 +02:00
goon-foss
b9898ba592 feat(movies): TPDB movie enrichment + dedup
Enrich existing movies (from paradisehill/dooplay, which mostly lack cast)
with metadata from TPDB's /movies API: cast, categories (tags), studio,
director + a canonical TPDB UUID for dedup. Chosen over IAFD after a
source-comparison research pass — IAFD has strong cast/studio but ZERO
categories, while TPDB /movies has ~11 tags/movie, cast, studio, director,
a canonical UUID (+ sparse phash), is already an integrated API (no
scraping/anti-bot), and covers ~75-85% of our western-DVD-feature catalog.

Enrichment only ever augments EXISTING movies and never creates new ones
(TPDB has no playback, so a standalone TPDB movie would be unplayable).
Writes to movie_performers / movie_tags / movie.studio_id, which the movies
API + mobile detail already render, so no schema/API/UI change is needed.

- connectors/tpdb.py: search_movies() + fetch_movie() + _parse_movie()
  reusing the existing _parse_studio/_parse_performer/_parse_tag.
- enrich/tpdb_movies.py: match our movie to a TPDB /movies result by
  token_sort_ratio on normalized titles (sort, not set, to reject the
  short-title-subset trap "Fantasies" -> "Tara's Fetish Fantasies") with a
  +/-2yr guard; then attach cast/tags/studio/director. Incoming performers
  deduped by external_id to avoid the performer_external_refs PK clash.
- resolve/movie_merge.py: merge_movies() mirror of scene_merge; two of our
  movies mapping to the same TPDB UUID are the same film -> merge.
- scheduler: _job_tpdb_movie_enrich every 6h, batch 200, prioritizing
  playable movies missing cast/studio.

Verified on a 150-movie batch: 119 enriched, 4 deduped, 26 no-match,
0 errors; matched titles/studios spot-checked correct (Big Butts Drive Me
Nuts 4 -> 33 tags, Seinfeld #2 -> 10 cast/17 tags, German BB Video titles
-> categories+studio).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 11:36:36 +02:00
goon-foss
cb0b843f48 refactor(review): dedup favorites/list visibility + SQL-aggregate count, share mobile isNew
Follow-up cleanup from the ultra-review (behaviour-preserving, verified equivalent):

- scenes.py: extract live_playback_exists / blacklist_clauses / stub_exclusion_clause;
  list_scenes and favorites now share ONE definition of "visible scene" (was a hand-kept
  copy in favorites -> the count-vs-list drift class). Verified identical: helper vs old
  inline both count 1,853,327 scenes.
- favorites.py: replace the two copy-pasted count blocks with one _new_counts(kind=...)
  that counts in SQL (count(*) FILTER over the windowed subquery) instead of streaming up
  to N*200 rows to Python; joins the favorite table for per-row last_seen. Deployed
  _new_counts verified == hand SQL (studios 364). Dropped now-unused imports.
- mobile: extract lib/newScenes.ts (isNewScene / sortNewFirst); SceneTile + Performer/
  StudioScenes use it (was triplicated, already drifted once in the deleted FavoriteSceneRow).
- SceneDetail: fold tube:pornxpph into the phoneResolver map (was a verbatim-duplicated block).
- deep_crawl: persist the cursor after every completed page (was once at run end), so a
  mid-page hard-kill past the soft budget can't lose progress (GOON-V hardening).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 09:41:37 +02:00
goon-foss
15e7c1646d fix(review): player recovery deadlocks, quick-play re-fire, backfill gaps, em-dashes
Addresses the ultra-review findings on this branch:

Player (PlayerScreen.tsx): the new recoveryPending mirrored the fallback-chain guards
by hand and could deadlock into a permanent "Reconnecting" spinner with no way to Mark
broken — for gone (410) sources on IP-bound tubes (re-resolve bails before setting
reResolveDone) and for any post-load error on those tubes (re-resolve is initial-load
only). Derive one reResolveApplicable flag (IP-bound AND initial-load AND not-gone) and
use it for both the chain gate and the spinner, so gone/post-load errors fall through to
proxy/WebView or the terminal error card. Seek-recovery now falls through to the chain
when player.replace() throws instead of returning.

Quick-play (SceneDetail): the autoplay route param persisted and autoPlay={i===0} re-fired
when the source list reordered (e.g. after Mark broken drops the dead source), bouncing the
user into the player. Consume it once via onAutoPlayConsumed -> nav.setParams({autoplay:false}).

Backfill semantics: performer-driven direct-scraper "backward fill" now tags scenes
backfill=True (search-by-name pulls the whole old catalog); merge coalesces backfill
(keep AND drop) so a fresh scene merged into a dead dup keeps NEW; deep-crawl only tags
backfill on a tube's FIRST sweep (swept_once) so re-sweep catalog growth stays genuine;
pilot script tags backfill.

Perf/migration: migration 0026 is now idempotent (IF NOT EXISTS; prod got the column via
manual ALTER) and adds ix_scene_performers_performer_id (favorites count filtered
performer_id with no index); index also created on prod.

Cleanup: deleted dead FavoriteSceneRow (unused import in two screens, stale isNew without
the backfill guard); removed em-dashes from all lines this branch added (user CLAUDE.md
rule), including the user-facing changelog / Settings / player-overlay strings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 09:20:40 +02:00
goon-foss
07cc3fdfba fix(favorites): exclude bulk-backfill scenes from "+N new" (tube fake dates)
Browse scrapers backfilling old catalogs stamp the tube's import/post date as
release_date, so old content (e.g. 83 MissaX classics via perverzija, ~3600/3 days
across eporner/youporn/etc.) fake-ranked as newest and flooded the favorites "+N".
NULLing the dates was a non-starter — the stub filter would hide 251k performer-less
scenes. Instead: a Scene.backfill flag marks bulk catalog imports; they stay visible but
never count as "new".

- scenes.backfill column (+ index, migration 0026); resolve_scene/_process_scene thread it.
- deep_crawl tags scenes from pages beyond the "latest" threshold (>2) as backfill;
  latest pages + TPDB/StashDB delta stay genuine. Cursor reset re-sweeps page 1 so real
  new content is always caught fresh.
- favorites +N (performers + studios) excludes backfill within the top-200 window.
- SceneOut exposes `backfill`; mobile NEW badge + NEW-first re-sort skip it (badge==count).
- Retroactive: tagged 439k existing scenes in bulk (>10 non-canonical / studio / day)
  clusters. Device check: favorite-studios +N 11626 (naive) -> 236.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 16:04:01 +02:00
goon-foss
449b7e30d3 fix(deep-crawl): soft per-run time budget so a slow tube can't hit the hard kill
run_deep_crawl picks one tube/run and crawls 60 pages under _job_deep_crawl's 3600s
hard timeout. A detail-fetch scraper on a slow patch (per-scene page fetch, e.g. via
proxy) could exceed it → the run is killed mid-page, the cursor is never saved (orphan
thread), and that tube makes zero progress — recurring Sentry GOON-V. Added a 3000s
in-run budget that breaks after a completed page, saves the cursor, and returns cleanly;
the next run continues. budget_hit surfaced in the summary log to spot the slow tube.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 11:21:41 +02:00
goon-foss
ac17350a67 fix(sources): require 25 telemetry attempts before a source can show OFFLINE
freshporno (5★ fresh+rich, verified working — 206/507MB) was labeled OFFLINE off 10
playback attempts that all failed in one unlucky window (a CDN-node blip; it resolves
fine now). 10 was too thin a sample to zero out a known-good source's stars. Raised
the telemetry-trust threshold 10→25; below it we fall back to the proxy/heuristic
health instead of declaring offline (user-report cb526949).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 10:23:29 +02:00
goon-foss
0f1f25393e feat(sources): 0-5★ ranking on Sites (freshness/metadata/plays) + playback telemetry
Rates each source on three axes the user asked for:
- freshness: how recently/often new content arrives (newest age + 7d volume)
- richness: metadata coverage (thumbnail/tags/performers/description/studio/duration)
- plays: does it actually play — from real playback telemetry when available,
  else a proxy from the resolve mechanism. 0★ = offline (gates the overall stars,
  so a fresh+rich source that doesn't play still ranks bottom — the hqfap/4k69 case)

Backend:
- playback_events: fire-and-forget telemetry POST from the app per playback attempt
  (origin + success/error + time-to-first-frame), append-only, 30d retention
- source_stats: per-origin computed scores, refreshed by a scheduler job (6h);
  /sources joins it and sorts by stars
- models + local migration 0025; new GOON_SCHED_SOURCE_STATS_HOURS setting

Mobile:
- Sites rows show ★ rating; tap the stars for a breakdown (axes + metadata %, plus
  whether "plays" is measured or estimated)
- PlayerScreen reports playback success/failure per source (native path only —
  symmetric, conservative); origin threaded through Scene/Movie play callsites

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 10:00:59 +02:00
goon-foss
abfabb6792 feat(scheduler): periodic title+duration dedup (missing-merge tube dupes)
Missing-merge duplicates (same performer + identical normalized title + identical duration-to-the-second) that bulk_dedup misses — tube re-scrapes and cross-tube re-ingests like porn00 pulling a video already present from xnxx (reports 28fe8181/32df33b1). Extracted the proven merge_exact_title_duration logic into app/scheduler/title_duration_dedup.py (script now a thin wrapper), wired a 12h scheduler job (playback-only = what users actually see, GOON_SCHED_TITLE_DEDUP_HOURS). Signal is near-certain (two different videos don't share byte-identical title AND exact duration); no shared performer = not merged (over-match guard). Verified: job registers (jobs=14), backlog currently 0 after the one-shot global merge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:20:48 +02:00
goon-foss
7b37561c49 feat(scheduler): hetzner bandwidth monitor + search-tube watchdog coverage
Two observability additions to the worker scheduler (intertwined in the same files): (1) ingest-watchdog now also covers performer-driven search scrapers (ALL_DIRECT_SCRAPERS) with a separate 7d threshold, not just browse tubes at 48h — several search tubes (perverzija, fpoxxx, porndish, ...) had frozen silently for weeks. (2) New Hetzner Cloud bandwidth monitor (app/scheduler/hetzner_monitor.py): polls outgoing_traffic vs included_traffic and fires a Sentry message at info/warning/error % thresholds with a per-level fingerprint. The config fields existed for ages but the monitor was never implemented. No-op until HETZNER_API_TOKEN + HETZNER_SERVER_ID are set in .env (verified: returns {enabled: False}, job registers as 'hetzner-monitor every 6h', jobs=13).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 09:18:59 +02:00
goon-foss
b08dc03bb9 feat(scheduler): per-origin ingest freshness watchdog -> Sentry
The global source monitor can't catch a single stalled tube because every tube scraper shares one Source row (tube-scraper), so an aggregate run still reports success while one origin freezes (freshporno browsing the rotating KVS homepage root, report 14f3a655). New watchdog checks max(created_at) per active browse-scraper origin (tube:<sitetag>); if a tube with history hasn't produced a new scene in > max_age_hours it fires a Sentry message with a stable per-origin fingerprint (age in extras, not the title, so it stays one grouped issue). Runs every 6h, 48h threshold, both env-tunable (GOON_SCHED_INGEST_WATCHDOG_HOURS / GOON_INGEST_WATCHDOG_MAX_AGE_HOURS). Verified: 0 stale at 48h post-fix, detects neporn at a strict 12h threshold.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:26:25 +02:00
goon-foss
13ba1632c5 feat(scheduler): periodic thumb-asset dedup (hdporn.gg/fullmovies.xxx)
The one-off cleanup merged ~13.5k same-video-different-title dupes, but they regrow as
these sibling tubes re-ingest under new titles. Wire the asset-id+duration merge into
the scheduler (every 12h, GOON_SCHED_THUMB_DEDUP_HOURS, 0=off) so it stays clean.

Shared logic lives in app/scheduler/thumb_dedup.py (run_thumb_asset_dedup); the one-shot
script now imports it. Same tight signature as the cleanup: family hosts only + identical
duration (the bare asset-id number is reused across unrelated CDNs, so cross-host/diff-
duration grouping is excluded). Reports 205b17d9 / 5a2944cb.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 14:56:45 +02:00
goon-foss
53778a248e fix(sxyprn): disable thumbnail refresh job — trafficdeposit token has ~1h TTL
CORRECTION: trafficdeposit thumbnail tokens are hour-bucketed and valid only ~1h
(verified 2026-06-10: stored ts=11:00 dead at 12:27, current ts=13:00 loads). Earlier
"~weekly rot" read was wrong. Storing/periodically-refreshing sxyprn thumbnail URLs
is futile — they expire within the hour. Default the refresh job OFF (kept in code).
The dead-marking sweep (Post Not Found → dead_at) it performed was still valid. Live
sxyprn thumbnails need on-demand resolution at serve time (future work).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-10 14:29:24 +02:00
goon-foss
e702e1893a feat(sxyprn): refresh rotting thumbnails from live post pages + scheduled job
CORRECTION to earlier "unrecoverable" call: the /post/<id> page is alive (200) and
DOES expose the scene's own fresh-signed poster via og:image / <video poster>
(post-id embedded, current timestamp) — only the STORED thumbnail URL had rotted.
Search/listings don't re-surface old posts (0 overlap), but per-post fetch works.

scripts/refresh_sxyprn_thumbs.py: iterate live sxyprn sources, fetch post page,
extract fresh og:image, UPDATE thumbnail_url (verified: refreshed URLs return 200).
_job_refresh_sxyprn_thumbs: every 12h refresh the 1200 least-recently-updated sources
(cycles the ~19k catalog within the expiry window). Pairs with the scene_resolver
overwrite fix so refreshed thumbnails stick.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-10 10:36:30 +02:00
goon-foss
862b2be68e fix(dedup): tighten cross-source candidate prefilter — kill 1800s hang (GOON-V)
_candidate used OR logic (studio OR date±7d OR dur±30s) → 938,950 pairs;
Etap-2 scoring at ~110/s never finished in 1800s → bulk_dedup_performers HUNG
every run, orphan thread leaked until restart. Require AND: same studio plus
(date±2d OR dur±30s). 939k→16k pairs, full run 213s. Real cross-source dup of
one master shares studio + near date/duration; rare studio_id-mismatch pairs
skipped on purpose — a job that COMPLETES beats one that times out merging nothing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 10:03:33 +02:00
goon-foss
b0de2c3d1b feat(dedup): merge exact-phash + same-duration + shared-performer duplicates
bug-report 2026-06-03 ("ten sam czas, ta sama miniaturka, czemu się nie mergują"):
duplicate scenes not merged at ingest. Exact phash alone is noisy here (95% are
collisions on shared thumbnails/intro frames — different scenes; bulk_dedup scorer
correctly gives 0 auto-merge). The safe subset is exact-phash AND same duration
(±3s) AND shared performer/title — near-certain same scene. Same-duration is key:
it excludes the false-merge pattern (short-clip-vs-full has DIFFERING durations).

- scripts/merge_phash_exact_dupes.py: one-off, dry-run by default, per-pair re-fetch
  (handles clusters). Applied: 30 merged.
- bulk_dedup: add `_pairs_exact_phash` (SQL O(N log N), not the O(N²) Hamming scan)
  + strategy "phash_exact" — gated by the normal scorer (surfaces review candidates,
  no risky auto-merge), schedulable for ongoing exact-collision review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 20:08:06 +02:00
goon-foss
d146303434 refactor(ingest): rename scraper Source name "pornapp" -> "tube-scraper"
The umbrella Source.name for all direct tube scrapers (deep-crawl, browse-latest,
performer-driven) was "pornapp" — a misleading leftover from the removed external
porn-app API. It read like a dependency on a third-party "pornapp" service; it is
not — these are our own scrapers hitting 25+ tubes directly (kind=scraper,
origin tube:<sitetag>). Renamed to "tube-scraper" via a single SCRAPER_SOURCE_NAME
constant; DB row renamed in place (UPDATE name, same id) so all ingest_runs +
external_records history stays linked. No behavior change — external_id keying
(sitetag:url) and dedup are unaffected.

NOTE: playback_sources.origin "pornapp:<sitetag>" prefix is a separate legacy
format (resolve_playback parses it) and is intentionally left untouched.

Verified on prod: row renamed (0 stray "pornapp"), new runs land on "tube-scraper".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 16:54:55 +02:00
goon-foss
9e317c1d34 fix(scheduler): bulk_dedup performers cross_source_only + hard-timeout (OOM)
_job_bulk_dedup_performers called run_bulk_dedup(strategy="performers") without
the cross_source_only guard whose docstring exists precisely to prevent this OOM.
At current catalog scale the unguarded path materializes N²/2 pairs per prolific
performer into a list → worker hit 6GB RSS and was OOM-killed every 12h (05:00/
17:00), taking down concurrent tpdb/stashdb/movie ingests as killed_by_restart
(0 new movies). Verified in prod: 05:00 run now completes (885k pairs scored, no
OOM) and ingests succeed (stashdb +241, tpdb +175).

Also wrap in _run_with_timeout like tpdb/stashdb (job had no hard-timeout).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 11:00:19 +02:00
goon-foss
671bd1d5a7 feat(deep-crawl): xvideos browse source (capped) + per-tube page cap
xvideos SSR's JSON-LD VideoObject (duration/title/uploadDate) + on-page /models/ (perf)
+ /tags/. Sample: median ~10.5min, 93% >=3min. Pilot (2 pages): 29 new, 100% playable +
visible + tagged (performers sparse — xvideos 'new' is amateur-heavy; /models/ tagged
mostly on studio rips).

- XVideosBrowseScraper (JSON-LD + page-parse models/tags), in ALL_BROWSE_SCRAPERS.
- deep_crawl._PAGE_CAP: per-sitetag depth cap; xvideoscom=1800 (~newest 50k). At the cap
  the tube is marked exhausted (reset -> incremental re-sweep) so a mega-tube cannot
  monopolize the round-robin or balloon the DB.
- ported yesporn.py into the public repo (was prod-only, like hdporngg) ending the
  __init__ public/prod divergence.

youporn rejected: JSON-LD lacks actor/keywords, its /pornstar//category/ links are A-Z
nav not scene-specific. xhamster: 429/Cloudflare from the VPS IP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 11:16:44 +02:00
goon-foss
c0c75ad3ce feat(deep-crawl): eporner via JSON API as SSR-rich source (Phase 2b alternative)
porntrex/hqporner rejected for deep-crawl: KVS sites with no SSR metadata (77% of
existing porntrex has no duration -> invisible under the app's >=60 filter). eporner
instead exposes a public JSON API (api/v2/video/search) returning title + length_sec
+ keywords + added per video; ~100k videos, ~100/page, no per-scene detail fetch.

- BaseBrowseScraper.crawl_page(page): factored out of latest_scenes; returns None
  (transient fail) / [] (catalog end) / [scenes]. API subclasses override it.
- deep_crawl drives via crawl_page (supports HTML-listing AND API sources).
- EpornerApiScraper: crawl_page hits the eporner API -> RawScene with duration+tags+
  date+thumb+playback; registered in ALL_BROWSE_SCRAPERS.
- Pilot (2 API pages): 192 new, 100% playable + tagged + visible (>=60); the <180s
  trailer filter dropped 6 short clips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 10:37:20 +02:00
goon-foss
86759c47dd feat(scheduler): deep-crawl full tube catalogs (Phase 2a — ingest-all)
We ingested only ~3% of each browse tube's catalog (porndoe >62k scenes; we had 1959)
because tubes were hit only by performer-search + top-N browse. Pilot (porndoe pages
64-110): 1119 new scenes, 100% playable + 100% tagged, 0% canonical overlap (purely
additive — content not in TPDB/StashDB).

- app/scheduler/deep_crawl.py: round-robin over ALL_BROWSE_SCRAPERS, per-tube page cursor
  in app/_state/deepcrawl_state.json (no DB migration), deep-paginate from the cursor,
  idempotent (resolver skips known by raw_hash), mark 'exhausted' at catalog end then
  reset cursors for an incremental re-sweep.
- _job_deep_crawl: hourly, 60 pages/run (~1860 scenes, ~22 min), wrapped in the 1h
  hard-timeout; registered in build_scheduler (jobs=10).
- config: sched_deep_crawl_hours=1, deep_crawl_pages_per_run=60, deepcrawl_state_path.
- scripts/pilot_porndoe_deepcrawl.py: one-off pilot used to validate the approach.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 09:26:44 +02:00
goon-foss
706191d60d fix(scheduler): hard-timeout heavy jobs + periodic stuck-run reaper
At the shared 05:00 anchor all heavy jobs fire together; tpdb/stashdb/performer-driven
had no timeout, so a hung connector blocked the whole job and — with max_instances=1 —
blocked every future fire of that job until a worker restart (incident 2026-06-02: 6 runs
hung 8.7h, movie mirrors 47h stale, tube ingest stalled).

- _run_with_timeout wraps tpdb/stashdb/performer-driven in a 30-min hard cap (same
  ThreadPoolExecutor pattern movie-ingest already uses): on timeout the job returns and
  frees the scheduler slot; the orphaned thread lives until restart.
- _job_reap_stuck: hourly reaper of 'running' >2h rows, registered in the scheduler —
  the startup-only reaper missed hangs while the worker stayed up for hours.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 16:17:50 +02:00
goon-foss
60bb53e444 fix(scenes): propagate playback duration to Scene + duration-consistent counts
Scene.duration_sec was NULL for ~74% of playable scenes (tube duration lives on
playback_source, never propagated to Scene), so the mobile min_duration_sec=60 filter
(Scene.duration_sec >= 60; NULL fails) silently hid them — surfaced as '119 in favorites,
14 after entering the performer' (Safira Yakkuza).

- resolver: _effective_duration() falls back to max live playback_source duration when the
  connector provides no scene-level duration (forward fix, used in create + update).
- scripts/backfill_scene_duration_from_playback.py: one-off idempotent backfill (recovered
  204,014 scenes).
- taxonomy_counts: scene_count now counts playable AND duration_sec >= 60, matching the
  always-60s-filtered scene lists, so favorites/performer/studio/tag badges agree with what
  the scene screen actually shows (Safira: 39 == 39).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 21:31:01 +02:00
goon-foss
a3720494c9 fix(movies): paradisehill delta date-granularity + browse cadence docs
- paradisehill.fetch_movies compared release_date coerced to midnight against the
  `since` timestamp, so the chronological crawl stopped at the first upload dated
  the same calendar day as `since` and silently dropped most new movies (0-2 seen
  per run; Movies tab stalled). Compare by DATE with a 1-day grace instead; idempotent
  external_records upsert dedups the re-fetched recent window.
- scripts/backfill_paradisehill_movies.py: one-off no-delta deep crawl to recover the
  backlog missed during the bug (idempotent, resumable).
- docs: correct stale 'raz dziennie/24h' browse-latest comments to 6h (4x/day), the
  actual configured cadence (config.py sched_browse_latest_hours=6).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 17:00:10 +02:00
goon-foss
fc4060310f feat(ingest): SQL phash match, tag inference + backfill, clip-store skip, browse tubes, watchdog
Resolver/perf:
- find_by_phash_within: nearest match via Postgres bit_count over bit(64) XOR
  instead of Python scan of all phash fingerprints (~20x faster per scene;
  unblocks long delta runs that were killed mid-run before since advanced).

Scheduler/reliability:
- reap ingest_runs stuck in 'running' on worker startup (killed_by_restart).
- smoke_test: per-source ingest health, stuck-run and browse-freshness checks
  -> Sentry; exclude killed_by_restart from the failed-run alarm.

Tags (ingest with tags + fill blanks):
- wire infer_tag_slugs into normalize_scene so tube scenes get title-inferred
  tags (was dead code); union with connector tags.
- scripts/backfill_inferred_tags.py: keyset/batched/idempotent backfill for
  existing tagless scenes (playable tag coverage 16% -> ~52%).

Clip-store:
- skip ManyVids/IWantClips/Clips4Sale/... from canonical sources at ingest
  (GOON_SKIP_CLIP_STORE, default on) — permanent orphans, ~56% of canonical
  ingest, never have a free-tube playback source.

Browse tubes:
- enable fullmovies + hdporn.gg: studio parsed from title prefix instead of
  the /networks/ sidebar (which always yielded the first listed network);
  drop phash compute (pilot: 0% canonical hit within Hamming 5 — auto-screenshots),
  matching relies on title/performer/duration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-01 15:07:35 +02:00
goon-foss
e2e7ab3e22 perf(taxonomy): denormalize scene_count for tags/performers/studios
Counts for /tags, /performers, /studios and /favorites were computed live
per-request by aggregating scene_tags / scene_performers with an EXISTS to
playback_sources. As the catalog grew to ~1.7M scenes (6.3M scene_tags) this
ran ~4.3s for /tags?order=popular (x2 incl. the total count) and ~950ms for
the default /scenes count, making those screens load in several seconds.

- migration 0019: add scene_count (+ DESC index) to tags/performers/studios
- background job _job_refresh_taxonomy_counts (every 3h) recomputes the counts
  in one UPDATE..FROM each (IS DISTINCT FROM to skip unchanged rows)
- /tags, /performers, /studios scenes path now read the column + ORDER BY the
  indexed scene_count; for_movies paths keep live aggregation (small tables)
- favorites read denormalized scene_count instead of a grouped EXISTS aggregate
- /scenes default count: 10-min in-process TTL cache (header is approximate)

Measured: /tags?order=popular&per_page=500 ~8s -> 66ms incl. serialization.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 17:53:48 +02:00
goon-foss
0e829844c8 fix(scheduler): per-connector hard timeout + reorder mangoporn-first
Bug-report 2026-05-30 "ingest znów się zawiesił". streamporn/pandamovies
wieszały się intermittentnie mid-run (zależnie od live-contentu danego dnia),
blokując sekwencyjny _job_movie_ingest → mangoporn (jedyny mirror z realnym
new-content: 72 nowych 05-28) nigdy nie startował. try/except chronił przed
wyjątkiem, NIE przed hangiem.

Fix:
- _job_movie_ingest: każdy connector w ThreadPoolExecutor z future.result
  (timeout=360s). Hang jednego źródła → log + shutdown(wait=False) + kolejka
  leci dalej. Healthy run ~50s, cap 6min = zapas.
- get_movie_connectors: reorder paradisehill, MANGOPORN, streamporn, pandamovies
  — mangoporn zaraz po canonical primary, przed wolniejszymi/wieszającymi się.

Zweryfikowane: pełny _job_movie_ingest przeszedł wszystkie 4 success w nowej
kolejności (mangoporn 2nd, 23s). 33 osierocone "running" rows (worker ubity
mid-run przy deployach) wyczyszczone osobno.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-31 11:19:13 +02:00
goon-foss
d586d45e5b session work: bug-report fixes + WIP cleanup
User-facing bugs resolved (per bug_reports table 2026-05-25):
- 40cd28aa (short-scene filter): mobile api.ts default min_duration_sec=60
  hides 6519 sub-60s scenes across all list endpoints (Performer/Site/Tag/
  Browse). Caller may override with explicit 0.
- 5e89ef7e (porndoe needs cookies/play click): INJECTED_JS in PlayerScreen
  now auto-clicks player-poster overlay (player-poster-play, big-play-button,
  vjs-big-play-button, jw-icon-display, btn-big-play, mejs__overlay-button,
  play-button, btn-play, videoPlayButton). Triggered same interval as
  consent-dismiss + ad-iframe removal.
- b1b5e1a2 (Mixdrop czarny ekran): re-enable mixdrop direct stream via VPS
  curl_cffi proxy (was: skip → WebView fallback → blank screen). Backend
  pipeline (mixdrop.py extract + stream_proxy._curl_cffi_stream with JA3 +
  auto-refetch on token expire) was already complete; just removed the skip
  in app/api/playback.py.

Plus ongoing WIP (paradisehill multi-part extraction, stream_proxy refetch
logic, gesture race fix for long-press 2x speed, anti-adblock INJECTED_JS
defenses, scripts for freshporno backfill, new sources API).
2026-05-25 22:02:52 +02:00
goon-foss
d8c44a0d1e Mobile 0.1.9: OTA enable, WebView cookie-dismiss fix, porndoe connector
Mobile / OTA:
- Enable Expo Updates (app.json + AndroidManifest) → api.goon-foss.org
- Bump 0.1.6 → 0.1.9 (build.gradle, app.json, appVersion.ts, main.py /version)
- backend.ts: default public backend auto-connect (no manual login)

WebView fallback fix (PlayerScreen INJECTED_JS):
- Auto-dismiss cookie/consent gates (hqporner et al. blocked kt_player init)
- Context-scoped: only clicks consent buttons inside cookie/gdpr containers
- Retry window for <source>.src polling raised 5→15 ticks (post-dismiss init)

Resolver:
- Series-position + modifier mismatch detector (Episode 2≠4, BTS/unedited)
  → composite_score hard-reject / cap; wired into scene_score + bulk_dedup
- aggregator-mode candidate query: LIMIT 500 + title-match ordering

Connectors:
- porndoe.com browse scraper (JSON-LD VideoObject) — theporndude audit pilot

landing: APK links → goon-v0.1.9.apk

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 11:20:57 +02:00
goon-foss
ad0284585b Initial commit
Goon — self-hosted aggregator for adult-content scene metadata.

Indexes scenes from TPDB, StashDB, and 30+ public adult tube sites.
Cross-source deduplication via perceptual hash + Levenshtein distance.
FastAPI backend + APScheduler worker + React Native (Expo) mobile client.

FOSS, ad-free, donation-funded. See README for details.
2026-05-20 10:10:22 +02:00