2bf9179467
galaxyporn.net carries paysite rips with a dedicated <div id="video-actors"> cast block (no sidebar pollution), studio + date in the title prefix, healthy /page/N/ pagination. Measured orphan risk is low: 92% of a 75-performer sample already have a tpdb/stashdb ref in our DB and 79% of titles match scenes we already hold. seekplayer_engine host regex widened to cover seekplays|4meplayer|ezplayer| seeks|upn and the pro|cloud|one TLDs. This is the same engine (identical AES key/IV and /api/v1/video endpoint), just newer domains; the whitelist is additive so existing hosts keep matching (verified, including that upns.evil.com is still rejected). Unlocks all four galaxyporn player hosts and the same family on other sites. Two things the extractor had to get right: - Call seekplayer_engine directly rather than via extract_stream_from_hoster: the wrapper verifies the resulting URL and the hotlink-guarded HLS 403s, so it discarded correctly decoded streams. - Referer must be the PLAYER origin, not galaxyporn.net. With the site referer the manifest 403s; with the player origin it returns 200. Verified end to end: manifest 200 -> variant 200 (570 segments) -> segment 206. Duration is absent from the HTML, JSON-LD and player payload, and a NULL duration would silently hide scenes behind the min_duration_sec filter (the porntrex incident), so it is computed from the player's thumbnail.vtt last cue. The player API throttles bursts, hence the retry/backoff: that took missing durations from 9/21 down to 3/21. The value runs ~2% short of the true length (one thumbnail interval), which beats having none. Pilot: 21 scenes/page with studio 21/21 and clean cast (max 3), ingest of 2 pages = 42 seen, 34 new, 8 merged, 0 errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
250 lines
14 KiB
Python
250 lines
14 KiB
Python
"""Stream URL extractors per-tube.
|
||
|
||
Public API:
|
||
- `try_extract(sitetag, page_url) -> list[StreamSource] | None`
|
||
- `StreamSource` (dataclass)
|
||
- `HosterDead` (exception)
|
||
- `extract_stream_from_hoster(iframe_url, *, referer)` — generic packer-based hoster extract
|
||
- `fetch_tube_html(url)` — Chrome TLS fingerprint fetch (curl_cffi)
|
||
- `browser_get(url)` — low-level
|
||
|
||
Architektura: każdy tube ma osobny moduł `app.extractors.tubes.<tube>` który eksportuje
|
||
`extract(page_url) -> list[StreamSource] | None`. Registry niżej mapuje sitetag →
|
||
modułowy extractor. `try_extract()` to thin wrapper z exception handlingiem.
|
||
|
||
Po removalu porn-app dependency, ten moduł jest jedynym mechanizmem rozwiązywania
|
||
streamów — playback.py nie wpada już do porn-app /stream API.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
from collections.abc import Callable
|
||
|
||
from app.extractors._fetch import browser_get, fetch_tube_html
|
||
from app.extractors._models import HosterDead, StreamSource, TubePageError
|
||
from app.extractors.hoster import extract_stream_from_hoster, unpack_packer
|
||
from app.extractors.tubes import (
|
||
_embed_iframe,
|
||
_vps_blocked_fallback,
|
||
_ytdlp,
|
||
eporner,
|
||
freshporno,
|
||
fullmovies,
|
||
galaxyporn,
|
||
hdporngg,
|
||
hqfap,
|
||
hqporner,
|
||
javflix,
|
||
neporn,
|
||
latestpornvideo,
|
||
paradisehill,
|
||
porn00,
|
||
porntrex,
|
||
supjav,
|
||
sxyprn,
|
||
vjav,
|
||
watchporn,
|
||
xhamster,
|
||
yespornvip,
|
||
youperv,
|
||
)
|
||
|
||
log = logging.getLogger(__name__)
|
||
|
||
|
||
# Sitetag → extractor function. Sitetag pasuje do format'u z origin: `pornapp:<sitetag>`
|
||
# (lub po Fazie 2 migracji: `tube:<sitetag>`).
|
||
#
|
||
# Mainstream tubes (pornhub/xvideos/xnxx/xhamster/redtube/youporn/porntrex) używają
|
||
# yt-dlp jako extractor — battle-tested, aktualizowane przez upstream przy zmianach
|
||
# HTML. Aggregator tubes (xmoviesforyou/watchporn/siska/...) używają generic
|
||
# embed-iframe extractor (page → /e/<id> iframe → P.A.C.K.E.R. unpack). Custom kod
|
||
# tylko tam gdzie tube ma niestandardowy schemat (eporner XHR, sxyprn URL transform).
|
||
_REGISTRY: dict[str, Callable[[str], list[StreamSource] | None]] = {
|
||
# hqporner — dedicated extractor zwraca multi-quality `<source>` mp4 URLs
|
||
# (bigcdn.cc / hqwo.cc / flyflv) z `force_proxy=True`. CDN URLs IP-bound do
|
||
# VPS, więc playback.py routuje przez proxy — mobile dostaje quality picker
|
||
# + natywny ExoPlayer, bez WebView.
|
||
# Bug-report e8ddd8d4: WebView fallback (`_vps_blocked_fallback`) ładował
|
||
# hqporner.com scene page w WebView, ale ta strona ma ad-iframes (adtng,
|
||
# goaserv, mavrtracktor) + pop-under-triggery → user klikał i widział
|
||
# reklamę zamiast video. INJECTED_JS w PlayerScreen.tsx nie chwytał
|
||
# popupów dośc szybko. Powrót do natywnego = `<source>` mp4 picker omija
|
||
# tę ścieżkę całkowicie.
|
||
"hqpornercom": hqporner.extract,
|
||
"epornercom": eporner.extract,
|
||
"sxyprncom": sxyprn.extract,
|
||
# Mainstream tubes — yt-dlp
|
||
# NB: 2026-05-18 cross-IP test potwierdził że xvideos/xnxx/pornhub/youporn/redtube
|
||
# CDN URLs są **time-bound** (nie IP-bound) — mobile_direct_ok auto-detect w
|
||
# playback.py daje mobile direct fetch, zero VPS bandwidth.
|
||
# pornhub + redtube — USUNIĘTE CAŁKOWICIE 2026-06-22 (user request). Scrapery były
|
||
# disabled od 2026-05-12 (0.4% canonical match — głównie skrócone amatorskie clipy),
|
||
# zamrożone sceny/źródła skasowane z DB. Brak ekstraktorów → zero resolve.
|
||
"xvideoscom": _ytdlp.extract,
|
||
"xnxxcom": _ytdlp.extract,
|
||
"youporncom": _ytdlp.extract,
|
||
# porntrex KVS — 2026-05-22 VPS znów dociera (HTTP 200). Dedykowany extractor:
|
||
# flashvars `video_url` → `get_file` 302 → CDN time-bound signed URL
|
||
# (`expires`+`md5`, NIE IP-bound) → mobile gra direct, zero VPS bandwidth.
|
||
"porntrexcom": porntrex.extract,
|
||
# fpoxxx — KVS, plain get_file + license. 2026-06-01 (task #20): get_file 302 →
|
||
# `videos3.fpo.xxx/remote_control.php?acctoken=<base64>` — zdekodowany acctoken
|
||
# zawiera WBITY IP serwera-resolvera → definitywnie IP-bound. WebView only.
|
||
"fpoxxx": _vps_blocked_fallback.extract,
|
||
# sxyland — embeduje playmogo.com/e/<id> (= klon DoodStream: doodcdn.io + pass_md5
|
||
# + niewidzialny CF Turnstile; Chrome-DevTools verify 2026-06-08, bug-report 827a50a1).
|
||
# Strona sxyland NIE jest Turnstile-gated (VPS curl wyciąga iframe URL z HTML), więc
|
||
# _embed_iframe wyłuskuje embed playmogo i oddaje jako type='hoster' → mobile
|
||
# doodstream.ts resolvuje phone-side (phone IP przechodzi invisible Turnstile) → direct
|
||
# mp4 → autoplay. Wcześniej _vps_blocked_fallback ładował CAŁĄ stronę sxyland w WebView
|
||
# (ads + klik-to-play + brak autoplay = dokładnie objaw z reportu 827a50a1).
|
||
"sxylandcom": _embed_iframe.extract,
|
||
# Aggregator tubes — generic embed-iframe → hoster unpacker
|
||
"latestpornvideocom": latestpornvideo.extract,
|
||
"xmoviesforyoucom": _embed_iframe.extract,
|
||
# watchporn — 2026-07-02 przebudowany na KVS (DoodStream-CAPTCHA zniknął).
|
||
# flashvars get_file direct mp4, same-session 302 resolve, token nie IP-bound.
|
||
"watchporn": watchporn.extract,
|
||
# youperv — fluidplayer, zwykły <source> mp4 na files.klubnichka-hd.com. Bez tokena
|
||
# i bez IP-bindingu; CDN pilnuje tylko Referera (cross-IP 206 z VPS) → mobile direct.
|
||
"youpervcom": youperv.extract,
|
||
# galaxyporn — iframe rodziny seekplayer (upns/4meplayer/seekplays); silnik już mamy,
|
||
# extractor tylko wyłuskuje iframe. HLS Referer+IP-bound → /proxy/hls.
|
||
"galaxyporncom": galaxyporn.extract,
|
||
"siskavideo": _embed_iframe.extract,
|
||
"porn4dayspw": _embed_iframe.extract,
|
||
"porndishcom": _embed_iframe.extract,
|
||
# xxxfreewatch — DELISTED 2026-05-18. 790 solo-orphan scen, 0% match, CF-walled z VPS.
|
||
"latestleaksco": _embed_iframe.extract,
|
||
"mypornerleakcom": _embed_iframe.extract,
|
||
# xhamster — 2026-06-08 PRZEPIĘTE z _vps_blocked_fallback na natywny server-side HLS.
|
||
# Re-test (DevTools + cross-IP): VPS pobiera scene page bez CF challenge, master m3u8
|
||
# w SSR HTML, manifest+segmenty time-bound (portable, nie IP-bound). Mobile gra HLS
|
||
# direct, multi-quality, zero VPS proxy/WebView/reklam. Patrz tubes/xhamster.py.
|
||
# ~155k solo-scen upgrade z WebView-z-reklamami na natywne. Wcześniej WebView fallback
|
||
# ładował ad-heavy stronę z phone IP (działało, ale gorszy UX + preroll VAST).
|
||
"xhamstercom": xhamster.extract,
|
||
# Freshporno KVS (function/0 + license). 2026-06-04 DevTools + cross-IP re-test
|
||
# NAPRAWIA błąd z #20: finalny cdn4.freshporno.org/remote_control.php jest PORTABLE
|
||
# (token time-bound nie IP-bound — VPS odtworzył token z residential → 206) ale
|
||
# wymaga browser-TLS (curl_cffi chrome/ExoPlayer → 206; plain curl → 000). W #20
|
||
# testowałem plain-curl-em poza sesją → 000 → błędnie „nieosiągalny" → WebView.
|
||
# Teraz backend-resolve jak yespornvip/pornditt (_kvs używa curl_cffi chrome).
|
||
# Native, multi-quality, zero proxy/WebView. (zweryfikowane na emulatorze przed deploy)
|
||
"freshpornoorg": freshporno.extract,
|
||
# porn00 — KVS (plain get_file + license). 2026-06-04 DevTools + cross-IP re-test
|
||
# NAPRAWIA błąd z #20: finalny fe.porn00.org/...?token=&expires= jest PORTABLE
|
||
# (token time-bound nie IP-bound — Bright Data residential proxy z innego IP → 206)
|
||
# ale wymaga browser-TLS (curl_cffi chrome → 206; plain curl → 403). W #20
|
||
# testowałem finalny URL plain-curl-em → 403 → błędnie „IP-bound" → WebView.
|
||
# Teraz backend-resolve przez _kvs (curl_cffi chrome), native multi-quality,
|
||
# ZERO proxy (wcześniej force_proxy łamał no-proxy). Same mechanizm co freshporno.
|
||
"porn00org": porn00.extract,
|
||
# pornxp — `<source> //sr.porn-xp.com/<token>/.../720.mp4` (redirect → xpxp.eu).
|
||
# 2026-06-01 (task #20): 403 cross-IP → token w path IP-bound. WebView only.
|
||
"pornxpph": _vps_blocked_fallback.extract,
|
||
# yesporn.vip — KVS engine. VPS znów dociera (HTTP 200, odblokowane jak porntrex),
|
||
# więc resolvujemy SERVER-SIDE: dekoduj flashvars `video_url`/alt/alt2 (function/0/ +
|
||
# license_code, algo kt_player) → follow get_file 302 → portable cdn5 url (time-bound,
|
||
# NIE IP/cookie-bound, zweryfikowane cross-IP 2026-05-31). Mobile gra direct natywnie,
|
||
# multi-quality, ZERO WebView/reklam/preroll. Wcześniej WebView fallback pokazywał
|
||
# ad-heavy stronę a scrape łapał preroll-reklamę (bkcdn) zamiast wideo.
|
||
"yespornvip": yespornvip.extract,
|
||
# Direct-scraping tubes (mają też search scraper w connectors/direct_scrapers/)
|
||
# — używają identycznego embed-iframe pattern dla streamingu.
|
||
# hdporn92com — DELISTED 2026-05-18. Scene pages to SEO shell bez player iframe,
|
||
# JS hijackuje kliki na popunder. Wszystkie playback_sources mass-marked dead.
|
||
# 0dayxx + pornditt + pornhat — USUNIĘTE CAŁKOWICIE 2026-06-22 (user request): orphan
|
||
# factories (0–0.2% canonical match), zastępujemy lepszymi źródłami. Dane skasowane.
|
||
# CF-protected tube — curl_cffi w fetch_tube_html bypassa JA3, embed-iframe pattern.
|
||
"perverzijacom": _embed_iframe.extract,
|
||
# Special: WebView-only (Yii2 session-bound player).
|
||
"paradisehillcc": paradisehill.extract,
|
||
# PornDoe — dołączony 2026-05-21 (theporndude audit). Stream URL nie inline w
|
||
# SSR HTML (player JS init po Play click), więc WebView fallback: mobile pobiera
|
||
# /watch/<id> z phone IP, player JS dekoduje video.src, INJECTED_JS scrape.
|
||
# 0 VPS bandwidth — zgodne z pre-public bandwidth/anonimowość priorytet.
|
||
"porndoecom": _vps_blocked_fallback.extract,
|
||
# fullmovies.xxx + hdporn.gg — BRAKOWAŁO extractora (try_extract→None→"no stream";
|
||
# fullmovies.xxx + hdporn.gg — ta sama platforma (`<source>/get_file/8512/`).
|
||
# 2026-06-04 (DevTools + cross-IP, naprawia „loading forever" + bug 19866e9e):
|
||
# get_file binduje fpvcdn do IP FETCHERA + jest stateless + ważny ≥90s, więc
|
||
# oddajemy get_file NIEZRESOLWOWANY (mobile_direct) — telefon follow-uje 302 →
|
||
# fpvcdn z IP telefonu → gra. POMIJAMY 4K (time-out 30s na fpvcdn = przyczyna
|
||
# „loading forever"; 720/480p gra ~1s). Native, multi-quality, ZERO proxy/WebView.
|
||
# (#19866e9e wcześniej źle: założyłem „get_file 403 IP-bound" testem plain-curl.)
|
||
"fullmoviesxxx": fullmovies.extract,
|
||
"hdporngg": hdporngg.extract,
|
||
# hqfap (PlayTube CMS) — JSON-LD contentUrl = direct mp4. USUNIĘTY 2026-06-25 (cała
|
||
# biblioteka serwowała `/upload/videos/video_down.mp4` stub), PRZYWRÓCONY 2026-07-07:
|
||
# strona wróciła na CDN vstor.top z realnymi plikami (portable cross-IP, zweryfikowane),
|
||
# user request. 4k69 zostaje usunięty (nie sprawdzany ponownie).
|
||
"hqfapcom": hqfap.extract,
|
||
# javflix (JAV, WordPress) — przyciski serwerów to `<a class="myLink" href="<embed>">`
|
||
# (streamtape/voe/doodstream/emturbovid). Generyczny _embed_iframe łapie je anchor-hoster
|
||
# patternem → type='hoster', telefon resolwuje (voe backend, dood/filemoon phone-side).
|
||
# Wrapper javflix.extract odsiewa placeholder players.mp4.
|
||
"javflix": javflix.extract,
|
||
# vjav (JAV, TXXX network) — videofile.php -> get_file HLS (patrz tubes/vjav.py).
|
||
"vjav": vjav.extract,
|
||
# supjav (JAV, embed-aggregator) — data-link reverse-hex -> lk1 -> hoster (type=hoster).
|
||
"supjav": supjav.extract,
|
||
# neporn — KVS function/0 + license (jak freshporno). Server-side _kvs resolve →
|
||
# data001.neporn.com/remote_control.php portable (cross-IP 206, 2026-06-10).
|
||
"neporncom": neporn.extract,
|
||
# superporn — `<source>` mp4 (cdnst*.superporn.com) token IP-bound do fetchera
|
||
# (403 cross-IP, test 2026-06-10), a sama strona CF-blocked z VPS. Resolve MUSI
|
||
# być phone-side: WebView ładuje stronę z residential IP telefonu, INJECTED_JS
|
||
# bierze video.src. Ingest HTML idzie osobno przez Bright Data proxy (scraper).
|
||
"superporncom": _vps_blocked_fallback.extract,
|
||
}
|
||
|
||
|
||
def try_extract(sitetag: str, page_url: str) -> list[StreamSource] | None:
|
||
"""Próbuje rozwiązać stream URL dla danego tube'a + page_url.
|
||
|
||
Zwraca listę StreamSource (różne quality/kontener) lub None gdy:
|
||
- brak extractora dla tego sitetag
|
||
- extractor zwrócił None / nie znalazł URL'a
|
||
|
||
Raises HosterDead gdy embed page wprost mówi że video deleted/not found —
|
||
caller (playback.py) łapie i oznacza playback_source.dead_at.
|
||
"""
|
||
extractor = _REGISTRY.get(sitetag)
|
||
if extractor is None:
|
||
return None
|
||
try:
|
||
return extractor(page_url)
|
||
except (HosterDead, TubePageError):
|
||
raise
|
||
except Exception as e:
|
||
log.warning("extractor for %s failed on %s: %s", sitetag, page_url, e)
|
||
return None
|
||
|
||
|
||
def supported_sitetags() -> tuple[str, ...]:
|
||
"""Zwraca listę sitetag-ów które mają zarejestrowany extractor."""
|
||
return tuple(_REGISTRY.keys())
|
||
|
||
|
||
def is_vps_blocked_fallback(sitetag: str) -> bool:
|
||
"""True gdy sitetag resolvuje się TYLKO przez WebView fallback (IP-bound CDN /
|
||
ad-heavy / CAPTCHA — np. fpoxxx, pornxpph). Takie źródła dają gorszy
|
||
UX (reklamy, czarny ekran) niż natywny KVS/direct resolve, więc UI powinien je
|
||
rankować NIŻEJ gdy scena ma też natywne źródło (bug-report 2026-06-07: scena
|
||
pokazywała fpoxxx-WebView przed działającym freshporno bo sort był alfabetyczny)."""
|
||
return _REGISTRY.get(sitetag) is _vps_blocked_fallback.extract
|
||
|
||
|
||
__all__ = [
|
||
"try_extract",
|
||
"supported_sitetags",
|
||
"StreamSource",
|
||
"HosterDead",
|
||
"TubePageError",
|
||
"extract_stream_from_hoster",
|
||
"unpack_packer",
|
||
"fetch_tube_html",
|
||
"browser_get",
|
||
]
|