b3e1092175
pornmike.com puts everything in JSON-LD @graph -> ItemPage.mainEntity (name, duration, uploadDate, actor[], genre[], keywords, description, thumbnail), so the parser reads one JSON blob instead of scraping markup. Cast is clean: the whole scene page carries exactly as many /pornstar/ links as there are actors, none of the sidebar pollution that got xxxfiles rejected. 82% of sampled performers already carry a tpdb/stashdb ref and 19-20 of 20 channels are studios we already know. Ingest is gated on a non-empty actor[]: 23% of the catalog has no cast and those scenes could neither be attributed nor deduped. Tags and categories come only from JSON-LD (keywords + genre), never from the HTML, which carries 38 /category/ and 22 /tag/ nav and sidebar links per page. Pagination is ?p=N only: the /N/ form 404s and ?page=N is silently ignored, returning page 1. Playback is the simplest in the portfolio: a plain <source> mp4 on twincdn with no token, no query string and no expiry. Verified 206 on a Range request from the VPS and from another machine in another country, both without a Referer, so it is neither hotlink-guarded nor IP-bound and the phone streams it directly. Two honest caveats recorded in the module docstring: these are 5-12 minute clips (median ~487s against 1800-2400s for the tubes we accepted), so for the ~20% of the catalog from Tushy/Blacked Raw/Milfy/Anilos they will sit as a short shadow next to full canonical scenes; and uploadDate is the tube's import date, not the studio release, so pages past the second are marked backfill. Pilot ingest of 3 pages: 127 seen, 49 merged into existing scenes, 66 new, 0 errors; on page one the gate passed 42 of 62 links with zero scenes missing cast, duration or studio. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
264 lines
14 KiB
Python
264 lines
14 KiB
Python
"""Stream URL extractors per-tube.
|
||
|
||
Public API:
|
||
- `try_extract(sitetag, page_url) -> list[StreamSource] | None`
|
||
- `StreamSource` (dataclass)
|
||
- `HosterDead` (exception)
|
||
- `extract_stream_from_hoster(iframe_url, *, referer)` — generic packer-based hoster extract
|
||
- `fetch_tube_html(url)` — Chrome TLS fingerprint fetch (curl_cffi)
|
||
- `browser_get(url)` — low-level
|
||
|
||
Architektura: każdy tube ma osobny moduł `app.extractors.tubes.<tube>` który eksportuje
|
||
`extract(page_url) -> list[StreamSource] | None`. Registry niżej mapuje sitetag →
|
||
modułowy extractor. `try_extract()` to thin wrapper z exception handlingiem.
|
||
|
||
Po removalu porn-app dependency, ten moduł jest jedynym mechanizmem rozwiązywania
|
||
streamów — playback.py nie wpada już do porn-app /stream API.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
from collections.abc import Callable
|
||
|
||
from app.extractors._fetch import browser_get, fetch_tube_html
|
||
from app.extractors._models import HosterDead, StreamSource, TubePageError
|
||
from app.extractors.hoster import extract_stream_from_hoster, unpack_packer
|
||
from app.extractors.tubes import (
|
||
_embed_iframe,
|
||
_vps_blocked_fallback,
|
||
_ytdlp,
|
||
eporner,
|
||
freshporno,
|
||
fullmovies,
|
||
fullvideosporn,
|
||
galaxyporn,
|
||
hdporngg,
|
||
hqfap,
|
||
hqporner,
|
||
javflix,
|
||
neporn,
|
||
latestpornvideo,
|
||
paradisehill,
|
||
porn00,
|
||
pornbusy,
|
||
pornmike,
|
||
porntrex,
|
||
supjav,
|
||
sxyprn,
|
||
vjav,
|
||
watchporn,
|
||
xhamster,
|
||
yespornvip,
|
||
youperv,
|
||
)
|
||
|
||
log = logging.getLogger(__name__)
|
||
|
||
|
||
# Sitetag → extractor function. Sitetag pasuje do format'u z origin: `pornapp:<sitetag>`
|
||
# (lub po Fazie 2 migracji: `tube:<sitetag>`).
|
||
#
|
||
# Mainstream tubes (pornhub/xvideos/xnxx/xhamster/redtube/youporn/porntrex) używają
|
||
# yt-dlp jako extractor — battle-tested, aktualizowane przez upstream przy zmianach
|
||
# HTML. Aggregator tubes (xmoviesforyou/watchporn/siska/...) używają generic
|
||
# embed-iframe extractor (page → /e/<id> iframe → P.A.C.K.E.R. unpack). Custom kod
|
||
# tylko tam gdzie tube ma niestandardowy schemat (eporner XHR, sxyprn URL transform).
|
||
_REGISTRY: dict[str, Callable[[str], list[StreamSource] | None]] = {
|
||
# hqporner — dedicated extractor zwraca multi-quality `<source>` mp4 URLs
|
||
# (bigcdn.cc / hqwo.cc / flyflv) z `force_proxy=True`. CDN URLs IP-bound do
|
||
# VPS, więc playback.py routuje przez proxy — mobile dostaje quality picker
|
||
# + natywny ExoPlayer, bez WebView.
|
||
# Bug-report e8ddd8d4: WebView fallback (`_vps_blocked_fallback`) ładował
|
||
# hqporner.com scene page w WebView, ale ta strona ma ad-iframes (adtng,
|
||
# goaserv, mavrtracktor) + pop-under-triggery → user klikał i widział
|
||
# reklamę zamiast video. INJECTED_JS w PlayerScreen.tsx nie chwytał
|
||
# popupów dośc szybko. Powrót do natywnego = `<source>` mp4 picker omija
|
||
# tę ścieżkę całkowicie.
|
||
"hqpornercom": hqporner.extract,
|
||
"epornercom": eporner.extract,
|
||
"sxyprncom": sxyprn.extract,
|
||
# Mainstream tubes — yt-dlp
|
||
# NB: 2026-05-18 cross-IP test potwierdził że xvideos/xnxx/pornhub/youporn/redtube
|
||
# CDN URLs są **time-bound** (nie IP-bound) — mobile_direct_ok auto-detect w
|
||
# playback.py daje mobile direct fetch, zero VPS bandwidth.
|
||
# pornhub + redtube — USUNIĘTE CAŁKOWICIE 2026-06-22 (user request). Scrapery były
|
||
# disabled od 2026-05-12 (0.4% canonical match — głównie skrócone amatorskie clipy),
|
||
# zamrożone sceny/źródła skasowane z DB. Brak ekstraktorów → zero resolve.
|
||
"xvideoscom": _ytdlp.extract,
|
||
"xnxxcom": _ytdlp.extract,
|
||
"youporncom": _ytdlp.extract,
|
||
# porntrex KVS — 2026-05-22 VPS znów dociera (HTTP 200). Dedykowany extractor:
|
||
# flashvars `video_url` → `get_file` 302 → CDN time-bound signed URL
|
||
# (`expires`+`md5`, NIE IP-bound) → mobile gra direct, zero VPS bandwidth.
|
||
"porntrexcom": porntrex.extract,
|
||
# fpoxxx — KVS, plain get_file + license. 2026-06-01 (task #20): get_file 302 →
|
||
# `videos3.fpo.xxx/remote_control.php?acctoken=<base64>` — zdekodowany acctoken
|
||
# zawiera WBITY IP serwera-resolvera → definitywnie IP-bound. WebView only.
|
||
"fpoxxx": _vps_blocked_fallback.extract,
|
||
# sxyland — embeduje playmogo.com/e/<id> (= klon DoodStream: doodcdn.io + pass_md5
|
||
# + niewidzialny CF Turnstile; Chrome-DevTools verify 2026-06-08, bug-report 827a50a1).
|
||
# Strona sxyland NIE jest Turnstile-gated (VPS curl wyciąga iframe URL z HTML), więc
|
||
# _embed_iframe wyłuskuje embed playmogo i oddaje jako type='hoster' → mobile
|
||
# doodstream.ts resolvuje phone-side (phone IP przechodzi invisible Turnstile) → direct
|
||
# mp4 → autoplay. Wcześniej _vps_blocked_fallback ładował CAŁĄ stronę sxyland w WebView
|
||
# (ads + klik-to-play + brak autoplay = dokładnie objaw z reportu 827a50a1).
|
||
"sxylandcom": _embed_iframe.extract,
|
||
# Aggregator tubes — generic embed-iframe → hoster unpacker
|
||
"latestpornvideocom": latestpornvideo.extract,
|
||
"xmoviesforyoucom": _embed_iframe.extract,
|
||
# watchporn — 2026-07-02 przebudowany na KVS (DoodStream-CAPTCHA zniknął).
|
||
# flashvars get_file direct mp4, same-session 302 resolve, token nie IP-bound.
|
||
"watchporn": watchporn.extract,
|
||
# youperv — fluidplayer, zwykły <source> mp4 na files.klubnichka-hd.com. Bez tokena
|
||
# i bez IP-bindingu; CDN pilnuje tylko Referera (cross-IP 206 z VPS) → mobile direct.
|
||
"youpervcom": youperv.extract,
|
||
# galaxyporn — iframe rodziny seekplayer (upns/4meplayer/seekplays); silnik już mamy,
|
||
# extractor tylko wyłuskuje iframe. HLS Referer+IP-bound → /proxy/hls.
|
||
"galaxyporncom": galaxyporn.extract,
|
||
# pornbusy — dispatch po embedURL: loadvid (POST-manifest przez /proxy/hls),
|
||
# rodzina seekplayer (istniejący silnik), zpi.cx (embedURL to gotowy plik).
|
||
# upload18 + ogon → None (token IP-bound, resolve wymusiłby proxy całego wideo).
|
||
"pornbusycom": pornbusy.extract,
|
||
# pornmike — gołe <source> mp4 na twincdn: bez tokenu, bez Referera, bez expiry.
|
||
# Cross-IP 206 z VPS i z innego kraju → mobile gra direct, zero proxy/WebView.
|
||
"pornmike": pornmike.extract,
|
||
# fullvideosporn (= sextu) — TXXX: videofile.php → dekod → get_file → 302 → znvcdn.
|
||
# Token CDN portable cross-IP → mobile direct. UWAGA: VPS dostaje od CDN 429
|
||
# (reputacja IP datacenter), więc health-check playbacku z VPS jest ślepy.
|
||
"fullvideosporn": fullvideosporn.extract,
|
||
"siskavideo": _embed_iframe.extract,
|
||
"porn4dayspw": _embed_iframe.extract,
|
||
"porndishcom": _embed_iframe.extract,
|
||
# xxxfreewatch — DELISTED 2026-05-18. 790 solo-orphan scen, 0% match, CF-walled z VPS.
|
||
"latestleaksco": _embed_iframe.extract,
|
||
"mypornerleakcom": _embed_iframe.extract,
|
||
# xhamster — 2026-06-08 PRZEPIĘTE z _vps_blocked_fallback na natywny server-side HLS.
|
||
# Re-test (DevTools + cross-IP): VPS pobiera scene page bez CF challenge, master m3u8
|
||
# w SSR HTML, manifest+segmenty time-bound (portable, nie IP-bound). Mobile gra HLS
|
||
# direct, multi-quality, zero VPS proxy/WebView/reklam. Patrz tubes/xhamster.py.
|
||
# ~155k solo-scen upgrade z WebView-z-reklamami na natywne. Wcześniej WebView fallback
|
||
# ładował ad-heavy stronę z phone IP (działało, ale gorszy UX + preroll VAST).
|
||
"xhamstercom": xhamster.extract,
|
||
# Freshporno KVS (function/0 + license). 2026-06-04 DevTools + cross-IP re-test
|
||
# NAPRAWIA błąd z #20: finalny cdn4.freshporno.org/remote_control.php jest PORTABLE
|
||
# (token time-bound nie IP-bound — VPS odtworzył token z residential → 206) ale
|
||
# wymaga browser-TLS (curl_cffi chrome/ExoPlayer → 206; plain curl → 000). W #20
|
||
# testowałem plain-curl-em poza sesją → 000 → błędnie „nieosiągalny" → WebView.
|
||
# Teraz backend-resolve jak yespornvip/pornditt (_kvs używa curl_cffi chrome).
|
||
# Native, multi-quality, zero proxy/WebView. (zweryfikowane na emulatorze przed deploy)
|
||
"freshpornoorg": freshporno.extract,
|
||
# porn00 — KVS (plain get_file + license). 2026-06-04 DevTools + cross-IP re-test
|
||
# NAPRAWIA błąd z #20: finalny fe.porn00.org/...?token=&expires= jest PORTABLE
|
||
# (token time-bound nie IP-bound — Bright Data residential proxy z innego IP → 206)
|
||
# ale wymaga browser-TLS (curl_cffi chrome → 206; plain curl → 403). W #20
|
||
# testowałem finalny URL plain-curl-em → 403 → błędnie „IP-bound" → WebView.
|
||
# Teraz backend-resolve przez _kvs (curl_cffi chrome), native multi-quality,
|
||
# ZERO proxy (wcześniej force_proxy łamał no-proxy). Same mechanizm co freshporno.
|
||
"porn00org": porn00.extract,
|
||
# pornxp — `<source> //sr.porn-xp.com/<token>/.../720.mp4` (redirect → xpxp.eu).
|
||
# 2026-06-01 (task #20): 403 cross-IP → token w path IP-bound. WebView only.
|
||
"pornxpph": _vps_blocked_fallback.extract,
|
||
# yesporn.vip — KVS engine. VPS znów dociera (HTTP 200, odblokowane jak porntrex),
|
||
# więc resolvujemy SERVER-SIDE: dekoduj flashvars `video_url`/alt/alt2 (function/0/ +
|
||
# license_code, algo kt_player) → follow get_file 302 → portable cdn5 url (time-bound,
|
||
# NIE IP/cookie-bound, zweryfikowane cross-IP 2026-05-31). Mobile gra direct natywnie,
|
||
# multi-quality, ZERO WebView/reklam/preroll. Wcześniej WebView fallback pokazywał
|
||
# ad-heavy stronę a scrape łapał preroll-reklamę (bkcdn) zamiast wideo.
|
||
"yespornvip": yespornvip.extract,
|
||
# Direct-scraping tubes (mają też search scraper w connectors/direct_scrapers/)
|
||
# — używają identycznego embed-iframe pattern dla streamingu.
|
||
# hdporn92com — DELISTED 2026-05-18. Scene pages to SEO shell bez player iframe,
|
||
# JS hijackuje kliki na popunder. Wszystkie playback_sources mass-marked dead.
|
||
# 0dayxx + pornditt + pornhat — USUNIĘTE CAŁKOWICIE 2026-06-22 (user request): orphan
|
||
# factories (0–0.2% canonical match), zastępujemy lepszymi źródłami. Dane skasowane.
|
||
# CF-protected tube — curl_cffi w fetch_tube_html bypassa JA3, embed-iframe pattern.
|
||
"perverzijacom": _embed_iframe.extract,
|
||
# Special: WebView-only (Yii2 session-bound player).
|
||
"paradisehillcc": paradisehill.extract,
|
||
# PornDoe — dołączony 2026-05-21 (theporndude audit). Stream URL nie inline w
|
||
# SSR HTML (player JS init po Play click), więc WebView fallback: mobile pobiera
|
||
# /watch/<id> z phone IP, player JS dekoduje video.src, INJECTED_JS scrape.
|
||
# 0 VPS bandwidth — zgodne z pre-public bandwidth/anonimowość priorytet.
|
||
"porndoecom": _vps_blocked_fallback.extract,
|
||
# fullmovies.xxx + hdporn.gg — BRAKOWAŁO extractora (try_extract→None→"no stream";
|
||
# fullmovies.xxx + hdporn.gg — ta sama platforma (`<source>/get_file/8512/`).
|
||
# 2026-06-04 (DevTools + cross-IP, naprawia „loading forever" + bug 19866e9e):
|
||
# get_file binduje fpvcdn do IP FETCHERA + jest stateless + ważny ≥90s, więc
|
||
# oddajemy get_file NIEZRESOLWOWANY (mobile_direct) — telefon follow-uje 302 →
|
||
# fpvcdn z IP telefonu → gra. POMIJAMY 4K (time-out 30s na fpvcdn = przyczyna
|
||
# „loading forever"; 720/480p gra ~1s). Native, multi-quality, ZERO proxy/WebView.
|
||
# (#19866e9e wcześniej źle: założyłem „get_file 403 IP-bound" testem plain-curl.)
|
||
"fullmoviesxxx": fullmovies.extract,
|
||
"hdporngg": hdporngg.extract,
|
||
# hqfap (PlayTube CMS) — JSON-LD contentUrl = direct mp4. USUNIĘTY 2026-06-25 (cała
|
||
# biblioteka serwowała `/upload/videos/video_down.mp4` stub), PRZYWRÓCONY 2026-07-07:
|
||
# strona wróciła na CDN vstor.top z realnymi plikami (portable cross-IP, zweryfikowane),
|
||
# user request. 4k69 zostaje usunięty (nie sprawdzany ponownie).
|
||
"hqfapcom": hqfap.extract,
|
||
# javflix (JAV, WordPress) — przyciski serwerów to `<a class="myLink" href="<embed>">`
|
||
# (streamtape/voe/doodstream/emturbovid). Generyczny _embed_iframe łapie je anchor-hoster
|
||
# patternem → type='hoster', telefon resolwuje (voe backend, dood/filemoon phone-side).
|
||
# Wrapper javflix.extract odsiewa placeholder players.mp4.
|
||
"javflix": javflix.extract,
|
||
# vjav (JAV, TXXX network) — videofile.php -> get_file HLS (patrz tubes/vjav.py).
|
||
"vjav": vjav.extract,
|
||
# supjav (JAV, embed-aggregator) — data-link reverse-hex -> lk1 -> hoster (type=hoster).
|
||
"supjav": supjav.extract,
|
||
# neporn — KVS function/0 + license (jak freshporno). Server-side _kvs resolve →
|
||
# data001.neporn.com/remote_control.php portable (cross-IP 206, 2026-06-10).
|
||
"neporncom": neporn.extract,
|
||
# superporn — `<source>` mp4 (cdnst*.superporn.com) token IP-bound do fetchera
|
||
# (403 cross-IP, test 2026-06-10), a sama strona CF-blocked z VPS. Resolve MUSI
|
||
# być phone-side: WebView ładuje stronę z residential IP telefonu, INJECTED_JS
|
||
# bierze video.src. Ingest HTML idzie osobno przez Bright Data proxy (scraper).
|
||
"superporncom": _vps_blocked_fallback.extract,
|
||
}
|
||
|
||
|
||
def try_extract(sitetag: str, page_url: str) -> list[StreamSource] | None:
|
||
"""Próbuje rozwiązać stream URL dla danego tube'a + page_url.
|
||
|
||
Zwraca listę StreamSource (różne quality/kontener) lub None gdy:
|
||
- brak extractora dla tego sitetag
|
||
- extractor zwrócił None / nie znalazł URL'a
|
||
|
||
Raises HosterDead gdy embed page wprost mówi że video deleted/not found —
|
||
caller (playback.py) łapie i oznacza playback_source.dead_at.
|
||
"""
|
||
extractor = _REGISTRY.get(sitetag)
|
||
if extractor is None:
|
||
return None
|
||
try:
|
||
return extractor(page_url)
|
||
except (HosterDead, TubePageError):
|
||
raise
|
||
except Exception as e:
|
||
log.warning("extractor for %s failed on %s: %s", sitetag, page_url, e)
|
||
return None
|
||
|
||
|
||
def supported_sitetags() -> tuple[str, ...]:
|
||
"""Zwraca listę sitetag-ów które mają zarejestrowany extractor."""
|
||
return tuple(_REGISTRY.keys())
|
||
|
||
|
||
def is_vps_blocked_fallback(sitetag: str) -> bool:
|
||
"""True gdy sitetag resolvuje się TYLKO przez WebView fallback (IP-bound CDN /
|
||
ad-heavy / CAPTCHA — np. fpoxxx, pornxpph). Takie źródła dają gorszy
|
||
UX (reklamy, czarny ekran) niż natywny KVS/direct resolve, więc UI powinien je
|
||
rankować NIŻEJ gdy scena ma też natywne źródło (bug-report 2026-06-07: scena
|
||
pokazywała fpoxxx-WebView przed działającym freshporno bo sort był alfabetyczny)."""
|
||
return _REGISTRY.get(sitetag) is _vps_blocked_fallback.extract
|
||
|
||
|
||
__all__ = [
|
||
"try_extract",
|
||
"supported_sitetags",
|
||
"StreamSource",
|
||
"HosterDead",
|
||
"TubePageError",
|
||
"extract_stream_from_hoster",
|
||
"unpack_packer",
|
||
"fetch_tube_html",
|
||
"browser_get",
|
||
]
|