feat(galaxyporn): browse scraper + extractor; widen seekplayer host regex
galaxyporn.net carries paysite rips with a dedicated <div id="video-actors"> cast block (no sidebar pollution), studio + date in the title prefix, healthy /page/N/ pagination. Measured orphan risk is low: 92% of a 75-performer sample already have a tpdb/stashdb ref in our DB and 79% of titles match scenes we already hold. seekplayer_engine host regex widened to cover seekplays|4meplayer|ezplayer| seeks|upn and the pro|cloud|one TLDs. This is the same engine (identical AES key/IV and /api/v1/video endpoint), just newer domains; the whitelist is additive so existing hosts keep matching (verified, including that upns.evil.com is still rejected). Unlocks all four galaxyporn player hosts and the same family on other sites. Two things the extractor had to get right: - Call seekplayer_engine directly rather than via extract_stream_from_hoster: the wrapper verifies the resulting URL and the hotlink-guarded HLS 403s, so it discarded correctly decoded streams. - Referer must be the PLAYER origin, not galaxyporn.net. With the site referer the manifest 403s; with the player origin it returns 200. Verified end to end: manifest 200 -> variant 200 (570 segments) -> segment 206. Duration is absent from the HTML, JSON-LD and player payload, and a NULL duration would silently hide scenes behind the min_duration_sec filter (the porntrex incident), so it is computed from the player's thumbnail.vtt last cue. The player API throttles bursts, hence the retry/backoff: that took missing durations from 9/21 down to 3/21. The value runs ~2% short of the true length (one thumbnail interval), which beats having none. Pilot: 21 scenes/page with studio 21/21 and clean cast (max 3), ingest of 2 pages = 42 seen, 34 new, 8 merged, 0 errors. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
goon-foss
parent
5a0b62c3e4
commit
2bf9179467
5 changed files with 321 additions and 3 deletions
|
|
@ -43,6 +43,7 @@ from app.connectors.direct_scrapers.youporn_browse import YouPornBrowseScraper
|
|||
from app.connectors.direct_scrapers.siska import SiskaScraper
|
||||
from app.connectors.direct_scrapers.sxyland import SxyLandScraper
|
||||
from app.connectors.direct_scrapers.sxyprn import SxyPrnScraper
|
||||
from app.connectors.direct_scrapers.galaxyporn import GalaxyPornScraper
|
||||
from app.connectors.direct_scrapers.watchporn import WatchPornScraper
|
||||
from app.connectors.direct_scrapers.youperv import YoupervScraper
|
||||
from app.connectors.direct_scrapers.xhamster import XHamsterScraper
|
||||
|
|
@ -158,6 +159,12 @@ ALL_BROWSE_SCRAPERS: list[type[BaseBrowseScraper]] = [
|
|||
# z linków, duration + release_date ISO, ~60-70 scen/dzień. Playback: direct mp4 bez
|
||||
# tokena (extractor youpervcom, VPS-side, CDN pilnuje tylko Referera).
|
||||
YoupervScraper,
|
||||
# GalaxyPornScraper — dodany 2026-07-26 (ocena 5/5). Ripy paysite, obsada w
|
||||
# wydzielonym `<div id="video-actors">` (zero pollution), studio+data z prefiksu
|
||||
# tytułu, paginacja zdrowa. Duration liczone z thumbnail.vtt playera (nie ma go
|
||||
# w HTML, a NULL ukryłby sceny pod filtrem min_duration_sec). Playback: iframe
|
||||
# seekplayer → istniejący silnik, HLS przez /proxy/hls.
|
||||
GalaxyPornScraper,
|
||||
# Browse równolegle do istniejącego search scrapera (wzorzec xvideos/eporner):
|
||||
# search zostaje (pokrycie back-catalogu performerów), browse gwarantuje świeżość
|
||||
# wprost z feedu (watchdog 48h zamiast 168h). Konwersja 2026-06-24 (user request).
|
||||
|
|
|
|||
228
app/connectors/direct_scrapers/galaxyporn.py
Normal file
228
app/connectors/direct_scrapers/galaxyporn.py
Normal file
|
|
@ -0,0 +1,228 @@
|
|||
"""galaxyporn.net — browse scraper (WordPress retrotube). Dodany 2026-07-26.
|
||||
|
||||
Ripy paysite ze studiami (BrazzersExxtra, EvilAngel, NubilesPorn, RealityKings,
|
||||
MYLF, BlackedRaw…). Ocena: orphan-risk LOW z twardym pomiarem — 92% performerów
|
||||
z próbki 75 ma już u nas ref tpdb/stashdb, 79% tytułów matchuje istniejące sceny.
|
||||
|
||||
Listing `/page/N/`, 21 scen/stronę, ~1180 stron, paginacja zdrowa (overlap p1/p2 = 0).
|
||||
|
||||
Metadane:
|
||||
- obsada: `<div id="video-actors">` — czysta, BEZ pollution (na całej stronie sceny
|
||||
są dokładnie te 2-3 linki `/actor/`, nie ma sekcji Related z obcymi performerami)
|
||||
- studio + data: prefiks tytułu `Studio YY MM DD Performer – Title` albo `[Studio]`
|
||||
- tagi: `<div class="tags-list">` (wycinamy slug studia, żeby nie robić tag=studio)
|
||||
- thumbnail: `data-main-thumb` / og:image
|
||||
|
||||
**Duration — uwaga, nie ma go w HTML.** Ani listing, ani detail, ani JSON-LD, ani
|
||||
payload playera nie niosą długości. NULL duration jest GROŹNY: `/scenes` filtruje
|
||||
`duration_sec >= min_duration_sec`, a NULL w SQL odpada z wyniku, więc sceny byłyby
|
||||
NIEWIDOCZNE w apce mimo "new: N, errors: 0" (dokładnie incydent porntrex, 6479 scen).
|
||||
Dlatego liczymy ją z `thumbnail.vtt` playera (ostatni cue), 1 dodatkowy request na
|
||||
scenę po odpytaniu API seekplayer. Wartość jest ~1 interwał miniatur krótsza od
|
||||
realnej (2226 s vs ~2276 s na próbce, ~2%) — akceptowalne, bo alternatywą jest brak.
|
||||
|
||||
Playback: iframe `#hash` rodziny seekplayer (galaxy.upns.online / galaxy.4meplayer.pro /
|
||||
sport.seekplays.com / news.upns.pro) → extractor `galaxyporncom` → istniejący
|
||||
`seekplayer_engine` (ten sam AES key/IV). HLS wymaga Referera → `/proxy/hls`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import date
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.connectors.base import (
|
||||
RawPerformer,
|
||||
RawPlaybackSource,
|
||||
RawScene,
|
||||
RawStudio,
|
||||
RawTag,
|
||||
)
|
||||
from app.connectors.direct_scrapers._browse_base import BaseBrowseScraper, meta_content
|
||||
from app.extractors._fetch import _DEFAULT_UA, browser_get
|
||||
from app.normalize.text import slugify
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_BASE = "https://galaxyporn.net"
|
||||
|
||||
_SCENE_URL_RE = re.compile(r'<a\s+href="(https://galaxyporn\.net/[a-z0-9\-]+/)"[^>]*title=', re.IGNORECASE)
|
||||
_H1_RE = re.compile(r'<h1[^>]*class="entry-title"[^>]*>(.*?)</h1>', re.IGNORECASE | re.DOTALL)
|
||||
_ACTORS_BLOCK_RE = re.compile(r'<div id="video-actors">(.*?)</div>', re.IGNORECASE | re.DOTALL)
|
||||
_ANCHOR_TEXT_RE = re.compile(r">([^<>]+)</a>")
|
||||
_TAGS_BLOCK_RE = re.compile(r'<div class="tags-list">(.*?)</div>', re.IGNORECASE | re.DOTALL)
|
||||
_THUMB_RE = re.compile(r'data-main-thumb="([^"]+)"', re.IGNORECASE)
|
||||
_IFRAME_RE = re.compile(r'<iframe[^>]+src="([^"]+)"', re.IGNORECASE)
|
||||
|
||||
# `BrazzersExxtra 26 07 26 Kayley Gunner & Elly Clutch – Swapping Off The Pervy Maid`
|
||||
_TITLE_STUDIO_DATE_RE = re.compile(r"^(?P<studio>[A-Za-z0-9][A-Za-z0-9._-]{1,30})\s+(?P<y>\d{2})\s+(?P<m>\d{2})\s+(?P<d>\d{2})\s+(?P<rest>.+)$")
|
||||
# `[Onlyfans] Some Title` / `[Darkkotv] …`
|
||||
_TITLE_BRACKET_RE = re.compile(r"^\[(?P<studio>[^\]]{2,30})\]\s*(?P<rest>.+)$")
|
||||
# `… [2018-09-28]`
|
||||
_TITLE_ISO_DATE_RE = re.compile(r"\[(\d{4})-(\d{2})-(\d{2})\]")
|
||||
|
||||
_VTT_CUE_RE = re.compile(r"(\d{2}):(\d{2}):(\d{2})[.,]\d+\s*-->\s*(\d{2}):(\d{2}):(\d{2})")
|
||||
|
||||
|
||||
def _clean_text(raw: str) -> str:
|
||||
text = re.sub(r"<[^>]+>", " ", raw)
|
||||
return html.unescape(re.sub(r"\s+", " ", text)).strip()
|
||||
|
||||
|
||||
def _resolve_duration(iframe_url: str, *, timeout: float, attempts: int = 3) -> int | None:
|
||||
"""Długość z `thumbnail.vtt` playera (ostatni cue). Patrz docstring modułu:
|
||||
duration nie ma w HTML, a NULL ukrywa sceny pod filtrem `min_duration_sec`.
|
||||
|
||||
API playera throttluje serie zapytań (przy ciągłym crawlu ~40% wywołań wracało
|
||||
puste), więc retry z rosnącym backoffem. Każdy błąd → None (scena i tak wejdzie,
|
||||
po prostu bez długości)."""
|
||||
from app.extractors.hosters.seekplayer_engine import _decrypt, matches
|
||||
|
||||
p = urlparse(iframe_url)
|
||||
if not p.hostname or not matches(iframe_url):
|
||||
return None
|
||||
hash_id = p.fragment.strip()
|
||||
if not hash_id:
|
||||
return None
|
||||
host = f"{p.scheme}://{p.hostname}"
|
||||
headers = {"User-Agent": _DEFAULT_UA, "Accept": "*/*", "Referer": host + "/"}
|
||||
|
||||
for attempt in range(attempts):
|
||||
if attempt:
|
||||
time.sleep(1.5 * attempt)
|
||||
try:
|
||||
r = browser_get(
|
||||
f"{host}/api/v1/video?id={hash_id}&w=1920&h=1080&r=",
|
||||
headers=headers,
|
||||
timeout=timeout,
|
||||
)
|
||||
if r.status_code != 200 or not r.text:
|
||||
continue
|
||||
data = json.loads(_decrypt(r.text))
|
||||
thumb = (data.get("thumbnail") or "").strip()
|
||||
if not thumb:
|
||||
return None # payload OK, ale brak miniatur — retry nie pomoże
|
||||
vtt_url = host + thumb if thumb.startswith("/") else thumb
|
||||
vr = browser_get(vtt_url, headers=headers, timeout=timeout)
|
||||
if vr.status_code != 200:
|
||||
continue
|
||||
cues = _VTT_CUE_RE.findall(vr.text)
|
||||
if not cues:
|
||||
return None
|
||||
h, m, s = cues[-1][3], cues[-1][4], cues[-1][5]
|
||||
return (int(h) * 3600 + int(m) * 60 + int(s)) or None
|
||||
except Exception as e: # pragma: no cover - best-effort
|
||||
log.info("galaxyporn: duration attempt %d failed (%s): %s", attempt + 1, iframe_url[:50], e)
|
||||
return None
|
||||
|
||||
|
||||
class GalaxyPornScraper(BaseBrowseScraper):
|
||||
sitetag = "galaxyporncom"
|
||||
|
||||
def _listing_url(self, page: int) -> str:
|
||||
return f"{_BASE}/" if page <= 1 else f"{_BASE}/page/{page}/"
|
||||
|
||||
def _extract_scene_urls(self, listing_html: str) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
for m in _SCENE_URL_RE.finditer(listing_html):
|
||||
url = m.group(1)
|
||||
if url in seen or url.rstrip("/") == _BASE:
|
||||
continue
|
||||
seen.add(url)
|
||||
out.append(url)
|
||||
return out
|
||||
|
||||
def _parse_detail(self, scene_url: str, detail_html: str) -> RawScene | None:
|
||||
h1 = _H1_RE.search(detail_html)
|
||||
title = _clean_text(h1.group(1)) if h1 else ""
|
||||
if not title:
|
||||
title = (meta_content(detail_html, property="og:title") or "").strip()
|
||||
if not title:
|
||||
return None
|
||||
|
||||
# Obsada — blok wydzielony, bez scopingu na siłę (potwierdzone: cała strona
|
||||
# ma dokładnie tyle linków /actor/ ile realnych aktorów).
|
||||
performers: list[RawPerformer] = []
|
||||
seen_p: set[str] = set()
|
||||
ab = _ACTORS_BLOCK_RE.search(detail_html)
|
||||
if ab:
|
||||
for m in _ANCHOR_TEXT_RE.finditer(ab.group(1)):
|
||||
name = html.unescape(m.group(1)).strip()
|
||||
sl = slugify(name)
|
||||
if not sl or sl in seen_p:
|
||||
continue
|
||||
seen_p.add(sl)
|
||||
performers.append(
|
||||
RawPerformer(external_id=f"{self.sitetag}:performer:{sl}", name=name)
|
||||
)
|
||||
|
||||
# Studio + data z prefiksu tytułu. `Studio YY MM DD …` albo `[Studio] …`.
|
||||
studio: RawStudio | None = None
|
||||
release_date: date | None = None
|
||||
studio_name: str | None = None
|
||||
if (m := _TITLE_STUDIO_DATE_RE.match(title)):
|
||||
studio_name = m.group("studio")
|
||||
try:
|
||||
release_date = date(2000 + int(m.group("y")), int(m.group("m")), int(m.group("d")))
|
||||
except ValueError:
|
||||
release_date = None
|
||||
elif (m := _TITLE_BRACKET_RE.match(title)):
|
||||
studio_name = m.group("studio")
|
||||
if release_date is None and (m := _TITLE_ISO_DATE_RE.search(title)):
|
||||
try:
|
||||
release_date = date(int(m.group(1)), int(m.group(2)), int(m.group(3)))
|
||||
except ValueError:
|
||||
release_date = None
|
||||
# Guard: prefiks nie może być nazwiskiem performera (wtedy to nie studio).
|
||||
if studio_name and slugify(studio_name) not in seen_p:
|
||||
studio = RawStudio(
|
||||
external_id=f"{self.sitetag}:studio:{slugify(studio_name)}",
|
||||
name=studio_name,
|
||||
slug=slugify(studio_name),
|
||||
)
|
||||
|
||||
# Tagi — bez slugu studia (retrotube wrzuca studio także jako zwykły tag).
|
||||
tags: list[RawTag] = []
|
||||
seen_t: set[str] = set()
|
||||
studio_slug = slugify(studio_name) if studio_name else ""
|
||||
tb = _TAGS_BLOCK_RE.search(detail_html)
|
||||
if tb:
|
||||
for m in _ANCHOR_TEXT_RE.finditer(tb.group(1)):
|
||||
name = html.unescape(m.group(1)).strip()
|
||||
sl = slugify(name)
|
||||
if not sl or sl in seen_t or sl == studio_slug or sl in seen_p:
|
||||
continue
|
||||
seen_t.add(sl)
|
||||
tags.append(RawTag(external_id=f"{self.sitetag}:tag:{sl}", name=name, slug=sl))
|
||||
|
||||
thumbnail_url = meta_content(detail_html, property="og:image")
|
||||
if not thumbnail_url and (tm := _THUMB_RE.search(detail_html)):
|
||||
thumbnail_url = tm.group(1)
|
||||
|
||||
duration_sec: int | None = None
|
||||
if (fm := _IFRAME_RE.search(detail_html)):
|
||||
duration_sec = _resolve_duration(fm.group(1).strip(), timeout=self._timeout)
|
||||
|
||||
return RawScene(
|
||||
external_id=f"{self.sitetag}:{scene_url}",
|
||||
title=title,
|
||||
release_date=release_date,
|
||||
duration_sec=duration_sec,
|
||||
url=scene_url,
|
||||
studio=studio,
|
||||
performers=performers,
|
||||
tags=tags,
|
||||
playback_sources=[
|
||||
RawPlaybackSource(
|
||||
origin=f"tube:{self.sitetag}",
|
||||
page_url=scene_url,
|
||||
duration_sec=duration_sec,
|
||||
thumbnail_url=thumbnail_url,
|
||||
)
|
||||
],
|
||||
)
|
||||
|
|
@ -30,6 +30,7 @@ from app.extractors.tubes import (
|
|||
eporner,
|
||||
freshporno,
|
||||
fullmovies,
|
||||
galaxyporn,
|
||||
hdporngg,
|
||||
hqfap,
|
||||
hqporner,
|
||||
|
|
@ -108,6 +109,9 @@ _REGISTRY: dict[str, Callable[[str], list[StreamSource] | None]] = {
|
|||
# youperv — fluidplayer, zwykły <source> mp4 na files.klubnichka-hd.com. Bez tokena
|
||||
# i bez IP-bindingu; CDN pilnuje tylko Referera (cross-IP 206 z VPS) → mobile direct.
|
||||
"youpervcom": youperv.extract,
|
||||
# galaxyporn — iframe rodziny seekplayer (upns/4meplayer/seekplays); silnik już mamy,
|
||||
# extractor tylko wyłuskuje iframe. HLS Referer+IP-bound → /proxy/hls.
|
||||
"galaxyporncom": galaxyporn.extract,
|
||||
"siskavideo": _embed_iframe.extract,
|
||||
"porn4dayspw": _embed_iframe.extract,
|
||||
"porndishcom": _embed_iframe.extract,
|
||||
|
|
|
|||
|
|
@ -54,13 +54,23 @@ log = logging.getLogger(__name__)
|
|||
_KEY = b"kiemtienmua911ca"
|
||||
_IV = b"1234567890oiuytr"
|
||||
|
||||
# Hostname matching: 6 base hosts × subdomains × TLD variants.
|
||||
# Hostname matching: base hosts × subdomains × TLD variants.
|
||||
# Examples:
|
||||
# my.embedseek.online, vip.seekplayer.vip, my.rpmplay.online,
|
||||
# my.upns.online, vip.player4me.vip, p.easyvidplayer.com
|
||||
#
|
||||
# 2026-07-26: dołożone `seekplays|4meplayer|ezplayer|seeks` + TLD `pro|cloud|one`.
|
||||
# Ta sama rodzina silnika (identyczny _KEY/_IV + `/api/v1/video?id=`), tylko inne
|
||||
# domeny — bez tego regexu `matches()` odrzucał je zanim doszło do dekodowania.
|
||||
# Zweryfikowane: galaxy.4meplayer.pro / sport.seekplays.com / news.upns.pro
|
||||
# (galaxyporn) oraz av.ezplayer.me / av.seeks.cloud / 4k.upn.one (pornbusy)
|
||||
# dekodują się tym samym silnikiem. Whitelist jest addytywna — istniejące hosty
|
||||
# matchują się dalej tak samo.
|
||||
_HOST_RE = re.compile(
|
||||
r"^(?:[a-z0-9]+\.)?(?:embedseek|seekplayer|rpmplay|upns|player4me|easyvidplayer)\."
|
||||
r"(?:online|vip|com|net|io|me|tv)$",
|
||||
r"^(?:[a-z0-9]+\.)?"
|
||||
r"(?:embedseek|seekplayer|seekplays|seeks|rpmplay|upns|upn|player4me|4meplayer"
|
||||
r"|ezplayer|easyvidplayer)\."
|
||||
r"(?:online|vip|com|net|io|me|tv|pro|cloud|one)$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
|
|
|||
69
app/extractors/tubes/galaxyporn.py
Normal file
69
app/extractors/tubes/galaxyporn.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
"""galaxyporn.net — iframe → seekplayer engine. Dodany 2026-07-26.
|
||||
|
||||
Scene page ma jeden `<iframe src="https://<host>/#<hash>">` rodziny seekplayer
|
||||
(galaxy.upns.online / galaxy.4meplayer.pro / sport.seekplays.com / news.upns.pro —
|
||||
100% próbki 50 scen z całej głębokości archiwum). Silnik mamy już w repo
|
||||
(`hosters/seekplayer_engine.py`, ten sam AES key/IV + `/api/v1/video?id=`), więc
|
||||
oddajemy iframe do generycznego `extract_stream_from_hoster` i nic nie dublujemy.
|
||||
|
||||
HLS jest hotlink-guarded na Referer (bez niego 403 na master i na segmentach),
|
||||
a token podpisany pod IP fetchera → manifest i segmenty idą przez `/proxy/hls`.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from app.extractors._fetch import fetch_tube_html
|
||||
from app.extractors._models import StreamSource
|
||||
from app.extractors.hosters import seekplayer_engine
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_BASE = "https://galaxyporn.net"
|
||||
_IFRAME_RE = re.compile(r'<iframe[^>]+src="([^"]+)"', re.IGNORECASE)
|
||||
|
||||
|
||||
def extract(page_url: str, *, timeout: float = 60.0) -> list[StreamSource] | None:
|
||||
html_text = fetch_tube_html(page_url, timeout=timeout)
|
||||
m = _IFRAME_RE.search(html_text)
|
||||
if not m:
|
||||
log.info("galaxyporn: no iframe on %s", page_url)
|
||||
return None
|
||||
iframe_src = m.group(1).strip()
|
||||
if iframe_src.startswith("//"):
|
||||
iframe_src = "https:" + iframe_src
|
||||
|
||||
# Wołamy silnik BEZPOŚREDNIO, nie przez `extract_stream_from_hoster`: wrapper
|
||||
# weryfikuje wynikowy URL, a HLS galaxyporn jest hotlink-guarded (403 bez
|
||||
# Referera) → wrapper kasował poprawnie zdekodowany stream (sprawdzone: engine
|
||||
# zwracał m3u8, wrapper None dla tych samych scen).
|
||||
sources = seekplayer_engine.extract(iframe_src, timeout=timeout)
|
||||
if not sources:
|
||||
# Nie oddajemy iframe'a jako type='hoster' — seekplayer to SPA, w WebView
|
||||
# user zobaczy pusty player. Lepiej None (źródło jako nierozwiązane).
|
||||
log.info("galaxyporn: seekplayer resolve failed for %s", iframe_src)
|
||||
return None
|
||||
|
||||
# Referer MUSI być hostem PLAYERA (np. https://galaxy.4meplayer.pro/), NIE
|
||||
# galaxyporn.net — zweryfikowane: ten sam manifest z Refererem galaxyporn = 403,
|
||||
# z Refererem playera = 200 + lista wariantów. Silnik ustawia go poprawnie, więc
|
||||
# go zachowujemy; fallback liczymy z hosta iframe'a.
|
||||
player_origin = f"https://{urlparse(iframe_src).hostname}/"
|
||||
out: list[StreamSource] = []
|
||||
for s in sources:
|
||||
raw = dict(s.raw or {})
|
||||
# Token HLS jest Referer+IP-bound → manifest+segmenty przez /proxy/hls
|
||||
# (bez mobile_direct_ok, inaczej telefon dostanie 403).
|
||||
raw["proxy_no_verify"] = True
|
||||
out.append(
|
||||
StreamSource(
|
||||
link=s.link,
|
||||
type=s.type or ("m3u8" if ".m3u8" in s.link.lower() else "mp4"),
|
||||
quality=s.quality,
|
||||
referer=s.referer or player_origin,
|
||||
raw=raw,
|
||||
)
|
||||
)
|
||||
return out
|
||||
Loading…
Add table
Reference in a new issue