d7442187c9
fullvideosporn.com is sextu.com rebranded, not a clone of fullmovies.xxx (different engine, different catalog, 0/140 title overlap with our fullmoviesxxx corpus). The site is only worth ingesting behind a gate: 65-75% of its catalog has no performer at all, and those scenes also carry SEO-spun titles, so they would never match canonical and would land as empty orphans. So we ingest only scenes with at least one performer. That keeps ~25-35% of the catalog (verified: 19 of 60 on page one) where the signal is good, since 88-89% of the performer names in the research sample already resolve to a canonical performer in our DB. Three site-specific traps, all handled: - Titles come from the player's vit:"..." field, not og:title/h1, which are sometimes an AI SEO rewrite rather than the real scene title. - Cast is read only from the <h3>Porn-stars:</h3> section; the page carries ~22 videos.php?q= links overall but only 1-2 real performers, the same pollution that got xxxfiles rejected. Porn Site / Porn Categories are separate h3 blocks and are parsed per-section so they don't bleed. - Every fetch passes a cookie gate: a fresh session gets HTTP 429 plus a small JS challenge, so we read the cookie out of it and retry on the same session. Hence the custom crawl_page instead of the base browser_get. The TXXX video_url decoder moved out of vjav into _txxx.py since both tubes share the engine; vjav keeps an alias and was re-verified after the move. Playback resolves videofile.php -> decode -> get_file -> 302 -> znvcdn, and the final CDN URL is portable cross-IP so the phone streams it directly. Note for future debugging: the VPS itself gets 429 from that CDN because of datacenter IP reputation, so playback health-checks run from the VPS will be falsely negative. Pilot ingest of 2 pages: 39 seen, 11 merged into existing scenes, 28 new, 0 errors; 0/19 without cast or duration on the sampled page. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
39 lines
1.4 KiB
Python
39 lines
1.4 KiB
Python
"""Wspólne elementy sieci TXXX (vjav, fullvideosporn/sextu, …).
|
||
|
||
Silnik TXXX oddaje URL pliku przez `GET /api/videofile.php?video_id=<id>&lifetime=N`
|
||
w polu `video_url`, zaciemnionym DWIEMA warstwami:
|
||
1. wielkie/małe litery łacińskie podmienione na cyrylickie homoglify (М→M, С→C, …),
|
||
2. custom alfabet base64: `,`→`/`, `~`→`=`, `-`→`+`.
|
||
|
||
Po odkręceniu obu i b64decode dostajemy `/get_file/...` (czasem absolutny URL).
|
||
Wyniesione tutaj, żeby vjav i fullvideosporn nie trzymały dwóch kopii dekodera.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import base64
|
||
|
||
# Cyrylickie homoglify → łacina. Bez tego b64decode dostaje śmieci.
|
||
HOMOGLYPHS = str.maketrans(
|
||
{
|
||
"А": "A", "В": "B", "С": "C", "Е": "E", "Н": "H", "К": "K", "М": "M",
|
||
"О": "O", "Р": "P", "Т": "T", "Х": "X", "У": "Y",
|
||
"а": "a", "с": "c", "е": "e", "о": "o", "р": "p", "х": "x", "у": "y",
|
||
}
|
||
)
|
||
|
||
|
||
def decode_video_url(obfuscated: str) -> str | None:
|
||
"""Zaciemniony `video_url` → ścieżka/URL `get_file`. None gdy dekod padnie."""
|
||
if not obfuscated:
|
||
return None
|
||
clean = (
|
||
obfuscated.translate(HOMOGLYPHS)
|
||
.replace(",", "/")
|
||
.replace("~", "=")
|
||
.replace("-", "+")
|
||
)
|
||
clean += "=" * (-len(clean) % 4) # padding do wielokrotności 4
|
||
try:
|
||
return base64.b64decode(clean).decode("utf-8", "ignore")
|
||
except Exception:
|
||
return None
|