fix(triage): theporndude candidate audit had two false-negative bugs
The domain fallback built its guess from the REVIEW slug: when pdude.link resolved to the porndudecams interstitial, "youperv-website-review" became youperv-website-review.com, which of course refused the connection and the candidate was filed as dead. That hit 15 of 144 entries, and among them were youperv and pornbusy, which we later found independently and shipped at 4.5/5. Now the -website-review / -review suffix is stripped, and 13 of those 15 answer 200. The audit also fetched with plain httpx, whose TLS fingerprint most Cloudflare-fronted tubes reject, so timeouts and 403s were being read as "dead" for sites that work fine in a browser. It now goes through browser_get (curl_cffi), the same path our scrapers use, so the triage finally measures what production sees. Worth knowing before trusting the output: the heuristic score does NOT predict a candidate's value to us. It matches cosmetic homepage markers, and on sites we have since evaluated properly it scored youperv 0 and pornbusy -1.5 (a false auth_wall) while giving 2.5 to hqfap, which we removed as an orphan factory. Use the refreshed file as a list of live, unexplored domains (95 of them) and judge each one on canonical match and metadata quality, not on this number. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
d7442187c9
commit
2ef54ad6ea
2 changed files with 521 additions and 349 deletions
|
|
@ -41,16 +41,43 @@ META_MARKERS = [
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# Slug w feedzie to slug RECENZJI ("youperv-website-review"), nie domena. Bez odcięcia
|
||||||
|
# sufiksu fallback budował "youperv-website-review.com" → conn_refused → wpis lądował
|
||||||
|
# jako "dead/skip". Tak wypadło 15 ze 144 kandydatów, w tym youperv i pornbusy, które
|
||||||
|
# potem niezależnie oceniliśmy na 4.5-5/5 i dodaliśmy. Fix 2026-07-27.
|
||||||
|
_REVIEW_SUFFIX_RE = re.compile(r"-(?:website-)?review$", re.IGNORECASE)
|
||||||
|
|
||||||
|
|
||||||
|
def _slug_to_domain(slug: str) -> str:
|
||||||
|
return _REVIEW_SUFFIX_RE.sub("", slug.lower().strip())
|
||||||
|
|
||||||
|
|
||||||
async def fetch_one(cli: httpx.AsyncClient, url: str) -> tuple[int, str]:
|
async def fetch_one(cli: httpx.AsyncClient, url: str) -> tuple[int, str]:
|
||||||
|
"""Fetch z Chrome TLS (curl_cffi) — NIE gołym httpx.
|
||||||
|
|
||||||
|
FIX 2026-07-27: pierwotna wersja szła czystym httpx, którego JA3 fingerprint
|
||||||
|
większość tubów za Cloudflare odrzuca. Efekt: 8 timeoutów + 4× 403 w wynikach,
|
||||||
|
czytane potem jako "dead/skip", choć strony działają w przeglądarce. `browser_get`
|
||||||
|
(curl_cffi, chrome120) przechodzi — to ta sama ścieżka, której używają nasze
|
||||||
|
scrapery, więc triage mierzy wreszcie to samo co produkcja.
|
||||||
|
"""
|
||||||
|
import asyncio as _asyncio
|
||||||
|
|
||||||
|
from app.extractors._fetch import browser_get
|
||||||
|
|
||||||
|
def _sync():
|
||||||
|
return browser_get(url, timeout=20.0, headers={"User-Agent": UA})
|
||||||
|
|
||||||
try:
|
try:
|
||||||
r = await cli.get(url, headers={"User-Agent": UA}, follow_redirects=True)
|
r = await _asyncio.to_thread(_sync)
|
||||||
return r.status_code, r.text[:200_000] # cap response
|
return r.status_code, (r.text or "")[:200_000]
|
||||||
except httpx.ConnectError:
|
|
||||||
return -1, "conn_refused"
|
|
||||||
except httpx.TimeoutException:
|
|
||||||
return -2, "timeout"
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
return -9, str(e)[:120]
|
name = type(e).__name__
|
||||||
|
if "Timeout" in name:
|
||||||
|
return -2, "timeout"
|
||||||
|
if "Connect" in name or "Resolve" in name or "DNS" in name:
|
||||||
|
return -1, "conn_refused"
|
||||||
|
return -9, f"{name}: {e}"[:120]
|
||||||
|
|
||||||
|
|
||||||
def analyze_html(html: str) -> dict:
|
def analyze_html(html: str) -> dict:
|
||||||
|
|
@ -152,7 +179,7 @@ async def main():
|
||||||
domain = r.get("domain") or ""
|
domain = r.get("domain") or ""
|
||||||
# Jeśli pdude.link daje porndudecams.com (interstitial), użyj <slug>.com
|
# Jeśli pdude.link daje porndudecams.com (interstitial), użyj <slug>.com
|
||||||
if not domain or "porndudecams" in domain:
|
if not domain or "porndudecams" in domain:
|
||||||
domain = f"{r['slug'].lower()}.com"
|
domain = f"{_slug_to_domain(r['slug'])}.com"
|
||||||
return {**r, **(await audit_one(cli, r["slug"], domain))}
|
return {**r, **(await audit_one(cli, r["slug"], domain))}
|
||||||
|
|
||||||
results = await asyncio.gather(*[worker(r) for r in new_candidates])
|
results = await asyncio.gather(*[worker(r) for r in new_candidates])
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
Loading…
Add table
Reference in a new issue