goon-foss/goon
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
goon/app/connectors/direct_scrapers
History
jtrzupek e42217773f feat(deep-crawl): xvideos browse source (capped) + per-tube page cap 
xvideos SSR's JSON-LD VideoObject (duration/title/uploadDate) + on-page /models/ (perf)
+ /tags/. Sample: median ~10.5min, 93% >=3min. Pilot (2 pages): 29 new, 100% playable +
visible + tagged (performers sparse — xvideos 'new' is amateur-heavy; /models/ tagged
mostly on studio rips).

- XVideosBrowseScraper (JSON-LD + page-parse models/tags), in ALL_BROWSE_SCRAPERS.
- deep_crawl._PAGE_CAP: per-sitetag depth cap; xvideoscom=1800 (~newest 50k). At the cap
  the tube is marked exhausted (reset -> incremental re-sweep) so a mega-tube cannot
  monopolize the round-robin or balloon the DB.
- ported yesporn.py into the public repo (was prod-only, like hdporngg) ending the
  __init__ public/prod divergence.

youporn rejected: JSON-LD lacks actor/keywords, its /pornstar//category/ links are A-Z
nav not scene-specific. xhamster: 429/Cloudflare from the VPS IP.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 11:16:44 +02:00
..
__init__.py feat(deep-crawl): xvideos browse source (capped) + per-tube page cap 2026-06-03 11:16:44 +02:00
_browse_base.py feat(deep-crawl): eporner via JSON API as SSR-rich source (Phase 2b alternative) 2026-06-03 10:37:20 +02:00
_search_base.py Initial commit 2026-05-20 10:10:22 +02:00
base.py Initial commit 2026-05-20 10:10:22 +02:00
eporner.py Initial commit 2026-05-20 10:10:22 +02:00
eporner_api.py feat(deep-crawl): eporner via JSON API as SSR-rich source (Phase 2b alternative) 2026-06-03 10:37:20 +02:00
fpoxxx.py Initial commit 2026-05-20 10:10:22 +02:00
freshporno.py Mobile 0.1.9: OTA enable, WebView cookie-dismiss fix, porndoe connector 2026-05-22 11:20:57 +02:00
fullmovies.py feat(ingest): SQL phash match, tag inference + backfill, clip-store skip, browse tubes, watchdog 2026-06-01 15:07:35 +02:00
hdporn92.py Initial commit 2026-05-20 10:10:22 +02:00
hdporngg.py feat(ingest): SQL phash match, tag inference + backfill, clip-store skip, browse tubes, watchdog 2026-06-01 15:07:35 +02:00
hqporner.py Initial commit 2026-05-20 10:10:22 +02:00
latestleaks.py Initial commit 2026-05-20 10:10:22 +02:00
latestpornvideo.py Initial commit 2026-05-20 10:10:22 +02:00
mypornerleak.py Initial commit 2026-05-20 10:10:22 +02:00
perverzija.py Initial commit 2026-05-20 10:10:22 +02:00
porn00.py Initial commit 2026-05-20 10:10:22 +02:00
porn4days.py Initial commit 2026-05-20 10:10:22 +02:00
porndish.py Initial commit 2026-05-20 10:10:22 +02:00
pornditt.py Initial commit 2026-05-20 10:10:22 +02:00
porndoe.py Mobile 0.1.9: OTA enable, WebView cookie-dismiss fix, porndoe connector 2026-05-22 11:20:57 +02:00
pornhat.py Initial commit 2026-05-20 10:10:22 +02:00
pornhub.py Initial commit 2026-05-20 10:10:22 +02:00
porntrex.py Initial commit 2026-05-20 10:10:22 +02:00
pornxp.py Initial commit 2026-05-20 10:10:22 +02:00
redtube.py Initial commit 2026-05-20 10:10:22 +02:00
shyfap.py Initial commit 2026-05-20 10:10:22 +02:00
siska.py Initial commit 2026-05-20 10:10:22 +02:00
sxyland.py Initial commit 2026-05-20 10:10:22 +02:00
sxyprn.py Initial commit 2026-05-20 10:10:22 +02:00
watchporn.py Initial commit 2026-05-20 10:10:22 +02:00
xhamster.py Initial commit 2026-05-20 10:10:22 +02:00
xmoviesforyou.py Initial commit 2026-05-20 10:10:22 +02:00
xnxx.py Initial commit 2026-05-20 10:10:22 +02:00
xvideos.py Initial commit 2026-05-20 10:10:22 +02:00
xvideos_browse.py feat(deep-crawl): xvideos browse source (capped) + per-tube page cap 2026-06-03 11:16:44 +02:00
xxxfreewatch.py Initial commit 2026-05-20 10:10:22 +02:00
yesporn.py feat(deep-crawl): xvideos browse source (capped) + per-tube page cap 2026-06-03 11:16:44 +02:00
youporn.py Initial commit 2026-05-20 10:10:22 +02:00
zerodayxx.py Initial commit 2026-05-20 10:10:22 +02:00