goon-foss/goon
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
goon/app/connectors/direct_scrapers
History
goon-foss 532c7e2bb4 fix(vjav): 429 parsowany jako XML, zrodlo stalo 12 dni bez zadnego bledu 
Audyt po fixie streamporn.vip. vjav (77 tys. zrodel) nie dodal nic od 2026-07-22 i
NIE zglaszal bledu - crawl_page oddawal 0 scen, licznik pokazywal errors: 0.

Przyczyna: vjav odcial IP VPS-a. Sprawdzone 2026-08-03 - sitemap, strona glowna i
/latest-updates/ daja 429 przy trzech probach z rzedu, a przez Bright Data 200 od
pierwszej. Kod tego nie widzial, bo browser_get NIE rzuca na 4xx, wiec 343-bajtowa
tresc bledu byla parsowana jako XML: zero sitemap, zero id.

Drugi blad, ktory to ukryl: przy niepowodzeniu _load_ids ustawialo _sorted_ids = [],
a crawl_page tlumaczy [] na "koniec katalogu" (poprawny stan), nie na awarie. Teraz
przy bledzie zostaje None, ktore crawl_page zwraca jako transient-fail.

Fix: proxy Bright Data na sitemap i API metadanych (ten sam host, ten sam 429), jawne
sprawdzanie statusu, [] tylko dla realnie pustego katalogu.

Po fixie crawl_page oddaje 120 scen zamiast 0. Nowej podazy to jeszcze nie dalo
(wszystkie 120 to pozycje sprzed blokady, skipped na niezmienionym hashu) - sitemap
vjav nie przesunal sie od czasu odciecia. Do obserwacji. Uwaga na jakosc: w tej
partii 58 proc. tytulow to slug-concat typu 0008363_JAV_..., przy 2,9 proc. w calym
katalogu.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-03 14:36:25 +02:00
..
__init__.py chore(hqfap): zdjete zrodlo - Cloudflare JS-challenge nie do przejscia 2026-08-02 18:01:19 +02:00
_browse_base.py docs: correct Bright Data proxy type (ISP, flat-rate not per-GB) 2026-06-10 19:18:40 +02:00
_playtube.py feat(tubes): add 4k69 + neporn browse scrapers, shared PlayTube base 2026-06-10 18:15:13 +02:00
_search_base.py feat(scrapers): extract tags + description from porndish scene pages 2026-06-06 21:32:10 +02:00
base.py refactor(ingest): rename scraper Source name "pornapp" -> "tube-scraper" 2026-06-07 16:54:55 +02:00
eporner.py Initial commit 2026-05-20 10:10:22 +02:00
eporner_api.py feat(deep-crawl): eporner via JSON API as SSR-rich source (Phase 2b alternative) 2026-06-03 10:37:20 +02:00
fpoxxx.py feat(ingest): revive fpoxxx — search→browse (KVS /new-N/) 2026-06-22 12:04:05 +02:00
freshporno.py fix(scrapers): freshporno browse from /latest-updates/ not homepage root 2026-06-15 09:59:40 +02:00
fullmovies.py feat(ingest): SQL phash match, tag inference + backfill, clip-store skip, browse tubes, watchdog 2026-06-01 15:07:35 +02:00
fullporner.py feat(fullporner): browse scraper pelnych scen + extractor bez tokenu 2026-07-27 12:04:34 +02:00
fullvideosporn.py feat(fullvideosporn): browse scraper behind a cast gate + TXXX extractor 2026-07-27 08:56:23 +02:00
galaxyporn.py feat(galaxyporn): browse scraper + extractor; widen seekplayer host regex 2026-07-26 18:23:13 +02:00
hdporn92.py fix(performers): czlon <3 znaki degenerowal filtr slug-matchu (over-attribution) 2026-07-28 09:52:41 +02:00
hdporngg.py feat(ingest): SQL phash match, tag inference + backfill, clip-store skip, browse tubes, watchdog 2026-06-01 15:07:35 +02:00
hqfap.py feat(hqfap): restore hqfap.com scraper + extractor (site back on new CDN) 2026-07-07 23:53:43 +02:00
hqporner.py fix(performers): czlon <3 znaki degenerowal filtr slug-matchu (over-attribution) 2026-07-28 09:52:41 +02:00
hqporner_browse.py feat(ingest): browse-latest scraper dla hqporner 2026-07-26 16:41:47 +02:00
javflix.py feat(jav): separate JAV section (tab + feed gating) + enable javflix ingest 2026-07-10 11:16:31 +02:00
kporner8.py feat(8kporner): browse scraper za obejściem CF + extractor JWPlayer sources 2026-07-27 11:15:32 +02:00
latestleaks.py Initial commit 2026-05-20 10:10:22 +02:00
latestpornvideo.py feat(ingest): disable hqfap/4k69 (broken playback), latestpornvideo → browse 2026-06-22 09:34:47 +02:00
mypornerleak.py fix(mypornerleak): revive search via /actor/ listing + metadata 2026-06-16 23:16:02 +02:00
mypornerleak_browse.py feat(ingest): add browse scrapers for porntrex + mypornerleak (alongside search) 2026-06-24 15:41:22 +02:00
neporn.py feat(tubes): add 4k69 + neporn browse scrapers, shared PlayTube base 2026-06-10 18:15:13 +02:00
perverzija.py feat(ingest): revive perverzija — search→WP REST API browse 2026-06-22 13:10:16 +02:00
porn00.py Initial commit 2026-05-20 10:10:22 +02:00
porn4days.py Initial commit 2026-05-20 10:10:22 +02:00
pornbusy.py feat(pornbusy): browse scraper + loadvid POST-manifest producer 2026-07-27 08:45:00 +02:00
porndish.py feat(ingest): revive porndish — search→WP REST API browse 2026-06-24 15:09:27 +02:00
porndoe.py Mobile 0.1.9: OTA enable, WebView cookie-dismiss fix, porndoe connector 2026-05-22 11:20:57 +02:00
pornmike.py feat(pornmike): browse scraper behind a cast gate + direct-mp4 extractor 2026-07-27 10:42:39 +02:00
porntrex.py Initial commit 2026-05-20 10:10:22 +02:00
porntrex_browse.py fix(playback+ingest): realny proxy fallback gdy CDN nieosiagalny + porntrex duration 2026-07-26 16:34:24 +02:00
pornxp.py Initial commit 2026-05-20 10:10:22 +02:00
sexu.py feat(sexu): scraper + extractor HLS, ale bez rejestracji w ingescie 2026-07-27 11:42:05 +02:00
shyfap.py Initial commit 2026-05-20 10:10:22 +02:00
siska.py feat(siska): convert to browse scraper, re-enable (search broken site-side) 2026-06-20 16:25:11 +02:00
superporn.py docs: correct Bright Data proxy type (ISP, flat-rate not per-GB) 2026-06-10 19:18:40 +02:00
supjav.py feat(jav): supjav.com scraper + extractor (embed-aggregator, 4th JAV source) 2026-07-10 14:10:43 +02:00
sxyland.py fix(sxyland): revive search via /actor/ pages + rich metadata 2026-06-16 23:11:44 +02:00
sxyprn.py fix(sxyprn): revive search via performer pages + rich metadata 2026-06-16 22:58:52 +02:00
vjav.py fix(vjav): 429 parsowany jako XML, zrodlo stalo 12 dni bez zadnego bledu 2026-08-03 14:36:25 +02:00
watchporn.py fix(thumbnails): watchporn thumbs at ingest + backfill; SceneDetail uses live sxyprn thumb 2026-07-07 23:21:59 +02:00
xhamster.py Initial commit 2026-05-20 10:10:22 +02:00
xmoviesforyou.py Initial commit 2026-05-20 10:10:22 +02:00
xnxx.py Initial commit 2026-05-20 10:10:22 +02:00
xnxx_browse.py feat(ingest): add xnxx browse scraper (JSON-LD only, alongside search) 2026-06-24 15:52:32 +02:00
xvideos.py Initial commit 2026-05-20 10:10:22 +02:00
xvideos_browse.py fix(xvideos): parse model name from nested span.name — recover 0-performer scenes 2026-06-08 10:13:21 +02:00
xxxfreewatch.py Initial commit 2026-05-20 10:10:22 +02:00
yesporn.py feat(deep-crawl): xvideos browse source (capped) + per-tube page cap 2026-06-03 11:16:44 +02:00
youperv.py feat(youperv): browse scraper + direct-mp4 extractor 2026-07-26 17:57:24 +02:00
youporn.py Initial commit 2026-05-20 10:10:22 +02:00
youporn_browse.py feat(ingest): add youporn browse scraper (JSON-LD only, alongside search) 2026-06-24 15:47:58 +02:00