07cc3fdfba
Browse scrapers backfilling old catalogs stamp the tube's import/post date as release_date, so old content (e.g. 83 MissaX classics via perverzija, ~3600/3 days across eporner/youporn/etc.) fake-ranked as newest and flooded the favorites "+N". NULLing the dates was a non-starter — the stub filter would hide 251k performer-less scenes. Instead: a Scene.backfill flag marks bulk catalog imports; they stay visible but never count as "new". - scenes.backfill column (+ index, migration 0026); resolve_scene/_process_scene thread it. - deep_crawl tags scenes from pages beyond the "latest" threshold (>2) as backfill; latest pages + TPDB/StashDB delta stay genuine. Cursor reset re-sweeps page 1 so real new content is always caught fresh. - favorites +N (performers + studios) excludes backfill within the top-200 window. - SceneOut exposes `backfill`; mobile NEW badge + NEW-first re-sort skip it (badge==count). - Retroactive: tagged 439k existing scenes in bulk (>10 non-canonical / studio / day) clusters. Device check: favorite-studios +N 11626 (naive) -> 236. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
39 lines
1.3 KiB
Python
39 lines
1.3 KiB
Python
"""scene backfill flag: exclude bulk catalog imports from "new"
|
|
|
|
Revision ID: 0026_scene_backfill_flag
|
|
Revises: 0025_source_ranking
|
|
Create Date: 2026-07-02
|
|
|
|
Tube'y podają datę importu jako release_date, więc masowy backfill starego katalogu
|
|
(deep-crawl głębokie strony) udawał świeżość i zawyżał licznik "+N nowych" w ulubionych.
|
|
`scenes.backfill` oznacza takie sceny — pozostają widoczne, ale nie liczą się jako nowe.
|
|
Ustawiane przy tworzeniu (deep_crawl page > próg); świeży latest-crawl + delta TPDB/
|
|
StashDB → False. Patrz app/scheduler/deep_crawl.py, app/resolve/scene_resolver.py.
|
|
"""
|
|
from collections.abc import Sequence
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
|
|
revision: str = "0026_scene_backfill_flag"
|
|
down_revision: str | None = "0025_source_ranking"
|
|
branch_labels: str | Sequence[str] | None = None
|
|
depends_on: str | Sequence[str] | None = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.add_column(
|
|
"scenes",
|
|
sa.Column(
|
|
"backfill",
|
|
sa.Boolean(),
|
|
nullable=False,
|
|
server_default=sa.text("false"),
|
|
),
|
|
)
|
|
op.create_index("ix_scenes_backfill", "scenes", ["backfill"])
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_index("ix_scenes_backfill", table_name="scenes")
|
|
op.drop_column("scenes", "backfill")
|