Improvements v0.2.0 #14
Loading…
Reference in a new issue
No description provided.
Delete branch "improvements-v0.2.0"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
f4619de).c99bd53).Changes
Simplify v0.2.0:
view_models: drop per-rowget_source()calls inentry_to_view/hit_to_view; carrysource_trustonEntryRow/SearchHitand compute badges from that.queries/health: collapsesource_qualityN+1 into one windowed query usingROW_NUMBER() OVER (PARTITION BY source_slug ORDER BY started_at DESC).plaintext: reuseSTREAM_ORDERfrompresentation.topicsandstream_headlinefromview_modelsinstead of duplicating the constants and headline map.rank_badges: remove unusedmerge_rank_badgeshelper and its test.article_dates: exposeURL_FUTURE_GRACE_DAYS;helpers.pyre-uses it asARCHIVE_FUTURE_GRACE_DAYSso the two 14-day constants cannot drift.routes/web: usepublic_base_urlfor the RSS feeds instead of a private duplicate.worker: collapse_persist_mit_news_feedinto_persist_feedvia aparserparameter.HN coverage:
collect_candidatesat the same number.test_fetch_source_requests_ninety_hn_top_storiesasserts 1 topstories call + 90 item fetches.Why
HN front-page page 1 only carried the first 30 items; anything past rank 30 (including most of
/news?p=2) never reached TILens. Bumping to 90 covers the first three pages without bloating the per-cycle work.Surface cs.IR papers (GenRec, recsys, ranking, search) under a new top-level topic so /topics/information-retrieval is useful instead of having to scroll the algorithm feed. Cover the same content from the Finxter-curated AI newsletter list with the active feed-only entries that still parse today. And fix the underlying defect: newsletter expanders were stamping every extracted link with the issue date, not the linked article date, so all entries from one issue piled up at the same second on the timeline. Topics - New app/backend/presentation/topics.py constant TOPIC_INFORMATION_RETRIEVAL, registered as a top-level category so it appears alongside Algorithms / General in topic_sections(). - tests/test_taxonomy.py extended to assert the new category is part of main_category_slugs(). Migration 009 - app/backend/migrations/009_information_retrieval_topic.sql: * insert topics(slug='information-retrieval', parent_slug=NULL) * set arxiv-cs-ir primary_topic_slug = information-retrieval * link arxiv-cs-ir to source_topics for information-retrieval, ai-research, ai * backfill existing arxiv-cs-ir entries with the same three entry_topics and make information-retrieval the primary * refresh entries_fts.topic_labels so the child topic label is immediately searchable All statements are guarded so the migration is a no-op on databases that pre-date the source row. Catalog - config/sources.csv adds arxiv-cs-ir (research, top-level IR) and six vetted AI newsletter feeds: - the-rundown-ai (50 entries, latest 2026-09-01) - the-neuron-ai (10 entries, latest 2026-09-01) - ai-weekly (20 entries, latest 2026-08-30) - true-positive-weekly (20 entries, latest 2026-08-31) - ai-agents-simplified (20 entries, latest 2026-08-25) - altern-newsletter (20 entries, latest 2026-08-22) Skipped for quality: the-marketing-newsletter (SEO/marketing), buzzrobot (stale), bens-bites-blog (stale), data-elixir (stale), finxter-global (broad/noisy). HTTP-only ai-weekly is left in HTTP because the worker SSRF-safe fetcher is HTTPS by default; the smoke test acknowledges this. Article-date helpers - New app/backend/adapters/article_dates.py (stdlib only): * parse_datetime handles RFC-822 and ISO with milliseconds and offsets, normalizing to UTC * published_from_url extracts /YYYY/MM/DD/, /YYYY-MM-DD-, and /YYYYMMDD URL paths with sanity bounds on the year * published_from_html handles article:published_time, JSON-LD datePublished / dateModified, <time datetime>, parsely, dc.date, citation_publication_date - unused today but ready for a future bounded fetcher * best_link_published_at prefers URL date, falls back to the supplied issue date, accepts None fallback so existing tests and callers can pass it without guarding. Adapter wiring - this_week_in_rust, pycoders_weekly, pythonhub_digest, jvm_weekly now resolve the per-link date via best_link_published_at, falling back to the issue RSS / Atom date when the URL has no date. - python_weekly adds _issue_slug_published_at so the Beehiiv slug /p/python-weekly-issue-<n>-<month>-<day>-<year> yields a date instead of datetime.now(UTC), and then resolves per-link dates on top. - ocaml_cwn intentionally unchanged - it expands weekly sections inside the same issue, so the issue date is correct. Tests - tests/test_article_dates.py: URL patterns, RFC-822 + offset, ISO with milliseconds, JSON-LD datePublished, <time datetime>, article:published_time, out-of-range and impossible-day rejections, fallback handling. - tests/test_migration_009.py: topic insertion, top-level status, primary-topic + multi-tag backfill for arxiv-cs-ir, idempotency, FTS topic_labels refresh. - tests/test_this_week_in_rust_adapter.py / pycoders / python_weekly / pythonhub / jvm_weekly: URL date overrides issue date when present; URL fallback preserved when missing; Python Weekly issue-slug parses; resolved-final-URL date wins in PyCoder's. - tests/test_feed_adapter.py: ISO with milliseconds + offset to UTC (locks The Neuron and arXiv cs.IR formats). - tests/test_catalog.py: arxiv-cs-ir active + topics; six newsletter sources active + feed + news + ai topic; smoke test that any of the v0.1.1 endpoints returns a usable feed. - tests/test_taxonomy.py: main_category_slugs includes the new Information Retrieval entry. Verification - uv run pytest tests/ -q -> 750 passed (was 691), 1 pre-existing failure in test_calendar_only_redirects_to_first_available_day (date drift, unrelated to v0.1.1; verified by stashing the diff). - uv run ruff check -> clean. - uv run python scripts/audit_sources.py -> 322 sources, 0 issues, 0 unsafe new endpoints. Version bump: app/__init__.py, pyproject.toml, package.json, package-lock.json, uv.lock -> 0.1.1.Address read-only review findings for the v0.1.1 implementation. Smoke test - tests/test_catalog.py::test_v0_1_1_newsletter_endpoints_return_usable_feed: * Any single broken endpoint now fails the test (previously tolerated one silent failure, contradicting the docstring). * Marked with the new 'network' marker so the live network test is opt-in (-m 'not network' excludes it from the default suite). - pyproject.toml registers the 'network' pytest marker. Article-date helper - app/backend/adapters/article_dates.py: * Drop unused parse_datetime and its tests; no production caller. * Drop unreachable _MIN_YEAR/_MAX_YEAR bounds (URL patterns already anchor the year to 20\d{2}); datetime() rejects impossible dates. * published_from_url now falls through to the next pattern when a regex match produces an invalid calendar date, so a URL carrying both an invalid slash-date and a valid dash-date returns the valid date instead of None. * Module docstring notes that linked-HTML parsing is intentionally out of scope until the bounded linked-page fetcher exists. Tests - tests/test_article_dates.py: * Remove the four assert-True placeholder tests for the dropped published_from_html helper. * Add test_published_from_url_falls_through_to_later_pattern for the invalid-calendar-date fall-through behavior. * Replace the past-date 'within grace window' test with two dynamic tests that actually exercise the future-within-grace accept path and the beyond-grace reject path. * Drop the four parse_datetime tests. Topics - app/backend/presentation/topics.py: add Information Retrieval to the top-level topic enumeration in the module docstring. Docs - docs/source-catalog.md: * Replace the static adapter/language enumerations with pointers to VALID_ADAPTERS / VALID_LANGUAGES so the table stays in sync with the validator. * Add a v0.1.1 section that records the new Information Retrieval topic and the seven new catalog rows (arxiv-cs-ir plus six AI newsletters), including that they have empty original_rows because they were vetted from the Finxter list. Verification - uv run ruff check -> clean. - uv run pytest tests/ -q -m 'not network' -> 747 passed. - uv run pytest tests/test_catalog.py::test_v0_1_1_newsletter_endpoints_return_usable_feed -> passes in isolation; opt-in for full CI runs.A feed item whose published/updated value parses to year 9999 (e.g. sentinel 9999-12-31) is now clamped to a sane bound before persistence. Without this guard the calendar picker surfaces a 9999 archive year tab, the daily edition can render entries from the year 9999, and any /?date=9999-12-31&calendar=9999-12 URL returned 500 because date math overflowed when picking the next month. Shared helper - app/backend/adapters/article_dates.py: * New sanitize_published_at clamps any datetime more than the 14-day future grace window past now to now, normalizes naive datetimes to UTC, and returns now when the input is None. Adapter boundary - apply sanitize_published_at at the parse/persist boundary in: * app/backend/adapters/feed.py * app/backend/adapters/pythonhub_digest.py * app/backend/adapters/jvm_weekly.py * app/backend/adapters/this_week_in_rust.py * app/backend/adapters/pycoders_weekly.py * app/backend/adapters/python_weekly.py (issue-slug dates too) * app/backend/adapters/tanstack_blog.py * app/backend/adapters/javacodegeeks.py * app/backend/adapters/substack.py * app/backend/adapters/anthropic.py - app/backend/storage/entries.py::upsert_entries / persist_source_outcome: replace c.published_at = sanitize(...) with c = replace(c, published_at=sanitize(...)) so the frozen EntryCandidate dataclass stays immutable. Read-side hardening - app/backend/queries/helpers.py: * New ARCHIVE_FUTURE_GRACE_DAYS constant. * Tighten parse_selected_date and parse_calendar_month to reject year > 2099 (was 9999). - app/backend/queries/digest.py: * available_archive_months and available_archive_counts filter published_at <= now + grace so a stray 9999 row cannot surface a phantom archive year or day bucket. * adjacent_archive_day adds the same ceiling so the next/prev day buttons never point at a sentinel date. Migration 010 (backfill) - app/backend/migrations/010_future_date_sanitization.sql: rewrite published_at to last_seen_at for entries whose date sits more than 14 days past last_seen_at. Idempotent and safe. Tests - tests/test_article_dates.py: sanitize_published_at covers sentinel, recent, grace-window, out-of-window, None, and naive inputs. - tests/test_migration_010.py: covers the backfill's sentinel replacement, idempotency, recent-date pass-through, and the within-grace-window pass-through. - tests/test_adjacent_archive_day.py: covers archive months, archive counts, parse-selected-date, parse-calendar-month, and adjacent archive day all rejecting the 9999 sentinel. - tests/test_shell.py: cover /?date=9999-12-31&calendar=9999-12 and /?date=2099-12-31 returning a non-500 status. Verification - uv run ruff check -> clean. - uv run pytest tests/ -q -m 'not network' -> 762 passed (was 747). - pre-existing calendar redirect test remains the only known failure; not related to this change.- Default-hide GitHub sources; enable via ?github=1 across pages, partials, canonical URLs, and RSS feeds. - Wire topic enrichment into upsert_entries and persist_source_outcome so entries gain multi-topic context (Django, FastAPI, React, Postgres, MongoDB, AI, IR) without duplicating primary topics. - Add ranking badges to article cards, stream rows, and search rows with first-3-visible + overflow reveal. - Surface source quality on /status with green/amber/degraded health based on success ratio and last-success freshness. - Add RSS 2.0 feeds for /feed.xml, /topics/{slug}/feed.xml, and /streams/{stream}/feed.xml. - Mitigate Unsloth 403 by warming the blog index before the sitemap request. - Normalize Calendar active palette to use accent-soft + ink-strong, matching the Topics dropdown. - Bump version to 0.2.0 across app, pyproject, package, uv.lock.