rdenadai/tilens
rdenadai/tilens
Watch
1
0
Fork
You've already forked tilens
 0
Code Pull requests Activity Actions

Improvements v0.0.7 #8

Merged
rdenadai merged 6 commits from improvements-v0.0.7 into main 2026-08-26 14:25:46 +00:00
Conversation 0 Commits 6 Files changed 26 +1174 -99
rdenadai commented 2026-08-26 14:25:32 +00:00
Owner
Copy link

Summary

Improvements v0.0.7: repair the broken source catalog, reactivate the Anthropic news source via a new adapter, fix the User-Agent string, make /sources show a two-band visual flag for failing cards, and stop the source filter chips from silently hiding the page body.

Ingestion

  • New anthropic adapter parses the Next App Router hydration embedded in https://www.anthropic.com/news and yields EntryCandidate records. anthropic-blog is reactivated (was unsupported).
  • Real User-Agent string built from app.__version__ and the public base URL, replacing the stale tilens.example placeholder in the core HTTP client and the GitHub / Hacker News adapters.
  • YouTube adapter derives the handle from the curated endpoint URL first, falling back to the slug; fixes youtube-docker resolving @docker instead of @dockertv.
  • HTTP client / HN / GitHub outbound UA tests, YouTube fetch-level regression test, and outbound UA tests for the GitHub and HN adapters.

Catalog

  • New rows: pyfound-blog, discuss-python-ideas, discuss-python-peps (all Python).
  • Endpoint repairs: discuss-python, planet-python, rust-users-forum, rust-internals, google-ai-blog, javarevisited, tanstack-blog, grafana-blog.
  • discuss-python reclassified from news to community since /latest.rss polls every Discourse category.
  • cisa-advisories reactivated through the public cisagov/CSAF GitHub mirror.
  • Retired with reason: docker-blog, last-week-in-aws, hf-papers, substack-technology (no public RSS or API; arXiv already covers the AI material).
  • VALID_ADAPTERS and ADAPTER_REQUIRES_ENDPOINT updated to admit the new anthropic adapter.
  • Catalog smoke test probes every repaired active endpoint for HTTP 200 plus parseable candidates.

Sources page UI

  • Always render one group per filter tab; empty groups show "No sources in this state yet." instead of dropping the body.
  • Two-band visual flag:1-2 failures render amber,3+ failures or any streak without a recorded success render red, with a soft red wash, bold red consec-fails number, and a small alert pip.
  • aria-label="failing" on failing cards plus data-fail-state for testability.
  • Filter script falls back to active when no chip matches.
  • Regression tests lock the topic-chip navigation at every entry point (/?topic=..., /topics/<slug>, /topics index).
## Summary Improvements v0.0.7: repair the broken source catalog, reactivate the Anthropic news source via a new adapter, fix the User-Agent string, make `/sources` show a two-band visual flag for failing cards, and stop the source filter chips from silently hiding the page body. ## Ingestion - New `anthropic` adapter parses the Next App Router hydration embedded in `https://www.anthropic.com/news` and yields `EntryCandidate` records. `anthropic-blog` is reactivated (was `unsupported`). - Real `User-Agent` string built from `app.__version__` and the public base URL, replacing the stale `tilens.example` placeholder in the core HTTP client and the GitHub / Hacker News adapters. - YouTube adapter derives the handle from the curated endpoint URL first, falling back to the slug; fixes `youtube-docker` resolving `@docker` instead of `@dockertv`. - HTTP client / HN / GitHub outbound UA tests, YouTube fetch-level regression test, and outbound UA tests for the GitHub and HN adapters. ## Catalog - New rows: `pyfound-blog`, `discuss-python-ideas`, `discuss-python-peps` (all Python). - Endpoint repairs: `discuss-python`, `planet-python`, `rust-users-forum`, `rust-internals`, `google-ai-blog`, `javarevisited`, `tanstack-blog`, `grafana-blog`. - `discuss-python` reclassified from `news` to `community` since `/latest.rss` polls every Discourse category. - `cisa-advisories` reactivated through the public `cisagov/CSAF` GitHub mirror. - Retired with reason: `docker-blog`, `last-week-in-aws`, `hf-papers`, `substack-technology` (no public RSS or API; arXiv already covers the AI material). - `VALID_ADAPTERS` and `ADAPTER_REQUIRES_ENDPOINT` updated to admit the new `anthropic` adapter. - Catalog smoke test probes every repaired active endpoint for HTTP 200 plus parseable candidates. ## Sources page UI - Always render one group per filter tab; empty groups show "No sources in this state yet." instead of dropping the body. - Two-band visual flag:1-2 failures render amber,3+ failures or any streak without a recorded success render red, with a soft red wash, bold red consec-fails number, and a small alert pip. - `aria-label="failing"` on failing cards plus `data-fail-state` for testability. - Filter script falls back to `active` when no chip matches. - Regression tests lock the topic-chip navigation at every entry point (`/?topic=...`, `/topics/<slug>`, `/topics` index).
- Bump version to 0.0.7
- Real User-Agent string built from app version and public base URL,
  replacing the stale tilens.example placeholder in core http and the
  GitHub/HN adapters
- YouTube adapter derives the handle from the curated endpoint URL and
  only falls back to the slug, fixing youtube-docker
- Catalog repairs: discuss-python switches to /latest.rss after the news
  category disappeared; planet-python and rust users/internals use https
  and current Discourse paths; google-ai-blog points to the new blog
  feed; javarevisited uses feedburner over https; tanstack-blog and
  grafana-blog use index.xml
- Add pyfound-blog, discuss-python-ideas, discuss-python-peps rows
- Tests for the User-Agent contract, YouTube handle derivation, and
  catalog repair expectations
Review fixes:
- tanstack-blog now points to https://tanstack.com/rss.xml
- rust-internals now points to https://internals.rust-lang.org/c/language-design/21.rss
  (previous ideas-deprecated feed was 2018-stale)
- discuss-python reclassified from news to community since /latest.rss
  pulls from every Discourse category
- _handle_for_source now typed with SourceRecord and accesses fields
  directly

Anthropic support:
- New anthropic adapter parses Next App Router hydration in /news HTML
  to extract _type:post records; produces EntryCandidate list using
  title, slug.current, publishedOn, summary
- anthropic-blog source reactivated as active with adapter anthropic,
  endpoint https://www.anthropic.com/news, topics ai-labs|ai
- Worker routes adapter=anthropic through fetch_news and surfaces
  adapter errors via fetch_source

Tests:
- Anthropic parser fixture tests and worker integration tests
- Outbound UA tests for GitHub and Hacker News adapters
- Fetch-level YouTube regression test verifying @dockertv handle from
  endpoint overrides @docker from slug
- Strengthened USER_AGENT assertion against app version and settings
- Catalog smoke test probing every repaired active endpoint for HTTP
  200 + parseable candidates, plus anthropic-blog activation check
- Repaired endpoints use https-only check
After running v0.0.7 locally, 191 of 196 active sources succeed. The
remaining five cannot be ingested from the worker's egress and have no
public RSS or API alternative:

- cisa-advisories: HTTP 403 from origin to worker egress
- docker-blog: HTTP 403 from origin to worker egress
- last-week-in-aws: HTTP 403 from origin to worker egress
- hf-papers: feed endpoint retired, /papers page is client-rendered
- substack-technology: no public Substack category RSS or API

Each row keeps a non-empty reason explaining the retirement, so the
public sources page surfaces the why instead of silently disappearing.

Tests:
- Drop hf-papers from test_ai_sources_declared_subtopics since the
  slug is no longer in report.active
- Add test_retired_blocked_sources_have_non_empty_reason asserting the
  five retired slugs carry a reason
Local compose verified the cisa-advisories feed at
https://www.cisa.gov/cybersecurity-advisories/all.xml still 403s to the
worker egress. Switch the source to the public cisagov/CSAF GitHub
mirror commit Atom feed, which is reachable from the worker and still
serves CISA's advisory publication signal.

Docker blog has no GitHub mirror and the worker egress hits HTTP 403 on
the entire docker.com domain; retire it again with an explicit reason.

Tests:
- Update test_retired_blocked_sources_have_non_empty_reason to include
  docker-blog alongside last-week-in-aws, hf-papers, substack-technology
- Add test_reactivated_sources_use_github_or_known_feeds asserting the
  cisa-advisories endpoint matches the GitHub mirror
The /sources filter chips used to silently hide every group when an
operator clicked an empty filter (Duplicate/Blocked/Retired), and
nothing distinguished a fresh failure from a long-running outage.

Backend:
- classify_failures() picks fail_state as "degraded" for 1-2 failures
  with a known last success and "failing" for 3+ failures or any
  failure streak without a recorded success
- build_source_directory now stamps fail_state on each record, returns
  one group per DIRECTORY_FILTERS entry with empty: bool, and validates
  the requested filter against DIRECTORY_FILTER_KEYS (was passing on
  unknown status strings)
- source_view() exposes fail_state on the detail page

Frontend:
- Source card renders tl-source-card--{degraded,failing} and the
  matching data-fail-state attribute plus an aria-label on failing
- A small tl-source-card__alert pip and "Failing" label appear on
  failing cards
- Every filter tab now matches a group; empty groups render
  "No sources in this state yet." so /sources?filter=blocked etc. no
  longer drop the page body
- activate() falls back to "active" when no tab has aria-pressed

CSS (two-band visual flag):
- Degraded: amber left border + amber consec-fails number
- Failing: red left border + soft red wash + bold red consec-fails
  number + the alert pip
- Empty-state copy uses muted dashed border

Tests:
- test_classify_failures_thresholds covers all six success/failure
  combinations
- test_build_source_directory_exposes_fail_state asserts every active
  record carries fail_state and every DIRECTORY_FILTERS key is
  represented in tabs and groups
- test_sources_page_marks_failing_and_degraded_cards asserts the
  rendered HTML contains the new CSS hooks and the empty-state block
  for filters with zero sources
The topic filter is plain anchor navigation (no JS), but until now no
test asserted the operator-visible behavior end-to-end:

- test_topic_chip_filters_digest_at_root_route hits /?topic=python,
  /?topic=rust, and the main catalog topic slugs to confirm the
  eyebrow text reflects the active topic and that no Python content
  leaks into the Rust filter (or vice versa).
- test_topics_index_links_match_topic_slug_targets scrapes every chip
  on /topics, asserts each href is unique, and hits it server-side to
  catch future bugs where a chip links to a stale or invalid slug.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rdenadai/tilens!8
No description provided.