Improvements v0.0.9 #11

Merged

rdenadai merged 6 commits from improvements/v0.0.9 into main 2026-08-30 01:10:47 +00:00

Conversation 0 Commits 6 Files changed 28 +5173 -170

rdenadai commented 2026-08-29 03:52:12 +00:00

Owner

Copy link

Summary

Custom parsers for four newsletter/news sources that previously emitted one entry per issue (or were broken). Each adapter now yields one entry per linked article so the timeline shows the actual news items.

Sources

Slug	Adapter	Endpoint	Notes
pycoders-weekly	html_pycoders_weekly	https://pycoders.com/feed	switched from generic feed
python-weekly	html_python_weekly	https://pythonweekly.com/	switched from broken feed
jvm-weekly	html_jvm_weekly	https://www.jvm-weekly.com/feed	new
java-code-geeks	feed_javacodegeeks_filtered	https://www.javacodegeeks.com/feed	new, drops [DEALS]/[FREE EBOOKS]/Meta JCG

Adapters

• html_pycoders_weekly — fetches latest 3 issues from RSS, parses issue HTML, resolves pycoders.com/link/<id>/web redirects via HEAD/GET, drops <p class="sponsor"> blocks and unresolved internal redirects, keeps external final_url even when the destination returns 429, checks subfetch HTTP status.
• html_python_weekly — walks pythonweekly.com/ archive, fetches each /p/python-weekly-issue-*, keeps Articles, Tutorials and Talks and Interesting Projects, Tools, and Libraries (h5 sections) while dropping events/sponsors/newsletter footers (h5 sections) and h6 article titles, strips utm_* / _bhiiv / bhcl_* / __... query params via urllib.parse.
• html_jvm_weekly — parses Substack content:encoded, keeps external article anchors, drops publisher/Substack/social/CDN hosts, strips tracking query params.
• feed_javacodegeeks_filtered — generic RSS parser with title pattern ([DEALS], [FREE EBOOKS]) and category (Meta JCG, Deals, Offers, eBooks) filters.

Errors and observability

When every issue subfetch returns an unhealthy HTTP status, both Python Weekly and PyCoder's now return error_class="issue_fetch_error" instead of a misleading success with zero candidates.

Wiring

• Added adapter IDs to app/backend/services/catalog.py (VALID_ADAPTERS, ADAPTER_REQUIRES_ENDPOINT) and app/backend/services/worker.py (adapter set + dispatch blocks).
• Updated config/sources.csv to switch pycoders-weekly and python-weekly to their new adapters and added jvm-weekly and java-code-geeks.

Tests

• New adapter unit tests in tests/test_pycoders_weekly_adapter.py, tests/test_python_weekly_adapter.py, tests/test_jvm_weekly_adapter.py, tests/test_javacodegeeks_adapter.py.
• Saved live Beehiiv fixture at tests/fixtures/python_weekly_issue.html for regression coverage.
• Worker dispatch tests for all new routes, including top-level HTTP error and FetchError propagation and issue-fetch error surfacing.
• tests/test_catalog.py::test_v0_0_9_newsletter_html_adapters_are_active.

## Summary Custom parsers for four newsletter/news sources that previously emitted one entry per issue (or were broken). Each adapter now yields one entry per linked article so the timeline shows the actual news items. ## Sources | Slug | Adapter | Endpoint | Notes | | --- | --- | --- | --- | | `pycoders-weekly` | `html_pycoders_weekly` | `https://pycoders.com/feed` | switched from generic feed | | `python-weekly` | `html_python_weekly` | `https://pythonweekly.com/` | switched from broken feed | | `jvm-weekly` | `html_jvm_weekly` | `https://www.jvm-weekly.com/feed` | new | | `java-code-geeks` | `feed_javacodegeeks_filtered` | `https://www.javacodegeeks.com/feed` | new, drops `[DEALS]`/`[FREE EBOOKS]`/Meta JCG | ## Adapters - **`html_pycoders_weekly`** — fetches latest 3 issues from RSS, parses issue HTML, resolves `pycoders.com/link/<id>/web` redirects via HEAD/GET, drops `<p class="sponsor">` blocks and unresolved internal redirects, keeps external `final_url` even when the destination returns 429, checks subfetch HTTP status. - **`html_python_weekly`** — walks `pythonweekly.com/` archive, fetches each `/p/python-weekly-issue-*`, keeps `Articles, Tutorials and Talks` and `Interesting Projects, Tools, and Libraries` (h5 sections) while dropping events/sponsors/newsletter footers (h5 sections) and h6 article titles, strips `utm_*` / `_bhiiv` / `bhcl_*` / `__...` query params via `urllib.parse`. - **`html_jvm_weekly`** — parses Substack `content:encoded`, keeps external article anchors, drops publisher/Substack/social/CDN hosts, strips tracking query params. - **`feed_javacodegeeks_filtered`** — generic RSS parser with title pattern (`[DEALS]`, `[FREE EBOOKS]`) and category (`Meta JCG`, `Deals`, `Offers`, `eBooks`) filters. ## Errors and observability When every issue subfetch returns an unhealthy HTTP status, both Python Weekly and PyCoder's now return `error_class="issue_fetch_error"` instead of a misleading success with zero candidates. ## Wiring - Added adapter IDs to `app/backend/services/catalog.py` (`VALID_ADAPTERS`, `ADAPTER_REQUIRES_ENDPOINT`) and `app/backend/services/worker.py` (adapter set + dispatch blocks). - Updated `config/sources.csv` to switch `pycoders-weekly` and `python-weekly` to their new adapters and added `jvm-weekly` and `java-code-geeks`. ## Tests - New adapter unit tests in `tests/test_pycoders_weekly_adapter.py`, `tests/test_python_weekly_adapter.py`, `tests/test_jvm_weekly_adapter.py`, `tests/test_javacodegeeks_adapter.py`. - Saved live Beehiiv fixture at `tests/fixtures/python_weekly_issue.html` for regression coverage. - Worker dispatch tests for all new routes, including top-level HTTP error and `FetchError` propagation and issue-fetch error surfacing. - `tests/test_catalog.py::test_v0_0_9_newsletter_html_adapters_are_active`.

rdenadai added 2 commits 2026-08-29 03:52:12 +00:00

feat(ingest): v0.0.9 expand digest and add personal/AI sources ... 7746d2f735

- Python Hub (html_pythonhub): expand weekly digest RSS into per-link entries
- OCaml Weekly News (html_ocaml_cwn): expand CWN RSS into per-section entries
- Unsloth blog (html_unsloth): sitemap + OG meta extraction for blog posts
- New rdenadai.dev feed source (python|ai|general)
- New Towards Data Science feed source (ai-research|ai|python)
- Bump version to 0.0.9

feat(ingest): v0.0.9 lobsters adapter and github commit-feed cleanup ... aa65340193

Lobsters:
- New html_lobsters adapter parses https://lobste.rs/ into one entry per
  story with original article URL, discussion URL, score, comment count,
  raw Lobsters tags preserved in entry.extra.
- Catalog tags only enter EntryCandidate.extra_topics when they match
  the curated TOPIC_LABELS set so the entry_topics topic_slug FK is
  never violated by Lobsters-side tags.
- Title-link parser handles HTML attribute order; HTML entities are
  decoded; timezones preserved as UTC offsets.

GitHub commit feeds:
- Mark vue-changelog, cisa-advisories, gh-aws-sdk-go-v2 unsupported
  with reason about commit feeds vs tags/releases.
- react-blog: switch endpoint from commits atom to react.dev/rss.xml.
- Add catalog guard test that active GitHub repo feeds must end in
  /releases.atom or /tags.atom.

Tests added: 16 Lobsters adapter tests and one storage FK safety test.

rdenadai added 4 commits 2026-08-30 01:07:05 +00:00

feat(ingest): v0.0.9 expand GitHub repos catalog ... eb79d094d7

Add 109 active GitHub /releases.atom sources across existing topics:
- Python/FastAPI/Django: pytest, flask, requests, poetry, sqlalchemy,
  celery, scrapy, httpx, DRF, manim, opencv, home-assistant, rlm,
  python-prompt-toolkit, anthropic-sdk-python, openai-python, toml
- Rust: serde, rust-clippy, mdBook, reqwest, hyper, diesel, tauri,
  proptest, wry, blitz, cuda-oxide, turbovec, llmfit, markdown-rs
- JavaScript/Web: next.js, vite, svelte, angular, webpack, eslint,
  pnpm, react-router, redux-toolkit, nuxt, nest, electron, three.js,
  d3, axios, deno, tailwindcss, playwright, opencode
- Go: kubernetes, gin, cobra, gorm, grafana, terraform, helm, cockroach
- Java: spring-boot, kafka, elasticsearch, netty, dubbo
- Databases: prisma, redis, cassandra, clickhouse
- AI/AI-tools: tensorflow, transformers, ollama, vllm, llama.cpp,
  mlflow, ray, jax, keras, whisper, unsloth, comfyui, langflow,
  open-webui, autogpt, openai-codex, gemini-cli, spec-kit, deepseek,
  ornith, openclaw, superpowers, skills catalogs, cep
- DevOps/Architecture: docker-compose, docker-buildx, argo-cd, istio,
  envoy, dapr, otel-collector, sentry
- DevSec: trivy, semgrep, osquery, grype

All sources use /releases.atom, verified >= 1k GitHub stars.
audit_sources.py: 303 active sources, no duplicate active endpoints.

fix(catalog): retire empty GitHub release feeds and normalize line endings ... cc53554a8f

Mark five active GitHub /releases.atom sources as unsupported after live
verification showed both /releases.atom and /tags.atom return empty Atom
documents (skills catalogs distribute via direct commits):
  - gh-ornith, gh-anthropic-skills, gh-openai-skills,
    gh-google-skills, gh-minimax-skills

Rename gh-deepseek-v3 / gh-deepseek-r1 display names from 'Tags' to
'Releases' since the rows use /releases.atom.

Normalize config/sources.csv line endings to LF (the previous mix of
CRLF and LF was hiding the structural diff as 166 unrelated rewrites).

Add tests/test_catalog.py::test_v0_0_9_added_github_repo_feeds_return_entries
to assert every newly active GitHub release source produces at least one
candidate when fetched live, and to keep the empty-feed skills repos
out of the active set.

Verification: 629 tests pass; audit_sources.py clean.

feat(ingest): v0.0.9 newsletter article adapters ... c3bd17cbe6

Add custom HTML/feed adapters that expand newsletter issues into one
entry per linked article instead of one entry per issue.

- html_pycoders_weekly: PyCoder's Weekly issue archive parser. Resolves
  pycoders.com/link/<id>/web redirects via HEAD/GET, drops sponsor
  blocks, drops unresolved internal links, checks subfetch HTTP status.
- html_python_weekly: Python Weekly Beehiiv archive parser. Treats <h5>
  as section delimiter and <h6> as article heading, keeps Articles and
  Interesting Projects sections, drops events/sponsors/newsletter
  footers, strips tracking query params, checks subfetch HTTP status.
- html_jvm_weekly: Substack feed parser. Extracts external article
  anchors from content:encoded, drops publisher/Substack/social/CDN
  hosts, strips tracking via urllib.parse.
- feed_javacodegeeks_filtered: Java Code Geeks feed parser. Drops
  [DEALS]/[FREE EBOOKS] titles and Meta JCG/Deals/Offers/eBooks
  categories.

Wire new adapter IDs into catalog allowlist and worker dispatch, swap
pycoders-weekly and python-weekly to custom adapters, and add new
jvm-weekly and java-code-geeks sources. Includes fixture tests for
h5/h6 markup, redirect 429 handling, /link/ URL guard, and HTTP error
propagation.

test(ingest): cover newsletter article adapter failure paths ... 687d337d83

Add tests for adapter behaviors that were previously silent or only
covered by happy-path fixtures:

- Python Weekly parser emits articles from a saved live-shape Beehiiv
  fixture (h5 sections + h6 article headings + post-h6 description).
- Python Weekly and PyCoder's adapters return error_class
  'issue_fetch_error' when every issue subfetch returns an unhealthy
  HTTP status, instead of a misleading success with zero candidates.
- PyCoder's adapter test asserts no pycoders.com/link/* URLs survive
  resolution.
- Worker dispatch tests cover top-level HTTP 503 propagation for the
  four new adapter routes plus the Python Weekly FetchError path.

rdenadai merged commit 82c225b08c into main 2026-08-30 01:10:47 +00:00

rdenadai referenced this pull request from a commit 2026-08-30 01:10:48 +00:00

Merge pull request 'Improvements v0.0.9' (#11) from improvements/v0.0.9 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rdenadai/tilens!11

No description provided.