Improvements v0.0.8 #9

Merged

rdenadai merged 7 commits from improvements-v0.0.8 into main 2026-08-26 21:54:28 +00:00

Conversation 0 Commits 7 Files changed 10 +1509 -18

rdenadai commented 2026-08-26 20:37:03 +00:00

Owner

Copy link

Summary

Adapters

• app/backend/adapters/tanstack_blog.py: indexes https://tanstack.com/blog, extracts /blog/<slug> anchors with a host allowlist, enriches each post with og:title / article:published_time / og:image (and JSON-LD fallbacks).
• app/backend/adapters/grafana_blog.py: indexes https://grafana.com/blog/, strips tracking query and .html, enriches with the same meta-tag set.
• Both adapters drop candidates whose publish date cannot be parsed instead of writing datetime.now(UTC), so storage no longer rewrites the rank of older posts with the current timestamp on every poll.
• Both adapters honor source.endpoint with the host allowlist and pass through settings.worker_fetch_concurrency so the article enrichment pool respects the configured worker budget.

Worker

• app/backend/services/worker.py routes html_tanstack and html_grafana through _adapter_outcome using the same error shape as the Anthropic and Substack adapters.

Catalog

• Active: tanstack-blog, grafana-blog.
• Reactivated after live probe: docker-blog, last-week-in-aws, baeldung. Their public RSS feeds parse cleanly today.
• Fallback: gh-aws-sdk-go-v2 switched to commits/main.atom after releases.atom started returning GitHub 502/503 to the worker.
• Retained as unsupported (with reasons): discuss-python (umbrella feed pollutes community), javarevisited (FeedBurner HTTPS->HTTP downgrade blocked), google-ai-blog (client-rendered origin, off-topic RSS), hf-papers (401), substack-technology (no public RSS / category data).

CI

• .forgejo/workflows/ci.yml: runs Ruff, pytest, and the asset build on PRs and main pushes against the ubuntu-latest runner. Per-ref concurrency group cancels stale runs on re-push.

## Summary ### Adapters - **`app/backend/adapters/tanstack_blog.py`**: indexes `https://tanstack.com/blog`, extracts `/blog/<slug>` anchors with a host allowlist, enriches each post with `og:title` / `article:published_time` / `og:image` (and JSON-LD fallbacks). - **`app/backend/adapters/grafana_blog.py`**: indexes `https://grafana.com/blog/`, strips tracking query and `.html`, enriches with the same meta-tag set. - Both adapters drop candidates whose publish date cannot be parsed instead of writing `datetime.now(UTC)`, so storage no longer rewrites the rank of older posts with the current timestamp on every poll. - Both adapters honor `source.endpoint` with the host allowlist and pass through `settings.worker_fetch_concurrency` so the article enrichment pool respects the configured worker budget. ### Worker - `app/backend/services/worker.py` routes `html_tanstack` and `html_grafana` through `_adapter_outcome` using the same error shape as the Anthropic and Substack adapters. ### Catalog - **Active**: `tanstack-blog`, `grafana-blog`. - **Reactivated after live probe**: `docker-blog`, `last-week-in-aws`, `baeldung`. Their public RSS feeds parse cleanly today. - **Fallback**: `gh-aws-sdk-go-v2` switched to `commits/main.atom` after `releases.atom` started returning GitHub 502/503 to the worker. - **Retained as unsupported** (with reasons): `discuss-python` (umbrella feed pollutes community), `javarevisited` (FeedBurner HTTPS->HTTP downgrade blocked), `google-ai-blog` (client-rendered origin, off-topic RSS), `hf-papers` (401), `substack-technology` (no public RSS / category data). ### CI - `.forgejo/workflows/ci.yml`: runs Ruff, pytest, and the asset build on PRs and main pushes against the `ubuntu-latest` runner. Per-ref concurrency group cancels stale runs on re-push.

rdenadai added 3 commits 2026-08-26 20:37:03 +00:00

feat(ingest): v0.0.8 html adapters for TanStack and Grafana + retirements ... c4a5bc406d

- Add html_tanstack adapter for https://tanstack.com/blog that
  indexes /blog/<slug> posts and enriches each with the article
  page's <h1> title, <time datetime> publish date, and
  <meta property=og:image> image. Falls back to slug-as-title and
  now() when the per-article fetch fails so a transient outage
  on one article does not zero out a successful index scan.
- Add html_grafana adapter for https://grafana.com/blog/ that
  indexes /blog/<slug>?pg=blog posts, stripping the tracking query
  and any trailing .html suffix. Enriches each with
  <meta property=og:title>, <meta property=article:published_time>,
  and <meta property=og:image>.
- Both adapters reuse _clip, _strip_html, _stable_id, MAX_TITLE,
  and MAX_URL from app.backend.adapters.feed. Per-article fetches
  run in a bounded ThreadPoolExecutor capped at
  min(MAX_ENTRIES, worker_fetch_concurrency) workers.
- Wire both adapters into services/worker.py::_adapter_outcome
  alongside the Substack and Anthropic branches, reusing the
  bounded HTTP client and SSRF defenses.
- Catalog: tanstack-blog and grafana-blog are active with the new
  adapters. Retirements with descriptive reasons:
  discuss-python (umbrella /latest.rss pollutes community surface),
  javarevisited (FeedBurner only redirects http and the safety
  policy blocks https->http downgrades),
  google-ai-blog (client-rendered origin, no stable post pattern,
  RSS endpoint 404 to worker),
  baeldung (client-rendered origin, RSS 403 to worker).
- Tests: parser fixtures, empty-body adapter_error, query-stripping,
  .html strip, title clipping, fallback to slug-as-title, worker
  integration, catalog assertions for the four retirements and the
  two new active rows.

fix(ingest): v0.0.8 correct html adapter URLs, drop undated posts, reactivate clean feeds ... 995893dac2

- app/backend/adapters/tanstack_blog.py:
  * post URL is now https://tanstack.com/blog/<slug>, not the duplicated
    https://tanstack.com/blog/blog/<slug> produced by the previous
    rstrip('/')/concat build
  * new host allowlist (tanstack.com, www.tanstack.com); external /blog/<slug>
    anchors on other hosts are rejected before slug extraction
  * honor source.endpoint with the same allowlist so a future catalog
    row can override the canonical index URL
  * parse og:title, article:published_time, og:image, with <h1>, JSON-LD
    headline, <time datetime>, and JSON-LD datePublished fallbacks; the
    JSON-LD parser ignores entries that lack the keys we need so it does
    not return the org-level @graph stub that ships first
  * drop candidates whose publish date cannot be parsed instead of
    falling back to datetime.now(UTC), so storage no longer rewrites the
    rank of older posts with the current timestamp on every poll
  * external_id is now the canonical URL, which keeps EntryCandidate.id
    stable when only the title or image changes

- app/backend/adapters/grafana_blog.py mirrors the same fix set:
  * strict endpoint validation against the Grafana host allowlist
  * og:title / article:published_time / og:image with JSON-LD fallbacks
  * drop undated candidates; external_id is the canonical URL
  * the regex set now accepts single-quoted attribute values

- app/backend/services/worker.py: html_tanstack and html_grafana pass
  settings.worker_fetch_concurrency so the article enrichment pool
  respects the configured worker budget.

- config/sources.csv:
  * reactivate docker-blog, last-week-in-aws, baeldung; their public
    RSS feeds respond 200 today and parse cleanly into Java / devops /
    aws topic coverage
  * gh-aws-sdk-go-v2 falls back to the commits/main.atom feed after the
    releases.atom endpoint started returning GitHub 502/503 to the
    worker
  * discuss-python, javarevisited, google-ai-blog, hf-papers,
    substack-technology stay retired: their surviving feeds are
    polluted with off-topic content or no longer reachable

- tests/test_tanstack_blog_adapter.py + tests/test_grafana_blog_adapter.py:
  * exact URL assertions
  * external-host anchor rejection
  * JSON-LD headline / datePublished fallback
  * candidate drop when the article body lacks a publish date
  * validate_endpoint() positive and negative cases
  * existing parser fixtures still pass

- tests/test_worker.py: html routing tests now key candidates by URL
  and assert articles without publish_date are dropped end-to-end.

- tests/test_catalog.py: retired set shrinks to the five sources that
  are still actually unreachable; v0.0.8 reactivation test asserts the
  four feeds and gh-aws-sdk-go-v2 commit feed endpoint.

ci(forgejo): run lint, pytest, and asset build on PRs and main ... 509780f3f3

Triggers on pull_request (opened/synchronize/reopened) and on push to
main. Runs on the 'docker' label with the node:22-bookworm container,
installs uv, then mirrors the local verification loop: uv sync,
npm ci, ruff check, pytest, npm run build:assets.

Uses a per-ref concurrency group so re-pushed PR branches and
follow-up pushes to main cancel the previous run instead of stacking
two jobs on the same tree.

rdenadai added 1 commit 2026-08-26 20:44:15 +00:00

ci(forgejo): mirror sigil runner setup ... fd4f7c8a2b

Switch the CI job to the runner pattern already in use on the sigil
repository: ubuntu:22.04 container, Node 22 installed via the NodeSource
apt repository, uv 0.8.17 via pip, actions/checkout@v5, and
uv sync --locked --extra dev.

This drops the manual astral.sh installer step and the homebrew uv
fallback, both of which assumed internet egress and a writable PATH
that the runner container does not always provide.

rdenadai added 1 commit 2026-08-26 20:46:14 +00:00

ci(forgejo): install pip before uv in the ubuntu container ... ffb7f77748

The ubuntu:22.04 container image used by the runner does not ship
pip, so 'pip install uv' fails with 'pip: command not found'. Install
python3-pip from apt and use --break-system-packages so the rest of
the pipeline can install uv and other Python tooling.

rdenadai added 1 commit 2026-08-26 21:21:48 +00:00

ci(forgejo): install uv via astral.sh instead of pip ... 257c7f9bf5

The ubuntu:22.04 container ships pip 22.0.2, which predates the
--break-system-packages flag, so 'pip install --break-system-packages
uv' fails with 'no such option'. Drop the apt-installed pip entirely
and bootstrap uv directly with the official astral.sh installer.

rdenadai added 1 commit 2026-08-26 21:25:31 +00:00

ci(forgejo): build frontend assets before pytest ... 9ed72358f0

tests/test_shell.py and tests/test_deployment_paths.py read the
generated app.css, favorites.js, chunks.js, and tex-chtml.js from
app/frontend/static/{dist,vendor}/*. They were failing in CI because
the workflow ran pytest before npm run build:assets.

Reorder so the asset build runs after npm ci and before lint and
pytest. Pytest now finds the built artefacts.

rdenadai merged commit f1181cf641 into main 2026-08-26 21:54:28 +00:00

rdenadai referenced this pull request from a commit 2026-08-26 21:54:28 +00:00

Merge pull request 'Improvements v0.0.8' (#9) from improvements-v0.0.8 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rdenadai/tilens!9

No description provided.