Improvements v0.3.0 #15

Merged

rdenadai merged 14 commits from improvements-v0.3.0 into main 2026-09-18 22:06:38 +00:00

Conversation 0 Commits 14 Files changed 26 +2028 -131

rdenadai commented 2026-09-18 20:56:46 +00:00

Owner

Copy link

Carries the worker resilience + proxy + WAF rework on top of the v0.2.0 release. v0.2.0 left 8 sources failing in prod (5 stuck on Cloudflare/CloudFront 403s, Foojay on a retired RSS path, 2 with broken endpoints); this branch addresses the root causes and adds defenses against the same class of failure going forward.

Worker

• Auto-block + 24h recovery probe. After TILENS_AUTO_BLOCK_AFTER_FAILURES consecutive failures with error_class in {http_error, fetch_error, sitemap_http_403, post_fetch_error, adapter_blocked}, the worker flips the row to status='blocked' and stamps last_blocked_at. A separate 24h probe picks the row back up and flips it to active on a 200/304. Operators can tune via TILENS_BLOCKED_PROBE_INTERVAL_SEC.
• Critical fix: sync_catalog now preserves auto-blocked status so the catalog can't undo the worker's block on the next cycle and collapse the 24h cadence to the worker interval (~6h). Migration 011 backfills last_blocked_at on pre-existing editorial-blocked rows.

Proxy

• TILENS_FETCH_PROXY_URL routes worker fetches through an HTTP CONNECT proxy when the egress IP lacks reputation with publisher CDNs. Validated at startup: http scheme, public hostname, port 1..65535, no path/query/fragment. Documented SSRF trade-off: with a proxy set, _validate_target still checks against worker DNS at fetch time; a hostile proxy operator could DNS-spoof destinations. Trust the proxy as you would trust a member of your own team.

WAF headers

• Bumped the Chrome signature 139 -> 142 and added Sec-CH-UA, Sec-CH-UA-Platform, Sec-Fetch-Dest/Mode/Site/User, Accept-Encoding with br/zstd, and Upgrade-Insecure-Requests. Allowlist expanded to osi-blog, mit-news-ai, mit-news-eecs. End-to-end tests pin every signal for every entry.

Catalog

• foojay-today endpoint: /today/feed/ -> /today/feed/atom/ (the old RSS path 302s to /index.xml which itself 404s; WordPress still serves the Atom equivalent).

Code review

Four review rounds covered: probe cadence bug (probe-every-cycle regression), SSRF regression under proxy (httpcore CONNECT tunnel ignores sni_hostname), success-branch duplication, port/path/query validation, threshold clamp, and the sync_catalog auto-block undo defect. All resolved. 270 tests pass.

Out of scope for this PR (follow-up)

• docs drift: ops.md / architecture.md / source-catalog.md predate v0.3.0 features. Can land as a separate docs-only PR.
• Sec-CH-UA shuffle for browser fingerprinting hygiene.
• Probe-only cycle log line undercounts source count (cosmetic).

Carries the worker resilience + proxy + WAF rework on top of the v0.2.0 release. v0.2.0 left 8 sources failing in prod (5 stuck on Cloudflare/CloudFront 403s, Foojay on a retired RSS path, 2 with broken endpoints); this branch addresses the root causes and adds defenses against the same class of failure going forward. ## Worker - **Auto-block + 24h recovery probe.** After TILENS_AUTO_BLOCK_AFTER_FAILURES consecutive failures with error_class in {http_error, fetch_error, sitemap_http_403, post_fetch_error, adapter_blocked}, the worker flips the row to status='blocked' and stamps last_blocked_at. A separate 24h probe picks the row back up and flips it to active on a 200/304. Operators can tune via TILENS_BLOCKED_PROBE_INTERVAL_SEC. - **Critical fix:** sync_catalog now preserves auto-blocked status so the catalog can't undo the worker's block on the next cycle and collapse the 24h cadence to the worker interval (~6h). Migration 011 backfills last_blocked_at on pre-existing editorial-blocked rows. ## Proxy - **TILENS_FETCH_PROXY_URL** routes worker fetches through an HTTP CONNECT proxy when the egress IP lacks reputation with publisher CDNs. Validated at startup: http scheme, public hostname, port 1..65535, no path/query/fragment. Documented SSRF trade-off: with a proxy set, _validate_target still checks against worker DNS at fetch time; a hostile proxy operator could DNS-spoof destinations. Trust the proxy as you would trust a member of your own team. ## WAF headers - Bumped the Chrome signature 139 -> 142 and added Sec-CH-UA, Sec-CH-UA-Platform, Sec-Fetch-Dest/Mode/Site/User, Accept-Encoding with br/zstd, and Upgrade-Insecure-Requests. Allowlist expanded to osi-blog, mit-news-ai, mit-news-eecs. End-to-end tests pin every signal for every entry. ## Catalog - foojay-today endpoint: /today/feed/ -> /today/feed/atom/ (the old RSS path 302s to /index.xml which itself 404s; WordPress still serves the Atom equivalent). ## Code review Four review rounds covered: probe cadence bug (probe-every-cycle regression), SSRF regression under proxy (httpcore CONNECT tunnel ignores sni_hostname), success-branch duplication, port/path/query validation, threshold clamp, and the sync_catalog auto-block undo defect. All resolved. 270 tests pass. ## Out of scope for this PR (follow-up) - docs drift: ops.md / architecture.md / source-catalog.md predate v0.3.0 features. Can land as a separate docs-only PR. - Sec-CH-UA shuffle for browser fingerprinting hygiene. - Probe-only cycle log line undercounts source count (cosmetic).

rdenadai added 12 commits 2026-09-18 20:56:46 +00:00

fix(worker): swap foojay endpoint to atom feed ... f19dc26a5e

The Foojay Today RSS endpoint /today/feed/ now 302s to /index.xml
which itself 404s. WordPress still serves the Atom equivalent at
/today/feed/atom/ (10 fresh entries, full ETag/Last-Modified
support), so swap the catalog row to that URL with a reason note
for the audit trail. Adapter stays as feed; feedparser handles Atom
identically to RSS.

feat(worker): expand WAF-safe request headers and allowlist ... 39ae8835b7

Bump the Chrome signature from 139 to 142 and add the modern
Sec-CH-UA, Sec-Fetch-*, Accept-Encoding (br/zstd), and
Upgrade-Insecure-Requests signals that publishing CDNs key on. MIT
News (ai + eecs) and OSI Blog join docker-blog, last-week-in-aws,
and unsloth-blog in the WAF allowlist so the override path applies
to all five currently-403'd sources. Tests pin the new headers and
each new allowlist slug.

feat(worker): add optional egress proxy support ... e9cba4ceb8

Introduce TILENS_FETCH_PROXY_URL for operators whose egress IP
lacks reputation with publishing CDNs. The worker validates the
URL at startup (http scheme only, host must resolve to a public
address; loopback, private, link-local, and metadata ranges are
rejected to prevent SSRF) and routes every fetch through an HTTP
CONNECT tunnel via httpx's built-in transport. Destination URLs
still pass through _validate_target at fetch time so the worker
cannot be tricked into reaching internal ranges through the proxy.
Credentials ship inline in the URL; the worker never logs them.

Operations doc + .env.example list recommended providers (Webshare,
Bright Data, residential VPS) and the URL shape. Tests pin the
helper, the validation surface, and the transport selection logic.

feat(worker): auto-block sources past failure threshold with 24h probe ... 017712af1c

Add a bounded retry cap so perpetually-failing sources stop
hammering the network. When consecutive_failures crosses
TILENS_AUTO_BLOCK_AFTER_FAILURES (default 25) and the error class
is in {http_error, fetch_error, sitemap_http_403,
post_fetch_error, adapter_blocked}, the storage layer flips the
row to status='blocked', stamps last_blocked_at, clears
next_due_at, and writes a reason explaining the transition.

Each ingestion cycle now also selects blocked sources whose
last_blocked_at is older than TILENS_BLOCKED_PROBE_INTERVAL_SEC
(default 24h) and probes them through the regular fetch pipeline.
A 200/304 result flips the row back to active (with the SQLite
SET expression order chosen so the cleanup CASE runs BEFORE the
status CASE, otherwise SQLite's left-to-right evaluation reads the
already-updated status). Other error classes keep retrying.

New migration 011 adds the last_blocked_at column plus a
partial index on (status, last_blocked_at) WHERE status='blocked'
to keep the probe scan cheap. SourceRow, list_blocked_sources_for_probe,
and the persistence layer carry the field end-to-end. Tests pin
the threshold boundary, the non-blocking error class escape
hatch, the successful probe restore, and the probe-window
skip behavior.

chore(release): bump version to 0.3.0 ... b8ff57027e

Two net-new features (egress proxy support, auto-block + 24h
recovery probe) and a worker behavioral change (WAF signature
upgrade + allowlist expansion) warrant a minor bump per SemVer.
Foojay endpoint swap is a bugfix that rides along. The rendered
app_version picks up the new value automatically from
app.__version__; the PWA service worker cache version stays at
v4 because no frontend assets changed.

fix(worker): refresh probe cooldown on failure and dedupe success-branch ... 796b1ff33c

Two correctness fixes discovered in code review:

1) When a 24h recovery probe runs on an already-blocked source and
   fails, _apply_failure_transition's auto-block UPDATE had a
   WHERE slug=? AND status='active' guard that matched zero rows
   for the already-blocked case. The code fell through to the
   generic failure UPDATE which set next_due_at to the regular
   backoff (~6h) but did NOT advance last_blocked_at. The probe
   selector (last_blocked_at <= cutoff) kept matching every cycle
   so the documented 24h cadence silently degraded to the worker
   cycle interval, hammering persistently-failing sources.

   Drop the status='active' guard so the auto-block UPDATE also
   matches already-blocked rows; the SET clause is idempotent for
   that case and the last_blocked_at refresh restores the 24h
   cooldown. The non-blocking fallback UPDATE now also refreshes
   last_blocked_at when the row is already blocked so a probe
   that returns a non-blocking error class still gates the row
   for the configured probe interval.

2) Extract _apply_success_transition helper. Both
   persist_source_outcome and record_fetch_outcome carried
   byte-identical SET clauses (including the SQLite left-to-right
   CASE ordering for the blocked->active cleanup). Drift risk
   was real; the helper is now called from both sites with the
   CASE ordering comment living in one place.

Other changes:
- list_blocked_sources_for_probe no longer defaults limit=10;
  run_once passes settings.worker_fetch_concurrency so the
  probe SELECT respects the same in-flight cap as active fetches.
- New regression tests:
  - test_run_once_failed_probe_refreshes_block_cooldown locks #1
  - test_run_once_skips_blocked_sources_with_future_last_blocked_at
    pins clock-skew safety
  - test_high_threshold_keeps_source_active pins threshold=999
  - test_invalid_threshold_falls_back_to_default pins clamp behavior

fix(worker): pin destinations through proxy and harden proxy URL validation ... be27d5b474

Two security improvements discovered in code review:

1) SSRF regression under proxy. When TILENS_FETCH_PROXY_URL is set,
   _build_transport was using httpx.HTTPTransport(proxy=...)
   directly, which bypassed _PinningTransport. The proxy was
   resolving the destination hostname on its own, while
   _validate_target only validates against the worker's DNS view.
   DNS-split-horizon / rebinding during the redirect window
   becomes possible.

   _PinningTransport now accepts an optional inner transport
   parameter; _build_transport wraps httpx.HTTPTransport(proxy=...)
   inside _PinningTransport. The pinning transport still rewrites
   the URL host to a worker-validated public IP and preserves the
   original hostname in the SNI extension; the proxy transport
   only owns the CONNECT tunnel to the pinned IP.

2) Proxy URL hardening. _coerce_proxy_url now rejects paths,
   query strings, and fragments (a proxy URL is a tunnel target,
   never a resource address) and validates the port falls in
   1..65535 (port=0 lets the kernel bind an ephemeral port on
   the worker's behalf). The credential-redaction test pins the
   error path; secrets in userinfo must never reach operator logs.

Network helpers (default_resolver, is_public_address,
public_addresses) extracted into app/backend/core/network.py to
break the circular import between config and http without
duplicating the SSRF guard logic. http.resolve_public stays as a
thin wrapper that raises FetchError for back compat with existing
tests.

Tests:
- test_build_transport_wraps_proxy_with_pinning pins #1
- New _coerce_proxy_url tests: path, query, fragment, port=0,
  port=99999, IPv6 loopback, credential redaction
- test_coerce_proxy_url_accepts_userinfo preserved (round-trip
  with credentials intact)

test(worker): end-to-end WAF coverage for new allowlist + source catalog cleanup ... d9a86e7904

The docstring contract for source_headers says adding a slug to
the WAF allowlist requires a regression test that proves the
override path is exercised end-to-end. The new osi-blog,
mit-news-ai, and mit-news-eecs entries only had static-fallback
unit tests; the new Sec-CH-UA + Chrome 142 signals were verified
end-to-end only for docker-blog. Add three sibling tests modeled
on test_fetch_source_injects_waf_headers_for_docker_blog so any
future regression that strips one of the new signals trips every
allowlist entry.

Also clear the foojay-today reason column now that the source is
healthy: the /sources directory renders reason alongside the
status chip and the explanation of the old /today/feed/ RSS
path is no longer useful on an Active row. The v0.3.0 release
that ships this branch carries the audit trail via git history.

uv.lock was not picked up by the version bump commit and is now
synced to 0.3.0.

fix(worker): revert proxy transport wrapping and document SSRF trade-off ... a1fb1c32e5

The previous SSRF hardening wrapped httpx.HTTPTransport(proxy=...)
inside _PinningTransport so the destination hostname would be
resolved against the worker's DNS view and the IP pinned for the
CONNECT tunnel. It broke HTTPS: httpcore's
TunnelHTTPConnection.handle_request hard-codes the TLS server_hostname
to self._remote_origin.host and never reads the sni_hostname
extension the pinning transport sets, so the SNI sent to the
destination became the pinned IP literal instead of the original
hostname. Cert verification failed for any host whose cert did not
also list the IP in its SAN. Every HTTPS fetch through the proxy
silently broke.

The minimal fix without rewriting httpcore's CONNECT tunnel
(fragile across httpcore updates) is to revert to
httpx.HTTPTransport(proxy=...) directly. The remaining SSRF surface
is the proxy's own DNS resolution of the destination; _validate_target
still rejects loopback / private / metadata ranges against the
worker's DNS at fetch time, and the proxy URL is operator-controlled
via TILENS_FETCH_PROXY_URL so the trust boundary matches the
documented contract. The trade-off is documented in the
_build_transport docstring.

Also drop the now-unused _public_addresses alias from config.py
(it was a temporary back-compat stub during the network.py
extraction). Fix the _apply_success_transition docstring: SQLite
reads each SET expression against the pre-UPDATE column snapshot,
so the ordering rationale was inverted; rewrite to explain the
legibility-of-cleanup-branch reading instead of the false SET-order
claim.

test: tighten proxy, WAF, and threshold coverage from review findings ... d1e9a55279

Findings from the post-fix code review:

- test_default_threshold_blocks_after_default replaces the
  misleadingly-named test_invalid_threshold_falls_back_to_default
  which never exercised _coerce_positive_int. The new
  test_auto_block_after_failures_clamps_invalid in test_config.py
  covers the actual env-var clamp via monkeypatched env reload.

- test_blocked_probe_interval_clamps_invalid covers the same clamp
  for TILENS_BLOCKED_PROBE_INTERVAL_SEC; both threshold and probe
  interval defaults are now locked against typos that would silently
  disable auto-blocking.

- _assert_modern_waf_signature now also asserts Sec-Fetch-Site and
  Sec-Fetch-User so every WAF-allowlist end-to-end test catches a
  regression on those signals.

- Three new positive proxy validation tests: hostname-only URL,
  public IPv6 host (Cloudflare DNS), and mixed-case scheme. All
  three are valid inputs that the previous suite didn't cover.

- test_run_once_failed_probe_refreshes_block_cooldown now also
  patches app.backend.storage.entries.settings. Without this the
  test happened to pass because the worker-layer threshold (25)
  coincided with the real default; the self-describing patch makes
  the threshold pin explicit and removes the coincidence.

- test_threshold_min_boundary_blocks_on_first_failure pins
  threshold=1 behavior so a future refactor that flips >= to > is
  caught at the boundary.

- test_build_transport_uses_pinning_when_no_proxy now also asserts
  the inner pool has no proxy URL, so a future regression that
  accidentally passes proxy= into the no-proxy branch fails fast.

fix(worker): preserve auto-blocked status across sync_catalog ... 777ca92a95

Release blocker found in code review: sync_catalog runs every
cycle and called upsert_source, which unconditionally writes
status and reason from the catalog value. After the worker
auto-blocked a source (status='blocked', reason='auto-blocked
after 25 consecutive failures', next_due_at=NULL), the very next
cycle's sync_catalog overwrote status back to 'active' and
cleared reason. list_due_sources then matched the row
immediately (next_due_at is NULL), the worker fetched it again,
got the same failure, and re-blocked it. The 24h probe cadence
documented in ops.md and pinned by the auto-block migration
silently degraded to the worker cycle interval (~6h) because the
row never stayed blocked long enough for the probe selector's
cooldown check to take effect.

Fix: upsert_source now reads last_blocked_at alongside the
existing etag lookup. When the row is currently auto-blocked
(last_blocked_at IS NOT NULL), the UPDATE writes editorial
metadata (publisher, name, homepage, endpoint, adapter,
language, trust, stream, interval_min, is_github_source,
primary_topic_slug) but preserves the worker-owned status,
reason, and last_blocked_at. Catalog-side status changes
(active, blocked, retired, etc.) only apply to rows that are not
in the auto-blocked state.

Migration 011 also grows a backfill UPDATE: pre-existing
editorial-blocked rows from before the migration ran get their
last_blocked_at stamped to NOW so the probe selector picks them
up after the configured cooldown elapses. Without this, those
rows were forever invisible to the recovery probe because the
partial index only contains rows where last_blocked_at is set.

Tests:
- test_sync_catalog_preserves_auto_blocked_status: the catalog
  says active, the row says blocked with last_blocked_at set,
  sync_catalog runs and the row stays blocked. This is the
  regression that locks the fix.
- test_migration_011_backfills_existing_blocked_rows: apply
  migrations 0..10, insert a blocked row, then apply just the
  migration 011 body — confirms the UPDATE sets last_blocked_at
  on the blocked row but leaves active rows untouched.
- test_upsert_source_preserves_auto_blocked_status: pure storage
  test that round-trips through upsert_source with the
  catalog-active / row-blocked state.

test: tighten WAF coverage and proxy URL validation edge cases ... 323717982b

Review-driven additions:

- _assert_modern_waf_signature now asserts Sec-CH-UA-Platform
  alongside the other Chrome 142 / Sec-CH-UA / Sec-Fetch-* / br+zstd
  signals. Every WAF-allowlist E2E test catches a regression on
  any single signal.

- test_fetch_source_injects_waf_headers_for_last_week_in_aws:
  the last allowlist entry without an E2E test. Now all six
  (docker-blog, unsloth-blog, last-week-in-aws, osi-blog,
  mit-news-ai, mit-news-eecs) exercise the full WAF signature
  end-to-end through the wrapped fetcher.

- test_coerce_proxy_url_rejects_double_slash_path: pins that
  urlparse('http://host:8080//') parses path='//' (not in
  {'', '/'}) and the validator rejects it. Locks the path
  regex against a future 'use parsed.path != ""' regression.

rdenadai added 1 commit 2026-09-18 21:51:43 +00:00

test: use file-relative migrations dir in migration 011 test ... 68f90b201e

CI failed with 'no such table: sources' because the test used
a hardcoded /ssd/projetos/tilens/... path that doesn't exist in
the CI workspace (which lives at /workspace/rdenadai/tilens/...).
The MIGRATIONS_DIR module-level constant was already file-relative
via PROJECT_ROOT; just reference it inside the test instead of
hardcoding the dev machine path.

rdenadai added 1 commit 2026-09-18 22:02:17 +00:00

test: fix stream calendar fallback test to match actual code behavior ... 7265b7a139

CI was failing with 'expected /streams/research?topic=python&date=2026-09-16
got /streams/research?topic=python&date=2026-08-01' because the test
assumed the redirect target is (today - 2 days). That assumption only
holds when today is in August 2026; with the env clock now in
September, the calendar view for 2026-08 has no matching entries
in the fixture, so canonical_filter_url falls back to the first
day of the calendar month via its start anchor.

The original test name 'fallback_uses_stream_scope' implied a
stream-scoped fallback that doesn't exist in the code (see
app/backend/presentation/url_canonical.py:169-180 where the
fallback is simply the calendar month's first day). Same author
fixed two sibling tests in 5723b01 to use datetime.now(UTC).date()
but missed this one.

Updated the assertion to pin the actual behavior: calendar=2026-08
with no entries -> date=2026-08-01. Renamed to drop the misleading
'stream_scope' wording. A future stream-scoped fallback feature
will trip this test together with the URL builder.

rdenadai merged commit 84c38a8883 into main 2026-09-18 22:06:38 +00:00

rdenadai referenced this pull request from a commit 2026-09-18 22:06:40 +00:00

Merge pull request 'Improvements v0.3.0' (#15) from improvements-v0.3.0 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rdenadai/tilens!15

No description provided.