rdenadai/hagalaz
rdenadai/hagalaz
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

feat: Add vLLM backend, pluggable architecture, and Docker build improvements #1

Open
rdenadai wants to merge 59 commits from rdenadai/improvements-v0.2.0 into main
pull from: rdenadai/improvements-v0.2.0
merge into: rdenadai:main
Conversation 0 Commits 59 Files changed 105 +25213 -3294
rdenadai commented 2026-05-28 11:24:08 +00:00
Owner
Copy link

Hagalaz v0.2.0: vLLM backend, chat UI, streaming audio, and service-layer refactor

TL;DR

This release turns Hagalaz from a single-backend transformers API into a multi-backend, production-oriented server. It adds a vLLM backend with continuous batching, a full Alpine.js chat UI with audio playback/recording, streaming TTS in multiple formats, a service-layer API refactor with dependency injection, multimodal model support for Gemma-4 / Qwen3.5-VL / Phi-4-multimodal, and a Docker overhaul with separate CUDA and vLLM images built on pyenv Python 3.13.13. The branch also adds comprehensive unit tests and rewritten documentation.

Major changes

vLLM backend

  • New src/core/backends/vllm.py implementing the backend protocol via vLLM AsyncLLMEngine.
  • New src/streaming/vllm_streamer.py for delta-based streaming with shared reasoning detection.
  • vLLM 0.22.0 / CUDA 12.9.2 / PyTorch 2.11.0+cu129 in dedicated docker/Dockerfile.vllm.
  • Configurable tensor parallelism, GPU memory utilization, max model length, quantization (AWQ/GPTQ/FP8), and dtype.
  • Chat-template fallback via --chat-template-model-id / CHAT_TEMPLATE_MODEL_ID for tokenizers that ship without one.
  • Full-stack LOAD=all support in the vLLM image alongside image/audio/TTS models.
  • System-message normalization and context-length pre-flight checks.

Chat web UI

  • New /chat route and static/chat.html.
  • Rewritten in Alpine.js (static/js/chat-app.js, static/css/chat.css) with:
    • Streaming markdown and code highlighting
    • Collapsible reasoning/thinking blocks
    • Sidebar conversation list
    • Browser audio recording for speech-to-text
    • IndexedDB-backed media storage for generated audio replay
    • SVG icons, responsive layout, and auth key UI

Audio / TTS streaming

  • New /v1/audio/speech/stream endpoint returning SSE with base64 audio chunks.
  • MP3, OGG, FLAC, OPUS, and WAV output formats.
  • Voxtral TTS support alongside Bark (mistralai/Voxtral-4B-TTS-2603).
  • Text ingestion for TTS: URL fetch, PDF, and HTML via readability (src/api/text_ingestion.py).
  • /v1/audio/voices endpoint lists model-specific voice presets.

API refactor

  • Routes in src/api/routes.py are now thin handlers delegating to services.
  • New service layer: ChatService, ImageService, AudioService, TTSService, HealthService, ModelListingService.
  • Pydantic request/response schemas in src/api/schemas.py and src/api/requests.py.
  • Input validation for chat and image endpoints (src/api/validation.py).
  • Dependency injection via app.state with ModelRegistry, ResourceMonitor, and request deduplication.
  • main.py split into focused helper functions with explicit lifespan management.

Backend architecture

  • src/core/backends/ with a shared base.py protocol, transformers.py, vllm.py, reasoning_builder.py, and processor_utils.py.
  • Unified ReasoningDetector in src/streaming/reasoning_detector.py for <think> / </think> detection across both backends.
  • ReasoningStreamer in src/streaming/streamer.py handles real-time token streaming, marker stripping, and thread-safe queueing.
  • ModelLoader orchestrates startup model loading.
  • Repository pattern for model loading in src/core/repositories/.

Model support and compatibility

  • Bumped to transformers>=5.1.0 for Qwen3.5, Gemma-4, and Phi-4-multimodal.
  • Multimodal image+text inputs via prepare_multimodal_inputs (Gemma-4, Qwen3.5-VL, Phi-4-multimodal).
  • GGUF auto-detection and --gguf-file selection.
  • 4-bit / BNB quantization fixes and CUDA numerical stability fixes.
  • Marked DeepReinforce Ornith models as community/experimental with template-fallback instructions.

Docker and deployment

  • All images migrated to pyenv Python 3.13.13 compiled with PGO+LTO.
  • New docker/Dockerfile.vllm with separate pyproject.vllm.toml and uv.vllm.lock.
  • Updated docker/Dockerfile.cuda (CUDA 12.4) with separate pyproject.cuda.toml / uv.cuda.lock.
  • docker/build.sh multi-image build script.
  • Updated docker-compose.yml with hagalaz-vllm service and env-var interpolation.
  • New docs: docs/DEPLOY_DOCKER.md, docs/DOCKERHUB.md, docs/VLLM.md, docs/MODELS.md, docs/RULES.md.

Configuration

  • src/config.py rewritten with pydantic-settings and environment-specific settings classes (LLMSettings, ImageSettings, AudioSettings, TTSSettings, etc.).
  • --load now supports comma-separated values (llm,image,audio,tts) in addition to all and both aliases.

Tests

  • Large expansion of unit tests under tests/unit/:
    • API: test_chat_service.py, test_middleware.py, test_routes.py, test_tts_service.py, test_schemas.py
    • Core backends: test_transformers_vllm.py, test_reasoning_builder.py
    • Streaming: test_streamer.py, test_reasoning_detector.py, test_vllm_streamer.py, test_base_iterator.py
    • Multimodal, model loader, audio, images, main app factory, manage_keys.
  • Torch-dependent tests skip cleanly outside Docker.

Breaking changes

  • Python: requires >=3.13.11 (Docker images use 3.13.13).
  • transformers>=5.1.0 is now required; v4.x is no longer supported.
  • Default LOAD: changed from tts,image to llm,image.
  • TTS response format: /v1/audio/speech now returns raw audio bytes by default instead of base64 JSON.
  • Config system: moved from manual argparse/env parsing to pydantic-settings with prefixed env vars.
  • Docker base images: migrated to pyenv-built Python; added a vLLM-specific image.
  • Project structure: docs moved to docs/, new src/core/backends/ and src/core/repositories/ packages.

Verification

  • Focused backend/streaming test suite passes:
    • tests/unit/core/backends/test_transformers_vllm.py
    • tests/unit/streaming/test_vllm_streamer.py
    • tests/unit/core/backends/test_reasoning_builder.py
    • tests/unit/streaming/test_streamer.py
    • tests/unit/core/test_multimodal.py
    • tests/unit/streaming/test_reasoning_detector.py
    • tests/unit/streaming/test_reasoning_streamer_final.py
  • ruff passes on changed source/test files.
  • vLLM Docker image builds successfully and reports:
    • vllm 0.22.0
    • torch 2.11.0+cu129
    • transformers 5.12.1
  • GPU smoke test reached vLLM model loading. The local test GPU (GTX 1060, sm_61) is below PyTorch cu129's minimum compute capability of 7.5, so full inference could not complete on that hardware.

Known limitations

  • vLLM does not support multimodal models or GGUF — use the transformers backend for those.
  • vLLM defaults to conservative context windows — set LLM_VLLM_MAX_MODEL_LEN for long-context models.
  • vLLM requires compute capability >= 7.5 (sm_75); GTX 10 series / Pascal GPUs are not supported.
  • Transformers backend serializes GPU access via asyncio.Lock — no concurrent LLM requests.
  • Full-stack vLLM on 24GB GPUs requires careful VRAM management (VLLM_GPU_MEMORY_UTILIZATION=0.60, small image/audio models).
# Hagalaz v0.2.0: vLLM backend, chat UI, streaming audio, and service-layer refactor ## TL;DR This release turns Hagalaz from a single-backend transformers API into a multi-backend, production-oriented server. It adds a **vLLM backend** with continuous batching, a full **Alpine.js chat UI** with audio playback/recording, **streaming TTS** in multiple formats, a **service-layer API refactor** with dependency injection, **multimodal model support** for Gemma-4 / Qwen3.5-VL / Phi-4-multimodal, and a **Docker overhaul** with separate CUDA and vLLM images built on pyenv Python 3.13.13. The branch also adds comprehensive unit tests and rewritten documentation. --- ## Major changes ### vLLM backend - New `src/core/backends/vllm.py` implementing the backend protocol via vLLM `AsyncLLMEngine`. - New `src/streaming/vllm_streamer.py` for delta-based streaming with shared reasoning detection. - vLLM 0.22.0 / CUDA 12.9.2 / PyTorch 2.11.0+cu129 in dedicated `docker/Dockerfile.vllm`. - Configurable tensor parallelism, GPU memory utilization, max model length, quantization (AWQ/GPTQ/FP8), and dtype. - Chat-template fallback via `--chat-template-model-id` / `CHAT_TEMPLATE_MODEL_ID` for tokenizers that ship without one. - Full-stack `LOAD=all` support in the vLLM image alongside image/audio/TTS models. - System-message normalization and context-length pre-flight checks. ### Chat web UI - New `/chat` route and `static/chat.html`. - Rewritten in Alpine.js (`static/js/chat-app.js`, `static/css/chat.css`) with: - Streaming markdown and code highlighting - Collapsible reasoning/thinking blocks - Sidebar conversation list - Browser audio recording for speech-to-text - IndexedDB-backed media storage for generated audio replay - SVG icons, responsive layout, and auth key UI ### Audio / TTS streaming - New `/v1/audio/speech/stream` endpoint returning SSE with base64 audio chunks. - MP3, OGG, FLAC, OPUS, and WAV output formats. - Voxtral TTS support alongside Bark (`mistralai/Voxtral-4B-TTS-2603`). - Text ingestion for TTS: URL fetch, PDF, and HTML via readability (`src/api/text_ingestion.py`). - `/v1/audio/voices` endpoint lists model-specific voice presets. ### API refactor - Routes in `src/api/routes.py` are now thin handlers delegating to services. - New service layer: `ChatService`, `ImageService`, `AudioService`, `TTSService`, `HealthService`, `ModelListingService`. - Pydantic request/response schemas in `src/api/schemas.py` and `src/api/requests.py`. - Input validation for chat and image endpoints (`src/api/validation.py`). - Dependency injection via `app.state` with `ModelRegistry`, `ResourceMonitor`, and request deduplication. - `main.py` split into focused helper functions with explicit lifespan management. ### Backend architecture - `src/core/backends/` with a shared `base.py` protocol, `transformers.py`, `vllm.py`, `reasoning_builder.py`, and `processor_utils.py`. - Unified `ReasoningDetector` in `src/streaming/reasoning_detector.py` for `<think>` / `</think>` detection across both backends. - `ReasoningStreamer` in `src/streaming/streamer.py` handles real-time token streaming, marker stripping, and thread-safe queueing. - `ModelLoader` orchestrates startup model loading. - Repository pattern for model loading in `src/core/repositories/`. ### Model support and compatibility - Bumped to `transformers>=5.1.0` for Qwen3.5, Gemma-4, and Phi-4-multimodal. - Multimodal image+text inputs via `prepare_multimodal_inputs` (Gemma-4, Qwen3.5-VL, Phi-4-multimodal). - GGUF auto-detection and `--gguf-file` selection. - 4-bit / BNB quantization fixes and CUDA numerical stability fixes. - Marked DeepReinforce Ornith models as community/experimental with template-fallback instructions. ### Docker and deployment - All images migrated to pyenv Python 3.13.13 compiled with PGO+LTO. - New `docker/Dockerfile.vllm` with separate `pyproject.vllm.toml` and `uv.vllm.lock`. - Updated `docker/Dockerfile.cuda` (CUDA 12.4) with separate `pyproject.cuda.toml` / `uv.cuda.lock`. - `docker/build.sh` multi-image build script. - Updated `docker-compose.yml` with `hagalaz-vllm` service and env-var interpolation. - New docs: `docs/DEPLOY_DOCKER.md`, `docs/DOCKERHUB.md`, `docs/VLLM.md`, `docs/MODELS.md`, `docs/RULES.md`. ### Configuration - `src/config.py` rewritten with `pydantic-settings` and environment-specific settings classes (`LLMSettings`, `ImageSettings`, `AudioSettings`, `TTSSettings`, etc.). - `--load` now supports comma-separated values (`llm,image,audio,tts`) in addition to `all` and `both` aliases. ### Tests - Large expansion of unit tests under `tests/unit/`: - API: `test_chat_service.py`, `test_middleware.py`, `test_routes.py`, `test_tts_service.py`, `test_schemas.py` - Core backends: `test_transformers_vllm.py`, `test_reasoning_builder.py` - Streaming: `test_streamer.py`, `test_reasoning_detector.py`, `test_vllm_streamer.py`, `test_base_iterator.py` - Multimodal, model loader, audio, images, main app factory, manage_keys. - Torch-dependent tests skip cleanly outside Docker. --- ## Breaking changes - **Python**: requires `>=3.13.11` (Docker images use 3.13.13). - **`transformers>=5.1.0`** is now required; v4.x is no longer supported. - **Default `LOAD`**: changed from `tts,image` to `llm,image`. - **TTS response format**: `/v1/audio/speech` now returns raw audio bytes by default instead of base64 JSON. - **Config system**: moved from manual argparse/env parsing to `pydantic-settings` with prefixed env vars. - **Docker base images**: migrated to pyenv-built Python; added a vLLM-specific image. - **Project structure**: docs moved to `docs/`, new `src/core/backends/` and `src/core/repositories/` packages. --- ## Verification - Focused backend/streaming test suite passes: - `tests/unit/core/backends/test_transformers_vllm.py` - `tests/unit/streaming/test_vllm_streamer.py` - `tests/unit/core/backends/test_reasoning_builder.py` - `tests/unit/streaming/test_streamer.py` - `tests/unit/core/test_multimodal.py` - `tests/unit/streaming/test_reasoning_detector.py` - `tests/unit/streaming/test_reasoning_streamer_final.py` - `ruff` passes on changed source/test files. - vLLM Docker image builds successfully and reports: - `vllm 0.22.0` - `torch 2.11.0+cu129` - `transformers 5.12.1` - GPU smoke test reached vLLM model loading. The local test GPU (GTX 1060, sm_61) is below PyTorch cu129's minimum compute capability of 7.5, so full inference could not complete on that hardware. --- ## Known limitations - **vLLM does not support multimodal models or GGUF** — use the transformers backend for those. - **vLLM defaults to conservative context windows** — set `LLM_VLLM_MAX_MODEL_LEN` for long-context models. - **vLLM requires compute capability >= 7.5** (sm_75); GTX 10 series / Pascal GPUs are not supported. - **Transformers backend serializes GPU access** via `asyncio.Lock` — no concurrent LLM requests. - Full-stack vLLM on 24GB GPUs requires careful VRAM management (`VLLM_GPU_MEMORY_UTILIZATION=0.60`, small image/audio models).
🚀 1
rdenadai changed title from rdenadai/improvements-v0.2.0 to feat: Add vLLM backend, pluggable architecture, and Docker build improvements 2026-05-28 11:25:43 +00:00
rdenadai force-pushed rdenadai/improvements-v0.2.0 from 33e3a813d0 to 3d0afc7708 2026-05-28 13:04:21 +00:00 Compare 
- Add vLLM 0.12.0 backend with AsyncLLMEngine for high-performance inference
- Fix vLLM tokenizer resolution (load HF tokenizer for chat template support)
- Add system role normalization for vLLM compatibility (Mistral/Phi models)
- Add context length pre-flight check with clear error messages
- Improve error logging across vLLM backend and chat service
- Optimize vLLM Dockerfile: CUDA 12.8 runtime, deadsnakes PPA Python 3.13
- Enable full stack support in vLLM image (LOAD=all for LLM+image+audio+tts)
- Update MODELS.md with context length configuration and vLLM examples
- Update DEPLOY_DOCKER.md with vLLM build instructions
- Add 38 new tests for vLLM backend, chat service, model loader
- Fix TRANSFORMERS_CACHE deprecation warning in Docker startup
- Update default model to Phi-4-mini-instruct (function calling, 128K context)
- Replace deadsnakes PPA with pyenv across all Dockerfiles
- Compile Python 3.13.13 with --enable-optimizations --with-lto
- Use -mtune=generic for CPU compatibility
- Create separate dependency files per image:
  * pyproject.local.toml + uv.local.lock (CUDA 11.8)
  * pyproject.cuda.toml + uv.cuda.lock (CUDA 12.4)
  * pyproject.vllm.toml + uv.vllm.lock (CUDA 12.8)
- Update documentation (DEPLOY_DOCKER.md, RULES.md, README.md, DOCKERHUB.md)
- All Docker images now use Ubuntu 22.04 base with pyenv
Add bounds checking for API parameters:
- max_tokens: 1-128000
- temperature: 0.0-2.0
- num_inference_steps: 1-50
- guidance_scale: 0.0-30.0
- width/height: 64-2048, multiple of 8
- n: 1-4 concurrent images

New src/api/validation.py provides reusable validators.
Add Unsloth Gemma 4 QAT (Quantization-Aware Training) variants:
- GGUF versions: E2B, E4B, 12B, 26B-A4B, 31B
- Unquantized BF16 versions for all sizes
- Updated VRAM estimates based on actual file sizes
- Added QAT models to Multimodal Models section
- Added QAT recommendations for RTX 4090 and L40S
- Added QAT-based model combinations

Verified all models exist on HuggingFace.
Add docker_only and requires_torch markers to pytest.ini.
Add module-level guards to 8 test files importing torch/CUDA.
Add Docker detection fixture to conftest.py for future use.
Remove redundant tests/unit/api/conftest.py.

Tests skip cleanly outside Docker with clear messaging about CUDA/cuDNN requirements.
- transformers: 4.52.4 -> >=5.1.0 in root/local/cuda pyproject.toml
- transformers: pinned <5 in vLLM image (vLLM 0.12.0 incompatibility)
- diffusers: >=0.32.0 -> >=0.37.0 (v5 support added in 0.37.0)
- huggingface-hub: >=0.27.0 -> >=1.0.0 (v5 requirement)
- Add protobuf>=3.20.0 to all images (fix SD 3.5 tokenizer load)
- Add orjson and xformers to docker local/cuda images
- processor_utils.py: normalize BatchEncoding return from apply_chat_template for v5 compat
- streamer.py: add fallback import for BaseStreamer (v5 path change)
- MODELS.md: mark Qwen3.5/Gemma4 as incompatible with vLLM backend
- README.md: clarify BaseStreamer import error for v5
- Regenerate all uv lockfiles

Fixes runtime dependency failures for newer models (Qwen3.5, Gemma4)
while maintaining vLLM 0.12.0 backward compatibility.
- transformers.py: filter out multimodal keys (mm_token_type_ids) that
  v5 processors include but generate() rejects. Prevents 500 errors on
  Qwen3.5 and other multimodal models.
- Replace deprecated torch_dtype= with dtype= in all from_pretrained()
  calls across models.py, audio.py, images.py. Both v4.57.6 and v5.x
  support dtype parameter.
- images.py: revert diffusers pipeline params from dtype back to torch_dtype
  (diffusers pipelines ignore dtype keyword, causing float32/float16 mismatch)
- audio.py: use device_map=device instead of low_cpu_mem_usage + .to(device)
  to avoid meta tensor copy errors in transformers v5 for Bark and Whisper
- models.py: restrict AutoProcessor detection to explicit multimodal
  architectures/model_type. Prevents text-only models like Qwen3.5 from
  using AutoProcessor and producing malformed inputs.
- chat_service.py: pass actual exception object to fail_dedup instead of
  string, fixing secondary TypeError in error handling.
Restores low_cpu_mem_usage=True for v5 compatibility while keeping
device_map=device to avoid meta tensor copy errors. Both flags are
supported together in transformers >=4.56 and v5.x.
Removes the defensive insertion of a dummy user message when the
conversation doesn't start with a user role. This was causing agent
tools (like opencode) to see artificial context and produce confused
reasoning. All models in MODELS.md are modern instruction-tuned models
that natively support system-first conversations via their chat templates.
Streamer:
- Remove incorrect next_tokens_are_prompt skip that truncated first token

Audio:
- Bark float16→float32 for soundfile compatibility
- Add max_length=None override to avoid config default conflicts
- Create attention_mask when missing

Transformers backend:
- Add attn_implementation=eager for BNB 4-bit numerical stability
- Disable use_cache for quantized models during generation
- Clamp temperature to min 0.01 to prevent softmax division-by-zero
- Create attention_mask when missing

Models:
- Replace SystemExit with RuntimeError for gated model access checks
- Add attn_implementation=eager to all model loading paths

API:
- Fix model ID extraction for audio/tts in /v1/models endpoint

Docker:
- Rewrite default Dockerfile to use nvidia/cuda:11.8 base image
- Tie default Dockerfile to root pyproject.toml and uv.lock
- Remove redundant docker/pyproject.local.toml and uv.local.lock
- Update docker-compose with env var interpolation for all 3 services
- Remove hagalaz-runpod service
- Add GPU reservations to default hagalaz service

Frontend:
- Add JavaScript to auto-replace YOUR_HOST placeholder with actual host

Docs:
- Update .env.example, README, DEPLOY_DOCKER.md for new Docker setup

Tests: 72 passed, 8 skipped (1 pre-existing failure unrelated to changes)
The extra }); after DOMContentLoaded listener caused:
- Uncaught SyntaxError: expected expression, got '}'
- YOUR_HOST:8000 placeholder never replaced with actual host

1 line removed. Fixes regression from previous commit.
Allow selective model loading via comma-separated lists:
  --load llm,image,audio
  LOAD=llm,image,audio

Also preserves legacy aliases:
  both -> llm+image
  all  -> llm+image+audio+tts

Changes:
- config.py: add _parse_load type with validation
- model_loader.py: add _normalize_load for backward compat
- validation updated to check set membership

Fixes Docker Compose default LOAD=llm,image,audio failing with
'invalid choice: llm,image,audio'.
A. Reorder quantization branch priority in _load_standard_model()
   - Native/pre-quantized models (Unsloth, GPTQ, AWQ) now load first
   - Fixes DeepSeek 1.5B being loaded in FP16 instead of native 4-bit
   - Prevents CUDA assert from numerical overflow in attention softmax

D. VRAM-based KV cache threshold (12GB)
   - >= 12GB: cache enabled (fast generation, RTX 3060+, L40S)
   - < 12GB: cache disabled (stable on RTX 1060 6GB)
   - Eliminates VRAM corruption on limited GPUs

E. Set max_length=None in generation kwargs
   - Resolves transformers warning about max_new_tokens vs max_length conflict
   - Prevents unnecessary memory allocation from model's default 131072 context

Impact:
- hagalaz (RTX 1060 6GB): Pre-quantized models load correctly, cache disabled
- hagalaz-cuda (L40S 48GB): Cache enabled, full speed
- hagalaz-vllm: No impact (uses vLLM's PagedAttention)
Fix 1: Native quantization loading
- Pass quantization_config explicitly in has_native_quant branch
- BNB models (e.g., unsloth/DeepSeek-R1-Distill-Qwen-1.5B-bnb-4bit) were
  loading in full FP16 instead of 4-bit despite log saying 'native quant'
- Caused CUDA assert from numerical overflow in attention softmax
- VRAM usage now ~1.5GB instead of ~4GB for 1.5B quantized models

Fix 2: Default LOAD for local Docker
- Changed from 'llm,image,audio' to 'llm,image'
- 6GB VRAM cannot fit all three models simultaneously
- Audio fails with meta tensor error when VRAM exhausted
- Users with more VRAM can override via LOAD env var

Tests: 72 passed, 8 skipped
The previous fix passed model_config.quantization_config (a plain dict)
which caused: 'model is quantized with BitsAndBytesConfig but you are
passing a dict config.'

Transformers auto-detects quantization_config from config.json and
instantiates the proper class internally. Explicit passing is redundant
and now causes type mismatch in v5.

Removed the parameter entirely from the native/pre-quantized branch.
Changed /v1/audio/speech to return a StreamingResponse with raw audio
bytes (Content-Type: audio/wav) matching OpenAI API specification.

Changes:
- Added audio_to_bytes() helper in src/core/audio.py
- Modified audio_to_base64() to reuse audio_to_bytes()
- Updated /v1/audio/speech endpoint to return StreamingResponse
- Added Content-Disposition header for file download

Previously returned JSON with base64-encoded audio string.
Now returns raw WAV file for direct playback/download.

Tests: 72 passed, 8 skipped
Eliminates transformers warnings:
- 'attention mask and the pad token id were not set'
- 'Setting pad_token_id to eos_token_id for open-end generation'

Changes:
- _generate_bark_speech(): add attention_mask + pad_token_id
- _generate_voxtral_speech(): add attention_mask + pad_token_id
- Both now create attention_mask when missing
- Both now set pad_token_id to eos_token_id when unset
- pad_token_id passed explicitly to model.generate()

Tests: syntax verified
- Extract non-chat business logic into services (image/audio/TTS/model-list/health)
- Add Pydantic request/response schemas for all endpoints
- Fix audio streaming yield indentation
- Fix vLLM fallback completion_tokens counting
- Add /chat.html to public endpoints
- Minimal TransformersBackend lock fix: hold lock until cleanup joins thread
- Move audio streaming sync work into executor
- Replace pending_futures with active task tracking for graceful shutdown
- Construct ChatService once at startup
- Offload image_to_base64 and audio_to_bytes from main thread
- Precompile streamer template-marker regex; use blake2b for cache/dedup keys
- Add include_reasoning to dedup key
- Add mypy strict config and type critical paths
- Fix system.py total_gb divisor
- Add mocked backend unit tests for TransformersBackend and VLLMBackend
- Remove Docker-only blanket skips from tests
- Delete empty tests/integration/
- Introduce Protocol-based DI for image/TTS/audio services.
- Move request parsing and SSE formatting out of routes.
- Remove ChatCacheService indirection; add ChatCache Protocol.
- Add core/repositories/ package for LLM/audio/image loading.
- Split VLLMBackend and TransformersBackend into focused helpers.
- Compose AsyncQueueIterator in ReasoningStreamer instead of MI.
- Fix chat UI TTS file/url handling and undefined Alpine state.
- Add dedicated tts_timeout config (default 300s) separate from audio_timeout.
- Reduce TTS chunk size via tts.max_chars_per_chunk config (default 150).
- Add /v1/audio/speech/stream endpoint returning chunked MP3.
- Move audio generator/encoder wrappers to src/api/audio_generators.py.
- Log actual device when loading Bark/Voxtral models.
- Clear generation_config.max_length in TransformersBackend to avoid warning.
- Update chat UI to stream TTS via MediaSource with non-streaming fallback.
- Add cache-busting query string to chat-app.js.
Add streaming TTS support, unified media generation bubbles, audio replay persistence, and local compose defaults for audio/TTS workflows.
- Buffer undecided text in reasoning-capable templates until think boundary
- Emit reasoning_content deltas only when include_reasoning is true
- Add typing queue for reasoning/answer with inline pending indicator
- Prevent duplicate reasoning text from appearing as answer
- Update reasoning block whitespace handling
Bump vLLM image from 0.12.0 to 0.22.0, torch 2.11.0+cu129,
transformers>=5.5.1, and CUDA 12.9.2 runtime base image.
Use venv python directly as entrypoint and add CUDA 13 libs
from the venv to LD_LIBRARY_PATH so vLLM extensions load.
Add --chat-template-model-id / CHAT_TEMPLATE_MODEL_ID to borrow a
chat_template from a base HF repo when the loaded tokenizer lacks one.
Prevents repetition loops on experimental models like Ornith GGUF repos.
Document Ornith models as community/experimental and note vLLM 0.22.0
GPU requirements (compute capability >= 7.5).
build_reasoning_response now calls detector.finalize so text buffered
without a think boundary is emitted as content instead of dropped.
Update streamer test to expect buffering behavior for reasoning-capable
templates until a think boundary or generation end.
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin rdenadai/improvements-v0.2.0:rdenadai/improvements-v0.2.0
git switch rdenadai/improvements-v0.2.0

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff rdenadai/improvements-v0.2.0
git switch rdenadai/improvements-v0.2.0
git rebase main
git switch main
git merge --ff-only rdenadai/improvements-v0.2.0
git switch rdenadai/improvements-v0.2.0
git rebase main
git switch main
git merge --no-ff rdenadai/improvements-v0.2.0
git switch main
git merge --squash rdenadai/improvements-v0.2.0
git switch main
git merge --ff-only rdenadai/improvements-v0.2.0
git switch main
git merge rdenadai/improvements-v0.2.0
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
rdenadai/hagalaz!1
No description provided.