---
type: reference
area: knowledge
status: reference
date: 2026-06-02
created: 2026-06-02
updated: 2026-06-02
tags:
  - knowledge
---
# STALGEN-AI FastAPI + Outlines Migration Plan

## Summary

- Goal:
  - Migrate the current Streamlit-first STALGEN app into a decoupled API-driven system.
  - Keep local Ollama as the LLM runtime.
  - Use Outlines to enforce structured JSON generation.
  - Use React as the frontend.
  - Deploy backend, frontend, and supporting services with Docker Compose on Ubuntu.

- Current workflow:
  - `main.py`
    - Starts the Streamlit UI.
  - `stalgen/ui/ui.py`
    - Handles upload, model selection, progress display, and download.
  - `stalgen/utils/processing.py`
    - Orchestrates the full document pipeline.
  - `stalgen/parsers/word_parser_pandoc.py`
    - Converts DOCX to Markdown with Pandoc.
  - `stalgen/extractors/data_extractor.py`
    - Extracts business need, product specs, and user stories.
  - `stalgen/llm/llm_system_refactored.py`
    - Builds prompts, calls Ollama, retries, caches, parses, and saves JSON.
  - `stalgen/llm/ollama/ollama_client.py`
    - Calls Ollama directly.
  - `stalgen/llm/shared/response_processor.py`
    - Cleans, parses, validates, repairs, and falls back for LLM responses.
  - `stalgen/converters/json_to_excel.py`
    - Writes generated scenarios into the `.xlsm` template.

- Target workflow:
  - React uploads `.docx`.
  - FastAPI accepts the file and starts processing.
  - Backend parses DOCX, extracts user stories, calls Outlines/Ollama, validates schemas, generates Excel.
  - React polls job status and downloads the final `.xlsm`.
  - Docker Compose runs backend, frontend, Redis, and optional local Ollama integration.

## Phase 1: Remediation Before Migration

- Purpose:
  - Fix correctness, structure, and performance issues before introducing FastAPI.
  - Keep behavior stable while isolating core logic from Streamlit.

- Existing files to refactor:
  - `stalgen/utils/processing.py`
    - Role now:
      - Full workflow orchestration.
      - Streamlit-style uploaded file handling.
      - Temp directory lifecycle.
      - LLM loop.
      - Excel generation.
    - Change:
      - Split into service classes.
      - Remove direct dependency on Streamlit upload objects.
      - Accept normal file paths or byte streams.
      - Return typed result objects instead of loose dictionaries.
    - Bugs and risks to fix:
      - Temp cleanup is global and may delete active work older than 1 hour.
      - Full request is synchronous and blocks the UI/API worker.
      - File names are used directly and need sanitization.
      - Partial LLM failures are mixed with successful workflow state.
      - Per-story JSON files are saved only as intermediate files.

  - `stalgen/extractors/data_extractor.py`
    - Role now:
      - Regex-based SFD extraction.
      - Stores mutable extraction state in `self.extracted_data`.
      - Writes structured Markdown files.
    - Change:
      - Make extraction stateless.
      - Move file writing out of the extractor.
      - Return typed `SFDStructuredData`.
    - Bugs and risks to fix:
      - `Path(markdown_content).exists()` can misinterpret long raw Markdown as a path.
      - Regex patterns are brittle against heading variation.
      - Empty user story list should be a typed validation error.

  - `stalgen/parsers/word_parser_pandoc.py`
    - Role now:
      - Converts DOCX to Markdown.
      - Checks Pandoc installation during class initialization.
      - Creates temporary filtered DOCX files.
    - Change:
      - Keep as an infrastructure adapter.
      - Add explicit timeout for Pandoc subprocess calls.
      - Return conversion metrics.
    - Bugs and risks to fix:
      - `subprocess.run` fallback lacks timeout in one path.
      - Page count is an estimate and should be named as such in API responses.
      - `.doc` should be rejected unless conversion support is confirmed.

  - `stalgen/llm/llm_system_refactored.py`
    - Role now:
      - Prompt loading.
      - Model availability check.
      - Message building.
      - Ollama call.
      - Retry logic.
      - Cache.
      - JSON saving.
    - Change:
      - Replace with smaller services.
      - Keep prompt assembly separate from model calls.
      - Remove MD5 cache or replace with bounded TTL cache.
    - Bugs and risks to fix:
      - Uses `hashlib.md5`; use SHA-256 for cache keys.
      - In-memory cache is unbounded.
      - Fallback response can hide real failures.
      - File saving does not belong in the LLM system.

  - `stalgen/llm/shared/response_processor.py`
    - Role now:
      - Cleans malformed LLM text.
      - Parses JSON.
      - Repairs missing fields.
      - Checks quality.
    - Change:
      - Keep only quality checks and legacy migration helpers.
      - Move strict schema validation to Pydantic models used by Outlines.
    - Bugs and risks to fix:
      - Auto-repair can convert invalid LLM output into false success.
      - Regex JSON extraction can pick the wrong JSON block.
      - Fallback data should be marked as degraded output.

  - `stalgen/converters/json_to_excel.py`
    - Role now:
      - Reads JSON files from a folder and fills the Excel template.
    - Change:
      - Accept typed scenario objects in memory.
      - Keep a file-based compatibility method only for tests or legacy mode.
    - Bugs and risks to fix:
      - Failed JSON files are logged and skipped, which can create incomplete Excel output.
      - Missing scenarios should produce a typed workflow error.
      - Output path should be controlled by the job workspace.

- New backend core files:
  - `stalgen/core/config.py`
    - Role:
      - Central app settings.
      - Reads environment variables.
      - Owns paths, limits, timeouts, model defaults, CORS origins.
    - Key settings:
      - `APP_ENV`
      - `MAX_UPLOAD_MB=10`
      - `JOB_TTL_SECONDS=3600`
      - `LLM_PROVIDER=ollama`
      - `OLLAMA_BASE_URL=http://ollama:11434`
      - `OLLAMA_MODEL=deepseek-r1:14b`
      - `LLM_TIMEOUT_SECONDS=180`
      - `MAX_CONCURRENT_LLM_REQUESTS=2`
      - `EXCEL_TEMPLATE_PATH=data/templates/UAT-NRM-CRONOS-OMS-XF_V2.2 - Template.xlsm`

  - `stalgen/core/errors.py`
    - Role:
      - Defines typed application exceptions.
    - Exception types:
      - `UploadValidationError`
      - `DocumentConversionError`
      - `ExtractionError`
      - `LLMGenerationError`
      - `LLMSchemaValidationError`
      - `ExcelGenerationError`
      - `JobNotFoundError`

  - `stalgen/core/logging.py`
    - Role:
      - JSON logging setup.
      - Adds request ID and job ID to logs.

  - `stalgen/domain/models.py`
    - Role:
      - Internal domain dataclasses or Pydantic models.
    - Models:
      - `UploadedDocument`
      - `ParsedDocument`
      - `UserStory`
      - `SFDDocument`
      - `MacroScenario`
      - `ScenarioBundle`
      - `ProcessingSummary`
      - `ProcessingResult`

  - `stalgen/domain/schemas.py`
    - Role:
      - Strict LLM output schemas for Outlines.
    - Models:
      - `GeneratedScenario`
      - `GeneratedScenarioBundle`
      - `GeneratedScenarioResponse`
    - Validation rules:
      - `macro_scenario`: 5 to 200 chars.
      - `test_cases`: 2 to 15 items.
      - Each test case: 20 to 500 chars.
      - Each test case starts with `Verify`, `Check`, `Validate`, `Ensure`, or `Confirm`.

  - `stalgen/services/document_service.py`
    - Role:
      - Validates uploaded file.
      - Saves file to a job workspace.
      - Calls DOCX parser.
      - Returns Markdown path and conversion metadata.

  - `stalgen/services/extraction_service.py`
    - Role:
      - Calls `SFDDataExtractor`.
      - Converts extracted data into domain models.
      - Fails clearly when no user stories are found.

  - `stalgen/services/scenario_service.py`
    - Role:
      - Generates scenarios per user story.
      - Coordinates prompt building and LLM provider.
      - Applies quality checks.
      - Returns typed scenario bundles.

  - `stalgen/services/excel_service.py`
    - Role:
      - Converts scenario bundles into final `.xlsm`.
      - Owns output workbook path.
      - Does not read intermediate JSON unless legacy mode is used.

  - `stalgen/services/pipeline_service.py`
    - Role:
      - Orchestrates the full backend workflow.
      - Creates job workspace.
      - Updates job status.
      - Calls document, extraction, scenario, and Excel services.
      - Produces final `ProcessingResult`.

- Tests to add in Phase 1:
  - `tests/unit/test_data_extractor.py`
    - Covers valid headings, missing business need, missing product specs, no user stories, heading variants.
  - `tests/unit/test_word_parser.py`
    - Covers invalid extension, missing file, skip pages, Pandoc failure.
  - `tests/unit/test_response_quality.py`
    - Covers too-short test cases, missing action verbs, repetitive output.
  - `tests/unit/test_excel_converter.py`
    - Covers missing template, missing worksheet, missing required columns, valid workbook output.
  - `tests/unit/test_pipeline_service.py`
    - Covers full service orchestration with mocked parser, LLM, and Excel converter.

- Phase 1 success criteria:
  - Unit tests pass.
  - Existing Streamlit path still works.
  - Core workflow can run without importing Streamlit.
  - LLM output has typed validation.
  - P95 local non-LLM processing time is measured.
  - No unbounded in-memory cache remains.

## Phase 2: FastAPI API Design

- Purpose:
  - Add an API layer without breaking the existing processing pipeline.
  - Keep API routes thin and move workflow logic into services.

- New API files:
  - `stalgen/api/main.py`
    - Role:
      - Creates FastAPI app.
      - Registers routers.
      - Configures middleware.
      - Configures exception handlers.
      - Exposes OpenAPI docs.

  - `stalgen/api/routes/health.py`
    - Role:
      - `GET /health`
      - Returns process health.
      - No dependency checks.
    - Response:
      - `{ "status": "ok" }`

  - `stalgen/api/routes/ready.py`
    - Role:
      - `GET /ready`
      - Checks required runtime dependencies.
      - Checks Excel template exists.
      - Checks Pandoc availability.
      - Checks Ollama availability.
    - Response:
      - `{ "status": "ready", "checks": {...} }`

  - `stalgen/api/routes/models.py`
    - Role:
      - `GET /v1/models`
      - Lists available Ollama models.
    - Replaces:
      - `get_available_models()` from `stalgen/ui/ui.py`.

  - `stalgen/api/routes/jobs.py`
    - Role:
      - `POST /v1/jobs`
      - Accepts `.docx` upload and options.
      - Creates background processing job.
      - Returns `job_id`.
      - `GET /v1/jobs/{job_id}`
      - Returns job status, progress, metrics, errors.
      - `GET /v1/jobs/{job_id}/download`
      - Downloads generated `.xlsm`.

  - `stalgen/api/schemas/requests.py`
    - Role:
      - Public request models.
    - Models:
      - `CreateJobOptions`
    - Fields:
      - `model_name: str | None`
      - `skip_pages: int = 0`
      - `use_cache: bool = true`

  - `stalgen/api/schemas/responses.py`
    - Role:
      - Public API response models.
    - Models:
      - `CreateJobResponse`
      - `JobStatusResponse`
      - `JobProgress`
      - `JobError`
      - `ModelListResponse`
      - `ReadinessResponse`

  - `stalgen/api/dependencies.py`
    - Role:
      - Provides settings, services, job store, and LLM provider.
      - Keeps route constructors clean.

  - `stalgen/api/exception_handlers.py`
    - Role:
      - Converts typed backend exceptions to stable JSON errors.
    - Error shape:
      - `{ "error": { "code": "...", "message": "...", "request_id": "...", "details": {...} } }`

  - `stalgen/api/middleware.py`
    - Role:
      - Adds request IDs.
      - Adds request timing.
      - Adds CORS.
      - Adds max body size protection if not handled by proxy.

- Job management files:
  - `stalgen/jobs/models.py`
    - Role:
      - Job status models.
    - Status values:
      - `queued`
      - `parsing`
      - `extracting`
      - `generating`
      - `exporting`
      - `completed`
      - `failed`
      - `expired`

  - `stalgen/jobs/store.py`
    - Role:
      - Job metadata storage interface.
      - Implementations can be in-memory first, Redis later.

  - `stalgen/jobs/in_memory_store.py`
    - Role:
      - Development job store.
      - Suitable for local tests only.

  - `stalgen/jobs/redis_store.py`
    - Role:
      - Production job store.
      - Stores job status, progress, error, output path, timestamps.

  - `stalgen/jobs/runner.py`
    - Role:
      - Runs pipeline jobs in background.
      - Updates status and progress.
      - Enforces concurrency limits.

- API behavior:
  - `POST /v1/jobs`
    - Accepts multipart form:
      - `file`: `.docx`
      - `model_name`: optional string
      - `skip_pages`: integer 0 to 100
      - `use_cache`: boolean
    - Returns:
      - `202 Accepted`
      - `{ "job_id": "...", "status": "queued" }`

  - `GET /v1/jobs/{job_id}`
    - Returns:
      - `200 OK`
      - Current job state.
      - Progress percentage.
      - Current step.
      - Counts for user stories, scenarios, test cases.
      - Error code if failed.

  - `GET /v1/jobs/{job_id}/download`
    - Returns:
      - `200 OK` with Excel file when completed.
      - `409 Conflict` when job is not completed.
      - `404 Not Found` when job does not exist.

- Phase 2 success criteria:
  - API can process one uploaded document end to end.
  - Streamlit no longer owns workflow logic.
  - All API errors use stable JSON shape.
  - Job polling works.
  - Download endpoint returns the generated `.xlsm`.
  - FastAPI routes have tests with `TestClient`.

## Phase 3: Outlines + Local Ollama LLM Integration

- Purpose:
  - Replace free-form JSON cleanup with schema-constrained generation.
  - Keep local Ollama as the model runtime.
  - Make LLM failures measurable and explicit.

- New LLM files:
  - `stalgen/llm/providers/base.py`
    - Role:
      - Defines LLM provider interface.
    - Interface:
      - `generate_scenarios(prompt: str, schema: type[BaseModel]) -> BaseModel`

  - `stalgen/llm/providers/outlines_ollama.py`
    - Role:
      - Uses Outlines with local Ollama.
      - Enforces Pydantic schema output.
      - Applies timeout and concurrency control.

  - `stalgen/llm/prompts/scenario_prompt.py`
    - Role:
      - Builds final scenario-generation prompt from one user story.
      - Uses existing prompt files from `docs/prompt`.
      - Removes JSON-format examples that conflict with Outlines schemas.

  - `stalgen/llm/quality.py`
    - Role:
      - Keeps quality checks from `ResponseProcessor`.
      - Does not repair invalid structure.
      - Produces warnings and metrics.

  - `stalgen/llm/cache.py`
    - Role:
      - Optional bounded cache.
      - Uses SHA-256 key.
      - Includes prompt, schema version, model name, and temperature in key.

- Existing files to keep temporarily:
  - `stalgen/llm/ollama/ollama_client.py`
    - Role:
      - Legacy adapter during transition.
      - Removed after Outlines path is stable.
  - `stalgen/llm/llm_system_refactored.py`
    - Role:
      - Legacy facade for Streamlit compatibility.
      - Deprecated after FastAPI path is complete.
  - `stalgen/llm/shared/response_processor.py`
    - Role:
      - Keep only quality checks and fallback helpers.
      - Stop using it for primary schema validation.

- Required schema:
  - `stalgen/domain/schemas.py`
    - `GeneratedScenario`
      - `macro_scenario: str`
      - `test_cases: list[str]`
    - `GeneratedScenarioResponse`
      - `output_json: list[GeneratedScenario]`

- Outlines integration snippet:
  ```python
  import asyncio
  from pydantic import BaseModel
  import outlines

  class GeneratedScenario(BaseModel):
      macro_scenario: str
      test_cases: list[str]

  class GeneratedScenarioResponse(BaseModel):
      output_json: list[GeneratedScenario]

  class OutlinesOllamaProvider:
      def __init__(self, model_name: str, timeout_seconds: int, semaphore: asyncio.Semaphore):
          self.model_name = model_name
          self.timeout_seconds = timeout_seconds
          self.semaphore = semaphore
          self.model = outlines.from_ollama(model_name)

      async def generate_scenarios(self, prompt: str) -> GeneratedScenarioResponse:
          async with self.semaphore:
              raw = await asyncio.wait_for(
                  asyncio.to_thread(
                      self.model,
                      prompt,
                      GeneratedScenarioResponse,
                      max_tokens=1200,
                  ),
                  timeout=self.timeout_seconds,
              )
              return GeneratedScenarioResponse.model_validate_json(raw)
  ```

- LLM failure handling:
  - Timeout:
    - Return API error code `LLM_TIMEOUT`.
    - Mark job as failed unless fallback mode is explicitly enabled.
  - Schema validation failure:
    - Retry once with a shorter repair prompt.
    - If still invalid, return `LLM_SCHEMA_VALIDATION_FAILED`.
  - Empty output:
    - Return `LLM_EMPTY_RESPONSE`.
  - Low-quality output:
    - Complete the job but include `quality_warnings` in job status.
  - Ollama unavailable:
    - `/ready` reports not ready.
    - Job creation fails with `LLM_PROVIDER_UNAVAILABLE`.

- Phase 3 success criteria:
  - JSON cleanup is no longer required for normal generation.
  - Schema validation success rate is measured.
  - LLM timeout rate is measured.
  - Retry rate is measured.
  - No invalid JSON reaches Excel conversion.
  - Legacy LLM path can be disabled with config.

## Phase 4: React Frontend

- Purpose:
  - Replace Streamlit with a dedicated client.
  - Keep frontend free of LLM logic.
  - Use API polling for long-running jobs.

- New frontend structure:
  - `frontend/package.json`
    - Role:
      - Defines React dependencies and scripts.
      - Scripts:
        - `dev`
        - `build`
        - `preview`
        - `lint`

  - `frontend/vite.config.ts`
    - Role:
      - Vite config.
      - Dev proxy to FastAPI.

  - `frontend/src/main.tsx`
    - Role:
      - React entrypoint.

  - `frontend/src/App.tsx`
    - Role:
      - Main page layout.
      - Upload form.
      - Progress panel.
      - Results summary.
      - Download button.

  - `frontend/src/api/client.ts`
    - Role:
      - Fetch wrapper.
      - Adds base URL.
      - Parses API errors.

  - `frontend/src/api/jobs.ts`
    - Role:
      - `createJob`
      - `getJob`
      - `downloadJob`

  - `frontend/src/api/models.ts`
    - Role:
      - `listModels`

  - `frontend/src/components/UploadForm.tsx`
    - Role:
      - File picker.
      - Model selector.
      - Skip-pages input.
      - Submit button.
      - Client-side file validation.

  - `frontend/src/components/JobProgress.tsx`
    - Role:
      - Shows current step.
      - Shows progress percentage.
      - Shows user story, scenario, and test case counts.

  - `frontend/src/components/ResultSummary.tsx`
    - Role:
      - Shows final metrics.
      - Shows quality warnings.
      - Shows download action.

  - `frontend/src/components/ErrorMessage.tsx`
    - Role:
      - Shows stable API errors.
      - Includes request ID when present.

  - `frontend/src/styles.css`
    - Role:
      - Global styling.
      - Replaces Streamlit-specific CSS.

- Frontend behavior:
  - On page load:
    - Call `GET /ready`.
    - Call `GET /v1/models`.
  - On upload:
    - Validate `.docx`.
    - Validate file size <= backend limit.
    - Submit `POST /v1/jobs`.
  - During processing:
    - Poll `GET /v1/jobs/{job_id}` every 2 seconds.
    - Stop polling on `completed` or `failed`.
  - On completion:
    - Enable download link.
    - Show metrics.
  - On failure:
    - Show error code and message.
    - Preserve selected file and options.

- Phase 4 success criteria:
  - React app can process and download a report.
  - Duplicate submits are blocked while a job is active.
  - Upload validation matches backend rules.
  - User sees clear state for queued, running, completed, and failed jobs.

## Phase 5: Docker + Deployment

- Purpose:
  - Containerize the backend and frontend.
  - Run repeatable deployments on Ubuntu.
  - Keep secrets and persistent job data outside images.

- New deployment files:
  - `Dockerfile.backend`
    - Role:
      - Builds FastAPI backend image.
      - Installs Python dependencies.
      - Installs Pandoc system package.
      - Runs Uvicorn.
    - Command:
      - `uvicorn stalgen.api.main:app --host 0.0.0.0 --port 8000`

  - `Dockerfile.frontend`
    - Role:
      - Builds React app.
      - Serves static assets with Nginx.

  - `docker-compose.yml`
    - Role:
      - Defines local production-like stack.
    - Services:
      - `backend`
      - `frontend`
      - `redis`
      - `ollama`
    - Volumes:
      - `job_data:/app/var/jobs`
      - `ollama_data:/root/.ollama`

  - `.env.example`
    - Role:
      - Documents required environment variables.
    - Values:
      - `APP_ENV=production`
      - `BACKEND_PORT=8000`
      - `FRONTEND_PORT=3000`
      - `OLLAMA_BASE_URL=http://ollama:11434`
      - `OLLAMA_MODEL=deepseek-r1:14b`
      - `MAX_UPLOAD_MB=10`
      - `MAX_CONCURRENT_LLM_REQUESTS=2`
      - `LLM_TIMEOUT_SECONDS=180`
      - `CORS_ORIGINS=http://localhost:3000`

  - `deploy/nginx.conf`
    - Role:
      - Optional reverse proxy config.
      - Routes frontend and `/api` backend traffic.
      - Sets upload body limit.

  - `deploy/ubuntu-setup.md`
    - Role:
      - Server setup instructions.
      - Docker install.
      - Firewall.
      - Environment setup.
      - First deployment.
      - Rollback.

- Backend dependency updates:
  - `requirements.txt`
    - Add:
      - `fastapi`
      - `uvicorn[standard]`
      - `python-multipart`
      - `pydantic-settings`
      - `redis`
      - `outlines`
      - `httpx`
      - `prometheus-client`
      - `pytest-asyncio`
    - Keep:
      - `python-docx`
      - `pypandoc`
      - `openpyxl`
      - `ollama` if Outlines local integration requires it.

- Deployment behavior:
  - Backend stores job workspaces under `/app/var/jobs/{job_id}`.
  - Redis stores job state.
  - Generated files expire after `JOB_TTL_SECONDS`.
  - Nginx or frontend container serves React.
  - `/ready` blocks rollout if Pandoc, template, Redis, or Ollama are unavailable.

- Phase 5 success criteria:
  - `docker compose up --build` starts the full stack.
  - `GET /health` returns `200`.
  - `GET /ready` returns `ready`.
  - React can upload a document and download Excel.
  - Container restart does not corrupt active job metadata.
  - Secrets are not committed.

## Phase 6: Observability, Performance, And Rollout

- Metrics to add:
  - `http_request_duration_seconds`
  - `job_total`
  - `job_failed_total`
  - `job_duration_seconds`
  - `llm_duration_seconds`
  - `llm_timeout_total`
  - `llm_schema_validation_failed_total`
  - `document_conversion_failed_total`
  - `excel_generation_failed_total`

- Logs to add:
  - Request ID.
  - Job ID.
  - Source file name.
  - File size.
  - Model name.
  - User story count.
  - Scenario count.
  - Test case count.
  - LLM latency.
  - Total job duration.
  - Error code.

- Performance targets:
  - `GET /health`: under 50 ms.
  - `GET /ready`: under 500 ms.
  - Upload validation failure: under 100 ms.
  - Non-LLM parsing for 10 MB file: measured and tracked.
  - LLM schema success rate: at least 99%.
  - LLM timeout rate: below 1%.
  - Job failure rate: below 3% after stabilization.

- Rollout steps:
  - Keep Streamlit path available during API migration.
  - Add FastAPI path behind separate command.
  - Validate API output against current Streamlit output on the same sample files.
  - Switch frontend users to React after parity checks pass.
  - Remove Streamlit only after one stable release cycle.

## Edge Cases And Required Mitigations

- LLM timeout:
  - Enforce `LLM_TIMEOUT_SECONDS`.
  - Limit concurrent LLM calls with `MAX_CONCURRENT_LLM_REQUESTS`.
  - Return `LLM_TIMEOUT`.
  - Mark job failed with clear retry guidance.

- Outlines schema validation failure:
  - Retry once with a compact repair prompt.
  - Return `LLM_SCHEMA_VALIDATION_FAILED` after retry.
  - Store failed raw output only in redacted debug logs.
  - Do not generate Excel from invalid scenarios.

- Concurrent API requests:
  - Use Redis job store.
  - Use LLM semaphore.
  - Return `429 TOO_MANY_REQUESTS` when queue limit is reached.
  - Track active jobs and queue depth.

- Large or invalid files:
  - Reject files over `MAX_UPLOAD_MB`.
  - Reject non-`.docx`.
  - Sanitize file names.
  - Return `UPLOAD_INVALID_FILE_TYPE` or `UPLOAD_TOO_LARGE`.

- No user stories found:
  - Fail extraction with `NO_USER_STORIES_FOUND`.
  - Return extracted section metrics for debugging.
  - Do not call the LLM.

- Pandoc unavailable:
  - `/ready` reports `pandoc=false`.
  - Jobs fail fast with `PANDOC_UNAVAILABLE`.
  - Docker backend image installs Pandoc.

- Excel template mismatch:
  - `/ready` verifies template exists.
  - `ExcelService` verifies worksheet and required columns.
  - Return `EXCEL_TEMPLATE_INVALID`.

- Job cleanup:
  - Cleanup only job directories owned by completed or failed jobs.
  - Never delete active jobs based only on directory age.
  - Mark expired jobs before deleting files.

## Test Plan

- Unit tests:
  - Test upload validation.
  - Test DOCX parser errors.
  - Test extraction with representative SFD Markdown.
  - Test no-user-story failure.
  - Test Outlines provider with mocked model.
  - Test schema validation failures.
  - Test quality warnings.
  - Test Excel output generation.
  - Test job store status transitions.

- API tests:
  - `GET /health` returns `200`.
  - `GET /ready` reports dependency status.
  - `GET /v1/models` handles Ollama available and unavailable.
  - `POST /v1/jobs` rejects invalid file type.
  - `POST /v1/jobs` rejects oversized file.
  - `POST /v1/jobs` creates queued job.
  - `GET /v1/jobs/{job_id}` returns progress.
  - `GET /v1/jobs/{job_id}/download` returns `409` before completion.
  - `GET /v1/jobs/{job_id}/download` returns file after completion.

- Integration tests:
  - Process one small sample `.docx` end to end.
  - Process document with skipped first page.
  - Simulate LLM timeout.
  - Simulate schema validation failure.
  - Simulate missing Excel template.
  - Simulate Redis unavailable.

- Frontend tests:
  - Upload form blocks invalid file.
  - Upload form blocks duplicate submit.
  - Job polling stops on completion.
  - Error message renders API error code.
  - Download button appears only after completion.

- Deployment tests:
  - `docker compose up --build` succeeds.
  - Backend health check passes.
  - Backend readiness check passes.
  - Frontend reaches backend through configured URL.
  - Generated Excel download works from containerized stack.

## Highest-Risk Step And Backup

- Highest-risk step:
  - Replacing current free-form Ollama JSON generation with Outlines schema-constrained generation.

- Why it is risky:
  - Current prompts may rely on JSON cleanup and repair.
  - Existing models may not follow the stricter schema consistently.
  - Strict validation may expose hidden quality issues that were previously repaired silently.

- Backup solution:
  - Keep `stalgen/llm/llm_system_refactored.py` as a legacy provider during rollout.
  - Add config:
    - `LLM_MODE=outlines`
    - `LLM_MODE=legacy`
  - Run both modes against the same sample documents.
  - Compare:
    - Schema success rate.
    - Scenario count.
    - Test case count.
    - Quality warnings.
    - Latency.
  - Use legacy mode only as a temporary fallback.
  - Remove legacy mode after Outlines reaches the target validation success rate.

## Assumptions And Defaults

- Frontend:
  - React with Vite.

- LLM provider:
  - Local Ollama through Outlines.

- API style:
  - Async job API with polling.
  - No synchronous long-running upload endpoint.

- Job state:
  - Redis in Docker and production.
  - In-memory store only for local tests.

- Upload limit:
  - 10 MB.

- LLM concurrency:
  - 2 concurrent LLM calls by default.

- Deployment:
  - Docker Compose on Ubuntu.
  - Optional Nginx reverse proxy.
  - Ollama runs as a Compose service unless the server already provides Ollama externally.

- Compatibility:
  - Keep Streamlit during migration.
  - Remove Streamlit only after React and FastAPI match current behavior.