Inside OpenRouter’s Tech Stack and Use of Effect | Louis Vichy | Cause & Effect 6

Overview

OpenRouter’s engineering stack discussion focuses on how they build and operate a high-scale “model switchboard” in TypeScript on Cloudflare Workers. It also covers why they adopt the Effect library—primarily for reliability, testing, and operational tooling—with an incremental plan to expand Effect usage over time.

---

Effect usage & testing infrastructure (key technical points)

• OpenRouter uses Effect heavily in internal tooling, not just “toy scripts.”
• Rather than relying on a generic “Python eval suite,” they wrote their own eval suite and a bulk test suite, and both are implemented with Effect, including Effect retry.
• They describe these tests as production-load operations run daily to validate endpoint behavior (testing across model/provider endpoints).

• They also use Effect in transformation pipelines, noting:

  “almost every respond transformation is using the pipe now.”

• Their adoption is incremental:

  • They introduce Effect patterns gradually (e.g., pipe + generators),
  • so they can transition more of the system to Effect later.
• Rationale: Effect’s retry and behavior composition improves reliability when interacting with external APIs.

---

Core product architecture (Cloudflare Workers + functional error handling)

The core OpenRouter API is hosted on Cloudflare Workers.

Request lifecycle model

• Authentication step: implemented as a function.
• Result/monad paradigm: many functions return a Result-like type; callers explicitly handle success/failure and return appropriate responses.
• Requests pass through multiple endpoints:
  • Their internal concept of an endpoint is a combination of model + context/weights/meta plus provider-specific serving configuration.
• Two layered stacks:
  1. Routing stack: orders/prioritizes endpoints using heuristics such as latency, throughput, or price.
  2. Plugin layers: transform requests and transform streamed responses in a FIFO-style layering.
• They iterate endpoints until one produces a usable success (for example, HTTP 200 + data).

Modularity emphasis

They highlight that lifecycle steps are functions and would be “trivial to rewrite into Effect down the line,” since the structure is already functional.

---

TypeScript-first + operational consistency

• Everything is TypeScript (backend, frontend, scripts), avoiding language mixing to preserve consistency and team velocity.
• They use TypeScript’s type system for:
  • Strict compile-time correctness across endpoints/models.
  • Refactor automation: when adding fields/models, TypeScript errors force updates across the codebase.
• Tooling decisions support development speed:
  • Switching Jest → Vitest (noted as 10x faster tests).
  • Switching formatting/linting Prettier → Biome.

---

Why Effect is not yet in the core (resistance and adoption strategy)

Core pushback is framed as “shiny object syndrome”:

• Engineers already have:
  • their own Result type
  • and a functional structure
• Adding Effect can feel like:
  • namespace/learning overhead
• Questions they receive include:

  “Why can’t we keep using our own result type and plain functions?”

Their counterpoint: Effect becomes most valuable when you need cross-cutting concerns, such as:

• observability (spans/loggers),
• retry policies with correct semantics,
• consistent behavior composition.

Adoption strategy:

• Start with small, targeted usage (e.g., pipe for composition),
• then move toward deeper rewrites later.

---

Retry semantics & “don’t retry malformed responses”

A critical operational caution:

• Retry only under appropriate failure modes (e.g., network errors).
• Avoid retry loops that retry on malformed responses, because decode/validation failures could persist and cause huge wasted retry counts.

---

Generators and handling sync vs async code

• They discuss generators as a way to keep code review-friendly and sequential-looking (similar mental model to await).
• They mention an anti-pattern:
  • avoiding “two representations” of code (sync vs async variants),
  • generally preferring a unified style.
• They say some async generator usage is acceptable when unavoidable async exists inside, to improve developer ergonomics (reducing repeated manual error unwrapping).

---

Observability (spans/traces) constraints at the edge

They want better observability, but note constraints:

• Running on Worker runtime limits span collection.
• They currently hook into Node.js diagnostic channel, which they view as imperfect on worker platforms due to memory limits.
• They argue trace spans become essential as call stacks grow across many functions.

---

Agentic workflows & long-running requests (distributed systems framing)

They connect agentic/LLM workloads to distributed systems:

• Requests may remain open for 30 seconds to minutes due to long reasoning and streaming (SSE keep-alive).
• They compare agentic behavior to actor models / entity systems and message passing.
• They discuss keep-alive requirements for SSE-style streaming and timeouts across layered proxies/CDNs.

---

Agentic coding steering: need deterministic control loops

A parallel theme is steering multi-step systems:

• Models alone may not reliably maintain correct loops.
• Deterministic outside loops help, for example:
  • run deterministic checks (type errors/tests),
  • then call an LLM again with feedback (e.g., “oracle not happy”).

They suggest this mirrors the broader theme of adding higher-order orchestration around LLMs.

---

Main speakers / sources

• Louis Vichy — Co-founder of OpenRouter
• Johannes Schickling — Host of Cause & Effect podcast (Effect creator/presenter)
• Michael — Effect co-creator mentioned as present in the episode