← All writing
// May 8, 2026

Running LLMs locally and in the cloud: a fallback strategy

  • Node.js
  • OpenAI
  • Ollama
  • Supabase

Document Explainer lets you upload a PDF and ask questions about specific passages — with chat, summaries, and study guides. One of the earliest decisions was: which model answers the question?

The answer turned out to be “it depends,” so I built for both.

Two modes, one interface

The app supports two backends:

  • Cloud — OpenAI, for the best quality and zero local setup.
  • Local — Ollama running Mistral on the user’s machine, for privacy and offline use (think: sensitive legal or medical documents).

Both implement the same provider interface, so the request layer never branches on which one is active. A ModelProvider has one method: given a prompt and context, return a stream of tokens.

The fallback chain

The interesting part is what happens when the preferred provider fails. A naive try/catch that silently switches to OpenAI would leak private documents to the cloud — exactly what someone running the local mode is trying to avoid. So the fallback is policy-aware:

async function answer(query: string, ctx: DocContext, policy: Policy) {
  const order = policy.allowsCloud
    ? [providers.openai, providers.local]
    : [providers.local]; // never escalate private docs to cloud

  for (const provider of order) {
    try {
      return await provider.complete(query, ctx);
    } catch (err) {
      if (isTransient(err)) continue; // timeout / rate limit → try next
      throw err;                      // hard error → surface it
    }
  }
  throw new Error("All providers unavailable");
}

Transient errors (timeouts, rate limits) fall through; auth or quota errors surface immediately. And if the document is marked private, the cloud provider isn’t in the chain at all — there’s no code path that can accidentally send it.

Retrieval over the whole document

“Explain this highlighted text” isn’t just a prompt — it needs surrounding context. I chunk the PDF at upload time, embed each chunk, and store vectors in pgvector via Supabase. A highlighted passage becomes a retrieval query; the top chunks get folded into the prompt. This is what lets the model reference a figure three pages away from the selection.

What surprised me

The local mode is more popular than I expected. I assumed everyone would default to the better cloud model, but for a lot of users “it never leaves my laptop” is the whole feature. Designing the fallback around their privacy preference — not just uptime — was the real lesson.