Calling LLMs from Python: LangChain, LiteLLM, and HF Transformers

Most "getting started" LLM tutorials end right after: call the API and print the response. That's like declaring your startup "enterprise-ready" because you bought a domain and set up a Slack workspace.

That's demo-ready. Not production-ready.

What comes next is the boring stuff that keeps your app alive after the first customer shows up: repeatability, reliability, cost control, and measurability. None of this is glamorous. All of it is what separates "cool prototype" from "system that runs for six months without waking you up at 2am."

We'll stay vendor-agnostic, but I'll name names when it helps.

1) Stop building prompts with string concatenation like it's 1997

If your prompts are built with f"..." plus three + signs, you're one refactor away from chaos. And yes, I've watched three different startups debug the same prompt injection bug because they were doing string concatenation. Don't be startup number four.

Do the boring, grown-up thing:

Use actual templates. Version them. Keep user data away from your instructions so you don't accidentally inject someone's email address into your system prompt.

If you like LangChain, its prompt tools are fine:
https://python.langchain.com/docs/concepts/prompt_templates/

If you don't, a tiny template works too:

from string import Template

# Keep instructions stable. Keep user content clearly separated.
PROMPT = Template(
    "You are a helpful assistant.\n"
    "Return JSON only.\n"
    "Question: $question\n"
)

prompt = PROMPT.substitute(question=user_question)

2) Retries + rate limits: be boring on purpose

APIs fail. Networks wobble. You will get rate-limited. Handle it like an adult. Exponential backoff with jitter is boring, which means it works.

import time
import random
import litellm  # https://github.com/BerriAI/litellm

def call_with_retry(**kwargs):
    # Yes, 6 attempts. Because 5 wasn't enough and 7 is overkill.
    for attempt in range(6):
        try:
            return litellm.completion(**kwargs)
        except Exception as e:
            # Backoff + jitter to avoid synchronized retry storms.
            sleep = (2 ** attempt) + random.random()
            time.sleep(sleep)

    raise RuntimeError("LLM call failed after retries")

LiteLLM is great here because it normalizes providers (OpenAI, Anthropic, etc.) behind one interface. I've used it on three different projects and the API stays consistent even when OpenAI decides to change their response format for the fifth time.

3) Stream responses for better UX

Even if the model takes 8 seconds, streaming makes it feel fast. Users care about "time to first token," not your feelings about latency budgets.

resp = litellm.completion(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    stream=True,
)

for event in resp:
    # Sometimes it's content, sometimes it's empty. Streaming is a journey.
    chunk = event["choices"][0]["delta"].get("content", "")
    print(chunk, end="", flush=True)

4) Cache deterministic calls (save money, reduce latency)

If temperature=0 and the input is the same, cache it. This is not "premature optimization." This is "not lighting money on fire."

Keep it simple:
Key by model, prompt, and settings. Store the response and some metadata. Expire it when you change your prompts. This isn't rocket science, it's just being cheap in a smart way.

5) Structured outputs: stop parsing vibes

Use JSON mode or tool/function calling. Free-text parsing is a hobby, not a strategy.

Yes, you can write a regex that parses JSON-ish text 80% of the time. You can also cut your own hair. Neither is a good idea when better options exist.

OpenAI structured outputs: https://platform.openai.com/docs/guides/structured-outputs
Anthropic tool use: https://docs.anthropic.com/en/docs/tool-use

If you're building more complex integrations, consider when to write an MCP versus a simple tool for your LLM application architecture.

6) Logging without leaking user data

Log enough to debug, not enough to end up in a compliance meeting explaining why customer social security numbers are in your application logs.

The practical version:
- Hash user identifiers (or use stable internal IDs)
- Redact PII (and do it before logs leave the box)
- Store prompts separately from raw user text
- Keep a "golden set" of safe test inputs you can replay anytime

7) Set up evals before you're lost

Nobody does this early enough. You'll spend two weeks tweaking prompts, then realize you have no idea if version 47 is better than version 12. Build the eval harness when you write your first prompt.

You don't need a PhD-grade benchmark. You need consistency:
- A handful of test cases
- Expected traits (correctness, tone, JSON validity)
- A score you can track over time

If you want a framework, look at OpenAI Evals:
https://github.com/openai/evals

Or roll your own. The point is you can ship changes without guessing.

None of this is glamorous. But the difference between a demo and a system that runs in production for six months is exactly this kind of boring infrastructure work. Build it early, while you still have the energy and before your customers become your monitoring system.

If you shipped your MVP quickly and are now dealing with the vibe coding hangover, these patterns will help you stabilize your system. And if you're working with async Python code for concurrent LLM calls, check out these Python asyncio recipes for production-ready patterns.

-Sethers

Calling LLMs from Python: LangChain, LiteLLM, and HF Transformers