Pin your model IDs: when a “latest” alias tripled our price

Two incidents in one quarter on ezychat.io came from the same habit: trusting a name we didn’t control, and a path we never exercised.

The first changed our costs overnight without a single deploy. The second meant a customer’s request for a quotation went unanswered. Both are fixed now, and both fixes are simple enough to copy.

Incident 1: the alias moved

Like most teams, we referred to some models by the vendor’s convenience alias, the -latest style name that always points at the vendor’s current recommendation. It’s handy. You get upgrades for free.

On 21 July 2026 the vendor pointed one of those aliases at a newer model. The new model had three times the input price and six times the output price of the model the alias had been serving. Every AI feature using the alias changed model, behaviour and cost at the same moment, with no deploy, no pull request and no changelog on our side.

An alias is the vendor’s to move. That’s the whole lesson, but it’s worth spelling out what it costs you:

• Cost. Your per-call price can change by a multiple overnight.
• Behaviour. A different model answers differently. Your prompts, your evals and your grounding thresholds were tuned against the old one.
• Attribution. If you don’t record which model actually served each call, you can’t even tell which answers came from which model.

That last point is why every model call on ezychat.io writes a usage event with the stage, the model that actually served it and the tokens. A per-call ledger is what makes a silent switch visible at all.

What we changed: models are data, and changes are migrations

Model choice now lives in a model_policies table, not in code defaults, and every entry is an exact model ID, never an alias. The defaults that remain in code (media understanding, OCR, extraction, the eval judge) are pinned the same way.

Changing a model is a database migration. A migration adds a new policy version rather than editing the old one, and every AI reply records the policy version it ran on. That gives us three things:

1. An audit trail. For any answer a customer received, we can say which model produced it and under which policy.
2. Comparable evals. Quality and cost can be compared across policy versions, on the same golden sets.
3. A clean rollback. A downgrade reactivates the previous version’s rows. Because replies recorded their version, history stays truthful.

The resolved policy is cached in Redis for 60 seconds, so a new policy serves within a minute of the migration, with no redeploy needed.

Incident 2: the dead failover

The second incident was quieter and worse.

Every call had a backup model, on a different vendor. On paper that’s resilient. In practice, production had only ever been configured with the primary vendor’s key. The function that built a client for the backup raised a “provider not configured” error before the failover loop’s try block. A single 503 from the primary model therefore failed the call outright. The backup was never tried, because constructing it crashed first.

On 2 October 2026 that path fired, and a customer’s quotation request went unanswered. Worse, the advanced tier’s reply model sat on the unconfigured vendor, so a business on that tier couldn’t have received an AI reply at all.

Failover code is the code you run least. If you’ve never exercised the path, assume it doesn’t work.

The failover rules now

A vendor with no key is skipped, not raised. It’s logged once per process, so the gap is visible, and the next candidate runs. A missing credential is a configuration fact, not an exception that should take down a customer’s turn.

Circuit breakers are per model, not per vendor. An outage is usually one model’s capacity, not the whole vendor. A per-vendor breaker would also block the same vendor’s backup model, which was still answering.

A call never falls back to its own model. When the backup model for a tier is the same model a call is already using, that call falls back to the next model in the policy instead. “Retry the thing that just failed” isn’t failover.

Budgets and back-pressure are per model too. Per-model request budgets live in Redis, and a 429 pauses that model rather than hammering it.

A simplified shape of the loop:

async def generate_with_failover(policy, request):
    errors = []
    for model in policy.candidates_for(request.model):   # never includes request.model twice
        client = clients.get(model.vendor)
        if client is None:                                # no key for this vendor: skip, don't raise
            log_once(f"policy names {model.id} but {model.vendor} has no API key")
            continue
        if breakers.is_open(model.id):                    # one breaker per model, not per vendor
            continue
        try:
            result = await client.generate(model.id, request)   # exact, pinned model ID
            breakers.record_success(model.id)
            ledger.record(model=model.id, policy_version=policy.version, usage=result.usage)
            return result
        except (RateLimited, ServiceUnavailable) as e:
            breakers.record_failure(model.id)
            errors.append(e)
    raise AllModelsFailed(errors)                         # the caller hands the conversation to a person

The last line matters as much as the loop. When everything fails, the conversation goes to a person, the same path described in No lost turns. Failover reduces how often that happens. It isn’t allowed to replace it.

Test the failure path, not just the happy one

Both incidents shared a root cause: a path that existed in code but had never been exercised in the conditions that mattered. A few cheap tests would have caught both:

• Remove the backup vendor’s key in a test and assert the call is skipped and logged, not raised.
• Force a 503 from the primary model and assert the backup answers.
• Assert every configured model ID is pinned. A one-line check that no policy row ends in -latest would have flagged the first incident before it shipped.
• Diff cost per call by model in your dashboards, so a model switch shows up as a step change you can see.

A checklist for production LLM features

1. Pin exact model IDs everywhere, including code defaults and eval judges.
2. Store model choice as versioned data, and record the version on every answer.
3. Keep a per-call ledger of which model served each request, with its tokens and price.
4. Skip unconfigured vendors instead of raising.
5. Trip circuit breakers per model, not per vendor.
6. Never fall back to the model that just failed.
7. When everything fails, hand the conversation to a person.
8. Test the failover path with a missing key and a forced outage.

If you’re putting LLM features into production and want them to survive vendor changes, that’s the work I take on.