Building an AI Chatbot With OpenAI-Compatible APIs

Building a chatbot used to mean either paying for a no-code platform that locks you in, or wrestling with a provider's proprietary SDK. The rise of the OpenAI-compatible API changed that: you can now build a capable chatbot in a few hundred lines, against a standard you can point at any model.

Here's the full path from "it replies" to "it's actually useful."

Why Build on OpenAI-Compatible APIs

The reason is leverage. The OpenAI-compatible chat completion shape — send a list of messages, get a reply — is supported by almost every provider. Write your chatbot once against that shape, and you can swap the underlying model by changing a base URL and a key.

This matters for three reasons: cost control (route cheap models for simple queries, expensive ones for hard ones), resilience (fall back to another provider when one fails), and no lock-in (switch providers without rewriting).

The Minimal Working Chatbot

The core loop is small. In concept:

messages = [{"role": "system", "content": "You are a helpful assistant."}]

loop:
    user_input = read_input()
    messages.append({"role": "user", "content": user_input})
    reply = chat_completion(model, messages)   # via base_url + api_key
    messages.append({"role": "assistant", "content": reply})
    print(reply)

That's the entire structure of a chatbot. Everything beyond this is about making it useful and reliable, not making it work.

What Takes It From Demo to Useful

1. A real system prompt

The system message is where you define the assistant's behavior, tone, and constraints. This is the single highest-leverage thing you can write. Be specific: what it is, what it should and shouldn't do, how it should respond.

2. Conversation history management

You can't send the whole history forever — there's a context limit, and cost grows with every message. Keep a window of recent messages, and summarize or drop older ones. This is where most simple chatbots quietly fall apart at scale.

3. Streaming for responsiveness

Users perceive a slow chatbot as broken. Streaming the reply token by token — which the OpenAI-compatible format supports — makes the chatbot feel instant even when the model is slow.

4. Error handling and fallbacks

Models fail, rate limits hit, providers have outages. A simple retry and a fallback to a second provider turn these from user-facing failures into non-events.

5. Tool use (if you need it)

If your chatbot needs to look things up or take actions, the OpenAI-compatible format supports function calling. This is where divergence between providers shows up most, so test it against each provider you use.

The Architecture That Scales

As it grows, a clean chatbot separates into a few pieces:

The router and fallback layer are what make it production-grade: they let you use cheap models for simple queries and keep things working when one provider fails.

What Didn't Work

Unbounded message history. My first version sent the full history every turn. Cost grew linearly and it eventually broke on context limits. A history window with summarization fixed both.

Ignoring streaming. The non-streaming version felt broken even when it worked, because the user waited in silence. Streaming changed the perceived quality dramatically with almost no code.

No fallback. The first provider outage produced a hard failure. A simple try-next-provider fallback made outages invisible. Build it before you need it.

Assuming tool use "just works." Function calling was the least standardized part of the "compatible" API. I had to test and adapt per provider. Don't assume; verify.

Over-engineering the system prompt. A giant prompt made the bot slower and not much better. Specific and concise beats exhaustive — the model follows a few clear rules better than fifty vague ones.

Verdict

Start with the minimal loop, then add the four things that make it real: a specific system prompt, history management, streaming, and fallbacks. Add tool use only if the job requires it, and verify it against each provider.

The OpenAI-compatible abstraction is the quiet win here — it turns "which model do I build on" from an architectural decision into a config value, and that flexibility is worth more than any single model choice.

FAQ

What do I need to build a chatbot? Very little — a language model API (OpenAI-compatible), and the ability to make an HTTP request and manage a message list. The minimal loop is a few dozen lines.

How do I keep costs down? Route simple queries to a cheap model and reserve the expensive one for hard queries. Also manage history aggressively — unbounded history is the most common source of cost creep.

Do I need tool use for a useful chatbot? No. Most useful chatbots are just good system prompts plus good history management. Add tool use only when the bot needs to look things up or act.

Can I switch models later without rewriting? Yes, if you build on the OpenAI-compatible shape — changing the base URL and key switches providers, and a router lets you do it per-request. That flexibility is the main reason to build this way.