How to Use LLM APIs: Build Your First Chatbot in Python

An LLM API is an HTTP endpoint that accepts text and returns generated text. A chatbot is a loop around that endpoint with one extra feature: you resend the conversation history on every turn. This guide builds that loop in Python, adds streaming and token management, and shows how to make it safe for real users.

Quick Takeaways

• LLM APIs are stateless. The model remembers nothing between calls. Your code stores the history and sends the full messages array each turn.
• A working chatbot needs about 40 lines of Python and one dependency: the official SDK.
• Three parameters control most behavior: max_tokens (output cap), temperature (randomness, 0.0-1.0 on Anthropic’s API), and the system prompt (persona and rules).
• Cost scales with tokens, not messages. Long histories get expensive, so trim or summarize them.

How LLM APIs Actually Work

Every request carries four things: a model name, a list of messages, a token limit, and optional settings. The server runs inference and returns a response object with the generated text and usage metadata: input tokens and output tokens.

A token is a chunk of text, roughly 3-4 characters of English. Providers bill input and output tokens separately. Output tokens usually cost more.

Three facts shape every design decision:

1. Statelessness. There is no server-side session. If you send only the latest question, the model has no idea what came before.
2. Context window. Each model accepts a maximum number of tokens per request (input plus output). Exceed it and the call fails or truncates.
3. Non-determinism. The same prompt can yield different outputs. Lower temperature to reduce variance.

Prerequisites and Setup

You need Python 3.9+, an API key from your provider’s console, and a virtual environment.

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install anthropic

Store the key as an environment variable. Never hardcode it or commit it to Git.

export ANTHROPIC_API_KEY="your-key-here"

The SDK reads ANTHROPIC_API_KEY automatically.

Step 1: Make Your First API Call

Start with a single request.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

response = client.messages.create(
    model="claude-sonnet-5-5",   # check docs.claude.com for current model IDs
    max_tokens=512,
    messages=[
        {"role": "user", "content": "Explain tokens in one paragraph."}
    ],
)

print(response.content[0].text)
print(response.usage.input_tokens, response.usage.output_tokens)

response.content is a list of content blocks. For plain text, the first block holds your answer. The usage fields tell you exactly what the call cost in tokens. Log them from day one.

Step 2: Add a System Prompt

The system prompt sets behavior that persists across the conversation. On Anthropic’s API it is a separate system parameter, not a message in the array.

SYSTEM = (
    "You are a concise Python tutor. "
    "Answer in under 120 words. "
    "Use code examples only when they clarify the answer."
)

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=512,
    temperature=0.3,
    system=SYSTEM,
    messages=[{"role": "user", "content": "What is a list comprehension?"}],
)

Write system prompts as constraints, not wishes. “Answer in under 120 words” beats “be brief.”

Step 3: Build the Chat Loop with Memory

Memory is just a Python list. Append the user message, call the API, append the assistant reply, repeat.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"
SYSTEM = "You are a friendly, concise assistant."

history = []  # list of {"role": ..., "content": ...}

def chat(user_input: str) -> str:
    history.append({"role": "user", "content": user_input})
    response = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        system=SYSTEM,
        messages=history,
    )
    reply = response.content[0].text
    history.append({"role": "assistant", "content": reply})
    return reply

if __name__ == "__main__":
    print("Chatbot ready. Type 'quit' to exit.")
    while True:
        text = input("You: ").strip()
        if text.lower() in {"quit", "exit"}:
            break
        if not text:
            continue
        print("Bot:", chat(text))

Run it, tell it your name, then ask “What’s my name?” It answers correctly because the full history list goes out with every request.

Rule: messages must alternate between user and assistant roles, starting with user.

Step 4: Stream Responses for Better UX

Without streaming, users stare at a blank terminal until the full response finishes. Streaming returns tokens as they are generated, which cuts perceived latency sharply.

def chat_stream(user_input: str) -> str:
    history.append({"role": "user", "content": user_input})
    reply = ""
    with client.messages.stream(
        model=MODEL,
        max_tokens=1024,
        system=SYSTEM,
        messages=history,
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)
            reply += text
    print()
    history.append({"role": "assistant", "content": reply})
    return reply

Swap chat() for chat_stream() in the loop. Time-to-first-token typically drops to a fraction of a second, even when the full answer takes several seconds.

Step 5: Control Context and Cost

Every turn resends the whole history, so input tokens grow with each exchange. A 20-turn chat can cost many times more than a 2-turn one.

Three strategies keep this in check:

Here is a minimal sliding window:

MAX_TURNS = 10  # 10 user + 10 assistant messages

def trimmed(history):
    window = history[-MAX_TURNS * 2:]
    # the array must start with a user message
    while window and window[0]["role"] != "user":
        window = window[1:]
    return window

Pass messages=trimmed(history) into the API call. For production bots, combine a window with a running summary.

Step 6: Handle Errors and Rate Limits

Production code fails in predictable ways. Handle these:

import time
import anthropic

def safe_call(messages, retries=3):
    for attempt in range(retries):
        try:
            return client.messages.create(
                model=MODEL, max_tokens=1024,
                system=SYSTEM, messages=messages,
            )
        except anthropic.RateLimitError:
            time.sleep(2 ** attempt)   # exponential backoff
        except anthropic.APIConnectionError:
            time.sleep(1)
        except anthropic.APIStatusError as e:
            print("API error:", e.status_code)
            raise
    raise RuntimeError("API unavailable after retries")

Common status codes: 400 (malformed request), 401 (bad key), 429 (rate limit), 5xx (provider-side). Retry 429 and 5xx with exponential backoff. Never retry 400 or 401.

Provider Comparison: Same Chatbot, Different APIs

The loop pattern is identical across vendors. Only the details differ.

Many open-source servers expose an OpenAI-compatible endpoint, so one abstraction layer can often serve several backends. Check each provider’s pricing page for current cost per 1M tokens, since rates change often.

Practical Workflows and Real-World Use Cases

Customer support triage bot

Use a system prompt that defines categories and an output format:

SYSTEM = """You classify support tickets.
Return JSON only: {"category": "billing|bug|account|other",
"urgency": 1-5, "summary": "<20 words"}."""

Set temperature to 0.0-0.2 for consistent labels, then parse the output with json.loads(). Add validation and a fallback for malformed JSON.

Retrieval-augmented support (RAG) to cut hallucinations

Models invent facts when they lack context. Enterprises reduce this by retrieving relevant documents first and inserting them into the prompt:

1. Embed the user question.
2. Search a vector store for the top 3-5 chunks (typically 300-800 tokens each).
3. Insert the chunks into the prompt with an instruction: “Answer only from the context below. If it’s missing, say you don’t know.”

That last instruction matters as much as the retrieval.

Prompt template for reliable answers

Role: You are a {domain} expert.
Task: {task in one sentence}.
Context: {retrieved or pasted material}
Constraints: Max {N} words. Cite the context section used. Say "unknown" if unsure.
Output format: {bullets | JSON | table}

Fixed structure makes outputs predictable and easier to test.

Safety checklist before shipping

• Keep API keys server-side. Never expose them in browser code.
• Set a hard max_tokens and per-user rate limits.
• Log token usage per session to catch runaway costs.
• Treat user input as untrusted. Guard against prompt injection when the bot reads external content.
• Add a human handoff for sensitive topics.

Next Steps: Extend Your Chatbot

Once the base loop works, add capabilities one at a time:

1. Persistence. Save history to SQLite or Redis keyed by user ID.
2. Tool use. Let the model call your functions (weather, database lookups) through the API’s tool-calling feature.
3. A web front end. Wrap chat_stream() in FastAPI and stream with server-sent events.
4. Evaluation. Build a test set of 20-50 prompts with expected behaviors and re-run it after every prompt change.

FAQ

How do I use an LLM API in Python?

Install the provider’s SDK, set your API key as an environment variable, and call the messages endpoint with a model name, a list of messages, and a max_tokens limit. The response contains the generated text and token usage.

Why does my chatbot forget previous messages?

LLM APIs are stateless. You must store the conversation in a list and send the entire history with each request. If you send only the newest message, the model has no earlier context.

How much does it cost to run an LLM chatbot?

Cost equals input tokens plus output tokens, each billed at the provider’s per-million-token rate. Long conversations cost more because the full history is resent every turn. Trim history, cache static prompts, and set max_tokens to control spend.

Which is better for beginners: a hosted API or a local model?

Start with a hosted API. Setup takes minutes and needs no GPU. Move to a local or self-hosted open-source model when you need data privacy, offline use, or lower cost at high volume.