Lesson 6 of 8 · 4 min read

How LLM function calling works

A language model can only do one thing: predict the next piece of text. It can't run code, open a website or check the time. Yet AI assistants "use tools" all the time. The trick is simple. The model writes text that asks for a tool, and an ordinary program does the rest.

The four pieces

Every function-calling system, from ChatGPT plugins to a 1B-parameter model in your browser, has the same four parts:

  1. Describe the tools to the model in its prompt.
  2. The model replies with text in an agreed format that names a tool and its arguments.
  3. Your code parses that text, runs the real function and gets a result.
  4. The result goes back to the model as more text, and it writes the final answer (or asks for another tool).

That's it. There's no hidden channel between the model and the tool. Let's look at each step with the exact text the playground sends.

1. Describe the tools

Tools are described with a name, a sentence about what they do, and a JSON Schema for the arguments. The description matters most: it's how the model decides when to use the tool.

{"name": "calculator",
 "description": "Evaluate a math expression. Supports + - * / % ^ ( ) and Math functions.",
 "parameters": {
   "type": "object",
   "properties": { "expression": { "type": "string" } },
   "required": ["expression"]
 }}

This JSON is pasted into the system prompt, together with instructions on how to call it:

To call a tool, reply with ONLY this, nothing else:
<tool_call>{"name": "<tool name>", "arguments": {...}}</tool_call>
You will then get the result inside <tool_response></tool_response>.

Tool definitions cost tokens. Every enabled tool takes space in the context window on every request, so a model with 20 tools has less room for your conversation.

2. The model writes a tool call

Ask "What is 23 × 19?" and a model that has learned this format replies with text like:

<tool_call>{"name": "calculator", "arguments": {"expression": "23*19"}}</tool_call>

To the model this is just more tokens. It chose these tokens because, during training, it saw many examples where a math question was followed by a call like this. Many open models (Qwen, Hermes and others) are fine-tuned on exactly this <tool_call> style.

3. Your code runs the tool

The app scans the reply for <tool_call>, parses the JSON, finds the tool by name and runs it with those arguments. In the playground, tools run in a sandboxed Web Worker with a 3-second limit. The model's arguments are untrusted input, so never run them with more access than they need.

The result is wrapped and sent back as a new message:

<tool_response>{"name": "calculator", "result": 437}</tool_response>

4. Back to the model: the agent loop

The model now sees the original question, its own tool call and the result. Usually it writes a plain-text answer: "23 × 19 = 437". Sometimes it calls another tool first. So the app runs a loop:

  1. Send the conversation to the model.
  2. If the reply contains a tool call, run it, add the result, and go to step 1.
  3. If the reply is plain text, that's the final answer.

The loop needs a limit (the playground stops after 5 model calls). Otherwise a confused model can call tools forever. This loop is what people mean by an AI agent: a model, some tools, and a loop.

What about "native" function calling?

APIs like OpenAI's accept a tools parameter and return structured tool_calls instead of text. It's the same mechanism with the formatting done for you. The server inserts your tool schemas into the prompt using the model's own chat template, and parses the model's output back into JSON. Underneath, the model is still generating tokens.

Why small models get it wrong

Small models (0.5B–2B parameters) follow the format less reliably. In the playground you'll see them:

Robust apps expect this. The playground accepts common mistakes (code fences, missing tags), sends errors back to the model so it can retry, and highlights every failure in red so you can see what went wrong.