Osaurus: A Feather-Light, Apple-Silicon-Only LLM Server That Runs Rings Around Ollama

Last updated: 26 Aug 2025

If you own an Apple-silicon Mac and want a truly local, offline chatbot that weighs less than a PDF, let me introduce Osaurus: a 7 MB, open-source, Swift-native LLM server built on Apple’s MLX framework. It claims to be 20 % faster than Ollama, speaks the OpenAI REST API fluently, and runs entirely on your laptop without a single cloud call.

Below you’ll find everything you need—no fluff, no hype—to decide whether Osaurus deserves a spot in your toolkit.

Table of contents

1. What exactly is Osaurus?
2. Compatibility checklist: will it run on my Mac?
3. Installation in three minutes (drag-and-drop or source)
4. First launch: a walk-through of the model manager
5. Talking to the server: curl, Python, and JavaScript examples
6. Benchmarks: time-to-first-token, throughput, and reliability
7. Power-user features: KV-cache reuse, tool calling, chat templates
8. FAQ: storage paths, Intel Macs, Whisper, and more
9. When to choose Osaurus—and when not to

1. What exactly is Osaurus?

In plain English, Osaurus is a tiny, native macOS app that turns your M-series Mac into an OpenAI-compatible inference endpoint. Think of it as Ollama’s younger, lighter cousin that only works on Apple Silicon and only runs locally.

Key differences at a glance

You’ll notice no Windows build, no Intel build, and no Docker image—Osaurus sticks to Apple’s ecosystem like glue, trading reach for raw speed.

2. Compatibility checklist: will it run on my Mac?

Ask yourself three quick questions:

If any answer is “no,” stop here—Osaurus won’t work. Otherwise, read on.

3. Installation in three minutes

Option A: ready-made DMG (no coding)

1. Visit the GitHub Releases page.
2. Download Osaurus.dmg.
3. Drag the app into /Applications.
4. First launch? macOS will warn you the developer is “unverified.” Go to System Settings → Privacy & Security → Allow and you’re done.

Option B: build from source (developers)

git clone https://github.com/dinoki-ai/osaurus.git
cd osaurus
open osaurus.xcodeproj

Choose the osaurus target, press ⌘R, and Xcode compiles the SwiftUI frontend plus the embedded SwiftNIO server in one shot.

4. First launch: a walk-through of the model manager

The UI is deliberately minimal:

Default storage: ~/Documents/MLXModels.
Override: launch with export OSU_MODELS_DIR=/Volumes/SSD/mlx.

5. Talking to the server: curl, Python, and JavaScript examples

Osaurus exposes three endpoints you already know:

5.1 List models with curl

curl -s http://127.0.0.1:8080/v1/models | jq

Sample JSON:

{
  "object": "list",
  "data": [
    {
      "id": "llama-3.2-3b-instruct-4bit",
      "object": "model",
      "created": 1724588800,
      "owned_by": "mlx-community"
    }
  ]
}

5.2 Non-streaming chat

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-3.2-3b-instruct-4bit",
        "messages": [{"role": "user", "content": "Write a haiku about dinosaurs"}],
        "max_tokens": 200
      }'

Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Ancient thunder lizard,\nshadows stretch across time,\nfossils whisper tales."
      },
      "finish_reason": "stop"
    }
  ]
}

5.3 Streaming chat (SSE)

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-3.2-3b-instruct-4bit",
        "messages": [{"role": "user", "content": "Explain quantum tunneling in one paragraph"}],
        "stream": true
      }'

The terminal will print deltas exactly like ChatGPT’s web UI.

5.4 Python snippet with the official OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="osaurus")

response = client.chat.completions.create(
    model="llama-3.2-3b-instruct-4bit",
    messages=[{"role": "user", "content": "Hello from Python!"}],
)
print(response.choices[0].message.content)

5.5 JavaScript (browser or Node)

const response = await fetch("http://127.0.0.1:8080/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "llama-3.2-3b-instruct-4bit",
    messages: [{ role: "user", content: "Hello from JavaScript!" }],
  }),
});
const data = await response.json();
console.log(data.choices[0].message.content);

6. Benchmarks: time-to-first-token, throughput, and reliability

The maintainers ran 20 iterations on the same Apple-silicon hardware (exact model not disclosed) and averaged the results:

What it means for you:

• TTFT (time-to-first-token) is slightly higher on Osaurus—likely because MLX’s memory mapping warms up slower than llama.cpp.
• Throughput (chars/sec) is highest on Osaurus once generation starts.
• All servers achieved 100 % success over the test suite, so stability is not a differentiator.

7. Power-user features

7.1 KV-cache reuse via session_id

Multi-turn conversations can be accelerated by reusing the KV cache. Just keep the same session_id across requests.

Example:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-3.2-3b-instruct-4bit",
        "session_id": "trip-planning",
        "messages": [
          {"role": "user", "content": "Suggest a two-day itinerary in Kyoto"}
        ]
      }'

Rules:

• The cache is opportunistically reused only if no other request is using that session_id.
• There is no manual eviction; unused sessions expire automatically after a short idle window.

7.2 Tool / function calling (OpenAI-compatible)

Osaurus supports the exact JSON schema that OpenAI clients expect.

Step 1: declare tools

"tools": [
  {
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get current weather by city name",
      "parameters": {
        "type": "object",
        "properties": { "city": { "type": "string" } },
        "required": ["city"]
      }
    }
  }
]

Step 2: send request

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-3.2-3b-instruct-4bit",
        "messages": [
          {"role": "system", "content": "You can call functions to answer concisely."},
          {"role": "user", "content": "What is the weather in Tokyo?"}
        ],
        "tools": [...],
        "tool_choice": "auto"
      }'

Step 3: execute the returned tool call on your side, then continue the conversation with a role: tool message.

Osaurus gracefully handles malformed JSON, code fences, and extra whitespace, so you don’t need to sanitize model output.

7.3 Chat templates (Jinja)

When a model’s tokenizer_config.json contains a chat_template, Osaurus renders it with:

• messages array
• add_generation_prompt: true
• bos_token and eos_token if defined

If the template is missing or fails to render, a simple transcript fallback is used:

User: ...
Assistant: ...

System messages are automatically prepended so you never need to manually format prompts.

8. FAQ: quick answers to common questions

Q: Where are the models stored?
A: Default is ~/Documents/MLXModels. Override with the environment variable OSU_MODELS_DIR.

Q: Can I run this on an Intel Mac?
A: No. Osaurus relies on Apple’s MLX, which only supports Apple Silicon.

Q: Is there Whisper support?
A: Not yet. The /transcribe endpoints are placeholders for a future release.

Q: How do I expose the server to my LAN?
A: Bind address is hard-coded to 127.0.0.1. Place a reverse proxy (e.g., Nginx) in front if you need external access.

Q: Does it work with LangChain or LlamaIndex?
A: Yes—any library that speaks the OpenAI REST API works out of the box. Just point the base_url to http://127.0.0.1:8080/v1.

9. When to choose Osaurus—and when not to

Osaurus is not trying to be everything for everyone. It is a laser-focused tool that squeezes every last drop of performance out of Apple Silicon while staying compatible with the OpenAI ecosystem you already know.

If that matches your hardware and your goals, download the latest release and give it a spin—you might find that 7 MB is all you need.