--- import Base from "../layouts/Base.astro"; import bench from "../data/tool-bench.json"; import agentCost from "../data/agent-cost.json"; /** * Each tool against the shell command it replaces. * * The bars are measured, not typed: `scripts/tool-bench.sh` runs both sides in a checkout and * writes `tool-bench.json`, which names the repository and the commit. Each shell command is also * put through the other agents' own output limits, because no agent sends its model a raw 8 MB * `git log`: the honest comparison is with what they forward. A tool whose win is behavior rather * than size gets the two behaviors side by side instead. */ type Contrast = { shell: string; luxe: string }; type Tool = { name: string; instead: string; text: string; bench?: string; contrast?: Contrast }; type Group = { id: string; name: string; tools: Tool[] }; const groups: Group[] = [ { id: "files", name: "Files", tools: [ { name: "read", instead: "`cat`, `head`, `sed -n`", text: "Several files in one call, with numbered lines. An image comes back as a picture.", contrast: { shell: "Sends the whole file, however long, and the model can't see an image.", luxe: "A page at a time, stopping at a line boundary and saying where to pick up.", }, }, { name: "edit", instead: "`sed -i`", text: "Changes exact text, several edits in one call, all or none. It won't touch a file the agent hasn't read, or one that changed since.", contrast: { shell: "Exits 0 when the pattern matches nothing, so a missed edit looks like a done one.", luxe: "Refuses a miss and points at the closest line, refuses an ambiguous match, and shows the diff after formatting.", }, }, { name: "write", instead: "`cat > file`", text: "Creates or replaces a whole file, and shows what it wrote.", contrast: { shell: "Replaces whatever was there, unseen.", luxe: "Won't overwrite a file the agent hasn't read, keeps its line endings, and formats it.", }, }, ], }, { id: "search", name: "Search", tools: [ { name: "glob", instead: "`find`", bench: "glob", text: "Skips whatever `.gitignore` skips, so build output and dependencies stay out. Sorted by path, capped at 200, with a count of the rest.", }, { name: "grep", instead: "`grep -rn`, `rg`", bench: "grep", text: "Capped per file and overall, with long lines trimmed. It can return just the files that match or just the counts, and says so when there's more.", }, { name: "ast_grep", instead: "`grep` on code", text: "Search and replace by code structure. A rewrite shows a preview before it writes.", contrast: { shell: "Matches text, so comments and strings count too.", luxe: "Matches syntax: `$X.unwrap()` finds the calls and nothing else.", }, }, ], }, { id: "shell", name: "Shell and tests", tools: [ { name: "bash", instead: "`sh`", bench: "bash", text: "Long output comes back as the lines that matter, with every gap marked and the untouched text saved to a file, each stream whole up to 8 MB. A command that runs out of time returns what it printed so far.", }, { name: "test", instead: "`cargo test`, `npm test`, `pytest`", bench: "test", text: "One line when everything passes. When something fails, each failure with its `file:line`, and which ones your change broke. Tests run the project's way, through its own `.venv` or package manager.", }, ], }, { id: "git", name: "Git", tools: [ { name: "git", instead: "`git`, `gh`", bench: "git", text: "The last ten commits, with a note when there are more. Diffs are capped and say how to narrow them. Pull requests, issues and CI checks come through `gh`.", }, ], }, { id: "code", name: "Code intelligence", tools: [ { name: "read_symbol", instead: "`cat`", bench: "read_symbol", text: "One function, struct or method by name, instead of the file it lives in.", }, { name: "module_report", instead: "`cat`", bench: "module_report", text: "A file's outline: each symbol and the lines it spans, so the agent reads only what it needs.", }, { name: "symbol_search", instead: "`rg 'fn name'`", text: "Where something is defined, from its name or part of it.", contrast: { shell: "Matches text, and the pattern changes with every language.", luxe: "Finds definitions by name in any language, ranked, with what kind of thing each one is.", }, }, { name: "impact", instead: "nothing", text: "What else a change touches, before the agent makes it.", contrast: { shell: "Callers of callers is a search per hop, and tests don't say what they cover.", luxe: "Lists what calls a symbol, what depends on a file, and which tests cover it, a few hops out.", }, }, { name: "lsp_diagnostics", instead: "`cargo check`, `tsc`", text: "Errors and warnings for the files you name. They also come back on their own after every edit.", contrast: { shell: "Builds the whole project, one language at a time.", luxe: "Asks the language server about one file, in any language that has a server.", }, }, { name: "lsp_navigation", instead: "`grep` for a name", text: "Definitions, references, types and renames, from the language server.", contrast: { shell: "Every line that spells the same name.", luxe: "The definition and the real references, by meaning rather than spelling.", }, }, ], }, { id: "debug", name: "Debugging", tools: [ { name: "debug", instead: "print statements", text: "Drives a real debugger, with breakpoints, stepping and variables, for Rust, C, C++, Python and Go.", contrast: { shell: "Add a print, rebuild, rerun, repeat.", luxe: "Stops at a breakpoint and shows the frame and its locals, with library frames folded away.", }, }, ], }, { id: "web", name: "The web", tools: [ { name: "web_fetch", instead: "`curl`", bench: "web_fetch", text: "Pages come back as markdown, not HTML. A GitHub repository is cloned into the project, with its links flattened and no language server started in it, so the agent can search the real files.", }, { name: "web_search", instead: "`curl` a search page", text: "Searches the web through the provider you have a key for, or a free one.", contrast: { shell: "A page of HTML built for a browser.", luxe: "Titles, links and the passage that matched.", }, }, { name: "screenshot", instead: "`chrome --headless --screenshot`", text: "Captures a page, so the agent can check the UI it just built.", contrast: { shell: "Writes a file the model never sees.", luxe: "Returns the picture to the model, from your dev server or any page.", }, }, ], }, ]; const beyond = [ { name: "todo", text: "A checklist that survives compaction, shown in the dock's PLAN." }, { name: "question", text: "Stops to ask you, with choices, instead of guessing." }, { name: "skill", text: "Loads a skill's instructions when the task calls for them." }, { name: "mcp", text: "Your MCP servers behind one tool, every call checked against your permissions." }, ]; // The primary repository draws the bars; a tool it has no case for (`web_fetch` is not about a // repository) falls back to the next one that does const [primary] = bench.repositories; const rows = new Map( [...bench.repositories].reverse().flatMap((repo) => repo.rows.map((row) => [row.id, row])), ); for (const row of primary.rows) rows.set(row.id, row); /** The tightest of the four other agents' limits on the same output: Codex's. */ const tightest = (row: { forwarded: Record }) => Math.min(...Object.values(row.forwarded)); /** `code` spans in copy, with everything else escaped. */ const inline = (text: string) => text .replace(/&/g, "&") .replace(//g, ">") .replace(/`([^`]+)`/g, "$1"); /** About four bytes a token, which is close enough to show the scale. */ const tokens = (bytes: number) => { const count = bytes / 4; if (count < 1000) return `${Math.max(1, Math.round(count / 10) * 10)} tokens`; if (count < 10_000) return `${(count / 1000).toFixed(1)}k tokens`; return `${Math.round(count / 1000)}k tokens`; }; const times = (shell: number, tool: number) => { const ratio = shell / tool; if (ratio < 1.1) return "the same"; return ratio < 10 ? `${ratio.toFixed(1)}× less` : `${Math.round(ratio).toLocaleString("en-US")}× less`; }; /** Luxe against the tightest cut another agent makes, said both ways round. */ const versus = (cut: number, tool: number) => { if (tool <= cut * 1.05 && tool >= cut * 0.95) return "about what the tightest cut sends"; if (tool < cut) return `${times(cut, tool)} than the tightest cut`; const ratio = tool / cut; return `${ratio.toFixed(1)}× more than the tightest cut, with nothing cut out`; }; /** * Whole tasks against the other agents: each agent's cost on the same task as a multiple of * Luxe's, charged as a cold start. */ type Other = { warm: number; cold: number; passed: number; runs: number }; type CostRow = { model: string; luxe: { passed: number; runs: number } } & Partial< Record<"claude" | "codex" | "opencode" | "pi", Other> >; const costRows = agentCost.rows as CostRow[]; const vendorAgent = (row: CostRow) => row.claude ?? row.codex; const multiple = (other?: Other) => (other ? `${other.cold.toFixed(2)}×` : "–"); /** A bar's share of its row, never so thin it disappears. */ const share = (shell: number, tool: number) => `${Math.max((tool / shell) * 100, 0.6)}%`; ---

Tools

Better than the shell.

An agent can do almost anything with a shell, and that's the trouble. A find in a real project returns half a megabyte, a sed that matches nothing reports success, and the model pays for all of it in tokens and turns.

So every tool in Luxe has to beat the command it replaces: by sending less, by getting it right the first time, or by doing something the shell can't.

Measured in a checkout of {primary.name} at {primary.commit}, the codebase the harness eval runs in, with its virtualenv in place. The shell row is the command's raw output; tightest cut is the least any other agent sends its model of it, which is Codex's 10 KB, head and tail kept. Pi and OpenCode keep up to 50 KB and Claude Code 30,000 characters. A token is about four bytes.

Those are single calls, and across a whole task the bars shrink, because most calls are small to begin with. What a whole task costs is below.

Whole tasks

Luxe, Claude Code, Codex, OpenCode and Pi each ran the first twelve Django tasks of SWE-bench Verified Mini, with the same model, the same prompt and the same keys, every call metered at the network and priced from one list. Each figure is what the other agent spent on the same task, as a multiple of what Luxe spent: above 1× costs more than Luxe.

{ costRows.map((row) => ( )) }
Model Claude Code or Codex OpenCode Pi
{row.model} {multiple(vendorAgent(row))} {multiple(row.opencode)} {multiple(row.pi)}

Every run is charged as a cold start, the way a session begins after a break. Run side by side on a warm cache, Claude Code and OpenCode cost up to 2.6× what Luxe does on Claude models. Pass rates overlapped on every model. Pi is the cheapest by design: four tools and a short prompt, with web search, MCP and the rest left to extensions. Graded on one machine rather than in SWE-bench's own containers, so these are not leaderboard scores. Measured at {agentCost.commit}.

{ groups.map((group) => (

{group.name}

{group.tools.map((tool) => { const row = tool.bench ? rows.get(tool.bench) : undefined; return (

{tool.name}

instead of

{row && (
$ {row.shell} {tokens(row.shellBytes)}
tightest cut {tokens(tightest(row))}
{row.tool} {tokens(row.toolBytes)}
{times(row.shellBytes, row.toolBytes)} {times(row.shellBytes, row.toolBytes) === "the same" ? "as" : "than"} the shell · {versus(tightest(row), row.toolBytes)}
)} {tool.contrast && (
Shell
Luxe
)}
); })}
)) }

Beyond the shell

    { beyond.map(({ name, text }) => (
  • {name}

    {text}

  • )) }

Every tool

  • Time limits

    Every call has one, and ctrl+esc stops the whole batch.

  • Forgiving

    A call with an argument the tool doesn't know still runs, and says which it ignored.

  • Yours to change

    /tools shows what the agent can use, and any tool but the core four can be turned off. See the guide.