Using a small model as a classifier, not a chat partner

A quietly common pattern among people building product pipelines: the model is not something you talk to, it is a component. It reads a row of data — a support ticket, a review, a signup — and emits a label, a category, a yes/no. Run it over three months of history once, then over new rows on a schedule. No chat, no streaming, nobody watching it work.

This is where small open models genuinely shine. A narrow task with a fixed prompt does not need frontier intelligence, and a lightweight model of the Gemma or Qwen class is often enough — cheap to run, small enough for a Mac you already own, and yours to keep. The open-weight part matters here for a practical reason: a classifier baked into a workflow should not silently change under you, and a downloaded model never does. A hosted model can be deprecated or retrained; the file on your disk is the same file next year.

Your labelled sample is the only benchmark that matters. Leaderboards rank models on general ability, and none of them know your categories. Label a few hundred rows by hand, hold some back, and measure the model against the held-out set — that number is real in a way no public score is. Re-check it on fresh rows every so often: the world your data describes drifts, even when the model doesn't.

Ask for structure, then verify it. Have the model answer in a fixed shape — a JSON object, a single label from a list you give it — and validate every response before it enters your pipeline. A classifier that occasionally writes a paragraph instead of a label is a bug you catch with three lines of validation, not with hope.

Where to run it. Batch work is the most forgiving workload there is: nobody is waiting, so a Mac churning through rows overnight is a perfectly serious deployment. Bigger backlogs fit the serverless GPU rung — pay for the hours the batch actually takes. And if the rows are customer data, the local option quietly answers the privacy questions before they're asked.

Where next: Small models in the catalogue · Buying a Mac for local models · Renting a GPU in the cloud