Foresound tracks music trends. At Velais we rebuilt the pipeline behind it. It collects posts from YouTube, TikTok, Reddit and Instagram, and one LLM step reads each post and returns its artists, its genres and a set of typed tags. Everything downstream, every trend and every chart, is built on that step.
Because everything rests on that one step, it has to be measured. It runs on millions of posts, so a prompt that is a little worse is a little worse everywhere, and a model that is a little more expensive is a lot more expensive in total. Reading a few outputs and deciding they look fine doesn't catch that.
From a script to a harness
The brief was to improve the tagging measurably without raising its cost by much, scaling up gradually from a hundred posts to a thousand. The obvious way to do that is a script. You plug in a model, run some posts and read what comes back. It answers one question, once.
We built a framework instead, where the model and the prompt are variables. The gold data stays fixed. Models live in one list, each with an endpoint, a request shape and a price. Prompts are versioned files. Every run gets scored the same way, with confidence intervals. A new model is a line in a list, a new prompt is a file, and any run can be compared to any other.
Regex, measured for the first time
The tagging that shipped before was regex, and nobody had measured it. The first run put it next to the LLM prompt, with regex as the baseline:
| artists | genres | music posts with no genre | |
|---|---|---|---|
| regex | X | X | X |
| LLM prompt | ≈ 4X | ≈ 2.3X | ≈ X / 12 |
Gold data, labelled by about 800 agents
The gold sets were most of the work, because an eval is only as good as its answer key.
Claude Code subagents labelled the first sets. They worked from a rubric written without looking at the production prompt, so they couldn't pick up its habits. A hundred posts got two independent passes, and how often those two passes agreed became the ceiling. I don't expect a model to beat two labellers who disagree with each other.
Names that appear only in a tag wall the post itself is not about — a reach-stuffing tag list naming twenty artists does not make the post about twenty artists.
These are not genres: production or format descriptions (
instrumental,live,remix,cover,8D audio,slowed + reverb); mood or function words (relaxing,study,workout,sleep,chill).Label channels: the label is not an artist. Find the act.
Never invent an artist or genre that nothing in the post supports.
I built the larger sets in Cursor. Tens of conversations each ran waves of ten parallel agents, and each agent reviewed five posts. That came to about 800 agents and sets from 100 to 3,000 posts. Some of those sets are held-out exams we never tune against, and if a person or an agent read a post during tuning, it comes out of the exam.
More than artists and genres
Writing the gold forced us to decide what correct means, and that cleaned up far more than the two fields we started with. "drake type beat" is not a genre. An artist is "Dave Lewis", not "Dave". Genres are only what the post states, because in a trend pipeline we shouldn't assume. Posts that aren't about music should get nothing at all, so contamination became a metric of its own. The typed tags, which track what shows up together with what, moved from regex to the model and got graded on their own.
What we got wrong
Most of what the harness taught us was that our guesses were off.
We assumed a shorter prompt would be cheaper. It wasn't. Most of the bill is what the model writes, not what it reads, and both shorter prompts we tried did worse on top of costing the same.
A test on 66 posts said low reasoning effort hurt genre accuracy. On the full set of 905 posts the same setting was 27% cheaper. Artists dipped by about 1% and genres went slightly up. If we had stopped at 66, we would have paid for reasoning we didn't need.
It went the other way too. The cheaper model we ended up choosing matched production on artists at 500 posts, then dropped about 9% on posts it hadn't seen. We withdrew that result. Parity came later, with a later prompt. Another gap between the two models came from the gold sets, not the models. One set was 55% YouTube and the other 93%, so comparing runs across them measured the platform mix. And a pile of what looked like malformed answers were rate limits. Retrying post by post fixed all but two.
Parity for under half the cost
On a held-out exam of a few thousand posts, a much cheaper model reached parity with the production model on artists and genres. Contamination was the one significant gap.
| production | cheaper | |
|---|---|---|
| artists | X | ≈ X |
| genres | X | ≈ X |
| tags | X | ≈ 1.06X |
| contamination | X | ≈ 1.8X |
| cost | X | ≈ 0.55X |
Then we looked at how the requests are sent. Batch processing trades latency for price, and tagging doesn't need an answer in real time. The exam table prices the exam's own requests. The next one uses what we were billed at scale, where the production model also came out cheaper per post, so its X is a different number. In the batch test, the cheaper model cost about 70% less per post than the production model. The real runs came in a little above that. The full backfill settled at about 62% less. Staging cost more per post, about 48% less, because its posts are mostly YouTube and each one names more artists and genres, so the model writes more. The whole project, the labelling included, came in under what we had planned to spend on it.
| cost | |
|---|---|
| production model, real time | X |
| cheaper model, real time | ≈ 0.64X |
| cheaper model, batch test | ≈ 0.3X |
| cheaper model, batch, full backfill | ≈ 0.38X |
| cheaper model, batch, staging | ≈ 0.52X |
Two tools, one repo
There was no clever integration between Claude Code and Cursor. Claude built the harness, ran the experiments and audited the gold. Cursor ran the labelling waves. What connected them was the repo: one AGENTS.md (CLAUDE.md is a symlink to it), skills in .agents/skills that both tools read, and plain handoff files each side picked up from the other.