Graham McNicoll

Token costs are exploding across our industry. Here's how we approach it. I definitely believe you should give your teams almost infinite token budgets, because token usage correlates very highly to productivity. Or at least it should. You'd never ask your team to work less hard on something, that just wouldn't come out of a manager's mouth. And yet here we are, constraining them on token budgets. There are times when tokens get spent inefficiently, and there are a few things we've done as a team to help with this. One is that every week we sit down as engineers and re-examine which models are best, and which model is best for which purpose. We share with each other what we're noticing, what tools we are trying out, etc. That helps us re-triangulate: oh, this new model 5 isn't actually that great at this, you should use 4.8, and you don't need the context window to be that large for these kinds of tasks. So you can spend your tokens a lot more efficiently if you get people to share what they've learned. There are some things AI still isn't great at, and some things it's absolutely amazing at, and knowing the difference helps.