Typing into a chat box leaves plenty of room for error. But if you run the same model in a tool-calling loop, whatever we’re calling that this month, it’s a different story. At that point, a handful of model behaviors stop being trivia and become real design constraints. These are the five I keep coming back to.
None of these are hard rules. There’s evidence against each one, and people sometimes state them more strongly than the research actually supports. Still, I pay attention to all five because each one helps me avoid making poor decisions.
A model cannot think without spending tokens
Models generate output one token at a time, and the reasoning you see is part of that process. That’s why saying “let’s think step by step” actually helps. Kojima and colleagues found that adding that single line to a zero-shot prompt boosted InstructGPT’s performance from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K.
Now there’s a theory to explain this. Li, Liu, Zhou, and Ma showed that chain of thought lets a decoder-only transformer do serial computation it otherwise can’t handle at low depth.
Using more tokens doesn’t always mean better results. Gema and colleagues created tasks where extending the reasoning length of Large Reasoning Models (LRMs) deteriorates performance. So, give the model enough space for multi-step tasks, but keep in mind the extra time and cost. Don’t assume longer answers are always better.
The system prompt is not a rule engine
The hierarchy is real, and it’s built into the model, not just a convention. Wallace and colleagues at OpenAI introduced it after noticing that models often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. OpenAI’s current order is system, then developer, then user, then tool, with tool messages treated as untrusted input.
There’s one point I want to make clear, since I often see it done the other way. Repeating an important instruction at the end of a user prompt doesn’t make it more important than the system prompt. The priority comes from the role, not where the instruction appears.
Just because the model is trained this way doesn’t mean it always works. OpenAI’s own IH-Challenge paper shows that under adaptive human red-teaming, GPT-5-Mini follows the hierarchy 63.8% of the time, and 88.2% after reinforcement learning on a special dataset. 88.2% is impressive for research, but it’s not good enough for a permission check.
So, keep validation in your application and permission checks in code that the model can’t bypass. Prompting is just how you ask for something. It’s not how you enforce it.
Formatting changes the input, and sometimes the answer
Tokenization reacts to spelling, capitalization, and whitespace. For example, “hello” and ” hello” can have different token IDs. This is easy to check with any tokenizer.
But don’t assume that different token IDs always mean a different answer. Models usually handle typos and spacing changes well, and a changed token ID doesn’t always lead to a very different output.
Sclar and colleagues changed formatting, like separators, casing, and spacing, and found performance differences of up to 76 accuracy points on LLaMA-2-13B. Their key finding for anyone running evaluations is that sensitivity remains even when increasing model size, the number of few-shot examples, or performing instruction tuning, and format performance only weakly correlates between models.
This is why standardizing formatting matters. It’s not about worrying over a stray space, but about knowing whether a change in results comes from the prompt or the template. It also helps make sure a format you tuned on one model still works on the next. It’s like a test that fails: you only learn something when you know what caused the result.
A large context window is not uniform retrieval
Liu and colleagues tested multi-document question answering and key-value retrieval, and found that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.
The effect depends on the model, training, task, and context length, and the paper doesn’t show that every model ignores the middle. What it does show is important: a large context window doesn’t guarantee reliable retrieval everywhere. Also, putting every instruction at the end isn’t always best. That’s a separate issue, and not what the research says.
This matters even more for agentic work. In a loop, tool outputs keep getting added, so positions shift. An instruction that starts near the top might end up in the middle after many turns. If the agent needs something buried deep in the transcript, that’s a retrieval problem. And retrieval has real techniques. Test where your own model actually keeps the information for your task, instead of relying on someone else’s rule of thumb.
Say what you want, not only what you don’t
This is a lot like how the human brain works. If you tell someone, “try not to remember an elephant”, the first thing they think of is an elephant.
But I want to be careful not to take this too far. This analogy is about how prohibitions draw attention to what they forbid, not about how a transformer handles the word “not”.
The measured version of this idea is more limited, but still important. The Computational Linguistics survey on instruction following says it clearly: “existing works found LLMs often fail to follow the negated instructions; some negations can even drop models’ performance”. Prompting guides say the same thing. Anthropic’s docs recommend telling Claude what to do instead of what not to do, like replacing “Do not use markdown in your response” with “Your response should be composed of smoothly flowing prose paragraphs.”
Describe the behavior you want, and don’t count on a prohibition to make it happen. Save explicit negatives for strict rules, and enforce those with automated checks, not just words.
None of this is a prompting problem
None of these five issues can be solved just by writing a better prompt. They all show why you can’t trust an agentic system to follow prompts alone. Instead, you need to focus on what you validate in code and what evidence you require before releasing.
This is the same point I make about delivery in general. Verification (defining contracts, picking the right evidence, and deciding what’s safe and worth shipping) already limits delivery, even without AI. Passing tests don’t tell you whether to ship, and neither does an agent that followed its system prompt during testing.
As authoring gets cheaper, verification isn’t just about running tests or reviews. The real challenge is defining the contract: what behavior is guaranteed, what evidence is enough, which security and operational rules must hold, and what must pass before release. For an agent, that means specifying what the output should look like, which tool calls are allowed, what happens if a tool returns an instruction, and what causes a release to fail. Coding was never the part that made shipping slow, and that hasn’t changed.
I might be wrong about how long this will last. Automated tests, formal verification, and AI review could keep verification from becoming a long-term bottleneck. And really, the honest summary of all five points isn’t a set of rules. What you can trust is measured performance for your model, your task, and your deployment. Not some universal property of language models.
