Advanced Techniques

LLMs internally cache prompt tokens in key–value pairs. Optimizing this cache reduces recomputation, making inference faster and cheaper.

• Keep prompt prefixes stable so cached tokens remain reusable across requests
• Avoid volatile tokens early (timestamps, session IDs) that invalidate the cache
• Use append-only updates to maximize cache hit rates

Large tool lists waste tokens and confuse models. Dynamic management controls which tools are visible without breaking consistency.

• Keep the full tool list but mask irrelevant ones during decoding
• Apply retrieval-based selection to expose only relevant tools per query
• Reuse tool schemas across workflows to avoid duplication

Agents offload large or persistent data to external files instead of overloading the context window. The prompt carries only references.

• Store documents, results, or web content in files on disk
• Keep only summaries or IDs in the active context window
• Retrieve full content on-demand when the task requires detail

Agents often lose track of goals in long tasks ("lost in the middle" problem). Recitation re-injects key objectives at each step.

• Maintain a persistent todo.md or goal file updated after every step
• Append current goals and plans to the end of the context window
• Use structured reminders to prevent drift or forgotten objectives

Repeating identical context structures can cause models to overfit and degrade. Small variations prevent stagnation.

• Vary phrasing or formatting of repeated context elements
• Rotate templates or synonyms for recurring instructions
• Randomize low-impact details to break monotony without changing meaning