For a couple of years, prompt engineering was the whole job. You’d fiddle with wording, add a “you are an expert” preamble, maybe throw in some ALL CAPS, and ship it. That era is mostly over. The term that replaced it is context engineering, and the shift in framing is this: it’s less about finding the right words for your prompt and more about answering a bigger question — what configuration of context is most likely to produce the behavior I want?
Context is just the set of tokens you include when you sample from a model. All of it: the system prompt, the tools, the message history, the MCP servers, whatever you pulled out of Postgres and stuffed into a string. Context engineering is the practice of curating that set, and the constraint that makes it interesting is that the set is finite.
Anthropic wrote the definitive piece on this, and most of the concepts below are theirs. What I want to do here is translate it into the world I actually work in: Rails backends, iOS and Android clients, and the kind of AI features that ship inside products people use every day.
Context rot is real, and your app makes it worse
The core finding is that models, like people, lose focus. As the number of tokens in the context window grows, the model’s ability to accurately recall information from that context degrades. This is called context rot, and it shows up across every model — some degrade more gracefully than others, but nobody is immune.
The reason is architectural. Transformers let every token attend to every other token, which means n tokens produce n² pairwise relationships. As context grows, the model’s attention gets stretched across more and more of those relationships. On top of that, models learn their attention patterns from training data where short sequences dominate, so they simply have less practice with context-wide dependencies. It’s not a cliff you fall off at 100k tokens. It’s a gradient — the model stays capable, but precision on retrieval and long-range reasoning erodes.
So: treat context as a budget. Every token you add spends some of it.
Here’s why this hits app developers specifically. When you build an AI feature into a real product, you’re not sitting at a terminal handing the model a carefully chosen file. You’re assembling context from a database, and the temptation is to just… include everything. I’ve seen (and written) code that looks like this:
# Don't do this
def context_for(user)
{
user: user.as_json,
orders: user.orders.as_json(include: :line_items),
support_tickets: user.support_tickets.as_json,
recent_events: user.events.last(500).as_json
}.to_json
endThat’s a to_json on an ActiveRecord object graph, which means you just handed the model created_at, updated_at, encrypted_password_salt, and every foreign key in your schema. It’ll work in the demo. Then a power user with 900 orders hits it and your agent starts confidently referencing the wrong order, because the one that mattered is buried 40,000 tokens deep.
The fix is boring and it’s the whole discipline: find the smallest set of high-signal tokens that gets you the outcome you want.
def context_for(user)
{
name: user.first_name,
plan: user.plan_name,
member_since: user.created_at.strftime("%B %Y"),
open_tickets: user.support_tickets.open.count,
recent_orders: user.orders.recent.limit(3).map { |o|
{ id: o.public_id, status: o.status, total: o.total_display, placed: o.created_at.to_date.to_s }
}
}.to_json
endSame feature. A fraction of the tokens. And notice the second version is easier for a human to read too, which is usually a sign you’re on the right track.
The right altitude for a system prompt
Anthropic describes system prompts as needing to hit “the right altitude,” which is the Goldilocks zone between two failure modes.
At one extreme, you hardcode brittle if-else logic into the prompt trying to nail down exact behavior. Every edge case a user hits becomes another paragraph. Six months later your system prompt is 4,000 tokens of accumulated scar tissue and nobody on the team will touch it.
At the other extreme, you write something vague — “you are a helpful shopping assistant” — and assume the model shares context it doesn’t have. It doesn’t know your return policy is 30 days, that “fulfilled” means shipped, or that you never discuss competitor pricing.
The target is specific enough to guide behavior, loose enough to let the model use its own judgment. In practice I organize prompts into clear sections and lean on Markdown headers or XML tags to delineate them:
class SupportAgent < BaseAgent
def system_prompt
<<~PROMPT
You are the support assistant for Acme, a subscription coffee service.
## Background
Customers subscribe to recurring shipments. Common issues are
delivery delays, plan changes, and pause requests.
## Instructions
- Use the customer context provided before asking clarifying questions.
- You may pause or resume a subscription with tools. You may not issue
refunds — hand those to a human with `escalate_to_human`.
- Keep responses under 3 sentences unless walking through steps.
## Tone
Warm and direct. No corporate hedging. Never apologize twice.
PROMPT
end
endMinimal doesn’t mean short. You still have to tell the agent what it needs to know. It means no token is in there without a job.
The workflow I’d recommend: start with the most capable model and a minimal prompt, see where it fails, then add instructions targeting those specific failures. Not the reverse. Writing 2,000 tokens of preemptive rules for problems you haven’t observed yet is how you end up with the scar tissue version.
Tools are the contract, so keep the set small
Tools are how your agent touches your app. They define the boundary between the model and your action space, which makes them worth more design attention than they usually get.
The most common failure I see is a bloated tool set — too many tools, with overlapping responsibilities and ambiguous decision points. The heuristic I like: if a human engineer on your team can’t say definitively which tool should be used in a given situation, the model won’t do better.
# Ambiguous. Which one does the agent reach for?
tools = [SearchOrdersTool, FindOrderTool, LookupOrderByEmailTool,
GetOrderStatusTool, OrderHistoryTool]
# One tool, clear parameters
tools = [FindOrdersTool]Tool output matters just as much as tool selection, and this is where app developers leak the most context. A tool that returns a serialized ActiveRecord object is dumping schema noise into the context window on every single call:
class FindOrdersTool < RubyLLM::Tool
description "Finds a customer's orders. Filter by status to narrow results."
param :status, desc: "Optional: pending, shipped, delivered, or cancelled"
param :limit, desc: "Max orders to return. Defaults to 5."
def execute(status: nil, limit: 5)
scope = current_user.orders.order(created_at: :desc)
scope = scope.where(status: status) if status.present?
orders = scope.limit(limit.to_i).map do |order|
{ id: order.public_id, status: order.status,
total: order.total_display, placed: order.created_at.to_date.to_s }
end
{ orders: orders, total_count: scope.count }
rescue => e
{ error: "Couldn't look up orders: #{e.message}" }
end
endFour fields per order instead of thirty. total_count lets the agent know there’s more without paying for it. And errors come back as data the agent can reason about rather than exceptions that blow up the loop.
Just-in-time context beats stuffing the window
Most AI-native apps today do some form of retrieval before inference — embed everything, search at request time, dump the top-k chunks into the prompt. That works, but the field is shifting toward what Anthropic calls just-in-time context.
The idea: instead of pre-loading all the data the agent might need, give it lightweight identifiers — IDs, paths, stored queries — and let it pull what it actually needs at runtime through tools. This is how Claude Code and Codex work. They don’t ingest your repo; they use glob and grep and read files on demand.
It mirrors how people work. You don’t memorize your inbox. You have a search box.
For a Rails app, this maps cleanly onto things you already have. A scope is a stored query. A public_id is a lightweight reference. Your agent doesn’t need the customer’s full order history in context — it needs the ability to ask for it:
class OrderDetailTool < RubyLLM::Tool
description "Fetches full detail for one order, including line items and tracking."
param :order_id, desc: "The order's public ID"
def execute(order_id:)
order = current_user.orders.find_by(public_id: order_id)
return { error: "No order found with ID #{order_id}" } unless order
{
id: order.public_id,
status: order.status,
placed: order.created_at.to_date.to_s,
tracking: order.tracking_number,
items: order.line_items.map { |li| "#{li.quantity}x #{li.product_name}" }
}
end
endThe summary list gives the agent enough to identify the right order. This tool gives it the depth, but only for the one order that matters. That’s progressive disclosure: the agent assembles understanding in layers instead of drowning in everything up front.
There’s a real trade-off, though. Runtime exploration is slower than pre-computed retrieval, and on a phone that latency is visible. Each just-in-time tool call is another round trip, and a user watching a spinner in a SwiftUI or Compose view doesn’t care that your agent is being elegant about context. On the web you can paper over some of this by streaming tokens the moment they exist; on iOS and Android the gap between tapping and seeing anything is the thing people actually judge you on.
Which is why the hybrid approach usually wins in product work: pre-load the small, stable, always-needed stuff, and let the agent fetch the rest on demand. Claude Code does exactly this — CLAUDE.md gets dropped into context naively, and everything else is retrieved just-in-time. For a support agent, the equivalent is loading the customer summary up front (you know you’ll need it, it’s 200 tokens, it saves a round trip) and leaving order details, ticket history, and shipment tracking behind tools.
Long-horizon work: compaction, notes, and sub-agents
Most in-app AI features are short — a few turns and done. But the moment you build something that runs for a while (an onboarding flow, a research task, an agent working through a queue), you hit the context window wall. Three techniques address this.
Compaction is summarizing a conversation that’s approaching the limit and reinitializing with the summary. Claude Code passes the message history back to the model, preserves the architectural decisions and unresolved bugs, discards redundant tool output, and continues. The art is entirely in what you keep versus discard — over-aggressive compaction loses the subtle detail whose importance only becomes obvious later. Tune your compaction prompt against real traces: maximize recall first, then trim for precision.
The lowest-hanging fruit here is clearing old tool results. Once your agent has called FindOrdersTool and acted on the answer, the raw JSON has no reason to stay in context. In a Rails app that’s a scope on your messages:
class Message < ApplicationRecord
scope :for_context, -> {
# Keep the last 20 messages intact; strip stale tool payloads before that
order(created_at: :desc).limit(50).reverse.map.with_index do |msg, i|
msg.tool_result? && i < 30 ? msg.compacted : msg
end
}
endStructured note-taking is the agent writing notes to persistent storage outside the context window and pulling them back later. Claude Code’s to-do list is this. So is a NOTES.md. In a Rails app you don’t need a file — you have a database. An agent_notes table with a tool to write and a tool to read gives you durable memory across context resets, and unlike a file it’s queryable, scopeable to a user, and survives deploys.
Sub-agent architectures are the third lever, and the one I’ve gotten the most mileage out of. Rather than one agent holding state for an entire task, specialized sub-agents handle focused work in clean context windows. The main agent keeps the high-level plan; sub-agents do the deep work and return a condensed summary. A sub-agent might burn 30,000 tokens exploring and return 1,500. All that search context stays isolated where it belongs.
I wrote about building this on Rails — an orchestrator that delegates to specialists via a DelegateToAgentTool, with sub-agents running ephemerally and returning just their results. At the time I framed it as a separation-of-concerns thing. It’s really a context engineering pattern: the orchestrator’s window stays clean because the messy parts happen somewhere else.
Which technique to reach for depends on the shape of the task. Compaction maintains conversational flow for back-and-forth work. Note-taking fits iterative tasks with clear milestones. Sub-agents win when exploration is parallelizable and the details don’t need to survive.
What this means for your app
If you’re shipping AI inside a web or mobile product, the practical version of all this is short:
- Serialize deliberately. Never
to_jsonan ActiveRecord object into a prompt. Hand-pick fields. - Cap everything. Every list your agent can see needs a limit and an honest count. Your prompt has to survive the user with 900 orders.
- Fewer tools, sharper boundaries. If you can’t say which tool fires in a given situation, neither can the model.
- Pre-load the small and certain, fetch the rest just-in-time. Latency is a UX constraint on mobile, not a footnote.
- Push the mess into sub-agents. Isolated context windows are cheap. A polluted main window is expensive.
The guiding principle holds all the way down: find the smallest set of high-signal tokens that maximizes the likelihood of the outcome you want. Every technique above is just a different way of doing that.
Models keep getting better, and the better they get, the less prescriptive engineering they need — that trend is real and it means some of these techniques will look quaint. But context is finite by construction. As long as attention has a budget, deciding how to spend it will be part of the job.
Most of the concepts here — context rot, altitude, just-in-time retrieval, compaction — come from Anthropic’s Effective context engineering for AI agents. The Rails opinions are mine.
