The Art & Science of Prompting AI Agents

Most people write AI prompts like they're sending a text message. Casual, unstructured, and they just hope the model figures out what they mean. Then they complain the output is generic. I've spent months building production AI agents that automate entire departments for enterprise clients, and I can tell you the single biggest factor in whether an AI agent works or doesn't is the prompt. Not the model. Not the temperature setting. Not how much you're paying for the API. The prompt.

And this isn't just my experience. There's hard research behind it. A 2024 study from Microsoft and MIT found that prompt formatting alone can change model accuracy by over 40 percentage points. Another study from the University of Washington found accuracy swings of up to 76 percentage points from formatting changes that didn't even change the meaning. They just adjusted separators, capitalization, and whitespace. Same content. Different structure. Wildly different results.

So how you structure a prompt matters just as much as what you actually say in it. Maybe more.

In this post I'm going to walk you through everything. Every major prompting method, why each one exists, the research behind it, and then the exact XML template I use for every AI agent I build. If you're deploying AI agents for businesses, building automation workflows, or even just trying to get better answers from ChatGPT, this will change how you think about prompting.

Why prompt format matters more than prompt content

Let me start with the research that completely changed how I think about this.

He et al. (2024) from Microsoft and MIT ran a proper experiment. They took the exact same prompt content and formatted it four ways: plain text, Markdown, JSON, and YAML. Then they ran the same tasks across GPT-3.5 and GPT-4.

The results were wild.

On international law questions, just switching from Markdown to JSON improved accuracy by 42%. On code translation tasks, format choice created a 40% swing. And here's the kicker: only 16% of responses were identical between Markdown and JSON for the same content on GPT-3.5. So 84% of the time, changing the format changed the answer. Same words, different wrapper, different output.

That wasn't an isolated finding either. Sclar et al. (2023, presented at ICLR 2024) went even further. They tested changes as small as swapping a separator character, adjusting capitalization, or adding whitespace. On LLaMA-2-13B across 50+ tasks, these tiny cosmetic changes produced accuracy swings of up to 76 percentage points. Seventy-six.

A 2025 WSEAS study looked at 400 programming challenges and found structured prompting gave 27.1% better solution accuracy and 21% higher code quality compared to just typing out instructions directly.

And it's not just about getting the right answer. A study in npj Digital Medicine found that structured prompts cut GPT-4o's hallucination rate from 53% to 23% on medical tasks. No fine-tuning. No retrieval augmentation. Just better structure in the prompt.

One important thing though. Structured input formatting consistently helps. But forcing strict output formatting can actually hurt reasoning. Tam et al. (2024, EMNLP Industry) showed that making LLMs generate in strict JSON mode hurts reasoning performance, especially on math tasks. The solution is simple: let the model think freely, then convert its output to structured format in a post-processing step.

The four levels of prompt engineering

Building AI agents for real businesses taught me that there are distinct levels to prompting. Each level exists for a reason, and each one works best for different types of tasks. Knowing when to use which one (and when to combine them) is what separates someone who plays with AI from someone who deploys it in production.

At its simplest, it looks like this:

You are an elite business consultant with 20 years of experience
in strategic advisory. Always respond concisely and back every
recommendation with reasoning.

Don't sleep on this. Even one sentence about who the model is changes the output significantly. When you tell the model it's a “senior Python developer specializing in async patterns,” it doesn't just change the tone. It changes which parts of its training data it pulls from most heavily. It activates different knowledge.

Role prompting is great for broad conversations where you want a certain level of expertise or communication style. And this is where 90% of people stop. It works fine for general use, but the moment you need precision, consistency, or multi-step task execution, it falls apart.

Chain-of-Thought was formalized by Wei et al. (2022) at Google. It was a real breakthrough. The original paper showed up to 18% improvement on arithmetic tasks. A follow-up technique called self-consistency, where you sample multiple reasoning paths and pick the best one by majority vote, added another 17.9% on math benchmarks.

A typical CoT prompt looks like:

Think step by step:
1. Identify the core problem
2. Break it down into sub-components
3. Analyze each component
4. Synthesize into a solution
5. Verify against the original problem

Now here's what most people miss. CoT's value is shrinking for newer reasoning models. Research from the Wharton Prompting Science series (Meincke & Mollick, 2025) found that CoT only gives +2.9% to +3.1% improvement for models like o3-mini and o4-mini, while adding 20-80% more latency. These models already think step by step internally. Telling them to do it explicitly is redundant and just makes them slower.

For standard models like Gemini Flash and Claude Sonnet though, CoT still gives you a solid +11-14% improvement. So the rule is straightforward: use explicit CoT for standard models, skip it for reasoning models.

{
  "role": "marketing expert",
  "task": "write ad copy",
  "audience": "startup founders",
  "tone": "bold and direct",
  "format": "3 variations, each under 50 words"
}

JSON is great when your AI agent needs to talk to tools and APIs. It's machine-readable, precise, and maps naturally to function calling. OpenAI's Structured Outputs feature (launched August 2024) guarantees model outputs match a JSON Schema exactly, which is critical for production systems where outputs feed directly into databases.

But JSON has real limitations for complex prompts. It's harder to read when you have long instructions, it's awkward to nest natural language inside, and the research actually shows forcing strict JSON output can hurt reasoning. Tam et al. (2024) found measurable performance drops on math and reasoning tasks when models were locked into JSON output mode.

So JSON is the right choice for output specification and tool integration. It's the wrong choice for the rich, multi-section instruction sets that production AI agents need.

<role>You are a world-class business strategist</role>
<task>Create a go-to-market plan</task>
<context>Startup in AI automation space</context>
<constraints>Low budget, fast execution</constraints>
<output_format>Action items with timelines</output_format>

Looks simple right? But the implications run deep. Let me explain why XML won.

Why XML won the format war

Here's something that caught my attention. The three biggest AI companies, Anthropic, OpenAI, and Google, have all independently converged on recommending XML as the go-to format for complex prompts. They came at it from different angles but ended up in the same place. When three competitors all arrive at the same conclusion independently, pay attention.

Anthropic (Claude)

Anthropic has been pushing XML from the start. Their docs call XML tags “a game-changer” for prompts with multiple components. They list four specific benefits: clarity, accuracy, flexibility, and parseability.

But the real reason goes deeper. Claude was trained on massive amounts of XML data. So XML tags aren't just another format for Claude. They're a native organizational system that the model recognizes structurally, not just as text. Anthropic's senior prompt engineer Zack Witten said it publicly: the model has seen more XML than other formats during training, so it processes XML more reliably.

And Anthropic practices what they preach. Their own internal system prompts for Claude are full of XML tags like <default_to_action> and <avoid_excessive_markdown>.

OpenAI (GPT-4, GPT-5)

OpenAI historically preferred Markdown. Their GPT-4.1 guide (April 2025) still says to start with Markdown titles for major sections. But in the same guide they admit XML “also performs well” and that they've “improved adherence to information in XML.”

Then look at their GPT-5 prompting guide. It uses XML-style tags like <code_editing_rules> and <guiding_principles>in its own examples. That's a big shift from pure Markdown. For their reasoning models (o1, o3, o4-mini), they explicitly list “markdown, XML tags, and section titles” as equal options.

Google (Gemini)

Google's Gemini guide says “XML-style tags (e.g., <context>, <task>) or Markdown headings are effective.” They treat both as first-class options but stress the importance of being consistent within a single prompt.

What makes XML actually better for complex prompts

Clear boundaries.Opening and closing tags leave zero ambiguity. The model never has to guess where your instructions end and your context begins. Markdown headers don't have closing tags, so the model has to infer section boundaries, which introduces errors.

Natural nesting. You can put <example> inside <examples>, or <step> inside <instructions>. Markdown's flat heading structure can't express hierarchy cleanly without conventions the model may or may not follow.

Metadata support. Tags can carry attributes like <example type="positive"> that add information without cluttering the text.

Easy to parse programmatically. When your agent produces output wrapped in XML tags, pulling out specific sections downstream is trivial with standard parsers. In production this is huge.

The trade-off is tokens. XML uses about 15% more tokens than equivalent Markdown. At massive scale that adds up. But for complex agent prompts where accuracy matters, it's almost always worth the cost.

Anatomy of a production-grade AI agent prompt

Knowing XML is the right format is step one. Step two is knowing what goes inside it.

I've studied how the best production agent systems are built. Anthropic's context engineering guide, OpenAI's prompting docs, real-world systems like Cline and Bolt.new, and Zapier's fleet of 800+ agents. They all share a consistent architecture. Here's what each component does and why it matters.

Identity and Role

A short statement of who the agent is, what it does, and how it communicates. Keep it to 1-3 sentences. This isn't ceremonial. It activates domain-specific knowledge in the model and sets the behavioral baseline for everything that follows. More than 3 sentences and you're diluting focus.

Context

This is the section most people skip. It's also the most important one. Context means: what triggered this agent, what's the current state of things, and what should the world look like when the agent is done.

Think about briefing a new employee. You wouldn't just say “write marketing copy.” You'd say “we're launching next month, our current messaging is too technical, and we need copy that speaks to non-technical decision-makers.” That's context. The more specific you are here, the less the model has to make up.

Tools

Every tool the agent can use, what it does, and what happens when it fails. This is critical for agentic systems. Anthropic's advice here is solid: use minimal viable toolsets. If a human can't clearly tell which tool to use for a task, the AI won't figure it out either. Fewer well-described tools beat a large confusing toolbox every time.

Instructions

Numbered steps. Not paragraphs. Not vague suggestions. Clear steps. This is the core of the prompt. Anthropic recommends finding the “right altitude” for instructions. Too specific (if-else logic for every possible scenario) and the agent breaks on edge cases. Too vague (“be helpful”) and it wanders. You want specific enough to guide behavior, flexible enough to handle surprises.

Output Format

The exact structure of what the agent should produce. Most people describe the task but forget to describe the deliverable. Here's a principle that changed everything for me: design the output first, then build the prompt that produces it. If you don't know what the output should look like, the model definitely doesn't.

Constraints and Rules

Hard lines. What the agent must never do. Data limits. Disclosure rules. When to escalate instead of answering. Important placement note: research shows models weight recent instructions more heavily. So your “never do this” rules should live near the bottom of your prompt, not the top. That's where they carry the most weight.

Chain of Thought

For standard models, adding explicit reasoning steps improves output quality a lot. For reasoning models like o3 or o4-mini, skip it. They do it internally. Include CoT when your agent needs to make decisions that benefit from step-by-step thinking.

Examples

Two to five input-output pairs showing what good looks like. Anthropic calls these “pictures worth a thousand words” and they're not wrong. One concrete example communicates more than ten abstract rules. Include edge cases and refusal cases, not just the easy happy path scenarios.

Placement matters

Anthropic's testing found that putting long documents and context at the top, with instructions and queries at the bottom, improved response quality by up to 30%. The model needs to absorb the situation before it hits the directives that reference it. Order isn't random. It's engineering.

The exact template I use for every agent

After testing everything (role prompting, chain-of-thought, JSON, basic XML, and dozens of hybrid approaches), I landed on a specific XML template that I now use for every production AI agent I build. Every section is there because the research supports it.

<role_identity>
You are a [Identity]. Your role is to [Role].
Tone: [Tone]
Style: [Style]
</role_identity>

<primary_objective>
[One clear mission. Not three. Not five. One.]
</primary_objective>

<context>
Trigger: [What starts this agent]
Current State: [What exists right now]
After State: [What should exist when it's done]
</context>

<tools>
- [Tool1_name]: [What it does]
  Fail switch: [What happens if this tool fails]
</tools>

<instructions>
1. [Step 1]
2. [Step 2]
3. [Step 3]
</instructions>

<output>
Type: [Format]
<structure>
[Exact output structure]
</structure>
</output>

<rules>
- NEVER: [Hard constraint 1]
- NEVER: [Hard constraint 2]
- ALWAYS: [Required behavior]
</rules>

<chain_of_thought>
1. [Reasoning step 1]
2. [Reasoning step 2]
3. [Reasoning step 3]
</chain_of_thought>

<examples>
<example_1>
Input: [Example input]
Output: [Example output]
</example_1>
<example_2>
Input: [Edge case input]
Output: [Edge case output]
</example_2>
</examples>

<extra>
[Anything unique to this use case]
</extra>

<guidelines>
- Use natural conversation flow within sections
- Ask about every variable before building
- Create examples if users don't have them
- Track completion of each section systematically
</guidelines>

Why this specific structure works

Role Identity goes first because the model needs to know who it is before it processes anything else. This sets the behavioral baseline for interpreting everything after it.

Primary Objective comes right after because ambiguity here cascades into every single output. One clear mission keeps the model from trying to optimize for multiple conflicting goals at the same time.

Context comes before Instructionsbecause of Anthropic's finding that context-first ordering improves quality by up to 30%. The model absorbs the situation before it encounters the directives that reference it.

Tools include fail switchesbecause in production, tools break. APIs time out. Databases go down. If your agent doesn't know what to do when a tool fails, it either makes something up or freezes. Defining what happens on failure for every tool is what separates a demo from a real system.

Rules sit near the end because models weight recent instructions more heavily. Your hardest constraints belong in the bottom third of the prompt where they have maximum impact.

Examples come after Rulesbecause they're the final calibration. They're the last thing the model processes before it starts generating, and they have enormous influence on output quality and format.

Every section is modular.Need to change the tools? Edit one section. Need to adjust the tone? Update Role Identity. Nothing else breaks. This modularity is essential when you're iterating on agent behavior in production. And trust me, you will be iterating constantly.

The shift from prompt engineering to context engineering

There was a big conceptual shift in this field in mid-2025 and most people missed it.

“I really like the term ‘context engineering’ over prompt engineering. It describes the core skill better: the art of providing all the context for the task to be plausibly solvable by the LLM.”
— Tobi Lutke, CEO of Shopify

Andrej Karpathy (former OpenAI, former Tesla AI lead) backed this up. He pointed out that people think of “prompts” as short task descriptions. But in any real production AI application, what actually matters is the complete information environment around each model call. He called it “the delicate art and science of filling the context window.”

This isn't just a rebrand. The distinction is real.

Prompt engineering is about what you say to the model. Phrasing, structure, examples.

Context engineering is about everything the model knows when you say it. That includes the system prompt, the conversation history, retrieved documents, user preferences, available tools, and output definitions. The full information environment.

Anthropic defines context as “the full set of tokens supplied to the model” and they warn about “context rot,” where accuracy drops as token count increases. The goal isn't to give the model maximum information. It's to give it the smallest possible set of high-quality, relevant tokens that produce the best output.

This reframing changes how you think about building AI agents. You're not writing a prompt. You're engineering an information environment. Every token matters. Every section either helps or hurts. And how you architect that environment (what goes where, in what order, with what priority) is the actual skill.

The template I shared above is really a context engineering artifact. It's not just a prompt. It's a complete information architecture designed to give the model everything it needs and nothing it doesn't.

How enterprises are doing this at scale

This isn't theory. The companies getting the most out of AI right now are the ones treating prompts like engineered artifacts. Versioned, tested, structured, shared across teams.

Zapierhas more than 800 AI agents deployed. That's more agents than employees. They hit 89% company-wide AI adoption across their 360-person team using structured “Agent Skills,” which are basically instruction folders that turn Claude into domain specialists for things like code review, git management, and ticket handling.

Uberbuilt a full prompt engineering toolkit with a model catalog, prompt builder, version control, and evaluation framework. One of their engineers described it as bringing “order to chaos” in a world where prompts were scattered across code, Google Docs, and random notebooks.

JPMorgan Chaseuses their COiN platform to process 12,000 commercial credit agreements in seconds. That work used to take 360,000 hours of manual legal review every year. They've cut compliance errors by an estimated 80%. Over 200,000 employees now use their AI tools.

The pattern is always the same. Organizations that treat prompts as software artifacts, with real engineering rigor, get dramatically better results than organizations that treat prompts as throwaway text. The gap isn't about clever tricks. It's about taking the medium seriously.

What this means for you

Three things.

Stop writing unstructured prompts.If you're building anything more complex than a simple chat response, use XML structure. The research is clear: format is a performance variable as significant as model selection. You are leaving 40%+ accuracy on the table with unstructured prompts.

Think in systems, not sentences. Your prompt is an information architecture. Every section should have a purpose. Every token should earn its place. Design the output first, then build the context that produces it.

Treat prompts as code.Version them. Test them. Iterate on them. Share them across your team. The companies winning with AI aren't the ones with the best models. They're the ones with the best prompt architecture.

I'm building an open-source tool called Prompt Craft that automates this whole process. It walks you through each section of the template, asks targeted questions about your use case, and builds production-ready prompts for you. More on that soon.

If you're building with AI and want to talk prompt architecture, find me on LinkedIn. My DMs are open.