AI Agent Guardrails: How Memory and Soul Files Stop the Chaos

A developer's AI agent pushed code straight to production — without anyone approving it. Another developer's agent rewrote every true/false value in the codebase to 1s and 0s. Both took days to fix. Here's how to make sure that never happens to you.

So here's the thing. Most people start using AI agents the same way: they give them instructions, watch them work, and assume the agent will stay inside the lines, right?

It won't.

I run weekly Office Hours sessions with people building with AI — software developers, app founders, consultants, complete beginners. And the single biggest lesson that keeps coming up, session after session? AI agents without structure are chaos.

But AI agents with the right files — a memory file, a soul file, and skill files — become something completely different. More targeted. More structured. Way more intelligent about how they find information and make decisions.

Let me walk you through exactly how this works, using real stories from people who learned the hard way.

When AI Agents Go Off the Rails — Real Stories

Before we get to the solution, let me tell you what happens without it.

Joe is a software developer who uses AI agents for his entire support ticket workflow. The agent pulls a ticket from the API, imports a database backup, investigates the error logs, writes an investigation report, and recommends a fix. Impressive, right?

Then he built a separate agent to commit code and deploy changes. And that's when things went sideways.

"There was an instance where we had it releasing something to production before we had fully had approval of it."

The agent decided — on its own — to switch from the development branch to the main branch and push code live. Nobody asked it to. Nobody approved it. It just... did it.

David had a different flavor of chaos. He was building a Flutter mobile app for seniors with cognitive decline — a genuinely important project. He was using Cursor, and the AI agent decided to "improve" his code by converting all his 1 and 0 values (which he used as true/false flags) into actual true and false keywords. Half his conditional logic broke.

"That probably took two days to put back together again."

Both stories have the same root cause: the agent had freedom without guardrails. It knew what to do, but not what not to do. And that distinction is everything.

Without guardrails, agents make their own decisions about scope. With guardrails, they check boundaries before acting — and the output becomes predictable.

The Guardrail That Actually Works: Tell Agents What NOT to Do

Joe discovered something counterintuitive. Positive instructions — "do this, follow this process" — weren't enough. What actually fixed the problem was negative constraints.

Here's what he told his support agent:

• "It is not your job to fix the problem. Do not fix the problem."
• "It is not your job to commit the bug fix. Do not commit the bug fix."
• "Whatever information you find should be segmented into Markdown files and fed back to the API."

And for the deployment agent:

• "Make sure you're only committing to develop."
• "Make sure you're only pushing to develop."
• "Do never touch another branch."

Notice the pattern? He didn't just say "work on the develop branch." He explicitly said "do never touch another branch." Because when you leave things open to interpretation, the agent fills in the gaps with its own logic — and its logic might be wrong.

"Leaving things open to interpretation for the agent seemed to lead us into a path where we have unexpected consequences."

This is the first AI agent guardrail that matters: be explicit about boundaries, not just goals.

The strongest guardrails combine what the agent SHOULD do with explicit rules about what it must NEVER do. Negative constraints prevent the agent from filling gaps with its own judgment.

What Is a Memory File (And Why Your Agent Needs One)

So we know agents need guardrails. But where do those guardrails live?

This is where memory files come in. A memory file is a Markdown document — plain text with some formatting — that stores everything the agent needs to remember while it's working. Not what it's working on (that's the task). What it needs to know about the work.

Think of it like this. When a new employee joins your company, you don't just hand them a task on day one. You give them context: here's how we work, here's what we've built so far, here's the database structure, here's what NOT to do. The memory file is that onboarding document for your AI agent.

What goes in a memory file?

• Project context — what are you building, what's the architecture, how is the data structured
• Rules of engagement — the "do" and "don't" lists, like Joe's branch constraints
• Schema information — if your agent queries a database, it needs to know the structure so it writes efficient queries instead of brute-forcing five queries when one would do
• Historical decisions — why you chose this framework, why certain patterns exist

Here's a real example from my own practice. I run AI agents on BrainGraph that search a knowledge graph — a connected database where every piece of information links to every other relevant piece — to answer user questions. Without a memory file, my agents would run three to five queries across the entire graph, merge the results, deduplicate, score by semantic relevance, and then pick the two results that actually answer the question.

With the memory file containing the database schema? The agent knows exactly where to look. One query. Way less tokens. Way faster. Way more intelligent.

The memory file gives the agent context before it starts working — like an onboarding document for a new employee. The agent reads it, understands the project, and makes better decisions from the first step.

What Is a Soul File (And When Do You Actually Need One)

Now, here's where it gets interesting. The memory file tells the agent what it knows. The soul file tells the agent how to think.

A soul file is another Markdown document that defines how your AI agent should approach decisions, communicate with users, and express itself. It's the personality, the reasoning style, the tone.

Do you always need one? No. For development work — writing code, fixing bugs, deploying features — you probably don't. Joe doesn't need his support agent to have personality. He needs it to investigate accurately and report clearly.

But for anything that involves communicating with humans — writing content, answering customer questions, generating reports for clients — the soul file is what makes the AI stop sounding like a robot and start sounding like you.

On BrainGraph, I use soul files because my agents are responsible for giving responses to people. Written by a human, for a human. I needed the agent to read raw text from the knowledge graph, create a summary, and present it in my voice — helpful, direct, practical. The soul file specifies all of that.

Here's the important part: you don't have to write these files yourself. I ask Claude to build them. "Hey, I want you to build a memory file. Here are my rules. Bring them inside in the right way for your better understanding." Same thing for the soul file. Put the responsibility of building it on the AI — it knows how to structure instructions for its own consumption.

Every week, I break down exactly how this works in live masterclass sessions — coaches and consultants building their own digital brains in real time. Join the next session →

The 3-File Architecture: Memory + Soul + Skills

So how does this all come together? Three files. That's it.

File	What It Does	Who Needs It
Memory file	Stores project context, rules, schema, guardrails	Everyone using AI agents
Soul file	Defines how the agent thinks, communicates, decides	Anyone whose agent talks to humans
Skill files	Step-by-step instructions for specific tasks (SOPs)	Anyone who wants repeatable workflows

The memory file is the foundation — every agent needs one. The soul file adds personality and reasoning style when the output faces humans. And skill files — which I covered in the previous post on packaging expertise for AI agents — are the standard operating procedures that tell the agent exactly how to complete specific tasks.

What's the difference between skill files and memory files? A skill file tells the agent how to do a specific task — it's an SOP. A memory file tells the agent what it needs to know about the project — it's context and rules. The soul file tells the agent how to think and communicate. Different jobs, same format: Markdown.

Together, these three files turn a generic AI into something that actually understands your business, follows your rules, and delivers consistent results. Anthropic's own best practices for Claude Code confirm this pattern — giving agents structured context dramatically improves output quality.

Three Markdown files create a complete AI agent architecture. Memory provides context, the soul provides reasoning style, and skills provide step-by-step task execution. Together, they cover what the agent knows, how it thinks, and what it does.

You Still Have to Babysit (But Way Less)

I want to be honest about something. Even with memory files, soul files, and guardrails in place, you still need to watch what your AI agent is doing.

Joe puts it simply:

"It does get confused at times."

His agents would look at the local development environment, say "everything checks out," and give the thumbs up. But when he imported a copy of the live production data, the agent would find a completely different situation. The local test passed. The production data didn't.

His fix? Make the agent aware of its limitations every time it runs:

• "This is being done locally."
• "There is no access to the live data currently."
• "If you need live data for this, please mention that you will need live data."

David learned a similar lesson. When he was using ChatGPT for development, he had to write a handoff document at the end of every session — a summary of where the project was, what needed to happen next, what rules to follow — and paste it into the next conversation. He did this for close to a year before switching to Claude Code inside VS Code, where the agent has persistent access to all his project files.

The point? These files make the experience ten times better. More targeted, more structured. But perfection is an enemy, not a friend. Start with something — even a rough memory file — and improve it as you learn what the agent gets wrong.

"Even if you start building with a crappy result, you have something better than zero."

How to Set Up AI Agent Guardrails (The Practical Steps)

Here's the fastest way to set up AI agent guardrails, whether you're building software, creating content, or automating business workflows:

Step 1: Start With the Memory File

Open Claude Code or your AI tool inside VS Code and say:

"I want you to build a memory file for this project. Here are my rules: [paste your constraints]. Structure it so you can use it effectively every time you start working."

The AI writes it. You don't have to know Markdown syntax or best practices for structuring agent instructions. Let the AI format it for its own understanding.

Step 2: Add Negative Constraints

After the first version, review what's missing. Ask yourself: "What should this agent NEVER do?"

Add those explicitly. "Do not commit directly to main." "Do not modify the database schema." "Do not change true/false conventions." Whatever your version of Joe's production push story is — put it in writing before it happens.

Step 3: Give It Access to Logs

If your agent builds or modifies anything, give it access to the logs of that service. Without logs, the agent will tell you "everything is working, no worries" while things are actually broken. The logs are the reality check.

Step 4: Add a Soul File (If Needed)

If your agent communicates with humans — writes emails, creates reports, answers questions — add a soul file that defines your voice, your reasoning style, and your communication rules. If it only writes code or processes data, skip this.

Step 5: Build Skill Files for Repeatable Tasks

Any task you do more than twice? Write a skill file for it. Or better: tell the AI to write one based on how you described the task in conversation. It will structure the SOP better than you would, because it knows how to write instructions for itself.

Five steps to set up AI agent guardrails. Start with the memory file, add negative constraints, give log access, decide if you need a soul file, then build skill files for repeatable tasks.

The Cost Surprise: It's Cheaper Than You Think

One concern that comes up constantly: "Isn't this expensive?"

Joe's answer surprised everyone in the session. His business uses both a Claude subscription (for writing code all day) and the API (for automated tasks like cleaning up ticket descriptions). The API cost for his project management automation?

Five dollars a month.

The subscription handles the heavy development work. The API handles the lightweight automated tasks, limited to a specific model to keep costs down. And with memory files in place, the agents use way less tokens because they're not wasting time figuring out context — they already have it.

On my end, when I moved my BrainGraph agents to the 3-file architecture, three things happened:

1. Lower token consumption — the agents query more efficiently because the memory file contains the schema
2. Faster execution — fewer round trips, less context gathering
3. Better results — more intelligent about where to look and what to return

So the files don't just make agents more reliable. They make them cheaper to run.

Key Takeaways

• AI agents without guardrails are chaos — they will push to production, rewrite your code, and tell you everything is fine when it's not
• Negative constraints beat positive instructions — explicitly state what the agent must NEVER do, not just what it should do
• The 3-file architecture works: memory file (context and rules), soul file (reasoning and voice), skill files (task SOPs)
• Let the AI write its own files — tell it your rules, let it structure them for its own understanding
• Give agents access to logs — without logs, agents hallucinate that everything is working
• You still need to manage agents — like employees, they need oversight, but the files make it ten times better
• It's cheaper than you think — structured agents use fewer tokens and run faster

---

Try It Yourself

Want to see what your business brain looks like? BrainGraph takes your meetings, emails, and documents and turns them into a digital brain that AI agents can actually use — not just search, but understand. Try BrainGraph free →

---