Voice AI Engineering · Episode 03
What Makes AI Agents Agentic?
The concrete difference between a chatbot that answers and an agent that acts—covering the agent loop, tool calling, memory, planning, and safety guardrails through a security engineer's lens.
Listen in my voice · AI narration (ElevenLabs clone)
On this page
Hey guys, I’m Chris Watkins, also known as Bingo Codes. I’m a security engineer transitioning into voice-first AI engineering while building Djembe AI and Big Mama—a culturally grounded voice-first agentic AI platform designed to help Black communities discover businesses, preserve culture, and help SMBs grow through intelligent AI systems.
A chatbot answers. An agent helps you get something done. That difference is where AI starts becoming product infrastructure. Today, I’m breaking down what makes an AI agent agentic, how I think about it as a security engineer, and what this means for Big Mama.
Assistant Versus Agent
Let’s begin with the simplest distinction. An assistant may respond to a prompt. An agent can pursue a goal across multiple steps, use tools, observe results, and adjust its behavior.
If I ask, “What is on my calendar today?” and the system answers, that is assistant behavior. The system retrieves information and presents it to me. It is helpful, but it is static. It does not take initiative or string together complex actions.
If I say, “Help me plan my day around this meeting, find a lunch spot nearby, and remind me to follow up with a client,” now we are moving into agent territory. The system must understand the constraints of my schedule, search for local businesses, evaluate options based on proximity and perhaps my past preferences, and then schedule a reminder. It is not just answering a question; it is executing a workflow.
An agentic system needs more than a model. It needs a loop. It needs the ability to reason about its environment, make decisions, and take actions that change that environment. This is a fundamental shift from the conversational AI we have grown accustomed to over the past few years.
The Agent Loop
In security and operations, we often talk about the observe, orient, decide, act pattern. This is a useful bridge into AI. When we build systems that can act autonomously, we need a structured way to think about how they process information and make choices.
| Loop Step | Meaning | Big Mama Example |
|---|---|---|
| Observe | Take in user input, context, tool results, or environment state. | User says they need to plan a Saturday outing. |
| Orient | Interpret the goal, constraints, preferences, and available data. | Big Mama checks location, time, business categories, and family preferences. |
| Decide | Choose the next action or ask a clarifying question. | Big Mama decides whether to search, recommend, or ask for budget. |
| Act | Use a tool, answer, schedule, retrieve, or update memory. | Big Mama returns options and offers to add the plan to a calendar. |
This loop can happen once or many times. The danger is that every additional step adds complexity, latency, and risk. In a voice-first system like Big Mama, latency is a critical factor. If the loop takes too long to execute, the conversation feels unnatural. If the loop makes a poor decision, the user loses trust. Therefore, optimizing this loop is a core engineering challenge.
Tools and Function Calling
Tools are how an agent interacts with systems outside the model. A tool might search a database, check a calendar, send an email, retrieve business information, create a task, or call an API. Function calling is a structured way for the model to request one of those actions.
“Without tools, the model is mostly talking. With tools, the system can do work.”
For Big Mama, tools might include a business directory search, event database, calendar, planner, CRM, messaging service, or analytics dashboard. The agent should not directly do everything. It should request specific tool actions with clear inputs, and the application should enforce permissions and validation.
When building these tools, we have to think like security engineers. We cannot just give an AI model unfettered access to a database. We must define strict interfaces. If Big Mama needs to look up a local Black-owned restaurant, it calls a specific search function with parameters like location and cuisine type. The application layer handles the actual database query, ensuring that the model cannot accidentally drop a table or access unauthorized data.
Memory Systems
Memory is what allows an agent to be useful across time. But memory is also where trust and privacy become serious. A system that remembers everything can be incredibly helpful, but it can also be deeply invasive if not designed with care.
There are three types of memory to consider:
| Memory Type | Purpose | Risk |
|---|---|---|
| Session memory | Keeps track of what is happening in the current conversation. | Can become confusing if the session gets too long. |
| User memory | Remembers preferences with permission. | Can violate trust if users cannot inspect or delete it. |
| Business memory | Stores business details, customer preferences, or workflows. | Can expose sensitive operational data if poorly secured. |
For Big Mama, memory should be transparent. Users should know what the system remembers and why. SMBs should know what business data is stored, where it comes from, and how it is used. If a user tells Big Mama they are allergic to peanuts, that should be stored in user memory so future restaurant recommendations are safe. But the user must have the ability to view, edit, or delete that memory at any time.
Planning and Decomposition
Planning means breaking a goal into steps. If a user says, “Help me get more customers this weekend,” Big Mama should not jump straight to generic advice. It might need to ask what kind of business they run, check their hours, look at upcoming local events, suggest a promotion, draft a post, and schedule reminders.
“Agentic AI becomes useful when it can turn a messy human goal into a sequence of practical steps.”
Planning does not mean total autonomy. Many systems should remain assisted. The agent can propose a plan and ask for confirmation before taking sensitive actions. This is crucial for maintaining user trust and ensuring that the AI does not take actions that could harm a business’s reputation or finances.
When Big Mama decomposes a task, it is essentially creating a mini-workflow on the fly. It must evaluate the dependencies between steps. For example, it cannot schedule a promotional post until the user has approved the draft. This requires a level of reasoning that goes beyond simple text generation.
Autonomous Versus Assisted Agents
Autonomy exists on a spectrum. A low-risk agent might automatically sort notes. A higher-risk agent should ask before sending messages, spending money, publishing content, changing calendars, or sharing personal data.
“For Big Mama, I want autonomy where it saves time and confirmation where the action affects someone’s money, reputation, privacy, or schedule.”
That line bridges product design and safety. We want the AI to handle the tedious parts of a task, like gathering information or drafting text, but we want the human to remain in control of the final decision. This is especially important when building tools for small businesses, where a single mistake can have significant consequences.
Safety, Alignment, and Reliability
Agentic systems need guardrails because they can act. The risk is not only a bad answer. The risk is a bad action. Safety means defining what the agent can and cannot do. Alignment means keeping the system’s behavior consistent with user intent and product values. Reliability means the system behaves predictably under normal and degraded conditions.
“The moment an AI system gets tools, it gets a blast radius. My job as a builder is to reduce that blast radius.”
This means implementing scopes, permissions, logging, rate limits, approval steps, and fallback behavior. If an API goes down, Big Mama should gracefully inform the user rather than crashing or hallucinating a response. If a user asks Big Mama to perform an action it is not authorized to do, it should politely decline and explain why.
As a security engineer, I look at an AI agent as a complex system with multiple attack vectors. We have to secure the prompts to prevent injection attacks. We have to secure the tools to prevent unauthorized access. We have to secure the memory to protect user privacy. Building an agentic system is not just about making it smart; it is about making it safe.
Big Mama Build Connection
Big Mama should be designed as an assisted agent first. It should help users discover businesses, understand options, plan tasks, and prepare actions. Over time, it can become more autonomous in low-risk areas while keeping user confirmation for sensitive actions.
“The goal is not to make Big Mama do everything without asking. The goal is to make Big Mama helpful, trustworthy, and aware of when it needs permission.”
By focusing on the agent loop, robust tool calling, transparent memory, and strict safety guardrails, we can build a system that truly serves its community. Big Mama is not just a technological experiment; it is a tool designed to empower Black communities and support the growth of small and mid-sized businesses.
Practical Takeaway
An AI agent needs a goal, context, tools, memory, a loop, and boundaries. If any one of those pieces is missing or poorly designed, the experience breaks.
| Component | Question to Ask |
|---|---|
| Goal | What is the user trying to accomplish? |
| Context | What does the system need to know right now? |
| Tools | What outside systems can the agent use? |
| Memory | What should the agent remember, and who controls it? |
| Loop | How does the agent observe results and adapt? |
| Boundaries | What requires confirmation, logging, or denial? |
When you are building or evaluating an AI agent, use this checklist. It will help you identify the weak points in the system and ensure that you are building something that is both useful and safe.
Closing
Next episode, I’m focusing on voice-first AI. We are going to talk about why voice changes the entire user experience, why latency matters, and why Big Mama has to be designed around conversation instead of screens.
If you are building in AI, security, voice infrastructure, or community-centered technology, follow along. This series is my public proof-of-work as I learn, build, and ship Djembe AI and Big Mama in public. Drop a comment with what you want me to build or explain next, and I’ll see you in the next episode.