For the past two years, the conversation about Artificial Intelligence has revolved around language models: which one reasons better, which one is faster, or which one performs better in the benchmarks.
However, when an organization incorporates AI agents into its business processes, it discovers a different reality: success depends much less on the model than we usually think and much more on classic aspects of enterprise architecture: data, security, governance, oversight, and process design.
An important clarification before we begin: some of the real-life cases that appear below correspond to conversational chatbots and not to agents in the current sense of the term. The difference is relevant (a chatbot mainly responds, an agent also consults data, makes decisions and executes actions on other systems), but these precedents illustrate problems that take on a much greater dimension when AI stops being limited to conversing and starts acting.
1. Thinking that everything can be solved with a good prompt
One of the most common mistakes is assuming that an agent is simply a language model that only needs to be provided with a good prompt. A prompt It can guide the model, but It cannot, on its own, guarantee that an action will always be carried out in the intended way..
The case of the chatbot of Chevrolet Watsonville, The system that accepted the sale of a vehicle for one dollar after following specific instructions from a user is a good example of this. The problem wasn't just the response it generated, but the lack of clear limits on what the system could compromise.
When an agent participates in discounts, approvals, or transactions, the response may be probabilistic, but the execution should not be. That's why architectures of hybrid reasoning, which combine the LLM's ability to interpret the context with deterministic logic to ensure that certain actions always respect the business rules.
2. Not preparing the data
The quality of an agent will never be higher than the quality of the information they use.
Many organizations spend weeks evaluating models and very little time reviewing the quality, consistency, and currency of their data. Feeding an agent with contradictory or outdated information increases the likelihood of incorrect responses and flawed decisions.
The case of Air Canada This is illustrative: their chatbot informed a customer of a refund policy that could not actually be applied retroactively. The company ended up being held liable for the information provided by the system.
The lesson is clear: an agent should not invent corporate knowledge. They should rely on it. governed, traceable and verifiable sources, so that we can always know the origin of the information it uses.
3. Forgetting that permissions also apply to AI
An agent should not access information that an employee cannot view or perform actions for which they are not authorized. It must operate within the existing identity, permissions, and governance model, not outside of it.
Many prototypes grant the agent unrestricted access to documents, databases, or applications, allowing it to decide what information it needs. This introduces two distinct risks.
The first one is the oversharingInformation that was already improperly accessible, but remained practically hidden among thousands of documents, becomes locatable and summarizeable in seconds using natural language.
The second one is even more important: Permissions should control not only what data an agent can access, but also what they can do with it.. An agent may be authorized to read a document, but should not be able to share it, modify it, or trigger an action on another system.
A recent case at Meta illustrated another dimension of this problem: an internal agent provided incorrect technical advice and posted it on a forum without the required human approval. An employee subsequently followed that advice and implemented a change that temporarily allowed unauthorized individuals access to sensitive information. The agent did not directly make the change, but the incident highlighted the importance of establishing controls not only over the data an agent can access, but also over how their advice is posted, validated, and used before it triggers actions that impact systems.
Security testing no longer consists solely of verifying that the agent cannot read restricted information. We must also verify What can they do with the information they have access to, what can they communicate without supervision, and what controls are in place before their recommendations become actions?.
4. Not measuring the results that really matter
A good demonstration doesn't mean an AI project has succeeded. An agent only generates value when it improves business indicators: reducing resolution times, decreasing errors, increasing customer satisfaction, or strengthening regulatory compliance.
The risk lies in measuring a single metric and mistaking it for the overall result. Automating more conversations, for example, can reduce costs but, at the same time, degrade the customer experience if it increases the need for follow-up or escalation to a human.
Control consists of defining from the beginning a balanced dashboard that combines productivity, quality, satisfaction, errors and business results. If we cannot quantify what has improved since the agent's implementation, we probably still don't know if the project is working.
5. Automate processes that were already inefficient
AI doesn't correct poorly designed processes; it simply allows them to be executed much faster.
If a procedure contains unnecessary steps, redundant approvals, or inconsistent rules, adding an agent can multiply the speed at which those problems occur.
The case of Zillow Offers, Although predating generative AI, it serves as an excellent precedent. The company automated home purchases on a large scale using valuation algorithms. When market conditions changed, the automation amplified the impact of decisions whose underlying assumptions were no longer valid.
Before automating a process, it's worth asking yourself a very simple question: If we designed it from scratch today, would we do it exactly the same way? If the answer is no, the first step is probably not to incorporate an agent, but to redesign the process.
Technology matters, but architecture matters more
Models will continue to evolve and become increasingly capable. But, as agents move from answering questions to executing complete processes, the challenge will be less about making the model "know more" and more about making it... the system behaves reliably, safely, and predictably.
This requires combining governed data, identity and permissions, deterministic business rules, validations, observability, and, when necessary, human oversight. And here lies one of the major challenges of this approach. do-it-yourselfBuilding a prototype connected to APIs is relatively simple; turning it into a robust, auditable, and maintainable enterprise system is much more complex.
The five errors above all point to the same issue: responsibility and control over the agents. The owner of an enterprise system must be able to define its limits of action, have complete observability of its behavior, and have mechanisms to escalate or interrupt its execution when it deviates from what is expected.
Because the real challenge of agentic AI is not just about building agents capable of doing things, but about building systems that an organization can rely on.
Perhaps the most important question is not «"Which model should we use?"», but another, much more difficult one:
Do we have the necessary architecture to trust what our agents will do?
Author
Manuel Vigil
Distinguished Solutions Engineer, Salesforce – LinkedIn
References
AIAAIC — Driver persuades chatbot to sell car for USD 1
Chicago Auto Trade Association — Chevrolet Dealer's Chatbot Agrees to Sell Tahoe for $1
Moffatt v. Air Canada — Civil Resolution Tribunal decision
Stanford GSB — Why Zillow's Algorithmic Home Buying Venture Imploded