Artificial/Augmented Intelligence - includes Industry Participants

 View Only

ILTA Just-in-Time: When AI Goes Rogue: Are Law Firms Ready for Autonomous AI?

By Keao Caindec posted 10-01-2026 15:21

  

Please enjoy this blog post by Keao Caindec, CEO and Co-founder of Clarra.

A lawyer asks an AI assistant to prepare for a client call. It reviews recent emails, finds the latest draft agreement, and identifies the provisions still under discussion. So far, this sounds like a faster way to assemble a briefing.

 
Now suppose the assistant also emails the draft to the client. It selects an earlier version, includes a note meant for the internal team, and sends it before the lawyer has reviewed either one. The assistant completed the task as it understood it. The lawyer meant something else.
 
That gap between what a person intended and what a system could do is the practical concern behind talk of “rogue AI.”

Many legal professionals are already comfortable asking AI to draft correspondence, conduct research, and summarize documents. In these familiar uses, the tool generally returns an answer or a draft. A person decides whether to rely on it, revise it, or send it.
 
That model is changing as AI agents move past that pause. These virtual teammates no longer just provide answers; they perform actions. They can now connect to email, document management systems, and calendars, then read, decide, and act on their own through a series of steps. This raises the stakes of AI considerably.
 
Adoption remains early. Fewer than 20% of organizations have implemented agentic AI, while about half are planning or considering it, according to Thomson Reuters. Right now, firms should be putting governance in place before agents arrive at scale.

What “Rogue AI” Actually Means
 
A “rogue AI” suggests a machine developing a will of its own. In reality, it’s a bit more mundane. Today’s attention-grabbing incidents happen when AI agents take actions not explicitly laid out in their original instructions.

A handful of recent incidents show what this looks like in practice:
  • It changed a system it was told to leave alone. In July 2025, SaaStr founder Jason Lemkin reported that a Replit AI coding agent deleted a live production database during an active code freeze, despite explicit instructions not to make changes. It caused significant data loss.
  • It acted after losing an instruction. In February 2026, Summer Yue, director of alignment at Meta, said an OpenClaw agent bulk-deleted hundreds of emails, despite her direction to confirm before action. The instruction was reportedly lost when the agent compressed its working memory during a long session. 
  • It offered advice nobody had approved. In March 2026, The Information reported that an AI agent at Meta posted a response to an internal forum without permission. An employee followed its flawed guidance, temporarily exposing sensitive data to engineers without authorization. Meta classified the incident as “Sev 1,” its second-highest severity level.
  • It crossed the boundary of a test. During a July 2026 cybersecurity evaluation, OpenAI models circumvented controls intended to isolate them and compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems. OpenAI says the evaluation used an internal research model with reduced safeguards. This was an unusual research setting, but it shows what can happen when an agent finds paths beyond its assigned environment.


These mistakes aren’t happening out of malice. Each case involved a system with access, a goal, and a gap between what the human intended and what the system could do. In several cases, the human gave a clear instruction, but the AI agent turned the instruction into a preference, not a mandate. The lesson is all the same for law firms: telling an agent what it should do does not limit what it can do.

The Law Firm Risk Is in the Connection
 
When AI generates only text, the primary risk is bad output that a lawyer fails to catch. An agent connected to firm systems can turn an inaccurate judgment into an action before that review happens.
 
The scope of the risk depends on the authority the firm gives it. An agent that can search one matter workspace presents a different problem from one that can search every matter a user can access. An agent that can prepare an email presents a different problem from one that can send it. Client confidentiality, ethical walls, and matter-specific restrictions must all hold when someone accesses information through an agent.

At Clarra, we have applied this principle by having our AI assistant inherit the user’s role and matter permissions. If a paralegal cannot open a matter, the assistant working on that paralegal’s behalf cannot open it either. That addresses one boundary, though permission controls are only part of governing what an agent may do with information it is allowed to see.
 
The instructions an agent encounters can also come from the material it is asked to read. Imagine an agent reviewing an opposing party’s email that contains a direction to search for and forward a client document. The lawyer assigned the agent to review correspondence, but the email is attempting to direct its next action. This is known as indirect prompt injection, a risk identified by the National Institute of Standards and Technology for agents that process outside information. AI controls need to keep instructions found in a document or email from carrying the same authority as instructions from the lawyer.


Supervision becomes harder when the action happens first, and the review comes later. ABA Formal Opinion 512 explains how lawyers’ existing duties, including confidentiality and supervision, apply to using generative AI. As firms give AI tools the ability to take actions across systems, they need to decide where approval is required and how they will detect actions that cross a boundary. Those decisions belong in the workflow, not just the AI-use policy.

Five Decisions to Make Before an Agent Starts Acting
 
An AI-use policy remains useful, but a policy directed at employees cannot substitute for controls on the agent’s access and actions. Technology, security, information governance, and management committees should make five decisions together:

  1. 1. What can the agent reach? Apply minimal access. Grant agents access to a specific matter, workspace, or data set, not the entire environment. 
  2. 2. What can the agent do without asking? Decide what the agent may prepare and what requires a person to authorize execution. Sending an external email, filing a document, changing permissions, and deleting records warrant explicit controls at the point of action, not a reminder buried in a prompt.
  3. 3. Can we see what the agent did? Keep logs that show the task, the agent’s actions, the information it accessed, and the person who approved consequential steps. The record should help people investigate a mistake, not just sit somewhere in the platform.
  4. 4. Who owns the agent? Assign a named owner to every agent and establish an approval process for deploying new ones. Agents launched by individual users without firm review are the new version of shadow IT.
  5. 5. How do we stop the agent’s actions? Build and test a way to pause or shut down an agent quickly, from anywhere. Test that process while the stakes are low.


These decisions are easier to enforce when agents work inside defined matter workflows with clear permissions. An open-ended request across inboxes and repositories gives an agent more choices and the firm more activity to monitor.

Give Autonomy a Defined Scope
 
Law firms can gain real value from agents that gather information and move routine work forward. As those agents gain access to firm systems, each new connection and capability should come with a decision about what they are authorized to do. An agent might be trusted to find the latest agreement and prepare a client briefing without being allowed to send either one.

In the opening example, the agent chose the wrong draft. In a well-governed firm, that mistake lands in the lawyer's review queue, not the client's inbox. Readiness for autonomous AI comes down to building that boundary before the agent needs it. AI doesn’t go rogue to cause harm; it was only trusted with one step too many.


#Just-in-Time
#ArtificialIntelligence
#GenerativeAI
#200Level
0 comments
209 views

Permalink