The Question Every Founder Is Now Asking

Someone in a venture group I belong to asked us a question last week, and I have not stopped thinking about it. What happens if your account is hacked, a wrong message goes out, or a stock-market trade is cleared without your authorisation, at a time when every company is rolling out AI and nobody seems to be accountable for it?

It is the right question, and it is being asked everywhere. Agents no longer only answer questions. They send emails, approve refunds, change infrastructure and place orders. McKinsey's State of AI 2026 found the share of large organisations scaling AI agents rose from 27% to 40% in a year. When one of those agents does something nobody approved, someone has to answer for it.

I spent 25 years building systems in banking technology, where "who authorised this?" is the first question after every incident. So I went through the court rulings, the regulators' own circulars and the major 2025–2026 surveys to answer it properly. This post is that answer. It is written for founders and CXOs first, and for the engineers who have to build the controls.

A note before we start: this is an engineering and governance analysis, not legal, financial or investment advice. I am not a lawyer or a SEBI-registered adviser. If you have a live dispute, speak to a lawyer.

The Short Answer: An AI Is Not a Legal Person

No major jurisdiction today treats an AI system as a legal person that can be sued, fined or held liable. So the question is never "is the AI responsible?" It is "which human or company stands behind it?" Three rulings show how courts answer that.

1. Air Canada tried to blame its chatbot, and lost

In Moffatt v. Air Canada (2024 BCCRT 149), the airline's website chatbot told a grieving customer, Jake Moffatt, that a bereavement discount could be claimed after travelling. That was wrong. When Moffatt claimed, Air Canada argued it could not be held liable for what its chatbot said. The tribunal was blunt. It described the airline as suggesting "the chatbot is a separate legal entity that is responsible for its own actions", called this "a remarkable submission", and held that Air Canada is responsible for all the information on its website. The amount was small, about C$650. The principle is large: the deployer owns what its agent says.

2. Delegating to software does not delegate liability

In Mobley v. Workday, a US federal court allowed discrimination claims over AI-driven applicant screening to proceed. Judge Rita Lin wrote that employers "cannot escape liability for discrimination by delegating their traditional functions, like hiring, to a third party." The ruling was at an early stage and decided no final liability (Workday denies the claims, and the case continues), but its logic applies directly to agents: handing a task to software does not hand over responsibility for the outcome.

3. The newest signal: under US anti-hacking law, the user is the one acting

In August 2026 the US Court of Appeals for the Ninth Circuit ruled on Amazon's attempt to block Perplexity's shopping agent. Lifting a preliminary injunction, the court held that under the US Computer Fraud and Abuse Act "it is the user who 'accesses' Amazon's computers, with the help of the Assistant." The court was careful to say it was not establishing a new legal regime for agentic AI, admitted there is "little to no existing caselaw" on how to assign responsibility for AI agents, and left other questions, including tort liability, open.

That last ruling cuts both ways. If courts carry the same reasoning further, "my agent did it" may not be a defence for you either.

Three Scenarios, Three Answers

Here is how the three scenarios map to rules that already exist.

ScenarioWho is accountableWhere it comes from
My bank account is hacked and money movesYour bank: zero liability if the bank was at fault, or if you report a third-party breach within three working days; the bank must prove any negligence on your part (this covers bank transactions, not broker or email accounts)RBI customer-liability rules, first issued 6 July 2017
A bot sends a wrong message to a customerThe company that deployed the botMoffatt v. Air Canada; RBI FREE-AI, Sutra 5
An algo or AI places a trade I never approvedIf the order came through an algo provider your broker has empanelled, the broker is the principal, the provider is its agent, and the broker must handle your grievance. If it came from your own agent using your own API key, expect it to be treated as your orderSEBI retail algo-trading circular, 4 February 2025

The corollary nobody mentions

RBI's zero-liability protection has a condition. Where the loss comes from the customer's own negligence, "such as where he has shared the payment credentials", the customer bears the loss until the transaction is reported. Now put that next to a US appeals court's reasoning, in an anti-hacking case, that when an agent logs in it is the user who is acting.

Many people now hand an AI assistant their broker login, their email password or their banking OTPs so it can "just handle it". Whether that counts as sharing credentials has not been tested in an Indian court, and I am not going to pretend to know how a court would rule. But it is exactly the question a bank or broker would raise. The protection you have as a victim of hacking could be weaker once you have voluntarily given your keys to an agent. That alone is reason to use scoped, revocable API access instead of your own credentials, which is how SEBI's framework is designed.

Before AI, there was Knight Capital

None of this is new. On 1 August 2012, a faulty trading system at Knight Capital sent millions of erroneous orders and lost more than $460 million in about 45 minutes. The US SEC did not pursue the code. It charged the firm, which settled without admitting or denying the findings and paid a $12 million penalty; the SEC's order found that Knight "did not have adequate safeguards". Automated systems that act without authorisation have always been the operator's problem. AI agents simply make it more common.

India Has Started Answering the Question

The worry was that nobody can be held responsible. In Indian financial services, that is no longer true. Regulators have started writing the answer down: SEBI in binding regulation, RBI and MeitY in guidance.

  • SEBI Regulation 16C (2025). The SEBI (Intermediaries) Amendment Regulations, 2025 added a regulation titled "Responsibility for the use of artificial intelligence". A regulated person using AI or ML tools, "either designed by it or procured from third-party", is solely responsible for data privacy and security, for "the output arising from the usage of such tools", and for legal compliance. Buying the model from a vendor does not move the responsibility.
  • SEBI's retail algo-trading framework. The February 2025 circular makes the broker the principal and the algo provider its agent. It requires every algo order to carry an exchange-issued identifier for the audit trail, bans open APIs, allows access only through a client-specific API key and whitelisted static IP with OAuth and two-factor authentication, and keeps a kill switch at the stock exchange for every algo ID. It applies to all brokers from 1 April 2026.
  • RBI FREE-AI (August 2025). The Reserve Bank's committee report on responsible AI in finance is explicit: "Accountability rests with the entities deploying AI", and they remain fully accountable "regardless of their level of automation". It also says customers must be told when they are dealing with AI.
  • India AI Governance Guidelines (MeitY, November 2025). Rather than a new AI law, the guidelines propose "a graded liability system" in which responsibility is proportional to the function performed, the risk anticipated and "the degree to which due diligence is undertaken".
  • DPDP Act, 2023. A data fiduciary is responsible for processing done on its behalf by a processor, "irrespective of any agreement to the contrary", and a failure to keep reasonable security safeguards will be able to attract a penalty of up to ₹250 crore once these obligations take effect in 2027.

Notice the phrase in the MeitY guidelines: the degree to which due diligence is undertaken. Due diligence is your defence, and due diligence has to be shown with evidence. Hold on to that; it is the key to the rest of this post.

The rest of the world points the same way. The EU AI Act requires high-risk systems to let a human "disregard, override or reverse the output" and "interrupt the system through a 'stop' button" (Article 14), and requires deployers to assign oversight to people with "the necessary competence, training and authority" (Article 26). The Digital Omnibus signed in July 2026 pushed those high-risk obligations back to December 2027 for stand-alone systems and August 2028 for AI embedded in products, and the proposed EU AI Liability Directive was withdrawn in 2025. The dates moved; the direction did not.

The Real Gap: You Cannot Prove What Happened

So the question was half right. The rules on who pays are clear. What is missing is the ability to prove what the agent was allowed to do, what it actually did, and who approved it. That is where the 2026 surveys are sobering:

  • Only 21% of organisations say they have a mature governance model for agentic AI (Deloitte, State of AI in the Enterprise 2026).
  • At large US companies, 26% of those using agentic AI cannot detect unauthorised AI agents operating internally, and 47% have bypassed their own AI governance policies to push urgent deployments (EY US AI Risk and Governance Survey, September 2026).
  • The share of organisations building controls into their agents fell from 43% to 30% in two quarters, and in the latest quarter alone their confidence rose from 57% to 73% (KPMG AI Quarterly Pulse, Q3 2026, US leaders).
  • Only 20% of US executives say they would trust AI agents with financial transactions (PwC AI Agent Survey, 2025).
  • The AI Incident Database recorded 362 incidents in 2025, up from 233 in 2024 (Stanford AI Index 2026).

A widely reported July 2025 incident shows what the gap looks like. According to SaaStr founder Jason Lemkin's account, an AI coding agent on the Replit platform deleted a live production database during an explicit code freeze. Replit's CEO called it "unacceptable and should never be possible", and the company said backups allowed a restore, offered a refund, and added automatic separation between development and production databases. Nobody debated who was responsible. The problem was that nothing technical had stopped the agent.

The research community reaches the same conclusion. An MIT Sloan Management Review and BCG expert panel in September 2026 found that 72% of its responding experts agreed governance that treats agents as autonomous decision makers will fail. As panellist Simon Chesterman put it, "Autonomy in the engineering sense is not autonomy in the moral or legal sense." Gartner says the same thing in plainer words: "accountability for outcomes remains with the organization."

The Accountability Stack: Five Controls That Make Responsibility Provable

This is the part I care most about as someone who builds and teaches agentic systems. Accountability is not created by a policy PDF. It is created by a chain of evidence that runs from a named human, through a mandate, to every action the agent takes:

Named human owner
   → signed mandate (what the agent may do, and up to what limit)
      → agent identity (its own, never a shared login)
         → scoped, short-lived token for each task
            → action (with an approval step if irreversible)
               → tamper-evident log entry
                  → owner reviews, or the circuit breaker trips

Each link maps to a control. Each control already has a deep-dive on this blog, so I will keep each one short.

1. Give every agent its own identity

If an agent uses your login, every action looks like yours, and nothing can separate what you did from what it did. Give each agent a distinct identity and tag every action with it. NIST's National Cybersecurity Center of Excellence published a concept paper in February 2026 for a proposed project on exactly this: identification, authorisation, auditing and non-repudiation for AI agents. The World Economic Forum's 2025 agent governance paper notes that "unique identifiers and output tagging support attribution, performance tracking and post-incident analysis". See Zero-Trust Security for AI Agents for the implementation.

2. Delegate scoped authority, not your keys

OWASP lists "Excessive Agency" (LLM06:2025) as a top risk for LLM applications, caused by excessive functionality, excessive permissions or excessive autonomy. The fix is least privilege: issue the agent a short-lived token that carries only the permissions and limits for this task. OAuth 2.0 token exchange (RFC 8693) can record in the token itself that an agent is acting on behalf of a user. SEBI's algo framework is a live example of the idea: client-specific API keys, a whitelisted IP and OAuth, never an open API or a shared password.

3. Tier autonomy and gate the irreversible

Not every action deserves the same freedom. Gartner describes four levels: observe, advise, act with approval, and act autonomously. Singapore's January 2026 framework for agentic AI asks organisations to define "significant checkpoints at which human approval is required". Put moving money, deleting data and external communication behind an approval step. Gartner also warns that human approvals "can degrade under time pressure or approval fatigue", so ask for approval only where it matters. Human-in-the-Loop AI: 4 LangGraph Patterns shows how to build these gates.

4. Keep a tamper-evident audit trail

SEBI's requirement that every algo order carries an exchange-issued identifier is the pattern to copy: every action is tied to an agent, a human owner and a mandate, in a log the agent itself cannot edit. When a dispute comes, and it will, this log is your evidence of due diligence. See AI Agent Observability in Production.

5. Circuit breakers and a named human owner

SEBI's framework keeps a kill switch at the exchange for every algo ID. The EU AI Act will require a stop button for high-risk systems from December 2027. Gartner calls for "circuit breakers that halt agent operation on threshold violations and clear ownership for agent behavior", and the MIT Sloan panel advises assigning clear ownership of agent outcomes to specific roles or individuals. A kill switch nobody owns will not be pulled in time. Build Your Enterprise AI Control Plane covers where these controls live.

A word on hacking. The first scenario was "my account is hacked". With agents, the attacker may not need your password at all. In NIST's agent-hijacking evaluations, attack success against a tested agent rose from 11% with baseline attacks to 81% with new red-team attacks, by planting instructions in content the agent reads. An agent's credentials need the same protection as a human's, and more. Read Prompt Injection Attacks on AI Agents before you give any agent write access.

What to Do on Monday

If you are a founder or individual using an AI agent

  • Never give an agent your own password or OTPs. Use broker-issued or bank-issued API access with limits you can revoke.
  • Keep a human approval step for anything that moves money or speaks for you publicly.
  • Check what the agent actually did, not just what it reports. If there is no log you can read, do not delegate.

If you build agents

  • Treat the five controls above as acceptance criteria, not a future hardening phase.
  • Log the mandate alongside the action, so you can prove the agent stayed inside it.
  • Test for hijacking with indirect prompt injection before any agent gets write access.

If you run a regulated entity

  • Map every agent to SEBI Regulation 16C or the RBI FREE-AI sutras, and name an accountable owner for each.
  • Keep a registry of every agent, its identity, its permissions and its owner. You cannot govern agents you cannot see.
  • Rehearse the kill switch. An untested kill switch is a hope, not a control.

Frequently Asked Questions

Can an AI agent be held legally responsible for its actions?

No. An AI agent is not a legal person in India, the EU, the UK or the US. Responsibility falls on the people and organisations that deploy, supply or instruct it. In Moffatt v. Air Canada (2024) the tribunal rejected the argument that a chatbot was a separate legal entity responsible for its own actions.

Who is responsible if an algo or AI places a wrong trade in India?

Under SEBI's retail algo-trading framework (circular of 4 February 2025, applicable to all brokers from 1 April 2026), if the order came through an algo provider your broker has empanelled, the broker is the principal, the provider acts as its agent, and the broker must handle your grievance. If the order came from your own agent using your own API key, expect it to be treated as your own order. Separately, SEBI's Regulation 16C makes a regulated entity solely responsible for the output of the AI and ML tools it uses, whether built in-house or procured.

Am I protected if money leaves my bank account without my authorisation?

Under RBI's customer-liability rules (first issued on 6 July 2017, now part of RBI's 2025 consolidated directions), you have zero liability if the bank was at fault, or if the breach was a third party's doing and you report it within three working days of the bank's alert. The bank must prove any customer liability. If you were negligent, for example by sharing payment credentials, you bear the loss until you report it. RBI has issued revised rules that apply from 1 January 2027.

Do the EU AI Act's human-oversight rules apply yet?

Not yet for most high-risk systems. The Digital Omnibus on AI, signed in July 2026, moved the high-risk obligations to 2 December 2027 for stand-alone systems and 2 August 2028 for AI embedded in regulated products. Articles 14 and 26 still describe what regulators will expect: a named, competent human overseer and a way to stop the system.

What is excessive agency in AI agents?

Excessive agency is entry LLM06 in the OWASP Top 10 for LLM Applications 2025. It describes an agent that can do more damage than intended because of excessive functionality, excessive permissions or excessive autonomy. The mitigations are least privilege, narrowly scoped tools and human approval for high-impact actions.

What is an AI agent kill switch?

A kill switch is a mechanism that halts an agent immediately and revokes its access. SEBI's algo-trading framework keeps a kill switch at the exchange for every algo ID, the EU AI Act requires a stop button or similar procedure for high-risk systems (from December 2027), and Gartner calls for circuit breakers that halt agent operation on threshold violations.

Conclusion

So, to the question from my venture group: when an AI agent acts without your authorisation, someone is always accountable, and it is almost never "the AI". It is the company that deployed it, the regulated intermediary that carried the action, or, increasingly, the person who handed over the keys. Air Canada, SEBI, RBI and the EU all point the same way.

The gap behind that question is real, but it is mostly not a gap in the law. It is a gap in evidence. Most organisations cannot yet show what their agents were allowed to do. Closing that gap is an engineering job: an identity per agent, scoped authority, approval gates, tamper-evident logs and a kill switch with a named owner. Build those, and accountability stops being a legal abstraction and becomes something you can prove.

Sources

This article is for general information and reflects the author's reading of public sources as of September 2026. It is not legal, financial or investment advice; the author is not a lawyer or a SEBI-registered adviser.