The legal world is grappling with the disruptive power of generative AI, where its capacity for “hallucinations”—creating non-existent information—has led to significant repercussions. From high-profile attorney sanctions for citing fabricated cases to federal judges admitting staff-generated errors, the legal system is being forced to confront the critical need for robust human oversight and ethical guidelines when integrating advanced AI tools.
The legal system, traditionally a cautious adopter of new technologies, is finding itself at a crossroads with the rapid advancement of generative artificial intelligence. While the promise of AI to streamline legal work is immense, its inherent flaws, particularly the tendency to “hallucinate” facts and sources, have plunged the profession into a series of embarrassing and costly incidents. These events are not merely cautionary tales; they are foundational moments reshaping how attorneys and even judges interact with AI, emphasizing that technological assistance must always be paired with rigorous human oversight and “actual intelligence.”
The Infamous Genesis: Mata v. Avianca
The most widely cited example of AI’s pitfalls in litigation originated last summer in an otherwise routine personal injury action, Mata v. Avianca. Facing an unfamiliar legal issue on a motion to dismiss, two New York attorneys turned to ChatGPT for research assistance. Unbeknownst to them, the AI chatbot conjured up half a dozen non-existent cases to support their client’s position, complete with “stylistic and reasoning flaws that do not generally appear in decisions issued by United States courts of appeals,” as dryly noted by Judge Kevin Castel of the Federal Southern District of New York. The judge less charitably described the AI’s legal analysis as “gibberish.”
The attorneys, Steven Schwartz and his co-counsel, relied on these fabricated citations in their briefing. Defense counsel, unable to locate the cases, brought the issue to the court’s attention, leading to a swift investigation. The episode culminated in a $5,000 sanction from the court for Schwartz and his firm, making Mata v. Avianca, 678 F. Supp. 3d 443, 453 (S.D.N.Y. 2023) a global headline in 2023 and one of AI’s most high-profile professional failures. This case highlighted a critical lesson: lawyers carry an undeniable oversight duty that cannot be delegated to an AI tool, however advanced it may seem.
A Spreading Problem: More Sanctions and Judicial Admissions
The Mata v. Avianca incident was not an isolated event. Subsequent cases have continued to expose the dangers of unchecked AI use:
- In an ongoing case involving HoosierVac, attorney Rafael Ramirez was ordered to pay $15,000 for including made-up cases in several briefs filed in October 2024. Despite his explanation that he was unaware AI could generate fictitious citations, U.S. Magistrate Judge Mark Dinsmore for the Southern District of Indiana stressed that Ramirez “did not make the requisite reasonable inquiry into the law.”
- In a separate January case against a hoverboard manufacturer and Walmart, attorneys cited eight non-existent cases. Following an inquiry by U.S. District Judge Kelly, Judge Rankin issued sanctions, including revoking one attorney’s pro hac vice admission and fining three others between $1,000 and $3,000 each.
Beyond attorney misconduct, the issue has even reached judicial chambers. In response to an inquiry by U.S. Senate Judiciary Committee Chairman Chuck Grassley, two federal judges admitted that their staff used AI to help prepare recent court orders, leading to “error-ridden” rulings. U.S. District Judge Henry Wingate in Mississippi stated that a law clerk used Perplexity as a “foundational drafting assistant,” while U.S. District Judge Julien Xavier Neals in New Jersey disclosed that a law school intern used OpenAI’s ChatGPT for unauthorized research. Both judges cited “human error” in the review process and have since implemented new AI policies and enhanced review procedures.
The Ethical Imperative: Lawyers’ Obligations in the Age of AI
Despite these incidents, the legal profession remains largely optimistic about AI. Research cited from Thomson Reuters indicates that 60% of attorneys consider themselves optimistic about AI entering the legal field. However, this optimism is tempered by a clear understanding of the ethical responsibilities involved. Panel discussions with former federal magistrate judges, like Andrew Peck and Ron Hedges, have underscored critical duties:
- Supervision is Paramount: Lawyers must understand how their AI tools work, supervise their teams, and meticulously check results. The senior-most attorney on a matter bears ethical responsibility, just as they would for a junior lawyer’s research. As Peck noted, “You can’t tell the judge, ‘I didn’t miscite this case, your honor, the AI miscited this case.’”
- Rule 11 and Professional Conduct: Attorneys must recall their obligations under Rule 11 of the Federal Rules of Civil Procedure for representations to a court, ensuring that all filings are factually and legally sound. Additionally, Rule 1.1 of the American Bar Association’s Model Rules of Professional Conduct, covering competence in client relationships, extends to the careful and competent use of AI tools. You can find detailed information on these rules from the American Bar Association.
- Confidentiality Concerns: Using public AI platforms for queries involving client data poses significant confidentiality risks. Lawyers must “take steps to prevent any information being leaked… into a large language model,” stressed Hedges.
- Vendor Scrutiny: Attorneys should inquire with vendors about how AI models are trained, how they work, and what biases they may display. The same questions should be posed to clients if their internal large language models are used.
These guidelines are crucial as AI moves into sensitive areas like document and privilege reviews, where “black box” concerns—the inability to fully understand an AI’s decision-making—could lead to new challenges. Judges and legal experts hope that early judicial decisions in these areas will be approving of AI’s proper use, as “one bad opinion… can set this back for years,” Peck stated.
Balancing Innovation with Prudence
The journey of AI in the legal profession reflects a broader trend of rapid technology adoption; ChatGPT, for instance, amassed 100 million users in a mere 65 days. This explosive growth necessitates a proactive approach from the legal community. While judges like Dinsmore are “a vocal advocate for the use of technology in the legal profession,” they consistently emphasize caution. “Much like a chain saw or other useful [but] potentially dangerous tools, one must understand the tools they are using and use those tools with caution,” Judge Dinsmore wrote, adding that “the use of artificial intelligence must be accompanied by the application of actual intelligence in its execution.”
The lesson from these early AI stumbles is clear for legal professionals: embrace experimentation with generative AI, but do so responsibly and “behind the scenes.” Rigorous verification of all AI-generated output is non-negotiable before anything is submitted to the court. By understanding AI’s limitations, upholding ethical duties, and applying sound judgment, the legal profession can harness AI’s power without falling victim to its digital deceptions.
