More than a buzzword, reliability is the cornerstone of professional-grade AI products

We’ve shared a lot about what’s vital in a generative AI solution for law, including: it’s specifically designed for and grounded in law, it’s built by both machine learning and AI and legal experts, and it adheres to the highest privacy and security standards.

Throughout these posts we’ve returned again and again to the idea of reliability. What most people think of when they hear “reliable” is “doesn’t hallucinate.” And of course freedom from hallucination—a phenomenon associated with all large language models (LLMs)—is crucial when using a tech solution to assist with legal work. No one wants to wind up like New York lawyer Steven Schwartz, who cited to cases hallucinated by ChatGPT in his federal brief.

But lack of hallucination is only one aspect of reliability. A great deal of the attention paid to the newest LLMs concerns their mistakes, but it’s just as important when vetting a legal AI product to think about what it does do—and who is doing it—as opposed to what it doesn’t do. 

Reliability means the law comes first

Building a tech product requires tech expertise. Building a domain-specific tech product, especially one for professionals, also requires at least equal expertise in that subject area. And if that profession is particularly specialized—like law—you should make sure the people behind the product have literally put themselves in their customers’ shoes. Are they attuned to the needs of a particular industry, or are they simply choosing a profession for which to use the latest attention-grabbing tech? Are they relying on user research and intelligence about what the professionals want and need when building a product? Or are they themselves current or former practitioners?

These details make the difference between a product that exists solely because of an opportunity in the market and a product that’s genuinely created to solve problems for professionals over the long term. From its founding over a decade ago, the Casetext team has been focused on advancing the efficiency and quality of work across every facet of a lawyer’s practice, transforming it by applying the power of AI. We’ve envisioned integrated technology that would empower attorneys to spend more of their limited time on complex work, ultimately elevating their practice. And for more than five years we’ve based decisions about how to focus the business on our conviction that LLMs were the key to doing this, long before LLM was the everyday term it’s become over the last year.

Reliability means a proven legal tech track record

While Casetext has most recently garnered recognition for CoCounsel, the first AI legal assistant, CoCounsel was only possible because of the decade of work that preceded it. Even before launch, more than 10,000 law practices—across sizes, types, and practice areas—used Casetext products. And when asked to take a chance on a new and nearly unbelievable piece of technology, those same lawyers trusted Casetext because of ten years’ worth of credibility, born of close attention and hard work led by Casetext’s founding team. 

These achievements were made possible because Casetext leadership prioritized hiring experienced attorneys (who have a combined 110+ years of experience), AI and machine learning experts, and product and design leaders. And many of the members of the Casetext team have experience in more than one of these disciplines. Together we’ve launched 6 products, all working with the power of AI long before it was a household word.

A trustworthy solutions provider also needs to have proven they’re above the hype. Are they thoroughly vetting new technologies before releasing products? Are they prudent about taking the burden of ensuring a solution is ready, rather than relying on early customers to essentially be paying beta users, even if it means missing the opportunity to be part of the latest “hype cycle”? For instance, we at Casetext have actually been working with generative AI—a term that almost no one had heard of before 2022—since 2018. And before deeming CoCounsel ready for launch, our expert attorneys and prompt engineers spent thousands of hours and several months honing it—filtering, ranking, and scoring results.

Because we’d “walked the walk” for so long, OpenAI selected Casetext to take the lead in applying the power of GPT-4 to the practice of law, giving us early access to the model and the opportunity to build CoCounsel and create the entire AI legal assistant category itself.

Reliability means providing partnership, not just product

While a trustworthy product that delivers what you need is vital, without substantive, sustained support you’re unlikely to get the most possible value from your investment. Especially when it comes to products built on the latest generation of generative AI, which can take time to use well. A vendor worth your time knows this and has invested the time and resources in building a success and support team that will be there for you long after you’ve signed a subscription agreement. 

Things to look for: Do they work with you to deeply understand your most prevalent use cases and how their solution will help you tackle them? Do they attend to the “small stuff,” ensuring all your users are able to log on, launch, and use the product, consistently? What happens when people run into problems, is there support available all the time? And what about keeping you informed of changes to the product, additions, improvements? Or continuing to create content you can use to increase your own knowledge, such as videos, webinars, and written materials?

And last, how open are they about their future vision for the product? Will it continue to grow with you, expanding capabilities, or simply make minor adjustments and optimizations to existing functionality? Do you feel as if your input matters to the product team in a real sense? Do you see customer feedback being applied to subsequent versions? 

Bottom line: Do you feel as if the product provider is as invested in you as you are in them? If you’re not sure, that tells you all you need to know.