By Jack Shepherd, Senior Principal Business Consultant, iManage
Focusing on gen AI return on investment
It has been two years since the world became enamoured with ChatGPT and generative AI. Since then, the focus on law firms and legal teams exploring the impact of AI has been relentless. Organisations have increased their innovation and technology budgets to make sure they are taking advantage of the opportunities AI might bring them.
Yet still, two years on, many organisations have struggled to move beyond the stage of learning what generative AI is, how it works ,and what benefits it might bring. While the industry is still very excited about generative AI, we have seen 2024 bring more realism. This has led innovators to discover where generative AI might be best deployed and where its use should be avoided rather than simply using it everywhere.
“many organisations have struggled to move beyond the stage of learning what generative AI is”
As 2025 goes on, we should expect to see less resource allocated to exploring AI for the sake of it, and more focus on what its return on investment might be. Generative AI is not immune from the harsh reality that technology is only as powerful as the value it delivers. To discover that value, there is no shortcut for organisations engaging with their users to find out what is preventing them from delivering better outcomes and then considering whether generative AI or any other technology might be well suited to delivering improvement. It all starts with problem and process analysis, not technology.
Obsessions with accuracy
We’ve seen a number of academic papers and press articles about various claims vendors are making regarding the degree to which their AI applications can avoid hallucinations. Many of these publications point out that some of the claims regarding AI accuracy made by legal technology companies are often overstated.
Some have drawn the conclusion from this that until AI accuracy metrics can be improved, it will have no real place in legal practice. This position holds true if you think that the only application of AI is to generate the first draft of legal research memos or contracts, or to otherwise automate away a large part of what lawyers do.
“until AI accuracy metrics can be improved, it will have no real place in legal practice”
Such a viewpoint takes too narrow an approach to the potential applications of AI. It is easy to jump to the conclusion that the only place for generative AI is to “generate legal work product”, but a more nuanced approach would acknowledge that it is neither possible, desirable, nor sensible to put generative AI in the driving seat of legal work. For example, if generative AI is not capable of generating a first draft of a legal research memo, can it be put to work in other ways, such as finding cases for lawyers to review, highlighting key parts of those cases for lawyers to consider, replacing repetitive “control F” exercises when reviewing contracts, or providing suggestions for the piece of work a lawyer has already written?
Instead of waiting for AI accuracy to improve, many firms are already looking at the more granular tasks that AI can assist with. Such an approach values incremental innovation over big-bang innovation but usually reaps benefits in the short and long term for organizations that adopt it.
Indeed, some organisations are wondering whether, even if AI were 100% accurate, would it be desirable for humans to be replaced by AI? Sometimes, the value is in the process rather than the end product. For example, if you need to do some technical legal research, what is more important: a lawyer who knows the law very well and can help facilitate a discussion, or a lengthy legal memo?
The practical guidance most firms should follow is to focus on the use cases and problems that matter to users rather than getting carried away with excitement and hype around technology.
Agentic AI on the cards
Perhaps all of this explains why proponents of AI have been so focused on renewing the excitement in AI by exploring what agentic AI might deliver. Agentic AI is nothing new per se but represents a new way of deploying generative AI models – specifically by allowing LLMs to take a prompt and break down the response into a series of steps that it feeds into itself. It is, in other words, an LLM conversing with itself and checking its own chain of thought
Agentic AI has the potential to be transformational as well as a huge disappointment.
Those deploying it sensibly will combine it with process design, looking at a process holistically, breaking it down into steps, and deploying agentic AI in steps where a more nuanced and flexible approach is appropriate.
Those who treat agentic as a silver bullet to solve processes will be sorely disappointed. Throwing agentic AI at a process without thinking about whether the process needs a complete redesign will result in chaos and inconsistency. These are things most legal teams should be trying to avoid.
Jack Shepherd
