Safeguarding against AI’s existential risks and small mistakes
Controlling for hallucination and error in AI
Joudrie emphasizes auditable and traceable calculations in SAM because that allows users to more clearly double-check the work done by the AI agent. AI agents have become a popular means of expediting workflows and introducing new angles of discussion between advisors and clients. While Joudrie sees why that popularity is growing, he stresses the need for the underlying data that the agents use to be verifiable by the advisor. Without that clarity the ‘small’ error of a hallucination can become a core fault in the plan and pose real long-term risks to the client.
Financial plans built in their software platform need to be living entities, pieces that can be revised and reworked based on clients’ changing circumstances. By ensuring that the work SAM does in building that plan can be traced back to initial inputs and run through the correct calculations, the plans can be revised without the risk of a core error. Joudrie insists that because of the emphasis on diligence and traceability, Conquest’s engine “doesn’t hallucinate.”
One of the sources of hallucination in other AI large language models is the sheer volume of information they pull from across the whole internet to answer a question. One of the tools Conquest has elected to use to avoid that is what they call a “small language model” which only pulls from conquest’s native environment. That more limited scope ensures that the tool is tied only to what has already been put into the system.
Managing existential risks in AI
Joudrie acknowledges that even for the kind of limited-scope and disciplined agents that his firm uses, there is a risk that AI agents can act in their own best interests rather than the interests of their users. This is one of the core risks being highlighted by AI CEOs like Anthropic’s Dario Ammodei in their statements about the risk of an AI-driven apocalypse. Some of the solution to this risk, especially in financial services, will come from regulators and dealers as they develop more complete forms of AI policy. However, Joudrie says that his team has factored this risk into their emphasis on traceability. If it appears that the AI agent is making suggestions or plans that serve its interests over the clients, the roots of those suggestions can be identified and corrected through those more traceable pathways.
Core cybersecurity principles also play a role in Conquest’s approach. They recently completed a new AI designation called ISO 42001, the world’s first international standard for an AI management system. They have also maintained a data sovereignty policy from their outset. All of Conquest’s data is stored in the country that their customers are based in. Canadian data is domiciled in Canada, UK in the UK, US in the US, all through Amazon Web Services in each country.
