Financial institutions are rapidly adopting AI within their inventory of complex models. We believe, along with most internal auditors and risk managers, that it is imperative to identify and manage the new business and regulatory challenges that accompany the use of AI.

At its core, AI models are simply another form of an End User Computing (EUC) Application.

For over 25 years, CIMCON Software has helped financial institutions reduce their risk from models/EUCs. CIMCON has leveraged its years of experience, technology and innovation to apply to AI models. The CIMCON tools can now be used to automatically identify and detect the prevalence of AI models in the organization and perform an automated risk assessment, as a precursor to maintaining an inventory and monitor the models for changes.

It is clear that AI has not only arrived but will become the dominant catalyst of change. In fact, according to a 2020 survey by the Economist, 77% of global bankers think that unlocking value from AI will differentiate bank winners and losers. However, as is often the case when companies are faced with the rapid adoption of technology as powerful as AI, the resultant increase in complexity, changes in data lineage, and enhanced data governance will challenge a company’s ability to determine the materiality/impact of its AI models. In short, AI will present a level of business, regulatory, and reputational risk that is unprecedented. This is why Gartner predicts that 85% of A.I. projects will deliver erroneous results.

We have already seen that errors in complex models and the lack of controls can cause disastrous results to financial institutions. In fact, a glitch in a trading algorithm cost Knight Capital $440 million when it accidentally bought and sold hundreds of stocks during a 30 minute period. As a result, Its stock price fell by 75% and a year later was acquired by its rival Getco. Only a few months ago, the lack of proper controls and unidentified errors in risk measurement models played a significant role in the collapse of Silicon Valley Bank.

What can firms do to mitigate AI risk?

Firms are moving into unchartered territory and without the appropriate updates to their policies, procedures, and controls, they will fail when deploying AI models within their organization. Moreover, any audit failures that result from flawed implementations will be significant and costly.

Based on our experience with 500+ clients over the last 30 years in Model/EUC Risk Management, CIMCON has developed a holistic approach to identify, assess and reduce AI Model risk. This approach is built to optimize the end user experience, accelerate business processes, and empower its users through actionable insight, intelligent automation, and powerful purpose-built workflows.

A Complete AI Model Risk Management Solution

CIMCON’s EUC Insight software delivers a holistic, end-to-end approach to managing and evidencing AI model risk management and regulatory compliance, by providing the following features and benefits in a single integrated platform:

1. Identification of AI Models: Identify the prevalence of AI models in the organization, including where they are being used, type of model being used, associated activity and frequency of use.

2. Automated AI Risk Assessment: Perform an automated risk assessment of the AI model using our proprietary algorithm based on a model’s complexity, code quality, dependencies, and other factors.

3. Model Inter-dependency: Explicitly called out in regulatory frameworks such as SS 1/23, a company must know how many models depend on the outputs of one or more other models. In this respect, input / output interdependence is of primary consideration when determining the inherent risk any particular model can present to an organization. By understanding the inherent risk and applying the necessary controls, an organization can understand whether its residual risk is within acceptable bounds.

CIMCON understands this critical relationship between inherent and residual risk and is able to customize an organization’s risk assessment models to ensure that this ‘balance of risk' is accurately managed. A key functional component of the CIMCON solution is to provide our customers with the ability to visualize model dependencies, both upstream and downstream including the ability to determine whether a ‘link’ is working correctly or is broken and requires the model owner’s attention to recover the link.

4. Inventory: Maintain an inventory of all your AI models, with configurable forms, workflows and alerts for periodic attestations.

5. Monitoring: CIMCON monitors the frequency with which a model is modified or accessed, who makes the changes, when and what is modified with a complete audit trail of all changes, that also includes a side by side compare of the before and after version.

In summary, by applying both quantitative and qualitative measurements to these key areas of risk, a company can implement a structured process of maintaining a balanced and accurate AI model risk management program.

Concluding thoughts

Large language models such as Chat GPT, deep learning text to image models such as DALL-E, as well as many others are transforming what is possible for us to accomplish as a society. For all of us to be a part of that future and reap the rewards that it can bring, we will need to embrace this change and become a part of this rapidly expanding future. We will need to collaborate, explore and creatively implement this new technology, while at all times remaining mindful that it is not perfect. AI Models can and will generate errors, which in turn could potentially hurt the organization’s bottom line, or even worst, damage its most valuable asset – the company’s reputation.

In the new normal, the world of model risk management has become exponentially more complex. Hence investing in automated tools that can identify, measure, and mitigate AI risk before any damage is caused has become the new imperative.