Gen AI: Industry engagement essential to understand how Gen AI is achieving positive consumer and market outcomes as well as how risks are being managed by firms

Daniel Trinder, Executive Director, Strategy and Design.

Generative Artificial Intelligence (Gen AI) is the latest technological breakthroughthat regulatorsmust grapple with. GenAI refers to large learning models (LLMs) that can both comprehendand generate human language text -text, audio, code, graphics - based on user-defined queries of the large quantitiesof data on which the model was trained.

As Gen AI technologies arestill maturing, their application is nascent. However,this "technology cycle"is not being driven by technology upstarts,but by the world'sbiggest technology firms with established adoption across the financial sector in New Zealand which gives potential to transform financial services.

Industry engagementwill beessential for the FMA to understand how Gen AI is achieving positive consumer and market outcomes,as well as how risks are being managed by firms.

Potential benefits includebetter consumer outcomes and personalised customer experience. For instance,through faster response times and claims pay-outsalong with efficiency gains to insurance firms deploying Gen AI sensibly.Advancements in transaction monitoring can result through improvements in detectingsuspicious activity. These tools shouldalso improve fraud detection and risk management.

However,with potential benefits come potential risks. New Zealand banks have said recentlythat Gen AI is a "double-edged sword". View the FIPS Banks: Review OF 2023 report from KPMG. Where there areopportunities,there'salso potential for cyber threats and fraud to be AI-weaponised by criminals and bad actors.

There may be the need for some firms to reassess their approachto governanceto ensure adequate oversight of AI tools. Theincorrect adoption of Gen AI applications can lead to undesired consequencesand reputationaldamage or may not contribute to the business value of the organisation.Appropriate governance of such applications mitigates the associated risks,holdstheappropriate peopleresponsible,and can increase longevity and security of applications.

Firms need to focus on implications for their own operational resilience, including any services that they outsource to third parties. Most Gen AI models are run on the cloud and powered by large data,creating digital infrastructure and third party-related stability risks that must be considered.Firms are also at risk from AI-powered cyber-attacks.

One of the defining features of Gen AI is its ability to process large volumes of data, and to detect andexploitpatterns in those data. Model risk management, and the risks that Gen AI exacerbateswith sample bias,andmodel drift are important considerations for firms.

There are also a broader set of issuesregardingdata usage, accountabilityand protection.

AI accountability is the ability to explain, justify, and take responsibility for the outcomes and impactsof an AI system.Governance and assurance over data privacy and ownership will also be paramount.

Last, but not least,aretheheightened risks to consumers. AI scamsare already on the riseand the deepfake technology raises risks to consumersfromscamsor fraudulent activities.

Despite potential risks, the FMA believes that firms can mitigate these risks through appropriate governancestructures, risk management, remainingvigilant over operational resilience implications and addressing data risks. All this shouldensure firms utilising Gen AI remain focused on achieving positive and safe outcomes for consumersand markets.That said,it is not the FMA'sjob to tell firms how to establishthe guardrails they need.

Overtime we will assess whether our regulatory framework needs strengthening to support better deployment of Gen AI. We welcome ongoing dialogue with all firms on their experiences of using Gen AI,and similar technologies,to enhance our understanding of the benefits and understand if there are risks materialising to consumers through the incorrect adoption of Gen AI.

The FMA is technology-neutral and pro-innovation. We are keen to see firms harness the opportunities and to ensure firms utilising Gen AI remain focused on achievingoutcomes that will benefit and enhanceconsumer's experience of financialmarkets and services.