Replika, a ‘virtual friendship’ AI chatbot, receives GDPR ban and threatened fine from Italian regulator over child safety concerns

Replika, a 'virtual friendship' AI chatbot, receives GDPR ban and threatened fine from Italian regulator over child safety concerns

Published 23 March 2023

The Italian Regulator (Garante) has recently issued an order banning the Replika app from processing Italian users' data following an investigation. Luka Inc (the developer of Replika) is an artificial intelligence ("AI") company based in the US which operates a 'virtual friendship' service based on customisable digital avatars whose responses are personalised and powered by AI to make its "human users feel better".

With the rise of ChatGPT and other AI tools, there are multiple privacy concerns, especially where some of the tools can be accessed by children. The Italian regulator has previously been at the forefront of challenging tech companies' processing of children's data and is usually quick to respond to any unlawful processing of data. The Italian regulator is the first to take formal action over the Replika app. To date, no other European data protection authorities and/or the UK's Information Commissioner have followed suit.

What is Replika? It is an AI chatbot which mimics user behaviour with an aim of serving as a virtual friend and/or a romantic partner. It's always there to respond to the users' questions and is "here to chat about anything, anytime". Replika's marketing campaigns make users believe they can find "AI soulmates" and pushes the idea that users can find a meaningful relationship with the AI chatbot. The app also supports the idea of a "romantic" partner confirming the "more you speak to Replika, the smarter it becomes". It is believed that Replika has more than 10 million users globally.

Although, the Replika app is aimed at users over the age of 17+, the Garante has evidence that this is being accessed by children under the age of 17. Further, the developer's terms of service only prohibit use by under 13s. The Italian DPA raised its concerns over the potential harm to children, because the app neither has age verification measures in place, nor user restrictions for users who declare themselves as underage. The Italian data protection authority also found that some of the chatbot's responses are deemed inappropriate for younger users and at "odds with the safeguards… children are entitled to". Further, there have been several reviews on the two main App stores that users have flagged sexually inappropriate content.

Notwithstanding the Italian DPA's focus on the risks posed to under-age users, the Garante also raised concerns with vulnerable adults as Replika's marketing language mostly appeals to vulnerable adults or those who would benefit from emotional support. Nonetheless, there is nothing on Replika's website which confirms how the app is approved to provide this type of mental health support. The app uses techniques to facilitate users in providing sensitive data by convincing users that they are in a real relationship and therefore users forget that the chatbot AI is nothing more than a language model-based programme and not a human. In fact, it encourages the users to share information in a safe space and manipulates users who may not ordinarily share such sensitive information. It follows that the app is collecting and processing high amounts of personal and sensitive data which is currently unlawful.

The Italian DPA further emphasised that the app does not disclose the essential information needed under GDPR to process personal data, in particular children's data, which is in violation of GDPR transparency provisions. The Italian DPA also found that there is no clear legal basis for data processing by Replika and has raised concerns over the data sharing with the US.

All of the above sparked grave concerns with the Italian data protection authority ("DPA") who has ordered the company to stop processing users' data effective immediately, with its primary concerns over the risks posed to children. The Garante has held that:

"Replika violates the European regulation on privacy, does not respect the principle of transparency, and carries out unlawful processing of personal data, as it cannot be based, even if only implicitly, on a contract that the minor is unable to conclude."[1] (free translation)

The order is an immediate ban on processing all Italian users' data (both adults and children) and should they fail to comply within 20 days, Replika will face a fine of up to 20 million euros or 4% of its annual turnover. Luka Inc has 60 days to appeal the decision.

With the development on new AI technologies ever growing, it is clear that many data protection authorities have concerns with how children's data is used. The Italian regulator has shown itself particularly sensitive to child safety concerns in the past. However, despite the GDPR enforcement around children safety issues, many campaign groups still argue that children are still not adequately protected. Closer to home, the UK is now working to pass the Online Safety Bill which in part responds to public concerns of what children are exposed to online. In terms of Replika itself, it is yet to be seen whether any other data protection authorities will follow in the Garante's footsteps and / or whether Replika will respond, but we will keep a close eye on developments. One thing we know for certain is that restrictions on the use of children's data is only going to get tighter.

The Order is available to read in Italian and in English here.

[1]https://www.garanteprivacy.it/web/guest/home/docweb/-/docweb-display/docweb/9852214#english

Public link

Please use the above public link if you want to share this noodl on another website.