China says no to suicide and gambling AI chatbots

Regulators in China are planning a clampdown on AI-powered chatbots that are pushing people into suicidal emotions, self-harm activities, and gambling. The development comes as two leading chatbot companies in China recently filed for IPOs in Hong Kong.

Reports suggest that the newly proposed measures that were announced on Saturday will apply to AI products or services that are offered to the public in China that simulate human personality and engage users emotionally via texts, images, audio or video.

China’s proposals are expected to protect minors from self-harm

According to the draft rules that were released on Saturday by the Cyberspace Administration, these are targeted at what it has termed “human-like interactive AI services,” as per CNBC translation of the Chinese-language document.

The draft rules have several proposals. For example, AI chatbots cannot generate content that encourages self-harm or suicide, engage in verbal violence or emotional manipulation that can damage users’ mental health.

Additionally, AI chatbots are not supposed to create obscene or violent and gambling-related content. According to the draft rules, if a user proposes suicide, the AI company is supposed to have a human who takes over the conversation and immediately contacts the user’s guardian or a designated individual.

The draft rules also propose that minors have guardian consent for emotional companionship use, with time limits on usage. Under the new rules, AI platforms are expected to decide if a user is an adult or a minor even if they do not disclose their age. In the event of doubts, platforms must apply settings for minors, while allowing for appeals.

Once finalized, these rules would mark the world’s first attempt to regulate AI with human or anthropomorphic characteristics, according to NYU School of Law professor Winston Ma. These developments come as businesses have rapidly developed AI companions and digital celebrities.

When comparing this with China’s 2023 generative AI regulation, Ma opined that this version “highlights a leap from content safety to emotional.”

The proposals come as two Chinese AI chatbot startups Z.ai and Minimax have this month filed for initial public offerings (IPOs) in Hong. Minimax is best known internationally for its Talkie AI app that lets users chat with virtual characters.

According to CNBC, the app and its domestic Chinese version, known as Xingye, accounted for more than a third of the firm’s revenue in the first three quarters of the year, with an average of over 20 million monthly active users during that time.

As for Z.ai, which is also known as Zhipu, it filed under the name Knowledge Atlas Technology, but did not disclose its monthly active users. However, the AI company revealed that its technology is on about 80 million devices, including smartphones, personal computers and smart vehicles.

As previously reported by Cryptopolitan, the two AI startups, both backed by Alibaba and Tencent are targeting to go public in early January next year on the Hong Kong Stock Exchange.