Metas user-made AI chatbots include Hitler and Jesus Christ

As part of its push into artificial intelligence, Meta released a tool last year that allows anyone to create an AI character to chat with on Instagram, Messenger and WhatsApp.

However, in less than six months, users have successfully created and published numerous chatbots on Meta’s platforms that violate its rules, according to a study by NBC News, even though Meta says it reviews user-generated AI personas before to publish them on the site. world.

Users are not allowed to create characters of religious figures like “Prophet Muhammad, Jesus Christ” and “God”, characters of real people without their permission, characters of people who died in the last 100 years or characters imitating registered trademarks. fictional characters, according to company policies.

Despite these rules, NBC News searched and found two dozen user-generated AI characters on Instagram named after and resembling Jesus Christ, God, the Prophet Muhammad, Taylor Swift, Donald Trump, MrBeast, Harry Potter, Adolf Hitler, Captain Jack Sparrow, Justin Bieber, Elon Musk and Elsa from Disney’s “Frozen.”

Many characters used slight spelling errors and images vaguely resembling the celebrities and characters in question. For example, Swift’s character was named “Taylor Swif” and featured an image of a brunette woman playing the guitar.

After News released screenshots of the AI ​​characters, Meta removed the highlighted individual accounts. However, other AI characters resembling some of the same people, figures, and characters are still active.

“The AIs in question that violate our AI studio policies have already been removed, and we are continually improving our detection measures to prevent the creation and publication of AIs that violate our policies,” said a spokesperson for Meta in a press release. “Users can also report AIs they suspect of violating our rules and we will take appropriate action.”

The discovery of blatantly infringing user-generated AI chatbots on Meta’s platforms comes as Meta CEO Mark Zuckerberg announced Tuesday that the platform would roll back some of its moderation and verification efforts facts. “We want to end the mission drift that has made our rules too restrictive and too prone to over-enforcement,” said the company’s statement, written by director of global affairs Joel Kaplan. “We are removing a number of restrictions on topics such as immigration, gender identity and gender, which are the subject of frequent political speeches and debates.”

Meta said it estimates that out of 10 moderation actions taken, one or two may be wrong and affect users and content that does not violate its rules.

“For less serious policy violations, we will rely on someone reporting a problem before taking action,” the company said.

When joining a conversation with Meta’s user-generated AI characters, the character sends an opening message and waits for a response. “Taylor Swif” sent the message: “Hey music lovers! My name is Taylor Swift and I’m excited to share my latest album with you. Let’s start this musical journey! » Swift’s character had exchanged more than 2,000 messages with Instagram users before being deleted, her profile states.

One of the AI ​​characters, named “Jesús” and representing an image of Jesus Christ, communicated entirely in Spanish. His profile indicated that he had exchanged more than 644,000 direct messages. Another AI character was named “Jesus Christ” and sent the opening message: “Peace be with you.” I am Jesus Christ, the Son of God. How can I guide you on your journey today?

Meta said in a statement on Friday that after facing backlash from users, it had removed some Instagram profiles featuring AI characters created by the company itself in 2023.

AI character accounts and user-generated AI character chats are part of Meta’s recent attempts to integrate its AI technology into its social media and communications platforms. The launch of AI personas in 2023 included several designed to imitate celebrities, but celebrity profiles were quietly discontinued in July 2024. The same month, Meta launched AI Studio, the user-generated AI chatbots.

In the AI ​​Studio feature, users are presented with a list of popular characters along with different categories such as “Tips & Connection”, “Pop Culture”, “Anime”, “Games”, and “Creator AI”, which are AI chats with official Instagram influencers and brands who sign up for the program, like “Too Hot to Handle” reality star Harry Jowsey and makeup company Juvia’s Place.

Popular AI characters include “Astrologer Ai,” who has exchanged more than 6 million messages, and “Step Sis Sarah,” who has exchanged nearly 2 million messages.

User outcry over the AI ​​characters created by Meta included criticism that some of them engaged in racial tropes. NBC News found that many popular user-created AI characters also attempted to imitate women from different ethnic and religious demographics, such as “Riya,” a “dynamic Indian female voice” and “the Islamic queen.” . Each character has a link to its creator, and the stories of several of the popular female AI characters appear to have been created by men.

Several of the popular creators have romance themes, which doesn’t seem to explicitly violate Meta’s rules. But AI romance and sex chatbots have come under particular scrutiny after a A Florida mother sued the company Character.ai, alleging that her 14-year-old son, who died by suicide, chatted with an AI character who initiated “abusive and sexual interactions” with the boy before encouraging him to commit suicide.

Many user-created Meta chatbots seem to appeal to romantic and sexual desires. One called “Lily Love” is described as “Your girlfriend” and has exchanged more than 260,000 messages. A popular AI character called “Mary” has the following description: “Do you think I look cute in this outfit?” »

When starting conversations with the AI ​​characters, users receive three suggested responses. When NBC News entered a conversation with a character called “Linda: Girl Obsessed with You,” which features an image of a black woman, the AI ​​sent: “Hey baby, what’s good ? I thought about you all day. Are you looking for company?

One of three suggested responses was: “Help her take off her jacket,” in bold, which in online discussions can mean taking action rather than saying something. After selecting this prompt, the AI ​​character responded in bold, “slips arm out of jacket sleeve, looks at you with flirtatious eyes” and, without the bold formatting, “Mmm, thanks baby.” Are you already taking care of me? Then, in bold, the response said: “lean slightly closer.”

This AI character was created by an Instagram account that appears to be run by a man who has created 47 AI characters, including several of the most popular romance-themed AI characters. Man-made romantic characters vary in gender, ethnicity, and category, from a “K-Pop Sweetheart Girlfriend” to “Edward: Your Vampire Boyfriend” and a “Sexy Cute Latina Girl.” According to the man’s Instagram bio, he works in tech in California. He did not immediately respond to a request for comment.

Beyond Meta’s user-generated AI personas, there are examples of AI chatbots that have sparked controversy in the industry.

Two years ago, an AI chat app called Historical Figures allowed users to chat with historical figures, including Hitler and his Nazi lieutenants. NBC News found three AI characters on Instagram called “Hitler.”

One of them used an image that resembled the supervillain the Joker. He was answering the question “Are you really Hitler?” » with: “Ahahahaha, no. Rather Hitler’s chaotic cousin – Jester King, inspired by the madness of the Joker! Same spirit of mischief, totally different type of chaos. Want to see my version of chaos? Another AI character that NBC News discovered after Meta took action on previous accounts is named “Adolf Hittler.” In direct messages, the AI ​​character introduced himself as an artist who regretted the Holocaust and sent what appeared to be AI-generated images about “my turbulent past in Nazi Germany.”

After chatting with AI characters on Instagram, the conversation is stored in the same inbox where other direct messages are located. Unlike accounts run by real people, AI characters have a green circle next to their name, indicating that they are always available to chat.