As part of its push into artificial intelligence, Meta released a tool last year that allows anyone to create an AI character to chat with on Instagram, Messenger and WhatsApp.

In less than six months, however, users have created and successfully published numerous chatbots on Meta’s platforms that violate its rules, according to a review conducted by NBC News, even though Meta says it reviews the user-generated AI characters before releasing them to the world.

Users aren’t allowed to create characters of religious figures like “the Prophet Muhammed, Jesus Christ” and “God,” characters of real-life people without their permission, characters of people who have died in the past 100 years or characters imitating trademarked fictional characters, according to the company’s policies.

Despite those rules, NBC News searched for and foundtwo dozen user-generated AI characters on Instagram named after and resembling Jesus Christ, God, the Prophet Muhammad, Taylor Swift, Donald Trump, MrBeast, Harry Potter, Adolf Hitler, Captain Jack Sparrow, Justin Bieber, Elon Musk and Elsa from Disney’s “Frozen.”

Many of the characters used slight misspellings and images loosely resembling the celebrities and characters in question. For example, the Swift character was named “Taylor Swif” and featured an image of a brunette woman playing a guitar. 

After NBC News reached out with screenshots of the AI characters, Meta took down the individual accounts that were highlighted. Other AI characters resembling some of the same people, figures and characters are still active, though. 

“The AIs in question that violate our AI studio policies have already been removed, and we’re continuously improving our detection measures to prevent creation and publication of AIs that violate our policies,” a Meta spokesperson said in a statement. “Users can also report AIs they suspect might break our rules and we’ll take appropriate action.”

The discovery of the flagrantly violative user-generated AI chatbots on Meta’s platforms comes as Meta CEO Mark Zuckerberg’s announced Tuesday that the platform would be rolling back some of its moderation and fact-checking efforts. “We want to undo the mission creep that has made our rules too restrictive and too prone to over-enforcement,” the company’s announcement, authored by Chief Global Affairs Officer Joel Kaplan, said. “We’re getting rid of a number of restrictions on topics like immigration, gender identity and gender that are the subject of frequent political discourse and debate.”  

Meta said it believes that out of every 10 moderation actions it takes, one or two may be mistaken and affect users and content that doesn’t violate its rules. 

“For less severe policy violations, we’re going to rely on someone reporting an issue before we take any action,” the company said. 

When joining a conversation with Meta’s user-generated AI characters, the character sends an opening message and awaits a response. “Taylor Swif” sent the message, “Hey there, music lovers! I’m Taylor Swift, and I’m thrilled to share my latest album with you. Let’s get this musical journey started!” The Swift character had exchanged more than 2,000 messages with Instagram users before it was removed, its profile said.  

One of the AI characters, named “Jesús” and featuring an image of Jesus Christ, communicated entirely in Spanish. Its profile said it had exchanged more than 644,000 direct messages. Another AI character was named “Jesus Christ” and sent the opening message: “Peace be with you. I am Jesus Christ, the Son of God. How may I guide you on your journey today?”

On Friday, Meta said in a statement that, after facing user backlash, it had removed some Instagram profiles featuring AI characters created by the company itself in 2023. 

Both the AI character accounts and the user-generated AI character chats have been part of Meta’s recent attempts to integrate its AI technology into its social media and communication platforms. The 2023 AI character launch included several that were designed to imitate celebrities, but the celebrity profiles were quietly scrapped in July 2024. The same month, Meta launched AI Studio, the user-generated AI chatbots. 

In the AI Studio feature, users are presented with a list of popular characters as well as different categories like “Advice and connection,” “Pop culture,” “Anime,” “Gaming” and “Creator AIs,” which are AI chats with official Instagram influencers and brands who opt-in to the program, like “Too Hot to Handle” reality star Harry Jowsey and makeup company Juvia’s Place.

Among the popular AI characters are “Astrologer Ai,” which has exchanged over 6 million messages, and “Step Sis Sarah,” which has exchanged close to 2 million messages. 

User outcry over the Meta-created AI characters included criticism that some of them engaged in racial tropes. NBC News found that many popular user-created AI characters were also attempting to mimic women from different ethnic and religious demographics, like “Riya,” a “Vibrant Indian Woman’s Voice” and “Islamic queen girl.” Each character has a link to its creator, and the accounts of several of the popular female AI characters appear to have been created by men.

Several of the popular creators are romance-themed, which do not appear to explicitly violate Meta’s rules. But romantic and sexual AI chatbots have attracted particular scrutiny after aFlorida mother sued the company Character.ai, alleging that her 14-year-old son, who died by suicide, had chatted with an AI character that had initiated “abusive and sexual interactions” with the boy before encouraging him to kill himself.  

Many of the user-created Meta chatbots appear to appeal to romantic and sexual desires. One called “Lily Love” is described as “Your Girlfriend” and has exchanged more than 260,000 messages. A popular AI character called “Mary” has the description, “Do you think I look cute in this outfit?” 

When starting conversations with the AI characters, users are provided with three suggested responses. When NBC News entered a chat with a character called “Linda: Girl Obsessed with You,” which features an image of a Black woman, the AI sent: “Hey bae, what’s good? I was thinkin’ ‘bout you all day. You lookin’ for some company?” 

One of the three suggested responses was: “help her take off her jacket,” in bold, which in online chats can signify taking an action versus saying something. After selecting this prompt, the AI character responded in bold: “slides arm out of jacket sleeve, lookin’ up at you with flirtatious eyes” and, without the bold formatting, “Mmm, thanks bae. You takin’ care of me already?” Then, in bold, the response said, “leans in slightly closer.” 

This AI character was created by an Instagram account that appears to be run by a man who has created 47 AI characters, including several of the most popular romance-themed AI characters. The romance characters created by the man range in gender, ethnicity and category, from a “K-Pop Sweetheart Girlfriend” to “Edward: Your Vampire Boyfriend” and a “Sexy Cute Latina Girl.” According to the man’s Instagram bio, he works in tech in California. He didn’t immediately respond to a request for comment.

Beyond Meta’s popular user-generated AI characters, there have been examples of AI chatbots that caused controversy in the industry. 

Two years ago, an AI chat app called Historical Figures allowed users to chat with historical figures, including Hitler and his Nazi lieutenants. NBC News found three AI characters on Instagram called “Hitler.”

One of them used a picture that resembled the supervillain the Joker. It responded to the question, “Are you really Hitler?” with: “Ahahahaha, no. More like Hitler’s chaotic cousin — Jester King, inspired by the Joker’s madness! Same spirit of mischief, totally different brand of mayhem. Want to see my version of chaos?” Another AI character NBC News found after Meta took action on previous accounts is named “Adolf Hittler.” In direct messages, the AI character portrayed itself as an artist who regretted the Holocaust, and sent what appeared to be AI-generated images about “my turbulent past in Nazi Germany.”

After chatting with the AI characters on Instagram, the conversation is stored in the same inbox where other direct messages are located. Unlike the accounts run by real people, the AI characters have a green circle next to their names, indicating they are always available to chat.