NYC Mayor Casually Announces He's Deepfaking Himself, Experts Horrified

New York City Mayor Eric Adams has been making audio deepfakes of himself to speak to New Yorkers in different languages.

At a press conference Monday that was meant to brief reporters on a recent “chatbot forum” at City Hall, Adams let slip that his office has been robo-calling New Yorkers with an algorithmically-generated version of his voice speaking in different languages that the mayor does not speak. 

The purpose of the Monday press conference was to announce the city’s impending use of “AI” voice chatbots that will speak to New Yorkers in their language of choice when they call 311 or other services, but the mayor casually mentioned that the city has already been deploying the technology for other purposes, and that New Yorkers are already being misled about the range of languages he speaks.

“We have started doing robocalls with my voice in many different languages. People stop me on the street all the time and say I didn’t know you speak Mandarin,” Adams said at the news conference. 

“The robocalls that we’re using, we’re using different languages to speak to the diversity of New Yorkers,” he said.

The city said in an email to Motherboard it has been sending AI robocalls since March 2022, including 5,000 in Spanish, 250 in Yiddish, 160 in Mandarin, 89 in Cantonese, and 23 in Haitian Creole. It used the calls to tell New Yorkers about hiring events as well as a free concert series.

But the robocalls do not include a clarification that they are artificially generated, leaving New Yorkers with the impression that Mayor Adams speaks Spanish, Mandarin, and Yiddish.

In a statement sent after the press conference Monday, the Surveillance Technology Oversight Project (STOP) condemned the robocalls, calling them deceptive. “This is deeply unethical, especially on the taxpayer’s dime,” executive director Albert Fox Cahn said in a statement. “Using AI to convince New Yorkers that he speaks languages that he doesn’t is deeply Orwellian. Yes, we need announcements in all of New Yorkers’ native languages, but the deep fakes are just a creepy vanity project.” 

STOP is also critical of the city’s other uses of AI, and related technologies, including facial recognition. “The Mayor continues to enable some of the worst AI abuses in the country, all while hiding behind the rhetoric of AI responsibility,” Cahn said in a statement.

One of the chief concerns with the use of algorithmically generated images, video and voices in political campaigns is that they can be misleading, and there have been calls to mandate disclosures so that constituents know what they’re seeing is artificially generated. A bill in the NY state assembly would mandate the disclosure of the use of such technology in political ads. But there’s no comparable legislation or regulation that would mandate disclosures in other forms of government communication, like robocalls or 311 requests. 

The city said the robocalls were developed using Voice Lab by Eleven Labs. The company, founded by former Google and Palantir employees, is a popular solution that allows users to clone their own voices by uploading an audio sample to its website.

In January, Motherboard found that 4chan users had been using a beta version of the service to generate voices of Joe Rogan, Ben Shapiro and Emma Watson making racist and transphobic comments. The company said it would add safeguards after discovering what it called “misuse cases. And in February, a Motherboard reporter used the service to bypass voice authentication and break into his own bank account. The company did not return a request for comment at that time.

Mayor Adams has touted what he says is the “ethical” use of AI, and he has made an effort to brand himself as someone who embraces buzzy technologies. 

“We are not running away from AI, we are going to properly govern how we use AI in a responsible way,” Adams said at the top of the press conference.  “You can use or abuse anything, and if we stay away from moving forward because we’re afraid someone is going to abuse it, you won’t get anything done.”