Effort to stem online extremism accidentally pushed people toward an anarchist

The idea began as a bait-and-switch: Give people searching online for terms like “join Oath Keepers” or “bomb instructions” some content that seemed to fit their request but instead offered an alternative to extremism.

The method, developed by Moonshot CVE, is called the “Redirect.” Through partnerships with Google as well as the Anti-Defamation League, the British firm’s goal is to dissuade users from pursuing conspiracies and violent rhetoric by luring them with advertisements to other sites.

It’s just one of dozens of ideas that tech companies are experimenting with as — in the wake of the Jan. 6 Capitol insurrection — they come under increasing pressure to prevent the spread of extremism on their platforms. In an effort to blunt federal regulation, California’s internet giants repeatedly have insisted they are working to solve the problem. The flow of dangerous content continues to proliferate.

And in an internet figure known as Beau of the Fifth Column, Moonshot thought it had found the perfect person to redirect online users to.

Broadcasting under a bare lightbulb from a shed packed with camping gear, Beau appeared as a Southern gun owner with progressive views. He had a calming demeanor, unkempt beard, military aura and huge YouTube following.

But Beau, whose real name is Justin King, also had a history of supporting anarchism and making statements that to many seem anti-Semitic. He is a felon.

The fiasco over Moonshot’s redirect experiment came to a head last week in Washington as federal legislators grilled the chief executive officers of Twitter, Facebook and Google about extremism and misinformation.

“They sent people who were already looking for violence to a convicted felon with anarchist and anti-Semitic views,” Rep. Morgan Griffith (R-Va.) said to Sundar Pichai, chief executive of Google’s parent company Alphabet, during a House hearing.

“It raises one of the key questions about moderating online platforms,” Griffith said afterward. “Who is vetting the vetters? We continue to need more transparency and accountability.”

Critics say the underlying issue is that internet companies don’t want regulators to target their secret algorithms, which drive billions in profit but also can guide online users to extremist sites. So they seek to use a patchwork of interventions that might not solve a problem and, in some cases, can create another one.

The stakes are high for the social media giants.

Several proposals in Congress would punish the firms for algorithms that promote extremism, including the Protecting Americans from Dangerous Algorithms Act championed by Rep. Anna G. Eshoo (D-Menlo Park). California legislators also have introduced Assembly Bill 587, which would require companies to file public reports on their efforts to stop extremism.

Pichai said during the House hearing that he was “disappointed” to hear of the problems with the redirect experiment and that “we will definitely look into it.” Google referred further requests for comment to Moonshot. A spokesman for the ADL said no one was available due to the Passover holiday.

The story of Beau’s intersection with Silicon Valley began with the 2020 election season.

Moonshot and the ADL teamed up last fall to dissuade internet users from pursuing conspiracies and violent rhetoric by luring them with an advertisement to content that included his video. Moonshot succeeded in creating 1,330 engagements with its target audience, intercepting thousands of searches across the U.S., including more than 600 people who may have viewed Beau’s clip after searching for content about armed groups.

The video focused on how armed Black men are treated with undue suspicion by law enforcement and stressed the need for people of all races to work together for change, according to the company’s report.

But in February, Moonshot received an anonymous complaint, alleging that King, the independent journalist who plays the character, had been involved in violent far-left rhetoric.

King said in an interview with The Times that he is not anti-Semitic. He describes himself as being “very anti-authoritarian,” and said that he considers himself politically left of the Democratic Party, but not an extremist. He views anarchy as an ideal, but he is not the type of anarchist to be “throwing rocks at cops,” he said.

“I would love a stateless society where everyone gets along,” said King, whose real accent is less pronounced than that of his Beau persona.

But in a post on his Fifth Column website, which began requiring a password on Saturday, King wrote in 2017 that it is “imperative that we are capable of violence at a greater level than those who would seek to place us back under the yoke of tyranny.”

In another article penned by King, he explained the reasons some feel violence against law enforcement officers is warranted. Though King said he has mellowed with age — he turned 40 recently — Beau’s very moniker nods to his ideology: A fifth column is a double-agent who seeks to sabotage a group or country from within.

In 2007, federal authorities charged him and three Russian co-conspirators in a scheme that brought mostly young Eastern European women to the Florida panhandle illegally to work as maids in local resorts. King was sentenced to 41 months in prison, and he and his accomplices were ordered to collectively forfeit $1 million in assets. It is, he said, a period in his life that has shaped his views on the criminal justice system and immigration.

Researchers at the Rutgers University Network Contagion Research Institute found his work on a number of sites they classify as disinformation, including Mint Press News, which the institute said promotes anti-Jewish conspiracy theories and which also posts copy from Russia Today and Sputnik, the Russian state-owned news agency. King said his content was licensed in a way that allowed almost anyone to run it without his consent.

King said that, while he would have given approval with advanced warning, the Moonshot experiment was run without his consent and he was concerned to learn that potential extremists may have been directed his way.

“There could be negative impacts for me to have my face shown to a bunch of people they thought could be violent,” he said. “I would like to be aware of that.”

Some warn that tech companies’ reliance on third-party contractors like Moonshot to solve their radicalization problem deflects scrutiny from their unregulated algorithms.

“It’s a Band-Aid, but I don’t blame Moonshot for that,” said Mia Bloom, an expert on extremism who has worked with the ADL. “A better response would be that the tech companies take responsibility and not have ... these pernicious algorithms.”

Moonshot said the company has been upfront about how its campaign for the ADL got tangled with Beau of the Fifth Column, and added that the Beau incident was an error, and exposed a breakdown in its vetting process.

But Moonshot is not ruling out using King again. Ross Frenett, a co-founder of Moonshot, said relying on formerly incarcerated or reformed extremists has long been a standard practice in deprogramming. It has been used with some success in diverting potential terrorists.

“We are comfortable working with people who have checkered histories,” he said. “But we like to know about it. In this case, our systems missed some of those elements from his past. It highlighted a gap in our systems.”

Alex Goldenberg, the lead intelligence analyst at the Network Contagion Research Institute, said that while the initial decision to include King’s content could be viewed as a mistake, he finds Moonshot’s response troubling — and indicative of a larger problem.

“This entire experiment is an extension of the same lack of transparency that is being demonstrated by many of the social media platforms themselves,” he said.