How U.S. video game companies are building tools for China's surveillance state

Software developers at American gaming companies are getting roped into building the tools of the Chinese social control state.

July 22, 2019 4 AM PT

Last October, software developers at Riot Games in Santa Monica fielded an unusual request. As with other video game makers, Riot’s success depends on its ability to make games that are compulsively playable, like its global hit “League of Legends.” But Tencent, the Chinese tech giant that owns Riot, needed a way to force some of its most enthusiastic customers to play less.

While it has owned a controlling stake in Riot since 2011, Tencent has generally been hands-off when it comes to the company’s products. But facing increasing pressure from Chinese state media and regulators over its role in a supposed epidemic of video game addiction, Tencent needed a way to track how much time individual gamers in China spent playing “League of Legends” — and kick out minors who exceeded two hours per day. If Riot engineers didn’t supply an “anti-addiction system” for “League of Legends,” they might lose access to the Chinese market altogether.

For the record:

10:00 p.m. July 29, 2019This article describes Sesame Credit’s service as a private pilot program for a national social credit system. Ant Financial, the program’s operator and an affiliate of the Chinese tech giant Alibaba, says the program is “in no way contracted to run or affiliated with China’s social credit system.”

Within weeks, an update brought these features to the Chinese version of “League of Legends.”

Over the last year, one game company after another has quietly acceded to Chinese government demands to limit the amount of time young people spend on their games. Chinese players of American hits such as “League of Legends,” “Fortnite” and “World of Warcraft” are having their playtime tracked according to their national ID number. Those younger than 18 face heavy in-game penalties or outright expulsions if they play too long.

Although it’s Chinese policy driving the restrictions, data privacy advocates say that for Americans to participate in the creation of these tools represents the crossing of a concerning new threshold. They view the moves as part of a problematic trend of Western technology firms redesigning their services to create China-friendly versions aligned with the country’s tighter social controls.

“For American companies, it really comes down to deciding whether or not you are willing to participate in this type of surveillance,” said Matt Erickson, executive director of the Digital Privacy Alliance. “If they do choose to take part, it makes these companies not unwitting but full-blown accomplices in the Chinese police state.”

Access to the world’s second-largest market is a powerful incentive, but for some companies, supporting Chinese censorship and social control efforts is not a matter of choice. As Chinese giants buy up American tech companies, from West Hollywood-based gay dating app Grindr to Motorola’s mobile phone business, regulators are raising questions about companies’ autonomy and ability to push back on requests that might violate their ethical principles.

Internal documents from Riot Games obtained by The Times offer a rare glimpse into how the Chinese government exercises influence over companies beyond its borders.

Tencent is the world’s largest game publisher and owns large or controlling stakes in a range of industry-leading developers including “Clash of Clans” maker Supercell and “Fortnite” developer Epic Games. Its self-developed title, “Honor of Kings,” was the world’s highest-grossing mobile game of 2018. The game’s success made it a lightning rod for growing Chinese government concerns of gaming addiction among Chinese youth, prompting Tencent to build its first ID-tracking playtime restriction system and pledge to incorporate similar systems on all of its games in 2019.

A digital presentation circulated via email among developers at Riot’s Santa Monica headquarters called for an “AAS [anti-addiction system] upgrade” for “League of Legends” in China. The presentation, authored in China, framed the request alongside accounts of growing Chinese government criticism of the gaming industry, official media attacks on Tencent, and a stark reminder that “League of Legends” “cannot [be] free from regulation.”

The request specified the need for features tagging teenage players in accordance with “future AAS regulation.” It also asked for the ability to kick certain players from the game at specified times and restrict time-based in-game rewards. The presentation’s author included mock-ups of “anti-addiction warning” pop-ups on “League of Legends,” with messages telling players they had reached their daily gaming limits or were forbidden to play between particular hours (9 p.m. to 8 a.m.).

It did not take developers long to produce the requested features. In a December 2018 post on Chinese social media site QQ, Riot China announced an update to “League of Legends” including most of the changes.

When asked about the U.S.-based staff’s level of involvement with the development of the ID-tracking and playtime restriction systems, Riot Games said in a statement, “Our lead engineers, based in California, are aware of every feature that we create for ‘League of Legends.’ We develop market-specific features collaboratively, with representatives from our engineering teams around the world.”

Tencent referred questions to an outside PR agency, which declined to provide responses.

While such systems are becoming standard in China, Jay Stanley, a policy analyst at the American Civil Liberties Union, said they constitute a granular privacy invasion that runs counter to American norms.

“American companies are part of American society and should be institutions that we can trust, abiding by American values,” Stanley said. “If these companies are running overseas and participating in authoritarian regimes, then it’s a real problem.”

Stanley acknowledged that it would be difficult for businesses like Riot to refuse to implement these systems at the cost of being locked out of the $36.5-billion Chinese gaming market, but said that failing to do so would help normalize invasive surveillance internationally. “This is going to put a lot of businesses in a bind, but we as a country need to defend American values,” Stanley said.

U.S. tech giants such as Google and Facebook have faced similar difficult choices about whether to create censored versions of their platforms in order to gain access to China. Google left the country in 2010 amid disputes over censored search results and a major hacking incident. Last year’s revelation that the company had secretly begun work on a censor-compliant Chinese search engine, code-named Project Dragonfly, sparked an outcry from Google employees and U.S. politicians. (A Google official told Congress last week that it has terminated the project.) Facebook has reportedly given up on entering China after years of courting Beijing failed to win the company a reprieve from a 2009 ban. Meanwhile, U.S. lawmakers have begun to call for an end to American investor money being channeled toward development of Chinese surveillance systems used in the repression of religious and ethnic minorities.

The anti-addiction regulations referenced in the Riot documents appear to refer to a Chinese Ministry of Education statement released last year. In it, the ministry said it would take steps to control the number of online games available, explore age-based restrictions and limit the amount of time adolescents spend gaming. No specifics or timeline were given, but many major game publishers in China rolled out their own solutions in anticipation of government action.

Tencent started requiring mandatory age and ID authentication on its top-grossing mobile game “Honor of Kings” in 2018, making players provide their Chinese national ID information for verification against police databases. The company has said the checks will be applied to all of its games this year and has been conducting trials of additional verification methods including mandatory facial recognition checks.

The Shenzhen-based firm, which also owns China’s largest social media platform, WeChat, is under intense pressure to address Chinese government concerns around the negative effects of gaming. Tencent’s market value fell by a record-breaking $271 billion last year partly because of a complete freeze on new game approvals from March through December of 2018. (The stock has since rallied as regulators began working through the backlog.) The halt came amid a flurry of official criticism blaming excessive gaming for everything from increasing rates of nearsightedness among youth to potential national security vulnerabilities from mobile gaming addiction among military personnel.

Lisa Cosmas Hanson, founder of Asia-focused gaming market intelligence firm Niko Partners, said that the freeze on game approvals and the resulting backlog have hurt many game companies in China, forcing some smaller operations to shut down. Although approvals have resumed, the process has become more difficult, she said.

“Publishers in China have told me things like that it used to be a 20-page application and now it’s 300,” Hanson said. According to research by her firm, only 75 imported games have been approved so far this year, down from 467 in 2017.

Although real-ID verification systems are relatively new, features designed to limit the amount of time Chinese gamers spend online have been around for more than a decade. Irvine-based Blizzard Entertainment put a three-hour playtime limit on the Chinese server for its massively multiplayer online role-playing game, “World of Warcraft,” in 2006. Gamers got around the restrictions by creating multiple accounts and switching between them, but the system has since been upgraded to require age and identity verification through a national ID.

“Fortnite,” which has held the title of the world’s largest and highest-grossing free-to-play online game for most of the last year, rolled out similar ID verification and playtime tracking restrictions in China earlier this year. Neither “Fortnite” publisher Epic Games, based in Cary, N.C., nor Blizzard Entertainment responded to requests for comment.

Looming over all this is China’s planned rollout of a social credit system, which will assign citizens a score incorporating both standard financial metrics, such as debt repayment, and more personal information such as shopping habits and online behavior. Once a nationwide system debuts in 2020, a citizen with a low score could reportedly face penalties such as losing the right to buy plane and high-speed rail tickets, take out loans or purchase property. Excessive time spent playing video games is one specific behavior that could lower your score with Sesame Credit — one of the system’s main private pilot programs. This applies not only to minors but gamers of all ages.

While there have as yet been no indications of ID-linked tracking systems being used for anything beyond limiting minors’ gaming time, some people remain concerned about the potential for future misuse, particularly because Tencent has already conducted its own short-lived pilot of a social credit system last year.

“There is no right to privacy in China,” said Erickson, of the Digital Privacy Alliance. “Any information collected to make sure kids aren’t playing too many video games will definitely be used by the government and the police for whatever purpose they see fit.”

Jack Poulson, founder of the advocacy group Tech Inquiry, says American tech workers — lacking visibility into executive-level decisions and other divisions of their sprawling conglomerates — aren’t always aware when they are participating in projects that might go against their values. “There’s no real protection in place to ensure that employees have an understanding of what they are helping build,” he said.

Poulson served as a senior scientist at Google’s research and machine intelligence department before publicly resigning last year in the wake of the Project Dragonfly revelations. He founded Tech Inquiry to help tech workers push back on unethical requests. He said the first step to ensuring that staff aren’t corralled into work they find morally objectionable is having a clear understanding of a project’s end goals, but he acknowledged that in Riot’s case the murky distinction between the Chinese public and private sectors makes this easier said than done.

“How do you even get a sense of what the likely uses of this data could be? It could be a government decision, it could be Tencent,” Poulson said. “With something like this, it’s obviously more complicated.”