Robert De Niro said no green screen. No face dots. How ‘The Irishman’s’ de-aging changes Hollywood

Few actors have undergone more dramatic physical transformations onscreen than Robert De Niro. For 1980’s “Raging Bull,” the actor, then in his mid-30s, famously put on 20 pounds of muscle to play boxing champ Jake LaMotta; then, over the course of several weeks, he packed on 60 pounds of fat to play the fighter as a bloated, washed-up older man. The extreme gain temporarily wrecked his health, but for his trouble he won his second Oscar — and the awed respect of every actor on the planet.

But not even the protean De Niro can reverse the relentless march of time.

To portray the hitman Frank Sheeran in Martin Scorsese’s gangster epic “The Irishman” — a role that spans more than five decades, from Sheeran’s service in World War II to his death in 2003 — the 76-year-old De Niro was put into a time machine unlike any seen in film history. The effort involved years of collaborative work from some of the industry’s top visual-effects artists, costume designers, makeup artists and even sound editors. That work may not only yield a slew of Oscar nominations but, in a very real sense, change the face of acting forever.

The road to the film’s cinematic fountain of youth began in November 2015, when Scorsese had dinner one night in Taiwan with Industrial Light & Magic visual effects supervisor Pablo Helman, with whom he was working on the drama “Silence.” Having struggled for years to crack “The Irishman,” Scorsese asked Helman how CGI might be used to help an actor like De Niro play the same character at several stages of his life.

Movies like “The Curious Case of Benjamin Button,” “Tron: Legacy” and “Star Wars: Rogue One” had used this type of digital plastic surgery, with varying degrees of success. But Helman knew that, to push it to the limits that “The Irishman” required would necessitate the development of completely new technology.

For years, visual effects artists have been refining the ability to “de-age” actors by digitally mapping their performances with dots placed on their faces, often capturing the data with head-mounted cameras and then manipulating the performance using advanced software. Skilled makeup work, combined with tracking dots and the use of younger actors filmed in key scenes for visual effects artists to use as a sort of digital paintbox in postproduction, is an approach used to create younger versions of Kurt Russell in “Guardians of the Galaxy Vol. 2” and Samuel L. Jackson in “Captain Marvel.”

But for “The Irishman,” Helman and his team needed to figure out a way to de-age De Niro, as well as costars Al Pacino (who plays union leader Jimmy Hoffa) and Joe Pesci (who plays crime boss Russell Bufalino) without resorting to such elaborate and obtrusive performance-capture tech.

“When I first met Bob De Niro, he said, ‘There’s no way we’re going to wear markers on our face or helmets with little cameras in front of us or gray pajamas,’ ” Helman says. “He said, ‘We’re going to be on set with each other, having a conversation in the moment, and you’re going to have to come up with the technology that allows us to do that.”

Helman brought the idea to his mentor, ILM creative director and eight-time Oscar winner Dennis Muren. “I put the script in front of him and I said, ‘We have the incredible opportunity to develop new software and to further filmmaking — what do you think?’ ” Helman recalls. “He said, ‘Risky.’ I said, ‘Do you remember how you felt when you did “Jurassic Park”? Didn’t you feel that was risky?’ And he said, ‘Yeah, you’re right. We should do this.’ ”

After creating a successful proof-of-concept by inserting the 70-something De Niro into a scene from 1990’s “Goodfellas,” Helman and his team spent two years developing a new type of camera rig that would enable Scorsese to shoot the actors on a real set under whatever lighting conditions the scene called for — no motion-capture suits or green screen required. Weighing 64 pounds, the rig had a standard camera flanked by two infared cameras that could capture the volumetric information that would normally have been picked up with the sorts of tracking dots that Scorsese and his cast refused to use.

Then, for Helman and a small army of visual effects artists, the real work began. To bring Sheeran and the other characters back in time was not simply a matter of digitally smoothing out wrinkles. The ILM artists team created younger likenesses for each age the actors would be depicting, then — using newly developed software — “re-targeted” their performances to those younger versions.

It was delicate, painstaking work, as much art as science, requiring a deep understanding of how the subtlest emotions are conveyed down to the way vibrations move through the face and neck when speaking. The difference between, say, a smile and a wince could be a matter of just a handful of pixels.

“Performances are a complex way of communicating, and we were not going to serve the story if we just painted on top of them,” Helman says. “For us to transfer those performances to the younger selves, we had to understand how Robert De Niro performs a concerned look or a happy expression or a guarded one. There’s a certain thing that makes us who we are, that makes De Niro go from a smile to a frown in a specific way that will immediately trigger that thing in an audience where they go, ‘That’s Robert De Niro.’ We were trying to get the heart of that behavioral likeness.”

CGI was used to de-age De Niro to his 20s, 30s and 40s. To take him back to his 50s — and forward to his 80s — makeup artist Carla White and her team stepped in, using the tools of their own trade.

“To bring him to his mid-50s, I used some prosthetics under the eyes to cover his own eye bags,” says White, who has worked with De Niro for 15 years. “To get rid of the aged neck that he has in the front, I pulled the neck back and had prosthetics to cover that. Then we used coloring and foundation. It was a lot for him to go through — the longest he had to sit in the chair was probably three hours. But he was a trooper.”

To further depict Sheeran’s transformation over the decades, costume designers Sandy Powell and Christopher Peterson developed scores of distinctive looks for Sheeran — 102 wardrobe changes in all — for each phase of his life. “We were tracking the way things changed through the decades with menswear,” Peterson says, “but we were also tracking character at the same time; Frank’s ascendance from a working-class teamster to one of the soldiers in the [crime] family and working his way up.”

“You tell the passage of time with clothing,” says Powell, who has earned 14 Oscar nominations, most recently for last year’s “The Favourite,” and three wins. “The gangsters in this film were a lot less flashy and ostentatious than in other movies Marty has done. That was the main brief, especially for Frank Sheeran, who has to be under the radar. A lot of it for him was about blending in and not standing out too much.”

Even sound came into play. Oscar-winning sound mixer Tom Fleischman experimented extensively to figure out how to de-age De Niro’s voice. “It had to be very subtle,” Fleischman says. “I tried adjusting the pitch of his voice in ‘The Irishman’ to a couple of scenes in [De Niro’s 1982 film] ‘The King of Comedy,’ but it sounded a little bit like he was on helium. What [sound editor] Phil Stockton ended up doing eventually was going in and editing out a lot of the breaths and the grunts and guttural things that De Niro does with his voice, particularly as he’s gotten older. Then we just did a subtle pitch change. That worked very well.”

As software improves and computing power increases, you can expect Hollywood to continue to push the envelope of de-aging. Earlier this year, with the sci-fi action film “Gemini Man,” director Ang Lee and his team used motion capture and other techniques to create a fully digital clone of 50-year-old Will Smith as he looked at 23. In what some may argue is a bridge too far, visual effects artists have even reached beyond the grave; the late actor Peter Cushing was digitally revived for 2016’s “Star Wars: Rogue One,” and outtakes from Carrie Fisher’s performance in “The Last Jedi” before her 2016 death were used as building blocks for “The Rise of Skywalker,” now in theaters. Meanwhile, James Dean, who died in 1955, is slated to be posthumously cast, via CGI, in an upcoming drama called “Finding Jack,” news that was widely greeted with dismay.

But all the digital technology in the world will be for naught if it fails to preserve the analog soul of a performance. With “The Irishman,” what Helman is most proud of is that his work enabled Scorsese and his cast to shoot the film largely as they would have in the old days. For all of the countless hours of work that went into de-aging De Niro, Pesci and Pacino, he says, at the heart of it, the performances belong entirely to the actors.

“The achievement here is allowing the actors to do what they do without the technology being in the middle of it, working in that organic way with the technology away from them,” he says. “I think it’s just going to keep going and we’re just going to keep trying to make this better and better. I just can’t wait for actors to look at this and say, ‘I don’t have to wear markers anymore. I don’t have to spend two hours in makeup. I can do what I was hired to do, which is perform.’ ”

In that spirit, Helman hopes that when you watch “The Irishman,” you don’t try to figure out how the magic trick was done. Don’t think about the 1,750 visual effects shots. Don’t contemplate the hours that De Niro spent in wardrobe fittings and makeup sessions. “The point of the whole thing,” Helman says, “is you shouldn’t be sitting there thinking about how we did it.”