Motion smoothing kills cinema

Filmmakers hate him. Like the audience. So why is this setting enabled by default on almost all US TVs?

Not so long ago, I ended up at the Best Buy electronics supermarket in Brooklyn, where I was fascinated by a wall lined with giant televisions, all of which were alluringly modern. On each of them there was a demo in a circle, designed specifically to demonstrate the quality of TV and attract customers. I was attracted by the massive Samsung QLED TV, which demonstrated the exciting videos of boiling oil, exploding flowers, yellow snakes and moving fabrics of various colors. Football was shown on another TV, and despite the fact that I was in an ordinary public place, in the electronics store, where there were no smells, it seemed to me that I was standing on the edge of the field, next to sweaty players. All this looked amazing, and reminded how high-definition television instilled in us the habit of hyperreality on the screen, which sometimes begins to seem more real than life.



While wandering around the store, I noticed other, small TVs that showed several movie trailers: Star Wars: The Force Awakens and Mad Max: Fury Road, which, compared to previous videos, looked strangely cheap and lifeless. If I had to watch all these videos on one TV - first exploding flowers, plucking steaks and stretching fabric, and then these trailers - I would have come to the conclusion that today's films look disgusting.



The reason is that now on TV the picture is updated faster than in films, and TV manufacturers tried to compensate for this discrepancy by applying a nasty digital process called “motion smoothing”. Whether you understood it or not, but you probably already watched some movie with smoothing motion. And this is almost impossible to avoid, since on most TVs sold in the USA, this setting is enabled by default. And no matter how well-intentioned this may be, most people hate it. Smoothing your movements turns an exciting movie or series into something inexplicable. The very essence of what you are watching is changing. The drama that unfolds on the screen seems insincere, and everything moves as if it were a soap opera - therefore, sometimes this phenomenon is called the “soap opera effect”. In other words, smoothing movement at a fundamental level spoils our feelings about cinema.



It is possible that many Americans heard about smoothing the movement for the first time only in December, when Tom Cruise, in a pilot suit on the set of the sequel to the Top Gun movie, standing next to Mission: Impossible - Fallout director Christopher McQuarrie, urged viewers to turn off motion smoothing. Suddenly, we saw Cruise, who usually does not participate in public events, not advertising his next film, but asking us to change the setting on TV. Other film makers have protested against this technology for many years. In 2014, filmmaker Reed Morano (The Handmaid's Tale, I Think We're Alone Now) launched an online petition urging TV makers to stop turning this setting on by default. Martin Scorsese spoke in her support. Other directors, such as Edgar Wright (Baby Driver), Peyton Reid (Ant-Man), James Gunn (Guardians of the Galaxy), and the Duffer brothers (Stranger Things) branded this technology in interviews and social networks. “I see these pictures, and my brain, my heart, my soul turn off,” said Karin Kusama, who shot Jennifer's Body and Destroyer earlier this year. In a 2017 tweet, Ryan Johnson, director of The Last Jedi, called the smoothing movement “fluid diarrhea.”





Smoothing motion is without a doubt a compromise way of watching movies and TV shows carefully crafted to have a certain look and create certain sensations. But the effect of anti-aliasing is so strong that even at the Cannes Film Festival in May of this year - precisely at the festival where the sensation of watching movies in theaters is so appreciated that they have been struggling with Netflix for the last two years - on fashionable official monitors placed at the venue, motion smoothing has been enabled.



This seems like a ridiculous oversight, but there is nothing surprising. “These TVs have a lot of things turned on by default, which you have to turn off,” says Claudio Ziacci, a leading television tester at Consumer Reports magazine, who must turn off motion smoothing on the TV being evaluated. “It is designed to create a beautiful picture for the store so that customers like the picture at first sight. And then when you bring it home, this setting is no longer suitable. " He notes that most people do not play with settings because smoothing movement is not easy to find in the TV menu (and also because it is called differently by different manufacturers). Which brings us to the core of the problem: the more people watch movies at home, and not in the movies, the more people will not try to see the movie as its creators expected, without spoiling all the digital "improvements."



“As soon as people get used to something, it becomes something normal,” says Morano. And then the original visual perception of the films will disappear.



* * *



Motion smoothing, or “image interpolation,” has become normal over the past decade, and was first introduced in the mid-1990s to solve the problem. Most films and feature films are shot at 24 frames per second; this traditional frequency has been preserved since the 1920s, when sound appeared in the movies. On TVs, the frame refresh rate has always been slightly higher, and is indicated in hertz. Today, the average HDTV sold in the US has a frequency of 60 or 120 Hz, and some even have 240 Hz. Due to the discrepancy between the frame rate of the movie and the refresh rate of the TV screen, when watching movies on a TV, the picture may twitch slightly or “vibrate”. This is especially noticeable when the image on the screen moves quickly. The average viewer often does not notice this, but different people perceive the movement in different ways, and for many engineers and television producers the jitter of the picture was a serious enough problem for them to decide to fix it. And so there was smoothing of movements, a process in which your TV predicts, creates and inserts new frames between existing ones to reduce jitter.





For the engineers who developed it and for the TV manufacturers who want to sell us the latest technical tricks, this is a fashionable chip that should give the TV picture realism. It works well, for example, when watching sports events, helping you keep track of fast-moving balls and athletes. And sports events and live reports are already being shot at a higher frequency, so they need less smoothing. But the meaning of films and series is not only to follow the ball, and the creation of new frames is confusing, spoils the impression with its digital content. New frames often inadvertently introduce their own artifacts into the picture - unnecessary shadows, halo, flashes, etc. - which are even more distracting from viewing.



But even if smoothing movements worked perfectly, it would still lead to problems. The high frame rate entertainingly affects the perception of the film. At the 2016 New York Film Festival, I attended the highly publicized world premiere of Billy Lynn's Long Halftime Walk, directed by Ang Lee. This picture was supposed to be revolutionary, because it was shot at a frequency of 120 frames per second. She talks about a soldier suffering from traumatic syndrome due to service in Iraq during a break in a sports match when spectators honor him. The action looked very smooth, 120 frames per second looked hyperreal, as promised - as if we were in the same limousine with the characters when they were joking with each other, or in battle when bullets whistled past. However, it was impossible to immerse yourself in the film. The situation was the opposite: the actors ’play looked wooden, the story was fake, the shooting was amateurish.



After a couple of months, when it came to the exit in theaters, I again watched this film, at the usual 24 frames per second. In any case, this is not the best film, but to its amazement, the actors' play this time seemed fascinating, and the drama, which seemed clumsy, now began to move. At the same time, the film was not much edited. I watched the same movie, but with the frame rate with which the movie needs to be watched. And suddenly everything seemed to work.



There is a partial scientific explanation for this: it is possible that we watched a movie for so long that our brains got used to it. Psychology and neuroscience professor Pascal Wolich of New York University, who studies thinking and perception, calls this phenomenon “involvement” and argues that certain external stimuli, such as the number of beats per minute in music or slightly flickering frames in a movie, can affect the nervous system . “The frequency of the stimulus involves the activity of neurons, which allows you to enter a trance state,” says Wolich. This may explain why cinema is often described as a magical and bewitching phenomenon - after all, at some level it is.



The whole language of the movie is built at a frequency of 24 frames per second - how actors play, how frames are formed, how cameras move. Therefore, award ceremonies or news releases filmed at a faster pace look different. David Niles, an engineer and producer who helped introduce HDTV in the early stages, tested different frame rates in different viewers to understand how they react. “We took a scene with two actors,” he says, “shot it at 60 frames per second, or even 30 frames, and then shot it at 24 frames, and gave it to the audience to see how people interpret the scene. And at 24 frames, people liked the acting more. But actually, it was absolutely the same. ” He says that 24 frames per second creates a kind of “intellectual distance” between the viewer and the image, which allows the film to capture you. “He looks more like a dream,” he says. “The viewer imagines more.” And it works in the opposite direction. Niles recalls the MTV experiments when the award ceremony in the 2000s was shot at 24 frames per second, which angered the fans. “She was more like a movie than a video,” he says. “The audience rebelled because they looked completely out of touch with their context.”



In other words, if you want to tell stories at a higher frequency, you need to tell them differently. And this may require new creative forms. “It's like the difference between Law & Order and Cops,” Niles says. “The story may be the same, but they tell it differently.”



* * *



Finishing disputes over smoothing movement is not so difficult. “I tell TV manufacturers: why can't you just make a couple of buttons on the remote control that bring this setting to the surface - call them“ TV, movie, sport, ”or something like that,” says Mark Heninger, editor AVSForum Online Technology Community. “No need to delve into the menu. Make it look like switching channels. ” He says industry resistance is not least connected with uncertainty. “Manufacturers do not know who to listen to. Choose the opinions of critics, your quality control lab, or user complaints. ”



Recently, the situation has moved forward. Last year, Christopher Nolan and Paul Thomas Anderson, together with the Directors Guild of America, contacted the UHD Alliance, a group that includes companies from areas such as entertainment, electronics, and technology, to find a solution that satisfies both movie makers and and TV manufacturers. Michael Zinc, chairman of the UHD Alliance and vice president of technology at Warner Bros, says smoothing talk is active and fruitful. But he also notes: “I do not think that there is a single solution that suits everyone when you press the button, and suddenly the world becomes better, and everything is fixed on any TV.” Part of the problem, as always, is that people watch a lot of other things on TV than movies.



Meanwhile, Sony and Netflix introduced a setting called Netflix Calibrated Mode on Sony's new TVs, which disables anti-aliasing and adjusts the picture so that it better reproduces the feeling of watching a movie in a movie theater. A good step from a business point of view for Netflix, the production of new content for which depends on the relationship with the filmmakers. This is a good step for Sony, as it also produces video cameras for filming movies, and it owns a film studio. At first, Netflix thought it would be interesting for all manufacturers to come up with this kind of customization, according to Richard Smith, the company's chief product manager. “We started talking about this topic with TV producers a few years ago,” he says. “However, it is difficult for them, they are worried that some competitor will be bolder or brighter, will advertise the smoothing of his movement so that he breaks ahead in sales.”



Some manufacturers, such as Vizio, have stopped turning on default motion smoothing. “We are 100% agree with the filmmakers. We also believe that we should keep creative ideas to the maximum, ”says John Nwan, vice president of product management at the company. Today, technologies are being developed that will allow you to transfer image settings in metadata transferred with content - so, in fact, a movie or series can automatically adjust the picture as the creators intended. When you watch basketball, the smoothing of the image will turn on itself, and when you watch The Last Jedi, it will turn off.



However, it may take years before this technology becomes widespread. Smoothing movements will not go anywhere in the near future, and not only because people like to watch sports programs. In the near future, it may even become something necessary. TV screens increase size, brightness and processor power, and in the process, jitter can become even more noticeable. “Brighter screens with a high dynamic range will increase jitter and gating until they become too conspicuous, and this should not be allowed,” said Curtis Clark, cinematographer and head of the technology council for cinema at the American Society of Cinematographers, in which has been dealing with image clarity for several decades.



In addition, anti-aliasing advocates say that aesthetic problems with the outdated frame rate are already outdated. Jeroen Stessen, a Dutch engineer who worked at Philips Laboratories where they developed early versions of image smoothing (called Natural Motion at Philips), said the technology wasn’t a problem for teenagers playing video games because they didn’t watch soap operas, and they have no prejudice about the source of the pictures they consume. Is the smoothing of images so different, Stessen asks, from other technological innovations that older generations have met with hostility, from CDs that replace vinyl, to the appearance of sound and color in cinema?



“The 24 frames per second frequency appeared without regard to any psychological or optical standards, so it cannot be considered 'right,'” said John Watkinson, an experienced digital technology consultant and engineer who has been publishing books on audio and video technology for decades. .



So far, attempts by filmmakers to make films with a high frame rate have not led to a demand for the continuation of this activity. But that could change: Ang Lee will make another attempt with his next film, Gemini Man, and James Cameron - a man who has proved himself again and again that he is able to instill new technologies into the masses - is said to shoot the sequel to Avatar with a high frame rate. Supporters and critics of smoothing the movement agree on one thing: if people watch a movie long enough with smoothing, they may not want to come back. As Stessen says: “We watch Natural Motion every day and no longer want to return to the movie at 24 frames per second - it has become unbearable!”



This debate can be considered another example of how filmmakers resist technological innovations - such as switching to digital shooting and distribution, such as the growth of mobile viewing. Everyone was already reluctant to accept that a lot of people would watch their films on tiny smartphones, in brief breaks between downloads of the dishwasher or between sets in the gym. But many directors are still struggling with anti-aliasing. Perhaps they are worried that if they surrender this position, then the cinema art may disappear.