Clever Geek Handbook

Why 3D Headache / Part 8 Defocus and the Future of 3D

S3D: No pain IS gain









Once upon a time in a distant galaxy Geektimes, a series of articles was published on why 3D movies are so difficult to shoot without a headache from the audience. As often happens, the cycle ended prematurely due to the lack of time for the author. Despite the fact that tons of materials remained lying (sometimes even carefully selected).



In December of this year, the 10th anniversary of the filming of 3D films will be marked by the triumphant success of the movie Avatar, which for the first time in history has raised more than $ 2 billion at the box office. Since then, according to reporters, "3D has died." In fact, of course, this is not so. Compared to the situation 13 years ago, the number of films produced in 3D has grown by an order of magnitude, and the number of halls where 3D can be shown has grown 400 times (!) And continues to grow rapidly with CAGR of about 12-14%. All technologies would have such a “death." Nevertheless, 3D movies obviously have problems. And we have been engaged in their analysis and correction for many years.



I would like to understand what is happening with these films. Why is “3D dead”, and are 3D cinemas actively being built?



One respected Russian stereographer has repeatedly publicly called us pathologists for how we examined the problems of films in detail. It sounds insulting, especially considering that we are actively developing “resuscitation” means. On the other hand, Avicenna, I remember, used to say to patients: “I exist, you and the disease. Whose side you take, he will win. ” Therefore, we will not go against the wisdom of the ages. If the patient said to the morgue, then to the morgue! Medicine is powerless here.









And for those who want to live, all these new 3D cinema halls are being built. As the area grows and will grow even stronger. Contrary to the forecasts of journalists who first inflate and then scourge hype.











Extremely brief summary of previous episodes



As the discerning reader already understood from the title, this is the 8th part of the cycle. Earlier it was:





The material lies still in about 7 parts, but the time in days is short, so how much will work.



And today about sharpness!



In an ideal world, the sharpness of objects on the right and left angles of stereo coincides, but in reality ...



The reasons for the difference in sharpness









In the fifth part of this series, geometric distortions were discussed, including the difference in scale that occurs due to the backlash of the lens. Often, when hovering over a lens with a zoom lens, it “leaves” not only “zoom”, but also sharpness. Moreover, if the difference in scale is corrected relatively easily, then the difference in sharpness is corrected already non-trivially.



The situation is aggravated by the fact that in the cinema a small depth of field is a very often used technique:





An example when flowers are shot with different depth of field. Obviously, in the second case, attention to flowers is attracted much more

A source



The blurred background is both drawing attention to the “gun hanging on the wall" (which is important for the dynamics of the film), and a purely artistic technique that can dramatically improve the aesthetics of the scene. Those who want to dive deeper into the subject can see the depth of field (Depth of the sharply depicted space) or bokeh .





An example where, using depth of field, we identified a specific person. The eye on it stops automatically. Moreover, if the whole frame were sharp, then we would most likely first look at the athlete closest to us

A source



When we watch the usual good old tube two-dimensional movie, we focus on the screen and perceive the image simply as a two-dimensional picture, which can be blurry somewhere. In the case of stereo, an interesting effect is possible - a person sees a three-dimensional scene and tries to “turn his eyes” to a blurred background object. At the same time, conditionally, the brain expects this object to become sharp (at least, in the ordinary world it always happened before), and the background remains blurred in a 3D film. And some people feel discomfort from this. You can, of course, say - “Watch 3D movies more often, and it will pass” (and this is actually so!). But the harsh reality is that the opposite is likely to happen: if the discomfort is great, a person will not go to 3D anymore and will not train this ability to watch poor 3D .



That is why 3D is usually shot with a greater depth of field and less background blur in general. The same drawing attention to a certain object in a scene in 3D is achieved not due to the difference in sharpness, but due to the fact that the scene is “correctly” built in three-dimensional space, that is, the same effect of attracting attention is achieved in a different way. True, this requires a good and still rare understanding of the three-dimensional layout of the scene at the stage of the script and storyboards.



But this is not so bad. The serious problem is that 2D rental is still dominant, and to shoot two versions of a picture - with the help of attention with sharpness and with the help of attention with depth - is too expensive and difficult. Plenoptic cameras that could allow this to be done “in one take” are still too bulky . As a result, today, in good films, both techniques are used simultaneously, that is, when shooting, the scene is blurred in sharpness (but less than in “normal” 2D) and attention is drawn in depth. This, of course, complicates the production, starting with the preparation of storyboards (which now need to think “three-dimensionally”) and ending directly with the shootings. By the way, it is for this reason that the first films shot by operators in 3D (they were many in 2010–2013) are often unsuccessful in terms of a comfortable stereo experience. Only after receiving feedback and experience do people begin to learn how to work adequately with volume. Moreover, they study with different efficiency, unfortunately.



Let's move on to concrete examples and start with the distant 1954, when this problem was prohibitively frequent, and there was no way to rectify the situation on digital post-production. To make the difference in sharpness clearly visible, fragments of the right and left foreshortenings are given in those places where the difference is most noticeable:









Traditionally, in the corner on the left is the name of the film, timestamp and metric value; in the corner on the right it is shown from which part of the frame this fragment is cut

Source: examples from films and examples of processing hereinafter - the materials of the author



The difference in sharpness can be easily obtained on modern cameras simply because of the backlash of the lens when hovering over. If you take into account that when shooting regularly you have to direct the camera to sharpness, it is clear where such frames come from. And in the worst case, the picture will be like this:









In the left view, the object behind is sharp, and in the right view, the object in the foreground

Source: hand-drawn examples hereinafter - the materials of the author



How this looks with real examples can be understood from this frame:





You must have animation enabled in your browser to see this example.



Take a closer look at the girl’s hair and the jacket of the actor closest to us: you can clearly see that their sharpness changes in phase with the main character in this scene, i.e. on the right side, the elderly gentleman and background are blurred more strongly, and on the left - the actors standing closer to us. Trouble is trouble.



A trained reader can politely notice that such a situation is a fairly common deviation in vision, for example, when one eye goes, conditionally, into myopia, and the other does not (and even into farsightedness). And this problem, indeed, is quite well compensated, as a result of which, for objects located far away, the parts are conditionally taken from one eye, and for close ones from the other, and we successfully see a sharp picture. This is certainly true. It takes several hours without getting up to sit in front of the computer for some shooter while writing a scientific article, as the eyes really “sit down”. Moreover, they often sit unevenly, that is, one eye is stronger than the other. And then this situation usually lasts all evening, until in the morning you wake up with restored vision. So what is the problem? As some professionals say, “the brain easily compensates for this!” The problem is, first of all, that if you strained your eyes and planted your eyesight, then for the whole evening you will have a change in "sharpness" in time and all the time "in one direction" between the eyes. And in real films, unfortunately, your scenes with a difference in sharpness change randomly. Conditionally, then everything is good, good, good, then the right angle blurred, then again good, then left, etc. This is an atypical mode for correcting differences in sharpness for our poor brain, in this mode they have never been trained to work in real life. As a result, after some time, our long-suffering 3D-viewer gets a headache.



And here an interesting effect occurs. With age, the human binocular visual system "gets used" to adjust the difference in sharpness in one direction. And, when the sharpness begins to change in different directions - the system begins to "rebel." As a result, when the grandmother goes with her grandchildren to a 3D film, the grandchildren are delighted, and the grandmother, pursing her lips, is silent and gives herself the vow not to go on this anymore. And this happens quite regularly.



Conclusions:





Examples of frames with a big difference



The examples below were automatically selected from 105 films using VQMT3D, a stereo video analysis tool.





Horror film "Piranha 3D", the sharpness of objects varies as the foreground and background; Typically, the problem is right in the center of the frame in the high-focus area





A major difference in sharpness throughout the frame in the second film of the Piranha 3DD series



In general, horror films are characterized by a greater number of sloppy scenes that will cause a headache. As they say, it will be not only scary, but also painful. Another horror movie:





A noticeable difference in sharpness of the actor in the foreground in the horror film "Jaws 3D"





The catastrophic difference in sharpness of the foreground object in the film "The Creature from the Black Lagoon" (and again a horror film)



Interestingly, similar examples are found in films that were not shot, but converted to 3D. Those. the difference in sharpness occurs due to an installation error. In particular, below the motion blur is superimposed on one angle and not superimposed on the second. The Nutcracker is not a horror movie, but ... in terms of the technical quality of the stereo, it is, unfortunately, terrible.





A big difference in sharpness in the domestic "Nutcracker". Interestingly, this is a conversion ...



You can even spoil the pavilion shooting, below is an example from the 3D cartoon “Coraline in the Land of Nightmares” (not that the cartoon was nightmare, but in some places ...):





It is clearly seen that the creators missed exposing the camera parameters in the puppet cartoon (the depth of field of the right and left angles do not match)



Interestingly, even in high-budget films, there are problem scenes. Fortunately - quite rarely. Below is an example from the Hobbit:





A noticeable difference in sharpness in the center of the frame on the subject that viewers will definitely look at



Well, rather as an announcement. In 2D to 3D conversion, in 99% of cases, the sharpness of objects coincides, but at the boundaries of objects it can differ markedly. This effect is called - strobe borders, they are very uncomfortable and it is easy to teach a layman to “see” them (that is, you will understand why you feel discomfort when you look at individual objects in stereo). Below are a couple of examples from a very high-quality converted film "Titanic":





The problem at the border is close to the center of the frame, there will be slight discomfort





And here it’s not just in the center of the frame, but close to the area of ​​attention (the main character’s face, see the full frame in the corner), not only the sharpness is different, but also the mask traces are visible and the background is inaccurately restored, i.e. strobing the border will be strong



However, more about the conversion problems will be in a separate part.



Conclusions:





Sharpness Trends



As usual, let's see what happens in films by year. The films below have been analyzed since 1954. Red means the worst 30% of films at this point in time, green means 30% of the best films.





Dependence of the difference in sharpness on the year of shooting (the higher the better)

Source: measurement graphs hereinafter - author's materials



It is clearly seen: before the Avatar (marked with a dotted line), most films were worse than him in this parameter, but later the quality of the vast majority of films became noticeably better. What pleases. Consider the dependence of the difference on the budget:





Dependence of the difference in sharpness on the budget (the higher the better)



It can be seen that Avatar is the worst high-budget film, i.e. the rest of the high-budget films (released after) were better in quality. Another good news is that there are very low-budget films that have a minimal sharpness difference at the level of the best expensive films. This was made possible thanks to improved cameras and simplified analysis of this problem both during filming and at the post-processing stage.





If you look at what is going on as a whole for the films, you can see that about half of the films are worse than Avatar. In some cases - much worse ...



Conclusions:





Correction



Often in a scene with a wide range of depths, we have sharp objects on the left-hand side, sharp on the right-hand side, and vice versa:









When correcting, we have several options, for example, bringing one angle in sharpness to the second:







Moreover, in practice, bringing objects in sharpness to the right angle or to the left is not all the same. If the “restoration” of blurring usually goes smoothly, then the restoration of sharpness can give rise to artifacts (primarily because when one of the objects is blurred, the difficulty of accurately matching objects increases).





In addition, we can, for example, bring both angles to sharp or, conversely, to blurry, resulting in 4 correction strategies. Moreover, arbitrary combinations of these strategies are also possible.



Interestingly, from a technical point of view, it is “profitable” for the algorithm (and the person applying it) to bring both angles to a blurry one. This is simpler (often “in one click”), and with minimal probability generates artifacts. And from the point of view of increasing the visual comfort of stereo, it’s “more profitable” to make both angles sharper (which, as a rule, is more expensive in terms of the amount of additional work, since it is usually not done with one click). And the operator may have a desire to choose to cast to the right or left angle. In general, the choice is rich, and the more the studio has a budget / qualification, the more high-quality result for the viewer we get.



Let's look at how it looks, using the example of one of the films of the International Moscow Stereo-Film Festival (which will be held for the 10th time in 2019):





It is clearly seen that due to an error during the shooting (from which no one is safe), the depth of the sharply depicted space on the right angle turned out to be greater than on the left



It can be seen that the human nose has the same sharpness between the angles, but whiskey, ears, a shirt, a jacket - already noticeably diverge in sharpness (also the angles diverge in color). Accordingly, we can fix everything in the difficult direction of sharpening, but we can equally blur it (all recovery examples below are ours):







An example of correcting the difference in color, sharpness, and geometry of Nicolas Cage’s brutal bristles in Drive Angry:





Sharpness Fix:









On the left, the angle is sharpened to the right, and on the right, colors and a small vertical shift to the left are corrected



It is clearly seen that after correcting the sharpness, color and vertical shift, the camera angles became much more similar and the discomfort of this scene will be noticeably lower.



Another sharp fix example from Drive Angry:





In this example, even without a strong approximation, a serious sharpness difference is clearly visible, corrected towards sharpening



And the last example from "Resident Evil", which also corrected the difference in sharpness and color:





To look at the girl’s shoulder and face and at the gravel to the right of the knee, note that the highlight on the hair is not affected (and this is correct, although not trivial)



Conclusions:





3D movie perspectives



Now let’s distract from the sharpness and, as promised at the very beginning, we will tell the most interesting thing - about the prospects of 3D cinema. What, in fact, has changed over the years since the death of 3D was repeatedly described by even the most lazy journalists.



A dramatic increase in the number of halls



As mentioned above, the number of cinemas where you can watch 3D from 400 to 2018 increased 400 times (by the way, about CAGR in the beginning there was a joke for those who understood it, the process, of course, was linear):





Source: Number of 3D cinema screens worldwide from 2006 to 2018



And if in 2006 in practice the film could be watched only in large cities of Europe and the USA, where the lion's share of pathetic 258 3D cinema halls was located, today, in 2019, the availability of halls has grown by two orders of magnitude. In addition, while in China, for example, there were practically no 3D halls, today it is one of the world leaders, i.e. 3D films as an industry have appeared in Asia.



Conclusions:





The advent of laser projection



It so happened that some time ago we "worked as projectionists", servicing the professional projectors Digital Projection , installed at Moscow State University. And I had to deal closely with the lamp market when projector manufacturers make money on lamps, much like cartridge printer manufacturers. Lamps are supposed to be changed often, and original lamps are VERY expensive (how do you like a lamp for half a million rubles? ). This situation spawned the release of Chinese non-original lamps at a price of 6-10 times lower, but with lower and less stable from batch to batch characteristics. Description of adventures with them is worthy of a separate saga, I had to get acquainted with professional forums, etc. Guess which lamps are most likely to stand in a movie theater near your home? Of course, the original Chinese.



For the viewer, this means a significantly darker screen, which affects the perception of 3D extremely sadly. I remember that the authors of Stalingrad (if I am not mistaken, the only IMAX 3D film produced in Russia) complained that many of the best cinemas went around in search of a hall for the first run, and everywhere (everywhere!) The image was dark.



The situation has changed dramatically for the better with the advent of laser projectors, the image of which is much brighter, and (importantly!) The degradation of brightness is much slower. Moreover, Christie Digital Systems , one of the market leaders in cinema projectors , notes that the second generation of laser projectors is already coming out right now (note that 10–18 thousand per year are quite consistent with the growth of 3D cinema halls in the world, taking into account the updating of outdated equipment in some 3D cinema halls ):









Source: Christie Taking RGB Laser Projection to Mainstream Cinemas



Accordingly, the current Laser Phosphor projectors (LP or LaPh) will very soon be replaced by RGB Laser projectors, which in a number of indicators, including brightness, are much better. In order not to offend anyone, we refer to the Christie - Barco rivals FAQ - in comparison of these two laser technologies . Note that in China alone there are already more than a dozen manufacturers of laser film projectors and the competition in this market, from the point of view of the consumer, is very interesting. However, this is a separate big promising story.



Conclusions:





IMAX,



But this is not all good news. Recently, completely new devices have appeared that fundamentally change the alignment. These are helmets designed specifically for watching movies, including 3D movies. They became possible thanks to the advent of new micro-screens (manufactured, in particular, Sony) with fantastic characteristics. By the number of dots per inch of the screen for the general public, the leaders are the marketing gurus - Apple with the iPhone X. However, these devices have a pixel density of 7 times higher than the iPhone X!





Source: Goovis helmet materials.



The main difference is that the optical system is not designed like in VR helmets, where a large viewing angle is important., but as if the screen is hanging in front of you (the viewing angle is smaller). And people do not understand what's wrong with that. And an incredible thing is happening. Thanks to this, in the literal sense of the word, a small miracle has the opportunity for the first time to massively watch 3D cinema at home with parallax of a large screen! Here is our old picture:





Source: this figure and the following graph - the author’s materials



The bottom line is simple - parallax 1% with a screen width of 6.5 meters - this is a shift of objects by 6.5 centimeters (distance between the eyes) and voila - the moon hangs for us in infinity and 3D scene mesmerizing with depth. And if we have a 3D TV with a screen width of 60 cm, 1% of the same content is 6 mm. Draw a proportion. We are sad. For the moon hangs a maximum of ten centimeters behind the screen (and even if you are located farther from the screen,the question was discussed in detail in the section on parallax ). There can be no talk of any “infinity”, because 6 mm cannot be turned into 6 cm, the visually perceived depth is small and the 3D cinema looks “flat” (because with the crashes from the screen to the viewer - the same trouble, they are reduced even if not an infinite number of times, but at times). Everything is flat and dull!



Is it possible to make parallax and the apparent depth of the film more?







The answer is yes, it is possible, but no one will do it. According to our measurements (see the graph above), the apparent depth is likely to fall, and on average it is noticeably smaller, even than the Avatar. You will not be told about this in an interview, but the film industry lives primarily on the cinema camps, and they won’t be able to cut the branch on which they sit (because torrents work like clockwork), releasing versions for viewing on a TV screen. So if you want a 3D effect to directly “Wow!” - welcome to laser IMAX (this, unfortunately, is not advertising, this is the minimum requirement). Raising parallax is also algorithmically possible. But this is the topic of a separate sad story.



However, down with sadness and sadness, it was so yesterday! And today helmets have appeared that allow you to watch at home “like in a movie”, including the 3D effect and FullHD. Thanks to the wonderful enthusiast Vladimir ( @ ED-209 ), my colleagues and I had the opportunity to test two models of such helmets. I really hope that we will write a large article about these devices, because they are worth it. Especially those who “hate 3D glasses” should like them:









Source: GOOVIS Pro Mobile 3D Cinema ; it can be seen that the screens are configured asymmetrically in distance from the bridge of the nose and different diopters are set.



The devices have a rare (for VR helmets) ability to configure diopters for each eye. As a result, you can forget about “our 3D glasses can be comfortably worn over your glasses” as a nightmare. And you can flexibly adjust the distance between the bridge of the nose and each (!) Eye, i.e. even the rare owners of broken noses and asymmetrical nose bridge (which is usually saved in VR helmets for comfort) can be customized. Moreover, a familiar person with a slight squint, for the first time in his life , was able to see surround stereo moviesthanks to this helmet and its flexible settings (in a 3D cinema, it does not see the volume). As a result, it will be convenient for everyone to watch, they even provided for a quiet cooler so that the screens do not fog up in 2 hours of the film. It’s just some kind of holiday, I want the development team to shake hands with everyone. They deserve it.



Well, a huge (huge!) Plus - 2 independent screens, i.e. you don’t need to use special technologies (spoiling life for both 3D cinemas and 3D TVs) to first display 2 angles on the screen at once, and then painfully and artifacts separate them back between the eyes.



As a result, we get:





Naturally, the devices also have problems. The main one is that the pixels of this miracle are very small and lack color depth . There was even a suspicion that there are 7 bits, but an autopsy (which included the non-trivial receipt of closed technical documentation from the manufacturer) showed that there are honest 8 bits, but the screen is so contrasted that they are noticeably lacking at times. I hope I will be able to tell in detail about these adventures and tests, as well as the forerunner of these helmets from Sony.



Conclusions:





The emergence of new software and algorithms



When Cameron did Avatar, there was practically no software for normal stereo operation. The take-off of Nuke was not surprising, there it is "native", and not ugly crutches, like in [cut out by censorship], it supports not only stereo, but also multi-angle video. And even plenoptic, for that matter. This is strong, and more recently, such an easy opportunity to work with new formats was not in principle.



A similar situation with the development of algorithms. Over the past 10 years, the algorithms for comparing angles , constructing an optical stream , processing matting the translucent boundary of objects, and auto-filling areas have been significantly less practical, less convenient, less fast. CUDA started corny just 12 years ago, and OpenCL - just 10 years ago. Moreover, GPU performance continues to grow amid a slowdown in CPU performance. NVIDIA promised a difference of three orders of magnitude by 2025:





Source: What is CUDA? Parallel programming for GPUs



Also, the changes that have occurred with video processing algorithms over 10 years are tectonic , since not just other algorithms, but other classes of algorithms became available with GPGPU . It would be great to write about it, because this is a direct area of ​​the author's work.



A separate powerful trend is neural networks:



Source: Top 10 Hot Artificial Intelligence (AI) Technologies



It is important not to overestimate them, especially in the short term, but in the long term - this is another revolution. The diagram above shows only the analysis of video and images, but modern processing (video processing, including stereo video processing), in fact, is based on analysis, so these trends also have a serious positive effect on it.



Conclusions:





3D shooting of the future



The complexities of 3D shooting described in color in detail and in previous articles were so great that 3D almost ceased to REMOVE. In fact, the lion's share of modern 3D films is converted. There, too, not everything is perfect in terms of quality and pain, but the situation is radically better, which, we note, was seen in the numerous graphs given in the previous parts, where the converted films almost always (because you can ruin everything) hung around the ideal.



The main reason for the massive use of conversion is a greater visual quality, lower and more predictable price, and a lower percentage of viewers experiencing a headache. In this regard, the shooting had no chance. Only a slow painful death.



Nevertheless, the conversion is also far from magical even in blockbusters (the author really hopes to publish examples), and most importantly, the depth there is not real, but the one that was considered the best (in all senses, including cost) in the studio.



The question is: is it possible to REMOVE 3D in such a way that there are not all the problems stated above (and a dozen more boring, but no less painful ones)?



The answer is yes! But for now it is too complicated.



The solution is to develop the so-called plenoptic cameras, about which there was a separate large and detailed post . Now it comes down to the presence of compact high-resolution sensors with a small edge (which can be placed close to each other) and low heat dissipation. Given the current fantastic speed of improving sensors, it will not take so long to wait.



Conclusions:





Instead of a conclusion



Lord! Some irresponsible citizen journalists periodically announce the neighborhood shouting "3D cinema has died !!!". Do not be alarmed. This is from illiteracy and lack of information.



As they said a couple of centuries ago (for another reason): “3D cinema has died! Long live 3D cinema! ”Now the time has not come yet for this phrase, but a lot has been done for this and continues to be very actively done. And the same comrades, it seems, even sincerely shout the opposite. In the United States, this was already 6 waves of 3D cinema in 100 years.



For example, Cameron has already removed Avatar 2 and 3, and judging by the numerous postponements of the release date (now it is December 21, 2021), something suggests that he aimed at least 3 billion fees from 2 and 3 parts. Now in a year there are more 3D-halls open than there were only at the time of the release of the first Avatar (and +2.5 years is a lot). At the same time, the technical image quality of the new halls is radically higher than the technologies of 10 years ago, especially in the second-generation laser projection, which, wangwu, will begin to be actively advertised in 1.5 years.



Carthage must be broken ... The whole video will be three-dimensional by the end of the century!



If there is enough time - I will write about the conversion.



Stay tuned!



Read also:





Acknowledgments



I would like to heartily thank:





More articles:


All Articles