Unique and patented technology solutions for real-time holography
With SeeReal’s extensive background in development and manufacture of Stereoscopic 3D (S3D) displays, it became obvious to the SeeReal team a long time ago that S3D inherently can never provide a comfortable long-term viewing experience of natural looking 3D scenes with properly scaled depth. S3D’s „2x 2D views“ principle (one 2D view for the left eye and one for the right eye) is not capable delivering all visual depth cues to the human sensory system. This applies to all Stereo 3D solutions – no matter if with Stereo glasses or without (Auto-Stereo). Also, the resulting requirement for depth-compressed 3D of S3D systems prohibits proper scale of 3D images which is either boring for consumers or prohibitive to professionals – just imagine a surgeon looking at an organ or an engineer working on a CAD design generated with improper depth. But even depth-compressed S3D with missing 3D cues generates eye fatigue or discomfort to a significant part of the population.read more
Therefore SeeReal developed solutions based on true holographic principles, using phase-modulating display hardware and interference-based light modulation for „reconstruction“ of 3D scenes in real 3D space. The principle objective of Holographic 3D (H3D) is generating the same information to the human eyes as a natural 3D scene would generate. The complete visual information of H3D generates all visual depth cues and enables a comfortable long-term viewing experience and a host of unique imaging features, not possible with any other imaging technology. Holographic 3D is typically considered „the holy grail“ of imaging.
While holograms, invented by Denis Gabor in the 1940’s, have been known and investigated for a long time, the classic approach of super-resolution media (starting with analog films), expensive illumination with long coherence length lasers and super computing, when ported to active display technology, was thought to be unsuitable for real-time imaging for the coming centuries … So SeeReal went out to create solutions eliminating the „classic requirements“ by starting with the human perception instead of the optical system. Soon SeeReal’s team realized that when limiting H3D information to what observers can see at any point in time, significant simplifications are possible for the optical system, illumination and hologram computation. Even for a natural view with large field of vision, a person sees through eye pupils a few millimeters in diameter … This was the starting point of an intense development of system concepts and designs, component technologies and corresponding software.
Many of SeeReal’s enabling solutions were implemented in prototype setups using a mix of off-the-shelf and custom components. Resulting were probably the largest real-time interactive H3D reconstructions ever created – on 20 inch screens, in HD resolution, and unlimited 3D depth. Mostly with components representative for a typical LC supply chain. These proof of principles, together with ongoing new developments and optimizations, now serve as basis for implementation of SeeReal’s H3D into first products with manufacturing and license partners.
True interference-based holographic reconstructions in 3D space
The term „hologram“ has been in frequent use – and misuse – since the „formal“ invention of holography in the 1960’s. So for the avoidance of doubt it must be emphasized that SeeReal’s solution to 3D displays is based on true interference-based holography in its literal meaning. Holographic 3D (H3D) designs, systems and prototypes of SeeReal are not based on mere imaging on flat screens or semi-transparent mirrors. SeeReal solutions reconstruct 3D objects and complex 3D scenes in real space. The result cannot be easily shown on 2D displays, as you probably use reading this. But bear with us and read on …read more
So imagine looking at a common Stereoscopic 3D (S3D) display like available for desktop, TV or movie cinemas. In S3D your left and right eye see two corresponding perspective images filtered by S3D goggles or optics in the displays. But the moment you take off the S3D glasses, the 3D effect collapses into the two overlapping perspective images on the flat screen. The 3D effect disappears also if you close one eye. S3D could be described as 2 times 2D (one 2D image for the left and one for the right eye).
Now imagine a real 3D scene in space as generated by a H3D display. Since the holographically reconstructed objects are really in 3D space, you may close one eye and still receive the same 3D depth cues to your visual system you would receive from a real object. This for instance includes eye focus (or accommodation) or motion parallax (relative movement of objects near to your eye or far away when you move). Of course the second perspective of the closed eye would be missing during this test, which makes it more difficult for your brain to process the 3D information at the same speed and precision. But even with one eye you can see 3D because all visual cues are presented by H3D. In fact, the additional information generated by H3D (compared to S3D) makes all the difference. In H3D you cannot just enjoy large and properly scaled depth 3D without eye strain or fatigue, you very quickly realize that this looks and feels as real as the real world.
For visualization of a H3D depth cue on a standard 2D display, two photos of a H3D scene in space are provided here (taken from a H3D SeeReal display prototype). Pictures were greatly magnified to make the reconstructed „3D pixels“ in space visible. Changing the focus of a common DSLR camera onto the simple H3D scene also changes blur and sharpness as you would expect and see using real objects. This is because H3D generates information in real space. Observer eyes receive proper 3D views onto real space objects. You may say H3D generates 2 times 3D (one proper 3D view per eye). This is not possible with S3D in its various embodiments, with or without S3D glasses.
Overcoming the super-resolution display requirement of „classic holography“
A long time ago, first holograms were made using high resolution analog film, easily with up to 5000 lines per millimeter resolution. Since such film is capable of generating very fine structures, diffraction angles can be very large. This translates into large viewing angles onto these „classic holograms“. Accordingly, 3D reconstructions from analog holograms are visible in a large area in front of the hologram. But transferring this principle to digital display technology would mean generation of extremely small displays pixels – at an extreme quantity. Even if this was possible (and scalable) computation requirements scale to PetaFLOP range for a decently sized display presenting holographic reconstructions with high H3D resolution and large 3D depth. Proper coherent illumination of such system is still another challenge.read more
But considering how much effort (in pixels, illumination and computation) would be needed for a large and real-time „classic“ H3D display to generate 99% information invisible to viewers of such displays, SeeReal turned to another and proprietary solution: Limit the information to what observers can see at a certain point in time – each eye during each frame.
SeeReal’s solution generates full holographic 3D views with high resolution and proper 3D depth each frame of an interactive game, movie, engineering design, night vision ride or video conference, but only at the eye positions of the observers. SeeReal has been calling these eye boxes virtual „Viewing Windows“ for over ten years, starting at a time when the first in-roads to the solution were developed by the SeeReal team. Effects of this unique concept are relaxed requirements on display pixel size, computation and illumination. Pixel resolutions found in today’s mobile devices suddenly become suitable for large H3D reconstructions with unlimited 3D depth and in real-time and high quality.
Properly implemented, and demonstrated at SeeReal’s laboratories, the Viewing Window principle eliminates crosstalk between left and right eye views of an observer as well as crosstalk to Viewing Windows of other observers. This provides not just the option of extreme viewing privacy but also that each viewer watches individual perspectives of an H3D scene – or even a totally different H3D data source, on the same display. For freedom of movement of viewers, viewer eye positions are tracked in space and Viewing Windows are properly placed each frame using the optical system of the H3D display.
Limiting H3D information to Viewing Windows is one of the many fundamental inventions of SeeReal and a key enabling aspect of H3D based on available technology.
Eliminating the need for super-computation resources of large high-resolution 3D holograms
As explained above, SeeReal limits the holographic information to what observers of a H3D display, or in fact of any real life environment, can see at a given point in time and space. By limiting the information to eye boxes or Viewing Windows, one also limits the area on the physical display, the hologram, required to encode information for individual portions of the 3D scene. One may say, SeeReal discretizes the hologram. In fact, each 3D scene point can be encoded in a very small individual hologram which are then super-positioned to generate the total 3D scene. SeeReal established the term „Sub-holograms“ for these hologram fractions.read more
While further optimizations had to be developed by SeeReal to scale computation requirements to a commercially viable level, one can imagine that small Sub-holograms play an important role.
For a typical 3D-scene with Full HD resolution, around 2 million Sub-holograms are computed and combined to a hologram frame. Accordingly H3D video streams contain full holographic representations of 3D scenes each frame. The visible 3D world is always present, no matter where a viewer looks at the display – same as in real life. You may consider a H3D display as a window onto the 3D world; objects can be far behind or even in front of the window.
SeeReal’s H3D display prototypes deploy these fundamental and proprietary principles of Viewing Windows and Sub-holograms for real-time interactive and HD resolution 3D reconstructions. Real-time code generation has been ported to various platforms, including CPU, GPU and FPGA.
Simplifying the light source coherence requirements for large holographic 3D reconstructions
Breaking down a 3D scene and corresponding hologram into small fractions, which SeeReal many years ago introduced as Sub-holograms, greatly simplifies the illumination requirements. In „classic holography“ each portion of the hologram (or H3D display) must be in sync, because all portions contribute to each portion of the 3D reconstruction; light from all physical display pixels interferes. This requires laser light sources with a coherence length at least as large as the physical display size, which calls for quite expensive lasers.read more
SeeReal’s Sub-holograms, each encoding only a fraction of the 3D scene, are much smaller than the display and they are independent from other Sub-holograms further away. So coherence length and correspondingly laser light sources can be greatly simplified. One may even use several lasers combined similar to common edge-lit LCD displays.
Core enabling technologies of H3D
SeeReal has developed holographic 3D (H3D) system designs combined with core enabling technologies for H3D displays for a wide range of sizes and applications. Core technologies are being continuously improved by SeeReal and in cooperation with partners for application in future (licensed) display products.
A portfolio of technology solutions has been developed for application in product-driven projects with key SeeReal customers and license partners. Read on below for some examples.
Some proprietary component solutions and services can be made available by SeeReal for third party application outside H3D displays. Please inquire via telephone or our contact form.
Laser display illumination
Some principles were explained further up this page, for instance the need for sufficiently coherent illumination of holograms, so that high quality H3D reconstructions are enabled. The required degree of light coherence is typically delivered by laser sources. While SeeReal’s hologram discretization and Sub-holograms greatly simplify laser light source requirements, which makes lasers cheaper, these spot sources must be expanded to larger areas to properly illuminate the hologram – ideally all in a slim package. Therefore, as part of SeeReal’s component development efforts, flat laser backlight designs were created, based on a mix of „classic“ refractive optics and diffractive optical elements (DOE).read more
Solutions for flat laser backlights were developed with the help of DOE properties. These backlights may use a single laser as source but also an array of smaller individual laser sources for combination across a large area.
It is important to note that the laser expansion inside the backlight and H3D display generates an inherently safe (class 1 laser) product for unlimited time use.
Large diffractive optical elements (DOE) based on volume Bragg diffraction
As explained in the previous paragraph, DOE are especially suitable for use with laser illumination and can generate very energy-effective systems. They also are instrumental to minimizing the physical size of a range of optical elements by replacing bulky glass elements with micron-thin films. SeeReal has developed solutions to illumination systems as well as imaging systems using DOE. This includes design and manufacture of a DOE family member, holographic optical elements HOE, as implemented in optical films with volume holograms. These can be made at partner sites but also in-house at SeeReal.read more
DOE can embody complex or simple optical functions in a single functional film of few microns thickness. Properly made, they exhibit very high light efficiency, which in turn translates to low crosstalk and stray light. Volume grating HOE, as made in SeeReal’s labs, steer light to a single diffraction order (i.e. light only goes where it is supposed to go). These elements can be made in direct exposure (as common for mastering or „tooling“) as well as in a master-copy-process (as required for mass replication). Sample replications from SeeReal masters exhibit the same optical efficiency as their master tools.
SeeReal’s in-house HOE component prototyping and manufacturing capabilities currently go up to approx. 350mm. Further scaling in size and quantities is done in partnership with large well-known optical components and systems companies.
Observer eye finding and tracking with high precision and speed
SeeReal’s tracked Viewing Window approach is based on repositioning observer eye boxes in real-time and unnoticed by the observer. As in real life viewing, one only requires the visual information passing through the few square millimeters of one’s eye pupils. So as long as seamless eye-tracking and positioning of H3D information at eye positions is performed, observers won’t miss anything and won’t even notice the reduction of data. Accordingly, a key enabling element to SeeReal’s H3D solution is fast and precise observer eye-finding and -tracking. Depending on application, this is implemented for single users (e.g. in desktop gaming) or for multiple users.read more
SeeReal’s H3D prototypes and display systems can be combined with any eye-tracking system as long as latency, precision and reliability match the display’s requirements. But for SeeReal H3D implementations and for potential use in third party products requiring eye-tracking, SeeReal has developed and made available a software solution based on gray-scale stereo cameras. SeeReal eye-tracking provides single millimeter precision, low latency and high reliability. The modular and portable software-system was designed to enable easy adaption to various hardware platforms and was already implemented in multi-core X86 (PC) and custom DSP-FPGA systems. The eye-tracking solution is applicable for Stereo 3D (S3D), Auto-Stereo 3D (AS3D), Holographic 3D (H3D) and other applications.
Configurations for high speed LC modulators
H3D display solutions, especially for multi-user applications, greatly benefit from higher frame rates. For LCD technology devices this is related to optimization of LC materials but also to backplane and driving architectures. SeeReal recognized this early in their development and dedicated resources to corresponding activities.
These included upgrading LCD systems to full lambda phase modulation or driving displays at high frame rates without the typical electrical losses of common active matrix architectures. Several LC material and system developments were conducted with partners in science and industry. But SeeReal also successfully conducted numerous electronics hardware and software projects with its dedicated in-house teams.read more
SeeReal’s LCD backplane develoment yielded in designs and practical hardware prototypes greatly exceeding common active matrix (AM) frame rates without suffering from typical AM scaling problems as electrical in-efficiencies.
For use in beam steering for observer eye tracking, SeeReal also developed active optical solutions which can be manufactured using common LCD industry supply chains.
Real-time computation of HD-quality H3D scenes of unlimited natural 3D depth
SeeReal’s patented Sub-hologram and Viewing Window technologies are key enabling elements to real-time holographic 3D scenes in high resolution, color and unlimited 3D depth. But even when using Sub-holograms, the classic approach to computing holograms, e.g. by using various Fourier transforms, would generate prohibitive computation requirements. Even Moore’s law or ingenious processor designs wouldn’t be enough for a very long time. Accordingly, SeeReal solved the problem with proprietary principles and algorithms which are much more efficient and easy to embed in semi-conductor hardware.read more
Content is King for any device. So it is good to know that hologram computation is immune to the type of content. Anything can be shown in H3D, no matter if it is CG data from a game or professional 3D application software, live action data from a sports event, entertainment show or remote drone camera or a 3D movie library. To SeeReal there are only pixels – 3D pixels in space. Examples are endless, but the unique 3D data handling of SeeReal’s algorithms also includes options as semi-transparent objects or smoke where multiple objects in various depths can co-exist with their individual properties, all without compromising the core H3D features as focus or motion parallax.
Standard 2D data is simply a sub-set of 3D, so common 2D content can be displayed in H3D. One could even move the available 2D plane close to one’s eyes. But for fully utilizing and benefiting from H3D capabilities, 3D data will be used. Data may come from a game engine and GPU but even common MPEG standards developed for Stereo 3D (S3D) can carry the data required for H3D visualization. This makes SeeReal’s H3D solution backwards compatible to existing content, standards, distribution channels and hardware. You may watch your legacy S3D movies in H3D quality without eye fatigue and 3D glasses. Concent providers may even consider scaling up the depth-compressed S3D videos to properly and more attractively scaled 3D depth. Interactive H3D from 3D-games or professional 3D applications could benefit from additional inputs as eye-tracking data and hand gestures to allow object manipulation, user interface control and object look-around. These and other options will enhance the user experience to a whole new level.
SeeReal’s H3D processing implementations are available for various platforms including GPU and FPGA and were prepared for porting to ASIC.
H3D display and component prototyping for system and product development
SeeReal has a proven track record for design and manufacture of auto-stereoscopic displays. While SeeReal has abandoned manufacture of stereoscopic 3D (S3D) displays, corresponding know-how and expertise in design and technology as well as components and integration have been utilized in numerous implementations. These range from component development to system prototyping, including not just optical and mechanical design and proprietary component manufacture but also electronics hardware and software.read more
In partner projects SeeReal often takes the lead in design, specification, component and system start-up or calibration and test. SeeReal also provides close support for all other aspects of cooperation, including supplier communication, component and system characterization, product development and manufacturing.
Many SeeReal prototype generations were made, some with limited purpose and lifetime. But impressive SeeReal H3D prototypes are available and are frequently used for demonstrations as well as development work.
Please note that SeeReal does not give public demonstrations as this would over-stretch our resources. Prototype demos are reserved for commercial discussions with project or license partners of SeeReal.