7.3 Correcting Optical Distortions

Recall from Section 4.3 that barrel and pincushion distortions are common for an optical system with a high field of view (Figure 4.20). When looking through the lens of a VR headset, a pincushion distortion typically results. If the images are drawn on the screen without any correction, then the virtual world appears to be incorrectly warped. If the user yaws his head back and forth, then fixed lines in the world, such as walls, appear to dynamically change their curvature because the distortion in the periphery is much stronger than in the center. If it is not corrected, then the perception of stationarity will fail because static objects should not appear to be warping dynamically. Furthermore, contributions may be made to VR sickness because incorrect accelerations are being visually perceived near the periphery.

Figure 7.14: A Fresnel lens (pronounced like ``frenelle'') simulates a simple lens by making a corrugated surface. The convex surface on the top lens is implemented in the Fresnel lens shown on the bottom. (Figure by Piotr Kozurno.)

How can this problem be solved? Significant research is being done in this area, and the possible solutions involve different optical systems and display technologies. For example, digital light processing (DLP) technology directly projects light into the eye without using lenses. Another way to greatly reduce this problem is to use a Fresnel lens (see Figure 7.14), which more accurately controls the bending of light rays by using a corrugated or sawtooth surface over a larger area; an aspheric design can be implemented as well. A Fresnel lens is used, for example, in the HTC Vive VR headset. One unfortunate side effect of Fresnel lenses is that glaring can be frequently observed as light scatters across the ridges along the surface.

Whether small or large, the distortion can also be corrected in software. One assumption is that the distortion is circularly symmetric. This means that the amount of distortion depends only on the distance from the lens center, and not the particular direction from the center. Even if the lens distortion is perfectly circularly symmetric, it must also be placed so that it is centered over the eye. Some headsets offer IPD adjustment, which allows the distance between the lenses to be adjusted so that they are matched to the user's eyes. If the eye is not centered on the lens, then asymmetric distortion arises. The situation is not perfect because as the eye rotates, the pupil moves along a spherical arc. As the position of the pupil over the lens changes laterally, the distortion varies and becomes asymmetric. This motivates making the lens as large as possible so that this problem is reduced. Another factor is that the distortion will change as the distance between the lens and the screen is altered. This adjustment may be useful to accommodate users with nearsightedness or farsightedness, as done in the Samsung Gear VR headset. The adjustment is also common in binoculars and binoculars, which explains why many people do not need their glasses to use them. To handle distortion correctly, the headset should ideally sense the adjustment setting and take it into account.

To fix radially symmetric distortion, suppose that the transformation chain $ T_{can}T_{eye}T_{rb}$ has been applied to the geometry, resulting in the canonical view volume, as covered in Section 3.5. All points that were inside of the viewing frustum now have $ x$ and $ y$ coordinates ranging from $ -1$ to $ 1$. Consider referring to these points using polar coordinates $ (r,\theta)$:

\begin{displaymath}\begin{array}{l} r = \sqrt{x^2 + y^2}  \theta = \atan2(y,x) , \end{array}\end{displaymath} (7.15)

in which $ \atan2$ represents the inverse tangent of $ y/x$. This function is commonly used in programming languages to return an angle $ \theta $ over the entire range from 0 to $ 2 \pi$. (The arctangent alone cannot do this because the quadrant that $ (x,y)$ came from is needed.)

We now express the lens distortion in terms of transforming the radius $ r$, without affecting the direction $ \theta $ (because of symmetry). Let $ f$ denote a function that applies to positive real numbers and distorts the radius. Let $ r_u$ denote the undistorted radius, and let $ r_d$ denote the distorted radius. Both pincushion and barrel distortion are commonly approximated using polynomials with odd powers, resulting in $ f$ being defined as

$\displaystyle r_d = f(r_u) = r_u + c_1 r_u^3 + c_2 r_u^5 ,$ (7.16)

in which $ c_1$ and $ c_2$ are suitably chosen constants. If $ c_1 < 0$, then barrel distortion occurs. If $ c_1 > 0$, then pincushion distortion results. Higher-order polynomials could also be used, such as adding a term $ c_3 r_u^7$ on the right above; however, in practice this is often considered unnecessary.

Correcting the distortion involves two phases:

  1. Determine the radial distortion function $ f$ for a particular headset, which involves a particular lens placed at a fixed distance from the screen. This is a regression or curve-fitting problem that involves an experimental setup that measures the distortion of many points and selects the coefficients $ c_1$, $ c_2$, and so on, that provide the best fit.
  2. Determine the inverse of $ f$ so that it be applied to the rendered image before the lens causes its distortion. The composition of the inverse with $ f$ should cancel out the distortion function.

Unfortunately, polynomial functions generally do not have inverses that can be determined or even expressed in a closed form. Therefore, approximations are used. One commonly used approximation is [119]:

$\displaystyle f^{-1}(r_d) \approx {c_1 r_d^2 + c_2 r_d^4 + c_1^2 r_d^4 + c_2^2 r_d^8 + 2 c_1 c_2 r_d^6 \over 1 + 4 c_1 r_d^2 + 6 c_2 r_d^4} .$ (7.17)

Alternatively, the inverse can be calculated very accurately off-line and then stored in an array for fast access. It needs to be done only once per headset design. Linear interpolation can be used for improved accuracy. The inverse values can be accurately calculated using Newton's method, with initial guesses provided by simply plotting $ f(r_u)$ against $ r_u$ and swapping the axes.

Figure 7.15: The rendered image appears to have a barrel distortion. Note that the resolution is effectively dropped near the periphery. (Figure by Nvidia.)

The transformation $ f^{-1}$ could be worked directly into the perspective transformation, thereby replacing $ T_{p}$ and $ T_{can}$ with a nonlinear operation. By leveraging the existing graphics rendering pipeline, it is instead handled as a post-processing step. The process of transforming the image is sometimes called distortion shading because it can be implemented as a shading operation in the GPU; it has nothing to do with ``shading'' as defined in Section 7.1. The rasterized image that was calculated using methods in Section 7.2 can be converted into a transformed image using (7.17), or another representation of $ f^{-1}$, on a pixel-by-pixel basis. If compensating for a pincushion distortion, the resulting image will appear to have a barrel distortion; see Figure 7.15. To improve VR performance, multiresolution shading is used in Nvidia GTX 1080 GPUs. One problem is that the resolution is effectively dropped near the periphery because of the transformed image (Figure 7.15). This results in wasted shading calculations in the original image. Instead, the image can be rendered before the transformation by taking into account the final resulting resolutions after the transformation. A lower-resolution image is rendered in a region that will become compressed by the transformation.

The methods described in this section may also be used for other optical distortions that are radially symmetric. For example, chromatic aberration can be partially corrected by transforming the red, green, and blue subpixels differently. Each color is displaced radially by a different amount to compensate for the radial distortion that occurs based on its wavelength. If chromatic aberration correction is being used, then if the lenses are removed from the VR headset, it would become clear that the colors are not perfectly aligned in the images being rendered to the display. The rendering system must create a distortion of pixel placements on the basis of color so that they will be moved closer to the correct places after they pass through the lens.

Steven M LaValle 2020-01-06