Abstract

Recent advances in novel view synthesis (NVS), particularly neural radiance fields (NeRF) and Gaussian splatting (3DGS), have demonstrated impressive results in photorealistic scene rendering. These techniques hold great potential for applications in virtual tourism and teleportation, where immersive realism is crucial. However, the high-performance demands of virtual reality (VR) systems present challenges in directly utilizing even such fast-to-render scene representations like 3DGS due to latency and computational constraints.

In this paper, we propose foveated rendering as a promising solution to these obstacles. We analyze state-of-the-art NVS methods with respect to their rendering performance and compatibility with the human visual system. Our approach introduces a novel foveated rendering approach for Virtual Reality, that leverages the sharp, detailed output of neural point rendering for the foveal region, fused with a smooth rendering of 3DGS for the peripheral vision.

Our evaluation confirms that perceived sharpness and detail-richness are increased by our approach compared to a standard VR-ready 3DGS configuration. Our system meets the necessary performance requirements for real-time VR interactions, ultimately enhancing the user's immersive experience.

Motivation

Gaussian rendering times increase drastically with number of splats, making full 3D Gaussian Splatting in VR difficult.



Thus for efficient rendering, we want a foveated approach, where looked-at areas are rendered in high quality and peripheral areas have lower rendering quality.
The periphery is sensitive to flickering. Gaussians however are inherently smooth, even at low primitive counts, making Gaussians perfect for peripheral rendering.


In the small time budget for the fovea, we can use neural point rendering (via TRIPS [Franke et al. 2024]) for crisper details.

Pipeline

Our method is as follows:

1. A smooth color image and approximate depth map are rendered from a limited set of 3D Gaussians with a temporally stable sorting active.
2. Afterwards the eye tracking system is queried to construct a subfrustum via an adapted projection matrix covering only the foveal region. We project a separate neural point cloud with the adapted matrix. Points, occluded by the denser Gaussian splats, are culled against the approximate depth maps and the result is interpreted by a shallow convolutional neural network.
3. Eventually, we blend the peripheral color output and the foveal region using an egde-aware mask.

The pipeline is trained end-to-end as a combined foveated representation.

Our combination function: An edge-aware mask blends peripheral and foveal renderings.

Results

SteamVR display mirroring

Our pipeline runs in 90Hz with full VR resolution of 2016x2240 pixels per eye.

User Study

Two-alternative forced-choice (2AFC) between OURS and VR-GS, a Gaussian Splatting VR solution (few primitives). Five Scenes (Garden, Bicycle, Bonsai, Family, Playground), twelve participants (aged 22 to 30, four female, eight male), 20 trials each.
We observe a significantly higher ratio of participant preferring OURS to VRGS (~76% preferred OURS, SD=0.196). Binomial testing showed significance with p < 0.005.

Table

Quantitative metrics. JOD (Just-Objectionable-Difference) [Mantiuk et al. 2021] are computed with FovVideoVDP for an HTC Vive Pro in foveated mode for the full images. LPIPS, PSNR and SSIM metrics are evaluated for the foveal regions only.

Videos

(red fixation point)