Free-viewpoint video has broad potential in immersive sports replay, telepresence, live performance capture, and robotics, where users need to view a dynamic scene from controllable viewpoints as it evolves. These applications require more than photorealistic rendering: the underlying 3D representation must also be updated efficiently enough to support streaming use. 3D Gaussian Splatting is a promising foundation for this setting because it represents scenes with explicit primitives that render quickly and can be locally optimized. However, many existing dynamic Gaussian pipelines still depend heavily on iterative photometric refinement to infer frame-to-frame motion. When objects undergo large displacement, much of the optimization budget can be spent recovering motion rather than improving appearance, making high-quality streaming updates expensive.

This thesis proposes OpticalSplat, a streaming Gaussian update framework that uses optical flow as a strong motion prior before refinement. The method estimates visible 2D displacement, lifts this evidence into group-level rigid motion proposals, and uses persistent ownership labels to move related Gaussians together. This motion-guided initialization places the Gaussian state closer to the next frame before optimization begins, allowing a small photometric refinement budget to focus on residual alignment, color, opacity, and local shape corrections.

OpticalSplat is evaluated against representative streaming and dynamic Gaussian baselines using reconstruction quality and per-frame update time. The experiments show a more favorable speed-quality trade-off: OpticalSplat reduces update time while preserving competitive reconstruction quality, narrowing the gap to more expensive tracking-based refinement without requiring the same optimization budget.

42 pages

Thesis Committee:
Fernando De la Torre (Chair)
Louis-Philippe Morency
Mihir Bala (Cornell University)

Jignesh Patel, Interim Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu