Extended Path Space Manifolds for Physically Based Differentiable Rendering

Published in Siggraph Asia, 2023

Jiankai Xing, Xuejun Hu, Fujun Luan, Ling-Qi Yan, Kun Xu

width=20%Paper width=20%Code(Mitsuba version here, LuisaRender version coming soon)

Fig. 1 Optimizing the 2D translation vectors of eight specimen objects via physically based differentiable rendering. Note that the specimen objects are placed inside a glass box and are viewed through nested reflection and refraction.

Abstract

Physically based differentiable rendering has become an increasingly important topic in recent years. A common pipeline computes local color derivatives of light paths or pixels with respect to arbitrary scene parameters, and enables optimizing or recovering the scene parameters through iterative gradient descent by minimizing the difference between rendered and target images. However, existing approaches cannot robustly handle complex illumination effects including reflections, refractions, caustics, shadows, and highlights, especially when the initial and target locations of such illumination effects are not close to each other in the image space. To address this problem, we propose a novel data structure named \emph{extended path space manifolds}. The manifolds are defined in the combined space of path vertices and scene parameters. By enforcing geometric constraints, the path vertices could be implicitly and uniquely determined by perturbed scene parameters. This enables the manifold to track specific illumination effects and the corresponding paths, i.e., specular paths will still be specular paths after scene parameters are perturbed. Besides, the path derivatives with respect to scene parameters could be computed by solving small linear systems. We further propose a physically based differentiable rendering method built upon the theoretical results of extended path space manifolds. By incorporating the path derivatives computed from the manifolds and an optimal transport based loss function, our method is demonstrated to be more effective and robust than state-of-the-art approaches in inverse rendering applications involving complex illumination effects.

Results

@article{Xing2023EPSM,
    title = {Extended Path Space Manifolds for Physically Based Differentiable Rendering},
    author = {Xing, Jiankai and Hu, Xuejun and Luan, Fujun and Yan, Ling-Qi and Xu, Kun},
    year = {2023},
    publisher = {ACM},
    booktitle = {SIGGRAPH Asia 2023 Conference Papers},
    url = {https://doi.org/10.1145/3610548.3618195},
    articleno = {62},
    numpages = {11}
}