CVPR 2022
DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Rendering
Ruizhi Shao1, Hongwen Zhang1, He Zhang2, Yanpei Cao3, Tao Yu1, Yebin Liu1
1Tsinghua University 2Beihang University 3Kuaishou Technology
Fig 1. Given only sparse multi-view RGB images (6 views for the left on the Twindom dataset, 5 views for the right on the real-world data), our method achieves high-fidelity neural human rendering for loose clothes and various poses.
Abstract
We introduce DoubleField, a novel representation combining the merits of both surface field and radiance field for high-fidelity human rendering. Within DoubleField, the surface field and radiance field are associated together by a shared feature embedding and a surface-guided sampling strategy. In this way, DoubleField has a continuous but disentangled learning space for geometry and appearance modeling, which supports fast training, inference, and finetuning. To achieve high-fidelity free-viewpoint rendering, DoubleField is further augmented to leverage ultra-high-resolution inputs, where a view-to-view transformer and a transfer learning scheme are introduced for more efficient learning and finetuning from sparse-view inputs at original resolutions. The efficacy of DoubleField is validated by the quantitative evaluations on several datasets and the qualitative results in a real-world sparse multi-view system, showing its superior capability for photo-realistic free-viewpoint human rendering.
[arXiv] [Code(comming soon)]
Overview
Fig 2. Comparison of different neural field representations. (a) Neural surface field in PIFu. (b) Neural radiance field in PixelNeRF. (c) The proposed DoubleField. (d) DoubleField with the raw ultra-high-resolution inputs.
Fig 3. High-fidelity neural human rendering based on DoubleField.
Results
Fig 4. High-fidelity rendering results from only 6 views on Twindom and THuman2.0 dataset.
Fig 5. High-fidelity rendering results from only 5 views on real-world data.
Technical Paper
Demo Video
Citation
Ruizhi Shao, Hongwen Zhang, He Zhang, Yanpei Cao, Tao Yu, and Yebin Liu. "DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Rendering". arXiv 2021
@misc{shao2021doublefield,
title={DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Rendering},
author={Shao, Ruizhi and Zhang, Hongwen and Zhang, He and Cao, Yanpei and Yu, Tao and Liu, Yebin},
year={2021},
eprint={2106.03798},
archivePrefix={arXiv},
primaryClass={cs.CV}
}