De-lighting Human Images

In real-world image processing, the original scene lighting is crucial to consider. Changes in lighting can significantly impact how a subject appears, potentially improving or diminishing the viewer's experience or the performance of image processing systems. Being able to adjust the lighting as a post-processing step is valuable for situations where controlling scene lighting is restricted. Significant progress has been made towards photo-realistic relighting of human portraits using deep learning. However, failure to remove real-world illumination features present in the input image has been a major drawback of recent work. To overcome these limitations, this research proposes various deep learning strategies for de-lighting single images of humans, that is, recovering the albedo, or a uniformly-lit image with all lighting artifacts removed. In Chapter 3, we developed a deep learning framework for de-lighting of portrait images, where we gain significant performance improvements by introducing three regularization strategies: masked loss, to emphasize high-frequency shading features; soft-shadow loss, which improves sensitivity to subtle changes in lighting; and shading-offset estimation, to supervise separation of shading and texture. Furthermore, we demonstrate how our de-lighting method can enhance the performance of light-sensitive computer vision tasks such as face relighting and semantic parsing, allowing the tasks to handle extreme lighting conditions. In Chapter 4, we transition to delighting full-body images, which pose greater challenges than portraits due to diverse poses and clothing variations. We design a semi-supervised deep learning framework for full-body albedo estimation that leverages both limited labeled data, and a vast dataset of unlabeled real-world photos. Here, we improve generalization by a novel data augmentation method for diversification, a sparsity constraint on the inferred shading for stable texture recovery, and dynamic gamma correction for handling complex shadows found in real-world images. Our experiments highlight that this approach outperforms state-of-the-art methods, improves shading removal, and maintains the integrity of underlying textures in real-world scenarios.