Whose Hands Are These? Hand Detection and
Hand-Body Association in the Wild

Supreeth Narasimhaswamy¹, Thanh Nguyen², Mingzhen Huang³, Minh Hoai^1,2

¹Stony Brook University, ²VinAI, ³University of Buffalo

CVPR 2022

Qualitative Results

Figure 1: We develop a method to detect hands and their corresponding body locations. Hands and bodies belonging to the same person have bounding boxes in the same color and identification numbers.

Abstract

We study a new problem of detecting hands and finding the location of the corresponding person for each detected hand. This task is helpful for many downstream tasks such as hand tracking and hand contact estimation. Associating hands with people is challenging in unconstrained conditions since multiple people can be present in the scene with varying overlaps and occlusions.

We propose a novel end-to-end trainable convolutional network that can jointly detect hands and the body location for the corresponding person. Our method first detects a set of hands and bodies and uses a novel Hand-Body Association Network to predict association scores between them. We use these association scores to find the body location for each detected hand. We also introduce a new challenging dataset called BodyHands containing unconstrained images with hand and their corresponding body locations annotations. We conduct extensive experiments on BodyHands and another public dataset to show the effectiveness of our method. Finally, we demonstrate the benefits of hand-body association in two critical applications: hand tracking and hand contact estimation. Our experiments show that hand tracking and hand contact estimation methods can be improved significantly by reasoning about the hand-body association.

BodyHands Dataset

                
                    @inproceedings{bodyhands_2022,
                        title={Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild},
                        author={Supreeth Narasimhaswamy and Thanh Nguyen and Mingzhen Huang and Minh Hoai},
                        booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
                        year={2022},
                    }
            
                    @inproceedings{handler_2022,
                        title={Forward Propagation, Backward Regression and Pose Association for Hand Tracking in the Wild},
                        author={Mingzhen Huang and Supreeth Narasimhaswamy and Saif Vazir and Haibin Ling and Minh Hoai},
                        booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
                        year={2022},
                    }

                    @inproceedings{contacthands_2020,
                        title={Detecting Hands and Recognizing Physical Contact in the Wild},
                        author={Supreeth Narasimhaswamy and Trung Nguyen and Minh Hoai},
                        booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
                        year={2020},
                    }