Natural interaction with virtual objects relies on two major technology components: hand tracking and hand-object physics simulation. There are functional solutions for these two components, but their hand representations may differ in size and skeletal morphology, hence making the connection non-trivial. In this paper, we introduce a pose retargeting strategy to connect the tracked and simulated hand representations, and we have formulated and solved this hand retargeting as an optimization problem. We have also carried out a user study that demonstrates the effectiveness of our approach to enable fine manipulations that are slow and awkward with naïve approaches.

@InProceedings\{SCVTO21,
  author       = "Sorli, Suzanne and Casas, Dan and Verschoor, Mickeal and Tajadura-Jiménez, Ana and Otaduy, Miguel A.",
  title        = "Fine Virtual Manipulation with Hands of Different Sizes",
  booktitle    = "Proc. of the International Symposium on Mixed and Augmented Reality (ISMAR)",
  year         = "2021",
  url          = "http://www.gmrv.es/Publications/2021/SCVTO21"
}