Markerless Motion Capture

Interacting Hands with Objects

Capturing the motion of two hands interacting with an object is a very challenging task due to the large number of degrees of freedom, self-occlusions, and similarity between the fingers, even in the case of multiple views observing the scene. In this paper we propose to use discriminatively learned salient points on the fingers and to estimate the finger-salient point associations and the hand poses at the same time. We introduce a differentiable objective function that also takes edges, optical flow and collisions into account.

Datasets and Results: [link]
Code: [link]

Results:





Publications:

Motion Capture of Hands in Action using Discriminative Salient Points

L. Ballan, A. Taneja, J. Gall, L. Van Gool, M. Pollefeys [PDF] [web] [bibtex] [supp] [dataset]               (Oral)
ECCV 2012, Firenze, Italy

Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

D. Tzionas, L. Ballan, A. Srikantha, P. Aponte, M. Pollefeys, J. Gall [arXiv pre-print] [video]








Interacting People

Input: 4 Video Streams Output: Pose Estimation
The project aims to track the pose of a human filmed by a set of video cameras. We explore a new approach to marker-less motion tracking of a priori known skinned meshes using both optical flow and silhouette information. We present a formulation which considers in a unified way both these two kinds of information and accounts for the non-rigid deformations of the object skin modeling them using the Skeletal Subspace Deformation (SSD). Once the pose of the human are estimated at each frame of the video, a free-viewpoint video of the entire action is genertated.
Introductory video: Soccer Juggling


Results:
Soccer Juggling Soccer Passing Drill Two People Boxing Handstand Pirouettes
Soccer Juggling Soccer Passing Drill Two People Boxing Handstand Pirouettes
 
Handshake Somersault Sword Swing Simple Break-dancing
Handshake Somersault
(result overlaid)
Sword Swing Simple Break-
Dancing


Datasets and Results: [link]
Code: [link]



Publications:

Marker-less Motion Capture of Skinned Models in a Four Camera Set-up using Optical Flow and Silhouettes

L. Ballan and G. M. Cortelazzo [PDF] [video] [bibtex] [dataset]
(Best Paper Award)
3DPVT 2008, Atlanta, GA, USA

Acquiring Shape and Motion of Interacting People from Videos

L. Ballan [PDF] [video] [bibtex] [dataset]
PhD's thesis, Department of Information Engineering, University of Padova, 2009