Grasp’D: Differentiable Contact-rich
Grasp Synthesis for Multi-fingered Hands

SDF representation. For the purpose of collision detection, the hand is represented by a set of surface points ${\text{X}}_{\text{h}}$, and the object to grasp is represented by its Signed Distance Function (SDF), $\varphi \left(x\right)$ (similar to [fuhrmann2003distance, macklin2020local, bender2014continuous]). The SDF maps a spatial position $x\in R^3$ to its distance to the closest point on the surface of the object, with a negative or positive sign for interior and exterior points, respectively [osher2006level]. The object surface can be recovered as the zero level-set of the SDF: $\left\{x|\varphi \right(x)=0\}$. The gradient of the SDF $\nabla \varphi \left(x\right)$ is always of unit magnitude, corresponds to the surface normal for $x$ on the object surface, and yields the closest point on the object as $x-\varphi \left(x\right)\nabla \varphi \left(x\right)$. SDF representations are well-suited to differentiable collision detection [macklin2020local], since contact forces can be written in terms of a penetration depth ($\varphi$) and normal direction ($\nabla \varphi$), for which gradients can be computed as $\nabla \varphi$ and ${\nabla }^2\varphi$, respectively.

Coarse-to-fine smoothing. To successfully optimize contact locations over non-smooth object geometry we employ surface smoothing in a coarse-to-fine way. At the start of each optimization, we define the object surface not as the zero level-set of the SDF, but as the radius $r$ level-set: $\left\{x|\varphi \right(x)=r>0\}$, which gives a smoothed and padded version of the original surface. As optimization continues, we decrease $r$ on a linear schedule until it reaches $0$, yielding the original surface. This coarse-to-fine smoothing allows gradient-based optimization to effectively move contact points across discontinuities and prevents the optimization from quickly overfitting to local geometric features. We set $r$ to approximately 10cm at the start of each optimization. Details are in Appendix A.2.

Contact forces. We use a primal (penalty-based) formulation of contact forces, which allows us to compute derivatives with autodiff [baydin2018automatic] and keep a consistent memory footprint. For a given point $x\in {\text{X}}_{\text{h}}$, the resultant contact force is

where $f_n$ is the normal component, proportional to penetration depth $\varphi \left(x\right)$, and $f_t$ is the frictional component, computed using a Coulomb friction model. $k_n$ and $k_f$ are the normal and frictional stiffness coefficients, respectively, $\mu$ is the friction coefficient, and $v_t$ is the component of relative velocity between hand and object at the contact point $x$ that is tangent to the contact normal $\nabla \varphi \left(x\right)$.

Leaky gradients. At any one time, most possible hand-object contacts are inactive – a property we refer to as contact sparsity. Since an infinitesimal perturbation to hand pose will not activate these contacts (i.e., will not make them touch the object), the gradient of their contact forces with respect to hand pose is zero, i.e., $\partial f_{\text{c}}/\partial q=\partial f_{\text{\%}c}/\partial \dot{q}=\partial f_{\text{c}}/\partial \ddot{q}=0$. When the hand is not touching the object, all contacts are inactive and gradient-based optimization can get stuck in a plateau. We work around this by computing a leaky gradient for the normal force term. From equation (4), we have $\frac{\partial \parallel f_n\parallel }{\partial q}=0$ if $\varphi \left(x\right)\ge 0$ but we instead set

where $\alpha \in [0,1]$ controls how much gradient leaks through the minimum. We set $\alpha =0.1$ in our experiments.

Simulation setup. To compute the grasp metric, we simulate the rigid-body interaction between a hand and an object. The hand is kinematic (does not react to contact forces), while the object is dynamic (thus subject to contact forces). The simulator state is given by the configuration vector $q$ and its first and second time derivatives $\dot{q},\ddot{q}$. $q$ is composed of hand and object components $q=(q_{\text{h}},q_{\text{o}})$ with corresponding spatial coordinates $u=(u_{\text{h}},u_{\text{obj}})$. The object is always initialized with the same configuration $q_{\text{o}}^{\left(0\right)}$: unrotated and untranslated at the origin. Given a state $q^{\left(t\right)}$, following equations (1) and (2), our simulator uses a semi-implicit Euler update scheme to compute subsequent state $q^{(t+1)}$.

Computing the grasp metric by simulation. To measure the quality of a candidate grasp $q_{\text{h}}$, we test its ability to withstand forces applied to the object. Given an initial state $q^{\left(0\right)}=(q_{\text{h}},q_{\text{o}}^{\left(0\right)})$, we apply an initial velocity ${{\dot{q}}_{\text{o}}}^{\left(0\right)}$ to the object. The hand is kept static, with ${\dot{u}}_{\text{h}}=0$. We run forward simulation to compute the object’s final velocity ${{\dot{u}}_{\text{o}}}^{\left(T\right)}$. A stable grasp will produce contact forces that resist the object velocity, so lower $\parallel {{\dot{u}}_{\text{o}}}^{\left(T\right)}\parallel$ indicates a more stable grasp. In fact, a stable grasp should be able to resist object velocities in any direction, so we perform multiple simulations with different initial velocities and average the results. This suggests the following basic grasp metric: for each set of $M$ simulations, indexed by $m=\{1,\dots ,M\}$, we set a different initial object velocity, run the simulation, and record $L_m=\parallel {{\dot{u}}_{\text{o}}}^{\left(T\right)}\parallel$. Then, averaging, we have

Since $L_{\text{grasp}}$ is a differentiable function of the output of a differentiable simulation, it is itself differentiable with respect to $q_{\text{h}}$, and we can compute loss gradients $\partial L_{\text{grasp}}/\partial q_{\text{h}}$ and use gradient-based optimization to find stable grasps.

Problem relaxation. Inspired by Contact-Invariant Optimization [mordatch2012discovery, mordatch2012contact] we relax the problem making it more forgiving to gradient-based optimization. Specifically, we introduce additional desired or prescribed contact force variables. This allows us to model physics violations as a cost rather than a constraint. For each surface point on the hand $x^i\in {\text{X}}_{\text{h}}$, we introduce a 6-dimensional vector ${\hat{f}}_{\text{c}}^i$ representing the desired hand-object contact wrench arising from contact at $x^i$.

Additional heuristic losses. We include some additional losses that improve the plausibility of resulting grasps. Most hand models have defined joint range limits. Let $q_{\text{h}}^{\text{low}}$ and $q_{\text{h}}^{\text{up}}$ be the lower and upper joint limits respectively. $L_{\text{range}}$ encourages hand joints to be near the middle of their ranges. $L_{\text{limit}}$ penalizes hand joints outside of their range. $L_{\text{inter}}$ penalizes self intersections of the hand.

The hand is kinematic, so it is not subject to contact forces. However, we still compute forces arising from contact between the hand links, for use in this loss term, as $f_{\text{link}}$. We ignore contacts between neighbouring links in the chain. For the purpose of computing $f_{\text{link}}$, we represent each hand link as both a point set and an SDF and compute $f_{\text{link}}$ according to equations (3), (4), and (5).

Evaluation metrics. We follow [hasson2019learning] and use contact area (CA), intersection volume (IV), and the ratio between contact area and intersection volume ($\frac{CA}{IV}$). We compute evaluation metrics that measure grasp stability and contact area. In addition, we measure the contact area each grasp creates and the volume of hand-object interpenetration. We compute two analytic measures of stability – the Ferrari-Canny (epsilon $ϵ$) [ferrari1992planning] and the volume metric (Vol) – and one simulated measure: the simulation displacement (SD) metric introduced in [hasson2019learning].

Hand parameterization. We use a differentiable PyTorch layer [hasson2019learning] to compute the 773 vertices of the MANO hand [MANO:SIGGRAPHASIA:2017] model. The input is a set of weights for principal components extracted from the MANO dataset of human scans [MANO:SIGGRAPHASIA:2017]. We find that this PCA parameterization provides a useful prior for human-like hand poses. We use the maximum number of principal components (44).

Data. We evaluate our approach to grasp synthesis by generating grasps with the MANO human hand [MANO:SIGGRAPHASIA:2017] model for 57 ShapeNet [chang2015shapenet] objects that span 8 categories (bottles, bowls, cameras, cans, cellphones, jars, knives, remote controls), and are each considered at 5 different scales (as in ObMan). See the Appendix A for details of mesh pre-processing, initialization, simulation, and optimization.

Results. Results are presented in Table 2. Grasps generated by our method (both top-2 and top-5) have a contact area of around $42{cm}^2$. This is higher than the $\sim 20{cm}^2$ area achieved with fingertip only grasps [brahmbhatt2019contactdb] and about $4\times$ higher than grasps from the ObMan dataset (top-2 or top-5). These contact-rich grasps achieve modest improvements in analytic measures of stability, and a significant reduction in simulation displacement ($\sim 3\times$ for top-2 grasps). Visualizations of our generated grasps in Figure 1 confirm that these grasps achieve larger areas of contact by closely conforming to object surface geometry, whereas the analytically generated grasps largely make use of fingertip contact only. These higher contact grasps have accordingly higher interpenetration, but the ratio between contact area and intersection volume is similar to the ObMan baseline.

Setting. One possible application of our method is to direct grasp prediction from RGB-D images by simulation on reconstructed object models. Currently, our method is too slow to be used online (about 5 minutes per grasp), but as simulation speeds increase and recent works in implicit fields push reconstruction accuracy higher and higher, we believe that grasp prediction by simulation models will become increasingly viable. To validate the plausibility of using our method with reconstructed object models, we present results from running our system on meshes reconstructed from RGB-D inputs. We synthesize grasps based on RGB-D (with camera pose) inputs from the YCB object dataset [calli2017yale]. In addition to reconstructed meshes, the YCB dataset provides the original RGB-D captures the meshes are based on. Each object was captured from 5 different cameras at 120 different angles for a total of 600 images. To confirm that our method can work with reconstructions done under more realistic assumptions, we limit our reconstructions to using 5 different angles from 3 cameras (2.5% of captures).

Data. For a subset of the YCB objects, we generate Poisson surface reconstructions and use our method to synthesize MANO hand grasps. Since the inputs are from cameras with a known pose, the object reconstruction is in the world frame. Details in the Appendix A.4.

Results. Our results confirm the viability of using simulation to synthesize grasps on reconstructed object models. Qualitative results are presented in Figure 4; additional results can be found in Appendix D. Although synthesis does not perform as well as with ground-truth models, plausible human grasps are discovered for many objects and the grasps appear well-aligned with the real-world object poses. Future work could take advantage of learning-based reconstruction methods to achieve grasp synthesis with fewer input images.

Results. We adopt the same data as in Section 4.2. Table 3 presents the results. Our relaxed problem formulation is key to our method’s success, and without it, performance greatly degrades by all measures, with discovered grasps creating very little contact (low contact area and intersection volume). Coarse-to-fine smoothing has a modest impact, with all metrics comparable with or without smoothing, except for simulation displacement, which is about 25% higher without smoothing. We did not include a variant without leaky gradient, since this variant would never make contact with the object (if the hand is not touching the object at initialization, there will be no gradient to follow and optimization will immediately be stuck in a plateau).