[

Most methods proposing to solve pose estimation problems at the level of objects using ellipsoid modeling are based on the projective geometry formalism. Any quadric $Q$ is thus linked to its projected conic $C$ by the Equation

where $P=K[R$ $t]$ is the camera projection matrix (Hartley2004) and $Q^{\ast },C^{\ast }$ are the dual forms of $Q$ and $C$ ($M^{\ast }=M^{-1}$ up to a scale factor for a non-singular symmetric matrix $M$). It is important noting that, under that formalism, ellipses and ellipsoids can be distinguished from other conics and quadrics only by certain algebraic conditions on $C$ and $Q$ entries, these conditions being difficult to leverage in practice.

Quadric-modeling of objects has often been implemented in the context of Semantic SLAM (ROB-059) for improving the process accuracy through multi-objective optimization (QSLAM; abs-2110-08977; abs-2004-05303). On a theoretical level, Crocco_2016_CVPR addresses the object-based Structure-from-Motion (SfM) problem and introduces an analytical solution to reconstruct both quadric and affine camera poses. The problem is solved in a least square sense with a matrix $P$ over-represented by a $6\times 10$ Kronecker product $P\otimes P$. This work is extended with CAD model priors in Gay_2017_ICCV, while 7919240 present a closed-form solution to the problem of reconstructing a quadric from three calibrated pinhole camera views in which the object projections are detected. However, in this method, nothing ensures that the reconstructed quadric is an ellipsoid, forcing the authors to add a costly post-processing non-linear optimization of the results.

Another limitation while using homogeneous quadrics and over-representation of $P$ appears in P12Q. In this method, the so-called gold-standard algorithm used to retrieve the camera projection matrix from 2D-3D point correspondences (Hartley2004) is adapted to conic-quadric correspondences. To compute $P$, 12 conic-quadric pairs are required whereas only 6 point pairs are sufficient in the same context.

We argue that these limitations may be due in part to the fact that ellipse and ellipsoid homogeneous formulations are not clearly enough distinguished from other members of their geometric families, and also to a non-minimal representation of the projection matrix. In our paper and to overcome these difficulties, we thus propose an ellipsoid-specific theoretical framework and highlight its advantages.

In WokesP10, a comprehensive study of the spheroid pose estimation problem is conducted. To this end, the authors introduce a spheroid-specific parameterization of the problem that enables solving it but prevent from extrapolating the method to the general case of ellipsoids.

In a nutshell, the authors demonstrates that the spheroid pose estimation problem has two distinct solutions. In 6.4, we retrieve the same result by restricting our general formulation to the case of spheroids.

Following the notations introduced in Eberly-backproj and presented in Fig. 1, we consider an ellipsoid $(C,A)$ defined by Equation

where $C$ is the center of the ellipsoid, $A$ is a real positive definite matrix characterizing its orientation and size, and $X$ is any point on it.

Given a center of projection $E$ and a projection plane of normal $N$ which does not contain $E$, the projection of the ellipsoid is an ellipse of center $K$ and of semi-diameters $a$ et $b$. Its principal directions are represented by unit vectors $U$ and $V$, such that $\{U,V,N\}$ is an orthonormal set.

While Equation (1) has been established in the camera coordinate frame, it is also valid in the ellipsoid coordinate frame, where matrix $A_{ell}$ is diagonal, and in the cone coordinate frame where $B_{cone}^{\prime }$ is diagonal. Since $A$ and $B^{\prime }$ are symmetric, both can be diagonalized using an orthogonal matrix $P$ such that $P^{-1}=P^T$. For that reason, Equation (1) remains the same whatever the choice of the coordinate frame (camera, ellipsoid or cone) in which matrices and vectors are expressed. In the following, if there is no restriction on the coordinate frame, we will adopt notations without subscript. Otherwise, subscripts $cam$, $cone$ or $ell$ will be used.

One can theoretically distinguish between camera pose estimation, which consists in estimating the pose of the camera with respect to its environment, and its reference frame counterpart, i.e. object pose estimation, which consists in estimating the pose of an object with respect to the camera. Fundamentally, the sought transformations are the inverse of each other thus, in this paper, we may focus on estimating the pose of the ellipsoid or the pose of the camera, according to the most convenient setup for mathematical developments, without loss of generality.

The givens of the problem are the ellipse detected in the image, the camera intrinsic parameters and the ellipsoid size (i.e. lengths of its three radii). Therefore, $B_{cone}^{\prime }$, $B_{cam}^{\prime }={}^{cam}{R}_{cone}{B}_{cone}^{\prime }{}^{cam}{R}_{cone}^{\top }$ and $A_{ell}$ are known, as well as $E_{cone}$, $E_{cam}$ and $C_{ell}$ (whose entries are all zero). Since expressions of $A$ and $B^{\prime }$ in specific coordinate frames are known, their eigenvalues are also known. In addition, matrix properties such as trace and determinant, that are fully constrained by the eigenvalues, are also known.

In the paper, we most often work in the camera frame. We thus aim at computing vector ${\Delta }_{cam}=E_{cam}-C_{cam}$ and matrix

from which we then retrieve the ellipsoid position $C_{cam}$ and orientation ${}^{cam}{R}_{ell}$.

In the other case, we aim at computing ${\Delta }_{ell}$ and $B_{ell}^{\prime }$ to then derive the camera position $E_{ell}$ and orientation ${}^{cam}{R}_{ell}^{\top }$.

Finally, vector $\Delta$ encodes the relative position between the ellipsoid and the camera, while couple $\{A,B^{\prime }\}$ characterizes their relative orientation, denoted ${}^{cam}{R}_{ell}$. Indeed, its expression in the camera frame is

while its expression in the ellipsoid frame is

recalling that $A_{ell}$ and $B_{cam}^{\prime }$ are known.

Solving Equation (1) therefore consists in determining the value(s) of $\sigma$ and the expressions of $A,B^{\prime },\Delta$ in a common coordinate frame (camera or ellipsoid).

Let $(A,B^{\prime },\Delta ,\sigma )$ be a set of solutions of Equation (1). Result 1 shows that they are also solutions of a generalized eigenvalue problem (GoluVanl96).

Since $B^{\prime }$ is invertible, the generalized eigenvectors and eigenvalues of pair $\{A,B^{\prime }\}$ are the eigenvectors and eigenvalues of matrix $B^{\prime -1}A$.

Let’s denote ${\sigma }_1$ (multiplicity 1) and ${\sigma }_2$ (multiplicity 2) the two generalized eigenvalues of $\{A,B^{\prime }\}$.

Result 4 demonstrates that the secondary scalar variable $\mu$ is also closely linked to the generalized eigenvalues of $\{A,B^{\prime }\}$.

Since ${\sigma }_1$ and ${\sigma }_2$ are of opposite signs, Equation (5) shows that $\mu <0$.

Result 5 now highlights the connection between $\mu$ and $\sigma$.

In this case, $A_{cam}$ or $B_{ell}^{\prime }$ is known. Since $A_{ell}$ and $B_{cam}^{\prime }$ are also known, eigenvalues of $\{A,B^{\prime }\}$, $\sigma$ (using Result 3) and $\mu$ (Result 4) can be retrieved. Result 7 then provides that $\Delta$ is unique and fully determined.

Result 7 highlights the fact that solving Equation (1) may consist only in determining $A_{cam}$ or $B_{ell}^{\prime }$. Indeed, $\sigma$ and $\Delta$ can then be uniquely derived. In other words, the relative position is fully constrained by the orientation.

As mentioned above, this result is of high practical interest. Indeed, getting the camera orientation, e.g. from physical sensors or image analysis, is usually easier than getting the camera position, and especially indoors where the GPS is useless. In a multi-object scene, the fact that only one ellipse-ellipsoid association is needed to compute the pose allows using a RANSAC-like strategy with low combinatorial cost both to detect the wrong associations and to choose the correct one when a label is shared by several objects.

A re-localization algorithm based on this strategy was presented in (ISMAR). The system operates in real time from YOLO detections (YOLOv3) and IMU data or vanishing points – both methods were assessed. Figure 2 shows a few qualitative results obtained with images from the standard RGB-D TUM dataset (SturmEEBC12). Quantitative results as well as a detailed analysis of the advantages and limitations of this algorithm can be found in (ISMAR). Other applications of Result 7 are presented in (IROS; IROS2; RAL).

In this case ${\Delta }_{cam}$ or ${\Delta }_{ell}$ is known. $A_{cam}$ and $B_{ell}^{\prime }$ are unknown, but since their eigenvalues are known, $det\left(A\right)$, $tr\left(A^{-1}\right)$, $det\left(B^{\prime }\right)$ and $tr\left(B^{\prime -1}\right)$ are known. $\sigma$ can thus be deduced from Result 6 and $\mu$ from Result 5. Result 8 then explains how to retrieve $A_{cam}$ or $B_{ell}^{\prime }$ and how to derive the orientation in closed-form, up to the cone or ellipsoid symmetries.

Once for instance $A_{cam}$ is retrieved, one can compute its eigenvalue decomposition:

For a triaxial ellipsoid ($a\ne b\ne c$), there are 4 solutions for ${}^{cam}{R}_{ell}$. Therefore, the ellipsoid orientation can be analytically derived from its position up to the ellipsoid symmetries.

In this paper, we address the full ellipsoid pose estimation problem, i.e. we cover every possible types of ellipsoids and thus cones (see Appendix D). However, a specific type of ellipsoid cannot necessarily be tangent to any type of cone, what we refer to as possible or impossible co-occurence.

Obviously, only circular cones can be tangent to a sphere. Furthermore, we are going to prove that only a non-circular elliptic cone can be tangent to a triaxial ellipsoid.

Let’s prove it by contradiction, and assume that the projection cone has a revolution axis (circular cone).

Let’s also assume that the ellipsoid center $C$ does not belong to that axis. Since the ellipsoid is tangent to the cone, any new ellipsoid obtained by rotating the original one around the cone revolution axis shall still be tangent to the cone, thus be solution of (1). Yet, in this case, the locus of ellipsoid centers would be a circle located in a plane orthogonal to that axis and whose center would belong to it. Every center would thus be at a fixed distance to the cone vertex $E$, whence there would be an infinite number of $\Delta$ solutions for the same $\sigma$, given (8). However, this contradicts Equation (19). Therefore, the center of the ellipsoid must belong to the revolution axis of the cone.

If the ellipsoid center belonged to the cone revolution axis, then $\Delta$ would be parallel to that axis, i.e. would be an eigenvector of $B^{\prime }$, whence of $A$ given (1’). However, in such a case, the symmetries of the cone-ellipsoid pair would impose that the tangent ellipse (intersection between the ellipsoid and the polar plane derived from $E$ (Wylie)) belongs to a plane orthogonal to the cone revolution axis, that is also a principal axis of the ellipsoid. Therefore, that tangent ellipse should be both a circle (orthogonal section of a circular cone) and a non-circular ellipse (section of an ellipsoid by a plane parallel to one of its principal planes), which is impossible. Therefore, the cone cannot have a revolution axis.

In brief, Table 1 summarizes the possible and impossible co-occurences between ellipsoid and cone types.

In the rest of this section, we determine the solutions of the Cone Alignment Equation (1), and derive the camera-ellipsoid relative poses. To this end, we distinguish between the three different types of ellipsoids (triaxial, spheroid, sphere). We demonstrate, in particular, that

• there is an infinite number of triaxial fixed-size ellipsoids that are tangent to a given backprojection cone (Fig. 2),

• as already demonstrated in WokesP10, there are only two fixed-size spheroids solutions (see Fig. 4).

In the first case, the infinite number of ellipsoids tangent to the cone (or, conversely, the infinite number of cones tangent to the ellipsoid) explains the infinite number of camera solutions (see Fig. 5), and provides a parameterization of them. In the second case, the infinite number of change of basis matrices between the spheroid and the camera explains the infinite number of camera solutions (see Fig. 6). The mathematical developments leading to these results are presented below.