Recovering Baseline and Orientation from Essential Matrix Berthold K.P. Horn January 1990 Abstract: Certain approaches to the problem of relative orientation in binocular stereo (as well as long-range motion vision) lead to an encoding of the baseline (translation) and orientation (rotation) in a single 3 × 3 matrix called the “essential” matrix. The essential matrix is defined by E = BR, where B is the skew-symmetric matrix that satisfies Bv = b × v for any vector v, with b being the baseline and R the orientation. Shown here is a simple method for recovering the two solutions for the baseline and the orientation from a given essential matrix using elementary matrix op-erations. The two solutions for the baseline b can be obtained from the equality bbT = 1 Trace(EET)I − EET,2 where I is the 3 × 3 identity matrix. The two solutions for the orientation can be found using (b · b) R = Cofactors(E)T − BE, where Cofactors(E) is the matrix of cofactors of E. There is no need to perform a singular value decomposition, to transform coordinates, or to use circular functions, and it is easy to see that there are exactly two solutions given a particular essential matrix. If the sign of E is reversed, an additional pair of solutions is obtained that are related the two already found in a simple fashion. This helps shed some light on the question of how many solutions a given relative orientation problem can have. 1. Coplanarity Condition in Relative Orientation Relative orientation is the well-known photogrammetric problem of recov-ering the position and orientation of one camera relative to another from five or more pairs of corresponding ray directions [Zeller 52] [Ghosh 72] [Slama et al. 80] [Wolf 83] [Horn 86, 87b]. Relative orientation has to be determined before binocular stereo information can be used to recover[  2 surface shape. The same is true of the use of image feature correspon-dences in long-range motion vision (but not in the case of short-range motion vision, where motion can be treated as infinitesimal and rotations can conveniently be represented by vectors [Horn & Weldon 88].) Let b be the baseline (translation of the right center of projection with respect to the left center of projection), while  and r are the rays from the left and right centers of projection to a given point in the scene. These vectors are all measured in the coordinate system of the left camera. The coplanarity condition expresses the fact that the rays from the left and the right centers of projection meet, that is, that for some α and β, α = b + β r . (1) The well-known triple product form of the coplanarity condition,  br] = 0, (2) is obtained by simply taking the dot-product of both sides of this equation with b × r . Since the triple product is the volume of the parallelepiped formed by the three vectors, we see that this is equivalent to requiring that the vectors be coplanar [Zeller 52] [Thompson 59]. Let r be the ray direction from the right center of projection measured in the right coordinate system. Then r= Rr, where R is the orientation (the rotation that aligns the right coordinate system with the left coordi-nate system). The coplanarity condition can then be written in the form  · b × (Rr) = 0, (3) or T BRr = 0, (4) where B is a skew-symmetric matrix defined by Bv = b × v for all vectors v [Thompson 59]. The coplanarity condition can thus be transformed into T Er = 0, (5) where E = BR is the so-called essential matrix [Longuet-Higgins 81] [Tsai & Wang 84]. Note that there must be three constraints on the nine elements of the essential matrix, since there are only six degrees of freedom (three for the baseline and three for the orientation). The essential matrix can be found given five correspondences be-tween pairs of rays in the left and right coordinate system—we do not, however, discuss here how this may be done [Horn 87b] [Faugeras & May-bank 89] [Holt & Netravali 90]. The essential matrix can also be found using linear methods, when eight pairs of corresponding rays are given [Longuett-Higgins 81]. But such methods do not enforce the required non-linear constraints on the elements of an essential matrix and thus will3 2. Decomposing the Essential Matrix produce a matrix that is not decomposable unless the data is absolutely perfect [Longuett-Higgins 84]. They also do not make effective use of the redundant information provided by eight ray pair correspondences. If T Er = 0, then T(kE)r = 0, for an arbitrary constant k. Thus if E is an essential matrix for a particular set of corresponding ray pairs, so is kE for non-zero k. A particular essential matrix has two unique de-compositions into a baseline and an orientation. A particular set of ray correspondences, however, does not fix the scale of E. Variations in the scale of the essential matrix are reflected in changes in the magnitude of the implied baseline b. Thus while a particular essential matrix corre-sponds to a fixed length of baseline, a set of ray correspondences does not. This is referred to as the scale-factor ambiguity. The essential matrix is primarily of theoretical interest, since it is useful only when exactly five ray correspondences have been found. In practice one typically applies least-squares methods to many more than five ray pairs in order to attain reasonable accuracy [Zeller 52] [Ghosh 72] [Slama et al. 80] [Wolf 83] [Horn 86, 87b]. If one has more than five ray correspondences, one could use least-squares methods to find a “best-fit” essential matrix, provided one enforces three non-linear conditions that ensure that the matrix is decomposable. This has not proven feasible. If, on the other hand, the three conditions are not enforced, then the resulting matrix will not be decomposable, and the methods described here (or elsewhere) should not be applied, since the elements of the matrix are then inconsistent. Applying an algorithm that assumes that the data is consistent can in this case lead to very poor results. At the very least, one should try to find a baseline and a rotation that yields an essential matrix as close as possible in the least-squares sense to the given inconsistent matrix. This too is a difficult problem. Since the “real” problem is a