Robust computation of projective structure

• Choose the first camera as
  $${\bf P} = \left[ {\bf I} \mid {\bf0} \right]$$
  which corresponds to the world coordinate system having its origin at the optical center of the first camera, and its axes aligned with the camera axes. It is always possible to make this choice.
• The choice of the second camera consistent with the fundamental matrix is
  $${\bf P'} = \left[ {\bf M'} \mid {\bf e'} \right]$$
  and the fundamental matrix is given as:
  $${\bf F} = {\bf e'}_{\times}{\bf M'}$$

• ${\bf e'}$ is the projection of the optical center of the first camera, $({\bf0}^T,1)^T$ , onto the second image
  $${\bf P'}\left[ \begin{array}{c} {\bf0} \\ 1 \end{array} \right] = {\bf e'}$$

• Similarly, the optical center of the second camera is is $(({\bf M'}^{-1}{\bf e'})^T,1)^T$ , and its projection onto the first image is ${\bf e}$ .
  $${\bf P}\left[ \begin{array}{c} {\bf M'}^{-1}{\bf e'} \\ 1 \end{array} \right] = {\bf M'}^{-1}{\bf e'} = {\bf e}$$

• Given ${\bf P}$ and ${\bf P'}$ , the 3D point ${\bf X_i}$ corresponding to a pair of image point ${\bf x_i}$ and ${\bf x'_i}$ can be computed by intersecting the back-projected rays
  $$\begin{array}{ccc} {\bf X_i}& =& \left[ \begin{array}{c} {\bf0} \... ...t[ \begin{array}{c} {\bf M'}^{-1}{\bf x_i} \\ 0 \end{array} \right] \end{array}$$

To summarize:

1. Compute the fundamental matrix ${\bf F}$ from ${\bf x_i} \leftrightarrow {\bf x'_i}$ .
2. Decompose ${\bf F}$ as ${\bf F} = {\bf e'}_{\times}{\bf M'}$ .
3. Compute 3D points ${\bf X_i}$ by intersecting back-projected rays using
   $${\bf P} = \left[ {\bf I} \mid {\bf0} \right]$$    and $${\bf P'} = \left[ {\bf M'} \mid {\bf e'} \right]$$