The question
Chapter 2 showed that a pixel only fixes a ray. With the same point seen in two or more calibrated cameras we can intersect the rays and recover the 3D point: triangulation. It is the step that turns matches into a point cloud in SfM (COLMAP) and SLAM, and how Gaussian Splatting gets its initial points.
1. Rays don't meet
With perfect pixels all rays pass through X. Real detections have noise of about a pixel, so in 3D the rays are skew lines that miss each other. Triangulation is therefore a least-squares problem: find the point that best agrees with all observations.
2. Linear triangulation (DLT)
Each view gives x ~ P X̃, i.e. x × (P X̃) = 0. Two of the three equations are independent:
Here pₖᵢᵀ is row k of Pᵢ. The lab uses normalized coordinates (P = [R | t], x = K⁻¹x) so that the numbers in A are well scaled. With noise-free data the smallest eigenvalue is exactly 0; noise makes it positive. The gold standard is to then refine X by minimizing reprojection error (non-linear least squares, as in bundle adjustment).
3. Baseline and depth uncertainty
When the rays are nearly parallel (small baseline compared to depth), a 1-pixel error moves their intersection a lot along the viewing direction. Roughly, for two views
— depth error grows with the square of distance and shrinks with baseline b and focal length f. Try Shrink baseline: the reconstructed house stretches along the viewing direction.
4. Three views: epipolar transfer
Given matches x₁ and x₂, where must the point appear in view 3? On the epipolar line of x₁ and on the epipolar line of x₂, so at their intersection:
A third view therefore adds a strong check: a wrong match from views 1–2 predicts a point in view 3 where there is nothing. The transfer breaks down when the 3D point lies in the trifocal plane (the plane through the three camera centers): the two lines then coincide. The complete three-view relation, valid even there, is the trifocal tensor.
5. Things to try
- Set noise σ to 0: the rays meet, the error is 0 and the smallest eigenvalue is 0.
- Raise σ to 3 px and toggle use view 3: averaging over three views reduces the error.
- Run Shrink baseline and watch the reconstruction smear in depth while the reprojection errors stay small.
- Hover a point and check that the two colored lines in each view cross near its observation.