What is this?
3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) represents a scene as millions of small, colored, semi-transparent 3D Gaussians and renders them by "splatting" each one onto the image as a 2D Gaussian. This page follows a single Gaussian through that projection; chapter 2 blends many of them into an image. It uses the pinhole camera from the Multi-view Geometry Explainer — K, R, t should be familiar.
1. One Gaussian, four attributes
- Position μ ∈ ℝ³ — the center.
- Covariance Σ ∈ ℝ³ˣ³ — size and orientation of the ellipsoid.
- Opacity o ∈ [0, 1] — how much it blocks what is behind it.
- Color — in 3DGS, spherical-harmonics coefficients so the color can change with viewing direction (here: one constant color).
Its density is G(x) = exp(−½ (x − μ)ᵀ Σ⁻¹ (x − μ)). The surfaces where G is constant are ellipsoids; the 1σ ellipsoid has the scales as its half-axes.
2. Why Σ = R S Sᵀ Rᵀ
A covariance matrix must be symmetric positive semi-definite. If you let gradient descent update the 6 entries of Σ freely, it quickly stops being one. 3DGS therefore stores a scale vector s (as log s) and a rotation as a unit quaternion q, and builds
which is valid for any values. The columns of R·S are the ellipsoid's half-axes (the colored axes in the 3D view).
3. Projecting a Gaussian
A perspective projection of a Gaussian is not a Gaussian. EWA splatting (Zwicker et al., 2001) linearizes the projection around the center and maps the covariance through that local affine map:
The approximation is excellent when the Gaussian is small compared to its distance, and degrades for big Gaussians close to the camera or far off the optical axis. Compare the white ellipses (EWA) with the orange dashed one, computed from actually projected random samples.
4. From Σ' to pixels and tiles
- The splat's opacity at pixel p is α = o · exp(−½ dᵀ Σ'⁻¹ d), d = p − μ'. 3DGS stores Σ'⁻¹ as the "conic", skips α < 1/255 and caps α ≤ 0.99.
- The eigenvalues λ₁ ≥ λ₂ of Σ' give the ellipse axes; the splat is cut at r = ⌈3√λ₁⌉ pixels.
- The image is split into 16×16-pixel tiles; each Gaussian is listed in every tile its 3σ square touches. The GPU then renders each tile independently — the key to real-time speed.
- The 0.3 px dilation makes every splat at least about a pixel wide, a cheap low-pass filter against aliasing of tiny Gaussians.
5. Things to try
- Make one scale tiny: the Gaussian becomes a flat disc — 3DGS scenes are full of such "surfels".
- Run Fly close with a large Gaussian: the orange (true) and white (EWA) ellipses separate.
- Run Pan off-axis: the splat stretches towards the image border even though the Gaussian did not change.
- Shrink all scales to the minimum and toggle the dilation.
Roadmap
- One Gaussian — parameterization and projection this page
- Rendering — sorting and α-blending many splats, tiles
- Optimization — fitting Gaussians to images with gradient descent, densification
- Gaussian Splatting SLAM — tracking and mapping with Gaussians (towards VarSplat)