From splats to an image
Chapter 1 turned one 3D Gaussian into a 2D splat with center μ'ᵢ, covariance Σ'ᵢ, opacity oᵢ and color cᵢ. A scene has millions of them. To color a pixel, 3DGS takes all splats that overlap it, sorted by depth, and blends them front to back:
This is the same volume-rendering equation NeRF uses, but evaluated on a handful of explicit primitives instead of hundreds of samples along a ray — which is why 3DGS renders at 100+ FPS.
1. Sorting
The formula is only correct if splats are in front-to-back order. 3DGS sorts once per frame by the depth of each Gaussian's center (a GPU radix sort on a key that combines tile id and depth). Turning sorting off in the Lab shows what happens otherwise: the ground, listed first, is blended as if it were in front of everything. Sorting by center depth is itself an approximation — two intersecting Gaussians can "pop" when their centers swap order.
2. Tiles and early termination
- The image is cut into 16×16 tiles. Each Gaussian is duplicated into every tile its 3σ square touches, so each tile gets its own short, sorted list, and one GPU thread block renders one tile.
- A pixel stops as soon as its transmittance drops below 10⁻⁴: nothing behind can change it visibly. Opaque foreground saves a lot of work.
- The heat map shows how many splats each tile has to process. Large splats (scale modifier) cost more.
3. Why it is differentiable
Every step — projection, α, transmittance, the sum — is a smooth function of μ, Σ, o and c (except the sort order, which is treated as constant). So the rendered image can be compared to a photo and the error back-propagated to every Gaussian. That optimization is the next chapter.
4. Things to try
- Hover the purple glass panel: you see it and what is behind it, each weighted by αT.
- Turn off sort by depth and orbit the camera.
- Run Build up: splats are added in depth order, nearest first.
- Lower the scale modifier to see the individual Gaussians, raise it to see them merge.