Understanding 3DGS, Part 2: How Is 3DGS Different from Photogrammetry?

In the previous article, we explained the differences between 3DGS and traditional 3D meshes.

To briefly recap, a 3D mesh represents a three-dimensional shape by combining vertices and faces. Textures are applied to its surface to reproduce colors, patterns, and surface details.

Many of the common 3D models used in games and video production are created using meshes. In traditional 3D production, specialists known as 3D modelers use dedicated tools such as Blender to build shapes and surface details.

3DGS, on the other hand, is a technology that reconstructs real-world spaces from multiple photographs and represents them as 3D scenes with a photographic appearance by placing large numbers of small elements called Gaussians.

However, 3DGS was not the first technology capable of turning real-world spaces into 3D from multiple photographs.

One representative example is photogrammetry.

Photogrammetry is a technology that analyzes multiple photographs to estimate the shape of a subject and the positional relationships of the cameras, then generates a 3D model.

In typical photogrammetry workflows, the final output is a textured mesh.

In other words, photogrammetry automatically generates a 3D model from photographs, but the resulting data has a structure similar to the traditional mesh-based 3D models introduced in the previous article.

So, are 3DGS and photogrammetry completely different technologies?

Not exactly.

They share some of the same processes. In particular, the initial stage of estimating camera positions and orientations from multiple photographs involves very similar processing.

So how much do they have in common, and at what point do they begin to differ?

In this article, we will review the basic photogrammetry workflow and examine its similarities to and differences from 3DGS.

First, an Overview of the Photogrammetry Workflow

Photogrammetry is a technology for reconstructing the 3D shape of a subject or environment from multiple photographs.

For example, by photographing a building from different directions and analyzing those images, it is possible to generate a 3D model of the building.

A typical workflow looks like this:

1. Capture photographs from multiple viewpoints
2. Match feature points between the photographs
3. Estimate the camera positions and orientations
4. Generate a Sparse Point Cloud
5. Generate a Dense Point Cloud
6. Generate a mesh and textures

Two particularly important processes here are SfM (Structure from Motion) and MVS (Multi-View Stereo).

SfM: Estimating Camera Positions and 3D Structure from Photographs

SfM is a technology that matches feature points appearing in multiple photographs and uses that information to estimate the positions and orientations of the cameras, as well as the 3D coordinates of those feature points.

For example, suppose the same pattern on a wall appears in several photographs.

By determining where that pattern appears in each image, it is possible to estimate the positional relationships between the photographs.

Using correspondences across multiple images, the system can then determine the position and orientation of each camera when the photograph was taken, as well as the 3D positions of the feature points.

The results of this process are camera position and orientation information, together with a Sparse Point Cloud.

A Sparse Point Cloud is a sparse collection of points centered mainly around feature points. At this stage, the point cloud is not yet dense enough to represent the entire surface of an object.

MVS: Reconstructing Dense 3D Geometry from Multiple Photographs

Once the camera positions and orientations have been determined through SfM, the next step is MVS.

MVS uses multiple photographs whose camera relationships are already known to estimate the 3D position corresponding to each pixel.

This makes it possible to generate a Dense Point Cloud, which contains far more points than a Sparse Point Cloud.

A mesh is then generated from the point cloud, and textures are applied to complete a typical photogrammetry-based 3D model.

In other words, photogrammetry first determines camera positions and a sparse 3D structure from photographs, then proceeds to reconstruct denser geometry.

3DGS Also Starts with SfM

Now let us look at the 3DGS workflow.

3DGS also generally uses multiple photographs as input.

In many standard 3DGS workflows, SfM is performed first.

Just as in photogrammetry, feature points are matched between photographs, and the positions and orientations of the cameras are estimated.

The Sparse Point Cloud generated through SfM is also used.

This is an important point.

3DGS does not necessarily build a 3D space from scratch using a method completely separate from photogrammetry.

At least in common 3DGS pipelines, the camera information and point cloud obtained through SfM serve as the starting point for the subsequent processing.

In fact, representative 3DGS implementations can use SfM results generated by COLMAP (*1) as input.

*1: COLMAP is open-source software capable of performing SfM and MVS. It is also widely used in photogrammetry.

After SfM, Photogrammetry and 3DGS Take Different Paths

So what happens after SfM?

This is where the two approaches begin to differ significantly.

In photogrammetry, MVS is typically used to generate a dense point cloud, which is then used to create a mesh and textures.

In 3DGS, by contrast, a Sparse Point Cloud and other information are used as initial values, and Gaussians are then placed and optimized.

Put simply, a Gaussian is a small ellipsoid-like representation element positioned in 3D space.

Each Gaussian contains information such as its position, size, orientation, opacity, and color.

A large number of these Gaussians are placed and optimized so that the rendered images reproduce the input photographs as closely as possible, resulting in a 3D Gaussian scene.

A simplified comparison of the two workflows looks like this:

Photogrammetry

Photography
↓
SfM (estimate camera position and orientation)
↓
Sparse Point Cloud
↓
MVS
↓
Dense Point Cloud
↓
Mesh Generation
↓
Texture Generation

3DGS

Photography
↓
SfM (estimate camera position and orientation)
↓
Sparse Point Cloud
↓
Gaussian Initialization
↓
Gaussian Optimization
↓
3D Gaussian Scene Generation

As shown above, the two approaches share many steps up to SfM, but they differ afterward in how they represent 3D space and how that representation is optimized.

How Is the Generated Data Different?

Now let us also look at the differences in the data generated by each approach.

As mentioned earlier, the output produced by photogrammetry is generally a 3D mesh, similar to models handled by conventional 3D modeling software.

It has clearly defined surfaces, and textures can be applied to those surfaces.

In 3DGS, by contrast, the rendering units are small elements with spatial extent, and the scene is represented using properties such as position, shape, color, and opacity.

An important point to keep in mind is that the Gaussians in 3DGS do not necessarily correspond precisely to the actual surfaces of objects.

3DGS places greater emphasis on reproducing the input photographs as naturally as possible from different viewpoints.

As a result, even if the rendered scene looks highly realistic, the geometry extracted from it is not necessarily an accurate 3D model.

This is one of the major differences from traditional photogrammetry.

Is 3DGS a Replacement for Photogrammetry?

Based on what we have discussed so far, is 3DGS a replacement for photogrammetry?

The answer depends on the intended use.

For example, 3DGS can be a strong option when you want to:

• Reproduce a real-world space with high visual quality
• View a scene freely from different viewpoints
• Prioritize a photographic appearance
• Achieve rendering close to real time

On the other hand, traditional photogrammetry-based mesh generation may be more suitable when you want to:

• Measure dimensions or shapes accurately
• Use the result as material for CAD or 3D modeling
• Use it for physical simulation
• Edit or modify it as a conventional 3D model

Of course, research and technologies for generating meshes from 3DGS are also advancing.

However, the scenes produced by 3DGS and the mesh models produced by traditional photogrammetry are fundamentally designed to achieve different goals.

Rather than simply asking which technology is better, it is important to choose between them based on what you want to represent in 3D.

Conclusion

In this article, we explored the relationship between 3DGS and photogrammetry.

The three key points are:

1. 3DGS and photogrammetry have much in common at the SfM stage, where camera positions and orientations are estimated from multiple photographs.
2. After SfM, photogrammetry proceeds to dense geometry reconstruction and mesh generation through MVS, while 3DGS proceeds to Gaussian initialization and optimization.
3. Photogrammetry primarily aims to reconstruct geometry, while 3DGS places greater emphasis on reproducing the visual appearance of photographs at high quality.

3DGS is not a technology that is completely unrelated to traditional photogrammetry.

Rather, it is easier to understand 3DGS as a technology that builds on established techniques such as SfM and combines them with a new approach to 3D representation and rendering.

In the next article, we will explore the differences between 3DGS, 2DGS, and NeRF, looking at the types of 3D representation they use and what each approach does best.