Understanding 3DGS, Part 0: How Do Photos Become 3D? — 170 Years of Trial and Error

Photography Was Not Just About Pictures — It Was a Tool for Measurement

Today, it is becoming possible to reconstruct buildings and landscapes in 3D from photos taken with a smartphone, without the need for specialized equipment.

In recent years, a technology called 3D Gaussian Splatting (3DGS) has attracted considerable attention. It can create three-dimensional views from many photographs that make you feel as though you were actually standing in the captured location.

At first, this may sound like a completely new technology made possible only by the rise of AI. In reality, however, attempts to create three-dimensional representations from photographs have a history spanning more than 170 years.

That history goes all the way back to the birth of photography itself.

The Birth of Photography and the Beginnings of Photogrammetry

Photography Was Introduced in 1839

One of the most important events in the history of modern photography was the public introduction of the daguerreotype in 1839.

Presented by the French inventor Louis Daguerre, this photographic process was groundbreaking because it made it possible to record the appearance of a subject as an image.

Even before photography, technologies such as the camera obscura had been used to project landscapes and buildings. However, being able to fix a scene permanently as an image and preserve it as a record represented a major change.

And when people saw this new technology, some began to ask a new question:

Could photographs be used to measure the actual shape and dimensions of real objects?

The Idea of Surveying from Photographs

Two people known as pioneers of photogrammetry are Aimé Laussedat, a French military officer and engineer, and Albrecht Meydenbauer, a German architect.

Around 1849, Laussedat began researching methods for surveying terrain using images and perspective drawings. At first, he also worked with drawings produced using an optical instrument called a camera lucida, while his experiments using an actual camera began from around 1852.

He attempted to use photographs to measure the position and shape of terrain and structures.

Meydenbauer, meanwhile, began working on architectural photogrammetry around 1858.

It is said that his interest in photographic measurement was inspired by a dangerous experience while directly measuring a building.

Measuring the walls of tall buildings from elevated positions was both dangerous and labor-intensive.

He therefore considered whether photographing a building and measuring its dimensions and shape from the photographs could reduce the need for direct on-site measurement.

This idea also connects closely with today's 3D digital archiving of cultural heritage and historic buildings.

The Birth of the Word “Photogrammetry”

The word Photogrammetry, as it is used today, appeared in the literature in 1867.

The term combines “Photo,” meaning light or photograph, “Gram,” meaning a record or drawing, and “Metry,” meaning measurement.

In other words, it refers to the technology of measuring through photographs.

Photogrammetry at that time did not involve processing images with computers as it does today.

Photographs were taken, and geometric relationships and optical instruments were used to determine the positions and dimensions of objects.

Even so, the fundamental idea of reconstructing real-world shapes from photographs is shared with modern photogrammetry.

Within only a few decades of the invention of photography, people had already begun using photographs not simply as records, but as tools for measurement.

Why Can Human Eyes See in 3D?

There has long been an important clue for creating three-dimensional information from photographs: our own eyes.

Try a simple experiment.

Close one eye and look at a finger held in front of you. Then switch to the other eye. Your finger should appear to shift slightly.

Your left and right eyes see the world from slightly different positions. Your brain uses this small difference to estimate distance and perceive depth.

In other words, we naturally perceive three-dimensional space by combining two slightly different images captured by our two eyes.

Could the same principle be applied to photogrammetry?

If we take two photographs from slightly different positions, we can calculate depth based on how the position of an object differs between the two images.

This is the basic principle of stereo photogrammetry.

This method eventually came to be used with aerial photography as well.

Photographs taken from aircraft are captured with partial overlap between neighboring images. By viewing adjacent photographs stereoscopically, it becomes possible to interpret terrain elevation and building height.

This made it possible to survey large areas three-dimensionally from the air—areas that could not be seen all at once from the ground.

It was a major advance for mapmaking.

However, accurate reconstruction required precise knowledge of where each photograph was taken, the orientation of the camera, and the characteristics of the lens.

Simply taking many photographs was not enough to automatically create an accurate 3D model.

What Does a Computer Compare Between Photographs?

As technology advanced, photography shifted from film to digital images.

The arrival of digital photography brought a major transformation to photogrammetry.

Computers could now automatically detect features within photographs and compare those features across multiple images.

Imagine, for example, that you take several photographs of a building.

One photograph shows the right side of a window. Another shows the window from the front. A third shows it from farther away.

A human can compare those images and recognize that they contain the same window.

A computer can do something similar by finding distinctive parts of an image, such as the corner of a window, a pattern on a wall, or the outline of a sign.

Once matching landmarks are identified across multiple photographs, the computer can begin calculating the positions from which each photograph was taken.

This is where a technology called SfM (Structure from Motion) comes in.

The name literally means reconstructing structure from motion.

By analyzing the relationships between feature points visible in multiple photographs, SfM estimates where the cameras were located and which direction they were facing. At the same time, it reconstructs the approximate three-dimensional structure of the scene.

What is particularly interesting is that the camera positions do not necessarily need to be recorded in advance.

By comparing the photographs with one another, the computer can work backward and estimate where each photograph was taken.

Simply by taking photographs, it becomes possible to infer how the camera moved through the scene.

It is almost as if the photographer's footsteps were left behind inside the images.

How Do Points in Photographs Become a Building?

At this point, SfM has given us the camera positions and an approximate structure of the scene.

However, what we have at this stage is still only a sparse collection of points.

Even if we photographed a building, there may only be points around distinctive areas such as window corners and wall patterns.

That is not enough to reconstruct the complete shape of the building.

What is needed next is a more detailed comparison of the photographs.

The same locations appearing in multiple images are matched, and their distances are estimated.

Rather than using only a small number of distinctive image features, the goal is to reconstruct the scene using as many pixels as possible.

This technology is called MVS (Multi-View Stereo).

While SfM mainly determines the positions of the cameras used to capture the images, MVS uses that information to reconstruct the surfaces of the scene in much greater detail.

As an analogy, if SfM is like placing survey markers around a building, MVS uses those markers to fill in the detailed shapes of the walls and roof.

The collection of points produced in this process is called a point cloud.

If those points are then connected to form surfaces, the result is a mesh, which represents the surfaces of buildings or terrain.

By applying colors and patterns taken from the photographs, the reconstructed object can begin to look very close to the real thing.

One piece of software that has made these kinds of processes easier to use is COLMAP.

When multiple photographs are provided, COLMAP can perform processes ranging from feature detection and camera pose estimation to three-dimensional reconstruction.

At this point, the long-standing dream of creating three-dimensional representations from photographs had become increasingly practical.

But another problem was beginning to emerge.

Is Reconstructing the Shape Enough?

Traditional photogrammetry was particularly good at reconstructing the shapes of objects.

It could reproduce information such as where a building's walls were located, the shape of its roof, and the elevation of the ground.

In other words, it could represent physical geometry in three dimensions.

However, when we look at a real scene, we perceive more than just shape.

We see light reflected in windows, metallic reflections, light passing through leaves, and the soft illumination of a sunset.

These visual effects are also part of how we experience a scene.

Accurately reconstructing geometry and reproducing a scene so that it looks like a photograph are, in fact, different problems.

A technology that approached this challenge from a new direction was NeRF (Neural Radiance Fields), introduced in 2020.

Instead of directly creating a surface from polygons, NeRF uses a neural network to learn what colors and light should be visible depending on a position in space and the direction from which it is viewed.

The mechanism may sound unusual, but the basic idea is relatively simple.

Using the captured photographs as clues, the computer learns:

“If this location is viewed from this direction, what color should it appear to be?”

This makes it possible to render scenes from viewpoints that were never actually photographed.

For example, if photographs were taken from the front and side of a building, the system can generate an image showing how the building might look from an angle between those two viewpoints.

It can effectively fill in the space between photographs and create images from new perspectives.

NeRF attracted considerable attention because it could generate highly photorealistic images.

However, rendering requires computation and can take time.

Even if a beautiful scene can be reproduced, displaying it smoothly enough to move around freely like in a game is not always easy.

Instead of “Drawing” a 3D Object, Fill the Space with Tiny Particles

Three years after NeRF appeared, a new method was introduced in 2023.

That method was 3D Gaussian Splatting (3DGS).

Its basic idea is somewhat unusual.

Instead of building the surfaces of objects from triangular polygons or having a neural network represent the entire space, 3DGS places a very large number of small, elliptical elements throughout the scene.

These elements are called Gaussians.

Each Gaussian stores information such as its position, size, orientation, color, and opacity.

They are projected onto the screen from the camera's viewpoint and composited together to produce a photorealistic image.

A useful way to imagine this is to picture the entire space filled with tiny, semi-transparent, colored particles.

Each particle is simple on its own, but when enormous numbers of them are combined, they can reproduce the outlines of buildings, the texture of trees, and even the way light falls across a scene.

One of the major characteristics of 3DGS is that it can render relatively quickly while maintaining a realistic appearance.

This makes experiences such as freely looking around a captured location or moving a virtual camera through the space much more practical.

The technology for creating three-dimensional scenes from photographs had entered another new stage.

Sometimes Looking Beautiful Is Not Enough

However, 3DGS also has limitations.

It is very good at reproducing photorealistic scenes using collections of particles, but that does not necessarily mean that the resulting geometry is accurate.

For example, a wall may look perfectly natural in the rendered image, while the surface of that wall may not actually be reconstructed as a mathematically accurate single plane.

Looking visually natural and having geometry accurate enough for surveying or design are not the same thing.

One technology proposed to address this issue is 2D Gaussian Splatting (2DGS).

Introduced in 2024, this method uses planar Gaussians rather than the three-dimensional Gaussian elements used by 3DGS.

By treating surfaces as planar elements, it aims to reconstruct objects while placing greater emphasis on consistency in shape and depth.

The goal is not only visual beauty, but also greater geometric correctness.

The technology for reconstructing three-dimensional scenes from photographs continues to evolve step by step.

Stepping Back into a Photograph

Looking back, the idea of creating three-dimensional information from photographs has existed since the early days of photography.

At first, surveyors and architects tried to use photographs to measure buildings and terrain.

Aerial photography later became widely used, and computers eventually became capable of identifying visual features in photographs and estimating the positions from which the images were captured.

Today, it is possible not only to reconstruct shape from photographs, but also to reproduce light, texture, and appearance in ways that make viewers feel as though they are actually present in the captured location.

These technologies are now used in many fields beyond mapping and surveying, including cultural heritage preservation, film and game production, VR, tourism, and digital archiving.

In the past, photographs were primarily a way to look back at scenes from the past.

Now that we can reconstruct three-dimensional spaces from photographs, we are beginning to experience photographed places from different angles and even move through them as if we were walking there.

Photography may be changing from a medium that simply records the past into a way of revisiting the spaces that once existed.

A single photograph was born in 1839.

From that small invention began the long effort to “measure the world with photographs.”

More than 170 years later, that effort is still continuing in new forms.