How VGGT Is Transforming 3D Reconstruction: From SfM to Geometry Foundation Models—and Their Limitations

Target Audience#

This article is intended for readers who:

• Are interested in generating point clouds or building 3D models from multiple images.
• Want to understand image feature matching, image alignment, change detection, and related computer vision techniques.
• Want to understand how SfM, NeRF, 3D Gaussian Splatting, and SLAM relate to modern deep learning-based 3D reconstruction methods.

Summary#

• The fundamental contribution of VGGT is that it moves much of the geometric computation and iterative optimization traditionally performed at inference time in SfM into unified predictions made by a pretrained Transformer.
• Building on research such as SfM/MVS, VGGSfM, DUSt3R, and MASt3R, VGGT enables the joint prediction of cameras, depth, point maps, and point tracks from multiple images.
• VGGT is not simply a replacement for COLMAP. It can serve as a general-purpose geometry backbone for NeRF, 3D Gaussian Splatting, SLAM, robotics, and other 3D applications.
• However, VGGT relies on important assumptions, such as input images observing a common 3D scene and learned geometric priors generalizing to unfamiliar environments. Being able to predict 3D geometry does not guarantee that the predicted geometry is physically correct.
• As of 2026, research is progressing toward Geometry Foundation Models through developments such as VGGT-Ω, MapAnything, and VGGT-SLAM. At the same time, geometric consistency, uncertainty estimation, outlier detection, and dynamic scene understanding remain important research challenges.

1. Where Does VGGT Fit into the History of 3D Computer Vision?#

VGGT (Visual Geometry Grounded Transformer) is a Transformer-based model designed to infer the structure of 3D scenes from images.

Introduced in 2025, it received the Best Paper Award at CVPR 2025. [1]

From a historical perspective, VGGT represents an important transition in 3D computer vision.

In simple terms, it symbolizes a shift:

From reconstructing 3D geometry through feature matching, explicit geometric computation, and iterative optimization, toward directly predicting scene geometry using a Transformer trained on large-scale 3D data.

However, this characterization requires an important qualification.

VGGT does not make geometry itself unnecessary. Nor does it replace every component of traditional Structure from Motion (SfM) with a mathematically equivalent neural operation.

Traditionally, reconstructing a 3D model from images required a pipeline combining feature detection, image matching, camera pose estimation, triangulation, and Bundle Adjustment.

VGGT learns many of the geometric relationships underlying these operations from large-scale training data. During inference, it predicts multiple types of geometric information using a shared Transformer.

As a result, VGGT can support not only point cloud generation but also image correspondence estimation, camera localization, 3D Gaussian Splatting, SLAM, and other applications.

At the same time, VGGT introduces a new question:

"When a pretrained model predicts a 3D structure, how do we know that the predicted geometry is actually supported by the observed images?"

Traditional SfM reconstructs geometry using image correspondences and explicit projective constraints.

VGGT can predict 3D geometry without explicitly solving all those constraints through iterative optimization at inference time.

However, producing a prediction and verifying that the prediction is correct are fundamentally different problems.

This article examines VGGT from several perspectives:

• What changed from traditional SfM?
• What does VGGT learn, and what does it predict?
• How does it relate to NeRF, 3D Gaussian Splatting, and SLAM?
• What assumptions does VGGT rely on?
• What happens when those assumptions are violated?
• What challenges remain as of 2026?

2. How Was 3D Reconstruction Traditionally Performed?#

2.1 3D Reconstruction with SfM and MVS#

Two foundational techniques in classical 3D reconstruction are Structure from Motion (SfM) and Multi-View Stereo (MVS).

SfM estimates camera positions, orientations, and 3D scene structure from multiple images.

MVS uses camera information and other geometric constraints to recover denser 3D geometry.

One of the most widely used implementations is COLMAP. [2]

A typical SfM/MVS pipeline looks approximately like this:

Images → Feature Detection → Feature Matching → Camera Pose Estimation → Triangulation → Bundle Adjustment → Dense MVS → 3D Reconstruction

Each stage serves a specific purpose:

1. Feature detection: Identify distinctive image locations that can be recognized across multiple images.
2. Feature matching: Establish correspondences between points depicting the same scene locations.
3. Camera pose estimation: Estimate camera positions and orientations from image correspondences.
4. Triangulation: Compute 3D coordinates from observations made from multiple viewpoints.
5. Bundle Adjustment (BA): Jointly optimize camera parameters and 3D point positions.
6. Dense MVS: Recover denser 3D scene geometry.

Actual COLMAP pipelines include additional operations such as image registration, local and global Bundle Adjustment, and outlier rejection.

This classical approach has a major advantage.

Because camera models, projective geometry, and epipolar constraints are explicitly represented, it is possible to mathematically evaluate how well the reconstructed geometry agrees with the original image observations.

2.2 What Does Bundle Adjustment Actually Guarantee?#

Bundle Adjustment is one of the central techniques in 3D reconstruction.

Consider the difference between a feature point observed in an image and the location obtained by projecting its estimated 3D position back into that image.

This difference is called the reprojection error.

In simplified form, Bundle Adjustment minimizes an objective such as:

[ E_{\mathrm{BA}} = \sum_{(i,j)\in\mathcal{O}} \rho\left( \left| \pi(K_i,T_i,X_j)-x_{ij} \right|^2 \right) ]

where:

• (K_i): Camera intrinsic parameters.
• (T_i): Camera position and orientation.
• (X_j): A 3D point.
• (x_{ij}): The observed image coordinate.
• (\pi): A function projecting 3D points onto the image plane.
• (\rho): A robust loss function that reduces the influence of outliers.
• (\mathcal{O}): The set of valid observations.

In other words, Bundle Adjustment modifies camera poses and 3D point positions so that the reconstructed geometry is as consistent as possible with observations across multiple images.

This mechanism is essential for improving geometric consistency.

However, Bundle Adjustment also has limitations.

If the underlying image correspondences are incorrect, optimization does not necessarily recover the true 3D structure.

Likewise, a low reprojection error does not automatically guarantee that absolute distances or object shapes are correct in the physical world.

Nevertheless, the ability to explicitly measure disagreement between observations and estimated geometry is a defining strength of classical SfM.

2.3 Strengths and Weaknesses of Classical SfM#

Classical SfM is highly effective when sufficient image correspondences can be established.

However, reconstruction becomes difficult under conditions such as:

• Images containing few distinctive features.
• Textureless surfaces.
• Large viewpoint differences.
• Reflective or transparent materials.
• Limited overlap between images.
• Scenes containing many moving objects.

Furthermore, the individual stages must be carefully integrated.

Failures in feature matching can propagate into camera pose estimation, triangulation, and subsequent optimization.

To address these limitations, deep learning was gradually incorporated into the reconstruction pipeline.

2.4 Replacing Individual Components with Deep Learning#

From the late 2010s to the early 2020s, deep learning was increasingly introduced into individual stages of SfM.

Initially, the dominant approach was not to abandon SfM entirely, but to replace selected components of the classical pipeline with neural networks.

Examples include:

• Neural feature detection.
• Neural feature matching.
• Neural depth estimation.

In other words, the classical reconstruction pipeline was improved by replacing individual components with learned models.

2.5 End-to-End SfM with VGGSfM#

VGGSfM (CVPR 2024) advanced this direction further. [3]

VGGSfM combines point tracking, camera estimation, and differentiable Bundle Adjustment to make the overall SfM pipeline trainable end to end.

This represented a substantial advance over previous approaches.

However, VGGSfM retained both the traditional SfM problem formulation and explicit geometric optimization through Bundle Adjustment.

Therefore, VGGSfM can be understood as an approach that integrates and improves the classical SfM pipeline using deep learning.

The next major conceptual shift came from methods that reconsidered the reconstruction problem itself.

One of the most important examples is DUSt3R.

3. How DUSt3R Changed the Way We Think About 3D Reconstruction#

DUSt3R (CVPR 2024) is one of the most important predecessors for understanding VGGT. [4]

Its underlying idea differs significantly from traditional reconstruction pipelines.

Conceptually, the classical approach follows this sequence:

Estimate Cameras → Find Correspondences → Triangulate 3D Points

DUSt3R instead adopts a different strategy:

Given two images, directly predict the 3D location corresponding to each pixel using a Transformer.

3.1 The Point Map Representation#

A central concept introduced by DUSt3R is the point map.

A point map assigns a 3D coordinate to each image pixel.

A conventional depth map represents the depth associated with each pixel.

A point map, by contrast, directly represents the corresponding 3D position.

DUSt3R predicts point maps from image pairs without requiring known camera poses or calibration parameters.

From the predicted point maps, it is also possible to derive information such as depth, correspondences, focal length, and camera pose. [4]

In traditional reconstruction, camera poses were estimated before recovering 3D geometry.

DUSt3R demonstrated that directly predicting 3D geometry could provide the information needed to infer camera poses and image correspondences.

This represented an important change in how the reconstruction problem could be formulated.

3.2 Why Is Direct 3D Prediction Possible?#

A key factor behind this approach is the use of learned 3D priors.

For example, humans can often infer approximate object shapes and spatial arrangements from a single indoor photograph.

However, the true depth of every object cannot generally be determined uniquely from a single image.

We can make these inferences because we have prior knowledge of the typical shapes and arrangements of walls, floors, desks, chairs, and other objects.

Learning-based models such as DUSt3R acquire relationships between images and 3D structures from large amounts of training data.

Consequently, they may predict plausible 3D geometry even when the observations are insufficient for traditional triangulation.

However, this introduces an important distinction.

Being able to predict 3D geometry is not the same as uniquely determining that geometry from observations.

This distinction becomes particularly important when evaluating the limitations of VGGT.

3.3 Limitations of DUSt3R#

DUSt3R has an important limitation.

Its fundamental processing unit is an image pair.

Suppose we want to reconstruct a scene from 10 or 100 images.

We must process multiple image pairs and then align their predicted point maps through a global alignment procedure.

As the number of images increases, pairwise processing and the integration of individual reconstructions become increasingly important challenges.

Furthermore, point maps predicted from separate image pairs may use different coordinate systems or scales.

They must therefore be aligned to construct one coherent 3D scene.

3.4 From DUSt3R to MASt3R#

MASt3R (ECCV 2024) extended the ideas behind DUSt3R toward image matching. [5]

MASt3R uses features grounded in 3D geometry to estimate accurate image correspondences, including between images with large viewpoint differences.

An important implication is that image matching and 3D reconstruction no longer need to be treated as completely separate problems.

Traditionally, feature matching was used as an input to 3D reconstruction.

DUSt3R and MASt3R demonstrated the reverse relationship as well:

Learned 3D representations can be used to improve image matching.

This is important for image alignment and change detection.

For example, when comparing photographs captured from different positions, simple pixel differences cannot reliably distinguish viewpoint changes from actual changes in the scene.

If correspondences can be established using 3D geometry, it becomes possible to compare images while accounting for viewpoint differences.

However, DUSt3R and MASt3R are not general-purpose change detection systems by themselves.

4. What Exactly Did VGGT Change?#

VGGT extends learning-based geometric estimation toward unified processing of multiple images.

It is closely related to developments represented by DUSt3R and MASt3R.

At the same time, VGGT also builds on research directions involving learned camera estimation and point tracking, including VGGSfM, Deep SfM, PoseDiffusion, and CoTracker. [1][10]

Therefore, describing VGGT solely as a direct successor to DUSt3R or MASt3R would oversimplify its research lineage.

It is more accurate to regard VGGT as a convergence of multiple research directions.

VGGT can process anywhere from a single image to hundreds of images and jointly predict several types of geometric information through one network. [1][6]

4.1 Predicting Multiple Geometric Outputs with a Single Transformer#

VGGT produces four main categories of output.

Output	Description	Related Applications
Camera Parameters	Camera intrinsic and extrinsic parameters	Camera pose estimation, 3D reconstruction
Depth Maps	Per-pixel depth estimates	Depth and distance estimation
Point Maps	Per-pixel 3D coordinates	Point cloud generation, 3D modeling
Point Tracks	Correspondences and trajectories across images	Image matching, alignment, tracking

Traditionally, these quantities were often estimated using separate algorithms.

VGGT predicts them using multiple output heads built on a shared Transformer representation.

In other words, camera pose estimation, depth estimation, 3D reconstruction, and point tracking are treated as related aspects of understanding a common 3D scene rather than completely independent problems.

4.2 Frame-wise Attention and Global Attention#

VGGT's network architecture is another important contribution.

Images are converted into tokens using a DINO-based encoder.

The model then alternates between Frame-wise Attention and Global Attention. [1][6]

Conceptually, the architecture can be expressed as:

Images → Encoder → Frame-wise Attention → Global Attention → Frame-wise Attention → Global Attention → … → Geometric Outputs

Frame-wise Attention processes information within individual images.

Global Attention exchanges information across multiple images.

This allows the network to learn relationships between images observing the same 3D scene from different viewpoints.

Classical SfM explicitly performs operations such as feature matching, epipolar geometry estimation, and triangulation.

VGGT's Global Attention can be interpreted as implicitly representing some of the relationships that these operations traditionally handled.

However, learning relationships between images does not guarantee that those relationships strictly satisfy the mathematical constraints of projective geometry.

Global Attention provides a mechanism for learning geometric consistency, but it does not mathematically guarantee geometric consistency.

4.3 The Idea of Over-complete Prediction#

One of VGGT's distinctive design choices is Over-complete Prediction.

If a depth map and camera parameters are known, a point map can be computed.

Conversely, point maps can be used to recover information about cameras.

These outputs therefore contain mathematical redundancy.

In conventional algorithm design, predicting the same underlying information in multiple forms might appear inefficient.

VGGT intentionally exploits this redundancy.

By jointly supervising cameras, depth maps, point maps, and tracks, the model learns an internal representation grounded in 3D geometry.

In other words, the model learns not only visual appearance but also geometric structure from several related perspectives: cameras, depth, 3D coordinates, and correspondences.

This is a central idea behind the name Visual Geometry Grounded Transformer.

4.4 Does Over-complete Prediction Guarantee Geometric Consistency?#

There is an important distinction between predicting several related quantities jointly and guaranteeing that those predictions are mathematically consistent.

For example, VGGT predicts depth maps and camera parameters.

These outputs can be used to calculate 3D points.

However, VGGT also predicts point maps directly.

Ideally, the following two representations should agree:

• 3D points computed from predicted depth and camera parameters.
• 3D points directly predicted as point maps.

Nevertheless, the model architecture does not guarantee that they always match exactly.

In fact, the original VGGT paper reports cases where point maps constructed from predicted depth and camera parameters are more accurate than directly predicted point maps. [16]

This suggests that Over-complete Prediction is an effective learning strategy, but output consistency and the appropriate use of each prediction remain separate questions.

This distinction is particularly important when using VGGT for measurement, 3D model construction, or image alignment.

5. Comparing the Evolution from SfM to VGGT#

The major approaches discussed so far can be summarized as follows.

Generation	Fundamental Idea	Inference Procedure	Basis for Geometric Correctness
SfM / COLMAP	Solve geometry explicitly	Matching → Camera Estimation / PnP → Triangulation → BA	Correspondences, projective geometry, reprojection error, BA
Deep SfM / VGGSfM	Make SfM components trainable	Neural Network + Geometric Optimization	Learned features and tracking combined with explicit optimization
DUSt3R	Directly predict 3D point maps	Image Pairs → Point Maps → Global Alignment	Learned 3D priors and pairwise geometric alignment
MASt3R	Extend learned geometry toward matching	Image Pairs → 3D-aware Matching	Correspondences grounded in learned 3D representations
VGGT	Jointly predict scene geometry	Multiple Images → Cameras + Depth + Points + Tracks	Unified representations learned from large-scale 3D supervision
Post-VGGT	Develop Geometry Foundation Models	Backbones for SLAM, 3DGS, Robotics, etc.	Learned predictions combined with geometric verification and optimization when needed

The historical progression can be divided into five broad stages.

5.1 Stage 1: Reconstructing 3D Through Geometric Computation#

Camera poses and 3D coordinates are recovered using feature matching and projective geometry.

The central approach is to explicitly solve geometric constraints.

5.2 Stage 2: Introducing Deep Learning into Classical Pipelines#

Neural networks replace components such as feature detection, matching, and depth estimation.

VGGSfM extends this direction by making the overall SfM pipeline trainable end to end.

5.3 Stage 3: Predicting 3D Coordinates Directly#

DUSt3R introduces direct point map prediction.

Instead of always estimating camera poses before triangulating points, learned 3D representations can be used to derive multiple geometric quantities.

MASt3R applies these representations to image matching.

5.4 Stage 4: Jointly Predicting Geometry from Multiple Images#

VGGT predicts depth, camera poses, and relationships across multiple images using a shared model.

Other methods, including Fast3R and CUT3R, also emerged around the same period, advancing the transition from pairwise processing toward multi-view reconstruction.

Fast3R appeared on arXiv in January 2025, while VGGT was released on arXiv in March 2025. [7][1]

This development should therefore not be understood as a single, linear research lineage.

Multiple research groups were exploring feed-forward multi-view reconstruction around the same time.

VGGT is one of the most representative achievements of this broader trend.

5.5 Stage 5: Moving Toward Geometry Foundation Models#

The next direction is to develop general-purpose models that can serve as foundations for SLAM, 3D modeling, neural rendering, and other tasks.

The important change is not simply improving reconstruction accuracy.

It is the shift toward pretraining general-purpose 3D representations that can support multiple downstream applications.

6. VGGT Represents a Transition Toward 3D Foundation Models#

VGGT's historical importance extends beyond improvements in reconstruction algorithms.

In image understanding, a broad research progression has been:

ImageNet Classifiers → ViT → DINO / CLIP → General-purpose Visual Foundation Models

Instead of building a separate specialized model for each task, researchers increasingly use general-purpose visual models pretrained on large datasets.

VGGT suggests that a similar transition may be possible for 3D geometry.

Traditionally, SfM, MVS, camera pose estimation, depth estimation, and point tracking have often been studied as distinct problems requiring specialized algorithms.

VGGT demonstrates a direction in which a Transformer trained on large-scale 3D data can learn a shared geometric representation that supports multiple tasks.

The VGGT paper also reports improvements when pretrained features are transferred to tasks such as non-rigid point tracking and feed-forward novel-view synthesis. [1][8]

A loose analogy can be drawn with the development of large language models.

Instead of relying on a collection of specialized analysis modules, Transformer models learn general-purpose representations that can support multiple tasks.

Of course, language and 3D geometry involve fundamentally different structures and constraints.

In particular, 3D geometry involves mathematically defined relationships such as projective geometry, coordinate transformations, physical scale, and spatial consistency.

Therefore, similarities with language models do not imply that Geometry Foundation Models will follow exactly the same development path.

Nevertheless, both fields exhibit a transition:

From improving task-specific algorithms toward pretraining general-purpose models that can be adapted to many applications.

6.1 What Does a Geometry Foundation Model Provide?#

Traditional 3D reconstruction systems often estimate camera poses, depth maps, point clouds, and feature correspondences using separate methods.

A Geometry Foundation Model aims to provide these quantities through a shared learned representation.

For example, the architecture of a downstream system might be:

Images → Geometry Foundation Model → Shared 3D Representation → Multiple Downstream Tasks

These tasks may include camera localization, point cloud generation, 3D modeling, novel-view synthesis, SLAM, and robotic perception.

Under this architecture, the geometry model itself becomes a reusable component.

However, the usefulness of a shared representation across multiple tasks does not automatically guarantee the accuracy or reliability required by every application.

For applications such as robotics or physical measurement, geometric verification and alignment with external references remain important.

7. How Does VGGT Connect to Modern 3D Modeling Technologies?#

VGGT has created new opportunities for integration with technologies such as NeRF and 3D Gaussian Splatting.

7.1 Relationship with NeRF#

NeRF (Neural Radiance Fields) learns a representation of a 3D scene from images and generates images from novel viewpoints.

The original NeRF assumes that camera poses are known. [9]

This means that camera poses generally need to be estimated using SfM or another method before training NeRF.

A traditional pipeline looks approximately like this:

Images → COLMAP Camera Pose Estimation → NeRF → Novel-view Rendering

With VGGT, camera pose estimation can be performed by a pretrained model.

Images → VGGT → Camera Parameters + Depth + Point Cloud → NeRF

The important point is that VGGT and NeRF are not competing technologies.

VGGT estimates 3D geometry from images.

NeRF represents a 3D scene and generates images from new viewpoints.

Their purposes and roles are different.

Furthermore, using camera poses predicted by VGGT does not automatically guarantee better NeRF results.

Any remaining camera pose errors may affect the quality of the learned NeRF representation.

7.2 Relationship with 3D Gaussian Splatting#

Camera poses and initial point clouds are also important in 3D Gaussian Splatting (3DGS).

Traditional practical 3DGS pipelines often use COLMAP to obtain camera poses and initial point clouds.

VGGT can be incorporated into this process:

Uncalibrated Images → VGGT → Cameras + Depth + Point Cloud → 3D Gaussian Splatting

The official VGGT implementation provides functionality for exporting predictions in COLMAP format, making it possible to use them with NeRF and Gaussian Splatting libraries such as gsplat. [10]

AnySplat (SIGGRAPH Asia 2025) is another relevant development.

It generates 3D Gaussians from uncalibrated images using a feed-forward model.

AnySplat also uses pretrained VGGT as a geometry prior during training. [11]

The relationship can be summarized as follows:

• VGGT: A model for estimating 3D geometry from images.
• 3D Gaussian Splatting: A representation and rendering technique for 3D scenes.

In simplified terms, VGGT helps a system infer 3D structure, while 3DGS provides a way to represent and render that structure.

They can therefore complement each other.

This distinction is particularly useful for applications involving 3D model generation from photographs.

However, visually convincing 3DGS rendering does not necessarily mean that the underlying geometry or distances are accurate.

Novel-view rendering quality and geometric measurement accuracy should be evaluated separately.

8. Applications to SLAM and Robotics#

One important application area for VGGT is SLAM (Simultaneous Localization and Mapping).

SLAM is the process of estimating a system's position while simultaneously building a map of its surroundings using cameras or other sensors.

It is widely used in robotics, autonomous systems, and augmented reality.

8.1 VGGT-SLAM#

VGGT-SLAM from MIT applies VGGT to SLAM. [12]

It divides long video sequences into smaller submaps and reconstructs each submap using VGGT.

The resulting submaps are then combined using geometric optimization.

Conceptually, a system might operate as follows:

First 30 Frames → VGGT → Local 3D Map

Next 30 Frames → VGGT → Local 3D Map

Following 30 Frames → VGGT → Local 3D Map

A SLAM backend then aligns the local maps to construct a larger global map.

The 30-frame segments are illustrative examples. Actual segment sizes and processing methods depend on the system design.

This submap-based approach is a fundamental strategy for handling long sequences.

8.2 Why Is Simply Combining Submaps Not Enough?#

VGGT-SLAM addresses more than the computational cost of processing many images.

Another important issue is that separately reconstructed submaps may not represent exactly the same geometric coordinate system.

For example, scale estimates and camera pose errors may differ between submaps.

Furthermore, when dealing with uncalibrated cameras, reconstruction may contain projective distortions that cannot be corrected using only rotation, translation, and scale.

VGGT-SLAM proposes an approach using more general projective transformations to address this issue. [12]

This demonstrates that simply combining multiple VGGT predictions does not necessarily produce a geometrically correct global reconstruction.

8.3 Development Toward VGGT-SLAM++#

VGGT-SLAM++ was introduced in 2026. [13]

It incorporates techniques such as Local Bundle Adjustment and Loop Closure into the use of VGGT predictions.

The system uses VGGT-generated submaps while applying frequent local optimization to reduce short-term camera pose drift.

This reflects an important direction in post-VGGT research:

Artificial intelligence does not necessarily replace classical geometric computation. Instead, the two approaches can be combined.

VGGT rapidly produces initial estimates of 3D structure.

Classical geometric optimization improves consistency between predictions and compensates for errors accumulated over longer sequences.

The roles can be summarized as follows:

• VGGT: Rapidly produces useful initial geometry and geometric priors.
• Classical Geometry: Improves long-term and global geometric consistency.

This combination is particularly suitable for robotics applications that must maintain representations of 3D environments over extended periods.

9. Relationship with Feature Matching, Image Alignment, and Change Detection#

Understanding VGGT is also valuable for applications involving feature matching and change detection.

However, it is important to distinguish between capabilities demonstrated directly in the research and possible applications built on top of those capabilities.

9.1 Traditional Feature Matching#

Traditional feature matching extracts keypoints and descriptors from two images and compares them to establish correspondences.

Geometric constraints are then used to reject incorrect matches or estimate the relative relationship between the images.

An important part of this process is verifying whether visually similar features also represent geometrically plausible correspondences.

9.2 Changes Introduced by DUSt3R, MASt3R, and VGGT#

DUSt3R predicts point maps directly from image pairs.

MASt3R uses representations grounded in 3D geometry to estimate image correspondences.

VGGT processes multiple images jointly and predicts geometric information, including point tracks.

As a result, image correspondence estimation is evolving beyond purely comparing 2D visual features.

It can increasingly incorporate information about the underlying 3D scene structure.

However, VGGT's ability to predict point tracks does not mean that all correspondences are correct or that every moving object can be tracked accurately.

9.3 Potential Applications to Change Detection#

Suppose we want to detect changes between two images captured at different times or from different camera positions.

Simple pixel differences are affected by several factors:

• Different camera positions and orientations.
• Changes in object appearance caused by viewpoint differences.
• Depth-dependent image displacement.
• Image differences in regions where no physical change has occurred.

In such situations, image correspondences and 3D geometry can be used to improve alignment.

VGGT outputs such as camera parameters, point maps, and point tracks may provide useful foundational information for this process.

For example, after aligning images using 3D scene structure, regions that remain inconsistent could be treated as candidates for actual scene changes.

However, VGGT is not itself a dedicated change detection model.

Additional processing is necessary to distinguish actual structural changes from illumination changes, object motion, occlusion, and reconstruction errors.

VGGT is therefore better understood as a foundational technology for establishing relationships between images using 3D geometry, rather than a complete change detection solution.

9.4 Challenges in 3D-based Change Detection#

Consider a component whose apparent image location differs by several pixels between two photographs.

There are multiple possible explanations:

• The component physically moved.
• The camera position changed.
• The estimated camera pose contains an error.
• The estimated depth contains an error.
• Lighting or reflections changed the appearance.
• Part of the component became occluded.

Simply estimating 3D geometry using VGGT is not sufficient to distinguish these cases.

A practical system might require a pipeline such as:

Images → VGGT → 3D Geometry Estimation → Geometric Verification and Coordinate Alignment → 3D Correspondence Estimation → Change Detection → Reliability Evaluation

This is an example of a possible system architecture, not a complete change detection pipeline demonstrated by VGGT itself.

The key challenge is distinguishing errors in 3D reconstruction from actual changes in the physical scene.

Incorrect 3D predictions may otherwise be interpreted as structural changes.

Therefore, evaluation must include not only 3D reconstruction accuracy but also change detection precision, recall, false-positive rates, and other downstream metrics.

10. From VGGT to Geometry Foundation Models#

Between 2025 and 2026, several studies extended the general ideas behind VGGT.

10.1 MapAnything#

MapAnything, developed by Meta and CMU, aims to build a general-purpose Transformer model for 3D geometry. [14]

If the VGGT approach is generalized further, the model does not necessarily have to rely only on images.

Additional inputs may include:

• Images.
• Known camera intrinsics.
• Camera poses.
• Depth.
• Partial 3D information.

MapAnything explores metric 3D reconstruction while accepting different combinations of available geometric information.

A single feed-forward model can address tasks such as SfM, MVS, metric monocular depth estimation, localization, and depth completion.

This represents a transition from specialized 3D reconstruction algorithms toward general-purpose models that accept and produce geometric information.

One important difference is that while VGGT primarily predicts geometry from images, MapAnything can use known geometric quantities as additional input conditions.

Instead of inferring everything from images, the model can exploit known cameras or depth information to reduce ambiguity in the reconstruction problem.

10.2 VGGT-Ω#

VGGT itself evolved into VGGT-Ω (VGGT-Omega) in 2026.

The official VGGT repository describes VGGT-Ω as "the next step of VGGT." [10]

VGGT-Ω is a CVPR 2026 study that improves reconstruction accuracy, efficiency, and dynamic scene handling compared with the original VGGT. [15]

It also reports that increasing model size and training data can improve 3D reconstruction performance in a predictable manner.

This is an important result.

Traditionally, 3D vision research often focused on designing specialized geometric algorithms for individual reconstruction problems.

The direction represented by VGGT-Ω suggests that pretraining sufficiently large geometry models on sufficiently large 3D datasets can become another major driver of performance improvements.

This resembles the scaling principles that have received attention in the development of large language models.

However, increasing model size and training data does not guarantee correct 3D reconstruction under all conditions.

In particular, if the training data contains geometric errors, the model may learn those errors as well.

VGGT-Ω discusses the influence of training data quality on model predictions. [15]

Furthermore, although dynamic scene performance has improved, challenging conditions such as severe motion blur, large changes in field of view, and lens distortion remain difficult.

10.3 Scaling with VGG-T³#

VGG-T³, introduced in 2026, addresses the computational and memory limitations of applying VGGT-style models to larger collections of images. [17]

VGGT's Global Attention is powerful for integrating information across images.

However, computational costs increase as the number of input views grows.

VGG-T³ aims to enable feed-forward 3D reconstruction at a larger scale.

It can be understood as extending the multi-view reasoning introduced by VGGT toward more practical, large-scale scene reconstruction.

10.4 Directions in Geometry Foundation Model Research#

MapAnything, VGGT-Ω, and VGG-T³ demonstrate several directions in the development of Geometry Foundation Models.

Research	Main Direction	Problem Addressed
VGGT	Integration of multiple 3D tasks	Joint estimation of cameras, depth, points, and tracks
MapAnything	Generalization of inputs and outputs	Multi-purpose 3D estimation using known geometric information
VGGT-Ω	Model/data scaling and dynamic scene handling	Improving accuracy, efficiency, and generalization
VGG-T³	Scaling to larger image collections	Computation and memory limitations with many views
VGGT-SLAM	Long-term, large-scale mapping	Geometric consistency between local maps
VGGT-SLAM++	Local optimization and long-term stability	Camera pose drift and global map stability

These developments show that VGGT is evolving beyond a specialized reconstruction method toward a reusable foundation for multiple 3D applications.

However, as the models become more capable, the question of how much their predictions can be trusted becomes increasingly important.

11. What Assumptions Does VGGT Implicitly Make?#

So far, we have examined the technical progress represented by VGGT.

However, to understand VGGT correctly, we must consider not only what it can do but also what assumptions its predictions depend on.

Based on the original VGGT paper and subsequent studies, several important assumptions can be identified.

Assumption	Description	Potential Problem When Violated
Common scene	Input images observe a shared 3D scene	Unrelated images may be incorrectly integrated
Approximately static geometry	Scene structure remains mostly consistent across images	Moving or deforming objects may cause inconsistencies
Camera model compatibility	Images can be represented by the assumed projection model	Fisheye or panoramic images may reduce accuracy
Generalization of learned 3D priors	Learned geometric knowledge applies to unseen environments	Unfamiliar objects or capture conditions may cause errors
Prediction under insufficient observation	Learned priors can support 3D estimation when evidence is limited	Geometry unsupported by observations may be generated
Joint supervision	Learning cameras, depth, and point maps together improves geometric understanding	Mathematical consistency between outputs is not guaranteed
Replacement of inference-time optimization	Learned inference can eliminate much of the need for iterative optimization	Additional optimization may still be necessary
Reliability of confidence estimates	Predicted confidence can indicate output quality	High-confidence predictions may still contain significant errors

Some of these assumptions are explicitly stated in the original VGGT problem formulation.

Others are research questions arising from the model's learning-based design.

11.1 Assumption 1: Input Images Observe the Same 3D Scene#

This is one of VGGT's most important assumptions.

The original paper formulates the problem using multiple RGB images observing the same 3D scene. [16]

For example, photographs of the same room captured from different viewpoints can be used to reconstruct a shared 3D structure.

But what happens if some images depict a different room?

What if the furniture has been rearranged significantly between photographs?

VGGT does not first mathematically prove that the input images belong to the same scene before attempting reconstruction.

Therefore, it may still predict some form of 3D geometry even when unrelated images are included.

This issue has become an important topic in subsequent research. [18]

11.2 Assumption 2: Scene Geometry Is Approximately Static#

The original VGGT formulation primarily concerns the reconstruction of static 3D scenes.

For buildings, indoor spaces, desks, and furniture, images captured from different viewpoints can often be treated as observations of the same geometry.

However, scenes containing pedestrians, vehicles, moving trees, or deformable cloth have geometry that changes over time.

In such cases, the same 3D point does not necessarily remain at the same spatial location across images.

Although VGGT predicts point tracks, this does not automatically provide a complete 4D reconstruction that separates camera motion from object motion.

Large non-rigid deformations were reported as a limitation of the original VGGT.

VGGT-Ω improves dynamic scene handling, but static 3D reconstruction and full 4D scene understanding remain different problems. [15][16]

11.3 Assumption 3: The Camera Projection Model Is Appropriate#

VGGT can estimate camera parameters from uncalibrated images.

However, it does not support arbitrary camera optics without limitations.

The camera representation in the original paper estimates quantities such as rotation, translation, and field of view while assuming that the principal point lies at the image center. [16]

The original model also has limitations when dealing with fisheye and panoramic images.

Performance degradation is reported for images with extreme rotations.

Therefore, saying that VGGT can estimate camera parameters from uncalibrated images is not the same as saying it supports every camera configuration.

The assumed camera model, lens distortion, capture method, and image preprocessing remain important.

11.4 Assumption 4: Learned Geometric Principles Generalize to Unseen Environments#

This is one of the most fundamental hypotheses behind VGGT.

VGGT learns relationships between images and 3D geometry using large-scale supervision.

However, the model does not learn only mathematically exact rules of projective geometry.

It can also learn patterns associated with object shapes, spatial layouts, camera configurations, sensor characteristics, and errors present in the training data.

For example, strong performance on ordinary indoor scenes does not necessarily imply equally strong performance on specialized industrial equipment, transparent components, extreme close-up photographs, or unfamiliar camera systems.

Generalization under such conditions must be evaluated independently.

11.5 Assumption 5: 3D Can Be Predicted Even When Observations Are Insufficient#

This is both a major advantage and an important reliability challenge.

Consider a single frontal photograph of an object.

The shape of the object's hidden back surface cannot generally be determined uniquely from that photograph.

Similarly, even with multiple images, depth estimation may remain ambiguous when the viewpoints are nearly identical and there is insufficient parallax.

Traditional SfM often struggles in these situations.

Learning-based models such as VGGT may still produce 3D predictions by using learned priors.

However, these predictions do not necessarily represent the only physically correct solution supported by the observations.

Distinguishing geometry directly supported by observations from geometry inferred through learned priors is an important research challenge.

11.6 Assumption 6: Jointly Learning Multiple Geometric Quantities Improves 3D Understanding#

VGGT's Over-complete Prediction jointly learns related geometric outputs.

The original paper provides experimental evidence supporting the effectiveness of this approach. [1]

However, jointly learning several quantities does not guarantee that the resulting predictions are mathematically consistent.

For example, 3D points computed from predicted cameras and depth may differ from directly predicted point maps.

It is therefore necessary to distinguish between using redundant outputs to improve learning and guaranteeing consistency between those outputs.

11.7 Assumption 7: Learning Can Replace Inference-time Geometric Optimization#

VGGT aims to substantially reduce the complex test-time optimization required by traditional SfM.

However, this does not mean that geometric constraints have become irrelevant.

A more accurate interpretation is that part of the geometric reasoning previously performed through explicit computation has been transferred into the parameters of a pretrained model.

The original VGGT paper also reports performance improvements from adding Bundle Adjustment. [1]

Therefore, while VGGT demonstrates strong predictive capabilities, it does not establish that inference-time geometric optimization is unnecessary in every situation.

11.8 Assumption 8: Confidence Estimates Reflect Prediction Quality#

VGGT predicts confidence values for outputs such as depth and point maps.

However, these confidence values should not automatically be interpreted as real-world measurement errors or calibrated probabilities of correctness.

For example:

• How small are the actual 3D errors for points assigned high confidence?
• Does confidence decrease appropriately for unfamiliar environments?
• Does confidence decrease when unrelated images are provided?
• Can the model recognize when the available observations are insufficient for reliable reconstruction?

These questions must be evaluated independently before confidence estimates can be used as reliability indicators in practical systems.

12. Challenges and Limitations Reported in the Original VGGT Paper#

This section focuses on limitations discussed in the original VGGT paper.

12.1 Fisheye Images, Panoramic Images, and Extreme Rotations#

The original VGGT has limitations when processing fisheye images, panoramic images, and other capture configurations that differ substantially from its assumed camera model.

Performance degradation is also reported for images with extreme rotations. [16]

These are not simply problems related to image content.

They also reflect the assumptions made about camera projection and image formation.

12.2 Large Non-rigid Deformations#

Scenes containing pedestrians, deformable objects, or substantial motion are difficult to reconstruct under a static-scene formulation.

The original VGGT paper identifies large non-rigid deformations as a limitation. [16]

VGGT-Ω improves dynamic scene performance, but this does not mean that all challenges in dynamic reconstruction have been solved. [15]

12.3 The Absolute Scale Problem#

The scale of a 3D scene predicted by standard VGGT does not necessarily correspond to an absolute physical scale such as 1.00 meter.

During training, VGGT normalizes 3D quantities using distances derived from point maps. [16]

This is related to the inherent scale ambiguity of multi-view reconstruction.

For example, scaling both the entire scene and the camera translation by the same factor can preserve the projected image geometry.

Consequently, even if the relative shape of a reconstructed scene is correct, its coordinates cannot automatically be treated as real-world measurements.

Applications requiring physical distances need additional information or procedures for establishing absolute scale, such as known reference lengths, external sensors, or calibrated camera information.

12.4 Computational Cost with Many Images#

VGGT uses Global Attention to integrate information across multiple images.

Although this architecture is powerful, computational cost and GPU memory requirements increase with the number of input images.

The original paper reports inference times and GPU memory usage for the Transformer backbone on an NVIDIA H100 using 336×518 images. [16]

Number of Input Images	Inference Time	Peak GPU Memory
10	0.14 s	3.63 GB
50	1.04 s	11.41 GB
100	3.12 s	21.15 GB
200	8.75 s	40.63 GB

These measurements apply to the backbone under the conditions specified in the paper.

They do not represent the total runtime of a complete reconstruction system, which may also include image loading, output conversion, point cloud generation, Bundle Adjustment, NeRF training, or 3DGS construction.

Subsequent implementation improvements may also change practical runtime and memory requirements.

VGGT-SLAM addresses large sequences by dividing them into submaps. [12]

VGG-T³ also addresses the scaling limitations of VGGT-style attention as the number of input views grows. [17]

Therefore, although VGGT advances direct multi-view inference, efficiently processing large image collections remains an active research topic.

12.5 Geometric Consistency#

VGGT uses learned knowledge to rapidly predict plausible 3D geometry.

However, its outputs are not automatically guaranteed to satisfy all projective geometry constraints exactly.

For example, predicted camera poses and 3D point maps may contain inconsistencies across images.

In such situations, Bundle Adjustment and other geometric optimization methods can improve agreement between observations and predictions.

The official VGGT implementation provides a workflow for exporting predictions in COLMAP format and optionally running Bundle Adjustment. [10]

This points toward an important practical architecture:

Rapid 3D Prediction with a Foundation Model → Geometric Optimization When Necessary → 3D Model Construction and Rendering

It is a hybrid approach that combines learned inference with classical geometric methods.

13. A New Problem Highlighted by Research in 2026: 3D Hallucinations#

One particularly important study for understanding VGGT's limitations is:

"Can These Views Be One Scene? Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate," published in 2026. [18]

This study investigates cases where learning-based 3D models predict geometry even when the input images do not form a valid set of observations of one coherent 3D scene.

13.1 Producing 3D Geometry Does Not Mean the Geometry Is Correct#

Consider a set of photographs captured in completely different locations.

These images do not necessarily represent different views of the same static 3D scene.

Nevertheless, learning-based 3D models may still produce geometric predictions for such inputs.

The study reports that models including VGGT, MASt3R, DUSt3R, and Fast3R may generate dense geometry or apparent cross-view consistency when given unrelated images, repeated images, or random noise. [18]

This phenomenon occurs when the model produces geometric structures that are not sufficiently supported by the actual observations.

Such behavior can be described as 3D hallucination.

This does not mean that VGGT always produces incorrect geometry under normal conditions.

Rather, the important implication is that the mere existence of a 3D prediction is not evidence that the prediction is correct.

13.2 Why Is This Different from Classical SfM?#

Classical SfM may fail when image correspondences are insufficient or geometric constraints cannot be satisfied.

Such failures are inconvenient, but they can also indicate that the input images are unsuitable for reconstruction.

In contrast, a feed-forward model such as VGGT may still produce a prediction when the input is inappropriate.

This is part of what makes learning-based reconstruction powerful.

It may recover plausible geometry in situations where traditional methods struggle.

However, this capability also makes it more difficult to distinguish successful inference from unreliable predictions.

In other words, classical SfM and VGGT can fail in different ways.

Perspective	Classical SfM	Feed-forward Models Such as VGGT
Reconstruction mechanism	Explicit geometry and optimization based on correspondences	Direct prediction with a pretrained model
Insufficient observations	Reconstruction may fail	Predictions may still be generated using learned priors
Unrelated images	Geometric verification may reject inconsistent inputs	Incorrect reconstruction may occur without explicit outlier handling
Output verification	Reprojection error, registration success, and other geometric measures	Independent verification is particularly important
Main strength	Explicit use of geometric constraints	Rapid prediction even under difficult observation conditions
Main challenge	Sensitive to correspondence failures and insufficient observations	Must detect incorrect predictions driven by learned priors

The two approaches have different strengths and weaknesses.

The important question is not whether one completely replaces the other.

Rather, it is how to combine learning-based prediction with classical geometric verification.

13.3 The Problem of Using a Geometry Foundation Model as Its Own Evaluator#

Another interesting challenge arises when models such as VGGT are used to evaluate whether a collection of images is geometrically consistent.

Suppose we want to determine whether several images can be explained by one coherent 3D structure.

If the model itself generates an incorrect shared structure, the resulting predictions may appear mutually consistent even when the original images are not.

This creates a problem for consistency metrics based on learned 3D predictions.

The study compares learning-based measures with approaches based on classical geometric verification.

It investigates indicators such as image matching, registration success, dense 3D support, and reconstruction failure. [18]

The results highlight the importance of separating two components:

A model that generates 3D geometry and a mechanism that evaluates whether the generated geometry is trustworthy.

14. Detecting Outlier Views: Robust VGGT#

VGGT assumes that input images observe a common 3D scene.

However, this assumption is not always satisfied in real-world image collections.

For example, a large collection of photographs may include unrelated images or images with almost no overlap with the others.

Such images are often called outlier views or distractor images.

The 2026 study "Emergent Outlier View Rejection in Visual Geometry Grounded Transformers" addresses this problem. [19]

14.1 Outlier Information Can Emerge in VGGT's Internal Representation#

The researchers analyze VGGT's internal attention patterns and feature representations.

They report that even though VGGT does not explicitly implement an outlier rejection mechanism, certain internal representations contain information useful for identifying outlier views.

The study proposes a way to exploit this behavior to reject outlier images without additional fine-tuning or supervised training. [19]

This is an interesting result.

VGGT does not necessarily perform explicit geometric verification as part of its basic inference procedure.

Nevertheless, its internal representation may contain information that can help assess geometric relationships between images.

14.2 Does This Solve the Outlier Detection Problem?#

Robust VGGT demonstrates a useful approach to outlier view rejection.

However, it does not establish that every inappropriate input can now be detected perfectly.

For example, further evaluation is needed for situations such as:

• Most input images being outliers.
• Images of different but visually similar scenes being mixed together.
• Significant structural changes occurring at the same location over time.
• Every input image consisting of noise or otherwise invalid data.

Therefore, outlier detection performance and the correctness of the generated 3D geometry should be evaluated separately.

Nevertheless, Robust VGGT suggests that internal representations learned for 3D prediction may also be useful for improving prediction reliability.

15. What Has Not Yet Been Sufficiently Validated?#

The studies discussed so far demonstrate that VGGT has strong 3D estimation capabilities.

However, some important questions cannot be considered fully answered by the original paper and the major subsequent studies alone.

Here, "not sufficiently validated" does not mean that no research exists.

It means that additional independent evaluation is necessary before conclusions can be generalized to broader environments or practical operating conditions.

Research Challenge	Key Question	Practical Importance
Geometric consistency	Are cameras, depth, and point maps consistent with a single 3D structure?	3D modeling, measurement
Uncertainty calibration	Do confidence values correspond to actual 3D errors?	Quality control, failure detection
Input validity	Can the model identify unrelated images or invalid image collections?	Automated reconstruction
Generalization	Does accuracy remain stable for unfamiliar objects, materials, and capture conditions?	Real-world deployment
Observation vs. inference	Which parts of the predicted geometry are supported by images, and which come from learned priors?	Reliability assessment
Metric accuracy	What physical error bounds can be evaluated or guaranteed in meters or millimeters?	Surveying, facility management
Long-term stability	Can consistent geometry be reconstructed from images captured at different times?	SLAM, change detection
Robustness to image selection	How do image count, order, duplication, and viewpoint selection affect predictions?	Automation
Dynamic 3D understanding	Can camera motion and object motion be separated correctly?	Robotics, 4D reconstruction
Downstream effectiveness	Do improvements in reconstruction accuracy translate into better change detection or control?	Practical systems

Three questions are particularly important.

15.1 Has Geometric Consistency Actually Been Learned?#

VGGT jointly predicts cameras, depth maps, point maps, and point tracks.

However, the degree to which these outputs satisfy strict geometric constraints should be evaluated independently.

Possible evaluations include:

• Reproject predicted 3D points into each image and measure errors relative to observed points.
• Compare 3D points computed from camera and depth predictions with directly predicted point maps.
• Compare 3D geometry predicted from different views.
• Measure how much geometric consistency improves before and after Bundle Adjustment.

However, a low reprojection error alone does not guarantee that the reconstructed geometry matches the real world.

Incorrect image correspondences may still produce an internally consistent but physically incorrect reconstruction.

Therefore, both internal consistency and comparison with independent ground truth are important.

15.2 Can Confidence Predict Actual Reconstruction Error?#

In practical applications, detecting prediction failures can be just as important as achieving high average accuracy.

Suppose VGGT assigns high confidence to a particular 3D point.

How accurately does that confidence predict the point's actual geometric error?

One way to investigate this is to measure the distribution of 3D errors for different confidence levels.

Another useful evaluation is to remove low-confidence points and examine how reconstruction accuracy changes among the remaining points.

The objective is not merely to measure average prediction accuracy.

It is to determine how well the model represents uncertainty in its own predictions.

15.3 Can VGGT Recognize When It Does Not Know the Answer?#

When an image is severely blurred or multiple views contain almost no parallax, there may not be enough information for accurate reconstruction.

Likewise, if the input images are unrelated, reconstructing them as a single scene may be fundamentally inappropriate.

If the model still produces geometry, downstream systems must distinguish between "a prediction was produced" and "the prediction is trustworthy."

The important objective is not to generate geometry for every possible input.

Instead, it is to recognize situations where reliable estimation is impossible and detect predictions that should not be trusted.

16. What Experiments Are Needed to Evaluate VGGT's Reliability?#

Average accuracy comparisons between VGGT and COLMAP are useful but not sufficient.

It is also important to investigate how VGGT behaves when the assumptions underlying its predictions are violated.

The following experiments are proposals derived from the research discussed in this article.

They should not be interpreted as experiments already demonstrated in the original VGGT paper.

16.1 Testing the Common-scene Assumption#

Start with a set of images depicting the same 3D scene.

Then introduce unrelated images into the set.

For example, vary the proportion of outlier images from 0% to 50% and observe how reconstruction quality changes.

Evaluation metrics may include camera pose error, reprojection error, 3D reconstruction accuracy, and outlier detection performance.

This would help determine how robust VGGT is to invalid image collections.

16.2 Investigating the Relationship Between Observation Quality and 3D Prediction#

Gradually change the viewpoint differences between input images.

Compare images captured from nearly identical viewpoints with images containing sufficient parallax.

Also reduce the number of input images and measure the resulting changes in depth and point map accuracy.

These experiments may provide insight into how strongly VGGT depends on direct observations versus learned geometric priors.

16.3 Evaluating Consistency Between Cameras, Depth, and Point Maps#

Compare different VGGT outputs independently.

For example, reconstruct 3D points from predicted cameras and depth maps, then compare them with directly predicted point maps.

Measure whether predictions of the same 3D points agree across multiple images.

Additionally, compare geometric errors before and after Bundle Adjustment.

This would help clarify the respective roles of feed-forward inference and explicit geometric optimization.

16.4 Testing Generalization to Unfamiliar Capture Conditions#

Evaluate fisheye images, reflective and transparent objects, low-light scenes, motion blur, and specialized industrial environments.

The objective should go beyond comparing average accuracy.

It is important to identify which conditions cause performance degradation and whether confidence estimates respond appropriately.

This would help define the practical operating conditions under which VGGT can be used reliably.

16.5 Testing Temporal Changes and Dynamic Objects#

Capture the same environment at different times.

Introduce controlled changes such as adding, removing, or moving objects.

Then evaluate how well the system distinguishes camera viewpoint differences from actual changes in 3D structure.

This experiment is particularly important for image change detection and long-term environmental monitoring.

16.6 Evaluating Independent Geometric Verification#

Compare VGGT alone with systems that combine VGGT and geometric verification.

Possible configurations include:

• VGGT only.
• VGGT + Outlier Rejection.
• VGGT + Bundle Adjustment.
• VGGT + Geometric Verification + Bundle Adjustment.

Compare reconstruction errors, failure detection rates, and computational cost.

Such experiments would clarify the practical value of combining VGGT with classical geometric computation.

Comparisons could also include COLMAP, DUSt3R, MASt3R, VGGT-Ω, and MapAnything.

However, fair evaluation requires careful control of input resolution, image count, camera calibration assumptions, and whether Bundle Adjustment is used.

17. Looking Back at the Research History: What Changed, and What Remains?#

The evolution of these technologies can be summarized chronologically.

17.1 Classical SfM / MVS#

Camera poses and 3D structure are computed using feature correspondences and projective geometry.

COLMAP is a representative implementation. [2]

17.2 Deep Features / Matching / Depth#

Neural networks improve individual stages of SfM, including feature extraction, matching, and depth estimation.

17.3 VGGSfM (2024)#

The SfM pipeline becomes trainable end to end.

However, geometric optimization such as Bundle Adjustment remains part of the architecture. [3]

17.4 DUSt3R (2024)#

Point maps are directly predicted from image pairs.

This changes the traditional sequence of first estimating camera poses and then triangulating points. [4]

17.5 MASt3R (2024)#

3D-aware features are used to improve image matching. [5]

17.6 Fast3R / CUT3R / VGGT and Related Methods (2025)#

Research progresses from pairwise processing toward models that directly handle multiple views.

VGGT jointly predicts cameras, depth, point maps, and point tracks from multiple images. [1][7]

17.7 Geometry Foundation Models (2025–2026)#

Systems and models such as VGGT-SLAM, AnySplat, MapAnything, and VGGT-Ω emerge. [11][12][14][15]

VGGT-style models are increasingly used as geometry backbones for SLAM, 3D Gaussian Splatting, robotics, and other applications.

17.8 Toward Trustworthy Geometry Foundation Models#

Research in 2026 also introduces VGGT-SLAM++, Robust VGGT, and studies investigating hallucinations in 3D Foundation Models. [13][18][19]

These studies address problems that go beyond improving reconstruction accuracy:

• Detecting outlier images.
• Improving geometric consistency.
• Correcting errors accumulated during long-term mapping.
• Verifying whether predicted geometry truly represents a shared scene.

This suggests a broader direction for practical systems:

Feed-forward Learned Geometry + Classical Optimization + Neural Rendering

Together with:

Geometric Verification + Uncertainty Estimation + Failure Detection

In other words, pretrained models can rapidly estimate 3D structure, classical geometric optimization can improve consistency when necessary, and NeRF or 3D Gaussian Splatting can provide scene representation and rendering.

At the same time, mechanisms for evaluating whether the geometry can be trusted become increasingly important.

18. The Fundamental Contribution of VGGT: From Computing 3D Geometry to Learning to Predict It#

The significance of VGGT is not simply that a faster alternative to traditional SfM has emerged.

Traditionally, reconstructing 3D geometry from images required sequential operations such as feature matching, camera pose estimation, triangulation, and Bundle Adjustment at inference time.

VGGT incorporates many of the geometric relationships underlying these operations into a pretrained model through learning on large-scale 3D datasets.

During inference, a single Transformer predicts multiple geometric quantities jointly.

Therefore, a central difference from traditional approaches is that some of the computations and optimizations previously performed explicitly at inference time have been transferred into pretraining.

However, this does not make geometry unnecessary.

On the contrary, combining learned geometry predictions with classical optimization, NeRF, and 3D Gaussian Splatting can lead to more practical reconstruction systems.

For applications such as point cloud generation, 3D modeling, feature matching, image alignment, and change detection, VGGT should not necessarily be regarded as a complete replacement for every individual processing component.

Instead, it can be understood as a foundational model that provides integrated information about 3D space.

In this sense, VGGT is not simply a replacement for COLMAP. It introduces a new geometry backbone that can be integrated before, after, or alongside technologies such as COLMAP, SLAM, and 3D Gaussian Splatting.

However, examining VGGT's limitations reveals another important change.

Predicting 3D geometry and verifying whether that geometry is correct are becoming increasingly important as separate problems.

Classical SfM uses image correspondences and geometric constraints to calculate 3D structure consistent with observations.

VGGT uses learned 3D priors to produce predictions even in conditions where traditional reconstruction may be difficult.

But whether those predictions are truly supported by the observed images is not always clear.

In particular, when observations are insufficient or the input images do not belong to the same scene, a plausible-looking 3D reconstruction may still be incorrect.

This distinction is essential when integrating VGGT into practical systems.

For example, in applications focused primarily on visually appealing 3D rendering, some geometric error may be acceptable.

However, in physical measurement, facility management, robotic control, and precise image change detection, geometric errors may directly lead to incorrect decisions.

Depending on the application, systems may therefore need to incorporate absolute scale calibration, geometric consistency verification, confidence evaluation, and failure detection.

Developments such as VGGT-Ω and MapAnything demonstrate that research is moving beyond specialized reconstruction algorithms toward general-purpose Geometry Foundation Models.

Meanwhile, studies such as VGGT-SLAM++ and Robust VGGT show that classical geometric computation and reliability evaluation remain important for deploying those models in real systems.

Looking back at the broader research history, the transition can be summarized as follows:

From an era in which 3D reconstruction was performed through a sequence of specialized algorithms, toward an era in which pretrained models directly predict 3D geometry.

Beyond that transition lies another important research direction:

Developing Geometry Foundation Models that can not only predict 3D geometry, but also determine whether their predictions are trustworthy.

VGGT is one of the most important studies for understanding this transition.

And the next major question for research following VGGT may be:

"VGGT has substantially advanced our ability to predict 3D geometry. But to what extent can the model itself provide evidence that its predictions are actually correct?"

The development of stronger 3D prediction capabilities and the development of methods for verifying prediction reliability are both likely to be important in the future of Geometry Foundation Models.

References#

[1] Jianyuan Wang et al., “VGGT: Visual Geometry Grounded Transformer,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5294–5306, 2025.
https://openaccess.thecvf.com/content/CVPR2025/html/Wang_VGGT_Visual_Geometry_Grounded_Transformer_CVPR_2025_paper.html

[2] COLMAP, “COLMAP Documentation,” Project Documentation.
https://colmap.github.io/

[3] Jianyuan Wang et al., “VGGSfM: Visual Geometry Grounded Deep Structure From Motion,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
https://openaccess.thecvf.com/content/CVPR2024/html/Wang_VGGSfM_Visual_Geometry_Grounded_Deep_Structure_From_Motion_CVPR_2024_paper.html

[4] Shuzhe Wang et al., “DUSt3R: Geometric 3D Vision Made Easy,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
https://openaccess.thecvf.com/content/CVPR2024/html/Wang_DUSt3R_Geometric_3D_Vision_Made_Easy_CVPR_2024_paper.html

[5] Vincent Leroy et al., “Grounding Image Matching in 3D with MASt3R,” Proceedings of the European Conference on Computer Vision (ECCV), 2024.
https://www.ecva.net/papers/eccv_2024/papers_ECCV/html/9080_ECCV_2024_paper.php

[6] VGGT Authors, “VGGT: Visual Geometry Grounded Transformer,” Official Project Page, 2025.
https://vgg-t.github.io/

[7] Yang et al., “Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass,” arXiv:2501.13928, 2025.
https://arxiv.org/abs/2501.13928

[8] Visual Geometry Group, University of Oxford, “Publications,” Research Publications.
https://www.robots.ox.ac.uk/~vgg/publications/index.php?idAuthor=131&pg=2

[9] Ben Mildenhall et al., “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” Proceedings of the European Conference on Computer Vision (ECCV), 2020.
https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/1473_ECCV_2020_paper.php

[10] Meta Research, “VGGT: Visual Geometry Grounded Transformer,” GitHub Repository, 2025–2026.
https://github.com/facebookresearch/vggt

[11] AnySplat Authors, “AnySplat,” SIGGRAPH Asia, 2025, GitHub Repository.
https://github.com/OpenRobotLab/AnySplat

[12] VGGT-SLAM Authors, “VGGT-SLAM,” arXiv:2505.12549, 2025.
https://arxiv.org/abs/2505.12549

[13] Avilasha Mandal, Rajesh Kumar, Sudarshan Sunil Harithas, and Chetan Arora, “VGGT-SLAM++,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1008–1018, 2026.
https://openaccess.thecvf.com/content/CVPR2026W/VOCVALC/html/Mandal_VGGT-SLAM_CVPRW_2026_paper.html

[14] MapAnything Authors, “MapAnything,” arXiv:2509.13414, 2025.
https://arxiv.org/abs/2509.13414

[15] VGGT-Ω Authors, “VGGT-Ω,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, Official Project Page.
https://vggt-omega.github.io/

[16] Jianyuan Wang et al., “VGGT: Visual Geometry Grounded Transformer,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, Paper PDF.
https://robots.ox.ac.uk/~vgg/publications/2025/Wang25/wang25.pdf

[17] Elflein et al., “VGG-T³: Offline Feed-Forward 3D Reconstruction at Scale,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.
https://openaccess.thecvf.com/content/CVPR2026/papers/Elflein_VGG-T3_Offline_Feed-Forward_3D_Reconstruction_at_Scale_CVPR_2026_paper.pdf

[18] Soumava Paul, Prakhar Kaushik, and Alan Yuille, “Can These Views Be One Scene? Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate,” arXiv:2605.18754, 2026.
https://arxiv.org/abs/2605.18754

[19] Jisang Han, Sunghwan Hong, Jaewoo Jung, Wooseok Jang, Honggyu An, Qianqian Wang, Seungryong Kim, and Chen Feng, “Emergent Outlier View Rejection in Visual Geometry Grounded Transformers,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427–437, 2026.
https://openaccess.thecvf.com/content/CVPR2026/html/Han_Emergent_Outlier_View_Rejection_in_Visual_Geometry_Grounded_Transformers_CVPR_2026_paper.html