Querying 3D Models with AI Agents

Executive Summary

High-fidelity 3D models provide rich visual records of physical spaces, but extracting insights still relies on manual human navigation. While autonomous agents can automate analysis, multimodal models process 2D images and cannot natively parse 3D formats like point clouds or Gaussian splats. We resolve this by generating a minimal set of 2D views covering the entire scene. By embedding each render into a structured scene matrix, we create a spatial data layer optimized for plain-language queries. Decoupling image embeddings from underlying geometry provides a scalable, practical foundation for autonomous spatial inspection.

Introduction

Gaussian Splatting ^[1] enables rapid 3D reconstruction and high-fidelity rendering, but these models still operate primarily as static visual records. Extracting compliance or safety data—such as verifying fire extinguisher placement or inspecting rack installations—requires manual human review.

Automating this analysis requires translating 3D spatial data into formats compatible with large multimodal models. While 3D-native vision-language models exist, they lack the extensive training datasets of their 2D counterparts ^[2]. Our prior methodology used original capture photography as an intermediary data layer: ranking source images against text queries, verifying matches via a multimodal model, and projecting coordinates into 3D space ^[3], ^[4].

Relying on original capture imagery, however, carries practical limitations:

• Missing Source Data: Imported spatial scans frequently lack original source photography.
• Misaligned Capture Intent: Original footage optimizes for structural reconstruction, not targeted component inspection.
• Fixed Viewpoints: Source photographs are locked to their initial poses. A poorly framed component cannot be reframed after capture without returning to the site.

Alternative systems embed language features directly into scene geometry, distilling image-text embeddings ^[5] into individual Gaussians or points (e.g., LangSplat ^[6], OpenGaussian ^[7]). This approach tightly couples geometry to a specific embedding model and demands significant preprocessing per scene. Our methodology preserves the original geometry, generating a discrete 2D data layer optimized for vision model processing.

System Architecture: Automatic Virtual Framing

We execute a pre-computation pipeline termed automatic virtual framing to construct the search index. The workflow operates in four steps:

1. Camera Initialization: Distribute candidate camera coordinates across the navigable free space within the 3D environment.
2. View Rendering & Evaluation: Render each candidate view and calculate its photographic fidelity. Because off-axis or distant renders introduce structural artifacts, a trained classifier operating on standard image embeddings ^[8] assigns a quantitative realism score to each frame. Candidates that narrowly fail are refined through small pose perturbations, retaining any adjustment that raises the score.
3. Coverage Optimization: Select views greedily, adding the high-realism view that observes the most uncaptured surface at each step to eliminate redundant camera angles.
4. Final View Set: Terminate selection once additional views yield only marginal coverage, producing a minimal, well-distributed set of virtual photographs that form the input to the scene matrix.

The Scene Matrix Data Structure

We process the finalized view set through a standard image-text embedding model ^[5]. The resulting outputs populate a scene matrix, allocating one row per virtual photograph and mapping each directly to its originating spatial coordinates.

During query execution, the agent embeds the search string and calculates similarity scores across the matrix to identify the highest-scoring camera poses. Isolating the embedding layer from the 3D geometry ensures system flexibility—updating the underlying embedding model requires only reprocessing the 2D renders. Every query result outputs a specific rendered view, providing a visual audit trail for human verification.

Operational Benchmarks

This decoupled architecture enables rapid indexing of imported spatial models that lack original source photography. We benchmarked this pipeline on a 1.8-million-Gaussian splat scan of a local cabin, intentionally withholding the original capture photographs.

The pipeline generated 122 virtual photographs, each rendered at a 1024 × 1024 pixel resolution, to achieve 95% surface coverage—a significant data reduction compared to the 362 original capture photos. The complete framing and embedding pipeline executed in less than two minutes.

Loading…

Find

Execution latency scales proportionally with Gaussian volume, as each candidate view requires sorting the scene's Gaussians prior to rendering. Across expanded testing, models containing 1–2 million Gaussians processed in one to two minutes, 5 million Gaussians in three minutes, and 7 million Gaussians in six minutes.

Conclusion

Digital twins deliver value when they can be analyzed and their insights directly impact business processes. Live exploration by an autonomous agent introduces high computational overhead on every query. By pre-computing a discrete network of interior renders and structuring them into an embedded scene matrix, we provide agents with an immediate, scalable mechanism for spatial search. This architecture establishes a practical data pipeline where every query result resolves directly to a specific, human-readable 3D coordinate and visual frame.

Citations