Unlocking the next level of 3D shape registration! Our cutting-edge method seamlessly aligns 3D shapes with 2D keypoints captured from multiple cameras, pushing the boundaries beyond single-image techniques. By leveraging a generalised camera model, we introduce three powerful approaches - each tailored to different levels of known spatial information and even incorporating object silhouettes for enhanced accuracy. This multi-view strategy significantly improves reconstruction precision, opening doors to new applications in medical imaging and handheld camera registration. Explore how our approach reshapes the future of 3D modeling!
Introducing Non-Rigid Shape Registration with Generalised Cameras
Non-rigid 3D shape registration—matching a deformable 3D template to observed 2D keypoints—has been extensively studied using standard camera models:
- Perspective cameras for real-world observations
- Orthographic cameras as a weak approximation for perspective cameras
- Weak-perspective cameras as a stronger approximation for perspective cameras
However, many real-world applications involve multiple cameras observing a deforming object, necessitating a multi-camera approach for joint registration.
Key Applications:
- Medical imaging: Multiplanar radiography requires shape registration for anatomical structures.
- Camera networks: Multi-camera setups observe moving objects, but traditional methods struggle with integrating multiple perspectives.
To solve this, we introduce generalised camera models, which:
- Represent each keypoint with a camera center and direction vector
- Overcome limitations of traditional projection models
- Enable accurate reconstruction from diverse multi-camera setups
The general problem setup is graphically summarised below:
Core Challenges and Innovations
Unlike rigid scenes, deformable objects lack standard epipolar constraints:
- Known extrinsics: Shape can be reconstructed using convex programming.
- Unknown extrinsics: Requires estimating both shape and camera parameters.
Our method provides:
- Higher accuracy than single-camera approaches
- Flexibility to handle varying camera setups
- Freedom to reconstruct volumetric objects from non-overlapping views
- The first convex solution for non-rigid shape registration with generalised cameras
This breakthrough broadens the scope of 3D shape registration, making it viable for diverse and challenging real-world scenarios.
Key Contributions
- Generalised Camera Model: Introduced for multi-camera non-rgid shape registration.
- Convex Shape Reconstruction: Solves registration with known camera poses (NS problem).
- Joint Shape & Pose Estimation: Estimates both shape and unknown camera poses (NSC problem).
- Silhouette-Based Refinement: Iteratively improves accuracy, ensuring optimal results.
Results
We show a sample result from a biplanar X-ray setup below:
