Recent probabilistic methods for 3D triangular meshes capture diverse shapes by differentiable mesh connectivity, but face high computational costs with increased shape details. We introduce a new differentiable mesh processing method in 2D and 3D that addresses this challenge and efficiently handles meshes with intricate structures. Additionally, we present an algorithm that adapts the mesh resolution to local geometry in 2D for efficient representation. We demonstrate the effectiveness of our approach on 2D point cloud and 3D multi-view reconstruction tasks.
DMesh++ can reconstruct complex 2D drawings from sample points extracted from them. On the left side of the image, both the "real" part (blue) and the "imaginary" part (gray) are rendered together. The imaginary portion is then discarded, leaving only the real part to define the final mesh.
DMesh++ can also reconstruct complex 3D shapes with colors from point clouds and multi-view images. On the bottom right, images are rendered for both the "real" part (blue) and the "imaginary" part (gray). By optimizing vertex-wise colors, we recover the texture information of the ground truth shape.
Computational speed comparison to DMesh [1].
DMesh++ can handle these complex shapes in 2D and 3D as it is based on the efficient Minimum-Ball algorithm.
Compared to DMesh, whose algorithm is inherently global, the Minimum-Ball algorithm considers local point configurations.
Therefore, DMesh++ achieves a much more efficient computational cost than DMesh.
Specifically, DMesh has a time complexity of O(N), while DMesh++ has a time complexity of O(logN), where N is the number of points.
[1] Son, Sanghyun, et al. "DMesh: A Differentiable Representation for General Meshes." arXiv preprint arXiv:2404.13445 (2024).
2D mesh complexity comparison to DMesh [1].
In 2D, DMesh++ can produce efficient yet accurate meshes that adapt to local geometry.
We introduce the Reinforce-Ball algorithm for adaptive resolution.
While DMesh also uses regularization to reduce mesh complexity, it often sacrifices geometric accuracy at lower complexities.
In contrast, DMesh++ achieves a much simpler mesh without compromising geometric accuracy.
[1] Son, Sanghyun, et al. "DMesh: A Differentiable Representation for General Meshes." arXiv preprint arXiv:2404.13445 (2024).
2D Font Reconstruction Process. In this task, we reconstruct 2D mesh from a given point cloud by minimizing Chamfer Distance between them. In this video, we render how the 2D mesh evolves during optimization. We render the global view of the mesh on the left, and the zoom-in view of it on the right. In the zoom-in view, we render the input point cloud together.
Reinforce-Ball Algorithm. In 2D reconstruction, we propose Reinforce-Ball algorithm to produce efficient mesh that adapts to local geometry. Note that the unnecessary vertices and edges are removed during the process. In this video, we illustrate real part and imaginary part together.
Left: Real part and imaginary part together / Right: Mesh (Real part) only.
In this task, we reconstruct a 3D mesh from multi-view color (or diffuse) and depth images taken from 64 viewpoints by minimizing a rendering loss. Below, we show videos of the 3D mesh as it undergoes reconstruction, side by side with four of the input images, each featuring a black background.
Started from grid (random) initialization (Left: Colored mesh, Right: Plain mesh with edges).
Started from point cloud initialization (Left: Colored mesh, Right: Plain mesh with edges).
We can reconstruct a small scene from multi-view images as shown below, and run physics simulation directly on the reconstructed mesh.
@misc{son2024dmeshefficientdifferentiablemesh,
title={DMesh++: An Efficient Differentiable Mesh for Complex Shapes},
author={Sanghyun Son and Matheus Gadelha and Yang Zhou and Matthew Fisher and Zexiang Xu and Yi-Ling Qiao and Ming C. Lin and Yi Zhou},
year={2024},
eprint={2412.16776},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2412.16776},
}