MIGU: Multimodal Instruction Grounding
under Uncertainty for Manipulation Planning

Mingke Lu1,2*, Anxing Xiao1*, David Hsu1

*denotes equal contribution, alphabetical order

1National University of Singapore2University of California, Los Angeles

Paper Task Videos

Key Message: Language and pointing provide complementary but uncertain cues. MIGU combines them into a unified grounding belief, enabling a robot to act on human instructions or ask for clarification when needed.

Framework Overview

MIGU framework: language and RGB-D observations produce geometric and semantic beliefs, which are fused to choose clarification or manipulation planning.
MIGU fuses geometric and semantic evidence into a grounding belief that supports behavior planning and downstream manipulation.

MIGU models pointing uncertainty in 3D and uses a vision-language model to estimate semantic priors over candidate objects and regions. Bayes-inspired fusion combines these cues into a shared belief. A behavior planner weighs direct execution against an attribute-based clarification question, and the grounded target becomes a goal for mobile manipulation or tabletop task-and-motion planning.

Real-World Benchmark Evaluation

Figure 7 compares MIGU with seven baselines across grounding accuracy, top-2 accuracy, scaled 2D deviation, negative log-likelihood, and Brier score. MIGU, shown in green, has the highest accuracy and top-2 accuracy, and the lowest deviation, NLL, and Brier score among the reported methods.
Grounding performance against baselines (Fig. 7). MIGU is shown in green. Higher is better for accuracy and top-2 accuracy; lower is better for deviation, negative log-likelihood (NLL), and Brier score. NLL is omitted for deterministic methods. Click the chart to enlarge.

Across 100 real-world instruction-grounding scenarios, MIGU combines language and pointing to outperform the evaluated baselines in grounding accuracy and spatial precision. The lower NLL and Brier scores also indicate better probabilistic predictions, providing a stronger basis for deciding when to act and when to ask for clarification.

Tasks