Uni-VLaT

Physical contact often determines how a humanoid should respond during loco-manipulation, yet vision and proprioception provide only indirect evidence of interaction, especially when the contact region is occluded. Distributed tactile sensing preserves spatially resolved contact patterns across the robot body.

Uni-VLaT adapts pretrained vision-language-action policies with a tactile pathway trained for both action generation and prediction of future tactile, proprioceptive, and visual representations. Contextualized tactile features anchor these complementary views of physical interaction. Across five real-robot tasks, Uni-VLaT achieves 75% average success, compared with 32% for No Tactile and 68% for tactile input without prediction. Evaluation with Isaac-GR00T and π_0.5, together with controlled ablations, supports the benefits of post-DiT tactile context and absolute future targets.

Tactile as a physical anchor. Tactile sensing directly observes contact at the robot body, proprioception captures the resulting body response, and vision captures changes in the scene. Uni-VLaT connects these complementary observations through future representation prediction while retaining the semantic, visual, and action priors of a pretrained VLA policy.

Distributed tactile sensors cover eight body regions: the chest, central back, left and right shoulders, left and right upper back, and left and right arms. Sensor readings are denoised and normalized by region. Independent regional MLPs map measurements to a common feature width. Eight learned attention queries aggregate the regional embeddings into eight spatial tactile tokens; each token can combine information from multiple body regions. Axial pooling reduces sensitivity to sleeve rotation while retaining contact location along the arm.

A lightweight temporal Transformer aggregates four causal tactile frames with learned time embeddings and a residual connection to the current frame. A learned scalar gate scales the tactile tokens before they join proprioceptive and noisy action tokens in the DiT trunk. Visual and language features enter through cross-attention.

After multimodal policy attention, the tactile tokens are pooled into a tactile-anchored cross-modal context. Three modality-specific predictors estimate four future steps of tactile, proprioceptive, and visual representations in their respective latent spaces.

The predictors learn absolute future latents in three separate modality spaces. Tactile and proprioceptive targets come from exponential-moving-average encoders; visual targets come from a frozen pretrained encoder. Flow-matching action supervision and the three predictive objectives jointly train the policy.

At deployment, the predictive heads and target encoders are omitted. The policy generates chunks of 64-dimensional motion tokens for the pretrained SONIC whole-body controller, which produces joint-position targets for coordinated upper- and lower-body execution.

On a Unitree G1 with Isaac-GR00T, Uni-VLaT achieves 75% average success across five contact-rich tasks, compared with 32% without tactile sensing, 68% with tactile input alone, and 69% with tactile-only prediction. Each task uses 50 demonstrations, and each main-table configuration is evaluated over 20 real-robot rollouts. Instability, falling, an emergency stop, or any human intervention counts as failure.

Compared with tactile input alone, Uni-VLaT improves Table Sweeping from 60% to 75% and Basket Loading from 65% to 80%. It reaches 80% on Human–Robot Hugging and 55% on Composed Cleanup. For Back-Tap Walking, tactile sensing is the dominant factor: all tactile-enabled variants perform well, and tactile-only prediction achieves the highest success rate of 90%.

Basket Loading tests adaptation to a changing physical load. The recorded Uni-VLaT rollout shows a lower contact response and changes around successive loading events, while No Tactile maintains a relatively high contact level after establishing support. These traces illustrate interaction behavior; they are not a calibrated force measurement or an aggregate safety metric.

The same tactile adaptation principle improves both Isaac-GR00T and π_0.5 on contact-guided sweeping and tactile-triggered walking. Within each comparison, methods share demonstrations, action targets, and the downstream control interface.

Context matters. Multimodal Prediction uses visual, tactile, and proprioceptive features to predict all three future modalities. Pre-DiT Tactile Prediction uses only pre-DiT tactile representations as the source for all three predictors. Full Uni-VLaT instead predicts from the contextualized post-DiT tactile states, achieving 80% average success versus 60% and 55% for these alternatives.

Persistent contact matters. Replacing absolute future latents with future-minus-current delta targets reduces average success from 80% to 35%. Absolute targets preserve persistent contact and body-state information as well as changes over time.

Uni-VLaT adapts pretrained VLA policies for contact-rich humanoid loco-manipulation by integrating whole-body tactile sensing and predicting future tactile, proprioceptive, and visual representations during training. Across five real-robot tasks, it achieves 75% average success with Isaac-GR00T. Cross-policy experiments and controlled ablations support the value of tactile-anchored multimodal context and absolute future targets for learning physical interaction.

Tactile arrays remain sensitive to sensor noise and mounting variation. Limited whole-body tactile simulation restricts large-scale robustness studies, and the real-robot evaluation covers five representative tasks. Tactile-as-instruction behaviors mainly benefit from access to contact signals; richer interactions demand finer tactile regulation and more precise contact control.