Vision, Language, Action

VLA models let robots act on what they see and what they are told, instead of following fixed paths. What holds them back today is less the model than the data: enough varied, realistic robot experience is hard to get. That is the problem I find most interesting, and I think simulation, real deployments and solid hardware all have to contribute to solving it.

Coming from mechatronics and autonomous robots, this is the field I follow most closely. A model that sees, understands an instruction and turns it into a physical action means robots no longer depend on rigid, structured environments. The more I look into it, the more it comes down to one bottleneck.

The data bottleneck

Training a capable VLA policy takes an enormous amount of interaction data, and each way of getting it has a catch. Simulation produces data at scale, but every simulated environment differs from the real world in physics, materials and sensor noise. This sim-to-real gap means a policy trained only in simulation often falls apart when it meets reality. Teleoperation on real hardware avoids that gap, but it is bound to real time: you can't parallelize a human operating a robot arm the way you can parallelize simulations on a GPU cluster, and every trajectory costs human time on an expensive system.

A third source is data from robots that are already working in the field, together with human video. It grows with every real application and comes from real conditions, but it is messy, unevenly distributed, and mostly available to those who operate robots at scale. I expect the mix of all three to matter more than any single source.

Where I look

On the simulation side, the lever is to build environments realistic and varied enough (better physics, better rendering, deliberate domain randomization) that a policy trained or finetuned there needs only a fraction of the real interaction data. Real data then goes where it counts most: closing the remaining gap.

This is why I follow what NVIDIA is doing with Omniverse and Cosmos. Omniverse provides physically accurate simulated environments, and Cosmos generates photorealistic, physically plausible synthetic video that can be used to post-train robot policies, which attacks the gap from the data-generation side. Genesis is an open-source physics engine that, according to its developers, runs 10–80x faster than GPU simulators like Isaac Gym, and it aims to generate scenes, tasks and trajectories from language prompts. If that works well in practice, environment variation becomes almost free. I'm watching how these tools hold up outside of demos.

To get hands-on, I built a small simulation in Genesis of a Franka Emika Panda arm picking up a cube and placing it elsewhere, using inverse kinematics for the approach and lift and position control for the gripper. Small in scope, but a good way to feel how quickly you can iterate on a manipulation task in simulation.

What I bring

None of these approaches closes the gap completely yet. My own background points at the same questions from the hardware side. At Evomotiv I developed autonomous robot platforms from the requirements list to testing, which showed me how much real-world detail a robot has to cope with. In my master's thesis I used surrogate models to reduce the number of expensive 3D simulations, which is about spending costly samples wisely. That is why I want to work where simulation, AI and physical robots meet.