Nvidia unveiled the Cosmos platform, powered by world models that Physical AI developers will use to train video analytics AI agents, AVs (Autonomous Vehicles), and robots. The company claims Cosmos world models use structured reasoning on images and videos to “understand the physical world like humans.”
The tech company said it was helping developers build foundational models. It explained that the Cosmos platform allowed developers to customize out-of-the-box pretrained models for specialized physical AI models. Nvidia boasts that Cosmos uses a “spatiotemporal understanding” of the physical world to curate data that trains decision-making in robotics and self-driving cars.
The company also added that the Cosmos Curator framework enabled developers to filter, annotate, and deduplicate massive amounts of sensor data. Developers use this data to create tailored datasets that meet specific physical AI needs. Cosmos world foundation models can also generate data for downstream pipelines in developing industrial vision systems.
Cosmos comes with Predict, Transfer, and Reason foundation models
According to the Nvidia team, the Cosmos platform has the Predict prediction model, allowing developers to generate continuous videos of up to 30 seconds. The videos are generated from multimodal inputs with strict adherence to prompts.
Transfer is a multicontrol model that allows developers to simulate different environments and lighting conditions. The tech company also said Transfer can accelerate 3D inputs from CARLA and Nvidia Isaac Sim physical AI simulation frameworks to enable “controllable data augmentation.”
Nvidia stated that Cosmos Reason used a fully customizable VLM (Vision Language Model) that understood the real physical world like humans. Reason powers video analytics agents that understand operations in industrial and city spaces. It curates the training data used for decision-making.
The tech company disclosed that developers could leverage the foundation models to generate data for training AI models in industrial and robotics applications, such as factory robots, automated warehouses, and AVs on highways or rough terrains.
Nvidia also said these foundational models were trained using unlabeled datasets to generate new data based on user inputs. It added that developers can use this generalizability to fine-tune the pretrained models using smaller datasets to build custom models. The developers can also train different autonomous machines to sense and interact with various surroundings.
Nvidia powers ‘digital twins’
The tech company announced the release of the Omniverse libraries on August 11. Nvidia added that the libraries were powered by its RTX PRO Servers and DGX Cloud, allowing developers to build physically accurate digital twins. Synthetic data can be generated by capturing and reconstructing the real world in simulation to build AI agents and train physical AI models.
Rev Lebaredian, the vice president of Omniverse and Simulation Technologies at Nvidia, said his company was committed to enabling developers to build tomorrow’s robots and AVs. He explained that AI and computer graphics were converging to transform the basic principles of robotics. Lebaredian believes these technologies will “transform trillions of dollars in industries.”
Nvidia disclosed that the Omniverse libraries and SDKs (Software Development Kits) were now available for developers to build and deploy robotics simulation and industrial AI applications. The SDKs enable data interoperability between OpenUSD (Universal Scene Description) and MJCF (MuJoCo) to allow robots to be simulated across platforms. The ‘RTX ray-traced 3D Gaussian splatting’ technique also allows developers to capture, reconstruct, and simulate physical environments in the real world in 3D using sensor data.
Nvidia claimed that Figure AI, Skild AI, Boston Dynamics, RAI Institute, Hexagon, and Lightwheel adopted Omniverse and the Isaac Suite (Sim and Lab) to accelerate their AI-driven robotics projects. Amazon Devices & Services also used these Nvidia systems to power its latest manufacturing solutions.
