Nvidia’s GANverse3D uses AI to create textured 3D models from 2D images

For the final day of GTC 2021, Nvidia revealed its latest AI developments. Utilising the Nvidia Omniverse platform and a new tool called GANverse3D, users can create 3D models, complete with mesh and textures, from a single 2D image, all using a trained AI. To showcase this tech in action, Nvidia recreated KITT – the AI powered car from 80's classic Knight Rider. 

GANverse3D is an application that takes flat 2D images and transforms them into realistic 3D models that can be visualised and controlled in virtual environments. A single photo of a car for example can be turned into a 3D model that can drive around a virtual scene with working functions, including headlights, tail lights and blinkers.

The AI was trained on a dataset of over 50,000 images showcasing multiple view points, allowing the AI to recognise different parts of a car. Nvidia has also trained the AI on birds and horses, but at this point in time, cars still work best. For instance, if you tried feeding the tool an image of a dog, a human or a tree, it would not be able to render a 3D model with the same level of accuracy due to geometric complexity.

Looking towards the future, this tool could enhance workflows for a number of industries, including architects and game developers. While the ‘main subject' of a scene would be modelled by hand, artists and developers could save time by using AI to generate ambient and background content without needing to do it by hand, or by purchasing ready-made assets.

The technology can be adapted to other datasets to produce different results, with Nvidia using birds and horses as the current example. For those really interested in the science behind all of this, you can check out the full ICLR paper, in a briefing with Nvidia attended by KitGuru, we were also told that code for this project will likely be released so other developers can collaborate and iterate on the technology and further AI research.

Discuss on our Facebook page, HERE.

KitGuru Says: Over the course of GTC 2021, we've seen a lot of really interesting developments in Nvidia's work with Artificial Intelligence. What do you think of the latest demo and the GANverse3D tool? 

Become a Patron!