SMPLify-X: Generate an accurate 3D model from single image

SMPLify-X is an open-source project that creates an expressive 3D body capture with detailed hands, face, and body from a single image. This project is a result of a research of 7 researchers and computer scientists at Max Planck ETH Center for Learning Systems.

The researchers train AI to rich dataset to provide detailed 3D reconstruction that recognize human body pose, hand gestures, and facial experssion. Unlike other methods which are using multiple cameras to reconstruct 3D model, SMPLify-X uses only one image as an input.

The team uses large dataset of 3D scans to produce expressive realistic model. They created a body pose prior that is based on a large dataset of thousands of body pose models.

The project uses PyTorch (an open source machine learning framework), and PyRender (a lightweight Python library for 3D rendering and visualization which is also open-source licensed under MIT license).

The Project also uses VPoser which is a variational human body pose prior by the same researchers. It's released also for free for non-commercial use.

Features

  • 2D body pose detector with support of body keypoints, hand keypoints, and facial landmarks.
  • Using detailed collision-based model for meshes.
  • Gender classification.
  • EHF (Expressive Hand Face) dataset for evaluation.
  • PyTorch implementation achieves a speedup of more than 8x over Chumpy.

License

The source code is released for free for non-commercial scientific research purposes. Contact [email protected] for commercial licensing (and all related questions for business applications).

Project Disclaimer

The original images used for the figures 1 and 2 of the paper can be found in this link. The images in the paper are used under license from gettyimages.com. We have acquired the right to use them in the publication, but redistribution is not allowed. Please follow the instructions on the given link to acquire right of usage. Our results are obtained on the 483 × 724 pixels resolution of the original images.

Citation code

References






Netron: Open-source Visualizer for Deep Learning, Machine Learning, and Neural Network Models

Netron is an open-source multi-platform visualizer and editor for artificial intelligence models. It supports many extensions for deep learning, machine learning and neural network models.  Netron is using Electron/ NodeJS and it has a bin.......Read more...

DeepFly3D: a deep learning software for 3D limb and appendage tracking

A group of scientists from EPFL (École polytechnique fédérale de Lausanne) in Switzerland have developed a deep-learning based motion-capture software that uses multiple camera views to model the movements of a fly (Drosophila melanogaster).......Read more...

Antimicrobial Resistance Prediction using Game Theory Algorithm

The increasing prevalence of antibiotic-resistant bacteria is a growing problem around the world. Every year, millions of people are infected with drug-resistant pathogens, and a lot of people die from pneumonia or bloodstream infections. I.......Read more...

IBM Open-Sourced three Cancer Research AI Projects

IBM Research lab in Zurich Open-Sourced three AI Projects to help accelerate cancer research. The first project named PaccMann (Prediction of anticancer compound sensitivity with Multi-modal attention-based neural networks), is a multimodal.......Read more...

Deep Learning Framework to delineate Organs at Risk in CT Images of Patients undergoing Radiation Therapy

Radiation Therapy is a widely used treatment modality for many types of cancer, but it has potential adverse effects to healthy surrounding organs. So, accurately identifying all Organs at Risk (OARs) is a very important step in Radiation T.......Read more...