Creating a 3D photo from a regular image using the 3D-Photo-Inpainting library

Source & Credits:

3 min readApr 23, 2020

3D Photography using Context-aware Layered Depth Inpainting
Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

3D Photo creation using 3d-photo-inpainting deep learning library

Description (How it works):

A color single RGB-D input image is used as input and the 3D photo is generated in MP4 format
3D photo generation is based on the multi-layer representation where these layers contain hallucinated color and depth structures in regions occluded in the original view.
The library uses Layered Depth Image (as input) with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that iteratively synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner.
The final 3D photo result image can be efficiently rendered with motion parallax using standard graphics engines.

Foundation Requirement:

python 3.7
Conda or virtualenv — A python runtime environment
GPU is good to have for speed however if you don't have CPU is still supported
Tested OK, on both CentOS Linux and OSX, (No Windows Test done)

Getting Started

Create a separate virtual python environment:

$ conda create -n 3DPhotoCreator python=3.7 anaconda
$ conda activate 3DPhotoCreator

Clone the repo

$ git clone https://github.com/vt-vl-lab/3d-photo-inpainting.git
$ cd 3d-photo-inpainting

Download the models and other required large files:

# NOTE: Make sure you are in 3d-photo-inpainting folder 
$ chmod +x download.sh
$ ./download.sh

The deep learning engine for this library is PyTorch which must be installed in your machine. To install 3D Photo Creator deep learning engine requirement as well as all other packages please try as below:

$ pip install -r requirements.txt
$ conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit==10.1.243 -c pytorch

Note: If you don't have GPU then you can remove “cudatoolkit==10.1.243” from the list of install above.

Applying CPU or GPU:

If you have GPU, the default setting is the use GPU, don’t need anything.
If you have NO GPU and want to use CPU do the following by editing the argument.yml as below:

$ vi argument.yml
# Note: Either set gpu_ids to  -1 or leave it with no or empty value
gpu_ids: -1
# or
gpu_ids:

Understanding various settings:

Please read the argument.yml file for the settings which you can modify based on your environment and requirement.

Using the target image file:

You must be using a JPG file as input. If your photo shows a view that clearly shows the layering object view, then your result 3D photo could be very realistic. Also, the object you want to highlight must be in the center of your input photo.

Take you .JPG photo and drop (or copy) into the image folder

# Note: You must be into
$ cp your_source_jpg_image_with_path image/

Start 3D Photo creation process:

Once everything is set for you above now you can start the 3D photo generation as below:

$ python main.py --config argument.yml
# Note: Make sure you have applied CPU/GPU specific setting is applicable to your environment.

Depending on your CPU/GPU setting and hardware capabilities and configuration the process will take 30 seconds to 2 minutes with GPU and 4–6 minutes with CPU.

3D Photo Output:

After 3D photo creation is completed, the final results will be available as video (.MP4) files showing a 3D effect added to your input photo. The video output shows the following 4 effects:

Circle
Swing
Zoom-in
Dolly-Zoom-In

Here is the example of each output types:

Circle 3D Photo Effect

Swing 3D Photo Effect

Zoom-in 3D Photo Effect

Dolly Zoom In 3D Photo Effect

Output/Results:

The final rendered videos will be stored into the video folder as below:

video/[your_file_name]_zoom-in.mp4
video/[your_file_name]_swing.mp4
video/[your_file_name]_circle.mp4
video/[your_file_name]_dolly-zoom-in.mp4

Best Practice:

Make sure your photo has some visible depth already in it
The object you highlight into 3D view must be at the center of the input image

Useful Resources:

3D-Photo-Inpainting

Home: https://shihmengli.github.io/3D-Photo-Inpainting
GitHub: https://github.com/vt-vl-lab/3d-photo-inpainting
Google Colab: https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz
3D Photography using Context-aware Layered Depth Inpainting (PDF): https://drive.google.com/file/d/17ki_YAL1k5CaHHP3pIBFWvw-ztF4CCPP/view