Creating a 3D photo from a regular image using the 3D-Photo-Inpainting library
Source & Credits:
Description (How it works):
- A color single RGB-D input image is used as input and the 3D photo is generated in MP4 format
- 3D photo generation is based on the multi-layer representation where these layers contain hallucinated color and depth structures in regions occluded in the original view.
- The library uses Layered Depth Image (as input) with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that iteratively synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner.
- The final 3D photo result image can be efficiently rendered with motion parallax using standard graphics engines.
- python 3.7
- Conda or virtualenv — A python runtime environment
- GPU is good to have for speed however if you don't have CPU is still supported
- Tested OK, on both CentOS Linux and OSX, (No Windows Test done)
Create a separate virtual python environment:
$ conda create -n 3DPhotoCreator python=3.7 anaconda
$ conda activate 3DPhotoCreator
Clone the repo
Download the models and other required large files:
# NOTE: Make sure you are in 3d-photo-inpainting folder
$ chmod +x download.sh
The deep learning engine for this library is PyTorch which must be installed in your machine. To install 3D Photo Creator deep learning engine requirement as well as all other packages please try as below:
$ pip install -r requirements.txt
$ conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit==10.1.243 -c pytorch
Note: If you don't have GPU then you can remove “cudatoolkit==10.1.243” from the list of install above.
Applying CPU or GPU:
- If you have GPU, the default setting is the use GPU, don’t need anything.
- If you have NO GPU and want to use CPU do the following by editing the argument.yml as below:
$ vi argument.yml
# Note: Either set gpu_ids to -1 or leave it with no or empty value
Understanding various settings:
Please read the argument.yml file for the settings which you can modify based on your environment and requirement.
Using the target image file:
You must be using a JPG file as input. If your photo shows a view that clearly shows the layering object view, then your result 3D photo could be very realistic. Also, the object you want to highlight must be in the center of your input photo.
- Take you .JPG photo and drop (or copy) into the image folder
# Note: You must be into
$ cp your_source_jpg_image_with_path image/
Start 3D Photo creation process:
Once everything is set for you above now you can start the 3D photo generation as below:
$ python main.py --config argument.yml
# Note: Make sure you have applied CPU/GPU specific setting is applicable to your environment.
Depending on your CPU/GPU setting and hardware capabilities and configuration the process will take 30 seconds to 2 minutes with GPU and 4–6 minutes with CPU.
3D Photo Output:
After 3D photo creation is completed, the final results will be available as video (.MP4) files showing a 3D effect added to your input photo. The video output shows the following 4 effects:
Here is the example of each output types:
Circle 3D Photo Effect
Swing 3D Photo Effect
Zoom-in 3D Photo Effect
Dolly Zoom In 3D Photo Effect
The final rendered videos will be stored into the video folder as below:
- Make sure your photo has some visible depth already in it
- The object you highlight into 3D view must be at the center of the input image
- Home: https://shihmengli.github.io/3D-Photo-Inpainting
- GitHub: https://github.com/vt-vl-lab/3d-photo-inpainting
- Google Colab: https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz
- 3D Photography using Context-aware Layered Depth Inpainting (PDF): https://drive.google.com/file/d/17ki_YAL1k5CaHHP3pIBFWvw-ztF4CCPP/view
Motivations (GitHub Projects)
Thank you so much for reading!!