Counting 3,742 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1


This is a pix2pix demo that learns from facial landmarks and translates this into a face. A webcam-enabled application is also provided that translates your face to the trained face in real-time.

Getting Started

1. Prepare Environment

# Clone this repo
git clone

# Create the conda environment from file (Mac OSX)
conda env create -f environment.yml

2. Generate Training Data

python --file angela_merkel_speech.mp4 --num 400 --landmark-model shape_predictor_68_face_landmarks.dat


  • file is the name of the video file from which you want to create the data set.
  • num is the number of train data to be created.
  • landmark-model is the facial landmark model that is used to detect the landmarks. A pre-trained facial landmark model is provided here.


  • Two folders original and landmarks will be created.

If you want to download my dataset, here is also the video file that I used and the generated training dataset (400 images already split into training and validation).

3. Train Model

# Clone the repo from Christopher Hesse's pix2pix TensorFlow implementation
git clone

# Move the original and landmarks folder into the pix2pix-tensorflow folder
mv face2face-demo/landmarks face2face-demo/original pix2pix-tensorflow/photos

# Go into the pix2pix-tensorflow folder
cd pix2pix-tensorflow/

# Resize original images
python tools/ \
  --input_dir photos/original \
  --operation resize \
  --output_dir photos/original_resized
# Resize landmark images
python tools/ \
  --input_dir photos/landmarks \
  --operation resize \
  --output_dir photos/landmarks_resized
# Combine both resized original and landmark images
python tools/ \
  --input_dir photos/landmarks_resized \
  --b_dir photos/original_resized \
  --operation combine \
  --output_dir photos/combined
# Split into train/val set
python tools/ \
  --dir photos/combined
# Train the model on the data
python \
  --mode train \
  --output_dir face2face-model \
  --max_epochs 200 \
  --input_dir photos/combined/train \
  --which_direction AtoB

For more information around training, have a look at Christopher Hesse's pix2pix-tensorflow implementation.

4. Export Model

  1. First, we need to reduce the trained model so that we can use an image tensor as input:

    python --model-input face2face-model --model-output face2face-reduced-model


    • model-input is the model folder to be imported.
    • model-output is the model (reduced) folder to be exported.


    • It returns a reduced model with less weights file size than the original model.
  2. Second, we freeze the reduced model to a single file.

    python --model-folder face2face-reduced-model


    • model-folder is the model folder of the reduced model.


    • It returns a frozen model file frozen_model.pb in the model folder.

I have uploaded a pre-trained frozen model here. This model is trained on 400 images with epoch 200.

5. Run Demo

python --source 0 --show 0 --landmark-model shape_predictor_68_face_landmarks.dat --tf-model face2face-reduced-model/frozen_model.pb


  • source is the device index of the camera (default=0).
  • show is an option to either display the normal input (0) or the facial landmark (1) alongside the generated image (default=0).
  • landmark-model is the facial landmark model that is used to detect the landmarks.
  • tf-model is the frozen model file.





Kudos to Christopher Hesse for his amazing pix2pix TensorFlow implementation and Gene Kogan for his inspirational workshop.


See LICENSE for details. Copyright (c) 2017 Dat Tran.