A production-grade pipeline for generating photorealistic lip-synced face videos from arbitrary speech audio, combining a SyncNet sync expert with a U-Net generator trained via perceptual, adversarial, and audio-visual sync losses.
Lip synchronization replaces the mouth region of a talking-head video to match a target audio stream — enabling dubbing, accessibility tools, and avatar animation. This project builds a full end-to-end pipeline: raw video is preprocessed into aligned face crops and 80-channel Mel spectrograms, a SyncNet audio-visual discriminator is trained to detect temporal alignment, and a Wav2Lip U-Net generator is supervised by a three-stage loss — L1 reconstruction, VGG perceptual, adversarial, and SyncNet sync — to produce high-quality, temporally coherent output frames.
Raw videos are segmented into 3-second clips at 25 fps, then processed through 8 sequential stages: face detection with RetinaNet-ResNet50, 5-point affine alignment to 288×288, FRCRN neural denoising, and librosa Mel extraction.
Dual-stream encoder producing 512-dim L2-normalized embeddings from 5-frame visual crops (lower half only) and Mel windows. Trained with BCE loss on cosine similarity. Frozen during generator training.
A 9-level U-Net takes a 6-channel masked face and a 512-dim audio embedding, producing 288×288 RGB frames via skip-connected transposed convolutions. Optional SAM spatial attention gates skip features to focus on the lip region.
Wav2Lip generator encoder–decoder with precise tensor shapes at each stage.
Conv2d → BatchNorm2d → ReLU; residual adds input before activation.
Three-phase warm-up strategy to stabilise generator before introducing adversarial and sync supervision.
Generator trained with L1 pixel loss and VGG perceptual (LPIPS) loss only. Discriminator and SyncNet gradients are off. Establishes basic face fidelity.
Discriminator (Wav2LipDiscriminator, 9 nonorm blocks, lower-half input) enters training alongside the generator. Sharpens lip texture realism.
Frozen SyncNet scores each generated frame against the driving audio. BCE on cosine similarity penalises temporal misalignment, completing the full training objective.
8-stage distributed pipeline (Ray) transforming raw video into model-ready tensors.
| Parameter | SyncNet | Wav2Lip | Description |
|---|---|---|---|
max_steps | 2,000,000 | 10,000,000 | Total training steps |
batch_size | 128 | 6 | Samples per step |
learning_rate | 5e-5 | 5e-5 | AdamW base LR |
betas | (0.5, 0.999) | (0.5, 0.999) | Adam momentum terms |
scheduler | ReduceLROnPlateau | None | LR schedule |
disc_start | — | 15,000 | Step to enable discriminator |
sync_start | — | 60,000 | Step to enable sync loss |
disc_wt | — | 0.025 | Adversarial loss weight |
sync_wt | — | 0.03 | Sync loss weight |
face_size | 288 (H/2×W) | 288×288 | Input face resolution |
mel_channels | 80 | 80 | Mel spectrogram bins |
mel_step_size | 16 | 16 | Mel time steps per frame |
embed_dim | 512 | 512 | Audio / visual embedding dim |