AI Lip Sync: How to Make Talking Photos and Multilingual Video Ads

What AI lip sync changes in a talking photo or video

AI lip sync generates or modifies mouth and lower-face movement to match a target speech track. In a talking photo, the system starts with a still portrait and creates a speaking-video performance around the supplied voice or script. In an existing video, it changes the visible performance so that a new or replacement audio track appears to come from the person on screen.

The narrow job is audio-visual alignment, not the creation of a fully verified message. The research behind Wav2Lip, for example, describes synchronizing an arbitrary face video with arbitrary speech and uses a discriminator intended to penalize mismatch between the sound and image. A convincing result therefore depends on two linked materials: an audio performance worth using and a face asset whose framing gives the system enough clear information to animate.

For a release team, that distinction matters. A lip-synced clip can have plausible mouth movement while still carrying an unapproved script, an awkward translated line, an unsuitable voice, or graphics that remain in the original language. Treat lip sync as one stage in a production and localization workflow, rather than as the final proof that an ad is ready to publish.

The production pipeline

Most workflows bring together a face image or video, a spoken audio track or text that will become speech, face detection, alignment between audio and visible speech, frame synthesis, and final compositing or facial restoration. The released Wav2Lip implementation, for instance, takes video and target speech as inputs and requires a face-detection model and FFmpeg.

The practical fork comes early. If the team has approved source footage and a separately recorded voice track, the task is to synchronize that audio to the person in the footage. If it has only a portrait and approved copy, the task is a talking-photo generation workflow: select the image, provide the text, choose a voice, then generate a new avatar video.

A useful brief identifies the workflow before it asks anyone to make a spokesperson speak. State whether the source is a still image or motion footage, whether the speech arrives as finished audio or is generated from text, which spoken language is required, and which script version is the rendering source; “Make this spokesperson speak” alone is not enough direction for a vendor or internal operator.

• Face asset: the selected portrait or the exact video edit to process.
• Speech source: approved recorded audio, or locked text plus the selected voice for text-to-speech.
• Alignment stage: the system matches visible lower-face motion to the selected speech.
• Output review: the team checks the rendered video against the approved language, performance, and release version.

Version control is particularly useful when the spoken copy changes late. A changed script can alter the duration and cadence of the voice track, which means the prior render is no longer the asset being approved. Name the portrait or footage source, script version, language, voice choice, and output file together so reviewers can identify what they are watching.

Talking-photo ads: build the voice and portrait as one performance

A talking photo is best planned as a compact performance, not as a static key visual with speech added at the end. The portrait supplies the visible identity and framing; the script and voice supply the pacing, emphasis, pronunciation, and duration that the generated face must carry. Short, final copy gives the team a clearer object to approve before it commits to a render.

D-ID’s Photo Avatars API quickstart illustrates this model: it accepts an image URL and text script with a selected voice provider, then returns a speaking-avatar video asynchronously. “Asynchronously” has a production consequence: create the job, retain its identifier or status in the production record, and plan the review after the result returns rather than treating generation as an instant edit.

A workable sequence for a promotional asset is:

1. Choose a portrait in which the face is clear and the intended speaker is unmistakable.
2. Lock the promotional script, including product names, offers, calls to action, and any required wording.
3. Select the voice and generate a test so the team can judge how the copy is performed.
4. Submit the image and approved script or speech source for generation.
5. Review the returned video with the script and master creative beside it; revise the source inputs rather than approving around a noticeable problem.
6. Place the approved avatar clip into the final ad edit, then review the complete deliverable again.

The final edit is a separate review point because end cards, supers, captions, and cuts can change the meaning or timing of the asset. The generated speaking section may be acceptable on its own and still fail to fit cleanly with the surrounding promotion.

Multilingual ad localization needs more than new lip movements

For a multilingual video, the working order is translate the spoken material, review and edit that translated script, create the replacement voice, apply lip sync where it suits the footage, and then handle visible language in the picture separately. Lip changes can help a dubbed speaker appear aligned with the replacement audio, but they do not translate a campaign by themselves.

This separation is explicit in Synthesia’s guidance on video dubbing: text baked into the original video, including titles, graphics, and burned-in captions, is not automatically translated. Build an inventory of every visible text element before rendering. That includes the opening title, product claims, lower-thirds, offer cards, end screens, subtitles already embedded in the source, and any text that appears only briefly between cuts.

The spoken script also deserves review before a final render. HeyGen notes that AI translation cannot guarantee 100% accuracy and offers proofread or review-and-edit workflows for correcting the translated script before rendering. Use that checkpoint for terminology, names, units, tone, and the intended action the viewer should take—not merely for obvious spelling errors.

A localized release package should therefore have distinct owners or explicit sign-off for spoken copy and on-screen copy. Otherwise, a team can approve a naturally moving face while missing an untranslated end card or a script change that has not been reflected in the edit.

Footage choices that determine whether lip sync will hold up

Source selection is an art-direction decision as much as a technical one. HeyGen recommends front-facing speakers, limited facial occlusion, one active speaker at a time, close-up framing, minimal camera cuts, and low background noise for video translation. Those conditions make the visible speaker and speech easier to isolate through the workflow.

A compelling master edit is not, by itself, a good candidate for lip-synced localization. Treat side profiles, hands or microphones covering the face, speaker switches, and complex conversations as early reject cases; rapid-cut conversations may call for a different editorial approach, or for segments to be evaluated individually rather than submitted as one continuous speaker performance.

For new shoots, preserve a clean, close speaker shot when the release plan may include later localization. For existing footage, identify the segments with one visible, unobstructed speaker and test those first. This is more useful than judging the whole video by a single favorable frame: the face must remain readable through the moments that will carry the replacement speech.

Approve language, performance, and visual sync separately

A usable review has at least three passes. First, compare the spoken language against the approved source and translated script. Second, listen for the performance the selected voice gives the copy: names, pacing, emphasis, and the call to action should be judged as spoken communication. Third, watch the rendered face with the sound on for audio-visual alignment and visible artifacts.

Keep the checks independent. Strong-looking mouth movement does not establish translation accuracy, and accurate wording does not establish visual realism. The distinction follows both the technical framing of lip sync as audio-visual matching and the translation-review warning in HeyGen’s guidance.

• Language check: Is the approved message present, and has a reviewer corrected the translated script where needed?
• Performance check: Does the selected voice deliver the intended names, emphasis, and promotional instruction?
• Visual check: Does the visible speaker remain clear, and do the mouth and lower-face movements track the audio through the full shot?
• Picture-text check: Have baked-in captions, graphics, and other on-screen text been replaced or localized separately?

Review the release file, not only a work-in-progress render. The relevant question is whether the exact language, voice, visual output, and visible text that viewers will receive have all passed their respective checks.

Frequently Asked Questions

Is one photo enough to make a talking avatar?

It can be. Talking-photo systems can accept a portrait and text script, then generate a speaking-avatar video using a selected voice. The result should still be reviewed as a generated performance, especially when the clip will carry promotional copy.

Does lip sync translate on-screen text?

No. Lip sync addresses the relationship between speech and the face. Titles, graphics, and burned-in captions in the original video need their own localization work.

What footage works best for multilingual lip sync?

Use a close, front-facing shot with one active speaker, little facial obstruction, few cuts, and low background noise where possible. Profiles, covered faces, speaker switches, and complex conversations are more difficult cases.

Do accurate lips guarantee an accurate translation?

They do not. A clip may be well synchronized visually while the translated wording still needs correction. Review and edit the translated script before the final render.

Can a video with multiple speakers be reliably localized?

It is a harder input than a single, clearly framed speaker. Evaluate speaker segments individually, because switches and complex conversations are identified as challenging conditions for this type of workflow.