Dub Videos into Another Language with AI: Whisper Transcription, LLM Translation and Kokoro Voiceover

A product demo, tutorial or brand video recorded in Chinese can reach buyers in other markets once it speaks their language. Professional dubbing is expensive and slow. This guide builds an automatic pipeline on your own server: faster-whisper transcribes the original speech with timestamps, a local language model translates it line by line, Kokoro speaks each translated line, and the script fits every line into its original time slot. You get a dubbed MP4, a translated subtitle file and an editable translation that you can correct before the final run.

Step 1: Install the Components

Set up /opt/tts as described in AI voiceover with Kokoro, including the extra packages for Japanese if you need that language. Translation runs on Ollama, see running a local LLM with Ollama. Then add speech recognition and the API client:

/opt/tts/bin/pip install faster-whisper "av<19" openai     # av<19: PyAV 19 is not yet compatible with faster-whisper 1.2
ollama pull qwen2.5:7b

Whisper understands about a hundred source languages; the target language must be one Kokoro can speak (see the voice table in the voiceover guide). For other target languages, translate the subtitles only.

Step 2: Create the Dubbing Script

cat > /opt/tts/dub.py <<'EOF2'
import os, subprocess, sys
import numpy as np
import soundfile as sf
from faster_whisper import WhisperModel
from kokoro import KPipeline
from openai import OpenAI

TARGET = os.environ.get("TARGET", "English")             # language name for the translator
LANG = os.environ.get("TTS_LANG", "a")                   # Kokoro language code of the target language
VOICE = os.environ.get("TTS_VOICE", "af_heart")
MAX_SPEED = float(os.environ.get("MAX_SPEED", "1.3"))    # never speak faster than this to fit a line in
ORIGINAL = os.environ.get("ORIGINAL_VOLUME", "0.15")     # keep the original audio quietly underneath (0 = remove)
RATE = 24000
client = OpenAI(base_url=os.environ.get("AI_BASE_URL", "http://127.0.0.1:11434/v1"),
                api_key=os.environ.get("AI_API_KEY", "ollama"))
MODEL = os.environ.get("AI_MODEL", "qwen2.5:7b")

def ts(t):
    ms = int(round(t * 1000))
    h, ms = divmod(ms, 3600000)
    m, ms = divmod(ms, 60000)
    s, ms = divmod(ms, 1000)
    return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"

def translate(text, previous):
    r = client.chat.completions.create(model=MODEL, temperature=0.2, messages=[
        {"role": "system", "content": f"Translate the line into natural spoken {TARGET} for a video voiceover. "
         "Keep it about as short as the original and keep product names unchanged. Return only the translation."},
        {"role": "user", "content": (f"Previous line, for context: {previous}\n" if previous else "") + f"Line: {text}"}])
    return r.choices[0].message.content.strip()

src = sys.argv[1]
base = os.path.splitext(src)[0] + "." + TARGET.lower()
cache = base + ".lines.txt"                              # translations, one per line: edit and run again
fixed = open(cache, encoding="utf-8").read().splitlines() if os.path.exists(cache) else []
whisper = WhisperModel(os.environ.get("WHISPER_MODEL", "small"), device="cpu", compute_type="int8")
segments, info = whisper.transcribe(src, vad_filter=True)
segs = [(s.start, s.end, s.text.strip()) for s in segments]
pipeline = KPipeline(lang_code=LANG, repo_id="hexgrad/Kokoro-82M")

def speak(text, speed):
    chunks = [np.asarray(a, dtype=np.float32) for _, _, a in pipeline(text, voice=VOICE, speed=speed)]
    return np.concatenate(chunks) if chunks else np.zeros(0, dtype=np.float32)

track = np.zeros(int((info.duration + 1) * RATE), dtype=np.float32)
srt, lines, cursor = [], [], 0.0
for i, (start, end, text) in enumerate(segs):
    line = fixed[i] if i < len(fixed) and fixed[i].strip() else translate(text, segs[i - 1][2] if i else "")
    lines.append(line)
    slot = (segs[i + 1][0] if i + 1 < len(segs) else info.duration) - start
    audio = speak(line, 1.0)
    if len(audio) / RATE > slot > 0:                       # too long for its place: speak a little faster
        audio = speak(line, min(MAX_SPEED, len(audio) / RATE / slot))
    at = max(start, cursor)                                # never overlap the previous line
    a = int(at * RATE)
    if a + len(audio) > len(track):
        track = np.concatenate([track, np.zeros(a + len(audio) - len(track), dtype=np.float32)])
    track[a:a + len(audio)] += audio
    cursor = at + len(audio) / RATE
    srt.append(f"{i + 1}\n{ts(at)} --> {ts(cursor)}\n{line}\n")
    print(f"{ts(start)} {text} -> {line}", flush=True)

sf.write(base + ".wav", track[:int(max(info.duration, cursor) * RATE)], RATE)
with open(cache, "w", encoding="utf-8") as f:
    f.write("\n".join(lines) + "\n")
with open(base + ".srt", "w", encoding="utf-8") as f:
    f.write("\n".join(srt))
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", src, "-i", base + ".wav", "-filter_complex",
                f"[0:a]volume={ORIGINAL}[o];[1:a][o]amix=inputs=2:duration=longest,loudnorm=I=-14:TP=-1.5[a]",
                "-map", "0:v", "-map", "[a]", "-c:v", "copy", "-c:a", "aac", "-b:a", "160k", "-ar", "48000", "-ac", "2",
                base + ".mp4"], check=True)
print("done:", base + ".mp4", base + ".srt")
EOF2

Each translated line starts at the time the original line started. If it is too long for its slot, it is spoken up to 30% faster; if it is still too long, the next line starts a little later, so lines never overlap. The original audio stays underneath at low volume, which keeps music and ambience.

Step 3: Dub a Video

cd /data/dub
TARGET=English TTS_LANG=a TTS_VOICE=af_heart /opt/tts/bin/python /opt/tts/dub.py promo-zh.mp4
TARGET=Japanese TTS_LANG=j TTS_VOICE=jf_alpha /opt/tts/bin/python /opt/tts/dub.py promo-zh.mp4

For each target language you get four files next to the video, for example promo-zh.english.mp4 (dubbed video), .srt (translated subtitles), .wav (voice track only) and .lines.txt (the translation, one line per original segment).

Step 4: Review and Correct the Translation

1. Watch the dubbed video together with promo-zh.english.lines.txt.
2. Correct wrong or too-long lines directly in that file, keeping one line per segment.
3. Run the same command again: the script uses your corrected lines instead of translating again, and rebuilds the voice, subtitles and video.

Useful settings: ORIGINAL_VOLUME=0 removes the original audio completely, MAX_SPEED=1.2 keeps speech calmer, and WHISPER_MODEL=medium recognises the original speech more accurately at the cost of time.

Step 5: Finishing Touches

# Burn the translated subtitles into the dubbed video
ffmpeg -i promo-zh.english.mp4 -vf "subtitles=promo-zh.english.srt" -c:a copy promo-zh.english-sub.mp4

• To remove the original voice but keep background music, separate the vocals first with a source-separation tool such as Demucs. It works on CPU but is slow.
• Translations are often longer than the original. Ask for shorter wording in the translation prompt, or shorten lines during review.
• Subtitles are created with automatic speech recognition, so names and model numbers in the original may be misheard; check them during review.

FAQ

Does it match lip movements?

No. This is voice-over style dubbing, which works well for product demos, screen recordings and narrated videos. Lip-sync needs other models, usually a GPU, and raises consent questions when real people appear on screen.

Can I use a cloud AI for the translation?

Yes. Set AI_BASE_URL, AI_API_KEY and AI_MODEL for any OpenAI-compatible API. The transcript then leaves your server, so use providers only in regions they support and within their terms.

How long does one video take?

Transcription, translation and voice generation all run on the CPU. Expect several minutes for a one-minute video with the small Whisper model and a 7B translation model; translation is usually the slowest part.

Still stuck after following these steps? Open a support ticket and the IMIDC 24/7 technical team will help. Include the server IP, OS and Python version, the command you ran and the full error output so we can pinpoint the issue faster.