
Hey 👋

I wanted to share WhisperSub, a command-line tool that generates, transcribes, and merges subtitles for video files.
I wanted a way to watch anime with both English translations and Japanese transcriptions, and built the tool to automate that workflow. This post focuses on the design choices and technical details of the journey.

## Scope and approach

WhisperSub is designed to be flexible and language-agnostic. I evaluated several Whisper implementations (standard whisper, faster-whisper, whisperX) and chose stable-ts for its accuracy and word-level timestamps. A major part of the work was broadening language support and handling romanization for different scripts:

- Japanese (Hepburn)
- Chinese (Hanyu Pinyin)
- Korean (Revised Romanization)
- Russian (GOST)
- Arabic (ISO 233)
- and others

Each language required specific handling and tests to ensure reasonable results across different inputs.

## Technical highlights

The main components are:

1. Audio extraction from video (ffmpeg)
2. Speech recognition using stable-ts for improved accuracy and word-level timestamps
3. Optional speech isolation with Demucs
4. Extraction of existing subtitles when present
5. Merging and exporting a consolidated subtitle file

Other features include adjustable sync tolerance and flexible I/O options. Optional GPU acceleration is available for faster transcription runs, but enabling and validating CUDA-based performance was not trivial: it requires compatible PyTorch/CUDA/ctranslate2 builds, careful dependency testing, and extra validation on different hardware/configurations.

## Subtitle synchronization challenges

Subtitle synchronization turned out to be one of the most intricate parts of the project. Ensuring that subtitles align perfectly with audio while maintaining readability required a combination of precise timing adjustments, format compatibility, and layering logic.

- **Timing adjustments**: WhisperSub uses a sync tolerance parameter to snap subtitles to the nearest matching audio event. This helps reduce drift but requires careful tuning to avoid introducing artifacts.
- **Layering and overlap handling**: The tool employs a smart layering algorithm to prevent overlapping subtitles. This is particularly important when merging multiple tracks or adding romanized text alongside translations.
- **Format-specific quirks**: The ASS format allows for advanced styling and positioning, which WhisperSub leverages to maintain visual clarity. However, converting to simpler formats like SRT can strip these features, making it impossible to preserve most styling options.
- **Iterative testing**: The logic for merging and aligning subtitles was refined through extensive testing with real-world media, encountering multiple edge cases (e.g., overlapping events, multi-line subtitles) that required custom handling.

## Lessons learned

The bulk of the engineering effort was about choosing and integrating the pieces that produce consistent results across noisy, real-world media. The two areas that taught me the most were the different Whisper implementations and audio separation tools:

- Whisper implementations and trade-offs
  - stable-ts: good accuracy and useful word-level timestamps, which simplifies subtitle alignment. It can be heavier on resources for larger models.
  - faster-whisper: significantly faster inference on CPU or smaller GPUs, but with trade-offs in some edge-case accuracy and stability across languages.
  - whisperX / standard whisper: robust and well-understood, but may require additional tooling for fine-grained timestamps and alignment.
  - Practical trade-offs: model size versus latency, GPU memory limits, and whether word-level timestamps are available dictate the best choice for a given workflow. Automatic language detection is convenient but not always perfect; explicit language selection can improve results.

- Audio separation (Demucs vs Spleeter and friends)
  - Demucs: generally produces higher-quality voice isolation and helps transcription accuracy in noisy scenes, but it is computationally intensive and slower, especially without GPU acceleration.
  - Spleeter: faster and lighter-weight, useful for quick experiments, but its separation quality is usually lower than Demucs.
  - Separation helps with clarity and downstream transcription quality, but it can introduce artifacts and increases processing time and complexity. I exposed separation as an optional step so users can trade quality for performance.

- Subtitle synchronization
  - Syncing subtitles is a non-trivial task, especially when dealing with overlapping events or multi-language tracks. The iterative process of refining the sync logic taught me the importance of balancing automation with user control.
  - Tools like `pysubs2` simplify subtitle manipulation but require careful handling of edge cases, such as multi-line subtitles or events with inconsistent timing.
  - Testing with diverse media files was crucial to uncover and address edge cases, ensuring that the tool performs reliably across different scenarios.

Overall, the project is about balancing accuracy, performance, and complexity. The implementation choices in WhisperSub aim to give users control over those trade-offs.

The project is open-source on [GitHub](https://github.com/ahernandezmiro/whispersub). If you try it and have suggestions or find issues, please open an issue or get in touch.

— Aleix


💬 WhisperSub: Building a Multilingual Subtitle Tool


Hey there! 👋

Welcome to the first post on my portfolio. I'm Aleix — a software engineer currently working as a Solutions Engineer in Oracle NetSuite's SDN team.\
I'm a passionate engineer with a long background in web development and backend scalability, integrating with third party tools and services, providing consultancy and technical support to partners and clients.

This blog is where I plan to share my thoughts, projects, and the occasional rant about the joys and pitfalls of the development world.

Like many portfolios, I started with a simple goal: create a minimalistic site to showcase my work and skills.
Initially, it was a straightforward SSR site:

- Next.js – for its flexibility and Vercel's free tier deployment
- React – to build interactive UI components
- Tailwind CSS – for rapid and consistent styling

I wanted to make the most of the project by learning as much as I could in the process, particularly on the frontend side.
For that reason, I decided not to use external components or libraries besides the basics provided by the framework.
I knew this limitation would impact the development speed, but I thought it’d be cool to do everything from scratch
(SPOILER: it did, *very much*, prolong the timeline of the project).

As I kept going, I couldn't resist the urge to add more features.
A simple animation on the landing page evolved into a full typewriting machine component, and a basic list of technical skills turned into a whole galactic SVG graph rendering experience.
Eventually, I ended up writing a simple MDX-based blog engine — the thing that’s making it possible for you to read this right now.

Each addition was a learning opportunity, pushing me to explore new facets of web development.
And since I had no actual pressure to publish or deploy anything, I just kept tweaking things, changing designs, and refactoring code.
I've also had other priorities in my free time, which not only include living my life, but starting and working on parallel projects without finishing others first too (I tend to do this more than I'd like to admit).

So yeah — building everything from scratch was definitely not the most efficient route, especially when there are plenty of templates and tools out there.
I’ve spent more time than I probably should have on this project.
But this approach gave me full control over the site's functionality and design.
It was a valuable exercise in problem-solving, and it really deepened my understanding of the stack and technologies I was working with.

I hope you like the site and have fun playing around.
And if everything goes well, I expect to present a new project shortly.

Stay tuned for more posts, and feel free to reach out if you have questions or suggestions!

Thanks for stopping by! \
— Aleix

Blog

💬 WhisperSub: Building a Multilingual Subtitle Tool

Hello world