Do Tri Nhan

Do Tri Nhan

AI Engineer in Speech and Signal Processing

Posts by Collection

portfolio

CUDA programing language

Published:

CUDA is architecture and programming model developed by NVIDIA to run parallel computing on graphics processing units (GPUs) CUDA is the acronym for Compute Unified Device Architecture

Vinorm

Published:

Python - NSW package for Vietnamese: Normalization system to convert numbers, abbreviations, and words that cannot be pronounced into syllables. I worked on this project at AILAB with an outstanding friend in APCS Program. Originally written in C++ then packaged into python for ease of use in research projects

Viphoneme

Published:

Pypi Package Viphoneme: Phonetization, Convert Vietnamese Grapheme to IPA. I did this project in my 3rd year of college, which converts raw text to phonemes of sound so that speech AI models can learn, this package is accessible and used by most projects Voice research project in Vietnam (not production)

Synthetic Speech Attribution - 2022 IEEE Signal Processing Cup

Published:

Fake synthetic speech audio tracks can be generated through a wide variety of available methods. Given an audio recording representing a synthetically generated speech track, to detect which method among a list of candidate ones has been used to synthesize the speech.

Voice Assessment

Published:

This project is customized to use the latest ASR model Whisper-large, with additional implementation of a React UI and several other signal processing tweaks.. .

post

publications

HCMUS at MediaEval 2020: Emotion Classification Using Wavenet Features with SpecAugment and EfficientNet

Published in MediaEval’20, 2020

MediaEval 2020 provided a subset of the MTG-Jamendo dataset, aimed to recognize mood and theme in music. Team HCMUS proposes several solutions to build efficient classifiers to solve this problem. In addition to the mel-spectrogram features, new features extracted from the wavenet model is extracted and utilized to train the EfficientNet model. As evaluated by the jury, our best result achieved of 0.142 in PR-AUC and 0.76 in the ROC-AUC measurement. With fast training and lightweight features, our proposed methods are potential to work well with deeper neural networks.

Author: Tri-Nhan Do, Minh-Tri Nguyen, Hai-Dang Nguyen, Minh-Triet Tran, Xuan-Nam Cao

Speedyspeech Model with Normalization for Faster Vietnamese Speech Synthesis

Published in VNUHCM-US_CONF_2020, 2020

End to end speech synthesis models have shown better results than traditional methods in terms of intelligence and spontaneity in recent years. However, these network models require a lot of data and training time, and the inference process consumes a lot of GPU resources. With the innovative experiments of SpeedySpeech, the speech synthesis time is shortened and the inference process has ability to run in real-time on the CPU. Authors apply this model for Vietnamese by standardizing the input, replace it with suitable Vietnamese phonemes, improve the embedding layer and the results showed that SpeedySpeech ‘s performance is asymptotic to the Tacotron2 model with significantly shorter training time. Training SpeedySpeech on Colab only takes 19 hours compared to 240 hours of training time on Tacotron2

Author: Do Tri Nhan, Nguyen Minh Tri, Cao Xuan Nam

Vietnamese Speech Synthesis with End-to-End Model and Text Normalization

Published in 7th NAFOSTED Conference on Information and Computer Science (NICS), 2020

Speech synthesis systems are now getting smarter and more natural thanks to the power of deep neural networks. However, each language has a different phonological and contextual characteristics, we have conducted experiments, statistics, and applied Vietnamese phonetics to improve speech synthesis systems based on Tacotron2 neural networks. Our methods achieve the accuracy of 97% in text normalization task, and the synthesized speeches with a MOS score of 3.97, asymptotic to 4.43 of the voices that are directly recorded. We also provide a library for standardizing Vietnamese text called Vinorm and a package that converts text into a phonetic format called Viphoneme, which is used as an input for end-to-end neural networks, make the synthesis process faster, more intelligent and natural than using character inputs.

Author: Do Tri Nhan, Nguyen Minh Tri, Cao Xuan Nam

DeepSpeechVC: Voice Cloning Framework with Speech Synthesis and Voice Conversion - Experiment with Speech2Speech techniques to make voice conversion from a small sample of the target speakers

Published in Thesis, 2021

With motivation of reconstructing voices for people who are mute after an accident or a person who has died and there is a small amount of data about their voices, this thesis aims to conduct experiments and apply new technologies, artificial neural networks to synthesize voices for Vietnamese. Different from speech synthesis models that require a onespeaker quality data set traditional voice cloning, we suggest a new voice clone sytem, which is able to synthesize any person’s voice with only a few sample audio input of that person’s voice.

Author: Tri-Nhan Do, Minh-Tri Nguyen under the advice of Prof. Vu Hai Quan, Msc. Xuan-Nam Cao

Vietnamese Speech-based Question Answering over Car Manuals

Published in Intelligent User Interfaces Conference, 2022

This paper presents a novel Vietnamese speech-based question answering system QA-CarManual that enables users to ask car-manual-related questions (e.g. how to properly operate devices and/or utilities within a car). Given a car manual written in Vietnamese as the main knowledge base, we develop QA-CarManual as a lightweight, real-time and interactive system that integrates state-of-the-art technologies in language and speech processing to (i) understand and interact with users via speech commands and (ii) automatically query the knowledge base and return answers in both forms of text and speech as well as visualization. To our best knowledge, QA-CarManual is the first Vietnamese question answering system that interacts with users via speech inputs and outputs. We perform a human evaluation to assess the quality of our QA-CarManual system and obtain promising results.

Author: Tin Duy Vo, Manh Tien Luong, Duong Minh Le, Hieu Minh Tran, Nhan Tri Do, Tuan-Duy Hien Nguyen, Hung Hai Bui, Dat Quoc Nguyen, Dinh Quoc Phung

FastSpeechStyle: Fast, Emotion Controllable and High-Quality Speech Synthesis

Published in Journal, 2023

The Non-autoregressive text to speech models such as Fastspeech2 can fast synthesize the high quality speech. This model also allows explicit control of the pitch, energy and speed of the speech signal. But, to control the emotion while maintaining the natural human-like speech is still a problems. In this paper, we propose an expressive speech synthesis model that can synthesize high-quality speech with desired emotion. The proposed model includes two main components (1) Mel Emotion Encoder extracts emotion embedding from the Mel-spectrogram of audio, (2) the FastSpeechStyle, a non-autoregressive model, which is modified from vanilla Fastspeech2. The FastSpeechStyle replaces normal LayerNorm with Style Adaptive LayerNorm to “shift” and “scale” hidden features according to emotion embedding, the model also used an improved Conformer block instead of vanilla FFTBlock to better model the local and global dependency in the acoustic model.

Author: Van Thinh Nguyen, Tri-Nhan Do, Hung-Cuong Pham, Tuan Vu Ho, Ngoc-Minh-Khanh Nguyen, Dang-Khoa Mac

Sound Event Detection with Soft Labels using Self-Attention Mechanisms for Global Scene Feature Extraction

Published in Detection and Classification of Acoustic Scenes and Events 2023, 2023

This paper presents our approach to Task 4b of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge, which focuses on Sound Event Detection with Soft Labels. Our proposed method builds upon a CRNN backbone model and leverages the benefits of data augmentation techniques to improve model robustness. Furthermore, we introduce self-attention mechanisms to capture global context information and enhance the models ability to predict soft label segments more accurately. Our experiments demonstrate that incorporating soft labels and self-attention mechanisms result in significant performance gains compared to traditional methods on data varying across different scenarios.

Author: Nhan Tri-Do, Param Biyani, Zhang Yuxuan, Andrew Koh Jin Jie, Chng Eng Siong

Enhancing Deepfake Detection: A Study Using WavLM and Advanced RawBoost Augmentation Techniques

Published in Paper, 2024

Automatic Speaker Verification (ASV) systems are increas- ingly vulnerable to sophisticated spoofing attacks, particularly those involving deepfake audio. This paper presents our ap- proach to the challenges in Task 1 of the ASVSpoof 2024 com- petition, focusing on deepfake detection to classify utterances is spoof or bonafide. To enhance model robustness, we employed self-supervised learning (SSL) models, specifically fine-tuning WavLM for feature extraction due to its superior performance in noisy environments. We utilized RawBoost augmentation techniques to simulate real-world audio distortions. Experimen- tal results demonstrate that our approach significantly improves detection accuracy, achieving an EER of 2.85% with WavLM and further reducing to 2.69% with a fusion of WavLM and Wav2Vec2 models.

Author: Nhan Tri Do, Loi Nguyen Hoang, Phuong Ta Viet, Kien Phan Trung

Voice Attacker Leveraging Multi-Head Factorized Attentive Reconstructor and Gradient Reversal for Random Prosody Anonymization

Published in Paper, 2025

This is the report for Team 04-SpeechWorld in the First VoicePrivacy Attacker Challenge. The attack methods aimed to verify speakers anonymized by two main anonymization systems: STTTS-based and NAC-based. The characteristics of the original audio were reconstructed using speaker embeddings from WavLM-Ecapa and codecs for the NAC system. Additionally, gradient reversal layers were incorporated to eliminate dependencies on prosody features that were randomly simulated by the anonymization models. The results show that the proposed attackers achieved a relative improvement of 26.49% in Equal Error Rate (EER) compared to the baseline, reducing it from 43.22% to 31.77% for the T12-5 attacker system.

Author: Nhan Tri Do

Unified Acoustic Representation Learning for Vietnamese Speech Classification Tasks

Published in MAPR 2025, 2025

Analyzing multiple attributes of Vietnamese speech concurrently, such as gender, dialect, and emotion, is an important yet challenging task. Traditional methods often isolate each task, potentially overlooking correlations and leading to suboptimal performance. This paper proposes a unified architecture that integrates features derived from two different representations of the same input audio signal: raw waveforms and Mel spectrograms. The architecture employs a fixed, pre-trained Wav2Vec 2.0 encoder processing the waveform and a trainable Vision Transformer (ViT) encoder processing the spectrogram. These representations are fused using Feature-wise Linear Modulation (FiLM). This single multi-task model is designed to concurrently classify gender, dialect, and emotion. We detail a practical training strategy for a multi-dataset scenario using probabilistic sampling and a masked loss function, alongside a tailored hybrid evaluation protocol (fixed test set for ViMD, 5-fold cross-validation for VNEMOS). The proposed approach achieves competitive Macro F1-scores: 98.74% (gender), 91.81% (dialect), and 95.53% (emotion, 5-fold average). This research demonstrates the effectiveness of fusing different acoustic representations within a unified multi-task model for complex Vietnamese speech analysis.

Author: Xuan-Truong Ha, Van-Huy Nguyen, Tri-Nhan Do, Trung-Kien Phan, Dang-Khoa Mac

Analyzing the Correlation and Impact of Speech Evaluation Metrics on Real-World Speaker Verification and Speech Recognition

Published in MAPR 2025, 2025

This study investigates the relationship between speech quality assessment metrics and the performance of downstream tasks such as Automatic Speech Recognition (ASR) and Speaker Verification (SV). We evaluate non-intrusive metrics and estimated versions of traditional intrusive metrics across multiple languages and noise conditions, focusing on their correlation with real-world task performance where clean reference audio is typically unavailable. Our experiments span Vietnamese and English datasets with varying noise types and levels. Results indicate that while estimated intelligibility metrics like STOI show strong correlation with ASR performance, the effectiveness of various non-intrusive metrics varies across languages and noise conditions. We explore enhanced prediction models combining multiple metrics to better estimate downstream performance. This work provides insights for optimizing speech processing systems in real-world applications.

Author: Tan-Loc Le, Van-Huy Nguyen, Tri-Nhan Do, Trung-Kien Phan, Dang-Khoa Mac

Adapting WavLM for Vietnamese Speaker Diarization in Real-world Conversations

Published in MAPR 2025, 2025

While end-to-end neural diarization (EEND) models have achieved state-of-the-art performance, their effectiveness in low-resource languages such as Vietnamese remains underexplored due to the lack of annotated conversational data. In this study, we adapt WavLM-based speaker diarization to Vietnamese by fine-tuning the DiariZen model with Vietnamese speech data. Additionally, we introduce ViYT-Diar, a high-quality benchmark of manually annotated Vietnamese dialogues. Experimental results show that our fine-tuned model achieves a DER of 11.76% on the English CALLHOME two-speaker test set and 2.38% on ViYT-Diar, significantly outperforming Pyannote 3.1 and Falcon API. Furthermore, our custom clustering pipeline maintains stable performance across chunk sizes (2.38% - 2.59% DER), whereas off-the-shelf models degrade from 9.73% to 3.75%. These results underscore the effectiveness of language-specific fine-tuning and tailored clustering for low-resource speaker diarization.

Author: Tuan-Duy Thang, Van-Huy Nguyen, Tri-Nhan Do, Quoc-Khanh Nguyen, Trung-Kien Phan, Dang-Khoa Mac

Language Identification with Additive Attention BiLSTM and LaBSE Embeddings

Accept in FDSE 2025, 2025

Language Identification (LangID) plays a critical role in multilingual NLP pipelines, particularly for large-scale web crawling, dataset filtering, and pre-processing in tasks such as machine translation and cross-lingual retrieval. With the growing volume of web-scale multilingual content, robust and scalable LangID systems are essential for curating clean and language-specific data. In this paper, we propose a hybrid neural architecture that combines sentence-level semantic representations from LaBSE embeddings with token-level sequential features modeled by a BiLSTM with additive attention. This design captures both global context and local subword cues, enabling accurate language classification even in short or noisy texts. Evaluated on the WiLI-2018 dataset spanning 235 languages, our model achieves 88.37% accuracy—outperforming strong baselines like FastText and XLM-RoBERTa. The proposed approach offers a compelling balance between performance, scalability, and deployability for real-world multilingual data pipelines.

Author: Tri-Nhan Do

ViInstruct: A Large-Scale Vietnamese Dataset for Instruction-Based Text-to-Speech Synthesis

Published in APSIPA ASC 2026, 2026

Instruction-based text-to-speech (TTS) enables natural-language control of speaking style without reference audio, but no comparable resource exists despite the language’s tonal prosody and substantial regional variation. This paper introduces ViInstruct, the first large-scale Vietnamese dataset for instruction-based TTS, comprising approximately 1,718 hours of speech and 1,224,945 utterance-instruction pairs collected from six corpora spanning general speech, regional accent, and emotion settings. We develop an automated pipeline that extracts acoustic features, preserves or infers speaker attributes, maps them to Vietnamese descriptors, and generates one natural-language instruction for each utterance with a large language model. We also provide a standardized benchmark, comprising two evaluation subsets and two baseline systems, to support reproducible evaluation of controllable Vietnamese TTS in a low-resource tonal language.

Author: Thang Ly, Tri-Nhan Do, Tu-Anh Nguyen, Dang-Khoa Mac

ViNonverbal: The First Vietnamese Nonverbal Speech Dataset via Scalable Pseudo-Labeling

Published in APSIPA ASC 2026, 2026

Nonverbal vocalizations, such as laughter and fillers, are essential for expressive speech recognition and synthesis, yet they are largely neglected in Vietnamese ASR and TTS systems. We introduce ViNonverbal, a Vietnamese nonverbal speech resource consisting of a manually annotated 500-sample benchmark and an additional 4,693 pseudo-labeled speech segments covering eight nonverbal event types. We fine-tune Qwen3-ASR and Whisper-large-v3 using extended nonverbal tokens to bridge this gap. Experimental results demonstrate that Qwen3-ASR achieves a superior F1-score of 57.21% for nonverbal event recognition, significantly outperforming the Whisper baseline. The model maintains high transcription stability with a WER of 6.16% on VIVOS and 13.61% on the ViNonverbal test set. These findings provide a robust foundation for developing more natural, human-like Vietnamese ASR and TTS systems that can understand and generate expressive nuances.

Author: Minh-Ngoc Nguyen, Tu-Anh Nguyen, Tri-Nhan Do, Hoang-Yen Nguyen, Dang-Khoa Mac

Vi-SparkRL: Enhancing Tonal Accuracy and Naturalness in Vietnamese TTS via Multi-Objective Rewards

Published in APSIPA ASC 2026, 2026

Recent advances in Large Language Models and neural audio codecs have significantly improved zero-shot Text-to-Speech (TTS) systems; however, maintaining tonal accuracy in languages such as Vietnamese remains a formidable challenge. Existing models often exhibit unstable prosody or phonetic biases that result in a noticeable foreign accent. In this paper, we propose Vi-SparkRL, a Reinforcement Learning enhanced framework designed to preserve Vietnamese tonal integrity in lightweight zero-shot Text-to-Speech models. Our approach introduces Group Sequence Policy Optimization, which performs sequence-level optimization to mitigate training instability and mode-collapse issues common in token-level reinforcement learning for continuous audio signals. Furthermore, we design a comprehensive multi-objective reward mechanism that integrates a specialized VietTone Reward for positional tonal matching, alongside constraints for intelligibility, speaker similarity, and acoustic quality. Evaluated on the PhoAudiobook dataset, Vi-SparkRL achieves a Word Error Rate of 2.85% and a tonal accuracy of 0.96, significantly outperforming traditional Supervised Fine-Tuning and competitive baselines. Our results demonstrate that sequence-level RL effectively aligns TTS models with the complex prosodic requirements of tonal languages while maintaining high naturalness.

Author: Xuan-Binh Dinh-Thi, Tri-Nhan Do, Tu-Anh Nguyen, Van-An Chu, Dang-Khoa Mac

talks

teaching