Center for Voice Intelligence and Security

Deducing faces from voices

Sounds produced in our vocal tract resonate in our vocal chambers. They are modulated by our articulators (tongue, lips, jaw etc.) and further influenced by the structures within the vocal tract. The dimensions of our vocal chambers are highly correlated with our skull structure, which in turn defines our facial appearance to a very large extent. Many physical properties and dynamics of our vocal structures also correlate well with our age, ethnicity, height, gender and so on -- which in turn influence our appearance. Thus, it is easy to see how the signatures of many physical factors that naturally influence our voice can also directly or indirectly inform about our facial structure.
Technologies that infer facial appearance and structure from voice leverage this web of information embedded in the voice signal. AI systems can also be designed to isolate this information.

Biomarker discovery

Biomarker discovery techniques identify or design/construct specific mathematical representations of voice signals that bear a strong correlation with a particular influence hypothesized to affect voice. As an example, consider a scenario where we theorize that the consumption of a specific medication must impact the voice. Yet in practical life this does not seem to be so -- these alterations, if present, remain undetectable to us, evading even the most direct voice signal analyses. In such instances, a biomarker discovery system could reveal the precise, distinctive signature indicative of the medication's influence on a given voice signal. This distinguishing signature might lie in a complex, high-dimensional mathematical space -- an imperceptible virtual representation that is only useful for computational purposes. Nevertheless, the ability to now extract the identified signature from new voice samples allows machines to learn and detect, merely from the speaker's voice, the recent consumption of that specific medication.

We began work on biomarker discovery techniques in 2016, well before the control of latent space representations using neural architectures became mainstream. Today we continue to expand the set of entities for which we have successfully created AI-based discovery pipelines.

Deducing vocal fold oscillations

Our vocal folds oscillate in a self-sustained manner during phonation (ie, when we produce voiced sounds like a sustained "aa"). There is a wealth of information in the fine-level details of how they oscillate. Understanding these details can reveal an astonishing amount of information aboout the state of the speaker. However, historically it was hard to measure the vocal fold oscillations of each person as they spoke. Doing so required specialized instruments, to be used in clinical settings. At CVIS, we have been developing techniques to deduce the vocal fold oscillations of speakers from voiced sounds directly from recorded speech signals. This opens doors to analyzing vocal fold osciallations on an individual basis, and studying ther changes in their patterns in response to various influencing factors -- from substances to mental problems to infectious diseases like Covid-19.

Based on our techniques, we built a live Covid-19 detection system in February 2020, and put together a protocol for analyzing a set of sustained vowel sounds and a couple of countinuous speech examples. This protocol is now globally used as a basis for sound analysis for Covid-19 and other conditions. We began this work in 2017 with the interesting goal of detecting voice disguise in-vacuo (without knowing what the original voice sounds like), and have made considerable progress in refining our techniques and exploring practical applications for it.

Deducing emotional, physiological, psychological and behavioral states and traits from voice

Vocal fold oscillations are not the sole cues that offer insights into the physiological alterations within our body. Subtleties embedded within our vocal production and control (vocal expression) are also linked with our emotional, behavioral, and psychological characteristics. The deduction of such "states" from vocal nuances has been an area of considerable research, spurred by centuries of observation and correlation between speech patterns and these characteristics.

Our work is two-faceted. In one, we join hands with researchers across the globe, contributing to mainstream techniques for deducing target states from voice. In the other, we move beyond these, aiming to discover the underlying traits: correlations that not only apply universally across populations but also relate specifically to an individual and their current state, and correlations with our genetic makeup. Our research aims to develop technologies to analyze diverse human states and traits, thereby enhancing our comprehension of the intricate relationship between vocal features and human characteristics. As an example, we deduce psychological traits from an aggregations of emotional states of a person -- the emotional spectrum -- and objective measurements of various voice qualities. These are also supported by measurements of low-level features.

Understanding personality bases

Understanding human personality plays a crucial role across numerous fields such as psychology, behavioral science, and many others. In order to understand personality, traditionally it is broken down into a handful of dimensions (or bases) like the Big-5 OCEAN traits. The way these bases have been agreed upon has been rigorous but has not been revisited for decades. In this work, we use newer AI methodologies like Large Language Models for word-base representations and K-Means and Neural Networks for understanding the underlying bases, in order to assess whether newer techniques agree with the widely accepted personality bases, and if not, what might be the optimal categorization for human personality.

Objective measurements of voice quality and personality OCEAN traits

The quality of human voice plays an important role across various fields yet it lacks a universally accepted, objective definition. Despite this subjectivity, extensive research across disciplines has linked these voice qualities to specific information about the speaker, such as health, physiological traits, and others. This work aims to objectively quantify voice quality and personality by synthesizing formulaic representations from past findings that correlate voice qualities to signal-processing metrics; and based on the known correlations of voice qualities to personality OCEAN traits, devising formulae to measure OCEAN traits objectively from voice.

In this work, we introduce formulae for 24 voice sub-qualities based on 25 signal properties, grounded in scientific literature, and also introduce formulae for the five OCEAN personality traits based on these 24 voice qualities, grounded in findings from past scientific studies. The soundness and correctness of voice quality and the OCEAN trait formulas are tested using various data-driven and human-expert-assessment methods.

Benchmarking VoIP applications

Our project on benchmarking VoIP applications is building a comprehensive performance evaluation framework tailored for businesses and developers overseeing real-time communication systems. Addressing the critical need to ensure high-quality voice transmission under varying network conditions, it assesses audio quality, latency, and system robustness against disruptions.

Unlike traditional benchmarking tools that focus only on data transfer metrics, our solution incorporates AI-driven analysis of speech intelligibility, noise suppression, and speaker recognition. This differentiation ensures that your voice communications are optimized for clarity and reliability, even in the most challenging network environments.

Personalized speech enhancement and separation

The work leverages personalized identity references—such as speaker embeddings, voiceprints, or auxiliary modalities like images, videos, or text annotations—to guide the enhancement process. These references provide crucial information about the speaker's unique vocal characteristics, including pitch, timbre, articulation style, accent etc. By incorporating these personalized features, the enhancement system ensures that the output not only suppresses blind background noise and distortions which include interfering speech, but also faithfully preserves the speaker's individuality.

Foundation models that can comparatively analyze speakers

Current speaker recognition models are often costrained to basic tasks and lack the capacity to generate descriptive speaker characteristics and context-rich results. In this work we build a novel Speaker Language Model (SLM) framework that addresses these limitations by combining a speaker encoder with prompt-based conditioning. The model then generates descriptive captions for speaker embeddings, accurately capturing key attributes such as dialect, gender, and age. By utilizing user-defined prompts, the model can produce customized captions tailored to specific requirements, such as regional dialect variations or age-related characteristics. Our approach leverages the complementary strengths of language models and speaker embedding techniques, and enables the model to produce descriptions that not only capture the unique attributes of each speaker but also provide nuanced and adaptable profiling.

This framework offers significant advancements in speaker recognition systems, offering enhanced descriptive profiling and adaptability across a broad range of applications.

Learning from imprecise labels

We introduce the Imprecise Label Learning (ILL) framework for audio processing, a unified approach to handle various imprecise labeling configurations, such as noisy labels, partial labels, and semi-supervised learning. By leveraging expectation-maximization (EM), ILL models imprecise label information as latent variables, enabling robust and scalable learning across diverse and mixed imprecise label scenarios, surpassing existing specialized methods. Many of the concepts in this work are easily extended to multimedia processing scenarios.

Framework to generate anomaly data for audio anomaly event detection

We introduce Audio Anomaly Data Generation (AADG), a framework leveraging Large Language Models (LLMs) to generate realistic audio data for anomaly detection and synthetic localization. Unlike existing datasets focusing on industrial sounds, this framework creates diverse, real-world scenarios, including anomalies, and provides a modular, scalable, and verifiable tool for generating synthetic audio data. Its impact lies in addressing the critical gap in general-purpose audio anomaly datasets, enhancing the training and benchmarking of audio anomaly detection models, and improving audio perception in safety-critical and real-world applications.