Center for Voice Intelligence and Security

Responsive autoregressive audio-image generation

Autoregressive (AR) image/audio generation has achieved notable progress and demonstrated promising performance recently. The current advanced image/audio generation paradigm typically requires a pre-trained image tokenizer that encodes the image into a more compact latent space, where an AR generator, e.g. transformers, is leveraged to model the latent distribution, for both efficiency and effectiveness in generation.

Unified speech enhancement and editing with conditional diffusion models

In this work we have built a unified Speech Enhancement and Editing (uSee) framework with conditional diffusion models to enable fine-grained controllable generation based on both acoustic and textual prompts. This model achieves superior performance in both speech denoising and dereverberation compared to other related generative speech enhancement models, and can perform speech editing given desired environmental sound text description, signal-to-noise ratios (SNR), and room impulse responses (RIR).

Continual learning paradigms for AI systems

Continual learning paradigms have transformative applications in developing AI systems capable of adapting to evolving tasks and environments without losing previously acquired knowledge. Existing continual learning metrics neglect the overall performance across all tasks, and do not adequately disentangle the plasticity versus stability/generalizability trade-offs in the model. We have created the Dual-transfer Matching Index (DMI) as a unified metric to evaluate all three properties of a continual learning system: stability, plasticity and generalizability. By employing the proposed metric, we demonstrate how introducing various knowledge distillations can improve these three properties.

Wheeler

Wheeler was built by Kelsey Joseph Harvey, in collaboration with General Robotics. Wheeler is a cornerstone of General Robotics' comprehensive research initiative in Embodied AI, representing our wheeled platform implementation. Working in parallel with our ROVER quadruped platform, Wheeler demonstrates how our AI technologies can adapt to and leverage different physical embodiments while maintaining consistent core capabilities across platforms.

Wheeler serves as a key platform in our broader Embodied AI initiative wihich includes testing and validation of cross-platform AI capabilities, physical manipulation tasks in real-world environments, computer vision integration with acoustic sensing, multi-modal learning and adaptation, real-time decision making and planning and development of transferable AI skills across different robot embodiments.

Adversarial robustness techniques for multimedia modalities

In this project we are developing techniques for measuring and enhancing the robustness of AI models to corruption, including adversarial attacks. Our work leverages biologically motivated neural network architectures and features that enhance robustness. For speech specifically, we have developed Speech Robust Bench, a comprehensive robustness benchmark for ASR.

Our systems include self-consistent activation (SCA) to address the critical challenge in deep neural networks: vulnerability to adversarial perturbations. By introducing fixed inter-neuron covariability patterns, SCA enhances the robustness of DNNs to adversarial attacks without compromising accuracy on natural data.

Learning methods for audio foundation models

We explore learning audio foundation models through the lens of language. We propose Contrastive Audio-Language Pretraining (CLAP) to learn joint audio-text representations. Once trained, the model can enable applications like zero-shot classification, text-to-audio retrieval, and audio-to-text retrieval. To support open-ended tasks like audio captioning and audio question answering, we introduce Pengi, an Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. Combining CLAP and Pengi, enables building an audio system that can perceive or understand multimodal input, and generate natural language responses.

Adaptation methods for audio foundation models

Audio foundation models (AFMs), though trained on millions of audio-text pairs, show near random performance on out-of-distribution/domain data. With this as backdrop, we are looking into adapting audio foundation models across different compute and data budgets. We have developed a test-time adaptation strategy for contrastive audio-language models, using unlabelled audio to learn soft tokens by minimizing entropy across different views. We are exploring few-shot learning scenarios with Pengi, focusing on achieving performance improvement with selected parameter updates. In this work we have also developed a method to counter missing modality at training time, allowing AFMs to be trained with a single modality or with generated data.

Reasoning, evaluation metrics, and benchmarks for audio foundation models

As audio foundation models (AFMs) scale, they begin to exhibit emergent abilities, with reasoning being the most impactful on downstream task performance.

Our work aims to enhance the reasoning capabilities of Audio Foundation Models (AFMs) and establish robust benchmarks for their performance. By leveraging emergent abilities, we explore innovative applications and metrics to address key challenges in audio processing. First, we demonstrate that AFMs can estimate audio quality without references using an antonym prompting strategy, paving the way for MelDiffusion, a diffusion model trained on spectrograms that achieves state-of-the-art audio generation quality. Building on this, we introduce audio entailment to evaluate deductive reasoning, addressing the reliance of models on Large Language Models (LLMs) by proposing a "caption-before-audio" strategy for improved audio-grounded logic. To measure comparative reasoning, we propose audio difference explanation, showing that smaller models can rival larger ones with optimized architectures and learning strategies. Finally, we introduce MACE, a novel audio captioning metric that incorporates audio information and achieves the highest correlation with human quality judgments. Together, these contributions form a unified approach to understanding, improving, and evaluating the reasoning and generative abilities of AFMs, supported by human-annotated datasets for comprehensive benchmarking.