Ask or search anything...
Equivariant Self-Supervision for Musical Tempo Estimation
03 Sep 2022
Self-supervised methods have emerged as a promising avenue for representation
learning in the recent years since they alleviate the need for labeled
datasets, which are scarce and expensive to acquire. Contrastive methods are a
popular choice for self-supervision in the audio domain, and typically provide
a learning signal by forcing the model to be invariant to some transformations
of the input. These methods, however, require measures such as negative
sampling or some form of regularisation to be taken to prevent the model from
collapsing on trivial solutions. In this work, instead of invariance, we
propose to use equivariance as a self-supervision signal to learn audio tempo
representations from unlabelled data. We derive a simple loss function that
prevents the network from collapsing on a trivial solution during training,
without requiring any form of regularisation or negative sampling. Our
experiments show that it is possible to learn meaningful representations for
tempo estimation by solely relying on equivariant self-supervision, achieving
performance comparable with supervised methods on several benchmarks. As an
added benefit, our method only requires moderate compute resources and
therefore remains accessible to a wide research community.
Not All Deepfakes Are Created Equal: Triaging Audio Forgeries for Robust Deepfake Singer Identification
20 Oct 2025
A two-stage pipeline called D ◦ S is introduced to robustly identify artists' vocal likenesses in both authentic and deepfake audio, specifically by triaging and prioritizing high-quality forgeries. This approach improves singer identification accuracy on deepfake-containing datasets, reducing the average Equal Error Rate from 24.79% to 16.19%.
View blog
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
Queen Mary University of LondonMultimodal models that jointly process audio and language hold great promise in audio understanding and are increasingly being adopted in the music domain. By allowing users to query via text and obtain information about a given audio input, these models have the potential to enable a variety of music understanding tasks via language-based interfaces. However, their evaluation poses considerable challenges, and it remains unclear how to effectively assess their ability to correctly interpret music-related inputs with current methods. Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music understanding in multimodal language models focused on audio. MuChoMusic comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets, and covering a wide variety of genres. Questions in the benchmark are crafted to assess knowledge and reasoning abilities across several dimensions that cover fundamental musical concepts and their relation to cultural and functional contexts. Through the holistic analysis afforded by the benchmark, we evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality, pointing to a need for better multimodal integration. Data and code are open-sourced.
SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding
21 Jun 2025
Joint embedding spaces have significantly advanced music understanding and generation by linking text and audio through multimodal contrastive learning. However, these approaches face large memory requirement limitations due to relying on large batch sizes to effectively utilize negative samples. Further, multimodal joint embedding spaces suffer from a modality gap wherein embeddings from different modalities lie in different manifolds of the embedding space. To address these challenges, we propose Siamese Language-Audio Pretraining (SLAP), a novel multimodal pretraining framework that allows learning powerful representations without negative samples. SLAP adapts the Bootstrap Your Own Latent (BYOL) paradigm for multimodal audio-text training, promoting scalability in training multimodal embedding spaces.
We illustrate the ability of our model to learn meaningful relationships between music and text -- specifically, we show that SLAP outperforms CLAP on tasks such as text-music retrieval and zero-shot classification. We also observe competitive downstream performance on several MIR tasks, including with larger or supervised models (genre and instrument classification, auto-tagging). Additionally, our approach has attractive properties, such as a quantifiably reduced modality gap and improved robustness to batch size variations on retrieval performance. Finally, its novel formulation unlocks large-scale training on a single GPU through gradient accumulation.
GD-Retriever: Controllable Generative Text-Music Retrieval with Diffusion Models
A framework called GD-Retriever employs conditional latent diffusion models to synthesize hypothetical audio latent queries from text, enabling a more controllable text-music retrieval experience. The method improves in-domain retrieval performance compared to contrastive baselines and allows interactive refinement of search results through negative prompting and DDIM inversion, all while preserving the fidelity of modified queries.
View blog
Contrastive Audio-Language Learning for Music
As one of the most intuitive interfaces known to humans, natural language has the potential to mediate many tasks that involve human-computer interaction, especially in application-focused fields like Music Information Retrieval. In this work, we explore cross-modal learning in an attempt to bridge audio and language in the music domain. To this end, we propose MusCALL, a framework for Music Contrastive Audio-Language Learning. Our approach consists of a dual-encoder architecture that learns the alignment between pairs of music audio and descriptive sentences, producing multimodal embeddings that can be used for text-to-audio and audio-to-text retrieval out-of-the-box. Thanks to this property, MusCALL can be transferred to virtually any task that can be cast as text-based retrieval. Our experiments show that our method performs significantly better than the baselines at retrieving audio that matches a textual description and, conversely, text that matches an audio query. We also demonstrate that the multimodal alignment capability of our model can be successfully extended to the zero-shot transfer scenario for genre classification and auto-tagging on two public datasets.
Leave-One-EquiVariant: Alleviating invariance-related information loss in contrastive music representations
Contrastive learning has proven effective in self-supervised musical representation learning, particularly for Music Information Retrieval (MIR) tasks. However, reliance on augmentation chains for contrastive view generation and the resulting learnt invariances pose challenges when different downstream tasks require sensitivity to certain musical attributes. To address this, we propose the Leave One EquiVariant (LOEV) framework, which introduces a flexible, task-adaptive approach compared to previous work by selectively preserving information about specific augmentations, allowing the model to maintain task-relevant equivariances. We demonstrate that LOEV alleviates information loss related to learned invariances, improving performance on augmentation related tasks and retrieval without sacrificing general representation quality. Furthermore, we introduce a variant of LOEV, LOEV++, which builds a disentangled latent space by design in a self-supervised manner, and enables targeted retrieval based on augmentation related attributes.
Not All Deepfakes Are Created Equal: Triaging Audio Forgeries for Robust Deepfake Singer Identification
20 Oct 2025
The proliferation of highly realistic singing voice deepfakes presents a significant challenge to protecting artist likeness and content authenticity. Automatic singer identification in vocal deepfakes is a promising avenue for artists and rights holders to defend against unauthorized use of their voice, but remains an open research problem. Based on the premise that the most harmful deepfakes are those of the highest quality, we introduce a two-stage pipeline to identify a singer's vocal likeness. It first employs a discriminator model to filter out low-quality forgeries that fail to accurately reproduce vocal likeness. A subsequent model, trained exclusively on authentic recordings, identifies the singer in the remaining high-quality deepfakes and authentic audio. Experiments show that this system consistently outperforms existing baselines on both authentic and synthetic content.
Towards a Unified Representation Evaluation Framework Beyond Downstream Tasks
A unified framework from QMUL and Universal Music Group researchers evaluates learned representations along four key axes (informativeness, equivariance, invariance, and disentanglement), revealing that models with similar downstream performance can exhibit drastically different structural properties when analyzed through controlled factors of variation in image and speech domains.
View blog
Learning music audio representations via weak language supervision
Audio representations for music information retrieval are typically learned
via supervised learning in a task-specific fashion. Although effective at
producing state-of-the-art results, this scheme lacks flexibility with respect
to the range of applications a model can have and requires extensively
annotated datasets. In this work, we pose the question of whether it may be
possible to exploit weakly aligned text as the only supervisory signal to learn
general-purpose music audio representations. To address this question, we
design a multimodal architecture for music and language pre-training (MuLaP)
optimised via a set of proxy tasks. Weak supervision is provided in the form of
noisy natural language descriptions conveying the overall musical content of
the track. After pre-training, we transfer the audio backbone of the model to a
set of music audio classification and regression tasks. We demonstrate the
usefulness of our approach by comparing the performance of audio
representations produced by the same audio backbone with different training
strategies and show that our pre-training method consistently achieves
comparable or higher scores on all tasks and datasets considered. Our
experiments also confirm that MuLaP effectively leverages audio-caption pairs
to learn representations that are competitive with audio-only and cross-modal
self-supervised methods in the literature.
The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation
22 Nov 2023
Université de Montréal
ByteDanceThe Song Describer Dataset (SDD) provides a new public corpus of 1106 human-written music captions for 706 unique 2-minute music recordings, explicitly designed for out-of-domain evaluation of music-and-language models. Benchmarking with SDD demonstrates significant performance drops for music captioning models and reveals generalization limitations for existing models when tested on unseen data distributions.
View blog
Robust Lossy Audio Compression Identification
31 Jul 2024
Previous research contributions on blind lossy compression identification
report near perfect performance metrics on their test set, across a variety of
codecs and bit rates. However, we show that such results can be deceptive and
may not accurately represent true ability of the system to tackle the task at
hand. In this article, we present an investigation into the robustness and
generalisation capability of a lossy audio identification model. Our
contributions are as follows. (1) We show the lack of robustness to codec
parameter variations of a model equivalent to prior art. In particular, when
naively training a lossy compression detection model on a dataset of music
recordings processed with a range of codecs and their lossless counterparts, we
obtain near perfect performance metrics on the held-out test set, but severely
degraded performance on lossy tracks produced with codec parameters not seen in
training. (2) We propose and show the effectiveness of an improved training
strategy to significantly increase the robustness and generalisation capability
of the model beyond codec configurations seen during training. Namely we apply
a random mask to the input spectrogram to encourage the model not to rely
solely on the training set's codec cutoff frequency.
Semi-Supervised Contrastive Learning of Musical Representations
Despite the success of contrastive learning in Music Information Retrieval, the inherent ambiguity of contrastive self-supervision presents a challenge. Relying solely on augmentation chains and self-supervised positive sampling strategies can lead to a pretraining objective that does not capture key musical information for downstream tasks. We introduce semi-supervised contrastive learning (SemiSupCon), a simple method for leveraging musically informed labeled data (supervision signals) in the contrastive learning of musical representations. Our approach introduces musically relevant supervision signals into self-supervised contrastive learning by combining supervised and self-supervised contrastive objectives in a simpler framework than previous approaches. This framework improves downstream performance and robustness to audio corruptions on a range of downstream MIR tasks with moderate amounts of labeled data. Our approach enables shaping the learned similarity metric through the choice of labeled data that (1) infuses the representations with musical domain knowledge and (2) improves out-of-domain performance with minimal general downstream performance loss. We show strong transfer learning performance on musically related yet not trivially similar tasks - such as pitch and key estimation. Additionally, our approach shows performance improvement on automatic tagging over self-supervised approaches with only 5\% of available labels included in pretraining.
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
Multimodal models that jointly process audio and language hold great promise in audio understanding and are increasingly being adopted in the music domain. By allowing users to query via text and obtain information about a given audio input, these models have the potential to enable a variety of music understanding tasks via language-based interfaces. However, their evaluation poses considerable challenges, and it remains unclear how to effectively assess their ability to correctly interpret music-related inputs with current methods. Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music understanding in multimodal language models focused on audio. MuChoMusic comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets, and covering a wide variety of genres. Questions in the benchmark are crafted to assess knowledge and reasoning abilities across several dimensions that cover fundamental musical concepts and their relation to cultural and functional contexts. Through the holistic analysis afforded by the benchmark, we evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality, pointing to a need for better multimodal integration. Data and code are open-sourced.
There are no more papers matching your filters at the moment.