Ask or search anything...
Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks
24 Oct 2025
UC Berkeley
Stanford UniversityWhat features neural networks learn, and how, remains an open question. In this paper, we introduce Alternating Gradient Flows (AGF), an algorithmic framework that describes the dynamics of feature learning in two-layer networks trained from small initialization. Prior works have shown that gradient flow in this regime exhibits a staircase-like loss curve, alternating between plateaus where neurons slowly align to useful directions and sharp drops where neurons rapidly grow in norm. AGF approximates this behavior as an alternating two-step process: maximizing a utility function over dormant neurons and minimizing a cost function over active ones. AGF begins with all neurons dormant. At each iteration, a dormant neuron activates, triggering the acquisition of a feature and a drop in the loss. AGF quantifies the order, timing, and magnitude of these drops, matching experiments across several commonly studied architectures. We show that AGF unifies and extends existing saddle-to-saddle analyses in fully connected linear networks and attention-only linear transformers, where the learned features are singular modes and principal components, respectively. In diagonal linear networks, we prove AGF converges to gradient flow in the limit of vanishing initialization. Applying AGF to quadratic networks trained to perform modular addition, we give the first complete characterization of the training dynamics, revealing that networks learn Fourier features in decreasing order of coefficient magnitude. Altogether, AGF offers a promising step towards understanding feature learning in neural networks.
PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models
30 Apr 2025
KU LeuvenGeospatial Foundation Models (GFMs) have emerged as powerful tools for
extracting representations from Earth observation data, but their evaluation
remains inconsistent and narrow. Existing works often evaluate on suboptimal
downstream datasets and tasks, that are often too easy or too narrow, limiting
the usefulness of the evaluations to assess the real-world applicability of
GFMs. Additionally, there is a distinct lack of diversity in current evaluation
protocols, which fail to account for the multiplicity of image resolutions,
sensor types, and temporalities, which further complicates the assessment of
GFM performance. In particular, most existing benchmarks are geographically
biased towards North America and Europe, questioning the global applicability
of GFMs. To overcome these challenges, we introduce PANGAEA, a standardized
evaluation protocol that covers a diverse set of datasets, tasks, resolutions,
sensor modalities, and temporalities. It establishes a robust and widely
applicable benchmark for GFMs. We evaluate the most popular GFMs openly
available on this benchmark and analyze their performance across several
domains. In particular, we compare these models to supervised baselines (e.g.
UNet and vanilla ViT), and assess their effectiveness when faced with limited
labeled data. Our findings highlight the limitations of GFMs, under different
scenarios, showing that they do not consistently outperform supervised models.
PANGAEA is designed to be highly extensible, allowing for the seamless
inclusion of new datasets, models, and tasks in future research. By releasing
the evaluation code and benchmark, we aim to enable other researchers to
replicate our experiments and build upon our work, fostering a more principled
evaluation protocol for large pre-trained geospatial models. The code is
available at this https URL
Near-Optimal Clustering in Mixture of Markov Chains
KAIST
MITResearchers from KAIST, MIT, and KTH establish the first instance-specific high-probability lower bound on the clustering error for Mixture of Markov Chains. They also propose a two-stage, parameter-free algorithm that achieves a near-optimal error rate, matching the derived lower bound asymptotically and demonstrating improved sample complexity compared to previous methods.
View blog
Multi-Marginal Flow Matching with Adversarially Learnt Interpolants
01 Oct 2025
Learning the dynamics of a process given sampled observations at several time points is an important but difficult task in many scientific applications. When no ground-truth trajectories are available, but one has only snapshots of data taken at discrete time steps, the problem of modelling the dynamics, and thus inferring the underlying trajectories, can be solved by multi-marginal generalisations of flow matching algorithms. This paper proposes a novel flow matching method that overcomes the limitations of existing multi-marginal trajectory inference algorithms. Our proposed method, ALI-CFM, uses a GAN-inspired adversarial loss to fit neurally parametrised interpolant curves between source and target points such that the marginal distributions at intermediate time points are close to the observed distributions. The resulting interpolants are smooth trajectories that, as we show, are unique under mild assumptions. These interpolants are subsequently marginalised by a flow matching algorithm, yielding a trained vector field for the underlying dynamics. We showcase the versatility and scalability of our method by outperforming the existing baselines on spatial transcriptomics and cell tracking datasets, while performing on par with them on single-cell trajectory prediction.
Code: this https URL.
Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect
We introduce Atlas-Chat, the first-ever collection of LLMs specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also known as Darija, we construct our instruction dataset by consolidating existing Darija language resources, creating novel datasets both manually and synthetically, and translating English instructions with stringent quality control. Atlas-Chat-2B, 9B, and 27B models, fine-tuned on the dataset, exhibit superior ability in following Darija instructions and performing standard NLP tasks. Notably, our models outperform both state-of-the-art and Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., our 9B model gains a 13% performance boost over a larger 13B model on DarijaMMLU, in our newly introduced evaluation suite for Darija covering both discriminative and generative tasks. Furthermore, we perform an experimental analysis of various fine-tuning strategies and base model choices to determine optimal configurations. All our resources are publicly accessible, and we believe our work offers comprehensive design methodologies of instruction-tuning for low-resource languages, which are often neglected in favor of data-rich languages by contemporary LLMs.
Scorch: A Library for Sparse Deep Learning
20 Jun 2024
Scorch integrates a compiler-driven approach for sparse tensor computation directly into PyTorch, enabling efficient and general sparse deep learning. It automates performance optimizations for sparse operations, demonstrating substantial speedups for models like GCNs, sparse autoencoders, and sparse transformers on CPU inference.
View blog
Earth-Adapter: Bridge the Geospatial Domain Gaps with Mixture of Frequency Adaptation
Shanghai Jiao Tong UniversityParameter-Efficient Fine-Tuning (PEFT) is a technique that allows us to adapt powerful Foundation Models (FMs) to diverse downstream tasks while preserving and unleashing their inherent capabilities. However, we have observed that existing PEFT methods, which are often designed with natural imagery in mind, struggle when applied to Remote Sensing (RS) scenarios. This is primarily due to their inability to handle artifact influences, a problem particularly severe in RS image features. To tackle this challenge, we introduce Earth-Adapter, the first PEFT method specifically designed for RS artifacts conquering. Earth-Adapter introduces a novel Mixture of Frequency Adaptation process that combines a Mixture of Adapter (MoA) with Discrete Fourier Transformation (DFT). By utilizing DFT, Earth-Adapter can decompose features into different frequency components, precisely separating artifacts from original features. The MoA then dynamically assigns weights to each adapter expert, allowing for the combination of features across various frequency domains. These simple-yet-effective approaches enable Earth-Adapter to more efficiently overcome the disturbances caused by artifacts than previous PEFT methods, significantly enhancing the FMs' performance on RS scenarios. Experiments on Domain Adaptation (DA), and Domain Generalization (DG) semantic segmentation benchmarks showcase the Earth-Adapter's effectiveness. Compared with baseline Rein, Earth-Adapter significantly improves 9.0% mIoU in DA and 3.1% mIoU in DG benchmarks. Our code will be released at this https URL.
DexDiffuser: Generating Dexterous Grasps with Diffusion Models
06 Nov 2024
Researchers at KTH Royal Institute of Technology developed DexDiffuser, a framework utilizing conditional diffusion models and refinement strategies to generate 16-DoF dexterous grasps for unknown objects from partial point clouds. It achieved a real-world grasp success rate of 68.89%, outperforming previous methods by 20.00% on diverse objects.
View blog
Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation
27 Mar 2025
ETH Zurich
University of TorontoSemLA introduces a training-free test-time adaptation method for open-vocabulary semantic segmentation that addresses domain shift. It builds a library of domain-specific LoRA adapters, indexed by CLIP embeddings, and dynamically merges the most semantically relevant adapters for a given test image to improve segmentation quality, achieving 2-4% mIoU gains over zero-shot baselines on a diverse 20-domain benchmark.
View blog
Can Generative Geospatial Diffusion Models Excel as Discriminative Geospatial Foundation Models?
12 Sep 2025
Self-supervised learning (SSL) has revolutionized representation learning in Remote Sensing (RS), advancing Geospatial Foundation Models (GFMs) to leverage vast unlabeled satellite imagery for diverse downstream tasks. Currently, GFMs primarily employ objectives like contrastive learning or masked image modeling, owing to their proven success in learning transferable representations. However, generative diffusion models, which demonstrate the potential to capture multi-grained semantics essential for RS tasks during image generation, remain underexplored for discriminative applications. This prompts the question: can generative diffusion models also excel and serve as GFMs with sufficient discriminative power? In this work, we answer this question with SatDiFuser, a framework that transforms a diffusion-based generative geospatial foundation model into a powerful pretraining tool for discriminative RS. By systematically analyzing multi-stage, noise-dependent diffusion features, we develop three fusion strategies to effectively leverage these diverse representations. Extensive experiments on remote sensing benchmarks show that SatDiFuser outperforms state-of-the-art GFMs, achieving gains of up to +5.7% mIoU in semantic segmentation and +7.9% F1-score in classification, demonstrating the capacity of diffusion-based generative foundation models to rival or exceed discriminative GFMs. The source code is available at: this https URL.
Emotional Speech-Driven Animation with Content-Emotion Disentanglement
26 Sep 2023
To be widely adopted, 3D facial avatars must be animated easily, realistically, and directly from speech signals. While the best recent methods generate 3D animations that are synchronized with the input audio, they largely ignore the impact of emotions on facial expressions. Realistic facial animation requires lip-sync together with the natural expression of emotion. To that end, we propose EMOTE (Expressive Model Optimized for Talking with Emotion), which generates 3D talking-head avatars that maintain lip-sync from speech while enabling explicit control over the expression of emotion. To achieve this, we supervise EMOTE with decoupled losses for speech (i.e., lip-sync) and emotion. These losses are based on two key observations: (1) deformations of the face due to speech are spatially localized around the mouth and have high temporal frequency, whereas (2) facial expressions may deform the whole face and occur over longer intervals. Thus, we train EMOTE with a per-frame lip-reading loss to preserve the speech-dependent content, while supervising emotion at the sequence level. Furthermore, we employ a content-emotion exchange mechanism in order to supervise different emotions on the same audio, while maintaining the lip motion synchronized with the speech. To employ deep perceptual losses without getting undesirable artifacts, we devise a motion prior in the form of a temporal VAE. Due to the absence of high-quality aligned emotional 3D face datasets with speech, EMOTE is trained with 3D pseudo-ground-truth extracted from an emotional video dataset (i.e., MEAD). Extensive qualitative and perceptual evaluations demonstrate that EMOTE produces speech-driven facial animations with better lip-sync than state-of-the-art methods trained on the same data, while offering additional, high-quality emotional control.
TopoX: A Suite of Python Packages for Machine Learning on Topological Domains
09 Dec 2024
Tolga Birdal
Maosheng Yang
Luca Scofano
University of California, Santa BarbaraWe introduce TopoX, a Python software suite that provides reliable and user-friendly building blocks for computing and machine learning on topological domains that extend graphs: hypergraphs, simplicial, cellular, path and combinatorial complexes. TopoX consists of three packages: TopoNetX facilitates constructing and computing on these domains, including working with nodes, edges and higher-order cells; TopoEmbedX provides methods to embed topological domains into vector spaces, akin to popular graph-based embedding algorithms such as node2vec; TopoModelX is built on top of PyTorch and offers a comprehensive toolbox of higher-order message passing functions for neural networks on topological domains. The extensively documented and unit-tested source code of TopoX is available under MIT license at this https URL}{this https URL.
Differential learning kinetics govern the transition from memorization to generalization during in-context learning
Princeton UniversityTransformers exhibit in-context learning (ICL): the ability to use novel
information presented in the context without additional weight updates. Recent
work shows that ICL emerges when models are trained on a sufficiently diverse
set of tasks and the transition from memorization to generalization is sharp
with increasing task diversity. One interpretation is that a network's limited
capacity to memorize favors generalization. Here, we examine the mechanistic
underpinnings of this transition using a small transformer applied to a
synthetic ICL task. Using theory and experiment, we show that the sub-circuits
that memorize and generalize can be viewed as largely independent. The relative
rates at which these sub-circuits learn explains the transition from
memorization to generalization, rather than capacity constraints. We uncover a
memorization scaling law, which determines the task diversity threshold at
which the network generalizes. The theory quantitatively explains a variety of
other ICL-related phenomena, including the long-tailed distribution of when ICL
is acquired, the bimodal behavior of solutions close to the task diversity
threshold, the influence of contextual and data distributional statistics on
ICL, and the transient nature of ICL.
Inverse Problems with Diffusion Models: A MAP Estimation Perspective
18 Sep 2024
Inverse problems have many applications in science and engineering. In Computer vision, several image restoration tasks such as inpainting, deblurring, and super-resolution can be formally modeled as inverse problems. Recently, methods have been developed for solving inverse problems that only leverage a pre-trained unconditional diffusion model and do not require additional task-specific training. In such methods, however, the inherent intractability of determining the conditional score function during the reverse diffusion process poses a real challenge, leaving the methods to settle with an approximation instead, which affects their performance in practice. Here, we propose a MAP estimation framework to model the reverse conditional generation process of a continuous time diffusion model as an optimization process of the underlying MAP objective, whose gradient term is tractable. In theory, the proposed framework can be applied to solve general inverse problems using gradient-based optimization methods. However, given the highly non-convex nature of the loss objective, finding a perfect gradient-based optimization algorithm can be quite challenging, nevertheless, our framework offers several potential research directions. We use our proposed formulation to develop empirically effective algorithms for image restoration. We validate our proposed algorithms with extensive experiments over multiple datasets across several restoration tasks.
Near-optimal Rank Adaptive Inference of High Dimensional Matrices
09 Oct 2025
KTH Royal Institute of TechnologyWe address the problem of estimating a high-dimensional matrix from linear measurements, with a focus on designing optimal rank-adaptive algorithms. These algorithms infer the matrix by estimating its singular values and the corresponding singular vectors up to an effective rank, adaptively determined based on the data. We establish instance-specific lower bounds for the sample complexity of such algorithms, uncovering fundamental trade-offs in selecting the effective rank: balancing the precision of estimating a subset of singular values against the approximation cost incurred for the remaining ones. Our analysis identifies how the optimal effective rank depends on the matrix being estimated, the sample size, and the noise level. We propose an algorithm that combines a Least-Squares estimator with a universal singular value thresholding procedure. We provide finite-sample error bounds for this algorithm and demonstrate that its performance nearly matches the derived fundamental limits. Our results rely on an enhanced analysis of matrix denoising methods based on singular value thresholding. We validate our findings with applications to multivariate regression and linear dynamical system identification.
CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models
11 Mar 2024
The Apex Data & Knowledge Management Lab at Shanghai Jiao Tong University developed CodeApex, a comprehensive bilingual (English and Chinese) benchmark for assessing large language models across programming comprehension, code generation, and correction tasks. Evaluations using CodeApex showed GPT-4 led in all tasks, for example, achieving around 69% accuracy in comprehension, while human experts consistently demonstrated superior performance, scoring approximately 80% in comprehension.
View blog
The Convergence of Sparsified Gradient Methods
27 Sep 2018
Distributed training of massive machine learning models, in particular deep
neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace.
Several families of communication-reduction methods, such as quantization,
large-batch methods, and gradient sparsification, have been proposed. To date,
gradient sparsification methods - where each node sorts gradients by magnitude,
and only communicates a subset of the components, accumulating the rest locally
- are known to yield some of the largest practical gains. Such methods can
reduce the amount of communication per step by up to three orders of magnitude,
while preserving model accuracy. Yet, this family of methods currently has no
theoretical justification.
This is the question we address in this paper. We prove that, under analytic
assumptions, sparsifying gradients by magnitude with local error correction
provides convergence guarantees, for both convex and non-convex smooth
objectives, for data-parallel SGD. The main insight is that sparsification
methods implicitly maintain bounds on the maximum impact of stale updates,
thanks to selection by magnitude. Our analysis and empirical validation also
reveal that these methods do require analytical conditions to converge well,
justifying existing heuristics.
Consistency Regularization Can Improve Robustness to Label Noise
04 Oct 2021
Consistency regularization is a commonly-used technique for semi-supervised and self-supervised learning. It is an auxiliary objective function that encourages the prediction of the network to be similar in the vicinity of the observed training samples. Hendrycks et al. (2020) have recently shown such regularization naturally brings test-time robustness to corrupted data and helps with calibration. This paper empirically studies the relevance of consistency regularization for training-time robustness to noisy labels. First, we make two interesting and useful observations regarding the consistency of networks trained with the standard cross entropy loss on noisy datasets which are: (i) networks trained on noisy data have lower consistency than those trained on clean data, and(ii) the consistency reduces more significantly around noisy-labelled training data points than correctly-labelled ones. Then, we show that a simple loss function that encourages consistency improves the robustness of the models to label noise on both synthetic (CIFAR-10, CIFAR-100) and real-world (WebVision) noise as well as different noise rates and types and achieves state-of-the-art results.
CAPGrasp: An Continuous Approach-Constrained Generative Grasp Sampler
07 Mar 2024
We propose CAPGrasp, an 6-DoF continuous approach-constrained generative grasp sampler. It includes a novel learning strategy for training CAPGrasp that eliminates the need to curate massive conditionally labeled datasets and a constrained grasp refinement technique that improves grasp poses while respecting the grasp approach directional constraints. The experimental results demonstrate that CAPGrasp is more than three times as sample efficient as unconstrained grasp samplers while achieving up to 38% grasp success rate improvement. CAPGrasp also achieves 4-10% higher grasp success rates than constrained but noncontinuous grasp samplers. Overall, CAPGrasp is a sample-efficient solution when grasps must originate from specific directions, such as grasping in confined spaces.
Neural Lattice Reduction: A Self-Supervised Geometric Deep Learning Approach
10 Feb 2025
Lattice reduction is a combinatorial optimization problem aimed at finding
the most orthogonal basis in a given lattice. The Lenstra-Lenstra-Lov\'asz
(LLL) algorithm is the best algorithm in the literature for solving this
problem. In light of recent research on algorithm discovery, in this work, we
would like to answer this question: is it possible to parametrize the algorithm
space for lattice reduction problem with neural networks and find an algorithm
without supervised data? Our strategy is to use equivariant and invariant
parametrizations and train in a self-supervised way. We design a deep neural
model outputting factorized unimodular matrices and train it in a
self-supervised manner by penalizing non-orthogonal lattice bases. We
incorporate the symmetries of lattice reduction into the model by making it
invariant to isometries and scaling of the ambient space and equivariant with
respect to the hyperocrahedral group permuting and flipping the lattice basis
elements. We show that this approach yields an algorithm with comparable
complexity and performance to the LLL algorithm on a set of benchmarks.
Additionally, motivated by certain applications for wireless communication, we
extend our method to a convolutional architecture which performs joint
reduction of spatially-correlated lattices arranged in a grid, thereby
amortizing its cost over multiple lattices.
There are no more papers matching your filters at the moment.