Ask or search anything...
Symmetry-Based Classification of Chern Phases in Honeycomb Photonic Crystals
05 Dec 2024
In this work, we develop a symmetry-based classification of Chern phases in honeycomb photonic crystals, considering arbitrary nonreciprocal couplings compatible with energy conservation. Our analysis focuses on crystals formed through nonreciprocal perturbations of photonic graphene. These perturbations, which can have arbitrary spatial variations, are generally described by scalar and vector fields. Using a tight-binding model, we consider the most general nonreciprocal interactions, including gyromagnetic, pseudo-Tellegen, and moving medium responses, and examine how the corresponding nonreciprocal fields influence the crystal's topology. Our findings reveal that nonreciprocal interactions alone are insufficient to induce a topologically nontrivial phase. Instead, a nontrivial p6m component in the nonreciprocal fields is required to open a bandgap and achieve a non-zero Chern number. These results provide a symmetry-based roadmap for engineering photonic topological phases via nonreciprocal perturbations of photonic graphene, offering practical guidelines for designing topological phases in graphene-like photonic crystals.
The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
26 Feb 2025
Chenglei Si (司程磊)
Pranav Dulepet
Stanford University
OpenAIResearchers from the University of Maryland, OpenAI, Stanford University, and other institutions present a systematic survey of prompt engineering, developing a comprehensive taxonomy of 58 LLM prompting techniques and a standardized vocabulary. Their work includes empirical benchmarking on MMLU, which generally shows performance improvements with more complex prompting, and a case study demonstrating that automated prompt optimization can outperform human-designed prompts.
View blog
Long-Context Generalization with Sparse Attention
27 Sep 2025
Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many settings, this density has been shown to be detrimental for tasks that demand precise focus on fixed-size patterns: as sequence length increases, non-informative tokens accumulate attention probability mass, leading to dispersion and representational collapse. We show in this paper that dynamically sparse attention mechanisms using -entmax can avoid these issues, due to their ability to assign exact zeros to irrelevant tokens. Furthermore, we introduce Adaptive-Scalable Entmax (ASEntmax), which endows -entmax with a learnable temperature parameter, allowing the attention distribution to interpolate between sparse (pattern-focused) and dense (softmax-like) regimes. Our empirical evaluation on synthetic tasks and language modeling demonstrates that ASEntmax substantially outperforms softmax, scalable softmax, and fixed-temperature -entmax baselines, achieving up to 1000 length extrapolation on synthetic benchmarks and superior long-context generalization on language modeling while preserving short-context performance, including better perplexity trends and higher retrieval accuracies at 8 training length.
-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation
19 May 2025
∞-VIDEO presents a training-free approach that enables existing video-language models to understand arbitrarily long videos by augmenting them with a continuous-time long-term memory consolidation mechanism. This method improves performance on long-video Q&A tasks by dynamically retaining relevant information and avoids the need for additional training or sparse frame subsampling.
View blog
M-Prometheus: A Suite of Open Multilingual LLM Judges
29 Oct 2025
Graham Neubig
KAISTThe use of language models for automatically evaluating long-form text (LLM-as-a-judge) is becoming increasingly common, yet most LLM judges are optimized exclusively for English, with strategies for enhancing their multilingual evaluation capabilities remaining largely unexplored in the current literature. This has created a disparity in the quality of automatic evaluation methods for non-English languages, ultimately hindering the development of models with better multilingual capabilities. To bridge this gap, we introduce M-Prometheus, a suite of open-weight LLM judges ranging from 3B to 14B parameters that can provide both direct assessment and pairwise comparison feedback on multilingual outputs. M-Prometheus models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation (MT) evaluation covering 4 language pairs. Furthermore, M-Prometheus models can be leveraged at decoding time to significantly improve generated outputs across all 3 tested languages, showcasing their utility for the development of better multilingual models. Lastly, through extensive ablations, we identify the key factors for obtaining an effective multilingual judge, including backbone model selection and training on synthetic multilingual feedback data instead of translated data. We release our models, training dataset, and code.
LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
24 Feb 2025
State space models (SSMs), such as Mamba, have emerged as an efficient
alternative to transformers for long-context sequence modeling. However,
despite their growing adoption, SSMs lack the interpretability tools that have
been crucial for understanding and improving attention-based architectures.
While recent efforts provide insights into Mamba's internal mechanisms, they do
not explicitly decompose token-wise contributions, leaving gaps in
understanding how Mamba selectively processes sequences across layers. In this
work, we introduce LaTIM, a novel token-level decomposition method for both
Mamba-1 and Mamba-2 that enables fine-grained interpretability. We extensively
evaluate our method across diverse tasks, including machine translation,
copying, and retrieval-based generation, demonstrating its effectiveness in
revealing Mamba's token-to-token interaction patterns.
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
19 Feb 2025
Beyza Ermis
Marzieh Fadaee
Shivalika Singh
Carnegie Mellon UniversityCultural biases in multilingual datasets pose significant challenges for
their effectiveness as global benchmarks. These biases stem not only from
differences in language but also from the cultural knowledge required to
interpret questions, reducing the practical utility of translated datasets like
MMLU. Furthermore, translation often introduces artefacts that can distort the
meaning or clarity of questions in the target language. A common practice in
multilingual evaluation is to rely on machine-translated evaluation sets, but
simply translating a dataset is insufficient to address these challenges. In
this work, we trace the impact of both of these issues on multilingual
evaluations and ensuing model performances. Our large-scale evaluation of
state-of-the-art open and proprietary models illustrates that progress on MMLU
depends heavily on learning Western-centric concepts, with 28% of all questions
requiring culturally sensitive knowledge. Moreover, for questions requiring
geographic knowledge, an astounding 84.9% focus on either North American or
European regions. Rankings of model evaluations change depending on whether
they are evaluated on the full portion or the subset of questions annotated as
culturally sensitive, showing the distortion to model rankings when blindly
relying on translated MMLU. We release Global MMLU, an improved MMLU with
evaluation coverage across 42 languages -- with improved overall quality by
engaging with compensated professional and community annotators to verify
translation quality while also rigorously evaluating cultural biases present in
the original dataset. This comprehensive Global MMLU set also includes
designated subsets labeled as culturally sensitive and culturally agnostic to
allow for more holistic, complete evaluation.
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
05 Jul 2024
Zheyu Zhang
Recent multilingual pretrained language models (mPLMs) have been shown to
encode strong language-specific signals, which are not explicitly provided
during pretraining. It remains an open question whether it is feasible to
employ mPLMs to measure language similarity, and subsequently use the
similarity results to select source languages for boosting cross-lingual
transfer. To investigate this, we propose mPLMSim, a language similarity
measure that induces the similarities across languages from mPLMs using
multi-parallel corpora. Our study shows that mPLM-Sim exhibits moderately high
correlations with linguistic similarity measures, such as lexicostatistics,
genealogical language family, and geographical sprachbund. We also conduct a
case study on languages with low correlation and observe that mPLM-Sim yields
more accurate similarity results. Additionally, we find that similarity results
vary across different mPLMs and different layers within an mPLM. We further
investigate whether mPLMSim is effective for zero-shot cross-lingual transfer
by conducting experiments on both low-level syntactic tasks and high-level
semantic tasks. The experimental results demonstrate that mPLM-Sim is capable
of selecting better source languages than linguistic measures, resulting in a
1%-2% improvement in zero-shot cross-lingual transfer performance.
Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs
20 Jun 2025
Unbabel's TOWER+ models demonstrate a successful approach to training multilingual Large Language Models that balance state-of-the-art machine translation performance with robust general-purpose capabilities. These models achieve high scores on both translation benchmarks and general instruction-following tasks like IFEval, often matching or surpassing leading proprietary and open-weight LLMs across various scales.
View blog
"I Never Said That": A dataset, taxonomy and baselines on response clarity classification
20 Sep 2024
Equivocation and ambiguity in public speech are well-studied discourse
phenomena, especially in political science and analysis of political
interviews. Inspired by the well-grounded theory on equivocation, we aim to
resolve the closely related problem of response clarity in questions extracted
from political interviews, leveraging the capabilities of Large Language Models
(LLMs) and human expertise. To this end, we introduce a novel taxonomy that
frames the task of detecting and classifying response clarity and a
corresponding clarity classification dataset which consists of question-answer
(QA) pairs drawn from political interviews and annotated accordingly. Our
proposed two-level taxonomy addresses the clarity of a response in terms of the
information provided for a given question (high-level) and also provides a
fine-grained taxonomy of evasion techniques that relate to unclear, ambiguous
responses (lower-level). We combine ChatGPT and human annotators to collect,
validate and annotate discrete QA pairs from political interviews, to be used
for our newly introduced response clarity task. We provide a detailed analysis
and conduct several experiments with different model architectures, sizes and
adaptation methods to gain insights and establish new baselines over the
proposed dataset and task.
-former: Infinite Memory Transformer
25 Mar 2022
Google DeepMindTransformers are unable to model long-term memories effectively, since the
amount of computation they need to perform grows with the context length. While
variations of efficient transformers have been proposed, they all have a finite
memory capacity and are forced to drop old information. In this paper, we
propose the -former, which extends the vanilla transformer with an
unbounded long-term memory. By making use of a continuous-space attention
mechanism to attend over the long-term memory, the -former's attention
complexity becomes independent of the context length, trading off memory length
with precision. In order to control where precision is more important,
-former maintains "sticky memories" being able to model arbitrarily
long contexts while keeping the computation budget fixed. Experiments on a
synthetic sorting task, language modeling, and document grounded dialogue
generation demonstrate the -former's ability to retain information from
long sequences.
Sparse Sequence-to-Sequence Models
12 Jun 2019
Sequence-to-sequence models are a powerful workhorse of NLP. Most variants employ a softmax transformation in both their attention mechanism and output layer, leading to dense alignments and strictly positive output probabilities. This density is wasteful, making models less interpretable and assigning probability mass to many implausible outputs. In this paper, we propose sparse sequence-to-sequence models, rooted in a new family of -entmax transformations, which includes softmax and sparsemax as particular cases, and is sparse for any . We provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. Our models are able to produce sparse alignments and to assign nonzero probability to a short list of plausible outputs, sometimes rendering beam search exact. Experiments on morphological inflection and machine translation reveal consistent gains over dense models.
Should We Still Pretrain Encoders with Masked Language Modeling?
04 Jul 2025
This paper presents a controlled study comparing Masked Language Modeling (MLM) and Causal Language Modeling (CLM) as pretraining objectives for text encoders across various model sizes and a fixed data budget. The research finds that while MLM generally yields stronger representations for downstream tasks, a two-stage CLM then MLM approach or continuing pretraining with MLM from a CLM checkpoint delivers optimal performance and improved fine-tuning stability.
View blog
MSTS: A Multimodal Safety Test Suite for Vision-Language Models
17 Jan 2025
Vision-language models (VLMs), which process image and text inputs, are increasingly integrated into chat assistants and other consumer AI applications. Without proper safeguards, however, VLMs may give harmful advice (e.g. how to self-harm) or encourage unsafe behaviours (e.g. to consume drugs). Despite these clear hazards, little work so far has evaluated VLM safety and the novel risks created by multimodal inputs. To address this gap, we introduce MSTS, a Multimodal Safety Test Suite for VLMs. MSTS comprises 400 test prompts across 40 fine-grained hazard categories. Each test prompt consists of a text and an image that only in combination reveal their full unsafe meaning. With MSTS, we find clear safety issues in several open VLMs. We also find some VLMs to be safe by accident, meaning that they are safe because they fail to understand even simple test prompts. We translate MSTS into ten languages, showing non-English prompts to increase the rate of unsafe model responses. We also show models to be safer when tested with text only rather than multimodal prompts. Finally, we explore the automation of VLM safety assessments, finding even the best safety classifiers to be lacking.
QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation
15 Oct 2024
An important challenge in machine translation (MT) is to generate
high-quality and diverse translations. Prior work has shown that the estimated
likelihood from the MT model correlates poorly with translation quality. In
contrast, quality evaluation metrics (such as COMET or BLEURT) exhibit high
correlations with human judgments, which has motivated their use as rerankers
(such as quality-aware and minimum Bayes risk decoding). However, relying on a
single translation with high estimated quality increases the chances of "gaming
the metric''. In this paper, we address the problem of sampling a set of
high-quality and diverse translations. We provide a simple and effective way to
avoid over-reliance on noisy quality estimates by using them as the energy
function of a Gibbs distribution. Instead of looking for a mode in the
distribution, we generate multiple samples from high-density areas through the
Metropolis-Hastings algorithm, a simple Markov chain Monte Carlo approach. The
results show that our proposed method leads to high-quality and diverse outputs
across multiple language pairs (English{German, Russian}) with
two strong decoder-only LLMs (Alma-7b, Tower-7b).
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
TOWER is an open multilingual large language model family developed by Unbabel and Instituto de Telecomunicações, designed to excel across various translation-related tasks. It was created using a multi-stage training recipe, achieving high translation quality that surpasses other open models and often rivals closed systems like GPT-3.5 and GPT-4 on benchmarks.
View blog
Differentiable Weightless Neural Networks
The University of Texas at AustinWe introduce the Differentiable Weightless Neural Network (DWN), a model
based on interconnected lookup tables. Training of DWNs is enabled by a novel
Extended Finite Difference technique for approximate differentiation of binary
values. We propose Learnable Mapping, Learnable Reduction, and Spectral
Regularization to further improve the accuracy and efficiency of these models.
We evaluate DWNs in three edge computing contexts: (1) an FPGA-based hardware
accelerator, where they demonstrate superior latency, throughput, energy
efficiency, and model area compared to state-of-the-art solutions, (2) a
low-power microcontroller, where they achieve preferable accuracy to XGBoost
while subject to stringent memory constraints, and (3) ultra-low-cost chips,
where they consistently outperform small models in both accuracy and projected
hardware area. DWNs also compare favorably against leading approaches for
tabular datasets, with higher average rank. Overall, our work positions DWNs as
a pioneering solution for edge-compatible high-throughput neural networks.
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions
25 May 2025
Emmy LiuGraham Neubig
Improvements in language model capabilities are often attributed to
increasing model size or training data, but in some cases smaller models
trained on curated data or with different architectural decisions can
outperform larger ones trained on more tokens. What accounts for this? To
quantify the impact of these design choices, we meta-analyze 92 open-source
pretrained models across a wide array of scales, including state-of-the-art
open-weights models as well as less performant models and those with less
conventional design decisions. We find that by incorporating features besides
model size and number of training tokens, we can achieve a relative 3-28%
increase in ability to predict downstream performance compared with using scale
alone. Analysis of model design decisions reveal insights into data
composition, such as the trade-off between language and code tasks at 15-25\%
code, as well as the better performance of some architectural decisions such as
choosing rotary over learned embeddings. Broadly, our framework lays a
foundation for more systematic investigation of how model development choices
shape final capabilities.
Teaching Language Models to Faithfully Express their Uncertainty
University of AmsterdamLarge language models (LLMs) often miscommunicate their uncertainty: repeated queries can produce divergent answers, yet generated responses are typically unhedged or hedged in ways that do not reflect this variability. This conveys unfaithful information about the uncertain state of the LLMs' knowledge, creating a faithfulness gap that affects even strong LLMs. We introduce Faithful Uncertainty Tuning (FUT): a fine-tuning approach that teaches instruction-tuned LLMs to express uncertainty faithfully without altering their underlying answer distribution. We construct training data by augmenting model samples with uncertainty hedges (i.e. verbal cues such as 'possibly' or 'likely') aligned with sample consistency, requiring no supervision beyond the model and a set of prompts. We evaluate FUT on open-domain question answering (QA) across multiple models and datasets. Our results show that FUT substantially reduces the faithfulness gap, while preserving QA accuracy and introducing minimal semantic distribution shift. Further analyses demonstrate robustness across decoding strategies, choice of hedgers, and other forms of uncertainty expression (i.e. numerical). These findings establish FUT as a simple and effective way to teach LLMs to communicate uncertainty faithfully.
Conformal Prediction for Natural Language Processing: A Survey
03 May 2024
This survey details the growing application of Conformal Prediction (CP) for uncertainty quantification in Natural Language Processing (NLP), demonstrating how it provides reliable uncertainty estimates with statistical guarantees. It shows CP successfully applied across diverse NLP tasks, including text classification and natural language generation, enhancing model reliability and supporting human-AI collaboration.
View blog
There are no more papers matching your filters at the moment.