Ask or search anything...
TopoBench: A Framework for Benchmarking Topological Deep Learning
25 Aug 2025
Paul Snopov
TopoBench introduces an open-source, modular framework designed to standardize benchmarking for Topological Deep Learning (TDL) and accelerate research in the field. Empirical evaluations using the framework demonstrate that higher-order neural networks frequently outperform traditional Graph Neural Networks on tasks benefiting from complex multi-way interactions across diverse datasets.
View blog
Long-Context Generalization with Sparse Attention
27 Sep 2025
Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many settings, this density has been shown to be detrimental for tasks that demand precise focus on fixed-size patterns: as sequence length increases, non-informative tokens accumulate attention probability mass, leading to dispersion and representational collapse. We show in this paper that dynamically sparse attention mechanisms using -entmax can avoid these issues, due to their ability to assign exact zeros to irrelevant tokens. Furthermore, we introduce Adaptive-Scalable Entmax (ASEntmax), which endows -entmax with a learnable temperature parameter, allowing the attention distribution to interpolate between sparse (pattern-focused) and dense (softmax-like) regimes. Our empirical evaluation on synthetic tasks and language modeling demonstrates that ASEntmax substantially outperforms softmax, scalable softmax, and fixed-temperature -entmax baselines, achieving up to 1000 length extrapolation on synthetic benchmarks and superior long-context generalization on language modeling while preserving short-context performance, including better perplexity trends and higher retrieval accuracies at 8 training length.
Spacetime Grand Unified Theory
26 Jul 2025
Gonçalo M. Quinta's Spacetime Grand Unified Theory proposes a first-principles derivation of the entire Standard Model from the free Dirac Lagrangian, using the Cℓ(8,0) Clifford algebra to explain the existence of three particle families via triality and to derive the Standard Model gauge group, predicting a Weinberg angle of sin²(θW) = 3/8.
View blog
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
19 Feb 2025
Beyza Ermis
Marzieh Fadaee
Shivalika Singh
KAIST
Carnegie Mellon UniversityCultural biases in multilingual datasets pose significant challenges for
their effectiveness as global benchmarks. These biases stem not only from
differences in language but also from the cultural knowledge required to
interpret questions, reducing the practical utility of translated datasets like
MMLU. Furthermore, translation often introduces artefacts that can distort the
meaning or clarity of questions in the target language. A common practice in
multilingual evaluation is to rely on machine-translated evaluation sets, but
simply translating a dataset is insufficient to address these challenges. In
this work, we trace the impact of both of these issues on multilingual
evaluations and ensuing model performances. Our large-scale evaluation of
state-of-the-art open and proprietary models illustrates that progress on MMLU
depends heavily on learning Western-centric concepts, with 28% of all questions
requiring culturally sensitive knowledge. Moreover, for questions requiring
geographic knowledge, an astounding 84.9% focus on either North American or
European regions. Rankings of model evaluations change depending on whether
they are evaluated on the full portion or the subset of questions annotated as
culturally sensitive, showing the distortion to model rankings when blindly
relying on translated MMLU. We release Global MMLU, an improved MMLU with
evaluation coverage across 42 languages -- with improved overall quality by
engaging with compensated professional and community annotators to verify
translation quality while also rigorously evaluating cultural biases present in
the original dataset. This comprehensive Global MMLU set also includes
designated subsets labeled as culturally sensitive and culturally agnostic to
allow for more holistic, complete evaluation.
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
05 Jul 2024
Zheyu Zhang
Recent multilingual pretrained language models (mPLMs) have been shown to
encode strong language-specific signals, which are not explicitly provided
during pretraining. It remains an open question whether it is feasible to
employ mPLMs to measure language similarity, and subsequently use the
similarity results to select source languages for boosting cross-lingual
transfer. To investigate this, we propose mPLMSim, a language similarity
measure that induces the similarities across languages from mPLMs using
multi-parallel corpora. Our study shows that mPLM-Sim exhibits moderately high
correlations with linguistic similarity measures, such as lexicostatistics,
genealogical language family, and geographical sprachbund. We also conduct a
case study on languages with low correlation and observe that mPLM-Sim yields
more accurate similarity results. Additionally, we find that similarity results
vary across different mPLMs and different layers within an mPLM. We further
investigate whether mPLMSim is effective for zero-shot cross-lingual transfer
by conducting experiments on both low-level syntactic tasks and high-level
semantic tasks. The experimental results demonstrate that mPLM-Sim is capable
of selecting better source languages than linguistic measures, resulting in a
1%-2% improvement in zero-shot cross-lingual transfer performance.
Linear stability analysis of hairy black holes in quadratic degenerate higher-order scalar-tensor theories: Odd-parity perturbations
23 Jul 2019
Kyoto UniversityWe study static spherically symmetric black hole solutions with a linearly time-dependent scalar field and discuss their linear stability in the shift- and reflection-symmetric subclass of quadratic degenerate higher-order scalar-tensor (DHOST) theories. We present the explicit forms of the reduced system of background field equations for a generic theory within this subclass. Using the reduced equations of motion, we show that in several cases the solution is forced to be of the Schwarzschild or Schwarzschild-(anti-)de Sitter form. We consider odd-parity perturbations around general static spherically symmetric black hole solutions, and derive the concise criteria for the black holes to be stable. Our analysis also covers the case with a static or constant profile of the scalar field.
LumberChunker: Long-Form Narrative Document Segmentation
25 Jun 2024
Arlindo Oliveira
LumberChunker introduces a novel LLM-driven method for segmenting long-form narrative documents by identifying natural semantic boundaries in text. This approach consistently outperformed traditional chunking methods in retrieval performance and led to an 88.89% accuracy in question-answering tasks within RAG pipelines, supported by the new GutenQA benchmark.
View blog
-former: Infinite Memory Transformer
25 Mar 2022
Google DeepMindTransformers are unable to model long-term memories effectively, since the
amount of computation they need to perform grows with the context length. While
variations of efficient transformers have been proposed, they all have a finite
memory capacity and are forced to drop old information. In this paper, we
propose the -former, which extends the vanilla transformer with an
unbounded long-term memory. By making use of a continuous-space attention
mechanism to attend over the long-term memory, the -former's attention
complexity becomes independent of the context length, trading off memory length
with precision. In order to control where precision is more important,
-former maintains "sticky memories" being able to model arbitrarily
long contexts while keeping the computation budget fixed. Experiments on a
synthetic sorting task, language modeling, and document grounded dialogue
generation demonstrate the -former's ability to retain information from
long sequences.
Should We Still Pretrain Encoders with Masked Language Modeling?
04 Jul 2025
This paper presents a controlled study comparing Masked Language Modeling (MLM) and Causal Language Modeling (CLM) as pretraining objectives for text encoders across various model sizes and a fixed data budget. The research finds that while MLM generally yields stronger representations for downstream tasks, a two-stage CLM then MLM approach or continuing pretraining with MLM from a CLM checkpoint delivers optimal performance and improved fine-tuning stability.
View blog
Cross-Chain Arbitrage: The Next Frontier of MEV in Decentralized Finance
18 Jun 2025
Researchers from Flashbots and the Technical University of Munich conducted the first large-scale empirical study of executed cross-chain arbitrage, quantifying its 8.65 million net profit across nine blockchains. Their analysis revealed a highly concentrated market where inventory-based strategies dominate due to bridging latency, raising concerns about vertical integration and systemic risks in decentralized finance.
View blog
SaulLM-7B: A pioneering Large Language Model for Law
07 Mar 2024
CNRSSaulLM-7B is presented as the first publicly available large language model specifically designed for the legal domain. It achieves state-of-the-art performance for 7-billion-parameter models on legal benchmarks by leveraging extensive legal data and a two-step training process, contributing an improved evaluation protocol, LegalBench-Instruct.
View blog
Local Density of States as a Probe of Multifractality in Quasiperiodic Moiré Materials
23 Oct 2025
Quasiperiodic moiré materials provide a new platform for realizing critical electronic states, yet a direct and experimentally practical method to characterize this criticality has been lacking. We show that a multifractal analysis of the local density of states (LDOS), accessible via scanning tunneling microscopy, offers an unambiguous signature of criticality from a single experimental sample. Applying this approach to a one-dimensional quasiperiodic model, a stringent test case due to its fractal energy spectrum, we find a clear distinction between the broad singularity spectra of critical states and the point-like spectra of extended states. We further demonstrate that these multifractal signatures remain robust over a wide range of energy broadenings relevant to experiments. Our results establish a model-independent, experimentally feasible framework for identifying and probing multifractality in the growing family of quasiperiodic and moiré materials.
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
TOWER is an open multilingual large language model family developed by Unbabel and Instituto de Telecomunicações, designed to excel across various translation-related tasks. It was created using a multi-stage training recipe, achieving high translation quality that surpasses other open models and often rivals closed systems like GPT-3.5 and GPT-4 on benchmarks.
View blog
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions
25 May 2025
Emmy Liu
Graham Neubig
Improvements in language model capabilities are often attributed to
increasing model size or training data, but in some cases smaller models
trained on curated data or with different architectural decisions can
outperform larger ones trained on more tokens. What accounts for this? To
quantify the impact of these design choices, we meta-analyze 92 open-source
pretrained models across a wide array of scales, including state-of-the-art
open-weights models as well as less performant models and those with less
conventional design decisions. We find that by incorporating features besides
model size and number of training tokens, we can achieve a relative 3-28%
increase in ability to predict downstream performance compared with using scale
alone. Analysis of model design decisions reveal insights into data
composition, such as the trade-off between language and code tasks at 15-25\%
code, as well as the better performance of some architectural decisions such as
choosing rotary over learned embeddings. Broadly, our framework lays a
foundation for more systematic investigation of how model development choices
shape final capabilities.
Can Relativistic Effects explain Galactic Dynamics without Dark Matter?
08 Oct 2025
This research rigorously demonstrates that General Relativistic effects, including gravitomagnetism and non-linear contributions, cannot explain observed galactic rotation curves or gravitational lensing without dark matter. The analysis shows that these relativistic effects are either geometrically inconsistent with observations or counteract the gravitational attraction required to solve the missing mass problem.
View blog
Model-Based Lookahead Reinforcement Learning for in-hand manipulation
10 Oct 2025
In-Hand Manipulation, as many other dexterous tasks, remains a difficult challenge in robotics by combining complex dynamic systems with the capability to control and manoeuvre various objects using its actuators. This work presents the application of a previously developed hybrid Reinforcement Learning (RL) Framework to In-Hand Manipulation task, verifying that it is capable of improving the performance of the task. The model combines concepts of both Model-Free and Model-Based Reinforcement Learning, by guiding a trained policy with the help of a dynamic model and value-function through trajectory evaluation, as done in Model Predictive Control. This work evaluates the performance of the model by comparing it with the policy that will be guided. To fully explore this, various tests are performed using both fully-actuated and under-actuated simulated robotic hands to manipulate different objects for a given task. The performance of the model will also be tested for generalization tests, by changing the properties of the objects in which both the policy and dynamic model were trained, such as density and size, and additionally by guiding a trained policy in a certain object to perform the same task in a different one. The results of this work show that, given a policy with high average reward and an accurate dynamic model, the hybrid framework improves the performance of in-hand manipulation tasks for most test cases, even when the object properties are changed. However, this improvement comes at the expense of increasing the computational cost, due to the complexity of trajectory evaluation.
TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
26 Jun 2021
Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predictions of a real-world bank account takeover fraud detection RNN model, and draw key insights from its explanations: i) the model identifies important features and events aligned with what fraud analysts consider cues for account takeover; ii) positive predicted sequences can be pruned to only 10% of the original length, as older events have residual attribution values; iii) the most recent input event of positive predictions only contributes on average to 41% of the model's score; iv) notably high attribution to client's age, suggesting a potential discriminatory reasoning, later confirmed as higher false positive rates for older clients.
From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?
27 Jan 2025
Large Language Models (LLMs) have found widespread applications in various
domains, including web applications, where they facilitate human interaction
via chatbots with natural language interfaces. Internally, aided by an
LLM-integration middleware such as Langchain, user prompts are translated into
SQL queries used by the LLM to provide meaningful responses to users. However,
unsanitized user prompts can lead to SQL injection attacks, potentially
compromising the security of the database. Despite the growing interest in
prompt injection vulnerabilities targeting LLMs, the specific risks of
generating SQL injection attacks through prompt injections have not been
extensively studied. In this paper, we present a comprehensive examination of
prompt-to-SQL (PSQL) injections targeting web applications based on the
Langchain framework. Using Langchain as our case study, we characterize
PSQL injections, exploring their variants and impact on application
security through multiple concrete examples. Furthermore, we evaluate 7
state-of-the-art LLMs, demonstrating the pervasiveness of PSQL attacks
across language models. Our findings indicate that LLM-integrated applications
based on Langchain are highly susceptible to PSQL injection attacks,
warranting the adoption of robust defenses. To counter these attacks, we
propose four effective defense techniques that can be integrated as extensions
to the Langchain framework. We validate the defenses through an experimental
evaluation with a real-world use case application.
Conformal Prediction for Natural Language Processing: A Survey
03 May 2024
This survey details the growing application of Conformal Prediction (CP) for uncertainty quantification in Natural Language Processing (NLP), demonstrating how it provides reliable uncertainty estimates with statistical guarantees. It shows CP successfully applied across diverse NLP tasks, including text classification and natural language generation, enhancing model reliability and supporting human-AI collaboration.
View blog
Native Logical and Hierarchical Representations with Subspace Embeddings
Traditional neural embeddings represent concepts as points, excelling at similarity but struggling with higher-level reasoning and asymmetric relationships. We introduce a novel paradigm: embedding concepts as linear subspaces. This framework inherently models generality via subspace dimensionality and hierarchy through subspace inclusion. It naturally supports set-theoretic operations like intersection (conjunction), linear sum (disjunction) and orthogonal complements (negations), aligning with classical formal semantics. To enable differentiable learning, we propose a smooth relaxation of orthogonal projection operators, allowing for the learning of both subspace orientation and dimension. Our method achieves state-of-the-art results in reconstruction and link prediction on WordNet. Furthermore, on natural language inference benchmarks, our subspace embeddings surpass bi-encoder baselines, offering an interpretable formulation of entailment that is both geometrically grounded and amenable to logical operations.
There are no more papers matching your filters at the moment.