Ask or search anything...
Long-Context Generalization with Sparse Attention
27 Sep 2025
Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many settings, this density has been shown to be detrimental for tasks that demand precise focus on fixed-size patterns: as sequence length increases, non-informative tokens accumulate attention probability mass, leading to dispersion and representational collapse. We show in this paper that dynamically sparse attention mechanisms using -entmax can avoid these issues, due to their ability to assign exact zeros to irrelevant tokens. Furthermore, we introduce Adaptive-Scalable Entmax (ASEntmax), which endows -entmax with a learnable temperature parameter, allowing the attention distribution to interpolate between sparse (pattern-focused) and dense (softmax-like) regimes. Our empirical evaluation on synthetic tasks and language modeling demonstrates that ASEntmax substantially outperforms softmax, scalable softmax, and fixed-temperature -entmax baselines, achieving up to 1000 length extrapolation on synthetic benchmarks and superior long-context generalization on language modeling while preserving short-context performance, including better perplexity trends and higher retrieval accuracies at 8 training length.
Plasmonic Brownian Ratchets for Directed Transport of Analytes
19 Sep 2025
Monash University
Imperial College LondonControlled long-range transport of micro- and nano-scale objects is a key requirement in lab-on-a-chip and microfluidic applications, enabling the efficient capture, concentration, manipulation, and detection of analytes. Traditional methods such as microfluidic pumps and optical trapping face challenges including high power consumption and limited range of action. This study introduces a plasmonic Brownian ratchet designed for the directed transport of dielectric nanometer-sized particles at low optical powers. Through numerical simulations, the ratchet geometry was optimized to enhance electric fields, optical forces, and trapping potentials. Experimentally, the plasmonic ratchet demonstrated the ability to rectify random thermal motion of 40 nm polysterene spheres over extended distances in a specific direction, achieving velocities up to 2.4 m/s at excitation powers as low as 0.785 kW/cm^2. This plasmonic ratchet offers a robust and efficient solution for the targeted delivery and concentration of nanoscale analytes on chips, with significant implications for advancing applications in the life sciences.
Cross-Chain Arbitrage: The Next Frontier of MEV in Decentralized Finance
18 Jun 2025
Researchers from Flashbots and the Technical University of Munich conducted the first large-scale empirical study of executed cross-chain arbitrage, quantifying its 8.65 million net profit across nine blockchains. Their analysis revealed a highly concentrated market where inventory-based strategies dominate due to bridging latency, raising concerns about vertical integration and systemic risks in decentralized finance.
View blog
Estimating Object Physical Properties from RGB-D Vision and Depth Robot Sensors Using Deep Learning
07 Jul 2025
Inertial mass plays a crucial role in robotic applications such as object grasping, manipulation, and simulation, providing a strong prior for planning and control. Accurately estimating an object's mass before interaction can significantly enhance the performance of various robotic tasks. However, mass estimation using only vision sensors is a relatively underexplored area. This paper proposes a novel approach combining sparse point-cloud data from depth images with RGB images to estimate the mass of objects. We evaluate a range of point-cloud processing architectures, alongside RGB-only methods. To overcome the limited availability of training data, we create a synthetic dataset using ShapeNetSem 3D models, simulating RGBD images via a Kinect camera. This synthetic data is used to train an image generation model for estimating dense depth maps, which we then use to augment an existing dataset of images paired with mass values. Our approach significantly outperforms existing benchmarks across all evaluated metrics. The data generation (this https URL) as well as the training of the depth estimator (this https URL) and the mass estimator (this https URL) are available online.
Multilingual Large Language Models: A Systematic Survey
19 Nov 2024
A systematic survey by researchers from Tianjin University and the University of Lisbon presents a comprehensive overview of multilingual large language models, categorizing research into six domains while detailing architectures, data strategies, tuning methodologies, and evaluation. The work synthesizes current progress and highlights critical challenges in achieving equitable performance across languages and ensuring responsible AI.
View blog
Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification
28 Jan 2025
Kyle Thompson
Formal verification using proof assistants, such as Coq, enables the creation
of high-quality software. However, the verification process requires
significant expertise and manual effort to write proofs. Recent work has
explored automating proof synthesis using machine learning and large language
models (LLMs). This work has shown that identifying relevant premises, such as
lemmas and definitions, can aid synthesis. We present Rango, a fully automated
proof synthesis tool for Coq that automatically identifies relevant premises
and also similar proofs from the current project and uses them during
synthesis. Rango uses retrieval augmentation at every step of the proof to
automatically determine which proofs and premises to include in the context of
its fine-tuned LLM. In this way, Rango adapts to the project and to the
evolving state of the proof. We create a new dataset, CoqStoq, of 2,226
open-source Coq projects and 196,929 theorems from GitHub, which includes both
training data and a curated evaluation benchmark of well-maintained projects.
On this benchmark, Rango synthesizes proofs for 32.0% of the theorems, which is
29% more theorems than the prior state-of-the-art tool Tactician. Our
evaluation also shows that Rango adding relevant proofs to its context leads to
a 47% increase in the number of theorems proven.
Large Language Models for Captioning and Retrieving Remote Sensing Images
09 Feb 2024
Image captioning and cross-modal retrieval are examples of tasks that involve the joint analysis of visual and linguistic information. In connection to remote sensing imagery, these tasks can help non-expert users in extracting relevant Earth observation information for a variety of applications. Still, despite some previous efforts, the development and application of vision and language models to the remote sensing domain have been hindered by the relatively small size of the available datasets and models used in previous studies. In this work, we propose RS-CapRet, a Vision and Language method for remote sensing tasks, in particular image captioning and text-image retrieval. We specifically propose to use a highly capable large decoder language model together with image encoders adapted to remote sensing imagery through contrastive language-image pre-training. To bridge together the image encoder and language decoder, we propose training simple linear layers with examples from combining different remote sensing image captioning datasets, keeping the other parameters frozen. RS-CapRet can then generate descriptions for remote sensing images and retrieve images from textual descriptions, achieving SOTA or competitive performance with existing methods. Qualitative results illustrate that RS-CapRet can effectively leverage the pre-trained large language model to describe remote sensing images, retrieve them based on different types of queries, and also show the ability to process interleaved sequences of images and text in a dialogue manner.
Hallucinations in Large Multilingual Translation Models
28 Mar 2023
Large-scale multilingual machine translation systems have demonstrated
remarkable ability to translate directly between numerous languages, making
them increasingly appealing for real-world applications. However, when deployed
in the wild, these models may generate hallucinated translations which have the
potential to severely undermine user trust and raise safety concerns. Existing
research on hallucinations has primarily focused on small bilingual models
trained on high-resource languages, leaving a gap in our understanding of
hallucinations in massively multilingual models across diverse translation
scenarios. In this work, we fill this gap by conducting a comprehensive
analysis on both the M2M family of conventional neural machine translation
models and ChatGPT, a general-purpose large language model~(LLM) that can be
prompted for translation. Our investigation covers a broad spectrum of
conditions, spanning over 100 translation directions across various resource
levels and going beyond English-centric language pairs. We provide key insights
regarding the prevalence, properties, and mitigation of hallucinations, paving
the way towards more responsible and reliable machine translation systems.
The Express Lane to Spam and Centralization: An Empirical Analysis of Arbitrum's Timeboost
26 Sep 2025
DeFi applications are vulnerable to MEV, where specialized actors profit by reordering or inserting transactions. To mitigate latency races and internalize MEV revenue, Arbitrum introduced Timeboost, an auction-based transaction sequencing mechanism that grants short-term priority access to an express lane. In this paper we present the first large-scale empirical study of Timeboost, analyzing over 11.5 million express lane transactions and 151 thousand auctions between April and July 2025. Our results reveal five main findings. First, express lane control is highly centralized, with two entities winning more than 90% of auctions. Second, while express lane access provides earlier inclusion, profitable MEV opportunities cluster at the end of blocks, limiting the value of priority access. Third, approximately 22% of time-boosted transactions are reverted, indicating that the Timeboost does not effectively mitigate spam. Fourth, secondary markets for reselling express lane rights have collapsed due to poor execution reliability and unsustainable economics. Finally, auction competition declined over time, leading to steadily reduced revenue for the Arbitrum DAO. Taken together, these findings show that Timeboost fails to deliver on its stated goals of fairness, decentralization, and spam reduction. Instead, it reinforces centralization and narrows adoption, highlighting the limitations of auction-based ordering as a mechanism for fair transaction sequencing in rollups.
On DESI's DR2 exclusion of CDM
30 Oct 2025
The DESI collaboration, combining their Baryon Acoustic Oscillation (BAO) data with cosmic microwave background (CMB) anisotropy and supernovae data, have found significant indication against the CDM cosmology. This can also be interpreted as the significance of the detection of the parameter that measures variation of the dark energy equation of state. DESI's DR2 article quotes exclusion of CDM for combinations of BAO and CMB data with each of three different and overlapping supernovae compilations (at 2.8-sigma for Pantheon+, 3.8-sigma for Union3, and 4.2-sigma for DESY5). We show that one can neither choose amongst nor average over these three different significances. We demonstrate how a principled statistical combination yields a combined exclusion significance of 3.1-sigma. Further we argue that, faced with these competing significances, the most secure inference from the DESI DR2 results is the 3.1-sigma level exclusion of CDM obtained from combining DESI+CMB alone, omitting supernovae.
FLORA: Efficient Synthetic Data Generation for Object Detection in Low-Data Regimes via finetuning Flux LoRA
29 Aug 2025
FLORA introduces an efficient, two-stage pipeline for generating synthetic data for object detection in low-data environments, leveraging Flux LoRA for object-centric inpainting. The method achieves superior object detection performance and image fidelity across multiple benchmarks using significantly less synthetic data and computational resources than prior state-of-the-art approaches.
View blog
XChainDataGen: A Cross-Chain Dataset Generation Framework
17 Mar 2025
The number of blockchain interoperability protocols for transferring data and
assets between blockchains has grown significantly. However, no open dataset of
cross-chain transactions exists to study interoperability protocols in
operation. There is also no tool to generate such datasets and make them
available to the community. This paper proposes XChainDataGen, a tool to
extract cross-chain data from blockchains and generate datasets of cross-chain
transactions (cctxs). Using XChainDataGen, we extracted over 35 GB of data from
five cross-chain protocols deployed on 11 blockchains in the last seven months
of 2024, identifying 11,285,753 cctxs that moved over 28 billion USD in
cross-chain token transfers. Using the data collected, we compare protocols and
provide insights into their security, cost, and performance trade-offs. As
examples, we highlight differences between protocols that require full finality
on the source blockchain and those that only demand soft finality
(\textit{security}). We compare user costs, fee models, and the impact of
variables such as the Ethereum gas price on protocol fees (\textit{cost}).
Finally, we produce the first analysis of the implications of EIP-7683 for
cross-chain intents, which are increasingly popular and greatly improve the
speed with which cctxs are processed (\textit{performance}), thereby enhancing
the user experience. The availability of XChainDataGen and this dataset allows
various analyses, including trends in cross-chain activity, security
assessments of interoperability protocols, and financial research on
decentralized finance (DeFi) protocols.
LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting
31 May 2023
University of CopenhagenLMCAP is a few-shot, retrieval-augmented language model prompting approach for multilingual image captioning that operates without task-specific training. It achieves competitive performance against fully supervised models on human-labeled multilingual datasets, notably outperforming them on Chinese captions.
View blog
Leveraging LLMs for On-the-Fly Instruction Guided Image Editing
04 Dec 2024
The combination of language processing and image processing keeps attracting
increased interest given recent impressive advances that leverage the combined
strengths of both domains of research. Among these advances, the task of
editing an image on the basis solely of a natural language instruction stands
out as a most challenging endeavour. While recent approaches for this task
resort, in one way or other, to some form of preliminary preparation, training
or fine-tuning, this paper explores a novel approach: We propose a
preparation-free method that permits instruction-guided image editing on the
fly. This approach is organized along three steps properly orchestrated that
resort to image captioning and DDIM inversion, followed by obtaining the edit
direction embedding, followed by image editing proper. While dispensing with
preliminary preparation, our approach demonstrates to be effective and
competitive, outperforming recent, state of the art models for this task when
evaluated on the MAGICBRUSH dataset.
Pegasus Simulator: An Isaac Sim Framework for Multiple Aerial Vehicles Simulation
Carnegie Mellon UniversityDeveloping and testing novel control and motion planning algorithms for aerial vehicles can be a challenging task, with the robotics community relying more than ever on 3D simulation technologies to evaluate the performance of new algorithms in a variety of conditions and environments. In this work, we introduce the Pegasus Simulator, a modular framework implemented as an NVIDIA Isaac Sim extension that enables real-time simulation of multiple multirotor vehicles in photo-realistic environments, while providing out-of-the-box integration with the widely adopted PX4-Autopilot and ROS2 through its modular implementation and intuitive graphical user interface. To demonstrate some of its capabilities, a nonlinear controller was implemented and simulation results for two drones performing aggressive flight maneuvers are presented. Code and documentation for this framework are also provided as supplementary material.
DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 generate correct code for LoRaWAN-related engineering tasks
07 Apr 2025
A study evaluated 16 Large Language Models (LLMs) for generating correct Python code specific to LoRaWAN engineering tasks, such as optimal UAV placement and received power calculation. It found that lightweight, locally executable models like Phi-4 and LLaMA-3.3 generated highly accurate code, demonstrating performance comparable to larger cloud-based models like DeepSeek-V3.
View blog
Euclid preparation. Sensitivity to neutrino parameters
09 May 2024
Arthur Loureiro
University of Toronto
University of AmsterdamThe Euclid mission of the European Space Agency will deliver weak
gravitational lensing and galaxy clustering surveys that can be used to
constrain the standard cosmological model and extensions thereof. We present
forecasts from the combination of these surveys on the sensitivity to
cosmological parameters including the summed neutrino mass and the
effective number of relativistic species in the standard
CDM scenario and in a scenario with dynamical dark energy ($w_0
w_a$CDM). We compare the accuracy of different algorithms predicting the
nonlinear matter power spectrum for such models. We then validate several
pipelines for Fisher matrix and MCMC forecasts, using different theory codes,
algorithms for numerical derivatives, and assumptions concerning the non-linear
cut-off scale. The Euclid primary probes alone will reach a sensitivity of
56meV in the CDM+ model, whereas the
combination with CMB data from Planck is expected to achieve
23meV and raise the evidence for a non-zero neutrino mass to at
least the level. This can be pushed to a detection if
future CMB data from LiteBIRD and CMB Stage-IV are included. In combination
with Planck, Euclid will also deliver tight constraints on $\Delta N_{\rm eff}<
0.144\LambdaM_\nuN_{\rm eff}\Delta
N_{\rm eff}< 0.063(w_0,
w_a)N_{\rm eff}$ remains stable, while that
to degrades at most by a factor 2. This work illustrates the
complementarity between the Euclid spectroscopic and imaging/photometric
surveys and between Euclid and CMB constraints. Euclid will have a great
potential for measuring the neutrino mass and excluding well-motivated
scenarios with additional relativistic particles.
State preparation by shallow circuits using feed forward
04 Dec 2024
In order to achieve fault-tolerant quantum computation, we need to repeat the
following sequence of four steps: First, perform 1 or 2 qubit quantum gates (in
parallel if possible). Second, do a syndrome measurement on a subset of the
qubits. Third, perform a fast classical computation to establish which errors
have occurred (if any). Fourth, depending on the errors, we apply a correction
step. Then the procedure repeats with the next sequence of gates. In order for
these four steps to succeed, we need the error rate of the gates to be below a
certain threshold. Unfortunately, the error rates of current quantum hardware
are still too high. On the other hand, current quantum hardware platforms are
designed with these four steps in mind. In this work we make use of this
four-step scheme not to carry out fault-tolerant computations, but to enhance
short, constant-depth, quantum circuits that perform 1 qubit gates and
nearest-neighbor 2 qubit gates. To explore how this can be useful, we study a
computational model which we call Local Alternating Quantum Classical
Computations (LAQCC). In this model, qubits are placed in a grid allowing
nearest neighbor interactions; the quantum circuits are of constant depth with
intermediate measurements; a classical controller can perform log-depth
computations on these intermediate measurement outcomes to control future
quantum operations. This model fits naturally between quantum algorithms in the
NISQ era and full fledged fault-tolerant quantum computation. We show that
LAQCC circuits can create long-ranged interactions, which constant-depth
quantum circuits cannot achieve, and use it to construct a range of useful
multi-qubit gates. With these gates, we create three new state preparation
protocols for a uniform superposition over an arbitrary number of states,
W-states, Dicke states and may-body scar states.
Why Is Anything Conscious?
15 Apr 2025
Michael Timothy Bennett
Australian National UniversityWe tackle the hard problem of consciousness taking the naturally selected,
embodied organism as our starting point. We provide a formalism describing how
biological systems self-organise to hierarchically interpret unlabelled sensory
information according to valence. Such interpretations imply behavioural
policies which are differentiated from each other only by the qualitative
aspect of information processing. Natural selection favours systems that
intervene in the world to achieve homeostatic and reproductive goals. Quality
is a property arising in such systems to link cause to affect to motivate
interventions. This produces interoceptive and exteroceptive classifiers and
determines priorities. In formalising the seminal distinction between access
and phenomenal consciousness, we claim that access consciousness at the human
level requires the ability to hierarchically model i) the self, ii) the
world/others and iii) the self as modelled by others, and that this requires
phenomenal consciousness. Phenomenal without access consciousness is likely
common, but the reverse is implausible. To put it provocatively: death grounds
meaning, and Nature does not like zombies. We then describe the multilayered
architecture of self-organisation from rocks to Einstein, illustrating how our
argument applies. Our proposal lays the foundation of a formal science of
consciousness, closer to human fact than zombie fiction.
Meta-prompting Optimized Retrieval-augmented Generation
04 Jul 2024
Retrieval-augmented generation resorts to content retrieved from external sources in order to leverage the performance of large language models in downstream tasks. The excessive volume of retrieved content, the possible dispersion of its parts, or their out of focus range may happen nevertheless to eventually have a detrimental rather than an incremental effect. To mitigate this issue and improve retrieval-augmented generation, we propose a method to refine the retrieved content before it is included in the prompt by resorting to meta-prompting optimization. Put to empirical test with the demanding multi-hop question answering task from the StrategyQA dataset, the evaluation results indicate that this method outperforms a similar retrieval-augmented system but without this method by over 30%.
There are no more papers matching your filters at the moment.