Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text. Through this, we expand on the out-of-the-box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks/modalities than existing ones and doing so without a loss in performance. This enables more fine-grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model. We successfully scale the training to a three billion parameter model using tens of modalities and different datasets. The resulting models and training code are open sourced at 4m.epfl.ch.

Accurate medical image segmentation is crucial for precise anatomical delineation. Deep learning models like U-Net have shown great success but depend heavily on large datasets and struggle with domain shifts, complex structures, and limited training samples. Recent studies have explored diffusion models for segmentation by iteratively refining masks. However, these methods still retain the conventional image-to-mask mapping, making them highly sensitive to input data, which hampers stability and generalization. In contrast, we introduce DiffAtlas, a novel generative framework that models both images and masks through diffusion during training, effectively ``GenAI-fying'' atlas-based segmentation. During testing, the model is guided to generate a specific target image-mask pair, from which the corresponding mask is obtained. DiffAtlas retains the robustness of the atlas paradigm while overcoming its scalability and domain-specific limitations. Extensive experiments on CT and MRI across same-domain, cross-modality, varying-domain, and different data-scale settings using the MMWHS and TotalSegmentator datasets demonstrate that our approach outperforms existing methods, particularly in limited-data and zero-shot modality segmentation. Code is available at this https URL

Artificial intelligence (AI) is vital in ophthalmology, tackling tasks like diagnosis, classification, and visual question answering (VQA). However, existing AI models in this domain often require extensive annotation and are task-specific, limiting their clinical utility. While recent developments have brought about foundation models for ophthalmology, they are limited by the need to train separate weights for each imaging modality, preventing a comprehensive representation of multi-modal features. This highlights the need for versatile foundation models capable of handling various tasks and modalities in ophthalmology. To address this gap, we present EyeFound, a multimodal foundation model for ophthalmic images. Unlike existing models, EyeFound learns generalizable representations from unlabeled multimodal retinal images, enabling efficient model adaptation across multiple applications. Trained on 2.78 million images from 227 hospitals across 11 ophthalmic modalities, EyeFound facilitates generalist representations and diverse multimodal downstream tasks, even for detecting challenging rare diseases. It outperforms previous work RETFound in diagnosing eye diseases, predicting systemic disease incidents, and zero-shot multimodal VQA. EyeFound provides a generalizable solution to improve model performance and lessen the annotation burden on experts, facilitating widespread clinical AI applications for retinal imaging.

Reconstructing cardiac motion from cine CMR sequences is critical for diagnosis, prediction, and intervention. Existing methods rely on complete CMR stacks to infer full heart motion, limiting their utility in intra-procedural scenarios where only sparse observations are available. We present TetHeart, the first end-to-end framework that unifies full 4D multi-structure heart mesh recovery from both offline full-stack acquisitions and intra-procedural sparse-slice observations. Our method leverages deep deformable tetrahedra, an explicit-implicit hybrid representation, to capture shape and motion in a coherent space shared across cardiac structures. It is initialized from high-quality pre-procedural or offline-acquired full stacks to build detailed, patient-specific heart meshes, which can then be updated using whatever slices are available, from full stacks down to a single slice. We further incorporate several key innovations: (i) an attentive mechanism for slice-adaptive 2D-3D feature assembly that dynamically integrates information from arbitrary numbers of slices at any position, combined with a distillation strategy from full-slice to sparse-slice settings to ensure accurate reconstruction under extreme sparsity; and (ii) a two-stage weakly supervised motion learning scheme requiring only keyframe (e.g., ED and ES) annotations. Trained and validated on three large public datasets and externally evaluated zero-shot on additional private interventional and public CMR datasets, TetHeart achieves state-of-the-art accuracy and strong generalization in both pre- and intra-procedural settings.

Decades of progress in radiofrequency (RF) transistors and receiver frontends have profoundly impacted wireless communications, remote sensing, navigation, and instrumentation. Growing demands for data throughput in 6G networks, timing precision in positioning systems, and resolution in atmospheric sensing and automotive radar have pushed receiver frontends into the millimeter-wave (mmW) and sub-mmW/THz regimes. At these frequencies, however, the noise performance of field-effect transistors (FETs) degrades rapidly due to parasitic effects, limited carrier mobility, hot electrons, and shot noise. Parametric transducers that couple electromagnetic signals to optical fields offer quantum-limited sensitivity at room temperature. Electro-optic materials enable receivers that convert RF signals into optical phase shifts. While early demonstrations used resonant devices and recent efforts have focused on cryogenic microwave-to-optical quantum transduction, room-temperature electro-optic receivers have yet to achieve noise figures comparable to their electronic counterparts. Here we demonstrate a room-temperature integrated cavity electro-optic mmW receiver on a lithium tantalate (LiTaO3) photonic integrated circuit with 2.5% on-chip photon-number transduction efficiency, achieving 250 K noise temperature at 59.33 GHz--matching state-of-the-art LNAs. We report the first direct resolution of thermal noise in cavity electro-optic transduction, showing the system is fundamentally limited by thermal photon occupation (~100) in the mmW cavity. Our work establishes integrated photonics as a path to surpass electronic LNAs while offering exceptional resilience to strong electromagnetic inputs and immunity to EMI, establishing cavity electro-optics as a low-noise, chip-scale, EMI-resilient receiver frontend for mmW applications and scalable analog processing in the optical domain.

We present a method to control a text-to-image generative model to produce training data useful for supervised learning. Unlike previous works that employ an open-loop approach and pre-define prompts to generate new data using either a language model or human expertise, we develop an automated closed-loop system which involves two feedback mechanisms. The first mechanism uses feedback from a given supervised model and finds adversarial prompts that result in image generations that maximize the model loss. While these adversarial prompts result in diverse data informed by the model, they are not informed of the target distribution, which can be inefficient. Therefore, we introduce the second feedback mechanism that guides the generation process towards a certain target distribution. We call the method combining these two mechanisms Guided Adversarial Prompts. We perform our evaluations on different tasks, datasets and architectures, with different types of distribution shifts (spuriously correlated data, unseen domains) and demonstrate the efficiency of the proposed feedback mechanisms compared to open-loop approaches.

We obtain and investigate theoretically a broad family of stable and unstable time-periodic orbits-oscillating Turing rolls (OTR)-in the Lugiato-Lefever model of optical cavities. Using the dynamical systems tools developed in fluid dynamics, we access the OTR solution branches in parameter space and elucidate their bifurcation structure. By tracking these exact invariant solutions deeply into the chaotic region of the modulation instability, we connect the main dynamical regimes of the Lugiato-Lefever model: continuous waves, Turing rolls, solitons, and breathers, which completes the classical phase diagram of the optical cavity. We then demonstrate that the OTR periodic orbits play a fundamental role as elementary building blocks in the regime of the intracavity field transition from stable Turing rolls to fully developed turbulent regimes. Depending on the cavity size, we observe that the chaotic intracavity field driven by modulation instability displays either spatiotemporal or purely temporal intermittancy between chaotic dynamics and different families of the OTR solutions, exhibiting locally the distinctive wave patterns and large amplitude peaks. This opens avenues for a theoretical description of optical turbulence within the dynamical systems framework.

Locomotion robots with active or passive compliance can show robustness to uncertain scenarios, which can be promising for agricultural, research and environmental industries. However, state estimation for these robots is challenging due to the lack of rigid-body assumptions and kinematic changes from morphing. We propose a method to estimate typical rigid-body states alongside compliance-related states, such as soft robot shape in different morphologies and locomotion modes. Our neural network-based state estimator uses a history of states and a mechanism to directly influence unreliable sensors. We test our framework on the GOAT platform, a robot capable of passive compliance and active morphing for extreme outdoor terrain. The network is trained on motion capture data in a novel compliance-centric frame that accounts for morphing-related states. Our method predicts shape-related measurements within 4.2% of the robot's size, velocities within 6.3% and 2.4% of the top linear and angular speeds, respectively, and orientation within 1.5 degrees. We also demonstrate a 300% increase in travel range during a motor malfunction when using our estimator for closed-loop autonomous outdoor operation.

Among all materials, mono-crystalline diamond has one of the highest measured thermal conductivities, with values above 2000 W/m/K at room temperature. This stems from momentum-conserving `normal' phonon-phonon scattering processes dominating over momentum-dissipating `Umklapp' processes, a feature that also suggests diamond as an ideal platform to experimentally investigate phonon heat transport phenomena that violate Fourier's law. Here, we introduce dilute nitrogen-vacancy color centers as in-situ, highly precise spin defect thermometers to image temperature inhomogeneities in single-crystal diamond microstructures heated from ambient conditions. We analyze cantilevers with cross-sections in the range from about 0.2 to 2.6 $\mathrm{\mu m}^2$, observing a relation between cross-section and heat flux that departs from Fourier's law predictions. We rationalize such behavior relying on first-principles simulations based on the linearized phonon Boltzmann transport equation, also discussing how fabrication-induced impurities influence conduction. Our temperature-imaging method can be applied to diamond devices of arbitrary geometry, paving the way for the exploration of unconventional, non-diffusive heat transport phenomena.

Accurate segmentation of pulmonary airways and vessels is crucial for the diagnosis and treatment of pulmonary diseases. However, current deep learning approaches suffer from disconnectivity issues that hinder their clinical usefulness. To address this challenge, we propose a post-processing approach that leverages a data-driven method to repair the topology of disconnected pulmonary tubular structures. Our approach formulates the problem as a keypoint detection task, where a neural network is trained to predict keypoints that can bridge disconnected components. We use a training data synthesis pipeline that generates disconnected data from complete pulmonary structures. Moreover, the new Pulmonary Tree Repairing (PTR) dataset is publicly available, which comprises 800 complete 3D models of pulmonary airways, arteries, and veins, as well as the synthetic disconnected data. Our code and data are available at this https URL

Fundus fluorescein angiography (FFA) is crucial for diagnosing and monitoring retinal vascular issues but is limited by its invasive nature and restricted accessibility compared to color fundus (CF) imaging. Existing methods that convert CF images to FFA are confined to static image generation, missing the dynamic lesional changes. We introduce Fundus2Video, an autoregressive generative adversarial network (GAN) model that generates dynamic FFA videos from single CF images. Fundus2Video excels in video generation, achieving an FVD of 1497.12 and a PSNR of 11.77. Clinical experts have validated the fidelity of the generated videos. Additionally, the model's generator demonstrates remarkable downstream transferability across ten external public datasets, including blood vessel segmentation, retinal disease diagnosis, systemic disease prediction, and multimodal retrieval, showcasing impressive zero-shot and few-shot capabilities. These findings position Fundus2Video as a powerful, non-invasive alternative to FFA exams and a versatile retinal generative foundation model that captures both static and temporal retinal features, enabling the representation of complex inter-modality relationships.

Second-harmonic generation (SHG) is a fundamental tool in modern laser technology, enabling coherent frequency conversion to remote optical bands, serving as the basis for self-referenced femtosecond lasers and quadrature-squeezed light sources. State-of-the-art SHG relies on bulk crystals and ridge waveguides, although continuous-wave (CW) SH efficiency in bulk crystals is limited by short interaction lengths and large mode areas. Ridge waveguides offer better performance with lower pump power requirements, yet must span several centimeters to deliver high output power, complicating fabrication and narrowing the bandwidth. Recently, SHG in periodically poled thin-film lithium niobate integrated photonic circuits has attracted significant interest, offering orders-of-magnitude improvement in SHG under CW pumping due to the stronger optical mode confinement. However, lithium niobate has a low optical damage threshold, even in MgO-doped substrates, which limits SH power output to well below the watt level. Here, we overcome this challenge and demonstrate 7 mm-long periodically poled thin-film lithium tantalate (PPLT) waveguides that achieve high SH output in the CW regime, with generated power exceeding 1 W and off-chip output above 0.5 W at 775 nm under 4.5 W pump power. PPLT offers a higher optical damage threshold than PPLN and supports watt-level operation. By optimizing electrode geometry and poling conditions, we obtain reproducible poling despite lithium tantalate's coercive field being nearly four times higher than that of MgO-doped lithium niobate. Although its effective nonlinearity is more than five times lower, we achieve watt-level CW output with a short waveguide, demonstrating the potential of PPLT circuits for high-power applications in integrated lasers, quantum photonics, AMO physics, optical clocks, and frequency metrology.