Removing soft and self shadows that lack clear boundaries from a single image is still challenging. Self shadows are shadows that are cast on the object itself. Most existing methods rely on binary shadow masks, without considering the ambiguous boundaries of soft and self shadows. In this paper, we present DeS3, a method that removes hard, soft and self shadows based on adaptive attention and ViT similarity. Our novel ViT similarity loss utilizes features extracted from a pre-trained Vision Transformer. This loss helps guide the reverse sampling towards recovering scene structures. Our adaptive attention is able to differentiate shadow regions from the underlying objects, as well as shadow regions from the object casting the shadow. This capability enables DeS3 to better recover the structures of objects even when they are partially occluded by shadows. Different from existing methods that rely on constraints during the training phase, we incorporate the ViT similarity during the sampling stage. Our method outperforms state-of-the-art methods on the SRD, AISTD, LRSS, USR and UIUC datasets, removing hard, soft, and self shadows robustly. Specifically, our method outperforms the SOTA method by 16\% of the RMSE of the whole image on the LRSS dataset. Our data and code is available at: \url{https://github.com/jinyeying/DeS3_Deshadow}

Night images suffer not only from low light, but also from uneven distributions of light. Most existing night visibility enhancement methods focus mainly on enhancing low-light regions. This inevitably leads to over enhancement and saturation in bright regions, such as those regions affected by light effects (glare, floodlight, etc). To address this problem, we need to suppress the light effects in bright regions while, at the same time, boosting the intensity of dark regions. With this idea in mind, we introduce an unsupervised method that integrates a layer decomposition network and a light-effects suppression network. Given a single night image as input, our decomposition network learns to decompose shading, reflectance and light-effects layers, guided by unsupervised layer-specific prior losses. Our light-effects suppression network further suppresses the light effects and, at the same time, enhances the illumination in dark regions. This light-effects suppression network exploits the estimated light-effects layer as the guidance to focus on the light-effects regions. To recover the background details and reduce hallucination/artefacts, we propose structure and high-frequency consistency losses. Our quantitative and qualitative evaluations on real images show that our method outperforms state-of-the-art methods in suppressing night light effects and boosting the intensity of dark regions.

Raindrops adhered to a glass window or camera lens can severely hamper the visibility of a background scene and degrade an image considerably. In this paper, we address the problem by visually removing raindrops, and thus transforming a raindrop degraded image into a clean one. The problem is intractable, since first the regions occluded by raindrops are not given. Second, the information about the background scene of the occluded regions is completely lost for most part. To resolve the problem, we apply an attentive generative network using adversarial training. Our main idea is to inject visual attention into both the generative and discriminative networks. During the training, our visual attention learns about raindrop regions and their surroundings. Hence, by injecting this information, the generative network will pay more attention to the raindrop regions and the surrounding structures, and the discriminative network will be able to assess the local consistency of the restored regions. This injection of visual attention to both generative and discriminative networks is the main contribution of this paper. Our experiments show the effectiveness of our approach, which outperforms the state of the art methods quantitatively and qualitatively.

Given a target function $H$ to minimize or a target Gibbs distribution $\pi_{\beta}^0 \propto e^{-\beta H}$ to sample from in the low temperature, in this paper we propose and analyze Langevin Monte Carlo (LMC) algorithms that run on an alternative landscape as specified by $H^f_{\beta,c,1}$ and target a modified Gibbs distribution $\pi^f_{\beta,c,1} \propto e^{-\beta H^f_{\beta,c,1}}$, where the landscape of$H^f_{\beta,c,1}$ is a transformed version of that of $H$ which depends on the parameters $f,\beta$ and $c$. While the original Log-Sobolev constant affiliated with $\pi^0_{\beta}$ exhibits exponential dependence on both $\beta$ and the energy barrier $M$ in the low temperature regime, with appropriate tuning of these parameters and subject to assumptions on $H$, we prove that the energy barrier of the transformed landscape is reduced which consequently leads to polynomial dependence on both $\beta$ and $M$ in the modified Log-Sobolev constant associated with $\pi^f_{\beta,c,1}$. This yield improved total variation mixing time bounds and improved convergence toward a global minimum of $H$. We stress that the technique developed in this paper is not only limited to LMC and is broadly applicable to other gradient-based optimization or sampling algorithms.

We introduce a new class of attacks on machine learning models. We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak significant private details of training points belonging to other parties. Our active inference attacks connect two independent lines of work targeting the integrity and privacy of machine learning training data. Our attacks are effective across membership inference, attribute inference, and data extraction. For example, our targeted attacks can poison <0.1% of the training dataset to boost the performance of inference attacks by 1 to 2 orders of magnitude. Further, an adversary who controls a significant fraction of the training data (e.g., 50%) can launch untargeted attacks that enable 8x more precise inference on all other users' otherwise-private data points. Our results cast doubts on the relevance of cryptographic privacy guarantees in multiparty computation protocols for machine learning, if parties can arbitrarily select their share of training data.

We investigate the cutoff phenomenon for Markov processes under information divergences such as $f$-divergences and R\'enyi divergences. We classify most common divergences into four types, namely $L^2$-type, $\mathrm{TV}$-type, separation-type and $\mathrm{KL}$ divergence, in which we prove that the cutoff phenomenon are equivalent and relate the cutoff time and window among members within each type. To justify that this classification is natural, we provide examples in which the family of Markov processes exhibit cutoff in one type but not in another. We also establish new product conditions in these settings for the processes to exhibit cutoff, along with new results in non-reversible or non-normal situations. The proofs rely on a functional analytic approach towards cutoff.

Discovering new superionic materials is essential for advancing solid-state batteries, which offer improved energy density and safety compared to the traditional lithium-ion batteries with liquid electrolytes. Conventional computational methods for identifying such materials are resource-intensive and not easily scalable. Recently, universal interatomic potential models have been developed using equivariant graph neural networks. These models are trained on extensive datasets of first-principles force and energy calculations. One can achieve significant computational advantages by leveraging them as the foundation for traditional methods of assessing the ionic conductivity, such as molecular dynamics or nudged elastic band techniques. However, the generalization error from model inference on diverse atomic structures arising in such calculations can compromise the reliability of the results. In this work, we propose an approach for the quick and reliable screening of ionic conductors through the analysis of a universal interatomic potential. Our method incorporates a set of heuristic structure descriptors that effectively employ the rich knowledge of the underlying model while requiring minimal generalization capabilities. Using our descriptors, we rank lithium-containing materials in the Materials Project database according to their expected ionic conductivity. Eight out of the ten highest-ranked materials are confirmed to be superionic at room temperature in first-principles calculations. Notably, our method achieves a speed-up factor of approximately 50 compared to molecular dynamics driven by a machine-learning potential, and is at least 3,000 times faster compared to first-principles molecular dynamics.

In dense foggy scenes, existing optical flow methods are erroneous. This is due to the degradation caused by dense fog particles that break the optical flow basic assumptions such as brightness and gradient constancy. To address the problem, we introduce a semi-supervised deep learning technique that employs real fog images without optical flow ground-truths in the training process. Our network integrates the domain transformation and optical flow networks in one framework. Initially, given a pair of synthetic fog images, its corresponding clean images and optical flow ground-truths, in one training batch we train our network in a supervised manner. Subsequently, given a pair of real fog images and a pair of clean images that are not corresponding to each other (unpaired), in the next training batch, we train our network in an unsupervised manner. We then alternate the training of synthetic and real data iteratively. We use real data without ground-truths, since to have ground-truths in such conditions is intractable, and also to avoid the overfitting problem of synthetic data training, where the knowledge learned on synthetic data cannot be generalized to real data testing. Together with the network architecture design, we propose a new training strategy that combines supervised synthetic-data training and unsupervised real-data training. Experimental results show that our method is effective and outperforms the state-of-the-art methods in estimating optical flow in dense foggy scenes.

We argue that models coming from a variety of fields, such as matching models and discrete choice models among others, share a common structure that we call matching function equilibria with partial assignment. This structure includes an aggregate matching function and a system of nonlinear equations. We provide a proof of existence and uniqueness of an equilibrium and propose an efficient algorithm to compute it. For a subclass of matching models, we also develop a new parameter-free approach for constructing the counterfactual matching equilibrium. It has the advantage of not requiring parametric estimation when computing counterfactuals. We use our procedure to analyze the impact of the elimination of the Social Security Student Benefit Program in 1982 on the marriage market in the United States. We estimate several candidate models from our general class of matching functions and select the best fitting model using information based criterion.

In this paper, we address a rain removal problem from a single image, even in the presence of heavy rain and rain streak accumulation. Our core ideas lie in the new rain image models and a novel deep learning architecture. We first modify an existing model comprising a rain streak layer and a background layer, by adding a binary map that locates rain streak regions. Second, we create a new model consisting of a component representing rain streak accumulation (where individual streaks cannot be seen, and thus visually similar to mist or fog), and another component representing various shapes and directions of overlapping rain streaks, which usually happen in heavy rain. Based on the first model, we develop a multi-task deep learning architecture that learns the binary rain streak map, the appearance of rain streaks, and the clean background, which is our ultimate output. The additional binary map is critically beneficial, since its loss function can provide additional strong information to the network. To handle rain streak accumulation (again, a phenomenon visually similar to mist or fog) and various shapes and directions of overlapping rain streaks, we propose a recurrent rain detection and removal network that removes rain streaks and clears up the rain accumulation iteratively and progressively. In each recurrence of our method, a new contextualized dilated network is developed to exploit regional contextual information and outputs better representation for rain detection. The evaluation on real images, particularly on heavy rain, shows the effectiveness of our novel models and architecture, outperforming the state-of-the-art methods significantly. Our codes and data sets will be publicly available.

Estimating the reflectance layer from a single image is a challenging task. It becomes more challenging when the input image contains shadows or specular highlights, which often render an inaccurate estimate of the reflectance layer. Therefore, we propose a two-stage learning method, including reflectance guidance and a Shadow/Specular-Aware (S-Aware) network to tackle the problem. In the first stage, an initial reflectance layer free from shadows and specularities is obtained with the constraint of novel losses that are guided by prior-based shadow-free and specular-free images. To further enforce the reflectance layer to be independent of shadows and specularities in the second-stage refinement, we introduce an S-Aware network that distinguishes the reflectance image from the input image. Our network employs a classifier to categorize shadow/shadow-free, specular/specular-free classes, enabling the activation features to function as attention maps that focus on shadow/specular regions. Our quantitative and qualitative evaluations show that our method outperforms the state-of-the-art methods in the reflectance layer estimation that is free from shadows and specularities. Code is at: \url{https://github.com/jinyeying/S-Aware-network}.

By virtue of being atomically thin, the electronic properties of heterostructures built from two-dimensional materials are strongly influenced by atomic relaxation. The atomic layers behave as flexible membranes rather than rigid crystals. Here we develop an analytical theory of lattice relaxation in twisted moir\'e materials. We obtain analytical results for the lattice displacements and corresponding pseudo gauge fields, as a function of twist angle. We benchmark our results for twisted bilayer graphene and twisted WSe$_2$ bilayers using large-scale molecular dynamics simulations. Our \textit{single-parameter} theory is valid in graphene bilayers for twist angles $\theta ~\gtrsim 0.7^\circ$, and in twisted WSe$_2$ for $\theta ~\gtrsim 1.6^\circ$. We also investigate how relaxation alters the electronic structure in twisted bilayer graphene, providing a simple extension to the continuum model to account for lattice relaxation.

We present a high-resolution kinematic study of the massive main-sequence star-forming galaxy (SFG) SDSS J090122.37+181432.3 (J0901) at z=2.259, using 0.36 arcsec ALMA CO(3-2) and 0.1-0.5 arcsec SINFONI/VLT H-alpha observations. J0901 is a rare, strongly-lensed but otherwise normal massive (log(M_star/M_sun)~11) main sequence SFG, offering a unique opportunity to study a typical massive SFG under the microscope of lensing. Through forward dynamical modeling incorporating lensing deflection, we fit the CO and H-alpha kinematics in the image plane out to about one disk effective radius (R_e ~ 4 kpc) at a ~600pc delensed physical resolution along the kinematic major axis. Our results show high intrinsic dispersions of the cold molecular and warm ionized gas (sig0_mol ~ 40 km/s and sig0_ion ~ 66 km/s) that remain constant out to R_e; a moderately low dark matter fraction (f_DM(R_e) ~ 0.3-0.4) within R_e; and a centrally-peaked Toomre Q-parameter -- agreeing well with the previously established sig0 vs. z, f_DM vs. Sig_baryon, and Q's radial trends using large-sample non-lensed main sequence SFGs. Our data further reveal a high stellar mass concentration within ~1-2 kpc with little molecular gas, and a clumpy molecular gas ring-like structure at R ~ 2-4 kpc, in line with the inside-out quenching scenario. Our further analysis indicates that J0901 had assembled half of its stellar mass only ~400 Myrs before its observed cosmic time, and cold gas ring and dense central stellar component are consistent with signposts of a recent wet compaction event of a highly turbulent disk found in recent simulations.

Bosonic codes encode quantum information into a single infinite-dimensional physical system endowed with error correction capabilities. This reduces the need for complex management of many physical constituents compared with standard approaches employing multiple physical qubits. Recent discussions of bosonic codes centre around correcting only boson-loss errors, with phase errors either actively suppressed or deferred to subsequent layers of encoding with standard qubit codes. Rotationally symmetric bosonic (RSB) codes, which include the well-known cat and binomial codes, are capable of simultaneous correction of loss and phase errors, offering an alternate route that deals with arbitrary errors already at the base layer. Here, we investigate the robustness of such codes, moving away from the more idealistic past studies towards a circuit-level noise analysis closer to the practical situation where every physical component in the device is potentially faulty. We extend the concept of fault tolerance to the case of RSB codes, and then examine the performance of two known error correction circuits under circuit-level noise. Our analysis reveals a significantly more stringent noise threshold for fault-tolerant operation than found in past works; nevertheless, we show how, through waiting-time optimization and the use of squeezing, we can restore the noise requirements to a regime achievable with near-term quantum hardware. While our focus here is on cat codes for concreteness, a similar analysis applies for general RSB codes.

Estimating 3D human poses from a monocular video is still a challenging task. Many existing methods' performance drops when the target person is occluded by other objects, or the motion is too fast/slow relative to the scale and speed of the training data. Moreover, many of these methods are not designed or trained under severe occlusion explicitly, making their performance on handling occlusion compromised. Addressing these problems, we introduce a spatio-temporal network for robust 3D human pose estimation. As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional networks (TCNs) to estimate 3D joints or keypoints. Furthermore, we design a spatio-temporal discriminator based on body structures as well as limb motions to assess whether the predicted pose forms a valid pose and a valid movement. During training, we explicitly mask out some keypoints to simulate various occlusion cases, from minor to severe occlusion, so that our network can learn better and becomes robust to various degrees of occlusion. As there are limited 3D ground-truth data, we further utilize 2D video data to inject a semi-supervised learning capability to our network. Moreover, we observe that there is a discrepancy between 3D pose prediction and 2D pose estimation due to different pose variations between video and image training datasets. We, therefore propose a confidence-based inference stage optimization to adaptively enforce 3D pose projection to match 2D pose estimation to further improve final pose prediction accuracy. Experiments on public datasets validate the effectiveness of our method, and our ablation studies show the strengths of our network's individual submodules.

We propose a scaling approach to evaluating the qubit resources required by concatenated fault-tolerant quantum computing. The approach gives closed-form expressions, which remain simple for multiple levels of concatenation, making it an ideal tool to compare and minimize the resource costs of different concatenation schemes. We then use it to study the resources required for normal and magic operations. Here, magic operations require the preparation, verification and injection of complicated states called "magic states", while normal operations do not. It is often expected that magic operations will dominate a computation's physical resource requirements (qubits, gates, etc.), although this expectation has been cast in doubt for surface codes. Our results show that this expectation is also wrong for typical concatenated codes, with magic operations rarely being costly. We give concrete examples for the concatenated 7-qubit scheme with Steane error-correction gadgets or a flag-qubits approach. While optimizations that affect all operations are naturally more effective than ones that affect only magic operations, the surprising conclusion of our work is that the former can reduce costs by several orders of magnitude whereas the latter contributes only marginal reductions. This is particularly surprising given the numerous works on optimizations that affect only magic operations.

More than 5,000 exoplanets have been confirmed and among them almost 4,000 were discovered by the transit method. However, few transiting exoplanets have an orbital period greater than 100 days. Here we report a transit detection of Kepler-167 e, a "Jupiter analog" exoplanet orbiting a K4 star with a period of 1,071 days, using the Unistellar ground-based telescope network. From 2021 November 18 to 20, citizen astronomers located in nine different countries gathered 43 observations, covering the 16 hour long transit. Using a nested sampling approach to combine and fit the observations, we detected the mid-transit time to be UTC 2021 November 19 17:20:51 with a 1$\sigma$ uncertainty of 9.8 minutes, making it the longest-period planet to ever have its transit detected from the ground. This is the fourth transit detection of Kepler-167 e, but the first made from the ground. This timing measurement refines the orbit and keeps the ephemeris up to date without requiring space telescopes. Observations like this demonstrate the capabilities of coordinated networks of small telescopes to identify and characterize planets with long orbital periods.