FAIR Universe: Weak Lensing Uncertainty Challenge

Winners

Phase 1 of the NeurIPS 2025 Weak Lensing ML Uncertainty Challenge has come to an end. We have announced the winners of the competition.

In the final evaluation, the final submitted models are tested on a new holdout dataset that contains new realizations of convergence maps obtained from (i) the new simulations with the original cosmological parameters in the training data, and (ii) the new simulations with the cosmological parameters that were not seen in the training data. The new parameters are sampled from the same prior distributions as the training data.

After extensive studies, we have decided to award the prizes based on the final rankings in three final leaderboards. The three leaderboards evaluate the model performance on (i), (ii), and the average of (i) and (ii), respectively, to reflect the strengths and robustness of each model.

1. Final leaderboard evaluated solely on (i):

2. Final leaderboard evaluated solely on (ii):

3. Final leaderboard from the average of the score obtained on (i) and (ii):

---

All winners will present at our NeurIPS 2025 Competition Workshop:

We will prepare a paper documenting the top solutions with the winning teams after the workshop. The dataset will also be released permanently to serve as a permanent benchmark.

Congratulations to all the winning teams!

About the Challenge

The large-scale structure of the universe—the cosmic web of galaxies, galaxy clusters, and dark matter spanning hundreds of millions of light-years—encodes essential information about the composition, evolution, and fundamental laws governing the cosmos. However, the majority of matter in the universe is dark matter, which does not interact with light and can only be observed indirectly through its gravitational effects. According to Einstein’s theory of general relativity, the gravitational field of this large-scale structure bends the path of light traveling through the universe. Weak gravitational lensing refers to the subtle, coherent distortions in the observed shapes of distant galaxies caused by the deflection of light as it traverses the inhomogeneous matter distribution of the universe. By statistically analyzing these distortions across large regions of the sky, weak lensing provides a powerful probe of the matter distribution and the underlying cosmological model that governs the expansion of the universe.

Traditional analysis based on two-point correlation functions can only capture limited amount of information from the weak lensing data (2D fields similar to images). To fully exploit the non-Gaussian features present in the cosmic web, higher-order statistics and modern machine learning (ML) methods have become increasingly important. These approaches, including deep learning and simulation-based inference, have been shown to extract significant more information in weak lensing maps than traditional techniques. However, different analyses assume different dataset setups and lead to different results, making it hard to directly compare with existing approaches. Furthermore, most (if not all) of these methods rely heavily on simulations that may not accurately represent real data due to modeling approximations and missing systematics.

This competition is motivated by the need to quantify and compare the information content that different analysis methods—ranging from classical statistics to ML-based models—can extract from weak lensing maps, while also evaluating their robustness to simulation inaccuracies and observational systematics.

The outcomes of this competition are expected to guide the development of next-generation weak lensing analysis pipelines, foster cross-disciplinary collaboration between the astrophysics and machine learning communities, and ultimately improve the reliability of cosmological inference from current and upcoming surveys such as LSST, Euclid, and the Roman Space Telescope. By explicitly addressing simulation-model mismatch and the need to quantify systematic uncertainties, this competition emphasizes scientific robustness and interpretability, aligning with the growing emphasis on trustworthy ML in scientific domains.

Organizers

We gratefully acknowledge the efforts of the FAIR Universe Team, composed of researchers dedicated to advancing high energy physics, cosmology, and machine learning for the benefit of the scientific community.

For inquiries, please contact us at: fair-universe@lbl.gov

Competition Material

Find all the competition resources below:

Phase 1 Codabench Leaderboard

Last updated: November 17, 2025 at 02:05 PM UTC