Astronauts on the International Space Station take thousands of photos each month, which are used for disaster management, climate change studies, and other earth science research. However, before a photo can be used, it must be localized: this was historically done manually, in a task that NASA defines as "monumentally important, but monumentally time-consuming job". This project provides an automatic localization pipeline, based on a large-scale image retrieval model (EarthLoc - CVPR 2024) and a coregistration method (EarthMatch - CVPRW 2024).
Once we have a location candidate from EarthLoc-based retreival, we apply the EarthMatch procedure. We begin with matching the astronaut photo query $Q$ (left) and the candidate $C_0$ (right) to produce a first homography $H_0$. Since applying this homography directly to $C_0$ may produce uncovered regions, if the query and candidate do not well overlap, we generate the candidate surroundings $C_s$ and apply $H_0$ to this larger area to produce $C_1$. $C_1$ is then matched with $Q$, producing $H_2$ which applied to a new $C_s$ yields $C_2$. This iterative process continues for a fixed number of iterations or until convergence. In general, the number of inlier matches increases steadily for each iteration. Use the slider to observe the different matching iterations. For more info, see Section 3.2 of the EarthMatch paper.. Below are four visualizations of a query and how the candidate changes across iterations while applying the computed homography.
@InProceedings{Berton_CVPR_2024_EarthLoc,
author = {Berton, Gabriele and Stoken, Alex and Caputo, Barbara and Masone, Carlo},
title = {EarthLoc: Astronaut Photography Localization by Indexing Earth from Space},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
}
@InProceedings{Berton_2024_EarthMatch,
author = {Gabriele Berton, Gabriele Goletto, Gabriele Trivigno, Alex Stoken, Barbara Caputo, Carlo Masone},
title = {EarthMatch: Iterative Coregistration for Fine-grained Localization of Astronaut Photography},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2024},
}