Automated Geolocalization Techniques for Astronaut Photography of Earth

Astronauts on the International Space Station take thousands of photos each month, which are used for disaster management, climate change studies, and other earth science research. However, before a photo can be used, it must be localized: this was historically done manually, in a task that NASA defines as "monumentally important, but monumentally time-consuming job". This project provides an automatic localization pipeline, based on a large-scale image retrieval model (EarthLoc - CVPR 2024) and a coregistration method (EarthMatch - CVPRW 2024).

EarthLoc: Astronaut Photography Localization by Indexing Earth from Space

1Politecnico di Torino, 2Jacobs Technology, NASA Johnson Space Center

EarthMatch: Iterative Coregistration for Fine-grained Localization of Astronaut Photography

1Politecnico di Torino, 2Jacobs Technology, NASA Johnson Space Center

From photo to geographic position

Each astronaut photo to be localized is passed through the trained EarthLoc model, to get a feature embedding. We take the top N nearest database embeddings as tentative location candidates. The database covers a huge area - the 2M square km visible at that time from the ISS. Then, the matching step takes the astronaut photo query and a candidate, decides if the two images overlap and, if they do, accurately coregister the photo.

Database and Training Set

In order to determine positions via image retrieval, it is necessary to have a global database of geotagged images. We create such a database using a publicly available compilation of Sentinel-2 satellite imagery, which offers cloudless coverage of landmasses worldwide with resolutions of up to 15 meters per pixel. As shown in the animation, images are sampled at different resolutions and are rotated 4 times, to simplify the retrieval task. These multi-scale and rotated images form our retreival database. For training, images are also sampled from different years, which makes the model more robust to natural variations in Earth features. For further details, see Section 3 of the EarthLoc paper.

Fine-grained Matching via Iteration

Once we have a location candidate from EarthLoc-based retreival, we apply the EarthMatch procedure. We begin with matching the astronaut photo query $Q$ (left) and the candidate $C_0$ (right) to produce a first homography $H_0$. Since applying this homography directly to $C_0$ may produce uncovered regions, if the query and candidate do not well overlap, we generate the candidate surroundings $C_s$ and apply $H_0$ to this larger area to produce $C_1$. $C_1$ is then matched with $Q$, producing $H_2$ which applied to a new $C_s$ yields $C_2$. This iterative process continues for a fixed number of iterations or until convergence. In general, the number of inlier matches increases steadily for each iteration. Use the slider to observe the different matching iterations. For more info, see Section 3.2 of the EarthMatch paper.. Below are four visualizations of a query and how the candidate changes across iterations while applying the computed homography.


GIF 1
GIF 2
GIF 3
GIF 4

Interactive qualitative results (try it yourself!)

Query
$C_0$
$C_s$
Comparison
Matching steps
Loading...

BibTeX

@InProceedings{Berton_CVPR_2024_EarthLoc,
  author    = {Berton, Gabriele and Stoken, Alex and Caputo, Barbara and Masone, Carlo},
  title     = {EarthLoc: Astronaut Photography Localization by Indexing Earth from Space},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
}
@InProceedings{Berton_2024_EarthMatch,
  author    = {Gabriele Berton, Gabriele Goletto, Gabriele Trivigno, Alex Stoken, Barbara Caputo, Carlo Masone},
  title     = {EarthMatch: Iterative Coregistration for Fine-grained Localization of Astronaut Photography},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month     = {June},
  year      = {2024},
}

Acknowledgements

Logo