fermi: migrate from LATDataQuery.cgi scraping to the new REST API by PazSheimy · Pull Request #3647 · astropy/astroquery

PazSheimy · 2026-08-18T13:13:35Z

Background

The Fermi LAT data server is replacing the LATDataQuery.cgi form endpoint
with a JSON REST API. Both will run in parallel for some time, but the REST
API will eventually replace LATDataQuery.cgi entirely, so this PR migrates
the module in advance for astroquery users.

Base URL: https://fermi.gsfc.nasa.gov/ssc/data/access/lat/query/api/v1
(docs). I work on the
Fermi data-server side and wrote this migration against the new API.

What changed

query_object_async() POSTs a JSON payload to {base}/query and returns
the server-assigned query_id. Previously it returned the URL of an HTML
results page
_parse_result() polls {base}/query/{id}/status and reads
{base}/query/{id}/results, so the synchronous
query_object() keeps its exact previous signature and return type
(a list of FITS file URLs) — existing user scripts are unaffected.
All regex/HTML scraping is removed, along with the module's direct
requests usage (everything now goes through BaseQuery._request).
New public methods: get_status(), list_results(), and get_file_urls()
(which accepts an optional max_wait to bound the wait).
New query options exposed by the module: zenithangle,
coordsystem (J2000/B1950/Galactic - previously only J2000), and all-sky queries
(radius > 60 deg with an observation window <= 24 h); literal "RA,Dec"
strings bypass name resolution.
Failed queries now raise RemoteServiceError;
a query that never completes can be bounded with max_wait
(raises TimeoutError).
Rejected queries surface the server's own error message: an HTTP error with
a JSON body like {"error": "Invalid energy range ..."} is raised as a
RemoteServiceError carrying that message (closes the old TODO from Fail with useful failure messages on genuine failures. #1849).
GetFermilatDatafile / get_fermilat_datafile kept as deprecated shims
delegating to get_file_urls().
Removed the "Experimental" import-time warning; the module now follows the
standard astroquery async/sync pattern.
Docs (docs/fermi/fermi.rst) and changelog updated.

Testing

Unit tests rewritten against mocked JSON responses (submit, status
running/complete/failed, results), including payload-shape tests, error
paths, and a forward-compatibility test for absolute file URLs:
15 pass offline.
Remote tests (--remote-data) run against the live API on 2026-08-14:
4/4 pass (basic query, async + status, zenith angle, all-sky).

codecov · 2026-08-18T13:20:20Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.87%. Comparing base (e0a0f4c) to head (f60bdc8).
⚠️ Report is 83 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3647      +/-   ##
==========================================
+ Coverage   73.45%   73.87%   +0.41%     
==========================================
  Files         230      230              
  Lines       21424    21639     +215     
==========================================
+ Hits        15738    15986     +248     
+ Misses       5686     5653      -33

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

PazSheimy · 2026-09-04T18:03:31Z

Note on CI: the devdeps job failure is unrelated to this PR

PazSheimy · 2026-09-21T12:54:28Z

Friendly ping @bsipocz — this PR is ready for review whenever someone has time. Since opening it I've synced with current main and all CI checks pass. I work on the Fermi data-server side, I'm happy to answer any questions about the API itself, and glad to adjust anything about the module's approach.

bsipocz · 2026-09-21T13:12:31Z

@PazSheimy - yes, it's somewhere at the top of my review queue, but there are a few other big PRs that preceed it.

bsipocz

Thanks for the PR!

I have some API related comments, mostly about aiming for API consistency with the other modules. I would think at least some of them need to be resolved before we merge this.

bsipocz · 2026-09-22T13:13:36Z

           ]
-
-import warnings
-warnings.warn("Experimental: Fermi-LAT has not yet been refactored to have "


bsipocz · 2026-09-22T13:42:04Z

+    # ------------------------------------------------------------------
+    # submission
+    # ------------------------------------------------------------------
    def query_object_async(self, *args, **kwargs):


have we make args and kwargs explicit? Only list the required positional argument and accepted keyword arguments in the signature and not have a signature that swallows everything

Also, this is a good spot to say we are moving away from the duplicated method and method_async approach, so I would say a substantial refactor like this PR can do that work and cleanup the API to only keep the non async methods around.

Happy to do this cleanup
I'll make query_object() an explicitly defined method with the full signature and move submission/polling to private helpers.

One question: should query_object_async be removed outright, or kept for one release as a deprecated shim? (Worth noting the old query_object_async returned an HTML results-page URL that won't exist once the CGI retires so pre-existing callers break either way; the shim would just make it a deprecation warning instead of an AttributeError.)

bsipocz · 2026-09-22T13:43:52Z

+_COORD_PAIR_RE = re.compile(r'^\s*[-+]?\d+(\.\d*)?\s*,\s*[-+]?\d+(\.\d*)?\s*$')
+

 @async_to_sync


you can remove this

Done. @async_to_sync is removed; query_object() is now an explicitly-defined method and submission/polling are private helpers. I kept query_object_async as a deprecated shim for now, matching the PR's existing deprecation pattern — happy to remove it outright instead if you prefer, it's a two-line change.

bsipocz · 2026-09-22T13:45:16Z

+
+        Notes
+        -----
+        Prior to the REST API migration this method returned the URL of an


No need to keep any mention of the old API here. If you want to you can keep some note boxes in the narrative documentation though and formal deprecations in the code itself.

Done — removed.

bsipocz · 2026-09-22T13:46:54Z

+        _raise_for_status(response, context=f"Fermi LAT results ({query_id})")
+        return response.json().get('files', [])
+
+    def wait_for_completion(self, query_id, *, check_frequency=None,


does this have to be user facing/public API?

Made it private (_wait_for_completion) — users still control polling via the check_frequency/max_wait keywords on the public methods.

bsipocz · 2026-09-22T13:55:56Z

+            time.sleep(check_frequency * 60)
+            elapsed_time += check_frequency
+
+    def get_file_urls(self, query_id, *, check_frequency=None, max_wait=None,


The method name here may need some coordination -- there are a couple other PRs in the works that deal with datalink urls and uris, but getting a conclusion on that should not be a blocker for this PR, it's just a heads up

Understood
happy to rename to match whatever convention comes out of the datalink discussions, here or in a follow-up PR.

bsipocz · 2026-09-22T14:28:27Z

-        return _fermi_format_coords(c)
-    except (u.UnitsError, TypeError):
-        raise Exception("Coordinates not specified correctly")
+def _raise_for_status(response, *, context):


everywhere else we just use response.raise_for_status(), I would do that here, too rather than override it.

Done — error detection now goes through response.raise_for_status(); the helper only catches the HTTPError and re-raises as RemoteServiceError with the server's JSON error message attached (that message is the #1849 fix, so I kept it). Happy to simplify to a bare raise_for_status() if you'd prefer.

bsipocz · 2026-09-22T14:33:34Z


-    def __call__(self, result_url, *, check_frequency=1, verbose=False):
-        self.result_url = result_url
+def _fermi_format_coords(c, *, coordsystem='J2000'):


do we really need this as a separate method, or could be merged into the previous one?

Merged into _parse_coordinates.

fermi: fix docstring markup for _parse_args reference Co-authored-by: Brigitta Sipőcz <b.sipocz@gmail.com>

Sheimy Paz Serpa added 7 commits July 17, 2026 16:08

fermi: migrate from LATDataQuery.cgi scraping to the new REST API

303277a

fermi: use full API URL in changelog

db18460

fermi: surface server error messages on failed queries

0a6f696

fermi: add changelog entry for error-message handling

46104aa

fermi: send search radius as 'shapefield', the name the API accepts

3b82c1c

fermi: clarify that LATDataQuery.cgi is still running in parallel

1be1bd5

fermi: add PR number to changelog entries

949a520

PazSheimy mentioned this pull request Aug 18, 2026

fermi: LATDataQuery.cgi is being replaced by a REST API — module update available #3646

Open

bsipocz added the fermi label Aug 18, 2026

Sheimy Paz Serpa added 2 commits September 4, 2026 11:41

fermi: cover error paths, polling loop, and deprecated shims in tests

6a740ef

Merge upstream main into fermi-rest-api

79e18ea

PazSheimy marked this pull request as ready for review September 4, 2026 18:02

Merge remote-tracking branch 'upstream/main' into fermi-rest-api

b6de4e6

bsipocz reviewed Sep 22, 2026

View reviewed changes

bsipocz added this to the 0.4.13 milestone Sep 22, 2026

PazSheimy and others added 7 commits September 25, 2026 12:01

Update astroquery/fermi/core.py

49265d5

fermi: fix docstring markup for _parse_args reference Co-authored-by: Brigitta Sipőcz <b.sipocz@gmail.com>

fermi: drop old-API historical note from docstring

7997142

fermi: make wait_for_completion private

d27643a

fermi: fix continuation-line indentation after rename

eb387f9

fermi: merge coordinate formatting into _parse_coordinates

f6f7f44

fermi: defer error detection to response.raise_for_status()

03a9e58

fermi: single query_object entry point with explicit signature

f60bdc8

		_COORD_PAIR_RE = re.compile(r'^\s[-+]?\d+(\.\d)?\s,\s[-+]?\d+(\.\d)?\s$')


		@async_to_sync

Uh oh!

Conversation

PazSheimy commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

What changed

Testing

Uh oh!

codecov Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

PazSheimy commented Sep 4, 2026

Uh oh!

PazSheimy commented Sep 21, 2026

Uh oh!

bsipocz commented Sep 21, 2026

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PazSheimy Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PazSheimy commented Aug 18, 2026 •

edited

Loading

codecov Bot commented Aug 18, 2026 •

edited

Loading

PazSheimy Sep 25, 2026 •

edited

Loading