Skip to content

Commit 989308b

Browse files
committed
Improved sagecal notebook
Improved behaviour of sum_visibility and predict_skycomponent_visibility with broadcasting magic.
1 parent a0cd5ab commit 989308b

9 files changed

Lines changed: 440 additions & 703 deletions

File tree

‎arl/calibration/sagecal.py‎

Lines changed: 40 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,22 @@
1111
1212
http://adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2011MNRAS.414.1656K&link_type=EJOURNAL
1313
14+
In this code:
15+
16+
- A single theta vector is taken to be the amplitude of a given point source and the gain (phase only) solutions for
17+
that point source for all solution point
18+
19+
20+
- The E step for a specific window is the sum of the window data model and the discrepancy between the observed data and the summed (over all windows) data models.
21+
22+
23+
- The M step for a specific window is the optimisation of the theta vector given the window data model. This involves fitting a point source (flux only for the moment) and fitting for the gain phases.
24+
25+
26+
To run sagecal, you must provide a visibility dataset and a set of skycomponents. The output will be the model
27+
parameters (source flux and gaintable for all skycomponents), and the residual visibility.
28+
29+
Sagecal should only be used once an initial phase calibration has been obtained using an isoplanatic approximation.
1430
"""
1531

1632
import logging
@@ -29,9 +45,11 @@
2945
log = logging.getLogger(__name__)
3046

3147

32-
def create_sagecal_thetas(vis, comps, **kwargs):
48+
def create_sagecal_thetas(vis: BlockVisibility, comps, **kwargs):
3349
"""Create the thetas
3450
51+
Create the data model for each window, from the visibility and the existing components
52+
3553
:param comps:
3654
:param gt:
3755
:return:
@@ -48,7 +66,9 @@ def create_sagecal_thetas(vis, comps, **kwargs):
4866
def sagecal_fit_component(evis, theta, gain=0.1, **kwargs):
4967
"""Fit a single component to a visibility i.e. A13
5068
51-
# Just do the amplitude for now
69+
This is the update to the component part of the window
70+
71+
Just do the amplitude for now
5272
5373
:param evis:
5474
:param theta:
@@ -64,6 +84,8 @@ def sagecal_fit_component(evis, theta, gain=0.1, **kwargs):
6484

6585
def sagecal_fit_gaintable(evis, theta, gain=0.1, niter=3, tol=1e-3, **kwargs):
6686
"""Fit a gaintable to a visibility i.e. A13
87+
88+
This is the update to the gain part of the window
6789
6890
:param evis:
6991
:param theta:
@@ -83,6 +105,8 @@ def sagecal_fit_gaintable(evis, theta, gain=0.1, niter=3, tol=1e-3, **kwargs):
83105

84106
def sagecal_e_step(vis: BlockVisibility, evis_all: BlockVisibility, theta, beta=1.0, **kwargs):
85107
"""Calculates E step in equation A12
108+
109+
This is the data model for this window plus the difference between observed data and summed data models
86110
87111
:param vis:
88112
:param theta:
@@ -100,6 +124,8 @@ def sagecal_e_step(vis: BlockVisibility, evis_all: BlockVisibility, theta, beta=
100124

101125
def sagecal_e_all(vis: BlockVisibility, thetas, **kwargs):
102126
"""Calculates E step in equation A12
127+
128+
This is the sum of the data models over all windows
103129
104130
:param vis:
105131
:param thetas:
@@ -116,10 +142,11 @@ def sagecal_e_all(vis: BlockVisibility, thetas, **kwargs):
116142
return evis
117143

118144

119-
def sagecal_m_step(evis: BlockVisibility, theta, gain=0.25,
120-
**kwargs):
145+
def sagecal_m_step(evis: BlockVisibility, theta, gain=0.25, **kwargs):
121146
"""Calculates M step in equation A13
122147
148+
This maximises the likelihood of the theta parameters given the existing model data
149+
123150
:param vis:
124151
:param theta:
125152
:param kwargs:
@@ -129,28 +156,16 @@ def sagecal_m_step(evis: BlockVisibility, theta, gain=0.25,
129156
sagecal_fit_gaintable(evis, theta, gain=gain, **kwargs))
130157

131158

132-
def sagecal_monitor(iter, thetas, window_index):
133-
"""Callback for monitoring
134-
135-
:param iter:
136-
:param thetas:
137-
:param window_index:
138-
:return:
139-
"""
140-
flux = thetas[window_index][0].flux[0, 0]
141-
qa = qa_gaintable(thetas[window_index][1])
142-
residual = qa.data['residual']
143-
print("\t Window %d, flux %s, residual %.3f" % (window_index, str(flux), residual))
144-
145-
146-
def sagecal_solve(vis, components, niter=10, tol=1e-8, monitor=sagecal_monitor, **kwargs):
159+
def sagecal_solve(vis, components, niter=10, tol=1e-8, gain=0.25, callback=None, **kwargs):
147160
""" Solve
148161
162+
Solve by iterating, performing E step and M step.
163+
149164
:param vis:
150165
:param components:
151166
:param gaintables:
152167
:param kwargs:
153-
:return:
168+
:return: The individual data models and the residual visibility
154169
"""
155170
thetas = create_sagecal_thetas(vis, components, **kwargs)
156171

@@ -159,10 +174,13 @@ def sagecal_solve(vis, components, niter=10, tol=1e-8, monitor=sagecal_monitor,
159174
evis_all = sagecal_e_all(vis, thetas, **kwargs)
160175
print("Iteration %d" % (iter))
161176
for window_index, theta in enumerate(thetas):
162-
evis = sagecal_e_step(vis, evis_all, theta, **kwargs)
163-
new_theta = sagecal_m_step(evis, theta, **kwargs)
177+
evis = sagecal_e_step(vis, evis_all, theta, gain=gain, **kwargs)
178+
new_theta = sagecal_m_step(evis, theta, gain=gain, **kwargs)
164179
new_thetas.append(new_theta)
165180

181+
if callback is not None:
182+
callback(iter, thetas)
183+
166184
flux = new_theta[0].flux[0, 0]
167185
qa = qa_gaintable(new_theta[1])
168186
residual = qa.data['residual']

‎arl/calibration/solvers.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,9 @@ def solve_gaintable(vis: BlockVisibility, modelvis: BlockVisibility = None, gt=N
4646
assert isinstance(modelvis, BlockVisibility), modelvis
4747

4848
if phase_only:
49-
log.info('solve_gaintable: Solving for phase only')
49+
log.debug('solve_gaintable: Solving for phase only')
5050
else:
51-
log.info('solve_gaintable: Solving for complex gain')
51+
log.debug('solve_gaintable: Solving for complex gain')
5252

5353
if gt is None:
5454
log.debug("solve_gaintable: creating new gaintable")
@@ -158,7 +158,6 @@ def solve_antenna_gains_itsubs_scalar(gain, gwt, x, xwt, niter=30, tol=1e-8, pha
158158
gain = 0.5 * (gain + gainLast)
159159
change = numpy.max(numpy.abs(gain - gainLast))
160160
if change < tol:
161-
print("solve_antenna_gains_itsubs_scalar: Converged in %d iterations" % iter)
162161
return gain, gwt, solution_residual_scalar(gain, x, xwt)
163162

164163
return gain, gwt, solution_residual_scalar(gain, x, xwt)

‎arl/image/operations.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@
1313
from astropy.wcs import FITSFixedWarning
1414
from astropy.wcs import WCS
1515
from reproject import reproject_interp
16+
from astropy.wcs.utils import skycoord_to_pixel
17+
1618

1719
from arl.data.data_models import Image, QA
1820
from arl.data.parameters import arl_path
@@ -259,12 +261,15 @@ def qa_image(im, context="") -> QA:
259261
return qa
260262

261263

262-
def show_image(im: Image, fig=None, title: str = '', pol=0, chan=0, cm='rainbow'):
264+
def show_image(im: Image, fig=None, title: str = '', pol=0, chan=0, cm='rainbow', components=None):
263265
""" Show an Image with coordinates using matplotlib
264266
265267
:param im:
266268
:param fig:
267269
:param title:
270+
:param pol: Polarisation
271+
:param chan: Channel
272+
:param components: Optional components
268273
:return:
269274
"""
270275
import matplotlib.pyplot as plt
@@ -282,6 +287,11 @@ def show_image(im: Image, fig=None, title: str = '', pol=0, chan=0, cm='rainbow'
282287
plt.ylabel('DEC--SIN')
283288
plt.title(title)
284289
plt.colorbar()
290+
291+
if components is not None:
292+
for sc in components:
293+
x, y = skycoord_to_pixel(sc.direction, im.wcs, 1, 'wcs')
294+
plt.plot(x, y, marker='+')
285295
return fig
286296

287297

‎arl/imaging/base.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -267,8 +267,7 @@ def predict_skycomponent_visibility(vis: Union[Visibility, BlockVisibility],
267267

268268
elif isinstance(vis, BlockVisibility):
269269

270-
nchan = vis.nchan
271-
npol = vis.npol
270+
ntimes, nant, _, nchan, npol = vis.vis.shape
272271

273272
k = numpy.array(vis.frequency) / constants.c.to('m/s').value
274273

@@ -280,10 +279,12 @@ def predict_skycomponent_visibility(vis: Union[Visibility, BlockVisibility],
280279
flux = convert_pol_frame(flux, comp.polarisation_frame, vis.polarisation_frame)
281280

282281
l, m, n = skycoord_to_lmn(comp.direction, vis.phasecentre)
282+
uvw = vis.uvw[..., numpy.newaxis] * k
283+
phasor = numpy.ones([ntimes, nant, nant, nchan, npol], dtype='complex')
283284
for chan in range(nchan):
284-
phasor = simulate_point(vis.uvw * k[chan], l, m)
285-
for pol in range(npol):
286-
vis.data['vis'][..., chan, pol] += flux[chan, pol] * phasor[...]
285+
phasor[:,:,:,chan,:] = simulate_point(uvw[...,chan], l, m)[...,numpy.newaxis]
286+
287+
vis.data['vis'][..., :, :] += flux[:, :] * phasor[..., :]
287288

288289
return vis
289290

‎docs/ARL_dask.rst‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,23 +24,16 @@ These are well-modeled in ARL. A example of three cycles of a major/minor cycle
2424
Conceptually we can think of one visibility set, which may be decomposed into different subsets in processing, and
2525
one image, which also may be subsetted for processing.
2626

27-
Dask.delayed is supported in various ways:
27+
Dask.delayed is used for distribution:
2828

2929
- :py:mod:`arl.graphs.graphs`: Graphs to perform various types of prediction and inversion of visibility data
3030
- :py:mod:`arl.graphs.generic_graphs`: Graphs to perform generic image or visibility unary operations
3131
- :py:mod:`arl.util.graph_support`: Graphs to support testing and simulations
3232
- :py:mod:`arl.pipelines.graphs`: Graphs to implement the canonical pipelines
3333

34-
Dask.bags is supported as follows:
35-
36-
- :py:mod:`arl.graphs.bags`: Bags to perform various types of prediction and inversion of visibility data
37-
- :py:mod:`arl.util.bag_support`: Bags to support testing and simulations
38-
- :py:mod:`arl.pipelines.bags`: Bags to implement the canonical pipelines
39-
4034
In addition there are notebooks in examples/arl.
4135

4236
- imaging-dask: General imaging example using Dask.delayed
43-
- imaging-bags: General imaging example using Dask.bag
4437
- imaging-pipelines: Pipelines run on small LOW observation (core only)
4538
- calibration-pipelines: Pipelines for self-calibration
4639

‎docs/ARL_definition.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,12 @@ Solvers
116116
.. automodule:: arl.calibration.solvers
117117
:members:
118118

119+
Sagecal
120+
+++++++
121+
122+
.. automodule:: arl.calibration.sagecal
123+
:members:
124+
119125
Fourier transform
120126
-----------------
121127

0 commit comments

Comments
 (0)