Near-final updates

[iannevans]

These updates are mostly those from issue #28, but include some changes from issue #29 and feedback from Pierre. There are additional changes because the TeX file that I downloaded from the repo included more changes than the PDF Preview against which I wrote issue #28. I have incorporated review and updates across these updates too for consistency. I made some significant changes to the UCDs section that I thought about deep and hard, and I will explain these and my reasoning in a separate note (probably tomorrow).

One issue I hit is that I had to comment out \makeglossaries ... \printglossaries because the document will not build for me with these in place. TeX fails with
(./HighEnergyObsCoreExt.gls
! Extra }, or forgotten \endgroup.
@endpbox ...finalstrut @arstrutbox \par \egroup
\ST@dimen =\ht \ST@pbox \a...
l.5 ...[]{page}\glsnumberformat{13}}}\glsgroupskip
however, I can't find an extra "}" in the .gls file so this appears to be a problem that will require more investigation to resolve.

[bkhelifi]

Suggested change

\gls{HEA} data include observations obtained using photon detectors covering X-ray (from $\sim$0.1 keV to $\sim$120 keV) through gamma-ray (from 120 keV up to $\gtrsim$ PeV) energies, as well as cosmic-ray and astrophysical neutrino ($\gtrsim$ TeV) detectors, or other messengers related to \gls{HEA} phenomena. The domain is now sufficiently mature to provide open data that are science-ready and work with open analysis tools ({\em e.g.\/}, CIAO \citep{2006SPIE.6270E..1VF} or Gammapy \citep{gammapy:2023}). The science output of the \gls{HEA} domain already includes high-level products such as images, cubes, spectra, and time series such as light curves and time-resolved spectra. Additional data products include fitted sky models with a spatial, spectral, and/or temporal component(s), along with their confidence intervals or confidence limits, and covariance matrices. Finally, multiple \gls{HEA} instruments produce source catalogs and surveys covering up to the full the sky, which include maps of photon or particle flux, exposure, sensitivity, and aperture-photometry likelihood profiles.

\gls{HEA} data include observations obtained using photon detectors covering X-ray (from $\sim$0.1 keV to $\sim$120 keV) through gamma-ray (from 120 keV up to $\gtrsim$ PeV) energies, as well as cosmic-ray and astrophysical neutrino ($\gtrsim$ TeV) detectors, or other messengers related to \gls{HEA} phenomena. The domain is now sufficiently mature to provide open data that are science-ready and work with open analysis tools ({\em e.g.\/}, CIAO \citep{2006SPIE.6270E..1VF} or Gammapy \citep{gammapy:2023}). The science output of the \gls{HEA} domain already includes advanced products such as images, cubes, spectra, and time series such as light curves and time-resolved spectra. Additional data products include fitted sky models with a spatial, spectral, and/or temporal component(s), along with their confidence intervals or confidence limits, and covariance matrices. Finally, multiple \gls{HEA} instruments produce source catalogs and surveys covering up to the full the sky, which include maps of photon or particle flux, exposure, sensitivity, and aperture-photometry likelihood profiles.

[bkhelifi]

As proposed by Mireille, let's try to use the term advanced products instead of high-level products (an event list is already a high-level product).

[bkhelifi]

Suggested change

In addition to {\em dataproduct\_type\/} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of ``advanced data products'' (typically with {\em calib\_level\/} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations) means that data providers may choose to provide such analysis products directly to the end user. For example, the Chandra Source Catalog includes 38 types of advanced data products (for a total of $\sim\!90$ million files) and $\sim\!50$\% of these data product types are not well represented by a {\em dataproduct\_type\/} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated progenitor observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type\/} (or {\em dataproduct\_subtype\/}) terms for these advanced data products, and note that these terms will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets):

In addition to {\em dataproduct\_type\/} terms that focus on event data, we note that existing ObsCore definitions do not adequately span the breadth of ``advanced data products'' (typically with {\em calib\_level\/} $\ge$ 3) that may be generated from astronomical observations by users or observatories. The computational complexity of analyzing \gls{HEA} data robustly in the extreme Poisson regime ({\em e.g.\/}, Bayesian X-ray aperture photometry applied simultaneously to multiple overlapping detections and observations, Frequentist adjustment of a model of electron populations on multi-wavelength data spanning from X-rays to PeV gamma rays) means that data providers may choose to provide such analysis products directly to the end user. For example, the Chandra Source Catalog includes 38 types of advanced data products (for a total of $\sim\!90$ million files) and $\sim\!50$\% of these data product types are not well represented by a {\em dataproduct\_type\/} value that allows for meaningful data discovery. Users will certainly want to discover these data products independently from the associated progenitor observation data (and many of these data products combine data from multiple observations). We therefore propose the following additional {\em dataproduct\_type\/} (or {\em dataproduct\_subtype\/}) terms for these advanced data products, and note that these terms will certainly be useful independent of waveband ({\em i.e.\/}, they can be equally applicable to UV/optical, IR, and radio datasets):

[bkhelifi]

I have added a VHE example, such that anyone can be convinced that the need of these products is from the whole HEA group.

[bkhelifi]

Suggested change

{\bf pdf}: a dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable. The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution.

{\bf pdf}: a dataset that records the probability density function of a quantity, for example the Bayesian marginal probability density function for a random variable or the DeltaTS associated to a quantity from a Frequentist analysis . The probability density function provides a robust estimation of the variable and allows arbitrary confidence intervals to be computed directly from the distribution.

[bkhelifi]

Suggested change

The {\bf measurements} {\em dataproduct\_type\/} is quite useful for many different types of advanced data products (that may be derived from multiple observations). But users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, measurements associated with sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = ``measurements'' in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''.

The {\bf measurements} {\em dataproduct\_type\/} is quite useful for many different types of advanced data products (that may be derived from multiple observations). But users of those products often may not be interested the progenitor datasets, especially if many advanced data products are extracted from a single or a few progenitors ({\em e.g.\/}, measurements associated with sources detected in a single observation field). We propose to delete the caveat associated with {\em dataproduct\_type\/} = ``measurements'' in the ObsCore IVOA Recommendation (\S~4.1.1) that requires the derived data products be exposed ``{\bf together} with the progenitor observation dataset''. The recovery of the progenitor observation dataset(s) can be achieved with the Provenance information.

[bkhelifi]

Suggested change

The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify more precisely the scientific nature of a data product. Although no vocabulary is defined for {\em dataproduct\_subtype\/}, we recommend that data providers formulate and use a standardized vocabulary for this attribute for data products that are commonly used in \gls{HEA}\null. We have proposed several terms in \S~5 for commonly used \gls{HEA} {\bf response-function} types ({\em e.g.\/}, {\bf aeff}, {\bf edisp}, {\bf psf}), but additional terms could be standardized for other common data products. For example, standardizing using {\bf exposuremap} for an exposure map would enable queries such as ({\em dataproduct\_type\/} = {\bf image}) AND ({\em dataproduct\_subtype\/} = {\bf exposuremap}) to work across multiple facilities. Other possible terms could include {\bf significancemap} for a significance map, and {\bf probabilitymap} for aprobability map.

The optional attribute {\em dataproduct\_subtype} may be used by the data provider to specify more precisely the scientific nature of a data product. Although no vocabulary is defined for {\em dataproduct\_subtype\/}, we recommend that data providers formulate and use a standardized vocabulary for this attribute for data products that are commonly used in \gls{HEA}\null. We have proposed several terms in \S~5 for commonly used \gls{HEA} {\bf response-function} types ({\em e.g.\/}, {\bf aeff}, {\bf edisp}, {\bf psf}, {\bf bkgrate}), but additional terms could be standardized for other common data products. For example, standardizing using {\bf exposuremap} for an exposure map would enable queries such as ({\em dataproduct\_type\/} = {\bf image}) AND ({\em dataproduct\_subtype\/} = {\bf exposuremap}) to work across multiple facilities. Other possible terms could include {\bf significancemap} for a significance map, {\bf probabilitymap} for a probability map, and {\bf exclusionmap} for the exclusion maps used to adjust TeV background models.

[bkhelifi]

Hello Ian,

Thank you @iannevans for this rich PR, containing a lot of great editing corrections and text improvements.

I have merged the PR of @loumir. So, the PDF generation should be fixed now. Let's see whether it works well once your PR will be merged.

Thanks for the update of the glossary. As you have noticed, glossary handling is not yet part of the standard CI of the IVOA latex files. I had to add it 'manually', as you have seem. So, thanks for the update of the gls files. To make it appears into the document: 1/ make 2/ makeglossaries HighEnergyObsCoreExt 3/ make.
@loumir or @mservillat, could you make a PR in order to make it append (modification of the Make and in the library to download by the CI)?

Here are my comments for the LateX files after a carefull last reading:

Core document:

• I have made minor inline suggestions in your PR (see above my current github comment). This is easier than long comments. You can yourself accept them (if you're OK with them).
• About the draws, I have a suggestion (to take or to ignore) in order to convince even more people that they can be used by other domains. One could add (e.g., in a footnote) a standard use case from cosmology: within the standard $\Lambda\textrm{CDM}$ cosmological model, some are using the intersection of confidence contours to have an estimation of a quantity. E.g., from https://arxiv.org/pdf/1801.00598, confidence contours of the dimensionless cosmological density parameters $\Omega_{\textrm{M}}$ and $\Omega_{\Lambda}$ derived from the Hubble diagram of quasars are superposed with the ones from the Type Ia supernovae sample in order to get an estimate of these cosmological quantities. These contours are draws. - I like this example because currently these confidence levels can not be published easily, and because an example from cosmology can convince people-
• For the region, I am really not sure that people will be convinced by this definition and its justification. Let's see...
• UCDs:
  • What is the dimention [M], the mass? If yes, I do not understand how it can be applied to, e.g., phot.flux.energy.density.sb that deals with photons... The same for phot.flux.energy, phot.flux.energy.density, phot.flux.energy.density.sb and phot.flux.energy.sb
  • You have proposed that the phot.count."fluxes" are for both counts and particles. There is a amsll issue here: how will do the cosmic rays experiments to publish counts fluxes and particles fluxes?? They can not! So, I have proposed inline corrections to remove the "or particles" in the current definitions. And, can you add entries for particles of all types of "fluxes" with the prefix phys.particle.?
  • Where are the photon fluxes in [L$^{-2}$ T$^{-1}$ E$^{-1}$], [L$^{-2}$ T$^{-1}$ E$^{-1}$ sr$^{-1}$], [E T$^{-1}$ L$^{-2}$], [E T$^{-1}$], etc?
  • One might have a quick Zoom call together to discuss that

Use Cases (I have focused my reading on the TeV use cases)

• Use Case A.1.3: given our analysis schemes and field of views, I propose to update the angular values of the query. It should be:

  (i) s_region position intersects with the source modeled by a circle of 1.5
  deg radius around (312.775, 30.683),"
  and
  NATURAL JOIN ivoa.obscore_hea
  WHERE (INTERSECTS(s_region, CIRCLE(312.775, 30.683, 1.5)) = 6)

  A tiny correction from a TeV analysis expert (that I missed before);)

• Use Case A.1.4: CTAO will bundle our products into a DataLink for the next Science Data Challenge. So, the dataproduct_type must be event-bundle. Could you make this change?

• Use Case A.1.6:

  • Like for the 1.3, the TeV expert that I am proposes to update the angle of query (minor change from 1 to 2 deg) as well as the angular size of CasA (5armin diameter) to:

  (target_name = "Cas A" OR
  CONTAINS(POINT(s_ra, s_dec), CIRCLE, 350.8584, +58.8113, 0.042) = 2)

  • after the different updates of the UCs, there is a kind of similarities between the UC A.1.6 and Use Case A.1.4. 1.6 is a search on a specific object, 1.4 is a blind spatial search with a query on >10TeV (ie, for experts, wuth data produced with the Small Size Telescopes of CTAO dedicated to the very/ultra high energy observations). So, for a reader, the UC 1.6 is a more classical search than the 1.4. I would propose just to reverse the appearance order: 1.6 in the place of 1.4, and vice-versa.

• Use case A.1.9: I think there is a difference between its title/aims and the query. We aim here to only get ALL IRFs, but the query return will return the event-list. So, one should change the query to:

  dataproduct_type = "event-bundle" or dataproduct_type = "aeff" or dat-
  aproduct_type = "psf" or dataproduct_type = "edisp" or dataprod-
  uct_type = "bkgrate",

  Could you make this change? I missed that before... my bad

[iannevans]

Hi Bruno,

I will incorporate your suggestions and resubmit this as a new PR. I suggest that the current PR gets rejected since it has not yet been merged.

A few comments.

Your inline suggestions seem OK, but I noticed a couple of grammar errors in those same paragraphs in what I originally updated, so will correct those too. Re: the examples of responses: I think that "({\em e.g./}, {\bf aeff}, {\bf edisp}, {\bf psf})" is sufficient (it's 3 out of 6) without adding bkgrate, especially since we reference section 5. Alternatively, we could drop the "e.g.," and just add all 6 in parentheses but using "e.g.," means that we can updated the list in section 5 subsequently without having to update this section also.

I'll update the text re draws to add the Lambda CDM example.

I'm not worried about region. There is no existing dataproduct_type that describes a data product that incorporates region definitions (MOC is an encoding, not a data product).

I think you might be misinterpreting the phot ({\em e.g./}, {\bf aeff}, {\bf edisp}, {\bf psf}) tree. phot is "photometry" not "photon". So phot.energy... are UCDs that describe energy flux (e.g., erg cm^-2 s^-1) and associated quantities. So M T^-3 is the correct dimensionality for energy flux. This highlighted an error in section 5.1.6, where energy flux (and associated radiance, flux density) is listed a W m^-2 s^-1. That should be J m^-2 s^-1. I'll fix that too.

I can add in particle flux and associated quantities similarly as phot.flux.particle... and this is a natural and consistent extension. phys.particle.flux is not the appropriate place for these. The phys UCD tree describes physical properties but what we are discussing here are photometric quantities. That highlights another issue that we are missing - we need to add phys.particle.photon !!! You would then define a counts flux (counts/cm^2/s) using "phot.count" (I know this one is inconsistent, but that's what happens when a standard accretes updates instead of being thought through at one time), a photon flux (photons/cm^2/s) would be "phot.flux.particle; phys.particle.photon", and an energy flux (erg/cm^2/s) would be "phot.flux.energy". I'll also correct the descriptions of the counts... UCDs to say just counts and not counts or particles.

I also don't want to add dimensionality to existing (basic) "phot.flux" and associated derived quantities because that has been "in the wild" for so long that there are probably a lot of uses out there that would not match any dimensionality that we suggest.

I'll update the use cases.

I'm going to leave \makeglossaries, \printglossaries commented out for now until ivoatex gets updated because the file won't pass the checks run by GitHub otherwise.

Thans,
--Ian

[bkhelifi]

Hello,
As you prefer, Ian. I will then wait for the new PR and we will ignore the current one.

About UCDs:

• phot is "photometry" not "photon: ah ok!!!! my bad... It makes sense then. A last question: what is the M dimension? a meter, a distance?
• "I can add in particle flux and associated quantities similarly as phot.flux.particle": +1
• well seen for phys.particle.photon
• " \makeglossaries, \printglossaries commented out": sure, no worry. This is an editing minor detail.

Thank you very much again :)

[iannevans]

what is the M dimension? a meter, a distance?

Mass. Using SI units, energy flux is J m^-2 s^-1 = N m m^-2 s^-1 = kg m s^-2 m m^-2 s^-1 = kg s^-3 so dimensionality = M T^-3.