table data in an Obscore column (related to issue # 4)

[loumir]

In the current version of the note , there are many suggestions to fill ObsCore columns with multiple values , corresponding to a set of columns in the dataset content, with the implicit convention that we order the various quantities in the same way .
e.g o_ucd , o_unit, o_calib_status, etc.
I believe we should refrain from this practice because it would imply an additional parsing step to analyse the content of these columns in the TAP query.
This supposes that we actually want to expose the list of columns from the data content, in ObsCore metadata i.e as an ObsCore table column .
This is not the goal of ObsCore .
As mentioned by François Bonnarel in issue #4 , we would need an extra table to express this list of columns, a kind of "observable profile" , with the ucd , units, as described in a VOTable FIELD , etc .

My question is :
Do we need this as a primary criterium to sort out datasets of interest ?
Is it enough to access and open the data files and check the FITS header ?

[msdemlei]

On Wed, Sep 10, 2025 at 10:41:39AM -0700, Mireille LOUYS wrote: e.g _o_ucd_ , _o_unit, o_calib_status_, etc. I believe we should refrain from this practice because it would imply an additional parsing step to analyse the content of these columns in the TAP query.

Yes. Doing this would break query patterns against these columns that we have documented so far. That would be a breaking change, and we would need really strong discovery cases to go that way (and carefully look for alternatives before that).

As mentioned by François Bonnarel in issue #4 , we would need an extra table to express this list of columns, a kind of "observable profile" , with the ucd , units, as described in a VOTable FIELD , etc .

That would be one possibility that has the striking advantage that it respects the first normal form <https://en.wikipedia.org/wiki/First_normal_form>. In practice, a certain amount of denormalisation may make matters simpler for both publishers and users. One option for that is to allow "hash lists" in these columns like EPN-TAP does it. See dataproduct_type in <https://ivoa.net/documents/EPNTAP/20220822/REC-EPNTAP-2.0.html#tth_sEc2.1.2> for an example. This would let old clients work with existing data collections, and going forward we would tell people to use ivo_hashlist_has with such columns. The downside is that it is hard (impossible?) to use indexes with these query patterns. With arrays, a slightly less ad-hoc denormalisation is available; but then we'd again be breaking existing clients querying existing data collections. Another denormalisation we should at least briefly consider is whether to more or less repeat entries that have different "aspects" to them. A clear downside here is that a single dataset would then have to have multiple publisher DIDs, which may or may not be problematic. And of course, you can define extension columns carrying the extra metadata you envision. Before we go into all this intellectual effort, however, we should be certain that strong and uncontroversial use cases exist that cannot be *reasonably* covered with obscore's existing schema. With "reasonably" I mean, roughly, "doable with an amount of human inspection that astronomers are likely to perform". Do we have these? Regrettably, I don't understand enough of the HE practices to be able to form an informed opinion here. The better spelled out the discovery cases in the document for having these multiple values are, the easier it will be to figure out good solutions with minimal collateral damage. The only thing I ask is that you always keep in mind that what makes a good standard in this field is guided by the principle: "Don't think too much about what you want to say; try to think about what others want to listen to."

[iannevans]

In the current version of the note , there are many suggestions to fill ObsCore columns with multiple values , corresponding to a set of columns in the dataset content, with the implicit convention that we order the various quantities in the same way . e.g o_ucd , o_unit, o_calib_status, etc. I believe we should refrain from this practice because it would imply an additional parsing step to analyse the content of these columns in the TAP query.

While I recognize that such a change might cause additional work on the part of the clients, I don't see that as a reason for the HEIG to not propose a change that we consider is necessary to support meaningful data discovery for HEA data. I use the term "meaningful data discovery" to mean that the set of data products identified in response to the user's query accurately fulfills the user's request without including large numbers and volumes of extraneous data products that do not satisfy the requirements). Several of the proposals in the document could in fact be considered to be breaking changes, and the move from ObsCore 1 to ObsCore 2 is an appropriate time to make such changes.

This supposes that we actually want to expose the list of columns from the data content, in ObsCore metadata i.e as an ObsCore table column . This is not the goal of ObsCore .

From a pragmatic perspective, ObsCore is how data are discovered in the IVOA, and so ObsCore should support meaningful data discovery for all wavebands and datasets. Ideally I don't think we want to have to use multiple different ways to discover data. That's not "seamless". With the current ObsCore definition (I suspect stemming from the current design of the ObsCore data model), I think for HEA event data this means that in some cases it will be necessary to dig into the data content to perform meaningful data discovery.

From what I can tell, the ObsCore data model appears to be implicitly based on assumptions that any dataset includes a single observable as a function of one or more spatial/spectral/temporal/polarization axes and that this information is all that is necessary for data discovery. However, event lists aren't like that. They embed multiple observables within a single data product, and the only "axis" is often the event number (i.e., row number in the table). That is one reason that ObsCore recommends that all xxx_xel be set to -1 for event lists. In one sense, a single HEA event list might be (roughly) equivalent to multiple datasets where there is a single observable (e.g., one can extract multiple images, spectra, and light-curves from a single event lists). It doesn't make sense to record an event list broken down this way in ObsCore, however, since an event list can be filtered and binned in any number of ways, depending on the science analysis being done.

Where possible, we have tried to leverage existing ObsCore definitions to address the most common cases. As a simple example, a user who is unfamiliar with HEA data analysis may only be interested in datasets calibrated in units of f_E vs. E, whereas others may be perfectly happy with counts vs. PHA. For typical soft X-ray astronomy missions, both types of datasets conform to what is typically considered to be spectrally "calibrated" data in this waveband, so both would likely have calib_level = 2. However, we suggest that they could be differentiated using em_calib_status.

Nevertheless, the wide range of HEA instrumentation covering different messengers and many decades in energy means that for some HEA datasets, there may not be sufficient information in the defined ObsCore columns to enable meaningful data discovery, and there can be many reasons for this. As a (perhaps somewhat more obscure) example, consider some high energy gamma ray detectors, which may be sensitive to 2pi sr, or neutrino or gravity wave detectors, which may be sensitive to 4pi sr (often fixed in an ECI reference frame so these are effectively "drift scans"). In cases such as these knowledge of the types of spatial coordinates that are recorded in the data products is essential to determining the usability of the data products for a particular task and so should be part of the query. While there may not be so many existing datasets like this, the numbers will certainly increase exponentially in the foreseeable future (consider, e.g., NANOGrav, LISA) and we take an encompassing view - we don't want to be re-doing this in 10-15 years.

As mentioned by François Bonnarel in issue #4 , we would need an extra table to express this list of columns, a kind of "observable profile" , with the ucd , units, as described in a VOTable FIELD , etc .

My question is : Do we need this as a primary criterium to sort out datasets of interest ? Is it enough to access and open the data files and check the FITS header ?

We have a simple example use case in the Appendix, querying to find event lists with o_ucd that includes both phys.polarization && pos.eq to identify datasets that include both polarization measurements (recall that for an event list, pol_xel = -1) and that have 2-D spatial world coordinates defined (vs. detectors that may only record a single spatial world coordinate in some observing modes).

I don't think requiring the user to access and open the data files and check the FITS headers meets my "meaningful data discovery" criterion in general. While it may be fine when only a small number of data products/small volume of data is expected, particularly when it comes to advanced data products (and especially catalog science, where queries typically will return large numbers of matches for statistical analyses) the numbers of matches and data volumes can increase significantly. I recall sending an example query earlier this year where eliminating one constraint increased the number of returned data products from ~6K to ~105K and the data volume increased from ~9GB to ~22GB. That's a lot of extra data files to download, open, and check the FITS headers.