PD_PEAK_OVERALL: should drop `_pd_peak_overall.diffractogram_id`

[jamesrhester]

As a result of #79, a link to _pd_diffractogram.id was added to PD_PEAK_OVERALL as a key data name. This means that, in order to identify a method of peak extraction, both an identifier for the extraction method (PD_PEAK_OVERALL) and for the diffractogram it was used on should be provided, implying that it is possible for more than one extraction method to be used on a single diffractogram and that this extraction method has an independent existence.

The "independent existence" qualifier is important, as normally, an identifier such as _pd_peak_overall.id would be the single key data name in a Set category, to identify instances of some object, concept, or logical grouping of properties. This is not the case for pd_peak_overall with its two key datanames.

Three places refer to _pd_peak_overall.id : pd_refln, pd_amorphous, and pd_background.

What happens if we drop the link to _pd_diffractogram.id from PD_AMORPHOUS_PEAK? For pd_refln and pd_background, nothing, as they are associated with a diffractogram already. For pd_amorphous, there is a link to _pd_peak.id, and each _pd_peak.id is associated with a _pd_diffractogram.id, so we also have no consequences.

Therefore, _pd_peak_overall.diffractogram_id is unnecessary.

[rowlesmr]

I can follow that.

[rowlesmr]

@jamesrhester But now there is no way of linking PEAK_OVERALL to PEAK?

As for some of the other Loop/Set category pairs we have, what about:

PD_PEAK:

• _pp.id: key
• _pp.overall_id: key
• _pp.thevalueswewanttorecord...

PD_PEAK_OVERALL

• _ppo.id: key
• _ppo.diffractogram_id
• _ppo.special_details

[jamesrhester]

@jamesrhester But now there is no way of linking PEAK_OVERALL to PEAK?

Indeed. So I agree we should have a _pp.overall_id that is not a key, unless we envision that the exact same peak will be extracted in a variety of different ways and we want to tabulate that.

I still don't see why we need _ppo.diffractogram_id, as I've outlined in my first message above how the diffractogram a peak belongs to is determined elsewhere.

[rowlesmr]

@jamesrhester But now there is no way of linking PEAK_OVERALL to PEAK?

Indeed. So I agree we should have a _pp.overall_id that is not a key, unless we envision that the exact same peak will be extracted in a variety of different ways and we want to tabulate that.

I think just to take advantage of autopopulating values.

data_block

_ppo.id                  "John's method"
_ppo.diffractogram_id    diffpat_01
_ppo.special_details     "a complicated explanation"

loop_
_pp.id
_pp.2theta_centroid
_pp.intensity # no need for explicit _pp.overall_id on each line
a   12.34   4567
b   13.54   2342
#...

if there are multiple extraction methods, then it would need to be listed explicitly

data_block

_ppo.id                  "John's method"
_ppo.diffractogram_id    diffpat_01
_ppo.special_details     "a complicated explanation"

_ppo.id                  "Mary's method"
_ppo.diffractogram_id    diffpat_01
_ppo.special_details     "another complicated explanation"

loop_
_pp.id
_pp.2theta_centroid
_pp.intensity
_pp.overall_id
a   12.34   4567   "John's method"
b   13.54   2342   "Mary's method"
#...

I still don't see why we need _ppo.diffractogram_id, as I've outlined in my first message above how the diffractogram a peak belongs to is determined elsewhere.

Sorry, I wasn't explicit. I'm proposing removing _pp.diffractogram_id and giving it to _ppo.diffractogram_id to be found via _pp.overall_id

[rowlesmr]

This is also partly competing philosophies on how to represent information which could be considered to be "constant" in loop categories.

Do we repeat ourselves in the loop, or delegate to a a Set OVERALL category.

I'm also cognizant of the fact that we shouldn't be looping linked Set categories* without changing the default _audit.schema, which I don't really want to have to change. So having key datanames in a category which are linked to (for example) _pd_phase.id or _pd_diffractogram.id allows these to be elided in many cases.

I'm not deliberately trying to be ornery, honest.

• I still don't really understand why this is a bad thing; the loop is still well-formed, and the key must be unique, so I don't now why you can't do it.

[rowlesmr]

@jamesrhester But now there is no way of linking PEAK_OVERALL to PEAK?

Indeed. So I agree we should have a _pp.overall_id that is not a key,

hangon. Wouldn't that require an _audit.schema Custom? It's not a key, so you can't autopopulate it, so you need to supply a value, but you're looping a linked Set category data name.

unless we envision that the exact same peak will be extracted in a variety of different ways and we want to tabulate that.

I think I can can agree with this. I can see an output where you might want to compare peak extraction methods, but the output is still going to be different peaks, not the same peak (in a database-id-sense),, and you would be comparing things like 2th values anyway, so having an overall_id key wouldn't be necessary

[rowlesmr]

Sorry for the smash posts, I'm reuploading to my head how all this interacts

Scratch my proposal.

I now advocate retaining _pd_peak.diffractogram_id and pd_peak.id as keys, and adding _peak.overall_id as at least a non-key item. This removes the need for _ppo.diffractogram_id

.

There's also a bit of "humans are bad" going on, too.

I want to label my peaks as "1", "2", "3",... and my diffractograms as "temp_100K", "temp_120K", ...

if we have _pp.diffractogram_id as a key, then peak "1", if you have the diffractogram, is always uniquely identifiable. You can then choose to give further information if you wish in the PPO category, but it isn't necessary to properly identify a peak

[jamesrhester]

I think just to take advantage of autopopulating values.

If there is only one peak extraction method (_ppo.id) then the data name could be dropped anyway, as there's no ambiguity as to which method was used. If there is more than one (how often does that happen?) we currently have to live with the extra column. I wouldn't like to shift data names around to make things more concise at the cost of incorrectly characterising the way data are thought about.

As for _pp.diffractogram_id, that will usually have an assumed value in the pd_peak loop, being a key data name, so there is no particular win for concision by putting it in ppo. And putting it in ppo as a non-key data name means that there will be a new _ppo.id for every diffractogram, which hides the fact that the method itself might not be any different.

hangon. Wouldn't that require an _audit.schema Custom? It's not a key, so you can't autopopulate it, so you need to supply a value, but you're looping a linked Set category data name.

The pp category is a Loop category, so no problem.

I still don't really understand why this is a bad thing; the loop is still well-formed, and the key must be unique, so I don't now why you can't do it.

Well, it is true that you can do it. All categories with keys can be treated as Loop categories, end of story, as long as software is written to expect any and all categories to be looped.

The reason for the existence of Set categories is only legacy: software written over the last 30 years that knows nothing of Set categories can still give useful results when fed data blocks that follow the rules. So structure display software given a structure block as described by our powder presentation recommendations will actually provide a sensible structure, whereas if the atom sites from all structures were in one loop that software would not do well. Instead of asking such software to be rewritten to expect loops all the time, we just ask that it checks the value of _audit.schema (and even getting that small change is probably a big ask).

mmCIF recognised that everything was a loop, so all their categories are officially looped, with keys. Except they use trickiness with _entry.id child data names to restrict certain categories to single rows.

This is also partly competing philosophies on how to represent information which could be considered to be "constant" in loop categories.

Maybe not even philosophy, but how much inefficiency we can tolerate. Given that most diffractograms will have thousands of angles but only hundreds of peaks, and our _ppo.id could be a single character, the relative expansion in file size seems minor. And in return for this "inefficiency" we express the data more naturally (it seems to me).

[jamesrhester]

I now advocate retaining _pd_peak.diffractogram_id and pd_peak.id as keys, and adding _peak.overall_id as at least a non-key item. This removes the need for _ppo.diffractogram_id

Agreed.

if we have _pp.diffractogram_id as a key, then peak "1", if you have the diffractogram, is always uniquely identifiable. You can then choose to give further information if you wish in the PPO category, but it isn't necessary to properly identify a peak

Agreed.

[jamesrhester]

I've thought about this some more. It is acceptable for PP.overall_id to be a key data name, but then all categories that refer to peaks must also include datanames that link to PPO.id so that the peaks can be uniquely identified using peak_id, diffractogram_id and PPO.id. On balance I think this is not worth the extra data names, given that most datasets will not attempt multiple different approaches to peak extraction.

[rowlesmr]

On balance I think this is not worth the extra data names, given that most datasets will not attempt multiple different approaches to peak extraction.

Agreed