Raw (nominally 2D or greater) data, _____ , PD_PROC

[rowlesmr]

This was raised in comments in #231.

The is a semantic hole* between the collection of raw 2D data and the presentation of a reduced 1D diffractogram, particularly this comment

I also note, and agree with, the comment: "this is too big a project to tackle right now".

There are no data names currently available to describe the manipulations done to the 2D data to get it to a 1D diffractogram, so currently, it just appears, nominally in PD_PROC.

But if it does that, there exist no data names in pdCIF to describe any further reduction/normalisation/... done on that 1D diffractogram.

For the purposes of discussion, I propose allowing reduced 2D data to appear in PD_MEAS, as this is the first appearance of the data as a 1D dataset. We can then apply the intensity/detector/angle calibrations to it, use _pd_proc.2theta_corrected to hold specimen displacement corrections, and record everything as we have previously. This also matches in with how TOF data have been historically recorded, with a detector bank being presented as a 1d PD_MEAS diffractogram.

* This would be a good name for a metal band.

[jamesrhester]

In going from 2D data to 1D data in our data reduction pipeline here at the neutron source, all corrections are applied to the 2D dataset (gain, normalisation, angular adjustment, vertical adjustment, geometry correction (not that I'm a fan of that)). The 1D dataset produced at the very end by summing vertically is what is fed straight into refinement software. So by the above suggestion PD_MEAS and PD_PROC for our datasets would be the same thing, and definitely not raw data.

As CIF tries to stick to the principle that the meaning of data names doesn't change based on the value of other data names, we probably can't multi-task with PD_PROC as suggested in Vol G - that is, different stages of processing can be captured by pd_proc loops in separate data blocks with some data name explaining where the loop was in the pipeline. We'd instead identify different data reduction steps with different categories, so the task is to define such categories and make sure they were universal enough to capture all calibrations and transformations applied. Which I suppose we're getting close to with our calibration categories.

An alternative approach would be to write dREL for the PD_PROC intensity and angle values that takes values from elsewhere in the CIF and uses them to calculate for each point in the PD_MEAS loop. That would give an idea of what intermediate categories might be useful to avoid the dREL growing out of control. Not that I'm suggested doing this right now.

[rowlesmr]

Yeah, nah. Also don't want to get in the situation of disentangling datanames with multiple contextual meanings.

I don't know how tractable it would be to list everything; it's essentially an openended problem. Maybe a _pd_proc.diffractogram_chain item that takes an ordered list of scan_id/diffractogram_id values and each data block names the things that were done to get to the data in that block from the previous id.

One of the final steps in the PROC pipeline would be something like 2Th values changing because of a refined specimen displacement/zero offset.

.

So given that we can't really do anything for it now, what is our advice for users?
"Put it in PROC, and explain as much as you can about what you did - _pd_proc.info_data_reduction"? This makes complete sense if the workflow is like yours. What if there are additional manipulations of the 1D data? Just add to the previous info?
What do we do about TOF?

[jamesrhester]

Yes, for now we can recommend that information about data reduction just goes in that human-readable text.

If we look at the situation for single crystal, the steps between raw images and extracted spots (diffrn_refln) is also not captured by datanames, and then the merging (diffrn_refln -> refln) is not all described either. We could at least aim to have as much of the 1D calibration information defined as possible, which we're almost at?