Detect and optionally fix <=0.1 fluorescence values (fl?_max)

[sdespina]

Events are missing when plotting fluorescent values using the logarithmic scale option.
Whenever the FL value of an event is negative, the event is completely omitted from the output graph. Results are therefore under-represented in the graphs

[paulmueller]

The general issue is described and tracked in DC-analysis/dclab#101. The task here would be to implement the UI that allows users to remove <=0.1 ==0.1 values from fluorescence max data.

Tasks:

• increment the dclab version in setup.py once Create a new check for negative fluorescence values dclab#101 is part of a release
• verify that the fl?_max checks show up when running the checks
• add a checkbox (which defaults to true) for setting <=0 values to 1 (or add an offset depending on how the discussion in Create a new check for negative fluorescence values dclab#101 goes)
• add a checkbox (which defaults to true) for "reparing" the fluorescence traces (see Create a new check for negative fluorescence values dclab#101 (comment))
  1. Determine the global offset by fitting a Poisson distribution or taking a median or whatever from all traces (see next comments)
  2. Determine the global offset by taking the traces for the 100 lowest peaks (or all peaks with 0.1) and computing their minimum.
  3. Only correct offsets that are <0 (as per @chrherold's comment in Create a new check for negative fluorescence values dclab#101 (comment)). Do not edit trace data (see comments below)
  4. Add this offset to all the traces
  5. Write the offset as an additional meta data keyword ["fluorescence"]: baseline ? offset (which first has to be introduced in dclab)
  6. Recompute fl_max for all events events that are 0.1 (we can just extract them, because we have the fl_pos feature) by adding the known offset to the trace data at that position.

[paulmueller]

Comments for @maxschloegel:

• Locate main.ui and add the checkbox somewhere on the right hand side. The checkbox should have a toolTip that explains what it does (you can edit the toolTip in designer).
• The user should be notified before conversion/compression if/which datasets will receive a repair for the fl?_max issue.
• You do not have to expect the fl?_max_ctc feature, only the fl?_max feature. This is because the cross-talk-corrected (ctc) features are usually created within dclab/Shape-Out2 and are not part of any raw data files. DCKit is for raw data files only.
• You will have to install dclab in editable mode (pip install -e path/to/root/of/dclab), because Create a new check for negative fluorescence values dclab#101 is not part of any release yet.
• DCKit always writes additional logs to the produced .rtdc files. The log files should have an entry stating by how much the baseline of each trace was shifted.
• Regarding (i) from above: You will probably have to look at example data to get an idea how the fluorescence data is usually distributed. I think something like a median should work for finding the baseline in the fluorescence trace data. (see comment below)
• Regarding (ii) from above: If there are still fluorescence values <0 when the baseline is >0, maybe it would still make sense to lift the baseline to a higher value. We would have to find out how the new version of Shape-In determines this baseline. (see comment below)

[paulmueller]

I just realized that the baseline should actually always be around zero. (see comment below) This setting is adjusted during RT-DC device maintenance for each sensor. This means we could still check for a non-zero baseline, but we should not modify any trace data, only issue a warning (maybe?). (see below)

[paulmueller]

That also means that the original issue (negative fl_max values) cannot be solved here, because those are absolutely expected for a basline at zero. This issue is now only about fl_max values that were set to 0.1. The other issue (data points cropped in log view) is actually just a visualization issue (see DC-analysis/dclab#56).

[paulmueller]

OK, now I got everything together.

1. "Baseline" is defined as the value in the trace from which the peak max is measured. This is always "0"!
2. "Offset" is defined as the lowest value in all traces, which should be set to zero. Sometimes this is not the case and we want to fix this here.
3. Negative peak maxima are the result of a faulty offset.
4. Actually, none of the fluorescence values (all of them, not only the maximum) should be positive (by definition).
5. We are not running into crosstalk-correction issues, because the offset is really small compared to actual fluorescence signals.

When we have determined the correct offset, we don't manipulate the trace data, but we write the offset to the meta data. We do however update the fl?_max features. We do not update the fl?_width features (the peaks are usually not triangular but rectangular, so we don't really make a mistake).