Summary stats prevents processing completion by default

[TFirth2]

By default:

summary_stats: run: true

which causes this error message in the terminal:

Traceback (most recent call last):
  File "/Users/t/miniconda3/envs/main_topostats/bin/topostats", line 8, in <module>
    sys.exit(entry_point())
  File "/Users/t/Documents/topostats/main_topostats/TopoStats/topostats/entry_point.py", line 1256, in entry_point
    args.func(args)
  File "/Users/t/Documents/topostats/main_topostats/TopoStats/topostats/run_modules.py", line 295, in process
    toposum(summary_config)
  File "/Users/t/Documents/topostats/main_topostats/TopoStats/topostats/plotting.py", line 395, in toposum
    result_option: tuple | None = topo_sum.sns_plot()
  File "/Users/t/Documents/topostats/main_topostats/TopoStats/topostats/plotting.py", line 231, in sns_plot
    sns.histplot(
  File "/Users/t/miniconda3/envs/main_topostats/lib/python3.10/site-packages/seaborn/distributions.py", line 1416, in histplot
    p.plot_univariate_histogram(
  File "/Users/t/miniconda3/envs/main_topostats/lib/python3.10/site-packages/seaborn/distributions.py", line 447, in plot_univariate_histogram
    densities = self._compute_univariate_density(
  File "/Users/t/miniconda3/envs/main_topostats/lib/python3.10/site-packages/seaborn/distributions.py", line 325, in _compute_univariate_density
    for sub_vars, sub_data in self.iter_data("hue", from_comp_data=True):
  File "/Users/t/miniconda3/envs/main_topostats/lib/python3.10/site-packages/seaborn/_base.py", line 948, in iter_data
    data_subset = grouped_data.get_group(pd_key)
  File "/Users/t/miniconda3/envs/main_topostats/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1103, in get_group
    warnings.warn(
FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.

This prevent the final stats files from being produced. I think it would be easier if this was just set to false by default.

[ns-rse]

Do you have details of the environment and a copy of the full .log this occurs with please @TFirth2 ? I can't reproduce this with minicircle.spm under my current environment and the above TraceBack only shows warnings rather than errors.

Output of...

pip freeze

...can be copy and pasted.

If you use triple backticks on separate lines around the code it will appear as a codeblock

```
this will be a code block
```

Should be able to attach a file using paste/drop or the button bottom left of comments box.

[ns-rse]

I suspect this is actually the problem that we are still seeing in the Continuous Integration tests which are failing on OSX...

topostats/plotting.py:309: in melt_data
    melted_data = pd.melt(df.reset_index(), id_vars=["grain_number", "basename"], value_vars=stat_to_summarize)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
frame =    level_0  index  ... total_contour_length  average_end_to_end_distance
0        0      0  ...         5.175174e-08  ...                    0.0
2        2      2  ...         8.687533e-08                          0.0
[3 rows x 13 columns]
id_vars = ['grain_number', 'basename'], value_vars = ['total_contour_length']
var_name = None, value_name = 'value', col_level = None, ignore_index = True
    @Appender(_shared_docs["melt"] % {"caller": "pd.melt(df, ", "other": "DataFrame.melt"})
    def melt(
        frame: DataFrame,
        id_vars=None,
        value_vars=None,
        var_name=None,
        value_name: Hashable = "value",
        col_level=None,
        ignore_index: bool = True,
    ) -> DataFrame:
        if value_name in frame.columns:
            raise ValueError(
                f"value_name ({value_name}) cannot match an element in "
                "the DataFrame columns."
            )
        id_vars = ensure_list_vars(id_vars, "id_vars", frame.columns)
        value_vars_was_not_none = value_vars is not None
        value_vars = ensure_list_vars(value_vars, "value_vars", frame.columns)
    
        if id_vars or value_vars:
            if col_level is not None:
                level = frame.columns.get_level_values(col_level)
            else:
                level = frame.columns
            labels = id_vars + value_vars
            idx = level.get_indexer_for(labels)
            missing = idx == -1
            if missing.any():
                missing_labels = [
                    lab for lab, not_found in zip(labels, missing) if not_found
                ]
>               raise KeyError(
                    "The following id_vars or value_vars are not present in "
                    f"the DataFrame: {missing_labels}"
                )
E               KeyError: "The following id_vars or value_vars are not present in the DataFrame: ['basename']"
/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pandas/core/reshape/melt.py:74: KeyError

Thus the proposed fix in #1273 doesn't address this and we need to investigate why on OSX alone the basename column is missing from the data frame.

[ns-rse]

Seem to have passing tests now on OSX for PR #1276 but the missing basename persists on OSX in PR #1273.

As I've noted in my comment on #1273 I don't think changing this default solves the problem (which may or may not be missing basename).

Aside from working out why basename is missing I think we should check that the CSV files are written to disk before calling toposum. The dataframes that are written to CSV should exist before running toposum as they are collated from individual images after parallel processing has completed.

Checking topostats.run_modules.process() we see that we conditionally run toposum() in the code block beginning on line 261.

However, the writing of CSV statistics comes after this on line 315 for grain_statistics.csv, lines 326/327 for branch_statistics.csv) and [lines 326/338](https://github.com/AFM-SPM/TopoStats/blob/416ec6c2f3a8c21cba8e936aafe09d3edaa3f0c7/topostats/run_modules.py#L338) for molecule_statistics.csv`).

Thus moving the code block that starts on line 261 to after these, and for good measure it can go after the write_yaml() which writes the configuration to disk, will mean that output files will be generated even if toposum() fails.

As it happens I'm half-way through getting CSV files written on my refactor of #1102 so will make sure to include that change there.

The issue of why basename is missing though remains and I will therefore leave this issue open so it can be investigated and addressed.

[ns-rse]

As it happens I'm half-way through getting CSV files written on my refactor of #1102 so will make sure to include that change there.

This is now the case, CSV files are written to disk before any attempt is made to plot grain statistics.