Additional validation of valid scans

[christophkern]

We are having a strange issue with data coming in from Mount Cleveland, AK. The telemetry link to this remote volcano is very slow so we are using custom tools to transfer the pak files to the computer running the NOVAC Program. We then use the "directory polling" option to analyze pak files as they come in.
For some reason, some of the pak files and resulting txt files / evaluation logs are corrupt. This is visible in that the pak files are larger than they should be (larger than the others), and the corresponding txt files contain duplicate lines. An example is attached.

MAYP115415_240711_0314_0.txt
MAYP115415_240711_0314_0.zip

It's possible that this issue stems from some strange issues with the telemetry and it may be difficult to fix these pak files in the NOVAC program. However, it would be extremely helpful if the software could perform an additional validation / check of the results. It shouldn't be too hard to find scans with duplicate lines and either fix or remove these from further consideration. As it stands, we are getting data spikes in the time series of SO2 emissions and have to manually check each one to see if it's real or not.

[alerner31]

On the topic of scan validation and error checking, we'd also like to request that scans be flagged if they are missing the Sky and/or Dark spectra and then categorize these scans as "corrupt" or some other term so that they are not included in normal data sequences.

Alternatively, or additionally, there can be a check that the .pak files contain the correct number of scans that are expected (reading in the step size in the configuration file and determining the appropriate number of scans from that).

Here are some example scan evaluations, one that is missing the sky + dark scans (plus some other scans) (MAYP115415_240727_1929_0.txt) and then the subsequent scan that looks normal (MAYP115415_240727_1935_0.txt).

MAYP115415_240727_1929_0.txt
MAYP115415_240727_1935_0.txt