Add eva comparison plots for observations

[mranst]

This PR adds comparison type plots for 3dfgat_atmos and 3dfgat_cycle. Any plot generated by the regular EvaObservations task is plotted in the same figure for two different experiments. I could use some help knowing how to handle the 'difference' aspect of the plots, for a lot of these I think a straight up difference isn't going to make much sense. We could handle it by plotting both sets on the same graph, but some of these plots are so dense that it probably isn't going to be legible

[Dooruk]

Some plots that came to my mind is that 1) two different experiments could be on the same scatter plot, with x-axis exp 1 ombg and y-axis exp2 ombg (or any other IODA groups), 2) Similar to the increment differences there could be difference plots of ombg and oman with 3 map panels (exp1 ombg, exp2 ombg and difference).

However I would think the better approach would be the different groups and applications figure out what type of plots they would want to create. For instance Megan can take a look at some of these EVA YAMLs (with our guidance nonetheless) and modify them accordingly. Otherwise I think this is a strong enough foundation, I haven't done a full review sweep though.

Regarding this PR, does the EVA yaml containing 3dfgat_atmos indicate those can be only used for those suites? And this finds the overlapping cycles correct?

[mranst]

Regarding this PR, does the EVA yaml containing 3dfgat_atmos indicate those can be only used for those suites? And this finds the overlapping cycles correct?

It finds the overlapping cycles automatically (though they can be manually specified). The eva yamls are differentiated by suite right now in swell, so I carried over that convention and changed the file name to distinguish them. We could potentially split up the compare suite, which is located in one directory right now, and have individual ones for each suite

[mranst]

I've added obs plots for 3dvar and 3dvar_atmos, and cost function plots for atmos. @rtodling does this look okay?

I added exp 2 vs 1 Oman and Ombg plots for obs as well

[rtodling]

This is looking really good.

I only have minor suggestions:

1. instead of designating exerperiments as 1 and 2 it might be better to allow the user to defined which is the CTL and which is the EXP and label, CTL and EXP.

2. I am not sure, when comparing two experiments, it should be needed to plot the individual plot in some of the cases. What I mean by is this: if there is a way to put the results together in a single figure, as for example in the case showing JoJc and Jb, create a single fig w/ both experiments' results combined should suffice.

3. The comment as in (2) applies for the scatter plots ... only the OMB(CTL) vs OMB(EXP) and OMA(CTL) vs OMA(EXP) are needed. The individual ones should not be needed in the comp suite - those we get where simply running the suite for a give experiment.

The suggestions above aim at reducing the number of plots generated - good for i-node count - but also for user sanity: the figs we have the harder it gets to look at them all.

[mranst]

I've reduced the number of plots under the jedi_log task, so they just display the information on one panel. By default, the experiment listed first will be referred to as 'CTL', and the second as 'EXP', I also added the ability to set custom tags to refer to experiments instead. You can now specify a dictionary to set these

[mranst]

Here's a few example plots:

[Dooruk]

Quick comment while testing, this task failed for me so I noticed for marine tasks data collection is jedi_log_test not JediLogTest. Sorry, this is annoying.

At some point we can make geos_cf, atm and marine consistent but that is not relevant for this PR.

[mranst]

Do the cost functions work differently for marine as well? I see jb and jojc isn't a variable in the dataset

[Dooruk]

ditto