can we avoid missing values in CLM by changing what fields are in the state?

[nancycollins]

Use case

one reason we allow missing values in the dart state is because CLM stores some values in arrays of layers and different ensemble members have different numbers of layers. however, i'd like to step back and ask if impacting individual layer values with the assimilation is the only solution here. i believe observations are of snow depth, or water content, or albedo, or other quantities which are modeled by the layers. could we instead preprocess the layers to construct one or more derived fields which are impacted by the observations during the assimilation and then redistributed to the layers before being passed back to CLM?

Is your feature request related to a problem?

allowing missing values in the model state has a code footprint in: forward operators, inflation, assimilation, smoother code. but possibly more important is that our current solution skips updating values for any items where one or more ensemble members have missing values. that means members without missing values contain their original values for those items at the end of the assimilation. that makes them inconsistent with layers which were successfully impacted by the observations. also adaptive inflation values for those items aren't updated.

Describe your preferred solution

can we look at what quantities (water content, crystal size, age, albedo, i don't know?) the model actually cares about, rather than the implementation details? i believe there is a 'redistribution function' in the model which spreads quantities across levels. can we impact the aggregated quantities with observations and then redistribute the updated values before returning to the model? the clm_to_dart program constructs derived fields, the assimilation updates the derived fields, and the dart_to_clm program uses them to update the prognostic fields in the restart files. we already have these programs to deal with CLM's inconsistent treatment of missing values in the restart files, so this would be an enhancement to what's already there.

Describe any alternatives you have considered

what is there is working but seems like it could be improved, both to simplify the dart code and to possibly give a better assimilation result.

[nancycollins]

note that this is very closely related to tim's issue #253 but i started looking at this from the 'can we avoid missing-in-state support' rather than knowing enough about the model to suggest how this could be done.

[braczka]

This makes a lot of sense -- also highly related to issue #253 (Updating snow in CLM). The 'redistribution' option of snow (SWE) is intended to be an added feature, as an alternative to updating layers directly, but we could discuss this as a complete replacement, or a necessary step before merging clm_main with main branch. Uncertain how long this would take to implement and test. It is much more heavy-handed approach and imparts repartitioning of the snow layers within the DART code -- something that has usually been left to the fusion/fission algorithm of snow in CLM.

[hkershaw-brown]

footprint in quad_utils_mod #249

[braczka]

To also add more context -- missing layers in CLM extend beyond just snow layers (above surface ice layers), but also to the subsurface layers. Many of the subsurface layers are missing (layers 13-37) if the soil depth is relatively shallow because of bedrock. The subsurface missing layers, behave differently from above surface snow layers, in that they are the same for a given location across ensemble members (soil depth is fixed). For that reason, the subsurface layers would be a bit easier to handle.

[hkershaw-brown]

noah has set_missing_ok(.true.)

DART/models/noah/model_mod.f90

Line 252 in 62ecfa2

call set_missing_ok_status(.true.)

Currently have both a filter_nml::allow_missing_clm and a subroutine set_missing_ok_status