bug: VERTISUNDEF in the cam-fv get_close routines

[kdraeder]

Describe the bug

The context for this is that Gio (CMCC) is trying to assimilate rttov radiances and initially gave them VERTISUNDEF. cam-fv/model_mod has a feature which reduces the size of increments in the top layers in order to play nicely with CAM ("damping"). It increases the distance between an ob and a state variable (or other ob) based on the vertical distance between them, but there's an exception for obs with VERTISUNDEF, because no one could figure out how to calculate the vertical distance. So there are unwanted increments in the damped region which could degrade the assimilation and which raise questions from users about unexpected behavior.

The original CAM+DART damping was designed to prevent all observations
from changing the model state in the region of CAM where artificial diffusion is applied.
The current CAM+DART get_close_ state prevents state variables with VERTISUNDEF
from have damping applied, for all obs. So far there are none of those.
Will there ever be model state variables with VERTISUNDEF?
Model parameter estimation experiments might use something like that.
But for now the test for VERTISUNDEF in get_close_state is never .true. and is misleading.

In get_close_ obs, my view is that it shouldn't even be a question,
because I can't think of an observation of anything that doesn't come from
a location or a region (which is an integral of locations).
We may not know how much each location in a region contributed to the ob,
but that just means that we need to make an educated guess:
Worst; assume it comes from a point (let the localization spread its influence)
Less bad; it comes uniformly from the region
Maybe not bad; get a plausible distribution from theory or empirical knowledge
If I understood Jeff's comments about this, the localization is more important than the location,
which might make this a secondary issue, except that we need to decide what VERTISUNDEF means
and whether and how to use it.
In my opinion obs should not be assigned VERTISUNDEF
and there is no need for a test of it around the damping algorithm.

In addition, I'm not convinced that there should be any damping in get_close_obs.
At first glance the idea is to treat the update of the estimated obs
the same as the update of the state, which has damping applied.
But an original estimated ob is the result of a forward operator,
which may use state variables from both inside and outside of the damped region.
I wouldn't know how to apply damping to an ob that's consistent with the mix of state variables.
It seems to me that the way to do this consistently is to use the updated state
to calculate updated estimated obs, but it seems like that's not how it's done.
Instead, the estimated obs are updated by direct regression with the actual obs.
Or am I misreading the code?
If I'm not, it seems that the direct regression method of updating the obs could introduce
inconsistencies with the updated state, unless the covariances between obs
are the same as between obs and state variablec, which seems unlikely.
So taking the damping out of get_close_obs might introduce inconsistencies
that are no worse than what's already being done.

Related to that, can someone point me to a source for learning about
why that is the right way to update the estimated obs?

Which model(s) are you working with?

cam-fv, any CAM

Version of DART

Which version of DART are you using? Manhattan, all versions to the present.

Have you modified the DART code?

No

[nancycollins]

i think taking care of top of model issues in the get_close() routines is a much better solution all the way around.

one thing to watch here - the get_close_state() and get_close_obs() routines get called with the current obs being assimilated and then need to return "close enough" state vector items and "close enough" other observations. the test for "obs too high" should only need to be on the base obs - the one currently being assimilated. if it is too high you return 0 close items for all of them - state and obs. since that observation doesn't change the model state, it doesn't need to change any of the unassimilated forward operator values.

the "too high" test for discarding obs doesn't need to know anything about the state.

you do need to decide what you want to do about obs which are lower than the "too high" limit but the effective radius (cutoff) lets that obs impact change values at the model top. that's what the separate ramping function could take care of. it't the ramping function that can't work correctly with VERTISUNDEF because the distances computed are only the horizontal. so you can't ramp it in the vertical. the shape of influence of a "normal" obs is an ellipsoid but for VERTISUNDEF it's a cylinder.

i've never (yet) seen VERTISUNDEF used for state, but certainly for obs for things like the vortex center location for a hurricane and vertically integrated values like total precipitable water. you might want to argue that radiance obs shouldn't be created with VERTISUNDEF but with a vert of pressure with the vertical location at the point where the absorption curve is the highest.

disclaimer: i'm on vacation so my thought process may not be 100% working.

[hkershaw-brown]

link #486

[hkershaw-brown]

moving to discussion