Using CESM's coupler history file ("forcing") fields in assimilations

[kdraeder]

This topic was sparked by a question from Nuo Chen (Cavallo) about generating synthetic obs
near CESM's surface (below the lowest CAM model level, which is normally an excluded region.
I thought about what CESM quantities might be available for that, and wondered whether the
fields in the coupler history files could be useful.

Jeff's initial thoughts were discouraging (see discussion below), but it's come up in 2 more conversations
in the last few weeks; Karspeck and Long at C-Worthy, and Danny Leung (CGD aerosol source estimation),
so I'm recording the discussion here.

---

Kevin Nov 10, 2023, 7:52 AM
Hi Jeff and Dan,
are you aware of anyone considering the possibility of using the coupler history file variables ("forcing")
as part of the, or a, model state?
Off hand I don't know whether the atmospheric forcing of the surface is calculated in CAM or in the coupler.
It could be that the coupler would need to be considered a model component and relevant observations assimilated into it.
I would hope that there's a much stronger correlation between these variables (and observations of them)
and the CLM fields than there is between the atmospheric
variables and CLM fields.

[kdraeder]

Jeffrey Anderson Sat, Nov 11, 8:58 AM
Kevin,
Not quite sure about the idea here. Are you talking about using info from the coupler files as inputs to compute forward operators? If so, we have chatted about that off and on through the years, but never identified a forward operator that seemed to justify a clearly nasty implementation.
Your message refers to treating coupling variables as model state. To the best of my understanding, these are not state variables in CESM, but maybe I'm not fully informed. Is this related to the question from Oklahoma about 2m temperature obs?
Jeff

[kdraeder]

Kevin Raeder Nov 13, 2023, 9:46 AM

Yes, this line of thought was sparked by Chen's question.
The forward operator occured to me first, when I was thinking about
the process of generating estimated obs.
But then it occurred to me that the process is somewhat dependent
on the "model" and what fields are available from it.

The fields (mostly surface fluxes) in the coupler restart files are generated
by one part of the model (CAM) and used by another part (e.g. CLM),
to calculate fields that then change the original part (CAM).
So that looks like a prognostic variable to me, just like T modifying
and being modified by various parts of CAM. So they could be candidates for assimilation.

It seems to me that in the strongly coupled context the model (CESM)
writes out a variety of files that each contain part of the whole state vector;
CLM restarts, CAM initials, ... coupler restarts.
That's an extension of what, I believe, CLM does for DART.
We may not end up implementing the assimilation that way, but I think it's a legitimate view.
We're already selective about which fields are included in the state vector,
so I'm guessing that some or most of the fields in the coupler restart files
would be excluded. And they're 2D fields, so they don't add much volume
to the state vector. It would take some work to figure out which are the important fields.

This view doesn't hold up in the weakly coupled environment;
each CESM component is viewed as a separate "model".
If the coupler is defined as a model, then inside of it the fluxes are not prognostic,
I believe.

[kdraeder]

Dan Amrhein Nov 13, 2023, 1:42 PM
Thanks for the clarifications Kevin. I know there's interest in using DA to change fluxes since we know that's where a lot of errors arise in coupled models. At the same time I think your question gets to the artificiality of treating different Earth System components as being distinct. I remember discussing this question with Judith early on in the M2Lines project since I know fluxes were/are a focus there.

So I understand, the idea is that for an ensemble member, the coupler takes two states (e.g. oce/atm) and computes fluxes between them, which are used to update component model states. For typical strongly coupled DA, if we had an ocean obs, we'd usually update oce/atm state. Instead, are you suggesting that we update oce and oce/atm fluxes, and then those fluxes would spread the ob impact to atm?
Is the idea that flux-ob covariances could be more robustly estimated than cross-component covariances?

[kdraeder]

Kevin Raeder Mon, Nov 13, 2:42 PM
Dan,
that's my understanding of current strongly-coupled DA.
I'm not suggesting replacing or igoring any cross-component covariances (yet),
but including the fluxes in the state vector seems theoretically justifiable
and could provide benefits such as you mention, as well as more straightforward
calculation of some estimated obs. And we might find that the flux-obs covariances
(or flux-component variable covariances) are significantly larger and more informative
than the cross-component covariances, especially give the small cross-component
covariances that Alicia found(?). It's conceivable that the cross-component
covariances could be ignored, which would simplify the assimilation and make it
cheaper to run.

[kdraeder]

Dan Amrhein Nov 14, 2023, 10:03 AM
Kevin, got it. I did my best to review the coupled DA literature when I got to NCAR, and this specifically doesn't ring a bell, but that doesn't mean it's not out there -- Jeff may know more. ECCO adjusts fluxes, but that's ocean-only. Sugiura et al. 2008 adjust flux parameters in a coupled model using 4DVAR, and there are other cases where folks have inferred parameters too (here's an ocean-only case Fred was on in 2009).

The FO motivation is clear. One thing I'm not sure about is the difference you would expect between adjusted fluxes and fluxes computed from an adjusted state.

My sense is that strongly coupled DA (at least ocean-atm) hasn't lived up to the hype yet and there's some head-scratching in the community about how to proceed.

[kdraeder]

Kevin Raeder Nov 14, 2023, 11:03 AM

The FO motivation is clear. One thing I'm not sure about is the difference you would expect between adjusted fluxes and fluxes computed from an adjusted state.

I don't know either, but if the states are not adjusted much because of weak covariances,
then directly adjusting the fluxes could more easily lead to larger adjustments.

My sense is that strongly coupled DA (at least ocean-atm) hasn't lived up to the hype yet and there's some head-scratching in the community about how to proceed.

Sounds like an opportunity!

[kdraeder]

Jeffrey Anderson Nov 16, 2023, 2:22 PM
By chance, this idea also came up in discussion with Alicia and Matt down at [C]worthy yesterday. The idea was to use the obs to directly correct the fluxes, rather than correcting the models and hoping they correct the fluxes. Exactly what obs might be related to the fluxes was less clear.

I will note again that the fluxes themselves are not state in the way we think of things, any more than the intermediate values in the convective parameterization are. The coupled dynamics are just a function of the individual model states. There are slight caveats possible related to the way in which the coupled model is stopped/started (almost another pause/resume complexity) and the way in which the coupling intervals relate to the model timesteps.

Does anyone know of obs that they think are closely related to the fluxes?

[kdraeder]

We each have an intuitive feel for what qualifies as a state variable, but is there an accepted definition or a list of characteristics that we could apply to these fluxes?
My view is that fluxes are already included in the conventional state vectors of fluid systems; wind is a flux of fluid.
We choose to characterize it in models as density (or pressure) and vector components of motion,
but that's for convenience and clarity.

It seems to me that the primary characteristic is whether a variable is required in order to start a model
without unacceptable shocks or spinup. We can start CAM without clouds, but that state doesn't act like
the atmosphere by some important measures, until the clouds build up, so we include clouds in the state vector. For some variables it's obvious, but for others we need to test in order to decide what's necessary.
That's what we did with the fields in the CAM initial files, which CAM says are required for starting,
but some of which are not worth including the state vector.

I also see the separation of the Earth system into models or components as somewhat arbitrary
and changing with time. So it's unclear what an intermediate versus final state is.
There are free-standing models of convection that can be used in DA in which the temperature
is obviously part of the state and has some final value. But when that model is embedded in a GCM,
then we usually choose to call it's final temperature an intermediate value. But its cloud amounts
may be the final values and are included in the GCM state.

[kdraeder]

Kevin Raeder Fri, Nov 17, 4:31 PM
Here's a short summary report of what I've found in the cpl history (forcing) files
and a foray into surface obs networks (thanks to Brett for suggestions).
I'll send a fuller report, if that would be useful. [Here it's cpl_fields+obs.txt]

There are ~40 distinct fields in the 5 forcing files. Many are fluxes, but a large fraction
are traditional state variables or quantities that could be traditional state variables.

Of those 40 fields, roughly 3/4 are directly or very closely observed in the NEON network.
I also looked at the ICOS (European) and TERN (Australian) ecosystem observation networks
and found similar overlaps, but those are harder to dig up details.
So there is somewhat global coverage, at least by the standards of other "global" observations we assimilate.
Other networks Brett mentioned are Ameriflux and Fluxnet, which I didn't explore yet.
And he mentioned that there are lots of other networks, which weren't coming to mind at the moment.

So there seems to be a rich array of observations of the quantities in the forcing files,
including traditional state variables, at least for the recent past.

[braczka]

Not sure I understand all the details of this conversation, but just to add to the flux conversation, the land based flux tower networks of NEON, Fluxnet (Ameriflux) come to mind. I have used them mostly in the context of carbon fluxes but energy fluxes (sensible, latent, irradiance) should be important in this context. There also should be flux reanalyses products available. FLUXCOM is an example of, where the gridded land surface flux product is based on the network of discrete tower-based observations. At least in land DA, assimilating site level observations for regional domains is difficult given the representation mismatch of the observation to the model state. These networks are land based, so not likely to bring utility to ocean-atmosphere coupling.

[nancycollins]

a couple of questions - one about the execution cycle of cesm (which has always been a point of confusion) and one about only updating some of the coupler fields.

is it clear exactly when cesm updates the coupler file values, and then when that data is used by the other components? if the coupler data is updated by an assimilation but then recomputed and overwritten during the next iteration of the cesm cycle then the assimilation wouldn't make any difference in the results.

if there are scalar quantities as well as vector (flux) quantities in the coupler files, then updating only the scalar fields seems like it would lead to inconsistent results. if you have a point observation and a flux state field, how to you adjust it? wouldn't you need multiple observation values (in time or space) to compute a vector quantity?