Fluxnet observation converter #478

braczka · 2023-05-24T20:01:23Z

Description:

Generates a new observation converter (Fluxnetfull_to_obs) for eddy covariance flux tower data (carbon, water energy fluxes) which is useful for land DA and has been used in the past for CLM-DART. This converter uses the new format provided by Fluxnet/Ameriflux, and also includes new features such as GPP and RECO carbon flux observations, different time aggregations (hourly through monthly), and provides estimates of random, ustar and partitioning uncertainty. As part of this PR, documentation changes will be made to the older, deprecated ameriflux converter (level4_to_obs) and the broken links will be fixed. New flux tower observation types will be added to accomodate the forward operator approach for time aggregated fluxes (daily through monthly).

Also includes pull requests #540 (Added get_channel to obs_def_rttov13_mod.f90) and #541 (bug fix: remove cloud_overlap integer from list of logicals to query)

Fixes issue

This addresses github issue #339 Location and format of L4 Ameriflux Observations have changed

Also #537, #532

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

I have compiled and run a test case based on the native hourly resolution flux data from Harvard Forest (AMF_US-Ha1_FLUXNET_FULLSET_HR_1991-2020_3-5.csv). This data can be downloaded from here:
(https://ameriflux.lbl.gov/data/download-data/)

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

Creating new observation converter for site level eddy covariance flux tower data (NEE,GPP,RECO,LH,SH). The existing Ameriflux_L4 converter is outdated and new database has new flux options and time aggregates

Necessary changes in syntax to allow fluxnetfull_to_obs.f90 to compile correctly. Also providing proper changes to support files input.nml and quickbuild.sh within work directory

fluxnetfull_to_obs converter compiles and runs testing on hourly flux fullset data. Header now includes timestamp which is treated differently than real values

braczka · 2023-05-24T20:09:48Z

@mjs2369 hoping you can take the lead on the review once I have pulled this PR off draft. The documentation still needs to be worked on a bunch by me, but I got the main converter (Fluxnetfull_to_obs) working today. It only functions on the hourly resolution flux data format, and I need to add the capability to convert time averaged observations, and also test it. This lays the foundation though.

mjs2369 · 2023-05-24T20:36:01Z

@braczka I can definitely do that. I'll keep an eye on it to review when the PR is no longer a draft.

Adds ability to use HH,HR,DD,WW,MM,YY flux data where test file format changes

Expanded functionality such that fluxnet converter can take in HH,HR,DD,WW,MM and YY time averages

Added more fixes and warnings to make code compatiable with coarser time resolution. File formatting changes based on time resolution.

QC values change from integers to fraction depending upon time resolution. Added code to convert fraction back to integer. Provided empirical fix when uncertainty values for H,LE and NEE are missing.

braczka · 2023-08-15T15:15:32Z

The code has been tested for all relevant time resolutions: hourly, daily, weekly, and monthly. The documentation has also been completed. Taking off the draft status -- ready for review.

braczka · 2023-08-15T15:40:25Z

Also a quicker link to get access to test data rather than the flux repository: /glade/scratch/bmraczka/Ameriflux_data

XueliHuo · 2023-08-16T19:26:37Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+real(r8)           :: maxgoodqc       = 3.0_r8
+! Always true except for latent,sensible heat and NEE for hourly time periods
+! Can only be false of HH or HR time resolution
+logical            :: gap_filled      = .true.


Lines 61 and 62 mean that for the coarse time resolution (DD, WW, MM), gap_filled for NEE, latent, sensible heat, GPP and RE is true, and for the native time resolution (HH, HR), gap_filled for NEE, latent, sensible heat, GPP and RE can be true or false, correct?

Yes, that is correct. If namelist variable gap_filled = .false., and time_resolution is set to any coarse resolution like DD, WW or MM, the code will error out. See lines 228-235.

XueliHuo · 2023-08-16T20:43:16Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+rewind(iunit)
+call decode_header(iunit, nwords)
+
+obsloop: do iline = 2,nlines


Could you please point me to the obs_seq.out that you generated in the test?

Sure -- I put them here: /glade/scratch/bmraczka/Ameriflux_data/

XueliHuo · 2023-08-17T18:35:07Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+   if (tower%hQC   < 0.0_r8) tower%hQC   = 2
+else  ! No QC values for energy balance corrected le and h.  Assign fair QC.
+   tower%leQC  = 2
+   tower%leQC  = 2


tower%hQC = 2

XueliHuo · 2023-08-17T18:58:27Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+   tower%leQC  = 2
+   tower%leQC  = 2
+endif
+


The leQC and hQC are assigned 2 in the if-else statement from lines 1139 to 1145 if the the observation le and hQC look Okay. I am wondering why there are no statements for leQC and hQC like lines 1155-1158 for gppQC and recoQC? Is it because in le and h are never negative in the downloaded data?

Both le (latent heat) and h (sensible heat) can be positive or negative (physically reasonable), thus I don't use this as a diagnostic to adjust the QC values. On the other hand, negative gpp and reco values are physically impossible, thus I force the code to reject them by making their respective QC values big.

Makes sense.

XueliHuo · 2023-08-17T19:14:57Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+      oerr      = oerr * umol_to_gC
+      ! Take average of night and day partitioning methods
+      tower%gpp = ((tower%gppDT + tower%gppNT) / 2)  * umol_to_gC    ! Matches units in CLM [gC m-2 s-1]
+      qc        = maxval((/real(tower%recoDTQC,r8),real(tower%recoNTQC,r8)/))


gppDTQC, gppNTQC. Or it doesn't matter since recoDTQC is always the same as gppDTQC, and recoNTQC is the same as gppNTQC?

Typo -- very good catch.

XueliHuo · 2023-08-17T19:45:52Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+   tower%gppNTQC = maxgoodqc + 100
+   tower%gppDTQC = maxgoodqc + 100
+   tower%recoNTQC = maxgoodqc + 100
+   tower%recoDTQC = maxgoodqc + 100


One last comment on lines 1168-1171. I understand the design logic you have. Say if I do would like to use the valid gpp and reco when the gap_filled is false at the HH or HR time resolution. Since the QC assigned to them in lines 1168-1171 are the same values as QC assigned when gpp, reco are negative. Do you think it is worthwhile assigning a different QC in lines 1168-1171 to distinguish the two situations so that I can pick a QC threshold to allow the valid gpp and reco when gap_filled is false at the HH or HR time resolution assimilated ?

If you want to use HH/HR GPP and RECO values, just set gap_filled=.true. Then all of the GPP and RECO values will be sent to your obs_seq file. By default they are assigned a QC=2 in the obs_seq file.

Also, to be clear, when gap_filled=.true. both gap_filled and 'true' observations are sent to the obs_seq file. 'True' observations are never excluded from the obs_seq file.

Adding better documentation on how user can adjust values that convert fractional QC to integer QC and also empirically derived variables related to uncertainty estimation for missing values

braczka · 2023-08-24T14:29:38Z

@XueliHuo Absolutely no rush on this review, but I wanted to follow up to see if there was anything else I needed to do (through documentation, code changes) to address any of your concerns? The last few commits in this PR hopefully fix inaccuracies in the documentation that you identified.

hkershaw-brown

Hi Brett,

Since the science review seems to be wrapping up, a few comments:

lower case for namelist and program names. (I put suggestions in for this, but ran out of steam at ~20. Just do a global find and replace). Fortran is case insensitive, various filesystems may or may not be, people may be using case-sensitive tools to generate scripts and namelists and chaos ensues. So lower case only for namelists program names.

add fluxnetfull_to_obs to .gitignore so people don't accidentally commit the executable to the repo.

You've got some duplicate code, from the obs_utilities_mod. Rather than duplicate these, just use the obs_utilities_mod versions:

use obs_utilities_mod, only: add_obs_to_seq, create_3d_obs

Cheers,
Helen

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.nml

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.rst

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.rst

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

fixes #537

…f-rttov13 Added get_channel to obs_def_rttov13_mod.f90

mjs2369 · 2023-09-14T20:29:55Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+use         types_mod, only : r8, MISSING_R8
+
+use     utilities_mod, only : initialize_utilities, finalize_utilities, &
+                              register_module, error_handler, E_MSG, E_ERR, &


Suggested change

register_module, error_handler, E_MSG, E_ERR, &

error_handler, E_MSG, E_ERR, &

mjs2369 · 2023-09-14T20:31:12Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+
+use      location_mod, only : VERTISHEIGHT
+
+use  obs_sequence_mod, only : obs_sequence_type, obs_type, read_obs_seq, &


Suggested change

use obs_sequence_mod, only : obs_sequence_type, obs_type, read_obs_seq, &

use obs_sequence_mod, only : obs_sequence_type, obs_type, &

mjs2369 · 2023-09-14T20:36:03Z

observations/obs_converters/Ameriflux/fluxnetfull_to_obs.f90

+! Initialize two empty observations - one to track location
+! in observation sequence - the other is for the new observation.
+
+iunit = open_file(text_input_file, 'formatted', 'read')


@braczka this call to open_file is missing a corresponding call to close_file

Gotcha -- took care of these in latest commits.

XueliHuo · 2023-09-14T23:53:11Z

@XueliHuo Absolutely no rush on this review, but I wanted to follow up to see if there was anything else I needed to do (through documentation, code changes) to address any of your concerns? The last few commits in this PR hopefully fix inaccuracies in the documentation that you identified.

Hi Brett, I went through the questions I posted and the response/modifications you provided/made. I approve this pull request.

hkershaw-brown

Switching my review to approve so I don't block anything.
Brett addressed my duplicate code and capital F comments.

@mjs2369 you're in charge of the release!

…t_converter

braczka · 2023-09-15T21:29:07Z

This is good on my end, @mjs2369 release at your convenience. Thanks all.

bug fix: remove cloud_overlap integer from list of logicals to query

braczka added 8 commits May 15, 2023 18:26

Generating new files for fluxnetfull obs converter

d7ae00c

Creating new observation converter for site level eddy covariance flux tower data (NEE,GPP,RECO,LH,SH). The existing Ameriflux_L4 converter is outdated and new database has new flux options and time aggregates

Adding new flux variables to converter

48d163b

Updating decode_header in Flux obs converter

9ef0a40

Modify stringparse for Fluxnet obs

794c850

Added GPP and ER obs to Fluxnet converter

a746d46

Update deprecated documentation for level4_to_obs

e383f80

Syntax changes to to fluxnetfull to compile

97cf892

Necessary changes in syntax to allow fluxnetfull_to_obs.f90 to compile correctly. Also providing proper changes to support files input.nml and quickbuild.sh within work directory

Edit flux converter to compile and run

3228690

fluxnetfull_to_obs converter compiles and runs testing on hourly flux fullset data. Header now includes timestamp which is treated differently than real values

braczka added Bug Something isn't working Enhancement New feature or request CLM Community Land Model labels May 24, 2023

braczka requested review from hkershaw-brown and mjs2369 May 24, 2023 20:01

braczka self-assigned this May 24, 2023

braczka added 8 commits May 25, 2023 15:33

Added option to ingest multiple time resolutions

ad0be7b

Adds ability to use HH,HR,DD,WW,MM,YY flux data where test file format changes

Fluxnet convert ingests multiple time averages

c026524

Expanded functionality such that fluxnet converter can take in HH,HR,DD,WW,MM and YY time averages

Add edit and warnings for coarse time resolution

c031964

Added more fixes and warnings to make code compatiable with coarser time resolution. File formatting changes based on time resolution.

Fixes to flux QC and flux uncertainty values

c8fbef6

QC values change from integers to fraction depending upon time resolution. Added code to convert fraction back to integer. Provided empirical fix when uncertainty values for H,LE and NEE are missing.

Updating work input.nml

5075b56

Merge remote-tracking branch 'upstream/main' into fluxnet_converter

f2a0ae2

Add documentation for fluxnetfull_to_obs converter

034ae1c

Corrections for fluxnetfull_to_obs documentation

674c9c6

braczka marked this pull request as ready for review August 15, 2023 15:15

braczka requested a review from XueliHuo August 15, 2023 16:30

XueliHuo reviewed Aug 16, 2023

View reviewed changes

Correcting misleading comments in code

7143044

XueliHuo reviewed Aug 17, 2023

View reviewed changes

Correcting typos in flux QC values

023b66d

XueliHuo reviewed Aug 17, 2023

View reviewed changes

Fixing typos and adding user documentation

9d68fe5

Adding better documentation on how user can adjust values that convert fractional QC to integer QC and also empirically derived variables related to uncertainty estimation for missing values

hkershaw-brown requested changes Aug 24, 2023

View reviewed changes

braczka and others added 6 commits August 29, 2023 14:32

changing fluxnetfull naming to lowercase

2d8210b

Removing duplicate obs_utilities_mod code

2dd12c5

fix: remove cloud_overlap integer from list of logicals to query

3dcd93e

fixes #537

Merge branch 'main' into fluxnet_converter

b48d15d

Added get_channel to obs_def_rttov13_mod.f90

2e1fecd

Merge pull request #540 from ann-norcio/fix-radiance_obs_seq_to_netcd…

cd5dc74

…f-rttov13 Added get_channel to obs_def_rttov13_mod.f90

mjs2369 reviewed Sep 14, 2023

View reviewed changes

last of the capital -> lower case changes for fluxnet

6432a86

hkershaw-brown approved these changes Sep 15, 2023

View reviewed changes

braczka added 3 commits September 15, 2023 15:11

Removing some unnessary function includes

e6f582a

Merge remote-tracking branch 'upstream/main' into fluxnet_converter

fb1af87

Merge remote-tracking branch 'upstream/fluxnet_converter' into fluxne…

883a768

…t_converter

mjs2369 and others added 2 commits September 18, 2023 12:27

Merge pull request #541 from NCAR/fix-int-log-rttov13

946fd26

bug fix: remove cloud_overlap integer from list of logicals to query

conf.py bump and CHANGELOG update

06c13cf

mjs2369 merged commit 6eeb453 into main Sep 18, 2023
3 checks passed

hkershaw-brown deleted the fluxnet_converter branch November 7, 2023 20:11

braczka mentioned this pull request Dec 7, 2023

Location and format of L4 Ameriflux Observations have changed #339

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fluxnet observation converter #478

Fluxnet observation converter #478

braczka commented May 24, 2023 •

edited by mjs2369

Loading

braczka commented May 24, 2023

mjs2369 commented May 24, 2023

braczka commented Aug 15, 2023

braczka commented Aug 15, 2023

XueliHuo Aug 16, 2023

braczka Aug 16, 2023

XueliHuo Aug 17, 2023

XueliHuo Aug 16, 2023

braczka Aug 16, 2023

XueliHuo Aug 17, 2023

XueliHuo Aug 17, 2023

braczka Aug 17, 2023

XueliHuo Sep 14, 2023

XueliHuo Aug 17, 2023

braczka Aug 17, 2023

XueliHuo Aug 17, 2023

braczka Aug 17, 2023

braczka Aug 17, 2023

XueliHuo Sep 14, 2023

braczka commented Aug 24, 2023

hkershaw-brown left a comment

mjs2369 Sep 14, 2023

mjs2369 Sep 14, 2023

mjs2369 Sep 14, 2023

braczka Sep 15, 2023

XueliHuo commented Sep 14, 2023

hkershaw-brown left a comment

braczka commented Sep 15, 2023

	register_module, error_handler, E_MSG, E_ERR, &
	error_handler, E_MSG, E_ERR, &


		use location_mod, only : VERTISHEIGHT

		use obs_sequence_mod, only : obs_sequence_type, obs_type, read_obs_seq, &

Fluxnet observation converter #478

Fluxnet observation converter #478

Conversation

braczka commented May 24, 2023 • edited by mjs2369 Loading

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

braczka commented May 24, 2023

mjs2369 commented May 24, 2023

braczka commented Aug 15, 2023

braczka commented Aug 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braczka commented Aug 24, 2023

hkershaw-brown left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XueliHuo commented Sep 14, 2023

hkershaw-brown left a comment

Choose a reason for hiding this comment

braczka commented Sep 15, 2023

braczka commented May 24, 2023 •

edited by mjs2369

Loading