cdo cmor operator For the CMIP 6 project

cdo cmor operator For the CMIP 6 project Fabian Wachsmann Deutsches Klimarechenzentrum (DKRZ) wachsmann@dkrz. de

cdo cmor operator 2 CMIP 6 CMIP standard • Recent era of Coupled Model Intercomparison Project • Convention about climate data format accepted in CMIP • https: //cmor. llnl. gov defines cdo • Collection of operators to process climate data • http: //code. zmaw. de/ projects/cdo/ creates is used by „Systematic analysis across models only easy to do if model output is written in - a common format - with files structured similary - and with sufficient metadata uniformly stored“ 27. 10. 2020

3 cdo cmor operator CMIP standard ncdump -h test. nc 27. 10. 2020

cdo cmor operator 4 CMIP standard ‚Raw‘ data Converted data 27. 10. 2020

5 cdo cmor operator CMIP standard ncdump -h test. nc Restructuring • Generating bounds • Adding attributes Renaming 27. 10. 2020

6 cdo cmor operator CMOR Climate Model Output Rewriter • What is CMOR? • Why no other tool? • Why cdo cmor? • CMOR ensures that output is CMIP compliant • Without CMIP standard no ESGF publication 27. 10. 2020

7 cdo cmor operator CMOR Climate Model Output Rewriter • What is CMOR? • Why no other tool? • Why cdo cmor? • • CMOR Use synergies • ensures Avoid repeating that work output is CMIP compliant • Without CMIP standard no ESGF publication 27. 10. 2020

8 cdo cmor operator CMOR Climate Model Output Rewriter All CMOR calls • What is CMOR? • Why no other tool? • Why cdo cmor? cmor_setup(param); cmor_dataset_json(param); cmor_load_table(param); cmor_set_table(param); cmor_axis(param); cmor_grid(param); cmor_set_grid_mapping(); cmor_time_varying_grid_coordinate(); cmor_zfactor() cmor_variable(param); cmor_set_deflate(); cmor_set_variable_attribute(); cmor_create_output_path(); cmor_write(param); cmor_close(param); are included by one cdo cmor operator 27. 10. 2020

cdo cmor operator 9 Project dependent check of attributes I. MIP-table II. Info-tables III. Mapping-table Mapping of variables Aspects of the cdo cmor Auxillary variables and bounds calculation Operator for producing CMIP compliant output Append. Mode 27. 10. 2020

10 cdo cmor operator MIP-Table „variable_entry“: { „ccb“: { „frequency“: „mon“, „modeling_realm“: “atmos“, cdo cmor, cmip 6_mip_tables_325/CMIP 6_Amon. json, i=infotable. txt, mt=mapping_table. txt test. nc Info table institude_id=MPI-M source_id=MPIESM-1 -2 -HR experiment_id=historical realization_index=1 physics_index=1 initialization_index=1 forcing_index=1 activity_id=CMIP 6 Mapping table &parameter cmor_name=clt c=164 units="1“ cm="m" / &parameter cmor_name=uas c=165 units="m s-1" cm="m" / &parameter cmor_name=vas c=166 units="m s-1" cm="m" / &parameter cmor_name=tas c=167 units="K“ cm="m" / 27. 10. 2020

11 cdo cmor operator Hands on 1. Preparation for hands on # connect to mistral ssh –X <usernr>@mistral. dkrz. de #create home directory mkdir ~/cdocmor_handson cd cdocmor_handson #copy examples directory cdo. Dir=/work/bm 0021/cdo_incl_cmor/ cp -r ${cdo. Dir}examples. / cd examples #link cdo cmor installation alias cdo=${cdo. Dir}cdo_handson_cmor 3 (alias cdo=${cdo. Dir}cdo_recent_cmor 3) 27. 10. 2020

cdo cmor operator 12 Hands on 2. Interface Minimum input for cdo cmor are MIP-table and infile: #abstract: #cdo cmor, MIP-table infile #example: cdo cmor, cmip 6_mip_tables_325/CMIP 6_Amon. json example_interface. nc • Parameter one must be the MIP-table • The outfile is generated by CMOR and must not be specified • Name and path of the outfile are generated according to project depending templates Filename Filepath CMOR 2 <variable_id><table_id><source_id> <experiment_id><variant_label> <grid_label> <activity_id><product><institute_id><model_id> <experiment_id><frequency><realm><variable_id> <member> CMOR 3 <variable_id><table_id><model_id> <experiment_id><member> <mip_era><activity_id><institution_id><source_id> <experiment_id><variant_label><table_id> <variable_id><grid_label><version> 27. 10. 2020

13 cdo cmor operator Hands on 2. Interface #example: cdo cmor, cmip 6_mip_tables_325/CMIP 6_Amon. json example_interface. nc • Parameter one must be the MIP-table • All further parameters are keyvalues with format: key=value(, value). • Keyword cmor_name (cn) selects a subset of variables to be processed. • All keys have short forms, e. g. cn for cmor_name. #examples: cdo cmor, cmip 6_mip_tables_325/CMIP 6_Amon. json, cn=tas example_interface. nc cdo cmor, cmip 6_mip_tables_325/CMIP 6_Amon. json, cn=tas, uas example_interface. nc 27. 10. 2020

cdo cmor operator 14 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 15 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 16 CMIP data standard I. MIP table cdo cmor, cmip 6_mip_tables/CMIP 6_Amon. json example_interface. nc • A MIP-table is CMOR-readable and contains the CMIP data standard • This standard is expressed by a header, variable and dimension information • MIP-tables are subdivided by variable aggregation frequencies (and realm). e. g. this is a subset of CMIP 6 tables: CMIP 6_Amon. json Extract of CMIP 6_Amon. json: CMIP 6_Ayr. json „header“: { CMIP 6_Omon. json „data_specs_version“: „ 01. 00. 06“, CMIP 6_Oyr. json } „variable_entry“: { „ccb“: { „frequency“: „mon“, „modeling_realm“: “atmos“, 27. 10. 2020

cdo cmor operator 17 Hands on CMIP data standard I. MIP table cdo cmor, cmip 6_mip_tables/CMIP 6_Amon. json example_interface. nc Exercise: Examine the MIP-tables, especially CMIP 6_Amon. json: 1. What is the cmor_name of variable with long_name „Air Temperature“ and what is the name of its vertical axis? 2. Your atmosphere model produces a variable „Not. In. Request“ in monthly frequency which is not in the data request. You would like to convert it to CMIP standard as well. What do you do? 3. Which table contains variable with cmor_name „opottemppmdiff“ and what is it? 1. ta and plev 17. 2. Copy tas and adopt standard for Not. In. Request 3. CMIP 6_Oyr and from comment: “Tendency of heat content for a grid cell from parameterized mesoscale eddy diffusion. Reported only for models that use potential temperature as prognostic field. ", 27. 10. 2020

cdo cmor operator 18 CMIP data standard I. MIP table cdo cmor, cmip 6_mip_tables/CMIP 6_Amon. json example_interface. nc • A MIP-table is CMOR-readable and contains the CMIP data standard • The information of the MIP-table is not available to the operator, only for CMOR • It does not read the table once more because • Avoid repetition of CMOR´s work • Much information is gained by CDO´s data interface • The user should be aware of the user´s input format and the required format instead of just fitting the data until CMOR works Some information has to be passed to the operator allthough it is available from the MIP-table. 27. 10. 2020

cdo cmor operator 19 CMIP data standard • A CMOR-variable is the unique combination of the cmor_name and the corresponding MIP-table which includes the cmor_name • Air temperature can have different requirements when monthly aggregated ( CMIP 6_Amon. json ) compared to when daily aggregated (CMIP 6_Aday. json). • The outfile variable name does not need to be the cmor_name. 27. 10. 2020

cdo cmor operator 20 CMIP (meta) data standard CMIP 6_CV. json A Controlled Vocabulary (CV) is a special MIP-table and defines - required and optional CMIP attributes - allowed values for attributes - restrictions resulting from a setting of attributes (e. g. min. simulation years of an experiment) - whether additional attributes must be specified for special cases (e. g. parent attributes ) CMIP 6_CV. json: { „CV“: { „required_global_attributes“: [ „Conventions“, „activity_id“, „creation_date“, „experiment_id“, 27. 10. 2020

cdo cmor operator 21 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 22 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 23 Project dependent check of attributes Hands on 3. Metadata standard • Values of required global attributes have to be passed to the operator by info tables • They are specified via keyword info • They contain additional control keywords. • The default info table is hidden and named „. cdocmorinfo“. It is taken from the current working directory (cwd). #display default information cat. cdocmorinfo The operator can create the path to the MIP-table with the help of keywords mip_table_dir and project_id so that it is sufficient to call: cdo cmor, Amon example_interface. nc , which is equivalent to: cdo cmor, cmip 6_mip_tables/CMIP 6_Amon. json example_interface. nc Info file format conventions are: • One keyvalue per line • key=value is of format word=string • String which contains commas or blanks must be specified in quotes 27. 10. 2020

cdo cmor operator 24 Project dependent check of attributes 3. Metadata standard Most important attributes… project_id experiment_id source_id institution_id CMIP 6, CMIP 5, CORDEX historical, pi. Control, amip MPIESM-1 -2 -HR, MPIESM-1 -2 -LR MPI-M, AWI . . and coupled attributes: experiment_id source_id institution_id experiment, member, required-time -units source, calendar, grid institution, contact 27. 10. 2020

cdo cmor operator 25 Project dependent check of attributes Hands on 3. Metadata standard A tool to create a cdocmorinfo file is provided at https: //c 6 dreq. dkrz. de/cdocmorinfo 27. 10. 2020

cdo cmor operator 26 Project dependent check of attributes Hands on 3. Metadata standard Exercise: 1. Create your own cdocmorinfo on c 6 dreq. dkrz. de/cdocmorinfo. html 2. Copy that file to CWD/. cdocmorinfo 3. Repeat: cdo cmor, Amon example_interface. nc 4. Have a look at differences between the files with ncdump –h. 27. 10. 2020

cdo cmor operator 27 Project dependent check of attributes 3. Metadata standard ‘Forced‘ (from CV) attributes in example experiments attributesexperiment_id 1 pct. CO 2 amip ssp 585 activity_id CMIP Scenario. MIP experiment 1 percent per year increase in CO 2 AMIP update of RCP 8. 5 based on SSP 5 sub_experiment_id none parent_activity_id CMIP no parent CMIP parent_experiment_id pi. Control no parent historical 27. 10. 2020

cdo cmor operator 28 Project dependent check of attributes 3. Metadata standard ‘Forced‘ (from CV) attributes in example experiments attributesexperiment_id 1 pct. CO 2 amip ssp 585 activity_id CMIP Scenario. MIP experiment 1 percent per year increase in CO 2 AMIP update of RCP 8. 5 based on SSP 5 sub_experiment_id none parent_activity_id CMIP no parent historical parent_experiment_id pi. Control If an experiment is based on a parent_experiment, one has to specify 6 parent attributes and also branch_time and branch_method. 27. 10. 2020

cdo cmor operator 29 Project dependent check of attributes 3. Metadata standard Project dependence of attributes attributeproject CMIP 6 CMIP 5 CORDEX MIP: activity_id project_id model: source_id model_id Institut: institution_id institute_id nominal_resoltuion cordex_domain The operator can control your input for the project you set 27. 10. 2020

cdo cmor operator 30 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table & CV II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 31 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table & CV II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 32 Mapping of variables 4. Conversion to CMIP data standard From model variable… …to CMOR variable mapping Mapping INFILE Selection cn • per command line • with a table CMOR ? 27. 10. 2020

cdo cmor operator 33 Mapping of variables 4. Conversion to CMIP data standard Adjustable variable and mapping attributes Only settable in MIP-table In Cmdline, only one value and one of both possible Keyword Short Value format name Default project_mip_table pmt String. name n String. Value is the variable name in infile. code c Three digits Integer. GRIB code of the variable in infile to be selected. units u String. Value must be readable by library udunits. E. g. : „W m-2“ cell_methods cm First character of string. One of ‘mean‘, ‘point‘, ‘climate‘, ‘diurnal‘ and ‘none‘. ‘mean‘ positive p First character of string. Value is one of ‘upward‘, ‘downward‘ and ‘ ‘ (blank) variable_comment vc String. 27. 10. 2020

cdo cmor operator 34 Mapping of variables Hands on 4. Conversion to CMIP data standard Assume, model output is available in GRIB-format and contains 2 m temperature. The corresponding cmor_name istas and should be processed. #One option is to use commandline mapping: cdo cmor, Amon, cn=tas, c=167, u=K, cm=m example_mapping. grb Note that • in this case, the infile variable selector is code=167 • command line mapping always requires cmor_name and one variable selector • if more than one value is specified, just the first one is processed 27. 10. 2020

cdo cmor operator 35 Mapping of variables Hands on 4. Conversion to CMIP data standard Assume, model output is available in GRIB-format and contains 2 m temperature. The corresponding cmor_name istas and should be processed. #Another option is to create a mapping table file mapping_table. txt and use it: echo ‘&parameter cn=tas c=167 u=K cm=m /‘ >>mapping_table. txt cdo cmor, Amon, cn=tas, mt=mapping_table. txt example_mapping. grb Note that • cmor_name and project_mip_table are the line selector keywords: A mapping table line which contains the same values for cmor_name and project_mip_table like specified in the command line is applied. • An infile variable selector can be specified in this mapping table line: name is taken for net. CDF files, a GRIB- code is taken for GRIB-files. • If no cmor_name is specified in the command line but a mapping table is specified: All lines are tested and all infile variable selector available from these lines are tested with infile variables. 27. 10. 2020

cdo cmor operator 36 Mapping of variables Hands on 4. Conversion to CMIP data standard Assume, model output is available in GRIB-format and contains Near-surface temperature. The corresponding cmor_name istas and should be processed. #Another option is to create a mapping table file mapping_table. txt and use it: echo ‘&parameter cn=tas c=167 u=K cm=m /‘ >>mapping_table. txt cdo cmor, Amon, cn=tas, mt=mapping_table. txt example_mapping. grb Mapping table format conventions are similar to a fortran namelist: • Each line begins with ‚&parameter‘ • keyvalues are separated by a blank • In general, just one value per keyword allowed 27. 10. 2020

cdo cmor operator 37 Mapping of variables Hands on 4. Conversion to CMIP data standard Exercise: 1. Convert the variable included in example_T_2 M. nc to CMIP standard. Use mapping_table. txt and overwrite the infile variable selector. 2. Convert the variable included in example_celsius. grb to CMIP standard. Use mapping_table. txt and adapt the correct model variable unit. 1. : cdo cmor, Amon, cn=tas, n=T_2 M, mt=mapping_table. txt example_T_2 M. nc 2. : cdo cmor, Amon, cn=tas, mt=mapping_table. txt, u=„deg. C“ example_celsius. nc #Convert many variables at once with a complete mapping table mt. PERFECT. txt: cdo cmor, Amon, mt=mt. PERFECT. txt example_collect. grb #Note that you can also specify mapping_table_dir and mapping_table in info files. 27. 10. 2020

cdo cmor operator 38 Mapping of variables Hands on 4. Conversion to CMIP data standard A tool is provided at c 6 dreq. dkrz. de to create and download mapping tables. After selecting an ESM-sub model of a DICAD partner, one can edit a set up for all variables requested from CMIP 6. 27. 10. 2020

cdo cmor operator 39 Mapping of variables 4. Conversion to CMIP data standard Control keywords for special requests processed like variable attributes: Name Short name Value format Default scalar_z_coord inate szc String. Value has the shape: ‚axisname_axisvalue‘, e. g. : ‚height 2 m_1. 5‘ The requested value of this coordinate is automatically taken. character_axis ca String. Value is a character axis, whose values needs to be registered as well in info files. E. g. : basin. * No default * basin=atlantic, pacific, indic must be in. cdocmorinfo. 27. 10. 2020

cdo cmor operator 40 Mapping of variables Hands on 4. Conversion to CMIP data standard Assume, your model calculates temperature at 3 m height and not on 2 m. Exercise: 1. Convert the variable included in example_T_3 M. nc to CMIP standard. Pass the correct height value to the operator 1. : cdo cmor, Amon, szc=height 2 m_3, mt=mapping_table. txt example_T_3 M. nc 27. 10. 2020

41 cdo cmor operator Hands on 5. CDO options Note that the operator can be executed in silent mode via cdo option –s or in verbose mode via cdo option -v: cdo –v cmor, Amon example_interface. nc cdo –s cmor, Amon example_interface. nc Note that operators can be chained: cdo –cmor, Amon –addc, 13 example_interface. nc 27. 10. 2020

cdo cmor operator 42 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 43 Append. Model simulation year 5. Operational application Workflow 1850 1851 1852 1853 Postprocessing 1850_ post CMIP 6 -compliant example variable tas_ 185001_185012 1851_ post 1852_ post tas_ 185001_185112 185001_185212 27. 10. 2020

cdo cmor operator 44 Append. Mode 5. Operational application Control attributes settable in infofiles Name short Value format name Default output_mode om Character. Either r for REPLACE or a for append. r last_chunk lc Strings. Chunk file names in the order of requested variables. Read from chunkdescription-file. max_size ms Integer. Upper limit for the size of an 2 output file in gb. 27. 10. 2020

cdo cmor operator 45 Hands on Append. Mode 6. Append mode From earlier examples, you should have created a tas-file for year 2001. Now you want to append year 2002: DRS=CMIP 6/CMIP/MPI-M/MPIESM-1 -2 - HR/historical/ri 1 ip 1 f 1/Amon/tas/gn/v 20171010/ cdo cmor, Amon, om=a, mt=mapping_table. txt, lc=${DRS}tas_Amon_MPIESM-1 -2 -HR_historical _r 1 i 1 p 1 f 1_gn_200101 -200112. nc example_append 2002. grb During the latter call, the outfile path and filename was saved in „CHUNK_FILE_tas_Amon_MPI-ESM_amip_r 1 i 1 p 1. txt“. This is the description for the default chunk in append mode. Now you can append the next year without defining lc: cdo cmor, Amon, om=a, mt=mapping_table. txt example_append 2003. grb 27. 10. 2020

cdo cmor operator 46 Project dependent check of attributes Auxillary variables and bounds calculation I. MIP-table II. Info-tables III. Mapping-table Development of the cdo cmor Operator for producing CMIP compliant output Mapping of variables Append. Mode 27. 10. 2020

cdo cmor operator 47 Auxillary variables and bounds calculation 6. Internal functions ncdump –h : Calculated according to frequency (in juldates for lower than day) and cell_methods (not for point or none) Calculated by interpolation if grid_type is lon_lat or curvilinear. 27. 10. 2020

Thank you! Questions at: Fabian Wachsmann Deutsches Klimarechenzentrum (DKRZ) wachsmann@dkrz. de

cdo cmor operator 49 Mapping für Variablenattribute Von der Modellvariablen. . INFILE . . zur CMOR Variablen mapping Auswahl Mapping Cmdlinemapping cdo cmor, Amon, cn=tas Infile cdo cmor, Amon, cn=tas, mt=maptabfile, n=t 2 m Infile &parameter name=temp 2 cmor_name=tas units=K cell_methods=mean 27. 10. 2020

cdo cmor operator 50 Mapping für Variablenattribute Von der Modellvariablen. . zur CMOR Variablen mapping cdo cmor, Amon, cn=tas Infile Fall 1, usecases: • Modelloutput enthält nahezu CMIP-konforme Variablen: Name, (UD-UNIT bekannte) Einheit, Achsen… • Interesse besteht an einer Subauswahl an Variablen Auswahl per cmor_name genügt 27. 10. 2020

cdo cmor operator 51 Mapping für Variablenattribute Von der Modellvariablen. . zur CMOR Variablen mapping cdo cmor, Amon, cn=tas, mt=maptabfile Infile Fall 2, usecases: • Variablen des Modelloutputs verfügen über wenig/keine attribute (bspw. GRIB Format), diese sind aber gesammelt abrufbar. • Es soll sämtlicher Modelloutput konvertiert werden Eine Mapping-Tabelle ist von Vorteil. &parameter name=temp 2 cmor_name=tas units=K cell_methods=mean &parameter name=ahfl cmor_name=hfls units=´W m-2´ pos=d Für DICAD-Modelle können auf c 6 dreq. dkrz. de Mapping. Tabellen erstellt und heruntergeladen werden 27. 10. 2020

cdo cmor operator 52 Mapping für Variablenattribute Von der Modellvariablen. . zur CMOR Variablen mapping cdo cmor, Amon, cn=tas, mt=maptabfile, n=t 2 m Infile Fall 3, usecases: • Eine Mapping-Tabelle eignet sich zur Konvertierung des gewünschten Modelloutputs, die Namen der Modellvariablen weichen jedoch von den dort eingetragenen ab. Kombiniertes Mapping mit Tabelle und über die Cmdline • Einstieg in die mt über cn • Auswahl der Infilevariable über n 27. 10. 2020

53 cdo cmor operator Projektabhängiger Check von Attributen Project dependent check of attributes CMIP 5/CMOR 2 CMIP 6/CMOR 3 In der Funktion cmor_dataset werden alle notwendigen Attribute übergeben. Deshalb können diese überprüft werden, bevor ein Datensatz eingelesen oder CMOR aufgerufen wird. CMOR 3 ermöglicht notwendige Attribute „selbst“ in einer CMIP 6_CV. json zu definieren. Hilfestellung: • Formular für CMOR-akzeptables Dataset • „branch_time_in_parent“ und „branch_time_in_child“ werden aus dem Anfangsdatum und Branch-datum des parent-Experiments berechnet. „CV“: { „required_global_attributes“: [ „Conventions“, 27. 10. 2020

cdo cmor operator 54 CMIP 6 Attribute CMIP 5 Äquivalent Beispielwert (CMIP 6) „Conventions“ CF-1. 7 CMIP-6. 0 „activity_id“ CMIP „creation_date“ „data_specs_version“ 01. 00. 11 „experiment“ pre-industrial control „experiment_id“ pi. Control „forcing_index“ 1 „frequency“ mon „further_info_url“ http: //furtherinfo. es-doc. org/bspwert „grid“ „grid_label“ gr „initialization_index“ 1 „institution“ Max Planck Institute for Meteorology, Hamburg 20146, Germany „institution_id “ MPI-M 27. 10. 2020

cdo cmor operator 55 CMIP 6 Attribute CMIP 5 Äquivalent Beispielwert (CMIP 6) „license “ langer String „mip_era “ CMIP 6 „nominal_resolution “ 1 km „physics_index“ 1 „product“ model-output „realization_index “ mon „realm“ atmos „source“ MPIESM 1. 2 -HR (2017): atmos: ECHAM 6. 3 (spectral T 127; 384 x 192 longitude/latitude; …) „source_id“ MPIESM-1 -2 -HR „source_type “ ESM 27. 10. 2020

cdo cmor operator 56 CMIP 6 Attribute CMIP 5 Äquivalent Beispielwert (CMIP 6) „source_type “ ESM „sub_experiment“ none „sub_experiment_ id“ none „table_id“ Amon „tracking_id“ hdl: 21. 14100… „variable_id“ tas „variant_label“ r 1 i 1 p 1 f 1 27. 10. 2020