Observation Screening Kertsz Sndor 3 DVARODB Training Budapest

Observation Screening Kertész Sándor 3 D-VAR/ODB Training Budapest, 7 June, 2006 3 D-VAR/ODB Training, Budapest

OUTLINE • ODB primer • Observation data flow • Screening QC and decisions in ODB • Screening inputs • How is screening working? • Output listings • Exercises 7 June, 2006 3 D-VAR/ODB Training, Budapest

ODB primer • • ODB is Observational Database Basic data structure is called table. Tables is made up of data columns Set of tables (with relations) forms a database type. E. g. ECMA that contains all the observations to be used in a DA system Information on tables, columns etc. • http: //www. cnrm. meteo. fr/gmapdoc/meshtml/DOC_odb/odb. html • In „Doc” directory on regatta: Assimilation_Chapter 9_v 2. 3. pdf • ODB training! 7 June, 2006 3 D-VAR/ODB Training, Budapest 3

ODB primer • • An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!) ODB data is divided into pools for parallel runs (to ensure load balance) To look into ODB you need to run the viewer with your view! The view define the retrieval! 7 June, 2006 3 D-VAR/ODB Training, Budapest 4

ODB primer • View definition (e. g. to retrieve AIREP data) View name Retrieved CREATE VIEW myview AS columns SELECT obstype, lat, lon, varno, obsvalue, status. blacklist@body FROM hdr, body Involved tables WHERE obstype == 2 Selection criteria 7 June, 2006 3 D-VAR/ODB Training, Budapest 5

ODB primer • The concept of sub-bases was invented to have the choice for handling observation types separately. E. g. : ECMA. conv, ECMA. amsua, ECMA. amsub etc… • • These ODB sub-bases must be merged into a new ODB Many applications are working on this kind of ODB that is called virtual basis 7 June, 2006 3 D-VAR/ODB Training, Budapest 6

Observation Screening • • • The observation screening is the last step in the preprocessing chain to produce observation inputs for the variational analysis Screening performs various quality control checks on data ALADIN screening is very similar to the global (ARPEGE) one Screening uses an ECMA ODB as input The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow … 7 June, 2006 3 D-VAR/ODB Training, Budapest 7

Obs dataflow: BATOR • • BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools We need: #pools = # CPUs for further applications! These ODB sub-bases must be merged into a new ECMA ODB using programme SHUFFLE Further applications are working on this ODB virtual basis 7 June, 2006 3 D-VAR/ODB Training, Budapest 8

Obs dataflow: LAMFLAG • • The resulting ODB is not suitable for ALADIN screening since it is not working with observations outside the C+I zone Abort in GEFGER! Solution: the input ODB must be restricted to the C+I zone programme LAMFLAG writes status flags into ODB E C+I zone@hdr=0 zone@hdr=1 7 June, 2006 3 D-VAR/ODB Training, Budapest 9

Obs dataflow: LAMFLAG namelists &NAMFCNT LOBSONLY=. FALSE. /END &NAMFGEOM ELAT 0=46. 2447006399999978, If. TRUE. : selection is based only on observation filter! Observation filter &NAMFOBS ELON 0=17. 00000000, LSYNOP=. T. , ELATC=46. 2447005948822678, LAIREP=. T. , ELONC=16. 999982550491, EDELX=12176. 1563281414165, EDELY=12176. 1563281414165, Model geometry settings LSATOB=. T. , LDRIBU=. F. , LTEMP=. T. , NDLUN=1, LPILOT=. T. , NDGUN=1, LSATEM=. T. NDLUX=229, LPAOB=. F. , NDGUX=205, LSCATT=. F. , Z_CANZONE=1500. , /END LVAR=. T. , LNEWGEOM=. F. , /END 7 June, 2006 Used for screening 3 D-VAR/ODB Training, Budapest 10

Obs dataflow: SHUFFLE reduction • • LAMFLAG puts flags into ODB but does not reduce it! SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB Then another SHUFFLE run is needed to put sub-bases together again (merging) This results in a virtual base ECMA ODB again that is now ready for used as screening input 7 June, 2006 3 D-VAR/ODB Training, Budapest 11

Obs dataflow: Screening • • Screening puts its quality control flags and decisions into the input ECMA ODB QC flags, decisions for reports (in table hdr): • status_t • event 1 report_event 1_t • blacklist report_blacklist_t status_t active bit 1 passive bit 1 rejected bit 1 blacklisted bit 1 …. 7 June, 2006 Passive and blacklisted are a subset of rejected! report_event 1_t no_data bit 1 all_rejected bit 1 bad_practice bit 1 rdb_rejected bit 1 rdb_activated bit 1 whitelist_activated bit 1 horipos_outrange bit 1 vertpos_outrange bit 1 time_outrange bit 1 redundant bit 1 land bit 1 sea bit 1 …. . report_blacklist_t obstype bit 1 statid bit 1 codetype bit 1 instype bit 1 date bit 1 time bit 1 lat bit 1 lon bit 1 stalt bit 1 scanpos bit 1 retrtype bit 1 …. . 3 D-VAR/ODB Training, Budapest 12

Obs dataflow: Screening • QC flags, decisions and additional infromation for observed values (in table body): datum_event 1_t • status_t vertco_missing bit 1 obsvalue_missing bit 1 fg_missing bit 1 • event 1 report_event 1_t rdb_rejected bit 1 • blacklist datum_blacklist_t rdb_activated bit 1 whitelist_activated bit 1 bad_practice bit 1 • fg_depar = obs-guess vertpos_outrange bit 1 status_t active bit 1 passive bit 1 rejected bit 1 blacklisted bit 1 …. 7 June, 2006 reflevel_outrange bit 1 fg 2 big bit 1 depar 2 big bit 1 obs_error 2 big bit 1 datum_redundant bit 1 level_redundant bit 1 …. datum_blacklist_t obstype bit 1 statid bit 1 codetype bit 1 instype bit 1 date bit 1 time bit 1 lat bit 1 lon bit 1 …. 3 D-VAR/ODB Training, Budapest 13

Obs dataflow: Compression • • • After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion A CCMA ODB consists of only those tables and column that are needed for the analysis The resulting CCMA ODB after SHUFFLE contains only the active reports and observations! This CCMA ODB is the input of the minimization (e 131) The observation data flow is finished! 7 June, 2006 3 D-VAR/ODB Training, Budapest 14

Obs dataflow summary BATOR ECMA (Full) LAMFLAG SHUFFLE (Reduc) ECMA. sb 1 ECMA. sb 2 … SHUFFLE (Merge) ECMA (C+I) Screening SHUFFLE (ECMA->CCMA) CCMA 7 June, 2006 3 D-VAR/ODB Training, Budapest 15

How to run screening? • • It is configuration 002: MASTER –c 002 –maladin –vmeteo –t 001 –aeul –e. MIN 1 Inputs: • ECMA ODB (observation, sigma-o, blacklisting) • First guess file under 5 different names: • ICMSHMIN 1 INIT, ICMSHMIN 1 IMIN, ICMRFMIN 10000, ELSCFMIN 1 ALBC • Constants, statistics • Namelists 7 June, 2006 3 D-VAR/ODB Training, Budapest 16

Constants • • errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF) rt_coef_atovs_neepred_ieee. dat: coefficients for the RTM (received from MF), binary file rszcoef_fmt. dat: the same the previous file but in ASCII format bcor_noaa. dat: bias correction file chanspec_noaa: obsolete? rmtberr_noaa. dat: obsolete? cstlim_noaa. dat: obsolete? 7 June, 2006 3 D-VAR/ODB Training, Budapest

Screening specific namelists • • These parameters should be set like this for 3 D-VAR: • NUPTRA = -1 , number of updates of the trajectory (NAMVAR) • NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR) • LOBS =. TRUE. , real obs (NAMCT 0) • LOBSC 1 =. FALSE. , (NAMCT 0), traj. is not computed • LSIMOB =. FALSE. , no simelated obs (NAMCT 0) • LSCREEN =. TRUE. , (NAMCT 0), screening is on! • LSCRE 4 D =. FALSE. , 3 D screening (NAMSCC) Tunable parameters are in NAMSCC and NAMOBS (discussed later)! 7 June, 2006 3 D-VAR/ODB Training, Budapest

How is screening working? Decisions are taken in subroutine: DECIS Independent decisions: performed separately for each report, RDB flag is assigned to reports rdbflag@hdr rdbflag@body Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected Dependent decisions: dependency on other reports or obs, or the previous decisions 7 June, 2006 3 D-VAR/ODB Training, Budapest

Independent decisions • Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT… • Blacklisting (BLACKCLN, BALCKSAT) • Background quality control (FIRST) 7 June, 2006 3 D-VAR/ODB Training, Budapest

Blacklisting • • • Blacklisting information is placed into ODB by BATOR In the screening a separate selection is applied for satellite data (BLACKSAT) Blacklisting is also performed in BLACKCLN: • Blacklisted obs will be rejected • Various blacklisting for SATOB (e. g. QI < 85) • Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is. T. ) • LSLRW 10: Wind 10 m, LSLRT 2: T 2 m, LSLRRH 2: THU 2 m (default is. T. meaning rejection over land. Will get a passive status!) 7 June, 2006 3 D-VAR/ODB Training, Budapest

Blacklisting • Also in BLACKCLN: • Height-based rejection. Controlled via NAMOBS LHDLREJ (default is. T. , rejected obs will get passive status!) • Orographic rejection limit is applied: • LHDRW 10: Wind 10 m, LHDRT 2: T 2 m, LHDRRH 2: RHU 2 m (default is. T. ) • AIREP data and significant levels for TEMP and PILOT are not used below the model surface! • RHU and Q is rejected above 300 h. Pa • AIREP is not used below a certain pressure 7 June, 2006 3 D-VAR/ODB Training, Budapest

Background QC (FIRST) • From the BLUE theory we know: • For a given location we expect: Values are read from file errgrib and interpolated to obs locations 7 June, 2006 Values are read from ODB 3 D-VAR/ODB Training, Budapest

Background QC (FIRST) • Flags are assigned to observations by checking: 0: correct 1: probably correct 2: probably incorrect 3: incorrect 7 June, 2006 Based on predefined flaglimits 4 flag values can be assigned to obs Flaglimits are defined in DEFRUN 3 D-VAR/ODB Training, Budapest

Dependent decisions • • Remove redundant surface levels from TEMP/PILOT (REDSL) Vertical consistency of profiles (VERCO) Removal of duplicated reports (DUPLI): two reports with the same ID, time an location … Redundancy check (REDUN) Thinning for AIREP (THIAIR) Specific scatterometer quality control Thinning of satellite data (THINN) 7 June, 2006 3 D-VAR/ODB Training, Budapest

Redundancy check (REDUN) • Applied for all active reports which are co-located and originate from the same station. Separate treatment for: • Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected • Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius • TEMP and PILOT • SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 h. Pa 7 June, 2006 3 D-VAR/ODB Training, Budapest

Thinning of AIREP data (THIAIR) • • • One flight consist of a set of reports Thinning is performed for each flight separately • 3 D boxes are constructed around model levels • In each box the report closest to the analysis date with the most active observations are selected • Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km! By default box size less than 50 km cannot be used in ARPEGE, code modification is needed 7 June, 2006 3 D-VAR/ODB Training, Budapest

Thinning of satellite data (THINNER) • • • For radiances a horizontal thinning is performed in two steps: • First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD 1 C(sensor)) • Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD 1 C(sensor)) Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, … Selection criterion: • Sea is preferred over land • Clear sky pixel is preferred over a cloudy one • Obs closer to the analysis date is preferred 7 June, 2006 3 D-VAR/ODB Training, Budapest

Thinning of satellite data (THINNER) • • • For SATOB a the thinning is performed: • Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km) • Vertically around model levels In each box the report closest to the analysis date with the most active observations are selected QI values are also used in the selection 7 June, 2006 3 D-VAR/ODB Training, Budapest

Sreening output • • Valuable summary about screening decisions can be found in the NODE file: Try „grep SCREENING STATISTICS” to get: • STATUS summary • EVENT summary • Number of variables, departures and missing departures • Diagnostic JO-table 7 June, 2006 3 D-VAR/ODB Training, Budapest

Sreening output • Summary at „grep SCREENING STATISTICS” Report and data status summary OB. TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED 1 725 44 0 681 0 2 0 0 0 OB. TYP DATA 0 ACTIVE 3 0 0 PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 4 0 0 0 1 0 0 0 5 61 60 02 3 0 0 0 6 61 59 0 2 0 0 0 0 7 0 0 04 9186 7271 1218 1915 705 8 0 0 05 0 0 6 3438 2050 64 1388 72 9 0 0 0 10 0 0 07 8 0 0 0 --------------------------------------------0684 0 0 0 TOT 847 163 09 10 0 0 ------------------------------------TOT 17200 9485 3128 7715 4639 7 June, 2006 3 D-VAR/ODB Training, Budapest

Sreening output • Summary at „grep SCREENING STATISTICS” Diagnostic Jo table Obstype 1 === SYNOP, Land stations and ships -------------------------Codetype 11 === SYNOP Land Manual Report Variable Data. Count Jo_Costfunction JO/n U 10 1380 2250. 937197524 1. 63 H 2 721 834. 1196058351 1. 16 Z 567 892. 5077244762 1. 57 T 2 721 2138. 950482967 2. 97 Q 721 1577. 382611689 2. 19 ------------------Obs. Type 1 Total: 4110 7693. 897622492 1. 87 7 June, 2006 Obs. Err 0. 200 E+01 0. 100 E+00 0. 785 E+02 0. 140 E+01 0. 998 E-03 Bg. Err 0. 000 E+00 3 D-VAR/ODB Training, Budapest

Sreening output • Observation monitoring available at: http: //pc 2088. met. hu/monitor/start. php 7 June, 2006 3 D-VAR/ODB Training, Budapest

Thank you for your attention! Any question? 7 June, 2006 3 D-VAR/ODB Training, Budapest 34

Exercises 1. Run screening for all the prepared observations with script ~/workdir/3 dvar/Assim and study the output listing! (You have to put an exit after screening in the script). 7 June, 2006 3 D-VAR/ODB Training, Budapest

Exercises 2. Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs (except Z) are really rejected, and find the reason for it. Try this view first: DEFINE VIEW bp AS SELECT obstype, status. active@hdr, varno, obsvalue, fg_depar, press, status. active@body, status. passive@body, status. blacklisted@body, status. rejected@body, event 1. land@body, event 1. datum_redundant@body FROM hdr, body WHERE (obstype == 1 OR obstype == 5) && statid = ‘ 12843 ‘ Guidance: A prepared viewer par file and sql file can be found in ~wshop 01/workdir/Odb. Viewer as bp. par bp. sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp. par” to run viewer. If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of event@body! 7 June, 2006 3 D-VAR/ODB Training, Budapest

Exercises 3. Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair. F 90 to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in include. inc! Compare the default and the new airep thinning in the output listings or in the ODB! 7 June, 2006 3 D-VAR/ODB Training, Budapest