DSARCH OVERVIEW Dataset Archiving Utility Overview By Zaihua

  • Slides: 24
Download presentation
DSARCH OVERVIEW Dataset Archiving Utility Overview By Zaihua Ji 1

DSARCH OVERVIEW Dataset Archiving Utility Overview By Zaihua Ji 1

Outline n n n Definitions and how DSARCH fits into RDA functions Purpose of

Outline n n n Definitions and how DSARCH fits into RDA functions Purpose of DSARCH Introduction of DSARCH Usage Procedures of using DSARCH to archive data Convert existing datasets Publish MSS/RDA Server data filelists 2

RDA Components 3

RDA Components 3

Definition of Metadata -- Metadata that summarizes a dataset, such as author, title, summary,

Definition of Metadata -- Metadata that summarizes a dataset, such as author, title, summary, etc. -- Metadata that defines external properties of RDA files, such as file locations on MSS/RDA Server, sizes, file packaging (tar, COS block, etc. ), dataset sub-groups of files, archive file type (P, B, W, …), etc. -- Metadata that defines internal properties of RDA files, such as data format (GRIB, ASCII, etc. ), variables, spatial & temporal ranges, internal metrics (eg. , number of grids or stations), etc. 4

Metadata Coverage 5

Metadata Coverage 5

Current Data Archive Flow 6

Current Data Archive Flow 6

Dsarch Work Flow for Existing dataset 7

Dsarch Work Flow for Existing dataset 7

Dsarch Work Flow for New Dataset 8

Dsarch Work Flow for New Dataset 8

What DSARCH For Archives/Retrieves data files to/from MSS and RDA server n Records/Retrieves dataset

What DSARCH For Archives/Retrieves data files to/from MSS and RDA server n Records/Retrieves dataset metadata (mainly VSN metadata) to/from RDADB n Organizes archived data files for one dataset into sub-datasets, called groups n 9

What DSARCH Does n n n n n Defines group and dataset information in

What DSARCH Does n n n n n Defines group and dataset information in RDADB Automatically selects MSS VSN by default, other options are available Archives data files on the MSS and/or RDA server and saves transaction records into the RDADB Retrieves dataset/group/file information from the RDADB Modifies/corrects information in the RDADB Copies MSS files back to local computers (RDA Server, bison, etc) Moves MSS/Web data files from one dataset/group to another Removes files from MSS or RDA server RDADB Maintenance Functions n n Backups dataset/group/file information from RDADB into CVS Archive Restores dataset/group/file information into RDADB from CVS Archive 10

Advantage of Using DSARCH 11

Advantage of Using DSARCH 11

General DSARCH Usage dsarch [[-DS] dsnnn. n] [Action Option] [Mode Options] [Information Options] n

General DSARCH Usage dsarch [[-DS] dsnnn. n] [Action Option] [Mode Options] [Information Options] n n n n Quotes [] indicate optional Three Option categories: Action, Mode, and Information (Info for short) Options. Action options - specify what tasks this utility to execute Mode options - modify behaviors of given actions Info options - pass information, one or multiple values, to run DSARCH An option is given in either short or long names, eg. -DS or -Dataset All Action and Mode options, as well as Info option -IF (-Input. File), must be given on command line; all other info options can be given either on command line or in one or multiple input files specified by option -IF 12

Categories of Action Options n n n Dataset Actions - create, modify and retrieve

Categories of Action Options n n n Dataset Actions - create, modify and retrieve dataset information in RDADB Group Actions - create, delete, modify and retrieve dataset group information in RDADB File Actions n n n archive files onto MSS and RDA server move and delete files on MSS and RDA server create, delete, modify and retrieve information about data files in RDADB File-Name Actions - generate, release, retrieve and archive MSS file names (VSN - Volume Serial Number) per RDADB card bank Info Actions - create, modify and retrieve all information for datasets, groups in datasets, on MSS and RDA Server Backup Actions - archive, restore, and check CVS backup history information for datasets, groups, and MSS and RDA Server files 13

Procedure for Creating a New Dataset n n n Create SCCS dataset archive file

Procedure for Creating a New Dataset n n n Create SCCS dataset archive file with ‘Search & Discovery Metadata’ only (for dataset main webpage) Create initial dataset record in RDADB per utility ‘filldataset’ Set flag for using RDADB to ‘Y’ and modify the dataset info Create group information if needed Archive local files onto MSS and/or RDA Server Set flag for using RDADB to ‘P’ or ‘I’, and publish dataset file lists per utility ‘publish_filelist’ 14

Procedure to Convert Existing Dataset n n n Set flag of using RDADB to

Procedure to Convert Existing Dataset n n n Set flag of using RDADB to ‘Y’ Edit VSN section from the SCCS dataset document to create a file named ‘dsnnn. n. sccs’ Use ‘myconvert’ utility to reformat ‘dsnnn. n. sccs’ into ‘dsnnn. n. mss’, which is an input file designed for RDADB Use DSARCH to enter the input file ‘dsnnn. n. mss’ into RDADB Set flag of using RDADB to ‘P’ or ‘I’, and use ‘publish_filelist’ utility to publish dataset file lists 15

Convert SCCS Metadata for An Existing Dataset myconvert dsnnn. n. sccs > dsnnn. n.

Convert SCCS Metadata for An Existing Dataset myconvert dsnnn. n. sccs > dsnnn. n. mss n dsnnn. n. sccs - MSS file information modified from VSN file section of SCCS dataset metadata file, by inserting conversion control keys for information about the dataset, groups and files. See examples later n dsnnn. n. mss - DSARCH input file holding dataset, group, and/or MSS file information 16

Conversion Key Categories n n Dataset Keys - mark public and/or internal MSS dataset

Conversion Key Categories n n Dataset Keys - mark public and/or internal MSS dataset notes Group Keys - build up group information, such as group index, group name (ID), group titles, and public and/or internal MSS group notes MSS File Keys - setup MSS file information, such as format key to specify what information should be collected from the description part of a file line, and keys for data and file formats A special key LB, which can be used to turn on (LB<=> , for example) or off (LB<=>) of a html linebreak symbol for multiple line notes 17

Dataset Keys DM - public MSS dataset note; description lines following ‘DM<=>’ are collected,

Dataset Keys DM - public MSS dataset note; description lines following ‘DM<=>’ are collected, including empty lines n DI - internal MSS dataset note; description lines following ‘DI<=>’ are collected, including empty lines n 18

Group Keys GN - group name or group ID, up to 20 characters. It

Group Keys GN - group name or group ID, up to 20 characters. It is given in format of GN<=>Group. ID, and is optional if GI is present, eg. GN<=>List-A n GI - group index number. It is automatically assigned in order, if GN is present. It is given format of GI<=>Group. Index, eg. GI<=>5 n 19

MSS File Keys n n n n DE - file note or description SD

MSS File Keys n n n n DE - file note or description SD - shared note for multiple files. Insert a line of 'SD<=>' to record description lines as a shared note LF - local file name for a MSS VSN file name FF - file format, up to 10 characters, eg. 'FF<=>BI. TAR' means the following files are binary COS-blocked and then tarred TF - data format, up to 10 characters, eg. 'TF<=>ASC. IMMA' means the following files hold data files in both ASCII and IMMA formats RL - length of each record in a file RN - number (count) of records in a file FMT - format for description part of file information lines; eg. 'FMT<=>LF, , DE' means that description part of each line is split into three columns by delimiter ', '; the first column is local file name, the second is ignored and the third is file note 20

Example of ds 540. 1. sccs p. TF<=>BINARY FF<=>TAR FMT<=>LF, GN<=>MSGSTD 2 p- MSG,

Example of ds 540. 1. sccs p. TF<=>BINARY FF<=>TAR FMT<=>LF, GN<=>MSGSTD 2 p- MSG, Standard Statistics, 2 x 2, 1800 -1997 py 61279 stdg 3. 1800. 1809. tar, less than. 5 MB py 61281 stdg 3. 1810. 1819. tar, less than. 5 MB …. FF<=>Z. TAR GN<=>MSGSTD 2_R 2. 2 p. MSG, Standard Statistics, 2 x 2, release 2. 2, 1998 -2004 p- Release 2. 2 is exclusively new data for 1998 -2004. For convenience, data from p- Group ID <a href="#MSGSTD 2">MSGSTD 2</a> for 1990 -1997 have been included in these files. py 88809 msg_2 deg/stdg 3. 1990. 1999. tar, 33. 983 MB by 88810 b/u y 88809 py 88811 msg_2 deg/stdg 4. 1990. 1999. tar, 34. 098 MB by 88812 b/u y 88811 …. 21

Snapshots of Web Display …. . 22

Snapshots of Web Display …. . 22

Publish MSS/Web Filelists Publish_filelist [-t] dsnnn. n n n Set flag of using RDADB

Publish MSS/Web Filelists Publish_filelist [-t] dsnnn. n n n Set flag of using RDADB to P or I before publishing a dataset (Y is fine for option -t to publish test filelists) Generates html file of MSS public filelist to root directory of given dataset Generates html file of MSS internal filelist to internal dataset directory Generates html index files of RDA Server data in data and sub-data directories of given dataset, unless manually created index html files exist already All html filelist files will be built dynamically and cached 23

Update Dataset With New Data Run DSARCH to archive new data files onto MSS

Update Dataset With New Data Run DSARCH to archive new data files onto MSS and/or RDA server n No need republish filelists n Filelists including the information of new archived data files will be recached automatically when the filelist web pages are accessed by users n 24