Htar Hpss Tape Archiver Client APIbased interface written

Htar Hpss Tape Archiver Client API-based interface written by Mike Gleicher Originally commissioned for LLNL in 2000 Now available as part of the HPSS Offering A File Bundling and manipulation tool that conforms to standard tar requirements: Bundles small files on a platform into large files in storage. The Htar is created in the storage system, each file is transferred directly into the Htar in storage.

Why use Htar Powerful command set that makes it easy to recursively archive files to storage: whole directories, files in lists, etc. with one command. A users directory structure is maintained in the “htar” file. Multiple files transfer faster with htar: HTAR –cvf htar. /mydir is optimized for simultaneous transfer HSI put –r. /mydir is recursive…. . But transfers files one at a time.

Htar@NERSC Authentication: Htar uses the same. hsipw file as HSI for authentication to NERSC storage systems. If you currently have a. hsipw file Htar will use it. If you don’t you will be prompted for your dce combo, and if it’s successful a. hsipw will be created in ~HOME. Availability: Installed on Seaborg /usr/common/mss/bin and on the PDSF /usr/local/bin

Htar Definitions Archive File - The large “tar” file that Htar creates out of files Member File - one of the files contained in the Htar Archive File. Index File - a catalog of the Member Files contained in the Archive File Consistency File - the file at the end of an Archive File used to verify the consistency of the Archive File and the Index File.

Instorage Htar -cvf myhtar. tar creates the following in storage: the archive file called myhtar. tar the index file called myhtar. idx a consistency file inside the archive file (looks like a member file but is treated differently) called /tmp/HTAR_CF_CHK…. . This consistency file can be used to verify the consistency of the index file and the archive file. It will not be extracted unless explicitly specified using the Htar command. (it would be extracted using the tar command)

Htar behavior Absolute path names: pathnames beginning with “/” can be used when creating the htar…when files are extracted from the htar the leading “/” is removed. All extracted files use pathnames that are relative to the current working directory. Htar makes use of the TMPDIR environment variable when creating temporary files. If TMPDIR is not set in the environment then “/tmp” is used. There is no default device for Htar the -f flag must be used.

Htar and tar Tar Similarities: Htar conforms to standard tar requirements. An Htar Archive File can be manipulated using the tar utility. File and directory name lengths and file sizes must conform to tar restrictions (POSIX 1003. 1 USTAR). Buffer path to 155 chars, name buffer to 100 bytes. Member file size <= 8 GBs. Tar Differences: Htar Can’t append a new Member File to an existing Archive File. Htar Can’t update/replace Member Files in an existing Archive File. Htar creates an index. idx file in the same dir as the archive file. and a consistency check file inside the Htar archive file

Htar examples Htar -cvf files. tar file 1 file 2 writes file 1 and file 2 into a new archive called files. tar in the current hpss home directory. Htar -cvf /nersc/user/homedir/htar. /dir 1 writes all the files in the current dir 1 directory into a new archive file called htar. tar in the hpss directory /nersc/user/homedir Htar -tvf htar. tar list files in the archive file htar. tar in the current hpss homedir.

Htar more examples Htar -xvf files. tar extract all the files from the archive files. tar into the current directory. Htar -xf proj 1. tar project 1/src extract all files in project 1/src directory of the archive file proj 1. tar into the current directory Htar -xf file 1 proj 1. tar extract the individual file : file 1: from the archive file proj 1. tar

Htar extensions Htar -X -f /nersc/user/hpsshomedir/tarfile creates an index for a tarfile in hpss. This file can now be accessed using htar. Htar -X -E -f localtarfile : creates an htar index for a local tar file. Htar -K -f htartest. htar : use the consistency file to verify the index and the archive file.

Recommended Usage Htar should replace all current tar transfers piped to storage. Htar should be used with data that is not changing. Htar makes two passes thru the data first to generate the index file, second pass to transfer the data. If files change before they are transferred and after the index is created a warning is issued and the file is truncated or padded to match the index. Htar should be used to bundle small files Htar should be used to archive files that are not frequently referenced (purely archival)

Usage for Performance Htar should be used with collections of data that will reaccessed as a collection. These Htar files should be < 6 GBs. Read access to files in storage: 1 second = 86400 accesses/day at XXMBs/sec 30 seconds = 2880 accesses/day at XXMBs/sec 60 seconds = 1440 …/etc The Htar will travel in the archive as one file to and from disk and tape. Access to any file in the Htar will stream the Htar to disk improving all subsequent accesses to the data to 1 second or less.