System Administration Ib File Systems File Maintenance File

System Administration Ib File Systems File Maintenance

File Systems

File systems • Directory structure with its own root • Disks can be divided into multiple file systems ▫ Separates data: �-base OS�-system programs and utils�-user data- ▫ Corruption of one file system will not affect another ▫ Each file system can be backed up independently

File Systems • Components ▫ boot block �boot program �partition table ▫ superblock �global info on file system �OS keeps a copy in memory ▫ inode blocks �keeps track of every disk block ▫ data blocks �all the data

Unix/Linux File Systems • Types ▫ s 5 ▫ ufs ▫ ext 2, ext 3 and ext 4 �Current linux systems ▫ iso 9660 or hsfs �CD ROM file system ▫ msdos or pcfs �original floppy for DOS ▫ swap ▫ bfs ▫ proc or procfs

File Systems • Mounting ▫ Mount Point: �A directory in one filesystem which contains another file system ▫ Classical mount points �/dev/fd �fd floppy disk �fd 0 first floppy drive �/dev/hda �hd ide drive �hda first ide �hda 1 first partition in hda �/dev/hdb �hdb second ide drive �/dev/sda �sd scsi drive ▫ also used for pata/sata drives �sda first scsi �sda 1 first partition in sda �/dev/tty. S 0 �a serial port

File systems • mount points may be different on various distros ▫ My Debian at home and the one at work have different /dev names for the same function

File Systems • Interesting note ▫ Can mount multiple drive blocks to make one larger file system �Combine partitions, drives �LVM �Logical Volume Manager �Similar in concept to RAID 0 or JBOD �LVM gives more control of how to combine

File Systems • Mounting ▫ Assign file blocks to a directory name • Unmounting ▫ Removes the mount • fstab file ▫ Lists all available disks and partitions �Where they should be mounted and how ▫ Can be used by mount

mounting • Connects a device to a file name • mount [opts] devname dirname ▫ must know device name ▫ dirname must exist �it should be empty �permissions set to at least 755 � 777 works but is less secure �some may be predefined • Notes: ▫ mount �If no options it lists all current mounts ▫ fdisk –l �will list available hard drives

mount example > mount /dev/hda 1 on / type ext 3 (rw, errors=remount-ro) tmpfs on /lib/init/rw type tmpfs (rw, nosuid, mode=0755) proc on /proc type proc (rw, noexec, nosuid, nodev) sysfs on /sys type sysfs (rw, noexec, nosuid, nodev) procbususb on /proc/bus/usb type usbfs (rw) udev on /dev type tmpfs (rw, mode=0755) tmpfs on /dev/shm type tmpfs (rw, nosuid, nodev) devpts on /dev/pts type devpts (rw, noexec, nosuid, gid=5, mode=620) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on /proc/fs/nfsd type nfsd (rw) /dev/sda 1 on /media/EGMONITOR type vfat (rw, nosuid, nodev, uhelper=hal, shortname=lower, uid=1000)

fdisk –l example > sudo fdisk -l Disk /dev/hda: 40. 0 GB, 40020664320 bytes 255 heads, 63 sectors/track, 4865 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0 x 93 dd Device Boot /dev/hda 1 * /dev/hda 2 /dev/hda 5 Start 1 4661 End 4660 4865 Blocks 37431418+ 1646662+ 1646631 Id 83 5 82 System Linux Extended Linux swap / Solaris Id 6 System FAT 16 Disk /dev/sda: 1018 MB, 1018167296 bytes 2 heads, 63 sectors/track, 15782 cylinders Units = cylinders of 126 * 512 = 64512 bytes Disk identifier: 0 x 017 c 9 ef 7 Device Boot /dev/sda 1 * > • Notes: Start 1 End 15783 Blocks 994288 ▫ hda is an ide drive with 3 partitions ▫ sda is a USB drive currently plugged in

unmounting • umount dirname ▫ can also use devname ▫ unmount before removing device �prevents loss or corruption of data �forces a sync ▫ note the name is umount, not unmount

fstab • File of default mount points • Gets mounted during the booting process ▫ Notes: �mount –a �will mount the fstab entries �mount /dirname �will mount to the device listed in fstab �umount –a �will try to unmount the entries in fstab �if device is busy will get an error �cannot unmount a directory you are in

fstab • Sample fstab: > cat /etc/fstab # /etc/fstab: static file system information. # #<filesys> <mount point> <type> <options> proc /proc defaults /dev/hda 1 / ext 3 defaults, errors=remount-ro /dev/hda 5 none swap sw /dev/hdc /media/cdrom 0 udf, iso 9660 user, noauto /dev/hdd /media/cdrom 1 udf, iso 9660 user, noauto /dev/fd 0 /media/floppy 0 auto rw, user, noauto > • Columns: ▫ 1: device name ▫ 2: mount point ▫ 3: file system type ▫ 4: mount options ▫ 5: enable backing up (dump) ▫ 6: fsck order <dump> 0 0 0 <pass> 0 1 0 0

File System Checking • superblock must be kept in sync between the version on the disk and the one in memory ▫ update does a sync every 30 seconds ▫ if there is a powerfailure before a change can be synced there can be discrepancies �Examples: �two or more inodes claiming same disk block �block marked free, but not listed in superblock �used block marked free �mismatch in sizes �file not having a directory entry • fsck can check for and fix these problems

Files system checking • fsck ▫ five sequential phases �each feeding the next phase ▫ if there is an error �fsck becomes interactive �user answers questions to fix

Manageing Disk Space

Managing Disk space • Fact: disks fill up! ▫ Need to monitor and maintain usage • Two good basic tools: ▫ df ▫ du

Managing Disk space: df • df ▫ reports free space on disk �by file system ▫ -h option makes it human readable > df -h Filesystem /dev/sda 1 tmpfs udev tmpfs /dev/sda 9 /dev/sda 8 /dev/sda 5 /dev/sda 6 /dev/sdb 1 /dev/sde 1 Size 323 M 2. 0 G 440 G 368 M 8. 3 G 2. 8 G 114 G 16 G Used Avail Use% Mounted on 160 M 147 M 53% / 0 2. 0 G 0% /lib/init/rw 236 K 2. 0 G 1% /dev 0 2. 0 G 0% /dev/shm 2. 9 G 415 G 1% /home 11 M 339 M 4% /tmp 4. 2 G 3. 7 G 54% /usr 1. 1 G 1. 6 G 41% /var 1. 9 G 106 G 2% /opt 7. 2 G 8. 1 G 48% /media/PENDRIVE

Managing Disk space: du • du ▫ reports disk usage �by directory ▫ Useful options: �-s: summary �-h: human readable > sudo du -sh /home/* 2. 7 G /home/ajkombol 8. 0 K /home/ftp 16 K /home/lost+found 1. 2 M /home/tkombol

Managing Disk space: du • du with the –sh option for the whole drive # sudo du -sh /* 1. 0 K /ajkubertooth 5. 5 M /bin 15 M /boot 236 K /dev 7. 6 M /etc 2. 7 G /home 0 /initrd. img 111 M /lib 2. 4 M /lib 32 0 /lib 64 12 K /lost+found 7. 1 G /media 2. 0 K /mnt 1. 6 G /opt 0 /proc 2. 5 M /root 3. 6 M /sbin 1. 0 K /selinux 1. 0 K /srv 0 /sys 755 K /tmp 4. 1 G /usr 1004 M /var 0 /vmlinuz

Managing Disk space: find • find: ▫ The adminstartor's friend • Syntax: ▫ find path_list select_criteria action �find /* –size 16 M ▫ Note: default action for Linux is –print �Most commonly used

Managing Disk space: find • Some Popular Options: ▫ -name: find files by name �find startdir –name filename ▫ -perm: find by permissions �find startdir –perm 755 ▫ -size: find files larger than a certain size �find startdir -size n[cwbk. MG] �find. –size 16 M ▫ -atime: find since last accessed �find startdir –atime n �n is the number of 24 hr periods ▫ -mtime: find since last modified f �find startdir –mtime n

Managing Disk space: examples • Find all users on size example ( >20 Mb): # find /home -size +20 M –print /home/ajkombol/Downloads/netbeans-7. 1. 2 -ml-php-linux. sh /home/ajkombol/Downloads/metasploit-latest-linux-x 64 -installer. run /home/ajkombol/Downloads/netbeans-7. 1. 2 -ml-linux. sh /home/ajkombol/. cache/chromium/Cache/data_3 /home/ajkombol/EYC Movie. iso /home/ajkombol/eyc. iso /home/ajkombol/Music/11 Epica/01 The Divine Conspiracy/13 The Divine Conspiracy. m 4 a /home/ajkombol/. mozilla/firefox/r 797 acrr. default/urlclassifier 3. sqlite • Find files on access time (500 days): # sudo find ~ -atime +500 /home/ajkombol/. bzf/cache/t 9 de 3 f 8 cab 7 b 93 d 8 aabfd 653089 a 2 fbde. bwc # cd. bzf/cache/ # ls -lu total 20 -rw-r--r-- 1 ajkombol 6012 Oct 12 18: 04 BZFS 0026_1 -Servers. bzs -rw-r--r-- 1 ajkombol 194 Oct 12 18: 04 CACHEDIR. TAG -rw-r--r-- 1 ajkombol 102 Oct 12 18: 05 Cache. Index. txt -rw-r--r-- 1 ajkombol 1876 Dec 31 1969 t 9 de 3 f 8 cab 7 b 93 d 8 aabfd 653089 a 2 fbde. bwc • Note: for ls –lu, the u returns last accessed date instead of modified date

Tar, Compress, Gzip, Zip

tar, compress, and gzip • Historically tar and compress were used to package and compress a set of files ▫ Back when tape was a popular backup medium • gzip is the modern equivalent of compress

tar, compress, and gzip • tar – tape, archive, restore ▫ General form: �tar main. Option other. Options �other. Options vary by main. Option �main. Options: �c – Create �r – Replace �t – Table of contents �u – Update �x – extract or restore

tar, compress, and gzip Main options c and r tar c [ b. Be. Ef. Fhiklnop. Pqvw. X [ 0 -7 ] ] [ block ] [ tarfile ] [ exclude-file ] {-I includefile | -C directory | file } c - Create • Writing begins at the beginning of the tarfile, instead of at the end. tar r [ b. Be. Ef. Fhiklnqvw [ 0 -7 ] ] [ block ] {-I include-file | -C directory | file } r - Replace • The named files are written at the end of the tarfile. A file created with extended headers must be updated with extended headers (see E flag under Function Modifiers). A file created without extended headers cannot be modified with extended headers. .

tar, compress, and gzip Main options t and u tar t [ Bef. Fhiklnqv. X [ 0 -7 ] ] [ tarfile ] [ exclude-file ] {-I include-file | file }. . . t - Table of Contents. • The names of the specified files are listed each time they occur in the tar file. If no file argument is given, the names of all files in the tarfile are listed. With the v function modifier, additional information for the specified files is displayed. tar u [ b. Be. Ef. Fhiklnqvw [ 0 -7 ] ] [ block ] [ tarfile ] file. . . u - Update. • The named files are written at the end of the tarfile if they are not already in the tar file, or if they have been modified since last written to that tarfile. An update can be rather slow. A tarfile created on a 5. x system cannot be updated on a 4. x system. A file created with extended headers must be updated with extended headers (see E flag under Function Modifiers). A file created without extended headers cannot be modified with extended headers.

tar, compress, and gzip Main option x tar x [ Bef. Fhiklmnopqvw. X [ 0 -7 ] ] [ tarfile ] [ exclude-file ] [ file. . . ] x - Extract or restore. • The named files are extracted from the tarfile and written to the directory specified in the tarfile, relative to the current directory. Use the relative path names of files and directories to be extracted. If a named file matches a directory whose contents has been writ ten to the tarfile, this directory is recursively extracted. The owner, modification time, and mode are restored (if possible); otherwise, to restore owner, you must be the super-user. Character special and block-special devices (created by mknod(1 M)) can only be extracted by the super user. If no file argument is given, the entire content of the tarfile is extracted. If the tar- file contains several files with the same name, each file is written to the appropriate directory, overwriting the previous one. Filename substitu tion wildcards cannot be used for extracting files from the archive; rather, use a command of the form: tar xvf. . . /dev/rmt/0 `tar tf. . . /dev/rmt/0 | grep 'pattern' ` • When extracting tapes created with the r or u functions, directory modification times may not be set correctly. These same functions cannot be used with many tape drives due to tape drive limitations such as the absence of backspace or append capabilities. When using the r, u, or x functions or the X function modifier, the named files must match exactly the corresponding files in the tarfile. For example, to extract. /thisfile, you must specify. /thisfile, and not thisfile. The t function displays how each file was archived.

tar, compress, and gzip • Typical use: ▫ tar –cvf archive. tar file 1 file 2 file. ext �Where: �-cvf: (c)reate an archive, (v)erbose, (f)ilename �archive. tar is the archive created �file 1, file 2 and file. ext are packed into archive. tar ▫ tar –xvf archive. tar �Where: �-xvf: e(x)tract, (v)erbose, (f)ilename ▫ tar –tvf archive. tar �Where the –t shows the contents of the tar �table of contents

tar - example > ls backup. Waittoread. sh file 1. bak mv. sh xfile 3. bak bigfile 1 file 3 myfirst. sh xfile 5 bigfile 2 file 3. bak myscp. sh xfile 5. bak bigfile 3 filelist noread xfilelist bigfile 4 filelist. save strlen. sh > tar -cvf bigfile. tar bigfile 1 bigfile 2 bigfile 3 bigfile 4 > rm bigfile 1 bigfile 2 bigfile 3 bigfile 4 > ls backup. Waittoread. sh file 3. bak myfirst. sh xfile 3. bak bigfile. tar filelist myscp. sh xfile 5 blurb. txt filelist. save noread xfile 5. bak > tar -xvf bigfile. tar bigfile 1 bigfile 2 bigfile 3 bigfile 4 > ls backup. sh bigfile. tar listwhenreadable. sh xfile 1. bak backup. Waittoread. sh file 1 logicalop. sh xfile 3 bigfile 1. bak mv. sh xfile 3. bak bigfile 2 file 3 myfirst. sh xfile 5 bigfile 3. bak myscp. sh xfile 5. bak bigfile 4 filelist noread xfilelist >

tar, compress, and gzip • compress ▫ "Obsolete" command �Not found on many Linux systems ▫ Compacts a file so that it is smaller ▫ Compressed file will be replaced: �With a same name file with the extension. z �Keeps all the same ownership modes ▫ Syntax: �compress [-c] [-f] [-v] filenames • uncompress ▫ Uncompressed files. ▫ Syntax: �uncompress [-c] [-f] [-v ] [file]

tar, compress, and gzip • gzip ▫ Replacement for compress ▫ Syntax: �gzip filename �compresses filename �replaces filename with filename. gz �Notes: �-c leaves original file �-r recursively compresses all files if filename is a directory • gunzip ▫ uncompresses ▫ Syntax: �gunzip filename. gz �replaces the filename. gz with filename ▫ same as gzip –d

gzip example # ls -l bi* -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 1 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 2 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 3 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 4 -rw-r--r-- 1 tkombol 10240 2013 -04 -09 15: 42 bigfile. tar # gzip bigfile. tar # ls -l bi* -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 1 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 2 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 3 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 4 -rw-r--r-- 1 tkombol 208 2013 -04 -09 15: 42 bigfile. tar. gz # gunzip bigfile. tar. gz # ls -l bi* -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 1 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 2 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 3 -rwxr-xr-x 1 tkombol 73 2013 -03 -28 18: 04 bigfile 4 -rw-r--r-- 1 tkombol 10240 2013 -04 -09 15: 42 bigfile. tar # tar -tvf bigfile. tar -rwxr-xr-x tkombol/tkombol 73 2013 -03 -28 18: 04 bigfile 1 -rwxr-xr-x tkombol/tkombol 73 2013 -03 -28 18: 04 bigfile 2 -rwxr-xr-x tkombol/tkombol 73 2013 -03 -28 18: 04 bigfile 3 -rwxr-xr-x tkombol/tkombol 73 2013 -03 -28 18: 04 bigfile 4 #

zip • zip and unzip ▫ PKZIP and PKUNZIP in the Unix/Linux world • zip ▫ Does both combining and compressing in one program ▫ -r �recursive �will pack and compress the subdirectories also • unzip ▫ non-interactive restore if restored file not present ▫ if file exists unzip asks permission to restore �[y]es, [n]o, [A]ll, [N]one, [r]ename options ▫ -v �view the contents of the zip file

# ls -l big* -rwxr-xr-x 1 tkombol tkombol # zip bigfile. zip big* adding: bigfile 1 (deflated adding: bigfile 2 (deflated adding: bigfile 3 (deflated adding: bigfile 4 (deflated # ls -l big* -rwxr-xr-x 1 tkombol tkombol -rw-r--r-- 1 tkombol # rm bigfile[0 -9] # ls -l big* -rw-r--r-- 1 tkombol # unzip bigfile. zip Archive: bigfile. zip inflating: bigfile 1 inflating: bigfile 2 inflating: bigfile 3 inflating: bigfile 4 # ls -l big* -rwxr-xr-x 1 tkombol tkombol -rw-r--r-- 1 tkombol # 73 73 2013 -03 -28 18: 04 bigfile 1 bigfile 2 bigfile 3 bigfile 4 75%) 68%) 73 73 613 2013 -03 -28 2013 -04 -10 18: 04 10: 35 bigfile 1 bigfile 2 bigfile 3 bigfile 4 bigfile. zip 613 2013 -04 -10 10: 35 bigfile. zip 73 73 613 2013 -03 -28 2013 -04 -10 18: 04 10: 35 bigfile 1 bigfile 2 bigfile 3 bigfile 4 bigfile. zip

• Example: ▫ zip one's home directory and log the results # cd ; zip -r tkombol. zip. > zip. log zip warning: name not matched: . kde/tmp-SIS-43 SPP 11 -PC zip warning: name not matched: . kde/socket-SIS-43 SPP 11 -PC zip warning: Permission denied zip warning: could not open for reading: . Xauthority. backup # ls -l total 150368 -rw-r--r-- 1 tkombol 12 2013 -04 -04 12: 32 ajktest drwxr-xr-x 4 tkombol 4096 2013 -04 -10 10: 41 Desktop drwxr-xr-x 11 tkombol 4096 2009 -11 -19 16: 53 grades -rw-r--r-- 1 tkombol 153626580 2013 -04 -10 10: 48 tkombol. zip -rw-r--r-- 1 tkombol 172323 2013 -04 -10 10: 48 zip. log # • Next pages: ▫ partial listing of zip. log � 2140 lines long ▫ partial view of unzip –v tkombol. zip �long

# cat zip. log adding: . bash_profile (deflated 39%) adding: . gnome 2/ (stored 0%). . . adding: . gconf/apps/evince/ (stored 0%). . . adding: . mozilla/firefox/jlammzq 3. default/Cache/4 A 74584 Bd 01 (deflated 0%). . . adding: . openoffice. org/3/user/config/standard. sob (deflated 2%). . . adding: . openoffice. org 2/user/registry/data/org/openoffice/Office/Recovery. xcu (deflated 70%). . . adding: . config/totem/state. ini (deflated 20%) adding: Desktop/ (stored 0%) adding: Desktop/my-ls. txt (deflated 62%). . . adding: Desktop/bashscripts/ (stored 0%) adding: Desktop/bashscripts/awk 1. awk (deflated 32%). . . adding: Desktop/bashscripts/xfile 1 (stored 0%) adding: Desktop/bashscripts/bigfile 2 (deflated 68%) adding: Desktop/a. htm~ (deflated 71%) adding: . dmrc (stored 0%). . . adding: . thumbnails/ (stored 0%) adding: . thumbnails/normal/7 cc 9 e 66 b 999 b 867 f 73 edf 672 dcd 4 eef 7. png (deflated 0%). . . adding: . thumbnails/normal/afb 4 bf 3 db 3 cb 1 f 1004599 f 801 ee 4 af 7 b. png (deflated 1%) adding: . fontconfig/ (stored 0%) adding: . fontconfig/e 13 b 20 fdb 08344 e 0 e 664864 cc 2 ede 53 d-x 86. cache-2 (deflated 83%). . . adding: . fontconfig/7 b 4 a 97 c 10 f 6 c 0166998 ddfa 1 cf 7392 fb-x 86. cache-2 (deflated 79%) adding: . lesshst (deflated 17%) adding: . gconfd/ (stored 0%). . . adding: grades/apache_pb 22_ani. gif (deflated 17%) adding: grades/_private/ (stored 0%) adding: . java/ (stored 0%) adding: . bash_history (deflated 70%) #

# unzip -v tkombol. zip > view. txt # ls -l total 150448 -rw-r--r-- 1 tkombol 12 2013 -04 -04 12: 32 ajktest drwxr-xr-x 4 tkombol 4096 2013 -04 -10 10: 41 Desktop drwxr-xr-x 11 tkombol 4096 2009 -11 -19 16: 53 grades -rw-r--r-- 1 tkombol 153626580 2013 -04 -10 10: 48 tkombol. zip -rw-r--r-- 1 tkombol 245847 2013 -04 -10 11: 10 view. txt -rw-r--r-- 1 tkombol 1481 2013 -04 -10 11: 03 zip. log -rw-r--r-- 1 tkombol 2534 2013 -04 -10 11: 00 zip. log~ # ls ajktest Desktop grades tkombol. zip view. txt~ zip. log~ # cat view. txt Archive: tkombol. zip Length Method Size Ratio Date Time CRC-32 Name ------- ---------414 Defl: N 254 39% 06 -07 -07 09: 25 2 f 4166 e 5. bash_profile 0 Stored 0 0% 04 -04 -13 16: 03 0000. gnome 2/. . . 0 Stored 0 0% 06 -16 -08 10: 27 0000. gnome/gnome-vfs/ 51 Defl: N 44 14% 03 -28 -13 18: 03 287 c 14 a 5. gnome/gnome-vfs/. trash_entry_cache 5922 Defl: N 1770 70% 04 -08 -13 11: 41 c 33 fdb 16. bash_history -------230245501 153152546 34% 2139 files #

Last Note • The preceding utilities are usually used to prepare files for offline storage • Originally they were meant to be sent to a tape ▫ Hence the t in tar • They are still a convenient way to package a set of files into a single file and then shrink the size of that file ▫ For storage or sending across a network