Zen and the Art of REQADM Maintenance Steve

Zen and the Art of REQADM Maintenance Steve Willoughby PMD Engineering Computing 12 April 2000 Copyright © 2000 Intel Corporation.

What This Course Is • Introduction to the REQADM system components for the REQADM Administrator. • Tips for configuring and maintaining the system. • Provides familiarity with system to help you troubleshoot your system. • A peek ahead at REQADM 2. 0, and how it’ll change what you do.

What This Course Is Not • User-level introduction to REQADM clients for either systems administrators or customers. • In-depth look at report generation. • How to organize a help center. • There are separate presentations available for each of those.

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

Agenda • What is REQADM, inside and out? – System Architecture • Directory Layout – Ticket States and Statuses – Overview of Clients – Server Design and Components

System Architecture • Server stores and manages tickets. – All access to ticket data goes through server. • Ensures reliable data, lock management, (minimal) security, sanity-checking, common interface. • Clients provide various interfaces to server. – New interfaces added independently, don’t require server or other client changes. – API library

Directory Layout /opt/reqadm* archive bin db lib Backup sbin spool queue 1 queue 2 Database files for ticket data files Configuration files, ASCII “Core” files REQADM for closed Supplimental scripts Nightly cron script Purged tickets REQADM server’s stored in directories *. pidcompiled files, etc. tickets system stored programs in and makes database archived here for later DBMS module. named the directories (server, clients, of 1, 000 programs, not “core” backup copies and retrievalfor (compressed REQADM queues tickets maintenance each. scripts). services. recovery files here. archives) *default name; YMMV Fixed D 0 D 1000

Agenda • What is REQADM, inside and out? – System Architecture • Directory Layout – Ticket States and Statuses – Overview of Clients – Server Design and Components

Ticket States New Ticket Assigned Prioritized Work in Progress On Hold Categorized Resolved Closed

Ticket Status • REQADM rates each ticket’s status: – Invalid: • Malformed ASCII file or other anomaly. Normally, you wouldn’t even see these, as the clients generally filter them from view. – Proto: • A new ticket still being built by REQADM. Not really a request yet. Again, clients tend not to show these to anyone until they’re full-blown tickets.

Ticket Status • REQADM rates each ticket’s status: – Re-queued: • Ticket has been moved to a different queue; this number is the old ticket number it used to have. • Kept for historical references. • Read-only copy; can’t be changed further. • REQADM 2. 0’s flat numbering space will render this obsolete.

Ticket Status • REQADM rates each ticket’s status: – Resolved: • Work completed. SLA “clock” stopped. Referred to customer for approval or denial. – Closed: • Work completed and approved by customer. – Pending: • “On hold” pending action by 3 rd-party or customer. SLA “clock” stopped while on hold, unless explicitly committed to a specific date and time.

Ticket Status • REQADM rates each ticket’s status: – New: • Submitted but not yet assigned to an owner. – Due: • Due (by SLA or commitment) within one hour. – Stale • Not modified within grace period (site configurable). – Active: • Any ticket not matching any of the above.

Agenda • What is REQADM, inside and out? – System Architecture • Directory Layout – Ticket States and Statuses – Overview of Clients – Server Design and Components

Overview of Clients • REQADM includes several standard clients which provide user interfaces to the system. • Also maintenance clients (discussed later). • Custom clients may be written locally in C++ or Perl.

ASCII Tools • reqadm – If ticket number is already known, can interactively operate on ticket, doing just about anything the X interface can do. – Batch-capable: can do all operations using command line arguments in non-interactive mode. – Can perform multiple operations on many tickets in one step.

ASCII Tools • reqadm-find – Displays queue list (one ticket per line). – Use to search for tickets or monitor queues. – Options to look for specific ranges of tickets. • Phonetic subject search. • Search by date ranges, people involved, etc.

ASCII Tools • reqadm-create – Creates new ticket for customer. – Support may invoke on behalf of customer. – ASCII input field editing. – Spawns external text editor for multi-line fields (user option) or uses line-at-a-time entry system.

GUI Tools • xreqadm – Integrated tool with actively-updated queue display and “editor” windows to work with individual tickets. – Can perform multiple operations on a single ticket, or a single operation on multiple tickets. – Arguably the most useful way for support personnel to use REQADM.

GUI Tools • reqadm-create – If a $DISPLAY variable is set in the environment, reqadm-create will attempt to shift to X mode rather than ASCII mode. – Displays an interactive GUI form to fill out.

GUI Tools • reqadm-prefs – Allows users (or support on behalf of a user) to customize their personal settings. – “Look and feel” of all REQADM clients affected. – E-mail sent to user (from whatever tickets) customized to their own preferences.

Web Tools • Most of the GUI functionality also available on web server (clients respond to being run as CGI rather than stand-alone). • CGI environment seriously hinders what tools can do, so this is not the most desirable way to access REQADM, especially for support personnel.

Mail Tools • reqadm-mail-receiver – Invoked to handle all incoming mail to REQADM, for new tickets and notes added to old ones. – Written in Perl, so lots of customization possible. – Invokes the batch “reqadm” commands to do the actual work.

Report Tools • reqadm-report – General-purpose reporting package for REQADM. – Pulls collection of tickets from server, analyzes them for various metrics. – Produces Structured ASCII, Post. Script or Comma-Separated ASCII outputs, suitable for printing or importing into spreadsheets and other programs.

Agenda • What is REQADM, inside and out? – System Architecture • Directory Layout – Ticket States and Statuses – Overview of Clients – Server Design and Components

Server Design • Server is the central manager of all activity within REQADM: – Storage and retrieval of tickets. – Management of simultaneous access between users (lock management, update management). – Tracks user customization and site configuration information. – Interacts with external services such as CDIS and LDAP.

Server Design Individual Client Connections. . . Client Session Management layer Server Command Interpreter Primitive Operations layer Lock Configuration Personnel DBMS Manager Connections to External Services

Server Subsystems • DBMS – Performs database management services. – REQADM 1. x uses this to store “overview” data (essential attributes of tickets) for fast queries and displays. Details still stored in ASCII files. – REQADM 2. 0 will move to all ticket data being stored in the database, for increased speed, and enhanced search capabilities.

Server Subsystems • Lock Manager – Arbitrates who gets to access tickets. – Currently supports both read (simultaneous) and write (exclusive) locks, but most common clients expect to update ticket, so always get write locks. • This means anyone viewing a ticket will lock out all other users from touching it. – Locks expire after a site-configurable length of time, or the client disconnects from the server.

Server Subsystems • Lock Manager, continued – Locks will be broken only if already expired another client wants the same lock. – Even if lock broken, original writer allowed to update if the intervening locker(s) didn’t modify the ticket. – REQADM 2. 0 will introduce more advanced locking, allowing simultaneous reads/writes.

Server Subsystems • Configuration Manager – Serves site configuration to clients. Discussed in detail later. • Personnel Manager – Handles user information in the database. – Serves user info to clients, allows clients and automated processes to update database from external sources (e. g. , Human Resources).

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

External Services vs. REQADM • REQADM does not operate in a vacuum. • Gets information about customers and support personnel from external personnel information servers and/or files. • Interacts with mail system to send and receive tickets and updates. • Web interface interacts with web server and authentication systems.

User Identification Infrastructure • User ID needed for tracking and security. • Initially loaded from static files – /etc/passwd – site “user table” CDIS Request Server LDAP Usertable* passwd *and CDIS and stdusers files. • Dynamic lookup possible if you use LDAP and/or CDIS servers, which REQADM will talk to.

User Identification Infrastructure CDISID” • All users are identified by “world-wide number. Request Server – Any number between 1 and 134, 217, 727. • (That’s 0 x 00000001 and 0 x 07 ffffff. ) LDAP – Allows usernames, full names, e-mail addresses, user ID numbers, etc. passwd to change usertable without losing ticket tracking history. • But how do we map a person to a WWID in the first place, and maintain that mapping?

Static User ID Files CDIS • The baseline REQADM system reads four static files to get this user information: Request Server – /etc/passwd (or passwd NIS map) LDAP • Defines full name, user ID number and username (therefore E-mail address) for each local user. – Site “usertable” file usertable passwd • Associates username with WWID number.

Static User ID Files • Usertable is a simple ASCII file, one. CDIS line per user, to map WWIDRequest to UNIX username: 1042: steve 1100: scaggs Server LDAP usertable WWID username passwd

Static User ID Files CDIS • The baseline REQADM system reads four static files to get this user information: Request – /opt/reqadm/lib/stdusers Server LDAP • Defines WWIDs (with names and mail addresses) for users not in the password file (including administrative accounts which wouldn’t be in the usertable passwd usertable file). – Site “CDIS” data file • Defines personal information (phone numbers, office location, etc. ) for each WWID. Optional.

Static User ID Files CDIS • /opt/reqadm/stdusers is formatted just like usertable, but is for defining non-user IDs Request within REQADM. May Server include additional LDAP fields: 10000: reqadmad@mydo main. com: REQADMusertable Administrator passwd WWID username E-mail Full name

Static User ID Files CDIS • /opt/reqadm/stdusers could be used for: – The REQADM administrator account (which Request Server probably won’t be in CDIS) – Other “non-human” accounts which may LDAP send or manipulate tickets – Accounts for customers or vendors who aren’t usertable passwd in your password file

Static User ID Files • CDIS file is supposed to be imported. CDIS on a regular basis from some. Request corporate HR Server or other database, customer database, LDAP appropriate source • Tab-delimited ASCII file with one line per usertable passwd user, identified by WWID number. • REQADM only uses certain fields from this file:

Static User ID Files • • 00 06 16 19 21 24 25 CDIS Full name of user Request E-mail address Server Mailstop LDAP Phone number for messages (secretary? ) Office location (room/cubicle number) usertable passwd Pager number Office phone number

Static User ID Files CDIS • 32 WWID number Request • 33 Cellular phone number Server • 39 Date and time entry was last updated LDAP – In format such as mm/dd/yyyy hh: mm. See Intro. txt for description of other legal formats. passwd – This date field is used usertable by REQADM to determine whether CDIS data is more recent than REQADM’s own database (triggering an update from this file into the REQADM database).

Static User ID Files CDIS • Nightly process updates REQADM database from these files. Request Server • This may not be timely enough, or contain LDAP all users who want to use REQADM. • Dynamic updates of user information possible using externalusertable services: passwd – LDAP – CDIS

E-Mail Address Resolution • When REQADM needs to identify a. CDIS user based only on E-mail address Request (such as when Server processing incoming mail messages): – If address looks like “domainusername”LDAP (NT style), we assume the local NT domains are in sync with the UNIX password so we drop usertable file, passwd the domain part and search for username: • In REQADM’s database of known usernames • Using a CDIS server, if one is available.

E-Mail Address Resolution CDIS of • Otherwise, if the address is only a string digits with no domain or. Request punctuation, we Server and look it up assume it is a WWID number, LDAP as explained below. – We have off-site users who authenticate to the web using their WWIDusertable number and a password, passwd so this is how they appear to the web-based REQADM clients.

E-Mail Address Resolution CDIS • Otherwise, we check REQADM’s database of known E-mail addresses, with and Request Serveraddress (we’ll without canonizing the mail try adding the local mail domain to LDAP unqualified addresses). usertable passwd

E-Mail Address Resolution CDIS • If all the above fails, we contact an LDAP directory server (if one is available) which Request is configured to translate. Server mail addresses to WWID numbers for our organization. LDAP • If no such user could be found, the client passwd may elect to generate ausertable temporary WWID to use for this user.

WWID Lookups • If we are given a WWID number for. CDIS a user, we look them up in the system Request by: Server we retrieve the – If the WWID is a group number, LDAP group information from the REQADM database (which comes from workgroups. conf). – Otherwise, if the WWID isn’t found in usertable passwd REQADM’s database already, a CDIS server is contacted (if available) to provide supplimental info for the user, and that is stored in REQADM’s database for future use.

User ID Resolution • Most clients look at the user ID they. CDIS are running under, and resolve that to a Request Server REQADM WWID number: LDAP – REQADM’s database of user ID’s is searched. – getpwuid() called to translate user ID to username. usertable passwd – Site “usertable” file opened and searched to get a WWID number for that username. – CDIS is contacted to get supplimental info and a new record is created in the database.

LDAP Server CDIS • The basic idea here is that you can set up an LDAP sever which contains all the valid ERequest Server your mail addresses known within organization, and associates them with. LDAP WWID numbers. usertable like passwd • Useful for large organizations Intel which have lots of users who may have accounts at multiple company sites.

LDAP Server CDIS • The LDAP server used by REQADM should accept queries of. Request the general form: – o=Organization-name Server – (mail=address-being-searched-for) LDAP • The response should include an attribute usertable passwd called “employeenumber” which contains the WWID for that address. • All of the above query components are site configurable to match your server settings.

CDIS Server • This is a simple server which serves. CDIS corporate personnel information in response Request Server to a query in one of the following formats: – iname – wnumber LDAP usertable passwd • If the specified local username or WWID number is not known, CDIS responds: – Key ’name’ not found.

CDIS Server • If a record was found, data lines are CDIS returned: WWID= 1234 Request Domain. Address= jrandom@mydomain. com Server Fname= JOHN LDAP Lname= RANDOM Mailstop= X 1 -111 usertable passwd MI= X Office. Loc= X 1 -1 -11 Msg. Num= 5551111

CDIS Server CDIS • Easy to implement if you have the corporate data somewhere you can. Request get to it (on-line database, static file, etc. ). Server LDAP • Can be written in Perl without much effort. • We intend to include a small sample CDIS usertable passwd server in REQADM 2. 0’s distribution.

Mail Server Interaction Incoming Ticket Update Notices Response to Notice Mail Server Approval/Denial DB Request Server New Ticket Receipts

Mail Server Interaction Incoming Ticket • Tightly integrated with mail services Update Notices • “Use the tool you’re Mail currently using” Response to Notice – Customer gets E-mail notice of ticket activity Server New Ticket Receipts – Can simply “reply” to mail—reply goes back to Approval/Denial REQADM, is noted in ticket history. • New commentary at top of each mail message—gives feel of casual E-mail Request conversation between parties. Server DB

Mail Server Interaction Incoming Ticketsends its own outgoing mail by • Each client contacting the SMTP server. Update Notices Mail Notice • Response Returntomail is spooled and fed into the Server New Ticket Receipts REQADM server by reqadm-mail-receiver Approval/Denial script – Generally run every 5 minutes via cron. DB Request Server

Web Server Interaction Browser Request Server Web Server Browser NT Auth Server htpasswd

Web Server Interaction • Most REQADM functions available via CGI. Browser • Problem Request of authenticating user to REQADM. – Server What functions to allow (security)? Browser Webaction on tickets? – Whom to log as taking Server Browser • Easiest is to make CGI directory passwordprotected, entries Browser NT Auth and make sure all users have in Server the web server password file. htpasswd

Web Server Interaction • We assume the web username is the same as: –Request UNIX username, Browser – Server Numeric-only WWID number (no “P” prefix), Browser – E-mail address, or Web Server Browser – NT-style username (domainusername) • It’s the web server’s responsibility to Browser NT Auth authenticate the user by whatever means Server necessary (NT auth server, . htpasswd file, etc. ) htpasswd

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

Installation • We won’t go through a tutorial of the installation process here. • But please refer to all these documents before installing: – README file – RELEASE-NOTES file • What’s in this release, known issues, etc. – Intro. txt file • Lots and lots of infrastructure details, etc.

Installation • However, there are some setup parameters you need to figure out before you install. – These will be “burned in” to the REQADM binaries when installed. Changing them will require a re-install. – Why? Mostly because it’s difficult to get so many customers to set environment variables or dotfiles to point REQADM clients to the local server and configuration directories. So we burn that info into REQADM directly.

Installation Parameters • REQADM server hostname and TCP port – Clients will connect sockets there to talk to server. • Pathname to Perl 5 interpreter – Written into #! lines of Perl scripts. • Pathname to GNU Tar utility – Written into some scripts which use tar, also used by the install program to unpack files.

Installation Parameters • REQADM home directory – Called “/opt/reqadm” here; server files will be installed there. • Pathname REQADM web images, HTML dir – Setup will install files there. • Pathname to client directory – Clients will look here to find help files.

Installation Parameters • Assigned site codes “A” and “B” – Together, these form a 64 -bit number which clients send to assure the server they are the correct clients for that server. (Not really air-tight security, but helps keep accidents from happening. ) – Each can be any number from 99, 999 to 2, 147, 483, 647. Just be consistent throughout all your installations on all platforms.

Installation Parameters • REQADM account user ID – Numeric UNIX user ID number for the account set aside to run the REQADM server. The setup program will chown all files to this account. • Web server account user ID – Numeric UNIX user ID number for the account which runs your web server. If a REQADM client is run by this account, it assumes it’s a CGI client and reacts accordingly.

Installation Parameters • Other parameters are configured at runtime and may be changed at will. • Caveat: web pages contain many of these runtime configuration values (e. g. , your organization name and phone number). – These are static HTML files, so you must configure these items before installing the web support. Changing these fields requires reinstalling the web support.

Installation Parameters • The fields affected are listed in Intro. txt. • In REQADM 2. (mumble), the static pages will all be replaced by a dynamic document server, which can get these values in real time from the REQADM server, so this won’t be a problem anymore.

Runtime Configuration • The parameters used to configure REQADM are stored in several files: – reqadm. conf – catcodes. conf – hours. conf – queues. conf – systems. conf – workgroups. conf

Configuration: reqadm. conf • This is the master configuration file for the REQADM system. • Values here available to any client. • Sets most of the policies and behavior of REQADM.

Configuration: reqadm. conf • REQADM is very touchy about mistakes: – A missing field will crash the server and/or client which was trying to access that field. – Be very careful to follow the file’s syntax rules. – Since this controls so much of REQADM’s internal code, be very sure you know what you are doing before changing these values.

Configuration: reqadm. conf • ASCII file, defining one “field” (i. e. , parameter) per line. • Comments begin with a “#” as the first character on the line (no whitespace allowed to the left). – “#” elsewhere on the line has no special meaning. • Lots of comments in the file, explaining what the fields do. Pay close attention to them.

Configuration: reqadm. conf • All fields have a four-character name. – Can actually be any four bytes, including nonprintable characters (but we recommend sticking to printable ASCII). – Case-sensitive. – REQADM will never define fields with names beginning with “$” (e. g. , “$ABC”), so you are free to use those names for custom clients which need central configuration fields.

Configuration: reqadm. conf • Text fields defined as: SUPN=Acme Widgets, Inc. Support Field name Field value

Configuration: reqadm. conf • Multi-line text fields defined as: CAT@=E-mail problem CAT@=Workstation crashed CAT@=Need software installed Field name It’s a multi-line (array) field if the field’s name ends with “@” Field value

Configuration: reqadm. conf • Numeric fields defined as: LTO#=3600 Field name It’s a numeric field if the field name ends with a “#” Field value

Configuration: reqadm. conf • Numeric fields may be in different bases: LTO#=3600 LTO#=$3 FF LTO#=@277 LTO#=%010111111 Decimal Hex Octal Binary

Configuration: reqadm. conf • This file sets policies enforced by REQADM: – Grace period before resolved tickets automatically close w/o customer approval, – Lock timeout interval, – Definition of “stale” ticket (how old), – Account to notify for all ticket resolutions and commitments

Configuration: reqadm. conf • This file sets identity information about your group: – Support center name and phone number, – Escalation mail address if customer not satisfied, – E-mail domains local to your group, – Return address and ID name for REQADM mail, – Your local area code

Configuration: reqadm. conf • This file sets resource location information: – Pathnames of other configuration files, – URLs of web components, – Host info for SMTP, CDIS, LDAP, etc. , – Commands for printing, paging, etc. , – Pathname to client help files, – Pathnames to static user ID files

Configuration: reqadm. conf • This file defines custom content: – HTML code to add to stock REQADM home page, – Text of error messages to be mailed out, – Which web graphics are used • Very skilled (and brave) admins can use these to change the appearance of the web forms used by REQADM. • REQADM 2. (mumble) will extend this to include GUI tools’ graphic images and colors as well.

Configuration: reqadm. conf • This file controls REQADM behavior: – Mail receiver parameters, alias names, etc. , – Category to use when merging tickets, – What queues customers can submit tickets into, – Input problem type list, – Required and/or edit-locked input fields, – Standard pre-define queue views for xreqadm

Configuration: reqadm. conf • Changes made to this file will be seen as soon as the reqadm server daemon is sent a HUP (1) signal. • Clients may have these values cached; they will need to be restarted to load fresh values again. – REQADM 2. 0 will include code to notify the clients when these values change, so that will no longer be necessary.

Configuration: catcodes. conf • Defines all root-cause categories available to categorize tickets. – Different than the problem type (CAT@) in reqadm. conf, which was the simple “what kind of complaint I have” data point supplied by the customer. – This is the more detailed “what actually caused the problem in the first place” data point supplied by the support tech resolving the ticket.

Configuration: catcodes. conf • These may be used to analyze the major issues and problem trends. • Hierarchial list of categories; for example: – Software Problems • Need upgrade • Bug in application – Workstation Problems • Dead hardware • Misconfigured system

Configuration: catcodes. conf • File defines one category per line. – Comments as with reqadm. conf – Use “/” to show hierarchy (like a pathname): sw sw/bug sw/upgrade ws ws/dead ws/config = = = Software Products Bug in application Upgrade needed Workstation Problems Dead hardware Mis-configured system

Configuration: catcodes. conf • File defines one category per line. – Comments as with reqadm. conf – Use “/” to show hierarchy (like a pathname): sw = Software Products sw/bug = Bug in application sw/upgrade = Upgrade needed ws = Workstation Problems Category code (used Category ws/dead = Dead hardware description Obsolete 2 nd internally and with field (can be ws/config = Mis-configured(multiple system command-line options any nonwords okay) space text)

Configuration: catcodes. conf • Don’t forget to include the category defined in reqadm. conf’s CDUP field. • Changes to this file may not be noticed until the server is restarted. This will eventually be fixed so that sending a HUP signal will suffice.

Configuration: hours. conf • Defines the hours your center is “open”. • Used for calculating SLA-based due dates for tickets. • A single line containing hour numbers for “open” and “close” time, Sunday through Saturday. • Will probably be moved into reqadm. conf at some point.

Configuration: hours. conf 0 0 8 17 … 8 17 0 0 Closed Sunday Open Monday 8: 00 -17: 00 Open Tuesday 8: 00 -17: 00 Open Friday 8: 00 -17: 00 Closed Saturday

Configuration: queues. conf • Defines the ticket queues used by REQADM. • File defines one queue per line. – Comments as with reqadm. conf. – First queue is the default queue for all clients. general restore project 005 010 040 400 0 010 010 000 000 000 0

Configuration: queues. conf • Defines the ticket queues used by REQADM. • File defines one queue per line. – Comments as with reqadm. conf. – First queue is the default queue for all clients. general restore project Queue name SLA hours (urgent) 005 010 040 400 0 010 010 000 000 000 0 Not used SLA hours (low)

Configuration: queues. conf • Removing a queue from this file will not delete it from REQADM. Once a queue is set up (especially if it contains tickets), REQADM is too paranoid to delete it. – You could, however, scratch and reload your database and just happen to fail to load that queue back in. – But beware if that (deleted) queue had tickets requeued to other active queues, or vice versa. REQADM will still need to have it around.

Configuration: queues. conf • Changes to this file will not be noticed by REQADM until the server is restarted. • This will eventually be fixed so sending a HUP signal will suffice.

Configuration: systems. conf • Defines an initial set of host names, locations and platform types to associate with tickets. • File defines one system per line. Intel: : Linux|lenny|John’s Office Intel: : Linux|laverne|Mary’s Office Sun: : 3/60|balthmus|Joey’s Office Intel: : Windows: : NT 4|pc 123|Accounting Intel: : Windows: : 98|demo 27|Sales Laptop

Configuration: systems. conf • Defines an initial set of host names, locations and platform types to associate with tickets. • File defines one system per line. Intel: : Linux|lenny|John’s Office Intel: : Linux|laverne|Mary’s Office Location Sun: : 3/60|balthmus|Joey’s Office Platform name. Intel: : Windows: : NT 4|pc 123|Accounting Note hierarchy Hostname Laptop with “: : ” Intel: : Windows: : 98|demo 27|Sales (unique in separating each database) level.

Configuration: systems. conf • Removing a system from this file will not remove it from the database. This file is just used to load into (merge with) the current database. • The contents of this file are re-read when the reqadm server is started or receives a HUP signal.

Configuration: workgroups. conf • Defines privileged users: those who can own tickets and have “administrator” access to the REQADM tools. – I. e. , your systems administration staff. • Also defines what group(s) each user belongs to. – Can then see a queue view for all tickets owned by the group (those owned by any members of the group), to simplify managing the group.

Configuration: workgroups. conf • Also defines “rotations” – “Fake” groups corresponding to areas of ownership (accounts, postmaster, etc. ). – Tickets may be assigned to the rotation group, then given to the actual owner who is in the rotation at that point in time. • Manually by a dispatcher or the recipient themselves • Or automatically via tools like REQMGR.

Configuration: workgroups. conf • One group/rotation/user per line. • Users must be defined after the group they belong to. – Multiple entries for the same user indicate they belong to multiple groups. • Groups may be nested. • Each group is assigned a number in the same range as WWID’s (but separate numbering space; will not conflict)

Configuration: workgroups. conf • Sample file: group 1 N support member 1042 steve group 2 N mailteam 1 member 1157 joey member 1291 beth group 100 Y postmaster group 101 Y accounts

Configuration: workgroups. conf • Sample file: group 1 N support member 1042 steve group 2 N mailteam. Group 1 name (shouldn’t be Start member 1157 joey the same as an existing defining user) member 1291 beth a group 100 Y postmaster 101 Y accounts Thisgroup is The group may not own group #1 tickets (not a rotation)

Configuration: workgroups. conf • Sample file: group 1 N support member 1042 steve group 2 N mailteam 1 Start member 1157 joey defining a UNIX username (must 1291 beth member of match with WWID in the support group 100 Y postmaster database) group 101 Y accounts WWID

Configuration: workgroups. conf • Sample file: group 1 N support member 1042 steve group 2 N mailteam 1 Start member 1157 joey definingmember 1291 beth This group is directly a group 100 Y postmaster under group #1 in the hierarchy (so “support” group 101 Y accounts This is includes everyone in group #2 “mailteam” too).

Configuration: workgroups. conf is • This Sample file: group #100 group 1 N support This group may own Start member 1042 steve tickets, so this is actually defining a defining 1 a group 2 N mailteam“rotation”. member 1157 joey member 1291 beth group 100 Y postmaster group 101 Y accounts

Configuration: workgroups. conf • Changes to this file will be reflected when the REQADM server receives a HUP signal. • You’d usually do this when hiring a new support person, adding them to the list of privileged users.

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

Maintenance • Normally, REQADM should run fairly smoothly on its own, with little or no supervision. • Maintenance tasks are automated and run from cron. • The only usual exceptions are reconfiguring or reacting to a problem.

Nightly Cron Script • /opt/reqadm/sbin/reqadm-nightly runs out of cron when users are not expected to need the system. – Closes requests which have not been approved – Suspends request server – Backs up database (only n days worth kept) – Updates personnel info from all static user ID files. – Allows server to continue running again.

Nightly Cron Script • Watch the mailed output from this script to make sure it’s not indicating that disk space is running low or any other problems are occurring.

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

Suspending the Server • Some of the following actions require suspending the REQADM server while they are performed (e. g. , any external access to the database files). • This is done with the reqadm-control command. • Like everything else we’ll discuss, this is in /opt/reqadm/sbin. Examples assume you’re in that directory.

reqadm-control • . /reqadm-control -locks – Prints a list of all tickets currently locked: G ID- TIME- QUEUE----- NUMBER---W 037 00: 05 request 130000 HOSTNAME/IP-NUM USERNAME COMMENT--192. 168. 1. 1 steve REQADM

reqadm-control • . /reqadm-control -locks – Prints a list of all tickets currently locked: G ID- TIME- QUEUE----- NUMBER---W 037 00: 05 request 130000 Lock grade (write) HOSTNAME/IP-NUM USERNAME COMMENT--192. 168. 1. 1 steve REQADM Session ID Lock Age Ticket

reqadm-control • . /reqadm-control -locks – Prints a list of all tickets currently locked: G ID- TIME- QUEUE----- NUMBER---W 037 00: 05 request 130000 HOSTNAME/IP-NUM USERNAME COMMENT--192. 168. 1. 1 steve REQADM System connecte d from Username Program running

reqadm-control • Use to see who’s getting in the way of your shutting down the server. • Also useful if people complain that they have requests locked: find the system/process where their client is still running.

reqadm-control • . /reqadm-control -suspend – Suspend the server when remaining locks released, and do something according to other options specified: -verbose – Print informational messages showing what’s happening. -time mm: ss – Abort attempt to suspend server if the specified time expires before locks are released.

reqadm-control • . /reqadm-control -suspend – Following the options is one of the following: mm: ss – Suspend for the specified length of time, then resume server operations. commands – Execute the shell commands specified. When they complete, resume server operations.

Checking Database Integrity • To check to see if the database is intact: . /dbchk. . /db/reqadm-db. sc – Verifies that database is internally consistent and valid for the DBMS to use. – Does not check that the data inside the database actually make sense. – Database password is “lao: ban 0”. • SUSPEND SERVER BEFORE RUNNING!

Clearing Database Lockfiles • If something died and left a lockfile behind, clear it with: . /dbchk -u. . /db/reqadm-db. sc – Be very certain the lockfile is not valid (i. e. , that the database server or other program isn’t really running, accessing the database now). – Database password is “lao: ban 0”. • SUSPEND SERVER BEFORE RUNNING!

Recovering Trashed Databases • If something catastrophic happened, you can recover the database from backups, to quickly get the system back up and running. • This is tricky (we need a script to automate this), and if you don’t have a huge number of tickets, it’s probably better to clear and reload the database instead. • In fact, I’m not going to teach it here until we do get a script for it. . .

Reloading Database From Scratch • Delete the entire contents of the database: . /dbinit. . /db/reqadm-db. sc – Database password is “lao: ban 0”. – If this won’t work because the database is locked, you can use a -u option just like with dbchk (and with the same cautions). • SUSPEND SERVER BEFORE RUNNING!

Reloading Database From Scratch • If the dbinit failed because the database is too corrupt even for that, you’ll have to recover an old reqadm-db. sc file. These are saved nightly as. . /db/Backup/REQADM_ DBS. *. *. A. Copy one of these to the. . /db directory under the name reqadm-db. sc and try the dbinit again.

Reloading Database From Scratch • Now that the database is empty, load all your personnel information from the static files: . /reqadm-update-db -people -verbose • For speed, this blasts the data directly into the database itself, bypassing the REQADM server (which must not be running). • SUSPEND SERVER BEFORE RUNNING!

Reloading Database From Scratch • Now we need to load all the tickets from the ASCII files into the REQADM database. • If you have time to spare, leave the server down and use this program to bulk-load all the tickets directly into the database (bypassing the server, which must not be running): . /reqadm-update-request-db • SUSPEND SERVER BEFORE RUNNING!

Reloading Database From Scratch • If time is not on your side, and the server must be started ASAP, you can load only the most critical tickets by naming them as explicit arguments to the update program (this loads only those tickets, and prevents the normal queue scan): . /reqadm-update-request-db queue/n. . . • SUSPEND SERVER BEFORE RUNNING!

Reloading Database From Scratch • What’s most critical? Maybe you want to load all the open tickets but not old, closed ones: cd. . /spool foreach queue (*). . /sbin/reqadm-update-request-db $queue/[0 -9]* end • SUSPEND SERVER BEFORE RUNNING!

Reloading Database From Scratch • If you only load partial tickets, there is one important set of tickets you must load before bringing up the server: always load the highestnumbered ticket in each queue! – These might be closed already, so hunt in the Fixed/D* directories for them. – Load them explicitly as explained already, but note they are specified as queue/number even if stored in the Fixed directories. • SUSPEND SERVER BEFORE RUNNING!

Reloading Database From Scratch • The reason for this is that if a new ticket is received, REQADM will assign a new ticket number higher than what it currently sees in the queue. If the highest existing ticket was not loaded, a duplicate ticket number will be issued by REQADM. • Once (at least) the highest ticket in each queue is loaded, you may restart the server.

Bulk-Loading Tickets • If you wish to bulk-load tickets while the server is up and running (perhaps to update status of all tickets, or maybe to finish loading a set of tickets partially loaded as noted a moment ago), run: . /bulk-loader – This scans all your queues, invoking reqadmbulk-load-request-db to load the tickets into the server.

Bulk-Loading Tickets • This takes longer, but can be done while the server’s up, and people are accessing the request system.

Adding/Correcting Individual Tickets • If you want to update the database to reflect what’s in a ticket (perhaps after editing the ASCII ticket file directly): . /reqadm-bulk-load-request-db q/no – This is done with the server up and running normally.

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

Reqmgr • Request queue manager – Monitors open requests – Automatically takes action based on state • Highly customizeable – Modular design – Uses Perl API REQMGR material by John Mechalas, PMD EC

Reqmgr • Useful tasks – notify people when a request is submitted – auto-assign request from “rotations” to people – escalate “ignored” requests (extended time w/o a human owner) – auto-assign requests with special “Subject: ” tags

Getting Started with reqmgr • reqmgr. conf – defines “trigger” conditions – module called when request state matches a rule – modules actually do the work

reqmgr. conf • Directives – “Add. Module” specifies a module to be loaded – “Match” determines whether or not multiple rules can match a single request – see the man page • Directive syntax Add. Module <modname> Match <single|multiple>

reqmgr. conf • Macros – substitue one string for another – expanded in rulesets only! – no quoting: value string taken literally – can be nested… – …but there’s no recursion protection • Macro syntax Define macroname value string …

reqmgr. conf • Variables – Sets a variable name to a text string – No quoting: value string taken literally – Visible to all reqmgr modules • Variable syntax Set varname value string …

reqmgr. conf • Rulesets – define “trigger” conditions that activate a module – processed in the order they are defined • Ruleset syntax Rule ruleid: field : : = target value Module modname

Rulesets • Rule expression – possible request fields defined in Reqmgr(3 p) – : : = is an equality, expression match, or list lookup operator (can be negated) – value is the data that field is compared to – see reqmgr. conf(5) for details • modname must be declared via Add. Module

Sample config Add. Module rotation Set rot_config rotation. conf Set rot_count_queues request incident restore Set rot_assign_msg (reqmgr assignment) Define ROTATION unixreq postmaster ntreq Rule 1: owner =@ ROTATION Module rotation

Sample Config • Activates module “rotation” if owner one of: unixreq postmaster ntreq • Sets variables to be used by “rotation” module

Sample Config • Module “rotation” – distributed with REQADM – uses config file to define rotation membership • day of week or date • time of day • rotation name – assigns requests to rotation members based on request load

reqmgr Modules • Module API – written in Perl – documented in Reqmgr. Module. API(3 p) – see sample modules for development hints

Agenda • • What is REQADM, inside and out? How does it interact with external services? How do I set up and configure REQADM? What standard maintenance is required? What can I do for special cases or problems? How can I automate tasks with REQMGR? How can I stay informed about changes?

Keep In Touch • Subscribe to the reqadm-users-l mailing list: – Send mail to majordomo@ichips. intel. com – In the message body place the text: subscribe reqadm-users-l end – You’ll receive announcements of new releases, problems discovered, enhancements planned. – Also a discussion forum for asking questions of interest to all REQADM users around the world.

Read the Files • When you download a new release of REQADM, be sure to read the RELEASENOTES file to see what has changed since the last release.

Further Reading • The REQADM distribution includes an unadvertised technical documentation section. • Assuming you installed REQADM’s web files to /reqadm on your server, go to: http: //. . . /reqadm-tech. html • Includes design information, database schemas, client/server protocol documentation, etc.

Copyright © 2000 Intel Corporation. All Rights Reserved. May be distributed under same terms and conditions as the REQADM distributions.