From LDAP to Patron Load Generating a patron

From LDAP to Patron Load Generating a patron update SIF file from an LDAP directory dump Michael Doran, Systems Librarian Endeavor Users Group Meeting, Chicago, IL Session 8, Thursday, April 26, 2007

Integrated library system patron database Cataloging Michael Doran 123 Elm St. Arlington, TX PATRON BLOCK patron DB Acquisitions Circulation Michael Doran, Systems Librarian doran@uta. edu

Getting patron data into Voyager D’oh! n Manual data entry via the Circulation client Michael Doran, Systems Librarian doran@uta. edu

Getting patron data into Voyager n n Manual data entry via the Circulation client Batch loading via the Pptrnupdt job q Technical User’s Guide n n 6 : Patron Update 19 : Patron Record Standard Interface File Michael Doran, Systems Librarian doran@uta. edu

The Patron SIF specification n Each record will consist of q 1 base segment - fixed length n n q 1 -9 address segments - fixed length n n q 19 fixed length fields one address is required 1 optional notes segment - variable length n q 52 fixed length fields certain fields are required 1 variable length field 1 end-of-record mark Michael Doran, Systems Librarian doran@uta. edu

Patron SIF – one record Michael Doran, Systems Librarian doran@uta. edu

Patron SIF – one record Michael Doran, Systems Librarian doran@uta. edu

Patron SIF documentation Michael Doran, Systems Librarian doran@uta. edu

Patron SIF documentation Item (field) number – 72 total Item (field) name Offset – starting point Item (field) description Format n - numeric s - string d - date Required field? Length of field Michael Doran, Systems Librarian doran@uta. edu

Patron SIF files the easy way… You send the Patron SIF documentation to your University IT department… Here’s your Patron SIF file. Enjoy! IF S Patron programmer … and voila Michael Doran, Systems Librarian doran@uta. edu

Patron SIF files the hard way… I want you to write a program that generates Patron SIF files. Enjoy! er g a y o V ls Manua your boss Michael Doran, Systems Librarian doran@uta. edu

So we have our marching orders Michael Doran, Systems Librarian doran@uta. edu

Somewhere on your campus is… n One or more directories of information on people registered to attend classes n One or more directories of information on people who work for the university students faculty and staff “patrons” Michael Doran, Systems Librarian doran@uta. edu

Directories, databases, protocols, APIs n directory an alphabetical or classified list (as of names or addresses) n database a collection of data organized for rapid retrieval (as by a computer) n protocol a standard that controls the connection, communication, and data transfer between two computing endpoints n application programming interface (API) an abstraction layer between the gory details of an application and the world outside that software Michael Doran, Systems Librarian doran@uta. edu

Directories, databases, protocols, APIs LDAP directory n directory an alphabetical or classified list (as of names or addresses) n database a collection of data organized for rapid retrieval (as by a computer) n protocol a standard that controls the connection, communication, and data transfer between Lightweight Directory Access Protocol two computing endpoints n application programming interface (API) an abstraction layer between the gory details of an application and the world outside that software Michael Doran, Systems Librarian doran@uta. edu

Directories, databases, protocols, APIs Voyager patron load n directory an alphabetical or classified list (as of names or addresses) n database a collection of data organized for rapid retrieval (as by a computer) n protocol a standard that controls the connection, communication, and data transfer “Standard Interface File” (SIF) between two computing endpoints n application programming interface (API) an abstraction layer between the gory Pptrnupdt details of an application and the world outside that software Michael Doran, Systems Librarian doran@uta. edu

Voyager patron database tables and relationships Michael Doran, Systems Librarian doran@uta. edu

The pieces and players LDAP server LDAP Protocol LDAP Net: : LDAP client Perl program ? patron SIF Pptrnupdt patron info Directory Voyager Michael Doran, Systems Librarian doran@uta. edu

The pieces and players LDAP server LDAP Protocol LDAP Net: : LDAP client Perl program ? patron SIF Pptrnupdt patron info Directory Voyager Michael Doran, Systems Librarian doran@uta. edu

Three easy(? ) steps… n Query the LDAP directory q n Process the records q q q n filter for valid patrons minor data formatting decision algorithms output records in SIF format Load SIF file into Voyager q Pptrnupdt Michael Doran, Systems Librarian doran@uta. edu

Three easy(? ) steps… n Query the LDAP directory q n Process the records q q q n filter for valid patrons Perl program minor data formatting decision algorithms output records in SIF format Load SIF file into Voyager q Pptrnupdt Michael Doran, Systems Librarian doran@uta. edu

Before we start Perl programming LDAP server LDAP Protocol LDAP ldapsearch Net: : LDAP client Perl program ? patron SIF Pptrnupdt patron info Directory Voyager Michael Doran, Systems Librarian doran@uta. edu

What you need to get started n command line client q n ldapsearch comes standard on Solaris LDAP server info q q hostname port numbers n n q n 389 – default ldap port 636 – default ldaps port LDAP Schema bind credentials q q Distinguished Name (DN) anonymous your own (e. g. Net. ID username & password) “superuser” (provided by LDAP administrator) Michael Doran, Systems Librarian doran@uta. edu

ldapsearch -h "ldap. uta. edu" -D "uid=doran, cn=accounts, dc=uta, dc=edu" -w "<password>" -b "cn=people, dc=uta, dc=edu" "uta. Account. Name=doran" Michael Doran, Systems Librarian doran@uta. edu

ldapsearch -h "ldap. uta. edu" <uid> -D "uid=doran, cn=accounts, dc=uta, dc=edu" -w "<password>" -b "cn=people, dc=uta, dc=edu" "uta. Account. Name=doran" LDAP Distinguished server Name Search Base Filter hostname Bind Credentials Michael Doran, Systems Librarian doran@uta. edu

ldapsearch output cedarid=1085691573581181000, cn=People, dc=uta, dc=edu uta. Account. Name=doran display. Name=doran, michael d uta. Prev. Account. Name=doran uta. Employee. Exemption=exempt uta. Primary. Account=doran uta. Employee. Phone=+1 817 272 5326 uta. Primary. Email=doran@uta. edu home. Postal. Address=123 Elm St. $Arlington$TX$76013 uta. Home. Street 1=123 Elm St. uta. ID=1000068572 uta. Employee. Title=Librarian uta. Employee. Percent. Time=100. 00 uta. Dept. Code=6800000 uta. Employee. Retirement. Code=9 uta. Employee. Status=active uta. Empl. ID=1000068572 uta. Employee. Campus. Box=19497 uta. Middle. Name=D edu. Person. Primary. Affiliation=staff object. Class=top uta. Home. City=Arlington object. Class=person uta. Employee. Title. Code=0100 object. Class=inet. Org. Person sn=Doran object. Class=uta. Person cedarid=1085691573581181000 object. Class=uta. Employee uta. Home. State=TX edu. Person. Scoped. Affiliation=employee@uta. edu home. Phone=+1 817 555 1234 edu. Person. Scoped. Affiliation=staff@uta. edu uta. Employee. Job. Category=prof_non_faculty uta. Person. Affiliation=employee cn=Michael D Doran uta. Person. Affiliation=staff uta. Employee. Type=monthly uta. Person. Affiliation=administrative uta. Home. Zip=76013 edu. Person. Affiliation=employee uta. Employee. Bldg=LIB edu. Person. Affiliation=staff given. Name=Michael search filter match “patron-worthy” search filter Michael Doran, Systems Librarian doran@uta. edu

LDAP search with Perl Net: : LDAP #!/usr/local/bin/perl -w use Net: : LDAP; $ldap = Net: : LDAP->new("ldap. uta. edu"); $mesg = $ldap->bind("uid=doran, cn=accounts, dc=uta, dc=edu", password => "<password>"); $mesg = $ldap->search( base => 'cn=people, dc=uta, dc=edu', filter => "(uta. Account. Name=doran)" ); foreach $entry ($mesg->all_entries) { $entry->dump; }; $ldap->unbind(); exit(0); Michael Doran, Systems Librarian doran@uta. edu

LDAP search with Perl Net: : LDAP #!/usr/local/bin/perl -w use Net: : LDAP; this Perl module will need to be installed $ldap = Net: : LDAP->new("ldap. uta. edu"); $mesg = $ldap->bind("uid=doran, cn=accounts, dc=uta, dc=edu", password => "<password>"); $mesg = $ldap->search( base => "cn=people, dc=uta, dc=edu", filter => "(uta. Account. Name=doran)" ); foreach $entry ($mesg->all_entries) { $entry->dump; }; $ldap->unbind(); LDAP Distinguished server Name Search Base Filter hostname Bind Credentials exit(0); Michael Doran, Systems Librarian doran@uta. edu

Perl Net: : LDAP output dn: cedarid=1085691573581181000, cn=People, dc=uta, dc=edu uta. Account. Name: uta. Primary. Account: uta. Primary. Email: uta. Home. Street 1: uta. Employee. Title: uta. Dept. Code: uta. Employee. Status: uta. Employee. Campus. Box: edu. Person. Primary. Affiliation: uta. Home. City: uta. Employee. Title. Code: sn: cedarid: uta. Home. State: home. Phone: uta. Employee. Job. Category: cn: uta. Employee. Type: uta. Home. Zip: uta. Employee. Bldg: given. Name: display. Name: . . . : doran@uta. edu 123 Elm St. Librarian 6800000 active 19497 staff Arlington 0100 Doran 1085691573581181000 TX +1 817 555 1234 prof_non_faculty Michael D Doran monthly 76013 LIB Michael doran, michael d. . . Michael Doran, Systems Librarian doran@uta. edu

Superuser & valid patron filtering #!/usr/local/bin/perl -w use Net: : LDAP; a superuser Bind DN $ldap = Net: : LDAP->new("ldap. uta. edu"); $mesg = $ldap->bind("cn=library, cn=app, dc=uta, dc=edu", password => "<password>"); $mesg = $ldap->search( base => 'cn=people, dc=uta, dc=edu', filter => "(|(uta. Student. Status=enrolled) (uta. Employee. Status=active) (uta. Employee. Status=retired))" ); foreach $entry ($mesg->all_entries) { $entry->dump; }; patron-worthy LDAP records . . . Michael Doran, Systems Librarian doran@uta. edu

Three easy(? ) steps… n Query the LDAP directory q n Process the records q q q n filter for valid patrons minor data formatting decision algorithms output records in SIF format Load SIF file into Voyager q Pptrnupdt Michael Doran, Systems Librarian doran@uta. edu

LDAP records simplified dn: cedarid=1085691573581181000, cn=People, dc=uta, dc=edu uta. Account. Name: uta. Primary. Account: uta. Primary. Email: uta. Home. Street 1: uta. Employee. Title: uta. Dept. Code: uta. Employee. Status: uta. Employee. Campus. Box: edu. Person. Primary. Affiliation: uta. Home. City: uta. Employee. Title. Code: sn: cedarid: uta. Home. State: home. Phone: uta. Employee. Job. Category: cn: uta. Employee. Type: uta. Home. Zip: uta. Employee. Bldg: given. Name: display. Name: . . . : attributes doran@uta. edu 123 Elm St. Librarian 6800000 active 19497 staff Arlington 0100 Doran 1085691573581181000 TX +1 817 555 1234 prof_non_faculty Michael D Doran monthly 76013 LIB Michael doran, michael d. . . values attribute-value pair Michael Doran, Systems Librarian doran@uta. edu

LDAP records made more complex n Not all records have all the same attributes additional attributes for employees generic attributes for all people additional attributes for students let’s consider “people” records Michael Doran, Systems Librarian doran@uta. edu

Attributes continued n n Some attributes are required, but many (most) are optional Some attributes have a constrained set of possible values Some attributes are “multi-valued” Attributes will not necessarily be sorted in any order for information… Michael Doran, Systems Librarian P A LD ma e h Sc doran@uta. edu

Sample schema page (UTA LDAP) Michael Doran, Systems Librarian doran@uta. edu

Sample schema page (UTA LDAP) Michael Doran, Systems Librarian doran@uta. edu

Processing records $mesg = $ldap->search( base => 'cn=people, dc=uta, dc=edu', filter => "(|(uta. Student. Status=enrolled) (uta. Employee. Status=active) (uta. Employee. Status=retired))", callback => &Process. Record ); foreach $entry ($mesg->all_entries) { $entry->dump; }; dumps records we want to process records The “callback” option allows you to process each record as it is received Michael Doran, Systems Librarian doran@uta. edu

Process. Record subroutine sub Process. Record { my ($mesg, $entry) = @_; if (!$entry) {&Finish(0); } $surname = $entry->get_value('sn'); $first_name = $entry->get_value('given. Name'); $middle_name = $entry->get_value('uta. Middle. Name'); $uta_id = $entry->get_value('uta. ID'); @uta_affil = $entry->get_value('uta. Person. Affiliation'); $email_addr = $entry->get_value('uta. Primary. Email'); $home_phone = $entry->get_value('home. Phone'); if ($home_phone) { $home_phone =~ s/+1 //; $home_phone =~ s/ /-/g; } #. . . yada, yada Michael Doran, Systems Librarian doran@uta. edu

Process. Record subroutine sub Process. Record { my ($mesg, $entry) = @_; single-valued assigning multi-valued LDAP attributes attribute =>=> values assigned to program totoa an scalar array variables if (!$entry) {&Finish(0); } $surname = $entry->get_value('sn'); $first_name = $entry->get_value('given. Name'); $middle_name = $entry->get_value('uta. Middle. Name'); $uta_id = $entry->get_value('uta. ID'); @uta_affil = minor $entry->get_value('uta. Person. Affiliation'); reformatting of data $email_addr = $entry->get_value('uta. Primary. Email'); $home_phone = $entry->get_value('home. Phone'); if ($home_phone) { $home_phone =~ s/+1 //; $home_phone =~ s/ /-/g; } #. . . yada, yada Michael Doran, Systems Librarian doran@uta. edu

LDAP record Find the Voyager patron group dn: cedarid=1085691573581181000, cn=People, dc=uta, dc=edu uta. Account. Name: uta. Primary. Account: uta. Primary. Email: uta. Home. Street 1: uta. Employee. Title: uta. Dept. Code: uta. Employee. Status: uta. Employee. Campus. Box: edu. Person. Primary. Affiliation: uta. Home. City: uta. Employee. Title. Code: sn: cedarid: uta. Home. State: home. Phone: uta. Employee. Job. Category: cn: uta. Employee. Type: uta. Home. Zip: uta. Employee. Bldg: given. Name: display. Name: . . . : doran@uta. edu 123 Elm St. Librarian 6800000 active 19497 staff Arlington 0100 Doran 1085691573581181000 TX +1 817 555 1234 prof_non_faculty Michael D Doran monthly 76013 LIB Michael doran, michael d. . . The LDAP record doesn’t have a patron group Michael Doran, Systems Librarian doran@uta. edu

Decision algorithms program patron group looks assignment at student attributes example first if ($stud_status eq "enrolled" && $stud_class) { if ($stud_class =~ /freshman/i || $stud_class =~ /sophomore/i || $stud_class =~ /junior/i || $stud_class =~ /senior/i ) { $patron_group = "und"; } elsif ($stud_class =~ /masters/i || $stud_class =~ /doctoral/i || $stud_class =~ /graduate/i) { $patron_group = "grad"; $expire_days = "180"; } } elsif ($stud_status eq "enrolled") { $patron_group = "und"; } student classification not a required attribute in LDAP record, so need a default Michael Doran, Systems Librarian doran@uta. edu

Decision algorithms program looks at employee attributes second if ($empl_status eq "active") { if (! $patron_group) { $patron_group = "staff"; } foreach my $affil (@uta_affil) { if (($affil =~ /faculty/i) || ($affil =~ /admin/i) || ($affil =~ /assistant/i)) { $patron_group = "fac"; $expire_days = "400"; } } default only if no } student patron group assigned Michael Doran, Systems Librarian doran@uta. edu

Decision algorithms continued multiple sets of address attributes employees work address & phone students all people home address & phone Michael Doran, Systems Librarian student address & phone doran@uta. edu

Three easy(? ) steps… n Query the LDAP directory q n Process the records q q q n filter for valid patrons minor data formatting decision algorithms output records in SIF format Load SIF file into Voyager q Pptrnupdt Michael Doran, Systems Librarian doran@uta. edu

A poetry break… “Our time is fixed, and all our days are number'd; How long, how short, we know not. ” Robert Blair, The Grave “The SIF fields are fixed, and all the fields are number'd; How long, how short, we better know. ” anonymous programmer Michael Doran, Systems Librarian doran@uta. edu

Patron SIF formatting simplified n For each patron SIF record q Format each field n n q q q s n n* d string – left-justified / blank-filled numeric – right-justified / zero-filled “special” numeric – if no value, then all blanks date – YYYY. MM. DD format Concatenate all fields into one long string Slap an end-of-record marker on string Write string to file Michael Doran, Systems Librarian doran@uta. edu

Remember this? Michael Doran, Systems Librarian doran@uta. edu

Formatting with sprintf* $first_name = sprintf("%-20 s", Michael); $patron_id = sprintf("%010 d", 123456); $concatenation = $first_name. $patron_id; print $concatenation; Michael 0000123456 *sprintf Michael Doran, Systems Librarian help doran@uta. edu

Three easy steps… n Query the LDAP directory q n Process the records q q q n filter for valid patrons minor data formatting decision algorithms output records in SIF format Load SIF file into Voyager q Pptrnupdt Michael Doran, Systems Librarian doran@uta. edu

Things I didn’t cover n Encryption n Data validation n Log files n Session file n Cron job setup n yada, yada… Michael Doran, Systems Librarian doran@uta. edu

You* can do this! *…you, or your resident geek LDAP protocol Patron SIF Perl programming geek stigmata Michael Doran, Systems Librarian doran@uta. edu

Resources n Voyager Technical User’s Guide q q n Perl Net: : LDAP module q q n download from CPAN documentation Perl Date: : Calc module q q n Patron SIF specification Patron Update (Pptrnupdt) download from CPAN documentation LDAP to patron SIF webpage (ldump. pl) Michael Doran, Systems Librarian doran@uta. edu

Questions? Please fill out the session evaluation! Michael Doran, Systems Librarian doran@uta. edu