Advanced Customisation Scripting EPrints Training Course Southampton May

Advanced Customisation: Scripting EPrints Training Course Southampton, May 3 -4 th 2007

Taking Control: the EPrints API x EPrints configuration files offer many opportunities for customisation and control x branding, workflow, controlled vocabs, authority lists, deposit types, metadata. . . x EPrints API offers many more opportunities x the more perl-intensive configuration files xe. g. eprint_render. pl x and beyond. . xplugins xcommand-line tools

Roadmap x Core API 1. manipulating your data 2. accessing data collections 3. searching your data x Scripting techniques 1. essentials – putting it all together 2. writing export plugins 3. writing screen plugins 4. writing command-line tools 5. writing CGI scripts

Part 1: Core API

About This Part of the Talk x Light on syntax xobject->function(arg 1, arg 2) x Incomplete x Designed to xgive you a feel for the EPrints data model xintroduce you to the most significant (and useful!) objects xhow they relate to one another xtheir most common methods xact as a jumping off point for exploring

Finding Documentation x EPrints modules have embedded documentation x Extract it using perldoc x perldoc perl_lib/EPrints/Search. pm

Core API: Manipulating Your Data

Data Model: 3 Core Objects x EPrint xsingle deposit in the repository x Document xsingle document attached to an EPrint x User xsingle registered user User EPrint Document

Data Model: Core Relationships x 1 User owns (deposits) many EPrints x 1 EPrint has many documents attached to it x 1 Document may contain many files, but these are not part of the API xe. g. PDF = 1 file xe. g. HTML + images = many files User 1 * EPrint 1 * Document

Data Model: Data. Obj x All data objects inherit from Data. Obj x Provides common interface to data Data. Obj User 1 * EPrint 1 * Document

Accessing Data: Data. Obj interface x get_id() x get_url() x. EPrint – abstract page x. User – user summary page x. Document – document download x get_type() x. EPrint – article, book, thesis. . . x. User – user, editor, admin x. Document – pdf, html, word. . .

Manipulating Data: Data. Obj Interface x get_value(fieldname) xget the value of the named data field xeprint->get_value( “title” ) x set_value(fieldname, value) xset the value of the named field xdoc->set_value( “format”, “pdf” ) x is_set(fieldname) xtrue if the named field has a value xuser->is_set( “email” )

Manipulating Data: Data. Obj Interface (2) x commit() xwrite any changes made to the object through to the database xe. g. after using set_value x remove() xerase the object from the database xalso removes any sub-objects and files xe. g. eprint->remove xremoves EPrint and associated Documents from DB xremoves Document files from filesystem

Getting Hold of Existing Data Objects x new(session, id) xreturns data object for an existing record x EPrints: : Data. Obj: : EPrint->new(session, 1) x EPrints: : Data. Obj: : User->new(session, 1) x EPrints: : Data. Obj: : Document->new(session, 1) x User object has extra options xuser_with_email(session, email) x user_with_username(session, username)

Creating New Data Objects x Slightly different for each data object x EPrint xcreate(session, dataset, data) x User xcreate(session, user_type) x Document xcreate(session, eprint)

Specific Methods x Each data object also has specific methods for manipulating their data

EPrint Methods x get_user() x get a User object representing the user to whom the EPrint belongs x get_all_documents() x get a list of all the Document objects associated with the EPrint x generate_static() x generate the static abstract page for the eprint xuseful when you’ve modified the eprint values! x in a multi-language archive this will generate a page in each language

User Methods x get_eprints(dataset) xget a list of EPrints owned by the user x mail(subject, message) xsend an email to the user

Document Methods x get_eprint() xget the EPrint object the document is associated with x local_path() xget the full path of the directory where the document is stored in the filesystem x files() xget a list of (filename, file size) pairs

Document Methods: Main File x get_main() x set_main(main_file) xget/set the main file for the document xthis is the file that gets linked to xin majority of cases, Document will have 1 file xe. g. PDF xbut there may be some cases where a Document has many file xe. g. HTML document =. html files, images, stylesheets xset main to top level index. html

Document Methods: Adding Files x add_file(file, filename) x upload(filehandle, filename) xboth add a file to the document xadd_file uses full path to file xupload uses file handle xin both cases the document will be named filename

Document Methods: Adding Files (2) x upload_url(url) xgrab file(s) from given URL xin the case of HTML, only relative links will be followed x add_archive(file, format) xadd files from a. zip or. tar. gz file x remove_file(filename) xremove the named file

Other Data Objects x Subject x a node in the subjects tree x Saved. Search x a saved search associated with a User x History x an event that took place on another data object x e. g. change to eprint metadata x Access x a Web access to an object x e. g. document download x Request x a request for a (restricted) document x Explore these using perldoc

Core API: Accessing Data Collections

Accessing Data Collections x We’ve looked at individual data objects x but a repository holds many eprints and documents and has many registered users x 2 key ways to manipulate data objects collectively: 1. built-in datasets x large fixed sets of data objects 2. by searching the repository x set of data objects matching specific criteria

Datasets x. All data objects in the repository are part of a collection called a dataset x 3 core datasets: xeprint xall eprints xuser xall registered users xdocument xall documents

Datasets (2) x. Also 4 subsets within eprint dataset which collect eprints in same state xarchive xall eprints in live archive xinbox xall eprints which users are still working on xbuffer xall eprints submitted for editorial review xdeletion xall eprints retired from live archive

The Data. Set Object x Gives access to all the data objects in a particular dataset x Also xtells us which data fields apply to that dataset xrecall get_value and set_value methods xa repository’s metadata is configurable so this gives us a way to find out: xwhich fields are available in a particular repository xthe properties of individual fields

Accessing Data. Sets x count(session) xget the number of items in the dataset x get_item_ids(session) xget the IDs of the objects in the dataset x map(function, args) xapply function to each object in the dataset xfunction is called with args: x(session, dataset, dataobj, args)

Fields in a Data. Set x has_field(fieldname) xtrue if the dataset has a field of that name x get_field(fieldname) xget a Meta. Field object describing the named field xget_fields() xget list of Meta. Field objects describing all fields in the dataset

Datasets and Meta. Fields x. A Meta. Field xis a single field in a dataset xtells us properties of the field xget_property(name) xset_property(name, value) xe. g. name, type, input_rows, maxlength, multiple. . . xbut not the field value xthe value is specific to the individual data object xe. g. eprint->get_value(“title”)

Core API: Searching the Repository

Searching the Repository x The Search object allows us to search datasets for data objects matching specific criteria x Provides access to the results

Starting a New Search x new(options) xcreate a new search expression xmust specify which dataset to search in xsearch = new Search( session => session, dataset => dataset, custom_order => “title” ) xmany other options can be specified xexplore with perldoc

Adding Search Fields x add_field(metafield, value) xadd a new search field with the given value (search text) to the search expression xadd as many fields as you like to the search criteria

Adding Search Fields: Example x Example: full text search x search->add_field( dataset->get_field(“title”), “routing”, “IN”, “ALL” )

Adding Search Fields: Example (2) x Example: full text search which matches word in title or abstract x search->add_field( [ dataset->get_field(“title”), dataset->get_field(“abstract”) ], “routing”, “IN”, “ALL” )

Adding Search Fields x Example: date search x search->add_field( dataset->get_field(“date”), “ 2000 -2004”, “EQ”, “ALL” )

Processing Search Results x Carry out a search using: xlist = search->perform_search() x Returns a List object which gives access to search results

The List Object x Any ordered collection of data objects xusually the results of a search

Processing Lists xcount() xget the number of results xget_ids(offset, count) xget_records(offset, count) xget an array if data objects, or just their ids xoptionally specify a range using count and offset xmap(function, args) xapply the function to each data object in the list

Manipulating Lists x newlist = list->reorder( neworder ) x newlist = list->union( list 2 ) x newlist = list->intersect( list 2 ) x newlist = list->remainder( list 2 )

Part 2: Scripting Techniques

Roadmap ü Core API ü manipulating your data ü accessing data collections ü searching your data x Scripting techniques 1. essentials – putting it all together 2. writing export plugins 3. writing screen plugins 4. writing command-line tools 5. writing CGI scripts

Scripting Techniques: Essentials

Putting it all together x Two essential objects 1. Session x connects to the repository x many useful methods 2. Repository x provides access to x datasets x session->get_repository->get_dataset(“archive”) x configuration settings x Explore using perldoc

Scripting for the Web x API provides lots of methods to help you build Web pages and display (render) data xthese methods return (X)HTML xbut not strings! x. XML DOM objects x. Document. Fragment, Element, Text. Node. . . x Build pages from these nodes xnode 1 ->append. Child(node 2) xwhy? it’s easier to manipulate a tree than to manipulate a large string

XML DOM vs. Strings x p = make_element(“p”) x p = “<p>” x text = make_text( x p += “Hello World” ) x p->append. Child(text) x p += “</p>” <p>Hello World</p> Can manipulate tree to add extra text, elements etc. p Hello World Difficult to make changes to the string – would need to find the right position first

Render Methods: Session x Session provides many useful Web page building blocks x make_doc_fragment() x create an empty XHTML document x fill it with things! x make_text(text) x create an XML Text. Node x make_element(name, attrs) x create an XHTML Element make_element("p", align => "right") <p align=”right” />

Render Methods: Session (2) x render_link(uri, target) x create an XHTML link 1. link = session-> render_link(“http: //www. eprints. org“) 2. text = session->make_text(“EPrints") 3. link->append. Child(text) <a href=”http: //www. eprints. org”> EPrints</a>

Render Methods: Session (3) x html_phrase(phraseid, inserts) x render an XHTML phrase in the current language x looks up phraseid from the phrases files x inserts can be used to add extra information to the phrase xmust be a corresponding <epc: pin> in the phrase x<epp: phrase>Number of results: <epc: pin name=“count“/></epp: phrase>

Render Methods: Session (4) x Many methods for building input forms, including: xrender_form(method, dest) xrender_option_list(params) xrender_hidden_field(name, value) xrender_upload_field(name) xrender_action_buttons(buttons) x. . .

Rendering Methods: Data Objects x render_citation(style) x render_citation_link(style) xcreate an XHTML citation for the object xif style is set then use the named citation style x render_value(fieldname) xget an XHTML fragment containing the rendered version of the value of the named field xin the current language

Rendering Methods: Meta. Fields x render_name(session) x render_help(session) xget an XHTML fragment containing the name/description of the field in the current language

Rendering Methods: Searches x render_description() xget some XHTML describing the search parameters x render_search_form(help) xrender an input form for the search xif help is true then this also renders the help for each search field in current language

Getting User Input (CGI parameters) x. Session object also provides useful methods for getting user input xe. g. from an input form xhave_parameters xtrue if parameters (POST or GET) are available xparam(name) xget the value of a named parameter

Scripting Techniques: Writing Export Plugins

Plugin Framework x EPrints provides a framework for plugins x registration of plugin capabilities x standard interface which plugins need to implement x Several types of plugin interface provided x import and export xget data in and out of the repository x interface screens xadd new tools and reports to UI x input components xadd new ways for users to enter data

Plugin Framework (2) x Not just a plugin framework for 3 rd party extensions! x Used extensively by EPrints itself x majority of (dynamic) Web pages you see are screen plugins x search, deposit workflow, editorial review, item control page, user profile, saved searches, adminstration tools. . . x all import/export options implemented as plugins x all input components in deposit workflow are plugins x subject browser input, file upload. . .

Plugin Framework (3) x EPrints is really a generic plugin framework x with a set of plugins that implement the functions of a repository x Gives plugin developers many examples to work from x find a plugin that does something similar to what you want to achieve and explore how it works Plugin Framework Backend (data model)

Writing Export Plugins x Typically a standalone Perl module in x perl_lib/EPrints/Plugin/Export/ x Writing export plugins 1. register plugin 2. define how to convert data objects to an output/interchange format

Export Plugin: Registration x Register x name x the name of the plugin x visible x who can use it x accept x what the plugin can convert x lists of data objects or single data objects (or both) x type of record (eprint, user. . . ) x suffix and mimetype x file extension and MIME type of format plugin converts to

Registration Example: Bib. Te. X $self->{name} = "Bib. Te. X"; $self->{accept} = [ 'list/eprint', 'dataobj/eprint' ]; $self->{visible} = "all"; $self->{suffix} = ". bib"; $self->{mimetype} = "text/plain"; x Converts lists or single EPrint objects x Available to all users x Produces plain text file with. bib extension

Registration Example: FOAF $self->{name} = "FOAF Export"; $self->{accept} = [ 'dataobj/user' ]; $self->{visible} = "all"; $self->{suffix} = ". rdf"; $self->{mimetype} = "text/xml"; x Converts single User objects x Available to all users x Produces XML file with. rdf extension

Registration Example: XML $self->{name} = "EP 3 XML"; $self->{accept} = [ 'list/*', 'dataobj/*' ]; $self->{visible} = "all"; $self->{suffix} = ". xml"; $self->{mimetype} = "text/xml"; x Converts any data object x Available to all users x Produces XML file with. xml extension

Export Plugin: Conversion x For a straight conversion plugin, this usually includes: 1. mapping data objects to output/interchange format 2. serialising the output/interchange format x e. g. End. Note conversion section: $data->{K} = $dataobj->get_value( "keywords" ); $data->{T} = $dataobj->get_value( "title" ); $data->{U} = $dataobj->get_url;

Export Plugin: Conversion (2) x. But export plugins aren’t limited to straight conversions! x. Explore: x. Google Maps export plugin xplot location data on map xhttp: //files. eprints. org/224/ x. Timeline export plugin xplot date data on timeline xhttp: //files. eprints. org/225/

Export Plugin: Template 1. Register x subclass EPrints: : Plugin: : Export x inherits all the mechanics so you don’t have to worry about them x could subclass existing plugin e. g. XML, Feed x define name, accept, visible etc. x in constructor new() of plugin module 2. Conversion x define output_dataobj function x will be called by plugin subsystem for every data object that needs to be converted

Writing Import Plugins x Typically a standalone Perl module in x perl_lib/EPrints/Plugin/Import/ x Reading input can be harder than writing output x need to detect and handle errors in input x many existing libraries available for parsing a wide variety of file formats x Writing import plugins 1. 2. register define how to convert input/interchange format into data objects x reverse of export

Scripting Techniques: Writing Screen Plugins

Plugins: Writing Screen Plugins x One or more Perl modules in x perl_lib/EPrints/Plugin/Screen/ x may be bundled with phrases, config files, stylesheets etc. x Writing screen plugins 1. register x where it appears in UI x who can use it 2. define functionality

Screen Plugin: Registration x Register xactions xthe actions the plugin carry out (if any) xappears xwhere abouts in the interface the plugin and/or actions will appear xnamed list xposition in list xwill be displayed as link, button or tab

Registration Example: Manage Deposits $self->{appears} = [ { place => "key_tools", position => 100, } ]; key_tools list

Registration Example: EPrint Details $self->{appears} = [ { place => "eprint_view_tabs", position => 100, }, ]; eprint_view_tabs list (each tab is a single screen plugin)

Registration Example: New Item $self->{appears} = [ { place => “item_tools", position => 100, action => “create”, }, ]; item_tools list (create action will be invoked when button pressed)

Screen Plugin: Define Functionality x 3 types of screen plugin 1. Render only x define how to produce output display x examples: Admin: : Status, EPrint: : Details 2. Action only (no output display) x define how to carry out action(s) x examples: Admin: : Indexer. Control, EPrint: : Move, EPrint: : New. Version 3. Combined (interactive) x define how to produce output/carry out action(s) x examples: EPrint: : Reject. With. Email, EPrint: : Edit, User: : Edit

Screen Plugins: Displaying Messages x Action plugins produce no output display but can still display messages to user xadd_message(type, message) xregister a message that will be displayed to the user on the next screen they see xtype can be xerror xwarning xmessage (informational)

Screen Plugin Template: Action Only 1. Register x subclass EPrints: : Plugin: : Screen x define actions supported x define where actions appear x define who can use actions x allow_ACTION function(s) 2. Define functionality x define action_ACTION function(s) x carry out the action x use add_message to show result/error x redirect to a different screen when done

Screen Plugin Template: Combined x render function usually displays links/buttons which invoke the plugin’s actions xe. g. EPrint: : Remove xregisters remove and cancel actions xrender function displays Are you sure? screen x. OK/Cancel buttons invoke remove/cancel actions

Scripting Techniques: Writing Command Line Scripts

Command Line Scripts x Usually stored in bin directory x Add batch/offline processes to your repository xe. g. duplicate detection – compare each record to every other record xe. g. file integrity - check stored MD 5 sums against actual MD 5 sums

Connecting to the Repository x Command line scripts (and CGI scripts) must explicitly connect to the repository by creating a new Session object xnew(mode, repositoryid) xset mode to 1 for command line scripts xset mode to 0 for CGI scripts x And disconnect from the repository when complete xterminate() xperforms necessary cleanup

Using Render Functions x XHTML is good for building Web pages xbut not so good for command line output! xoften no string equivalent xuse tree_to_utf 8() xextracts a string from the result of any rendering method xtree_to_utf 8( eprint->render_citation)

Search and Modify Template x Common pattern for command line tools 1. Connect to repository 2. Get desired dataset 3. Search dataset 4. Apply function to matching results x modify result x commit changes 5. Disconnect from repository

Example: lift_embargos x Removes access restrictions on documents with expired embargos 1. Connect to repository 2. Get document dataset 3. Search dataset x embargo date field earlier than today’s date 4. Apply function to matching results x remove access restriction x clear embargo date x commit changes 5. Disconnect from repository

Scripting Techniques: Writing CGI Scripts

CGI Scripts x Usually stored in cgi directory x Largely superceded by screen plugins but can still be used to add e. g. custom reports to your repository x Similar template to command-line scripts but build Web page output using API render_ methods

Building Pages x In Screen plugins, mechanics of sending Web pages to the user’s browser are handled by the plugin subsystem xneed to do this yourself with CGI scripts xmethods provided by the Session object x build_page(title, body) xwraps your XHTML document in the archive template x send_page() xflatten page and send it to the user

Summary ü Use the core API to manipulate data in the API ü individual data objects üEPrint, Document, User ü sets of data objects üData. Set, List, Search ü Wrap this in a plugin or script ü Session, Repository ü Web output using render_ methods ü templates ÞNext: hands-on exercises designed to get you started with these techniques