CVS Reorganization Installation Reorganization A Simple Build System

CVS Reorganization, Installation Reorganization & A Simple Build System Steve Fischer October 24, 2002

Goals o Make GUS portable n n o Self-explanatory CVS structure n n o o Projects wait for other projects only when necessary Easily maintain released versions while development continues Support development on a unified file system (new file server) n o Improve collaborators’ ability to understand our code structure And ours (particularly new folks) Improved ability to release projects on a schedule n o Easily build and configure runnable instances of our projects Avoid conflicting with the pre-existing structure of external sites Don’t rely on local file systems having different installed versions Encourage a move away from individual ownership of projects

CVS Reorganization

CVS Structure q At the root there are projects, which have components, which have parts: $CVSROOT/ Project 1/ Component 1/ bin/ lib/ Component 2/ Projectn/ parts

Projects o Units of software development having their own release schedule n o May depend on other projects n n o Have their own release number, eg, GUS 3. 1. 2, Annotator 1. 0. 0 Either on a specific release or ‘latest’ No circular dependencies Dependency example: n Do. TS->GUS->CSP

Proposed Projects (phase 1) o o o o o All. Genes Annotator (new) CSP (CBIL Style Police: web form assistance) DJob (distributed job controller) Do. TS (build, including Do. TS related plugins) GLE (grammars for transcription) GUS Para. DBs Plasmo. DB RAD (plugins & website) (or should these be separate? )

Projects Contain Components o For example: GUS/ Common/ DBAdmin Model/ o o A component is usually formed from one or a couple of package directories (eg, java packages). In addition to code, components have: n n o Documentation TODO list Configuration Required data A component may depend on others within its project

Proposed GUS/ Components o o o o Seq. Match/ Common/ DBAdmin/ Gen. Bank/ GOPredictor/ Model/ Obj. Rel. J/ Obj. Rel. P/ Pipeline/ Plugin. Mgr/ Schema. Browser/ Util/ Web. Dev. Kit/

Proposed GUS Components in Detail - 1 o Seq. Match/ n o Common/ n o Data model for genbank parser GOPredictor/ n o Database admin scripts Gen. Bank/ n o Commonly used schema dependent stuff, such as common plugins DBAdmin/ n o Blast, BLAST 2, and BLAT object models Predict GO function Model/ n n Schema definition files Hand edited objects (eg, GUS: : Model: : Do. TS: : Assembly. pm. man)

Proposed GUS Components in Detail - 2 o Obj. Rel. J/ n o Obj. Rel. P/ n o Web based schema browser Util/ n o GA and friends Schema. Browser/ n o Pipeline management API Plugin. Mgr/ n o Perl object layer: superclasses and code generator Pipeline/ n o Java object layer: superclasses Commonly used schema-independent utilities Web. Dev. Kit/ n Servlet and/or JSP based

Components Contain Standard Parts o As needed: Component 1/ bin/ cgi-bin/ config/ data/ doc/ htdocs/ lib/ src/ test/

Component Parts in Detail - 1 o Bin n o Cgi-bin/ n o Cgi executables Config/ n n o executables Properties files for the component Files typically named component. prop Doc/ n n n Specifications User guides TODO

Component Parts in Detail - 2 o Data/ n o Htdocs/ n o Extra data needed by component, eg, matrices, Static web pages Test/ n Test data

Component Parts in Detail - 3 o Lib/ n n Contains linkable object files Perl includes full package path (Project: : component: : module) lib/ perl/ Project/ component/ *. pm xml/ *. xml

Component Parts in Detail - 4 o Src/ n Contains source that needs to be compiled or otherwise transformed before it can be installed src/ c/ java/ edu/ cbil/ project/ component/ org/ gusdb/ component/

Process to Migrate CVS Structure S CV ut ko ec ch copy orig. CVS/ GUS/ perl/ … www/ perl/ lib/ Fasta. pm CSP/ orig from. CVS/ GUS/ perl/ … www/ perl/ lib/ Transform script Fasta. pm CSP/ pruned new. CVS/ Do. TS/ Common/ lib/ perl/ Common/ Fasta. pm CSP/ GUS/ new

Process to Migrate CVS Structure o Do transform n n n Check out old to orig. CVS/ Copy to from. CVS/ Run script to transform from. CVS/ to new. CVS/ o o moves files (thus pruning from. CVS/) Validate transform by: n n Examing transform script (see where YOUR stuff is going to be) Examine from. CVS/ to see what I left behind o Create new CVSROOT with new. CVS/ (losing history) o How can I get everybody’s OK on the transform? ?

Releases o o A project is the unit of release When it is released, it is tagged in cvs with a release number, eg 2. 1. 1 At that time, other projects can declare a dependency on that particular release (discussed in detail later) Bug fixes are applied to the tagged branch

Installation Reorganization

Objectives o Install into a single relocatable location: $GUS_HOME n n n o Be able to find (almost) all GUS installed resources there: n n n o o Executables Libraries Documentation Configuration Some data Third party resources Support multiple running instances on a machine, eg dev, beta Make explicit the versions of each included resource n o Don’t conflict with the site’s pre-existing structure Easy to install and uninstall Avoid path and classpath conflicts In a file called $GUS_HOME/versions Also have installation targets for websites and appl. servers

Installing Project 1 to $GUS_HOME P 1/ C 1 C 2 $GUS_HOME/ Bin/ Lib/ Doc/ P 2/ C 1 C 2 P 3/ C 1 C 2 = dependencies

Understanding $GUS_HOME o o o The home for a single GUS related installation Contains one or more projects, and the projects they depend on Including third party resources (such as Bio. Java or Bio. Perl) May contain projects that are running separately, as long as they don’t have conflicting dependencies (May even contain the entire set of GUS projects) It contains a version file, eg: Do. TS 1. 2. 1 GUS 2. 0. 0 CSP 3. 0. 1 Bio. Java 1. 1. 1

Sample $GUS_HOME/ bin/ dotsbuild extract. Seqs config/ dotsbuild. prop csp. prop gus. prop data/ matrices/ doc/ Do. TS/Dotsbuild/TODO GUS/Model/uml/*. uml CSP/User. Guide. html (cont’d)…

Sample $GUS_HOME (cont’d) $GUS_HOME/ … lib/ java/ gusmodel. jar guswdk. jar perl/ Do. TS/Dots. Build/*. pm GUS/Model/*. pm GUS/Obj. Rel. P/*. pm GUS/Web. Dev. Kit/*. pm CSP/*. pm xml/ *. xml

Installation Targets - Website o Contains cgi-bin and htdocs $/world/www. allgenes/ cgi-bin/ *. pl htdocs/ *. html

Installation Targets – Application Server o o EG, Tomcat Will copy. jar files and other stuff as needed

Configuration o o o Code never refers to absolute file paths Many resources are conveniently located relative to $GUS_HOME, so don’t need configuration Otherwise, code relies on property files for configuration We provide a perl and java Property. Set object to make this easy Property files are found in… $GUS_HOME/config Named after the component that is using them: n o gusmodel. prop Consider using properties instead of macro substitutions when possible

A Simple Ant-based Build System

Objectives o o o o Install projects to installation targets Serves two kinds of installs: n From $PROJECT_HOME to installation targets (developers) n From. tar file to installation targets (external users) Support inter-project dependencies Support dependencies on third party resources Handle object layer code generation Be easy to use Be relatively easy to maintain Be generic across projects and components, but also flexible

$PROJECT_HOME -> $GUS_HOME (developers) o Install the build system n n n o Check out the project of interest n o o o Create $PROJECT_HOME Check out the install/ project Copy build to a local bin/ build Do. TS install $GUS_HOME –co Now $PROJECT_HOME is an image of CVS containing your project and all projects it depends on. Edit away Install for testing n build Do. TS install $GUS_HOME

. tar -> $GUS_HOME (external users) o Untar the download file cd install/ setenv $GUS_HOME /usr/local/somewhere. /build Do. TS install $GUS_HOME o Also works for installing to a website o o o n . /build Gene. DB installweb /world/www. genedb

The Mechanics o Uses Ant (http: //jakarta. apache. org/ant/) n n o o Each project contains its own build. xml file The install/ directory contains the main build. xml file n n o Uses xml to declaratively specify the build Used extensively in the java community Calls the appropriate project’s build. xml Houses “subroutines” The project build. xml file n n n Specifies the project’s dependencies on other projects Specifies the project’s component’s dependencies on each other Calls default project and component build routines, unless custom ones supplied

The Mechanics (cont’d) o Java compilation n o Copying from $PROJECT_HOME to target (eg, $GUS_HOME) n n o o The build compiles java into the classes/ directory of a component It places. jar files into lib/java/ directory of a component Does compilation in dependency order Copies and merges all dirs, such as bin/, lib/ doc/ Can perform macro substitution on the way Creates version file Because Ant is kind of slow, build can do just a component instead of a whole project

Special case: object layer code generation o Generates into: n Perl: o o o n Java: o o GUS/Model/Do. TS/Assembly. pm. man Only generates if files are older than “schema definition” file: n o $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/dots $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/core Etc Hand created files have. man suffix, eg: n o $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/Do. TS $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/Core Etc $PROJECT_HOME/GUS/Model/schema/definition. sql This file is generated by schema modification process (and lives in CVS)

Making it All Happen

Goals o o Getting it done quickly With a minimum of disruption

Overarching strategy o o Work on projects and components one at a time Start with ones that are most central and not in development n n n o o o GUS/Obj. Rel. P GUS/Model/lib/perl/Model/Do. TS GUS/Plugin. Mgr GUS/Common Do. TS DJob Get that core working with new build system and $GUS_HOME In parallel, convert to GUS 3. 0 In parallel, bring in GUS/Obj. Rel. J and Annotator (Dave) Upgrade Web. Dev. Kit install apparatus Bring in Web sites one at a time

Inconveniences o During the time that a project is being upgraded: n n Everybody will need to commit their work to cvs Will need to maintain 2 copies in cvs: o o Old New Will need to merge changes to both Hopefully this will be short for most projects