Bio Data Development Tech Talk Project Structure Application

Bio. Data Development Tech Talk • • • Project Structure Application Features Technology Stack Development Methodology/Practices Tooling

Project Overview/Objectives • A secure, modern, web-oriented, extensible and adaptable system for the capture, storage, curation, and distribution of bioassessment and supporting habitat data commonly collected for NAWQA and other USGS projects. • Version 1 to support stream macroinvertebrate, algal periphyton, fish-community and instream habitat data collected using protocols of USGS National Water-Quality Assessment Program (NAWQA) and the USEPA National Rivers and Streams Assessment (NRSA).

Goal In this meeting the Development Lead (Mike Stern) for the Biodata project, will discuss several key areas of the project, and what he views as keys that led to a successful project. Discussion and questions are encouraged. About Me • • • Work for Comprehensive Computer Consulting (CCC) out of Sun Prairie, WI Work out of Wisconsin Science Center…co-working with CIDA in Middleton, WI Java developer for 12 years, mostly enterprise level in-house applications Development lead on Bio. Data—prefer server side/persistence, web api Love doing code reviews, and refactoring

Project Structure • Clients: Project managers/Business analysts • Development team: (Small, never > 8 devs at one time) – CCC devs are prime (3 -5 active developers) – USGS devs rotate in and out, (1 -4 active developers) – CIDA administration/hosting services (OS, Oracle, network, desktop & infrastructure) and coworking facility • Agile--Both Client and Developer buy in 2 week iterations. 3 -4 meetings per iteration: planning, mid iteration, preplanning, quarterly planning. • Daily standups (separate client and developers) • 3 main projects (Data Entry, ETL, Reporting/warehousing) • Integrations with Legacy Inputs (NWIS Sites, Bio. TDB, Labs)

Clients • Adopted the Lean/Agile methodology into their processes • Story and Acceptance Test Driven Development • Confluence (WIKI) and JIRA (task management system) are main communication tools. • Understand the iterative development cycle and have adopted Agile practices within their group

Developers • Small Team (< 8 devs at all times usually around 4 active). • Easy communication, IM, standups, low protocol • Let team self assign. Strengths quickly become evident & let members gravitate towards them (benefits project and developer alike) • Constant design discussions, little diagraming aside from whiteboards. Let code speak rather than outdated documents • Low formality code review

Agile / Lean Software Development – Mary & Tom Poppendieck Lean Principals Eliminate Waste Build Quality In Create Knowledge Defer Commitment Deliver Fast Respect People Optimize the Whole

Daily standups • About 15 minutes max… 1 -2 minute individual reports • What did you do yesterday, what are you doing later today. Any blockers? • Encourages cross communication and global awareness of changes in the system (key to integration) • Table lengthy discussions to be discussed immediately after standup. • Separate meetings for development and business analyst staff members.

Project Components • 3 separate ‘Projects’ • Biodata Input—Data entry, auto save, allow ‘invalid data’, continuous validation • Biodata Retreival — Canned reports with flexible (and reusable) query criteria • Biodata ETL — Transforms Biodata input and Legacy data into a more normalized format for use in retrieval…inject visibility rules • Quick Demo

Application Features • REST services to JS front end. . . clear decoupling • Auto Save with Real time validation (avoid save buttons) • Top Down/drill down paradigm. Start general work narrow: Project List of Field Samples Filter to narrow Edit/View Sample • Quick Search using index tables for ‘random’ access

DEMO • Biodata Input (QA): https: //igsarmewfsbd 3 b. er. usgs. gov: 8443/bio data. Web/ • Biodata Retreival (QA) https: //igsarmewfsbd 3 b. er. usgs. gov: 8443/bio share. Web/

REST: Quick Explanation Based solely on HTTP – the lingua franca of the Internet Everything is implemented as a PUT, POST, GET, or DELETE Everything is a RESOURCE, and everything has a URI • Dynamic object hierarchy inflation (with api) – https: //igsarmewfsbd 3 b. er. usgs. gov: 8443/biodata. Web/se rvice/project/11528? inflate. Option=: (project. Owner, project Staff)

Technology Stack • layered java, REST-ful webservice, ext. JS UI, Oracle • JBoss AS 5. 1, Apache, cent. OS • Key Java libraries: Spring, Ibatis 2, RESTeasy, Jasper Reports • Oracle XML tables using hierarchical queries, prepared statements

System Structure

Reference Application Architecture

Development System

Development Methodology/Practices • Test Driven Development--Make it Break, Make it Work, Make it Pretty • Continuous integration: good coverage, encourage early and often check ins • Push Button Deploy and 'releases' • Light weight process (code reviews, build and deploy, documentation) • Optimize when necessary

Testing Unit testing – All significant custom code has unit tests using the Junit test framework. We don't write unit tests for third-party java frameworks, just for the code we write. Encourage Test Driven Development - Tests are written before the implementing code…helps work out the API & design Tests are the ultimate place to start when onboarding new devs…can exercise the stack starting with tests…debug through Require tests for custom code check-in. Cobertura used to report on test coverage. Not religious about coverage, but it’s a good data point. Developers encouraged to run tests locally, continuous build and integration runs all tests each time code is checked into SCM.

Continuous Build and Integration Jenkins (formerly Hudson) used to implement continuous build, test, integration, on-demand deploy. Entire system is built after each developer checks in code. Developers are encouraged to check code in multiple times a day to reduce merge and integration problems. Unit and integration tests are run each time changes to SCM are detected. May include one or more check-ins. If build “breaks” entire team is informed. Members may check Hudson “dashboard” to determine who needs to do what. Release plugin encourages process for code elevation and following the Trunk, snapshot, branches workflow SVN loves so much. Also reinforces maven type development snapshot, Release, and Hotfix mentality.

Tooling: Rely on automation to make the developers job 'easy'. Leverage open source tools. • Confluence • JIRA • Eclipse • SVN • Maven • Jenkins • Jboss • IReport • Selenium • Remedy