Washington State Archives Digital Archives MultiState Preservation Consortium

Data Challenges • Files and metadata are often in different formats. • Metadata is

Transfer Challenges • Depending on the size of data (MB, GB, or even TB)

Administration Challenges • Users do not always do what we want them to do.

Apps to the rescue! • Archive This! – Responsible for transmitting and validating metadata

Archive This • Transforms a variety of data formats into one data format. •

Archive This Washington State Archives Documenting Democracy

Auto Todd (part 1) • Why Auto Todd? • Continually monitors for newly transferred

Auto Todd (part 2) • Processes and stages transferred data for ingestion into the

Intranet Admin Site • Ingestion coordinator can: – Manage users and their access to

Intranet Admin Site Washington State Archives Documenting Democracy

From Start To Finish • Ingestion coordinator adds an agency and its appropriate collections

Contacts Dan Waterbly dan. waterbly@sos. wa. gov Justin Jaffe justin. jaffe@sos. wa. gov Washington

Slides: 18

Download presentation

Washington State Archives, Digital Archives: Multi-State Preservation Consortium Digital Archives: We have apps for that! Presented by: Daniel Waterbly dan. waterbly@sos. wa. gov July 2010, Washington DC Washington State Archives Documenting Democracy

Data Challenges • Files and metadata are often in different formats. • Metadata is often incomplete or erroneous. • Linking mechanism between individual files and metadata is often broken. Washington State Archives Documenting Democracy

Transfer Challenges • Depending on the size of data (MB, GB, or even TB) copying/uploading times can be very long. • When data arrives it must be virus scanned and copied to the appropriate server and backup locations. • No guarantee that the files transferred to the Digital Archives are valid or adhere to a consistent directory structure. Washington State Archives Documenting Democracy

Administration Challenges • Users do not always do what we want them to do. • Need custom SQL scripts to monitor progress. • Many systems are involved creating configuration problems. Washington State Archives Documenting Democracy

Apps to the rescue! • Archive This! – Responsible for transmitting and validating metadata and files from a variety of data sources. • Auto Todd – Automates the ingestion process from start to finish. • Intranet Admin Site – Administers collections and ingestion apps. Washington State Archives Documenting Democracy

Archive This • Transforms a variety of data formats into one data format. • Validates metadata and files. • Transfers files to hard drive or to remote FTPS server and captures a digital fingerprint to ensure authenticity. • Ensures that files are copied or uploaded in a directory structure that the Digital Archives expects. Washington State Archives Documenting Democracy

Archive This Washington State Archives Documenting Democracy

Auto Todd (part 1) • Why Auto Todd? • Continually monitors for newly transferred data from Archive This. • Virus scans new data with up to seven virus scanners. • Verifies the integrity of the transferred data using the digital fingerprint file (contains a hash of every file that was sent) created by Archive This. Washington State Archives Documenting Democracy

Auto Todd (part 2) • Processes and stages transferred data for ingestion into the Digital Archives. • Monitors the ingestion system and sends notifications to the data submitters upon completion. • Backups data to tape for disaster recovery purposes (utilizes the digital fingerprint to ensure every bit is copied without error). • Fully automated, no humans needed! Washington State Archives Documenting Democracy

Intranet Admin Site • Ingestion coordinator can: – Manage users and their access to Archive This. – Manage collections. – View incoming data and its status. – Control ingestion systems and bring them online or offline for maintenance purposes. – Move data to backup tape. – Access a variety of custom reports using SQL Reporting Services. Washington State Archives Documenting Democracy

Intranet Admin Site Washington State Archives Documenting Democracy

From Start To Finish • Ingestion coordinator adds an agency and its appropriate collections to the database. • Ingestion coordinator creates a user account for the agency for Archive This. • User transforms their data using Archive This and uploads it to the Digital Archives. • Auto Todd is alerted when the upload finishes and processes the data. • Auto Todd notifies the user of completion. Washington State Archives Documenting Democracy

Contacts Dan Waterbly dan. waterbly@sos. wa. gov Justin Jaffe justin. jaffe@sos. wa. gov Washington State Archives Documenting Democracy