Evaluation of a Danish Digital Migration Project Danish

Evaluation of a Danish Digital Migration Project Danish National Archives (DNA)

Goals of the evaluation Report to the Ministry of Culture Spread the word internationally (are we the first ones? ) Lessons learnt for future migration projects

Background Legal and preservation perspectives The Danish National Archives decides in which formats and structures public institutions submit their data for ingest It hasn’t always been like that, and data submitted before 1998 had very heterogeneous datastructures and -formats A standardisation of these data was therefore necessary in order to comply with today’s preservation standard

The digital migration project Challenge and purpose of the Migration Project -Standardise records to make future operations automatic Records from before 1998 Preservation standard (Execution Order no. 342) s s ce Ac Mig rati AUTOMATION on

The digital migration project The big numbers 2005 -2008; 30 person years 10 -15 specialised project members Investment costs: 190. 000 $ Total costs: 2. 6 million $ 1. 7 GB data from 1960 -90’ies in over 200 different database structures totaling over 2. 200 Information Packages

The digital migration project The Object of the Migration Period Content Amount of data No. of files No. of Information Packages Datafiles on tape and 1963 -1998 paper documentation 171 GB 3, 109 650 Datafiles on cd-r and disks and paper 1998 -2000 documentation 419 GB 8, 078 641 2000 - Electronic filing systems and registries 1, 187 GB - 942 Total - 1, 777 GB 11, 187 2. 233

The digital migration project Prerequisites #1 Media Migration project

The digital migration project Prerequisites #2 Development of preservation standard

The digital migration project Knowledge management The knowledge challenge Need of disappearing knowledge (ancient databases, code pages, COBOL) Need of particular manpower capable of executing long, repetitive manual tasks in a structured and disciplined manner The knowledge management setup A favourable framework for easy access to knowledge via online databases and dynamic intranet Two major documents dedicated to migration principles and working procedures. Updated at all times. Immense importance.

The digital migration project Migration method and procedure Establishment of Software development specification requirements What to do - Requirements for the underpinning of the manual work: The Migration Principles How to do it - Underpinning of the manual work: The Working Procedures

The digital migration project Preparation of data and documentation Preparation of data Restructuring of data and documentation into new IP structure Registration of the IPs metadata into the preservation databases Transfer to an intermediate storage, waiting for migration Preparation of documentation Documenting the IPs Establish a documentation standard Gather, create and systematise documentation Scanning and registration of documentation Documenting the migration Logfiles

The digital migration project Software development In-house software development for reasons of control and specialisation of task 1 flexible migration tool was developed for all the pre 1998 archives, and 1 for the post 1998 ones Enabling automatic migration for the older archives required the development of special pre -migration software that could handle the myriads of different formats and structures.

The digital migration project The Migration - Illustration

The digital migration project Conclusions Migration (of old data) is expensive! The transformation of 1. 7 GB of data (1, 825, 361, 100 bytes) and its prerequisites took 50 person years (65, 000 hours) or 27. 4 KB per hour to migrate

The digital migration project Conclusions Meeting preservation principles is expensive! The Danish National Archives do not dispose of archives once they’ve been appraised and deemed worthy of preservation The salvage operation of the magnetic tapes cost $ 9, 000 per decayed tape and $ 62 per intact tape It was 253, 5 times as long to migrate 1 GB of a non standardised data as 1 GB of standardised data (63, 36 hours <-> 0, 25 hours)

The digital migration project Conclusions Standardisation is the key! The goal of every migrating preservation institution must be to standardise data so that it can be migrated with standardised software The DNA had to develop 175 applications in order to migrate 165 non standardised IPs The development of migration tools for non standardised IPs represented 80 % of the expenses pertaining to software development

The digital migration project Conclusions Quality control is expensive but worth it! A retrospectively unsatisfactory random check was performed on the magnetic tapes in 1995 and resulted in postponing the media migration project Testing that data match your preservation standard on ingest is very important influences preservation cycle In an ideal world, there is thus 100% compliance between the preservation standard and all other preservation tools – from Ingest to Access

The digital migration project Conclusions Project management at service for everyday work! Rigorous management, loose method The project was meticulously planned, but there was room for changing methods subsequently to lessons learnt Knowledge management priority The exchange of and access to knowledge was facilitated by direct online access, weekly meetings and dynamic documents Project management as instigator for an intense and enthousiastic project process Teamwork, enthousiasm, importance of task, responsibility and social commitment were succesfully implemented by the project management

The digital migration project Conclusions Make sure your ancillary activities and preliminary examinations are adequate! Hardware The hardware used to process the migrations wasn’t powerful enough Data and development Examination of the data wasn’t thorough enough, and resulted in inadequate specification requirements for the software The project could have examined the market for open source or commercial software solutions Organisation The agreements and coordination within the organisation weren’t in place from the beginning

The digital migration project Conclusion The migration project was a succes But is automation a possibility today? Almost, but… Anyways, we saved the data!

THANK YOU! Köszönöm szépen! Migreringsprojektet Konklusioner on ati alu Ev po Re rt http: //www. sa. dk/content/dk/forskning_og_udvikling/udviklingsprojekter/format-_og_strukturkonverteringsprojektet