Project Name: Meta-morphosis: building a framework for electronic ingest

Lead Institution: Institute of Education

Project Lead: Sarah Aitchison 

 

The overarching aim of the project was to create a set of best-practice procedures and documentation for accepting, processing and storing digital collections in the Institute of Education archive. This work built upon the findings of several pilot projects around personal, organisational and IOE archives. 

 

Background

 

Aims and objectives

 

The overarching aim of the project was to create a set of best-practice procedures and documentation for accepting, processing and storing digital collections in the Institute of Education archive. This work built upon the findings of several pilot projects around personal, organisational and IOE archives.

 

The main objectives were:

 

• Reading of JISC and other background materials and the production of an annotated bibliography on the subject of digital archives.
• Meetings and interviews with members of IT, Library and Directorate staff at the IOE to determine their requirements and potential input to the storage and processing of digital archives.
• Creation of best practice guidance regarding ingest of IOE archives using our E-prints-based digital storage system (named IDEA, or Institute Digital Education Archive) and management of semi-current records 
• Creation of best practice guidance regarding acquisition of deposited electronic archives including review of current documentation 
• Design and delivery of one-to-one training for Directorate based on best practice guidelines
• Development of  IDEA front end and functionality to support ingest of digital archives, including the potential for self-ingest
• Dissemination of work and findings through blogs, training and presentations 

 

Context

 

Newsam Library and Archive Services is regarded as world-leading and world renowned. The largest education research library in Europe, it holds extensive collections of current and historical materials on education and related areas of social science that are not held elsewhere and it plays a vital international role in preserving and curating these publications, archives and documentation.

 

The Newsam Library and Archives is divided into four sections: User Services, Technical Services, Collection Development Services and Archives. The Transformations project was run by the Archives team, which comprises 3 people.

 

The IOE Archive contains over 140 donated collections relating to education and the Institute’s own archive which dates from foundation in 1902 to the current day.  The most pressing issue currently affecting the work of the Archive Section is that of digital records - both donated and institutional.

 

In the past year, several archive depositors have approached us with hybrid collections of paper and electronic documents. These have fallen into several categories:

 

Archives of organisations

 

We initiated a pilot project with the General Teaching Council for England (GTCE) and worked extensively with them to accept their large collection of mainly electronic documents. This has involved revision of our standard agreements, the purchase of storage space using e-prints software (in conjunction with the University of London Computing Centre or ULCC), and the development of metadata spreadsheets and methods of transferring metadata from our archive CALM system (this is our archival management system, which holds our catalogue) to E-prints.

 

Archives and semi-current records of the IOE

 

Through the JISC Grant Funding ‘Digital Infrastructure Portfolio’, the Archives Section also worked with the IOE’s central administrative department (Directorate) to survey future requirements for digital preservation of archive material created by this department. This project has resulted in the creation of a departmental retention schedule, a survey of preservation metadata and a business plan for digital preservation. We have also undertaken a short project looking at the organisation of the Library and Archives shared drive and the potential use of this for records management.

 

Archives of individuals

 

We have also taken in our first personal digital archive. This was done in a fairly ad-hoc manner and resulted in 25 GB of completely unsorted electronic material, including programme files, favourites, duplicates, multiple versions and many other extraneous materials. 

 

During the process of undertaking these projects, the key findings centred on the requirement for systematic ingest procedures for digital records, both donated and institutional. In the future it is likely to be the case that whole archive collections will be created and retained in digital form, so it is vital that we retain the ability to continue collecting them for future researchers. 

 

The business case

 

• Development of digital archive preservation and delivery will ensure the IOE is able to build on its strategic priorities such as extending global reach through our on-line resources; and advancing learning and research through the development of a unique research resource and the accessibility of learning materials. The project also fits into the Library Strategy by preserving unique material and material of importance to future IOE work in a sustainable way; and by helping to find the right balance between print and electronic delivery for current use and future preservation of both library and archival material. 
• The project will have a positive impact on the curation of donated archives, allowing us to deal efficiently with hybrid collections and to continue expanding our archives and making them available to researchers irrespective of format.  
• In addition to a heightened profile amongst the academic community and potential student base, the completion of the proposed work will also inform new strategies in the management of internally created electronic records ensuring the IOE is capable of conforming to legislation, standards in information security and best practice. The project will help ensure the organisation is better placed to manage its core electronic records freeing up storage space on servers, increasing efficiency in the workplace and ultimately making financial savings. This work is currently of particular importance as a recent IT service review has stated these areas need to be addressed. 
• All aspects of the project will help to streamline the records life-cycle at the Institute and will feed into the developing records management programme, which we intend to roll out to the rest of the organisation over time. Creation of standard procedures for semi-current and archive storage of electronic records will lead to institutional efficiencies.  

 

Key drivers

 

• Ability to continue to build the unique research collections held in the IOE Archive, irrespective of format, over the long-term
• Ability to make unique archive collections available to researchers from all over the world, irrespective of format 
• Requirement for long-term storage and preservation of digital archives
• Streamlined management of IOE institutional records, leading to cost savings in storage and staff time
• Ensure legal compliance for IOE records, including Data Protection, Freedom of Information and Information Security 

 

JISC resources/technology used

 

• Digital Repositories Infokit – to develop repository policy and procedures; configuration and development of a public interface. This resource mainly relates to repositories housing research outputs and teaching materials, so was not hugely relevant for the purposes of this project. We already had a respoitory based on E-prints, so much of the material around chossing a system and interoperability was irrelevant. The areas on policy and buy-in may be useful as we progress further with the IOE's own digital archive.
• Records and Information impact calculator – to measure the long-term impact and value of the project on the efficient management of archival records by professional staff and the Change Management infokit – to provide guidance on how to best manage changes in workflows of IOE professional staff. The Impact Calculator and Change Management infokit both proved useful for recording the long-term impact of the project. Although we would have recorded this anyway for use in future development and fundraising work, it was useful to have a systematic method of doing this.
• Future proofing toolkit (enabling practical preservation of born-digital records): to create preservation copies of core business documents that require permanent preservation
• JISC/SPRUCE Mashups: one of the archive staff attended the London event in September 2012. Her blog post about the event shows that it was extremely useful in providing ideas for technical solutions to archival challenges. 

 

The reports and publications of various JISC-funded projects were reviewed and form part of the Literature Review attached below as an appendix:

 

• JISC funded Version - LSE 
• JISC – Developing an Institutional repository for Leeds Met Uni 
• JISC Cairo Project
• JISC-funded Carcanet Email Preservation Project May 2012, University of Manchester Library 

 

Outcomes

 

Achievements

 

Documentation 

 

At the close of the project, we have managed to combine findings from the various pilot projects and best practice gleaned from extensive reading and have created the following documentation relating to the intake of digital archives:

 

• Revised donation and deposit agreements for archive material. These are the standard agreements signed by anyone that gives us their archive, which now include clauses relating to the ownership and preservation of, and access to, digital materials. (See appendix). 
• An additional guideline document for depositors with regard to their digital archives, which will include: initial questions and considerations around issues such as size, format, hardware and software; actions to undertake before the material can pass to the archive; and an outline of what the Institute of Education will be able to take. 
• A step-by-step procedural document for the intake and processing of digital materials by archive staff – in other words, the workflow for taking them into the Archive. 
• A metadata template to give to depositors incorporating all the fields the archivists need to add digital materials to CALM and e-prints. 
• Naming convention and version control guidance for depositors. 

 

E-prints, digital storage and organisational archives

 

In addition, we have been extremely successful in managing to adapt E-prints (which was designed as a repository for published work by academics) to house our first deposited digital archive. There are now around 7,000 electronic documents comprising the archive of the General Teaching Council for England in our storage system, preserved for researchers. Through some complicated and technically difficult development work, colleagues in Archives and the Technical Services Section of the Library devised a method for the import of metadata from the GTCE's spread sheet to our CALM database and also into IDEA (the digital storage system). They also found a way of uploading the actual digital documents into IDEA and linking them to both sets of metadata. This is a huge and possibly unique achievement and will be the basis of future publications once it has been tested further.

 

Personal digital archives and appraisal

 

Another colleague worked hard to develop a set of procedures for personal digital archives. Even in paper form, personal archives tend to be less structured and messier than those of organisations, and our first electronic personal archive was no different. There were several challenges, such as:

 

• The collection was completely unsorted comprising the contents of a number of PCs
• The size of the collection made it impossible for us to survey it manually
• There was very little descriptive metadata – no meaningful titles; dates have been recorded as the date the collection was copied over to the archives; and there are no details of authors so it is difficult to know what has been created by the depositor and what is third-party content
• There was a huge amount of duplication including exact duplicates, and numerous versions of the same ‘record’ often with very little difference in the content

 

Much of the available literature around digital archives focuses on initial ingest, preservation or storage - we found comparatively little practical advice about digital archive appraisal (sorting which materials need to be kept permanently in an archive). Comparatively easy to do with paper records, appraisal of a large-scale unsorted digital archive proved much more difficult. It proved impractical to open every document to get an idea of content (many had no meaningful file names) so we had to devise new methods of sorting and appraising such collections. Valuable help and advice was provided at a SPRUCE mash-up, but we found it impossible to replicate some of the more technical aspects of this without dedicated IT support. Instead we turned to open-source software and discovered Karen's Printer Directory (http://www.karenware.com/powertools/ptdirprn.asp). The software had already been used by a number of archive services and recommended as a useful tool by the AIMS and Paradigm project teams (see Literature Review in the appendix). The software can be used to create descriptive metadata which can be imported into Excel enabling further manipulation of the data to aid appraisal. Two spreadsheets were created, one to be retained permanently as a record of the whole collection and the second for use as a working document for the appraisal. The software was used to locate whole sections of the collection that could be marked for destruction including: music files; internet links, personal photographs, family documents, set up files for various systems, software and external media including cameras. It also allowed us to find absolute duplicates and versions of the same document. We have retained just 6.5% of the collection (so far) and 2% should probably be retained but is currently in an obsolete format. There is still some way to go, but the work completed so far has allowed us to create a workflow for personal digital archives and has informed us of the large number of issues to be addressed with depositors before we can accept digital collections. This in turn has fed into our documentation.

 

Records management training

 

Another success has been the development of our records management training. Conceived in the original bid as small scale one-to-one training for Directorate staff, this training has now reached 121 members of staff, both administrative and academic, at the IOE. At the end of 2012 we were approached by our Staff Development department and asked to run a series of sessions around records management and associated legislation  - they had assessed need based on returns from the IOE's appraisal system where staff were asked to list their training requirements. We devised a one-hour programme based around Information Management, Data Protection and Freedom of Information and delivered it in five sessions in May and June. A revised version of the session will feature on the permanent training offer for IOE staff from September of this year.

 

Linked into our records management activities, we were concerned to explore methods of managing semi-current and archival records using the IOE's shared drive. In the absence of a document management system, our aim is to use existing technologies and methods of managing and sharing files. The shared drive is used by all staff at the IOE to store documents and share them with colleagues. We undertook a pilot to reorganise the Library and Archives section of the shared drive, removing constrictive permissions and opening it up to allow for greater joint working amongst staff. We also appraised the existing folders and files and worked with staff to undertake a large-scale weeding programme of out-of-date material. Next we set up 'records management' folders for semi-current material that needed to be kept for anywhere between 1 and 7 years, so that this was not cluttering up current activities. This project is almost completed and will inform our work with other departments in the IOE.

 

Outreach

 

Throughout the Transformations project, staff members posted regularly on the Library and Archives blog. This series of posts was intended to be both informative (giving details of the project and progress) but also reflective. Thus many of them asked questions or reflected upon the success or otherwise of particular strategies. As each member of the team was working on a different area, this method of reporting worked well.

 

Several articles are in progress based on work undertaken during the project, particularly the ingest of the GTCE archive and the development of IDEA and the appraisal of personal digital archives.

 

We have undertaken a large number of presentations about the work of the project. As well as the 121 staff reached during records management training, team members gave papers at the IOE's Learning and Teaching Conference; AIM25 meetings; the Friends of the Newsam Library and Archives; and many others. 

 

Benefits

 

• We now have a set of detailed workflows around the ingest of digital archives which will allow a seamless process from the moment we are approached by a potential depositor right up to the point their archive goes into our digital storage system. This will allow for consistency of approach and will lead to efficiencies in the work of the Archive section.
• The ability to continue to collect important education archives, no matter what their format, is of huge benefit to the IOE and researchers around the world. It means that we will be able to continue to develop our outstanding research collections and preserve them for future researchers.
• The development of procedures and training in records management and the management and storage of semi-current digital materials will create efficiencies in the work of IOE departments and lead to cost and time savings. More developed records management procedures will also reduce risk in relation to legal accountability.
• The technical development of E-prints software for archival storage provides a relatively cheap and practical solution to digital archive storage and preservation which could have huge future benefit for all archive services and the wider HE community. 

 

Drawbacks

 

• As the project ran over an entire year and was worked on by all of the archive staff, we had to timetable it quite tightly to achieve our objectives. At times, especially when faced with technical challenges out of our control (for example when work had to be done by the external host) or beyond our extertise, this proved difficult.
• E-prints documentation was not adequate for our needs and the wiki proved difficult to use even for experienced technicians. Our attempts to customise the most basic things, for example, the appearance of E-prints, proved frustratingly difficult. The link between CALM and IDEA was also complicated and only in-house technical expertise provided the solutions. Due to the innovative nature of the work, there was no blueprint or user base for us to turn to!
• The E-prints preservation plug-in is not yet fully developed and is an absolute requirement if e-prints is to become a long-term solution for digital archive preservation and storage.
• We struggled at times to gain management buy-in for the records management aspect of this work . Really focussed development of this area of the work will require staffing and/or basic project funding.

 

Key lessons

 

• This project was only achieved through great team working. Due to the varied nature of the objectives, we assigned tasks to specific individuals with specialist skills and arranged regular reporting meetings. This gave the whole team an overview and a chance to input into all areas, whilst developing the specialist skills of each team member. We will use this method of project management again.
• Use what you have and develop it for need. Much of what we were attempting to achieve had to be done with no extra staffing and a small budget. For this reason, we focussed our efforts on existing resources which were being used for another purpose (shared drive, E-prints) and adapted them for our needs. Whilst some of the functionality may not be ideal, this kind of working is of growing importance in the current economic climate and actually makes the work of wider relevance.
• A holistic approach was required: if this seems a wide-ranging project, that's because digital archive curation requires changes to every aspect of our work. from initial work with donors right up to the way we preserve materials for the future.
• Pilots are vital. Because of the wide-ranging nature of this work and the requirement for it to be undertaken within existing staffing and budgets, the only way to be realistic about what can be achieved, and to provide accurate costings and timings, is to run short pilots in several areas. 
• Records management needs to be achieved through personal contact, at least in the first instance. You can write as many policies as you like, but until you talk to the people creating the records and explain the benefits and its relevance, very little will be achieved.
• There is a huge amount of literature available on the subject of digital archives and their collections and preservation. Having an overview of at least some of this is vital to prevent repetition of effort. However, much of it is theoretical and unsuited to small archives with limited resources. There need to be more practical case studies.
• One size doesn't fit all. Although this project has allowed us to develop a basic work-flow for digital  archives, we will need to keep developing different versions of this for different types of archives. For example, the processes for sorted and unsorted personal archives will be very different, as will those for the IOE's own archives.
• We are going to have to work extremely closely with depositors regarding their digital archives due to the limitations of the medium. It is likely that we will ask them to have more involvement in areas such as organisation, appraisal, file  naming, version control, virus and format checking and provision of descriptive metadata (especially in the area of confidentiality and access).
• Digital archives are NOT the same as paper! Until recently, it has been argued that the archival principles are exactly the same for all formats. Our work is tending to prove the opposite, especially with regard to appraisal and cataloguing. For some examples of this, see these blog posts:

          http://newsamnews.ioe.ac.uk/?p=4392#more-4392

          http://newsamnews.ioe.ac.uk/?p=3740

          http://newsamnews.ioe.ac.uk/?p=3600    

          http://newsamnews.ioe.ac.uk/?p=3519#more-3519 

 

Looking ahead

 

There are several areas of development that have come out of this project and that we would like to take forward in the next few years:

 

• There is scope for a project to look at the potential of E-prints for archival storage and much more work needs to be done on the development of the E-prints preservation plug-in.
• Several areas of this work would benefit hugely from dedicated technical support and we will be developing some projects to look at specific technical aspects, including redesign of the IDEA front end.
• We plan to set up a small pilot working with one of our existing archive depositors to test our recently developed workflow for the ingest of personal archives.
• With regards to records management at the IOE and management of semi-current and archive records, we will be working towards self-ingest for IOE staff so that they can upload documents for permanent retention directly into IDEA.
• We will be working to set up access to digital archives in our reading room, with the provision of a dedicated terminal for researchers.  

 

Sustainability

 

The work we have undertaken during the Transformations project has been collated to form a complete set of documentation which is now embedded into our day-to-day working practices. Whilst we will continue to develop and improve these processes, they represent a permanent change in the way we handle digital archives.

 

We will continue to embed records management into practice at the IOE, through training and personal contact as well as a programme of working with individual departments to draw up records retention schedules. As we continue to develop IDEA, IOE departments will be able to deposit digital archives by themselves, leading to the creation of a digital institutional archive. The embedding of records management and self-ingest for digital archives will lead to substantial changes in working practices at the IOE, most notably in the way digital materials are stored and preserved. Records management training will now form part of the core staff training programme for the IOE.

 

Appendix

 

Revised deposit and donation agreements for archives:

2013 Agreement for the gift of records

2013 Agreement for the deposit of records

Digital archives workflow: WorkflowsDiagram

Version control/naming conventions document: VersionControlNamingConventions

Literature Review: MetaMorphosisLitReviewJISC