Jisc case studies wiki Case studies / Flexible Services for the Support of Research (FleSSR)
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • You already know Dokkio is an AI-powered assistant to organize & manage your digital files & messages. Very soon, Dokkio will support Outlook as well as One Drive. Check it out today!

View
 

Flexible Services for the Support of Research (FleSSR)

Funded by the: JISC Flexible Service Delivery programme.

Lead Institution: University of Oxford.

Partner Organisations: University of ReadingThe Science & Technology Facilities Council (STFC)EduservEoverI Ltd.Canonical and Eucalyptus Systems Inc.

Key Words: Cloud.

 

Background

 

Aims and Objectives

 

There were two parts to this project:

 

  1. A technical aspect including the development of a hybrid cloud infrastructure and the federation of private cloud within a Higher Education Institute (HEI) and public cloud in Eduserv. This required several services to be built to support these, primarily around accounting and monitoring. We built on top of the open standards compliant accounting system developed by the UK National Grid Service (NGS) and operated by the STFC, with a schema developed specifically for the recording of accounting information for IaaS (Information as a Service) cloud systems. Through the Zeel/I software we have developed a number of NAGIOS plug-ins that can be used to test the availability, responsiveness and ultimately the performance of different clouds that are available to a user community.
  2. The second part was the construction of two different use cases.
    1. The first, a multi platform software build and test environment, which allows large national and international research projects to develop their complex IT systems software in a way that they are able to remove some of the problems around platform dependence. This also developed the project knowledge around the management and operation of large numbers of virtual instances in an IaaS cloud.
    2. The other use case looks at the management of storage within a cloud environment, principally the management of smallish amounts of storage and the different software interfaces that are available to access it. This also gave the project insights into different methods of migration within hybrid cloud configurations.

 

Context

 

The groups participating in this project are a mix of central IT services, research computing groups and national research organisations. Lessons learned from this project will feed into the best practices established through the pilot services run by the UK NGS.

 

Key drivers

 

Research computing must be able to support an increasingly diverse range of applications and ICT utilisation models. The only way that this is practical in the long term is through the increased utilisation of virtual resources, either as gateways onto other more standard resources such as computational clusters or storage. As leading research-focused universities this is of prime importance in the near, medium and long term to sustain research excellence, and participation in international projects building on the EU Distributed Computing Initiatives.

 

Establishing and maintaining senior management buy-in

 

The public cloud infrastructure deployed is being expanded through the University Modernisation Fund (UMF) programme, the Oxford cloud is being moved for operation through the Oxford Supercomputing Centre (OSC), Oxford central research computing service and Reading who are investigating continued support and provision of a cloud solution.

 

Technologies used

 

Eucalyptus IaaS cloud within the Ubuntu Enterprise Cloud distribution and EoverIs Zeel cloud management software. NAGIOS systems monitoring and NGS Accounting server.

 

Outcomes

 

Achievements

 

The project has built and demonstrated utilisation of a hybrid cloud solution for two different research specific use cases. These have not only demonstrated useful applications within the cloud that are not domain-specific, and general enough to be reused by a number of different communities, but have also shown methods by which the management of large numbers of cloud-based instances and disparate storage blocks might be managed.

 

We have devised a candidate extension to the recommended Open Grid Forum (OGF) usage record standard for accounting IaaS clouds and a plugin for the NAGIOS monitoring system that is used by the UK NGS which is being packaged for inclusion into Ubuntu Enterprise Cloud (UEC).

 

Benefits

 

Tangible

 

  1. Advanced installation documentation for UEC as best practice for private cloud installations including network configurations in an HEI environment.
  2. Monitoring plugin for NAGIOS for UEC cloud.
  3. Accounting plugin to publish Usage Record based accounting information.
  4. Candidate specification for cloud extensions to the OGF Usage Record standard.
  5. A set of candidate business models by which an institution can operate and support a sustainable private cloud to support research.
  6. Installation software for software development environment and distributed storage services to support research. 

 

Intangible

 

This project expected to achieve the following intangible benefits:

 

  1. Experience of working with one of the leading private cloud systems available at the moment.
  2. Kick-start a movement within UK HEIs to work towards a federated cloud infrastructure between private HEI located services and public based services such as Eduserv or commercial suppliers.
  3. Build capability in user communities that are willing to utilise services that are externally provided and possibly could be made available across different HEI providers and consumers.

 

The following intangible benefits were actually achieved:

 

  1. Knowledge of different types of cloud software available during the early stages of the project. Real experience of installation of the software. This could benefit systems administrators within HEIs when they are required to provide cloud-based services to different research groups within their institutions. HEIs benefit through the provision of flexible storage services; this can change the mindset of researchers towards recognising that centralised services rather than those provided by themselves may easily serve their needs.
  2. The project has provided a spectrum of business cases (completed by Eduserv) so that HEIs do not need perform this work themselves.

 

Drawbacks

 

We have learned that large production clouds cannot currently be provided through open source/free software. 

 

Key Lessons

 

Though we have managed to develop our use cases, built software to support them and added services to cloud instances to start to interact with national e-infrastructure services, there currently are no open source/free software solutions available on which true production private cloud services can be based.

 

With the future development of Openstack and its support through multiple large and small providers that are engaged with the development of this platform it will undoubtedly be developed into a great product. It is, however, not currently in a state to provide this type of service. 

 

Looking Ahead

 

The installations of private clouds in both HEIs will be continued to support the user communities that were participating with the use cases and the software systems developed will be supported through configured instances which can be deployed in further cloud systems. The project participants are also leading alongside the UK NGS an effort for a UK-wide federated cloud project.

 

The experiences that this project has gained with accounting and monitoring will be used to lead global efforts for standardisation in these areas including the addition of these extended standards into a IaaS cloud profile that will be proposed through the project Principal Investigator (PI) and Director of the European Grid Infrastructure (EGI) project as a mechanism by which the cloud paradigm will be used to allow for the easy expansion of the European e-infrastructure activities and enable the easy support of multiple divergent user communities.

 

Sustainability

 

We have ongoing activity within our department through other Research Councils UK (RCUK) and locally funded projects to provide cloud services. This includes participation in UMF-funded Dataflow project which will make use of some of the knowledge that has been gained through the data storage use case combined with ongoing activities in other JISC-funded projects. The UMF cloud installation at Eduserv has been informed by our experiences in the provision of services using Eucalyptus-based clouds where they will be installing a production VM-Ware based system and an Openstack development cloud.

 

Summary and reflection

 

The project has, in a short length of time, developed services which will make private cloud solutions easier to fit into production e-infrastructure environments particularly around accounting and monitoring of a cloud system.

 

We have built two different use cases, each of which has allowed the project team to develop, in parallel, knowledge of instance and storage block management within a cloud system for different user communities as well as instance design and deployment expertise.

 

Unfortunately, we have not been able to develop and deploy an Open Cloud Computing Interface (OCCI) instance onto Eucalyptus though in collaboration with Canonical we have given input into their technical roadmap.

 

Overall the project has been a success though it would have been useful to take this knowledge directly into UMF with all partners providing exemplar infrastructure to the community beyond what was a fairly short project.

 

Appendix

 

Project Website

 

http://flessr.blogspot.com

 

Project Outputs