Funded by the: JISC Flexible Service Delivery programme.
Lead Institution: De Montfort University.
Partner Organisations: Southampton Solent University.
Key Words: Getting more from existing investments.
Background
Aims and Objectives
Traditionally, universities have chosen application systems from a variety of suppliers to support their administrative and learning processes. IT departments then had to develop a complex web of interfaces between systems. Typically, the student record system alone might have more that 10 sets of interfaces making it expensive and time-consuming to maintain and upgrade.
To meet the demands for more lean and agile systems the sector needs a technology shift towards Service Oriented Architecture (SOA) which basically implements a ‘backbone’ (an Enterprise Service Bus or ESB for short) to which any application makes a single connection. Information can then be passed between any two university applications or to an application residing on the Internet via the ESB. New systems can then be connected or disconnected easily from the backbone. Within the commercial world, SOA has driven huge productivity gains in application developments.
There has been much discussion about SOA during the past several years but a limited amount of take-up and practical realisation within the HE sector. The FSD programme aims to dramatically increase the use of both SOA and Enterprise Architecture concepts across the UK HE community.
A few universities have pioneered developments towards an SOA environment. There have been some successes but limited further take-up of SOA because the approach is perceived as time-consuming, difficult and because each solution has been uniquely geared to a single university.
This project was funded to demonstrate how SOA based solutions could be rapidly constructed within an existing university landscape. The project implemented different Enterprise Service buses (ESBs) as the backbone communications channel at two different universities. It connected 4 quite different applications at each university to the ESB and demonstrated how information could be combined from each application. The applications were deliberately chosen to have different technical challenges.
The project and all of its documents, designs, plans, notes etc are all available (the intellectual property of third party systems integrators has been transferred) so that it can be re-used by any UK university.
The project demonstrates the ease and also the difficulties of developing and implementing an SOA-based system within a traditional university environment.
The ‘pitch’ to the funders emphasised the fact that STEP-C, would be a fast-track demonstration of how best practice SOA principles could be realised across multiple universities and that the outcomes of the project would be fully available to any member of the sector wishing to develop their own specific SOA-based projects.
Senior managers at De Montfort University backed the project because it would enable the staff of the IT department to learn about SOA and new approaches to managing IT developments rapidly. The management at Solent University had already committed to a major University Transformation Programme which would require substantial changes to their core IT applications – so they viewed this project as a means to rapidly investigate SOA-based architectures before having to develop production-strength solutions during the next couple of years.
Context
The IT Director at DMU decided that it would be vitally important to select a second university which was significantly different to De Montfort in terms of its size, its geographic location and its technical architecture because it would be important to learn about the ‘generic’ issues of implementing SOA rather than discovering problems which were unique/specific to a single institution.
In conversations, Southampton Solent University expressed interest in taking part in the Pilot. They were an ideal fit in terms of being very different to De Montfort in many respects and they added the opportunity of being willing to select a different Enterprise Service Bus to that which De Montfort were interested in using.
The governance structures of the two Universities were different. However, the projects were managed by a local Project manager (Kenton Wheeler at Solent, Paul Hopkins at DMU) with an overall Programme Manager (Paul Hopkins, De Montfort University).
The project was run as a fast-track project – hence there were weekly project reports/summaries with email and telephone liaison between the two Universities and the Fulcrum development and implementation staff, globally.
Key drivers
The key driver within the DMU IT department for this project was that it represented a way of investigating new IT technologies without having to win significant funding from within the University. It also enabled the University to get access to technical specialists who were looking to develop a show-piece solution. Also, by creating a ‘Demonstrator’, or ‘Proof of Concept’ project, it was possible to develop an exciting prototype solution – without committing the University to dramatically change its architecture and the skill sets of its IT staff.
By reducing the element of risk (financial, personal, architectural etc) it was relatively easier for all parties within the University to regard the project as a ‘sand-pit’ to explore the art of the possible with regard to SOA approaches.
By coincidence, the University was developing a production system to measure whether students were actually engaged within the University, so this FSD programme had some genuine business analysis behind it – though the actual scenarios developed within the STEP-C project were necessarily simplified.
Designing the project approach
The project was designed to be a physical demonstration of how an SOA architectural approach could be used within the HE sector. It was not intended to create a full-strength production-ready solution. It aimed to demonstrate how SOA solutions could be implemented in any university by solving ‘generic’ rather than specific problems.
The scenario behind the project was to measure ‘Student Tracking and Engagement’. This scenario would require a report to be created which pulled information from a variety of different applications (eg the Student Record system, a finance system, a VLE, the library system, an email system etc).
The project demonstrated a complex solution in two quite different universities which run quite different application systems. It also used 2 different ESBs from different vendors.
The actual application systems selected within both Solent and DMU were chosen in order to demonstrate situations in which there were quite different types of connections required eg an application might:
- already be ‘web-services enabled’
- already have an API (application programming interface) which could be utilised
- have no ready-made connector available – hence requiring a direct link to the underlying database tables
To make the situation even more challenging, the project sought to test whether a single report could retrieve the same basic information from either university system without any programming changes ie that it should be ‘blind’ to the fact that the two universities were running completely different application systems!
Southampton Solent and De Montfort Universities collaborated to define the set of requirements and then actively managed the team provided by a systems integrator, Fulcrum Logic. The project insisted that all project documentation, ideas and intellectual property were to be transferred to the HE sector – so that all lessons could be made available to every UK university.
The project sought to demonstrate in real terms how many different types of applications could be connected to an Enterprise Service Bus to enable information to be transferred between multiple applications to create a common output. A test situation was constructed to observe whether a student was engaged with her university by accessing a series of applications to see when the student had last performed a transaction. In addition, the project wished to demonstrate how the above scenarios could be implemented in a second university which was operating completely different application systems. The situations are shown below:
The scenarios did not require complex data manipulation – rather they just required certain data fields to be retrieved from specific application systems. Therefore the bulk of the development activity was in creating appropriate ‘connectors’ between the application and the underlying Enterprise Service Bus (ESB) and in developing the necessary ‘process logic’ within the ESB to meet the needs of the solution (... ie ‘Go to the Finance system and retrieve the financial records for each of the students that have been selected’ then ‘ Go to the Library system to retrieve the latest date on which each of these students borrowed a library book’ etc).
The project integrated broadly similar data sets in each university although the data definitions etc were different. The reporting output was however identical in both cases.
Project Timetable
The key phases of the project were:
| August/September 2009 |
Get JISC approval for the project |
| September/October 2009 |
Initiation to tender for project contractor |
| November 2009 |
Detailed project planning |
| November 2009 - January 2010 |
Project execution |
| January 2010 |
Sign-offs at Universities and presentation to FSD workshop |
Project Methodology
The project methodology involved:
- Extracting a set of records (for 200-300 students) from each of the applications to be modelled.
- Within Fulcrum’s development labs, identical data bases were recreated to contain the extracted records. The architects/developers then created and tested their solutions based upon the ESBs which were to be used at each University.
- Within Solent and DMU IT test environments, Fulcrum implemented the relevant ESBs and test environment remotely – but utilising relevant University technical support staff.
- After (b) and (c) were completed, a Fulcrum developer visited each University to install the developed software and to link the solutions directly in to full-size test applications (eg student record, VLE, email etc).
- The Fulcrum on-site developer tested the application against copies of ‘live’ data, resolving any issues locally and getting ‘sign-off’ with the relevant analyst for each of the applications concerned.
- The Fulcrum on-site developer then created and demonstrated the integrated solution to the local IT department to get ‘sign-off’ that the full solution had been successfully demonstrated.
- The Fulcrum team created and presented the full solution to the FSD Workshop in late January, then packaged up all relevant notes and documents and transferred them to De Montfort University.
The project had to overcome difficulties in developing a solution while using off-shore contractors. The key issues involve the Data Protection Act which expressly forbids the export of personal data outside of the European Union without the express permission of the ‘data subject’ – unless the service being provided is of benefit to the data subject. Hence, whilst call centres and applications maintenance can be argued to be of benefit to the data subject, it is not possible to claim that a new application development will be of benefit to any individual student. The project team developed a very pragmatic solution which is partially described in the project methodology above, which kept the project within both the spirit and letter of the Data Protection Act and without impacting adversely on the project deadlines. Essentially, this involved ‘spoofing/anonymising’ personal data before exporting it to Fulcrum’s Global Development Centre and then ensuring that the subsequent installation/testing of the solution was performed completely on-campus and without direct support from any staff based off-shore.
Technologies used
De Montfort has Agresso Finance and Student Records systems, Blackboard as its VLE and uses Google Mail. Data from all of these systems was used in the project. The ESB used was Microsoft Biz Talk Server. De Montfort was ‘agnostic’ about the choice of ESB but felt it more of a challenge to demonstrate the proof of concept with one of the main proprietary solutions rather than an open source product.
Southampton Solent has Ex Libris as its Library system, CampusIT Student Records and Moodle VLE all of which provided data for the project. Solent considered using the Biz Talk ESB but opted instead for IBM Websphere because their University Transformation Programme (part-funded by an £8.3m HEFCE grant) was likely to be based upon IBM’s Websphere product hence it was more sensible for the STEP-C project to begin with this product.
Outcomes
Achievements
Proof of Concept
The project was completed within three calendar months and successfully demonstrated at a JISC Flexible Services Delivery workshop in January 2010 thus demonstrating that applying SOA does not necessarily have to be complex and time-consuming.
The introduction of SOA thinking into any university represents an enormous change in the ways in which application systems will be chosen, implemented, maintained and replaced. IT departments are as ‘conservative’ and reluctant to embark on major changes of concepts as any other part of the sector. Hence, many universities have not been able to find a way to get started with this approach. There are numerous myths that it is ‘expensive’, ‘complex’, ‘takes a long time’, ‘would not fit my particular institution’ etc. This project was funded with the express objective of exploding these myths and creating the lessons and tools which could be readily used by all other universities – hence the need to ensure that all Intellectual Property and design information was transferred from the systems integrator to the sector. By opening up the doors into the SOA environment, this project has begun to demonstrate how the sector can improve its efficiency in this key area.
This project enables any university to explore building a SOA-enabled IT environment by providing ‘cook-book’ advice on how to design and implement SOA-based architectures. It also enables the sector to begin to define those standards of connectors/interfaces which the sector will require any application vendor to meet when connecting their application to any ESB – so that it will be much easier for universities to mix, match and replace application systems.
The project has already borne fruit in terms of proof of concept as it has convinced HEFCE that the RMAS (Research Management and Administration System) shared services development project, led by Exeter University, should be encouraged to use this approach.
Increased Flexibility in terms of changing processes/systems
The impact of this project upon De Montfort and Southampton Solent Universities will be substantial but different and may take several years to come to fruition. De Montfort University have created a relatively stable IT architecture based upon Agresso’s Student Record, Finance and other systems. The key IT developments during 2010-2012 were more likely to come from adding in additional modules (eg CRM, Business Intelligence, Document management etc). Hence, the University was looking to use SOA-based solutions to further refine and improve its business process. Southampton Solent, however, were about to begin a thorough review of the core business processes across the University which could lead to significant changes in their supporting IT applications.
Discussions are underway to examine the further developments for use within the sector however De Montfort will be using the knowledge gleaned from this project to explore placing future application systems up in the ‘Cloud’ and connecting them directly to those applications which will remain running in-house. Initially, these already include Google Mail but may soon include the University Website, a new document management system and many more.
Southampton Solent is undertaking a major Transformation Programme to restructure many of its University processes. This will lead to major changes/replacements of their IT applications. Hence the experiences gained within this project will have a substantial influence on the new IT landscape which they will have to build.
A step change in how we think about systems
SOA obliges us to think about our systems in a very different way. Point to point interfaces tend to evolve over time without anybody ever really taking a step back and thinking about their design. SOA requires us to do this. It is a significant change for most IT staff and will require considerable investment in change management and staff development.
It also requires us to think about governance as there is no point in building the architecture if the governance to go with it isn’t there.
Capacity Building in the sector
All Intellectual Property developed within this project has been transferred from Fulcrum Logic to De Montfort University as part of the formal Agreement between the parties. Then, as an explicit condition of the Grant Letter, all relevant documents and intellectual property etc was transferred to the HE sector. This was intended so that further projects in the FSD programme could have access to any/all of the materials created within this project. The project team explicitly requested Fulcrum’s developers to explain WHY they made certain decisions (eg ‘Why BizTalk?’) or HOW they designed certain aspects of the project – hence all of the design documents are linked within the Appendix to this report.
Changed relationship with suppliers
The approach this project has taken to ownership of IPR represents step change in relationships with suppliers. Many interested suppliers felt unable to bid because of the requirement to hand over ownership of the IPR to the sector. The fact that the project has achieved this type of arrangement sets a precedent for the future.
The successful proof of concept also sets a precedent to suppliers who try to put obstacles in the way of flexibility (especially the flexibility to change systems easily). The project team describes this as being able to negotiate with some ‘dirt under your fingernails’ rather than from a theoretical standpoint.
Potential to improve student retention and achievement
The project was a technical proof of concept but it has nonetheless demonstrated the potential of the approach to have a direct impact on some of the key strategic priorities of the universities. The proof of concept shows that it is possible to undertake sophisticated tracking of students via their interactions with a range of different systems. This enables the identification of students at risk because it can be seen they aren’t logging on to the VLE, visiting the library etc and interventions can be put in place to address this before students begin failing and/or dropping out. This will be important in the future for Southampton Solent which has a strategic project on student engagement.
Potential to avoid data duplication
The proof of concept produced only reporting output but it has shown the potential to use the data in the middleware to populate other systems eg a new CRM system could be implemented and kept up to date without the need for duplicate data entry. The ESB is a kind of ‘information superhighway’ and you can choose whether it acts as a staging post or actually stores the data.
A Green approach
The approach is also a green one. The ESB has a single input and output so it is not bandwidth heavy in contrast to multiple point to point connections that consume a lot of bandwidth. How you design the architecture is of course important in this regard. For example if a query picks one student record at a time you may need to repeat it 200 times to get the set of records you require. This can be solved by effective business logic and query design. ‘Loose coupling’ of systems means you only query the relevant data every time.
How are you evaluating the FSD approach?
The critical measure of the project was that it delivered a public demonstration of the application systems at both Universities to an FSD workshop in late January 2010. To this extent, the project was successful in that it clearly ‘worked’! However, the project also required that ALL documentation, notes and other relevant materials would be collated and made available to all UK institutions (ie that any/all Intellectual Property would be surrendered and explained to help other institutions to build upon the STEP-C achievements. This has also been delivered and all materials have been transferred to JISC infoNet for storage (apart from the actual source code for the applications which is stored on the application servers at the two Universities!).
The success of the project must ultimately be measured by what further teams are able to build upon the successes of STEP-C. Several tentative suggestions have already been made to both JISC and to HEFCE – eg to place the ESB up in the ‘Cloud’, to expand the STEP-C scenarios so that they update the base applications, to replicate STEP-C in different universities, to extend the range of applications considered etc.
A key measure of the success of the STEP-C project was that HEFCE used the project as the basis for confirming that the RMAS (Research Management and Administration System) shared services development project, led by Exeter University, should be developed and built as a fully SOA-architected solution. RMAS is also part of the FSD programme and details of the project can be found on the FSD web pages.
Drawbacks
It is not necessarily a disadvantage or a drawback but any institution considering SOA should be cognisant of the implications for security. In order to understand this it is necessary to take a brief look at the characteristics of SOA and some basic security concepts:
SOA
Service Oriented Architecture (SOA) is way of enabling dissimilar subsystems to interact. Conceptually, this interaction between subsystems in an SOA is achieved by a messaging system, often referred to as an Enterprise Service Bus (ESB). The subsystems are equipped to send messages to, and receive messages from, the ESB messaging system. The subsystems do not communicate with one another directly; all subsystem to subsystem communication is carried over the ESB messaging system. SOA seeks to make the interaction between subsystems readily achievable. In particular, it seeks to make it relatively straightforward to adapt an existing setup to bring new subsystems aboard in response to changing needs. In short, SOA is an enabler of interaction and an enabler of change. Any functioning SOA will have at its core an implementation of an Enterprise Service Bus such as Microsoft's BizTalk, IBM's WebSphere or RedHat's JBoss. The technical success of any SOA project rests heavily on good configuration of a good ESB implementation.
Authentication
Computers act as agents for individuals. Authentication is the means whereby some degree of assurance is obtained that a particular computer is acting on behalf of a particular individual. Authentication requires presentation of a computer identity (user id) together with something only the person would know (egpassword), would have (eg smartcard) or was (eg fingerprint). The assurance that a particular computer is acting on behalf of a particular individual fades over time following an authentication event.
Authorisation
Authorisation is the granting of privilege to a user to access a particular information asset in a particular way. In effect, this is access control. Authorisation is distinct from authentication. However, where the information asset being accessed is anything other than ‘public’, authorisation should follow shortly after authentication.
Security is concerned with allowing access to those who deserve it and denying access to the rest. SOA makes the identification of ‘those who deserve’ problematic. Information is flowing from one subsystem (with one view of deserving identities) via the ESB (with another view of deserving identities) onto another subsystem (with yet another view of deserving identities). SOA systems are also intended to be rapidly reconfigurable. The security systems throughout the SOA implementation therefore need to be even more rapidly configurable if the appropriate allow/deny posture is to be sustained.
Limitations of proof of concept
The initial proof of concept shows that subsystems which should interact, can interact. It does not show that subsystems that should not interact, cannot interact. No assertions about the security of an HE SOA can be made based on this proof of concept. The next phase of the proof of concept should be to:
- establish security policies for the overall proof of concept, each discrete subsystems and the ESB
- operationalise the security policies in the proof of concept
- penetration test this hardened proof of concept system to ensure that information is accessible only to those who deserve access
The STEP-C project did not demonstrate solutions to the above issues. However, it did lead towards discussions of how a cloud-based Enterprise Service Bus together with the UK Access Federation could be used as part of a production-strength solution. The investigation and development of such a solution was deemed to be outside the scope of the STEP-C project.
Looking Ahead
The STEP-C project, as originally conceived, is now completed. There are however several areas which could be examined for further development at a Proof of Concept/Demonstrator level. These include:
- Replace one of the ESB buses with a different one to show that the application can continue without ANY changes to process logic
- Add in some additional applications to either Solent or De Montfort
- Replicate what has been done at a third University
- Remove one of the applications from the Process Logic and show that the basic system can continue to get information from the other application systems without change
- Change the scenario to demonstrate how UPDATES could be achieved within this scenario
Of the above, only the last two concepts would materially increase the value of the learning which can come from the STEP-C project.
A more relevant set of opportunities will arise when the STEP-C work is used as a foundation for moving forward. Key suggestions for further work may include:
- Create a small scenario which may involve just a small number of application systems. Develop a simple problem which can be developed in to a much richer situation involving creating/amending/reading and deleting records. Then consider implementing this solution in several different institutions.
- Building upon the STEP-C connector demonstration, begin to develop a set of HE-Sector standards for connecting applications to a service bus. By taking the same type of application (e.g. a student record, finance, library, VLE etc) from several vendors of the same application, examine the difficulty/ease of creating look-up tables so that they can each be converted to utilise a standard set of HE-specific names.
Summary and reflection
The project achieved all of its goals, resulting in a presentation at an FSD programme workshop in late January 2010. This report is intended to summarise the project achievements and point towards where all of the project materials can be found.
There were many lessons learned by the participants in this project which will enable them to work together on further assignments. These included utilising fast-track project methods and learning how to engage with an off-shore development company. However, lessons which may be of direct relevance to other HE institutions include:
Data
Data accuracy, integrity and consistency becomes even more important within an SOA environment than ever. When applications (and their associated data stores) are connected via an ESB then there will be an increasing volume of data being shared across the enterprise. The concepts of ‘which version of data is the Master?’ will be of increasing importance. In itself, SOA does little to improve data accuracy – but any production quality systems will need to ensure that processes are created which clean up the data which will be used.
This work also highlights the need for more defined standards and naming conventions for common data fields. This will of course only be achieved if the sector has a will to do so. We achieved this for UCAS data but universities are yet to see a compelling case for this in terms of facilitating students moving between institutions.
Vendor Claims
Be wary of claims from certain application vendors that their products are already ‘Web Services enabled’. It is likely that they designed the Web Service connectors to satisfy specific, rather than generic, problems. For example, at DMU it was noted that the Agresso Student Record application was ‘WS-enabled’ but that the connector would only return a single student record. It had not been designed to return a list (or ‘all’) student records. Similarly, it is conceivable that a pre-existing connector may not return certain data fields that the new SOA application requires.
Change
The reaction of existing Business Analysts to the introduction of SOA is important. The current generation of Analysts are extremely familiar with toolsets that enable them to develop applications successfully. Persuading them to switch their analysis and development methods to one based upon SOA principles will be challenging. This appears to be both an educational and also a cultural issue. It is certainly an area which will require substantial investigation – especially because there may well be (according to UCISA statistics) 2,500 staff engaged within the Applications sections of UK Universities.
Security
Security policies require clear boundaries and SOA has fuzzy boundaries. Defining the scope of any security policy will require considerable care. In particular, the concept of a ‘data owner’ may prove problematic as data becomes widely shared between subsystems.
Digital certificates should be used between any two communicating devices. They provide mutual authentication, and through encryption, maintain the integrity and the confidentiality of the message being communicated. Encryption however makes the monitoring of information leakage difficult.
The computers running the SOA services should be hardened to have only the software necessary to run those services. All nonessential ports should be closed. The services should run with minimum privilege and run within the equivalent of a chroot jail.
All firewalls should operate a default deny posture. Information should be permitted through firewalls, only where the SOA explicitly needs it.
Policies defining what a subsystem is permitted to do should be defined and demonstrably operational before a subsystem is permitted to join the SOA implementation. These policies should pay particular attention both to what happens to information after the subsystem receives it, and to what happens to information, before the subsystem sends it.
The overall system configuration should be such that it is easier for individuals to do the right things, than the wrong things.
Appendix
