Funded by the: Jisc e-Learning programme.

Lead Institution: University of Oxford.

Learner Provider Type: Higher Education

Project Duration: January 2012 - March 2013

Key Words: Course Data

Case study tags: course data, process improvement, course information, university of oxford

Note: This is an abridged version of this project's final report.  The full version is available here.  

 

Project Summary

The OXCAP project implemented the systems and processes required for the publication of XCRI-CAP 1.2 http://www.xcri.org/wiki/index.php/XCRI_1.2  feeds for Graduate Training in the University of Oxford.

 

Providers of such training within the University supply details about courses in electronic format (XML via a Share point data entry form, an XCRI-CAP feed or an .XSLX (Excel/spread sheet) file) to Oxford’s Open Data Service (OODS) which is managed by Oxford’s IT Services department. As part of the project, many training providers have made modifications to their systems to produce data in the appropriate format and OOSDS have been transitioned into a fully supported service.

 

Information about the courses is held within OODS and is made available as a series of data feeds (XCRI-CAP, RDF-XML, Turtle, etc.); there is an authenticated feed for internal use and an Open Government licensed (http://www.nationalarchives.gov.uk/doc/open-government-licence/) public feed (containing slightly fewer courses) which can be consumed by anybody that has a need.

 

The project has also developed supporting software artefacts, these include a VLE (Sakai) tool (SES), a Java Script library for displaying courses on websites, a SharePoint data entry form with an associated transformation library to produce XCRI and a website: http://tinyurl/oxcap/ .

 

Process and policy guidance is being developed in conjunction with the training providers to ensure all the outputs from the project are sustainable and will be submitted to the University’s Education Committee for approval. The Steering Group that has been guiding and monitoring the progress of the project will remain in place as a consultative body after the formal end of the funded project.

 

The service, which has been in pilot since January 2013, has been very well received by students, supervisors and the training providers. It currently holds 1400 courses of which well over 100 are classified as ‘upcoming’. Collecting all graduate training information in one place in a common format is seen as a huge success and should make a very real difference to the academic life of current and prospective students.

 

Advantages for Oxford:

• The public display of courses will show the academic world how much excellent graduate training is on offer, and will help Oxford to attract the best students
• The VLE tool will allow students to locate and take full advantage of the training on offer leading to ‘better’ trained researchers and academics.

 

What did you learn?

In some ways the project has been a victim of its own success. Before the project students were having trouble finding training, now they are having trouble deciding which course they should be attending. On many occasions, host departments will not accept students from other departments on their courses and, as it may not immediately be obvious that this is the case, students get frustrated and complain about having their time wasted. To address this, based on early feedback from the pilot we are planning to collect data for an additional XCRI description element of type ‘targetAudience’. This will be plain text that says who is able to attend the course; it will be displayed before the course description.

 

Concern has been expressed that if one or more training providers fail to supply data about their courses then a false picture of training at Oxford will emerge. Training providers can see the benefits of supplying their data,. More work is needed to converge on a common timetable for providing data, that meets the needs of students and supervisors. Sometimes skills training for taught postgraduates is difficult to separate from the formal courses that a student is registered on, which may also make it look like there are gaps in training provision.

 

Another related concern regards the issue of data quality. If some training providers provide elaborate descriptions and concise meaningful titles and other use short descriptions and rambling or very generic titles, then students may become frustrated with the tool and / or the information it provides training providers who supply less than perfect data. In order to counteract this we have undertaken a data evaluation exercise (see appendix) and will use this to provide further guidance to help training providers to improve the quality of their data.

 

Related to this, is the issue that some training providers don’t attach dates to their courses. , they may run a course when there is sufficient demand or, ‘sometime in Michaelmas term’ or simply do not want to specify dates in case non-registered students turn up. Because of this we have had to develop an algorithm for deciding when a course is ‘upcoming’, see http://wp.me/p1K8WQ-DL for details.

 

Once the outputs of the project have ‘bedded down’ we will conduct a survey of students to gauge their reaction to the SES tool, its usefulness and the data contained therein.

 

Another one of the many useful aspects of the project is that we have done a lot of preparatory work in the area of course data interoperability (processes, infrastructure, etc.). As mentioned earlier, Oxford is in the process of moving to a new student information system and once this system is in place, it is likely that Oxford will want to produce a course data feed of its formal courses. This project has supplied much of the required infrastructure and will be able to inform any such initiative in many different areas.

 

As part of this project we have had to model XCRI-CAP as an RDF vocabulary which is something that others have not done and involved a great deal of practical research. The nature of this modelling will be highly beneficial to others modelling course data, whether as RDF or not. We are gradually documenting this for wider dissemination through a variety of routes.

 

As part of the retrieval of course data from the University's SharePoint installation, OODS development work created an open source python SharePoint library which helps in extracting InfoPath XML from SharePoint. This is freely and openly developed at https://github.com/ox-it/python-sharepoint and will be extremely useful for other (internal) OODS projects since it enables greater use of Share Point for data entry.

 

One of the main problems with software development for this project has been the transformation of imperfect or flawed data sources to a common data model. This has entailed a large number of dataset-specific fixes and development. There have been a number of instances where requirements have changed slightly and these could have been improved with an even more detailed specification of work shortly after the start of the project.

 

Immediate Impact

The project has enabled the University to bring together and display in one place for the first time all of its extensive graduate training provision, which was previously displayed separately on several different websites or else was not immediately visible.

 

The training information can be viewed by date or department, and can be searched by skills training category or key words, and sorted and filtered. This means that graduate students and their supervisors, as well as training providers and administrators, can see information about all training opportunities together, and they can make better informed choices about the most appropriate training, in particular across disciplinary boundaries. It helps support the opening up of training opportunities across departments. This meets a key requirement of students, identified in recent surveys undertaken by the Student Union (OUSU), to make all training opportunities more accessible. The project also provides the University with the opportunity to learn more about handling course data generally and the use of XCRI-CAP in particular.

 

The information on the University’s graduate training provision will be presented externally for prospective students and others. This will help enable prospective students to make informed choices about where to study. The initiative addresses the requirement of the Research Councils which fund doctoral training, and the Doctoral Training Centres, to support training by making information about opportunities readily accessible to students.  The project also provides an example of the use of XCRI-CAP for graduate training information.

 

The University’s graduate training site, in its VLE, now has over 350 upcoming courses listed, if the project had not taken place there would only be 186. The current items of graduate training originate from all of the departments, libraries and central services, whereas previously training opportunities were listed on numerous different sites, and some were not online at all. The University will shortly undertake a survey of student and staff views, but already informal feedback is that this initiative is welcomed by graduate students.

 

The departments and services see the advantages to advertising their graduate training widely to students across the University and externally, and to opening up opportunities to students more widely. It also helps raise awareness about how training opportunities are presented.

 

Future Impact

As the initiative to display all graduate training opportunities in one place becomes embedded, it is expected that internally the provision of training itself will adjust as provision is rationalised and more clearly differentiated, and new opportunities and requirements are more readily identified. This will contribute to optimising the provision of training opportunities and the use of resources.

 

Externally, this initiative will contribute to improving the availability information about graduate training opportunities, which may in turn lead to greater sharing of expertise and resources between institutions.

 

One rough measure of success will be to track the number of distinct users visiting the graduate training sites within the VLE. Tracking has been turned on for the last two years so the increase in traffic since the introduction of the fruits of this project should be readily quantifiable. (However, we may not be able to accurately track whether overall attendance at courses has also increased.)

 

Conclusions

The project has been very well received at Oxford both from a staff and student perspective. An initiative such as this has been long overdue and as mentioned earlier, has addressed a number of long term issues. In terms of impact it is still early days as the new service has only been live since January and most training takes place in the first term (October – December) but this gives us a long lead-in time to get the service embedded within institutional processes.

 

There is no doubt in the project team’s minds that the service offered by the project is well overdue and that it will have a positive impact on taught post graduate students.

 

Regarding the project, the management / coordinator role incurred significantly more effort than was initially envisaged. We vastly underestimated the amount of contact with individual training providers and hands-on micro-management that would be needed.

 

We also underestimated the amount of disruption that would be caused by the lead department (Computing Services) merging with 2 other Oxford University departments – all of a sudden project managers were no longer responsible for staff working on the project and everybody had new priorities. This caused a delay in the introduction of the service and is still impacting significantly on most project activities including this final report.

 

One of the lessons learnt is that RDF Linked Open data is most likely a better approach for all sorts of institutional data than an increasing plethora of divergent XML formats. This does not mean they don't have their place, but the experience of OODS has been that using a single approach (such as RDF Linked Open Data) is greatly beneficial for institutional data. In the creation and development of XCRI there has necessarily been a number of reinventions of the wheel. There is a lack of vocabularies for some categories of information, such as saying what level or type of course it is (undergraduate / postgraduate / continuing education, etc.). In undertaking this development OODS has now become easier to use for a variety of additional datasets, created a much more resilient infrastructure, and further developed methods of easily updating and maintaining datasets. The extraction of data from Share Point is much easier and will benefit other institutional data projects which can now use Share Point / Info Path as a data entry system.

 

Recommendations

Recommendations for Jisc & Course Data Community

In order to make it easier for our students to find the training they are looking for, we categorised each item of graduate training using a set of skills devised locally by Oxford University training providers. We thought that we ought to map our local categories to the nationally recognised Vitae Researcher Development Framework (VRDF) (http://www.vitae.ac.uk/rdf) because this alternative categorisation would better understood by UK researchers.  We also felt this would be of greater use in the future when other institutions start producing XCRI-CAP feeds containing graduate training – if all feeds use a common categorisation, then potential students will be better placed to make comparisons and choose the best course to study.

 

Early on in the project we discovered there were licensing issues with VRDF; the conditions of use of the VRDF (http://www.vitae.ac.uk/researchers/274251/RDF-conditions-of-use.html) appear to restrict such use. Despite our best efforts, we have been unable to come to an agreement with Vitae that will allow us to categorise items of training using both Oxford skills and the Vitae RDF: it would appear that Vitae consider the mapping of a local skill set to VRDF to be a breach of their terms of use. Vitae were prepared to grant permission for us to use the dual mapping within a password protected network, but not within a public XCRI-CAP feed. They also granted permission for us to categorise our courses solely in terms of the VRDF. Unfortunately, because we need to display items of graduate training on public websites hosted within the Oxford domain and therefore need the Oxford categorisations to be present, we were not able to comply with Vitae’s request.

 

Due to this impasse, we have made the decision to remove all references to the VRDF from our XCRI-CAP feeds – we feel this reduces the value of our feeds to the UK researcher community and are quite unhappy that we were compelled to take this action.

 

We would strongly recommend that Jisc start a dialogue with Vitae’s in order to find a way around the dual categorisation problem.

 

We have expressed concern at how feeds will be displayed, what elements of a course / presentation will be used by course aggregators? For example, we have a number of short course summaries – these could be considered to be abstracts but alternatively list learning outcomes. We plumped for calling them learning outcomes as they are sometimes blocks of HTML code and are generally longer than 140 characters permitted by the <abstract> element. What if every other course has an abstract and due to the brief nature, this is used instead of a description? Oxford course will appear next to others as sorely lacking in detail. It would have been useful for all projects if there had been a document that prescribes how each field should be used.

 

Recommendations for Oxford

There are a small number of issues worthy of mention here,

 

• Training providers are responsible for supplying course data in a timely fashion. There is no centrally-funded resource to manage this process so training providers need be pro-active in supplying data. Failure to do this risks students losing confidence in the completeness of the list of courses and starting to bypass the VLE tool. When incomplete or out-of-date data is viewed externally, it will appear that certain parts of the University do not offer graduate training. If a training provider needs to commission a new ‘courses database’, it will be necessary to ensure that the new system is still able to supply data in the relevant format.
• Training providers need to maintain and improve the quality of their data; titles should be not be ambiguous; descriptions need to be improved, and should contain the same information and be of comparable length across the institution; hyperlinks need to be maintained and relevant audiences identified. Without this students may become frustrated with the VLE tool.
• Once the funded project has finished, there will (as yet) be no further IT resource for enhancing the processes or tools developed during the project. Every attempt will be made to fix bugs in a timely fashion but significant developments to the service will not be possible without further funding.
• The project Steering Group should remain in place after OXCAP has finished (indeed it was in place before the project, started overseeing graduate student enrolment for training). This Group should continue to monitor the project outcomes and ensure that it is properly embedded within the University’s overall Student Enrolment System (SES). Furthermore, it will be important for all training providers to act in unison to ensure that students get the best possible information regarding training opportunities.

 

Internal Recommendation

• Training providers will be responsible for supplying course data in a timely fashion. There is no centrally-funded resource to manage this process so training providers need to be pro-active in supplying data. This is very important to ensuring that students have confidence in the completeness of the list of courses and find the tool useful. Likewise it is important that the information that is presented externally, is comprehensive and up to date, otherwise it will appear that certain parts of the University do not offer graduate training. If a training provider needs to commission a new ‘courses database’ it will be necessary to ensure that the new system is still able to supply data in the relevant format.
• Training providers need to maintain and improve the quality of their training information: titles need be clear and unambiguous; course descriptions need to meet the standard requirements so that they provide students with consistent, comparable information across the University; the relevant audience needs to be identified, so that students can readily identify what training is relevant and open to them; broken hyperlinks need to be repaired promptly.
• Once the funding project has finished, no further IT resource has been allocated for enhancing the processes or tools developed during the project. While every attempt will be made to fix bugs in a timely fashion but significant developments to the service will not be possible without further funding.
• The project Steering Group should remain in place after OXCAP has finished (indeed it was in place before the project started overseeing the student enrolment for graduate training). This group should continue to monitor the project outcomes, to oversee the maintenance of training information, and ensure that the facility is properly embedded within the overall Student Enrolment System (SES). Furthermore, it will be important for the training providers to contribute and collectively to ensure that graduate students get the best possible information regarding training opportunities.

 

External Recommendations

Licence of Vitae RDF

We found that Vitae’s RDF was too lengthy to use as a skills classifications schema and that students are not familiar with the language so we developed a more lightweight vocabulary – we mapped this skill set to Vitae Researcher Development Framework (VRDF). Since we define the mapping as a two way process we can translate between local and national frameworks.

 

Early on in the project we discovered there were licencing issues with use of the VRDF. We wanted to annotate our courses using terms from the VRDF, these course descriptions would then be made available under an open license. However, the terms of use of the VRDF http://www.vitae.ac.uk/researchers/274251/RDF-conditions-of-use.html would restrict such use. (Vitae’s main concerns are that someone might use it for commercial gain and they want to maintain its integrity and their control over it.)

 

We obtained special permission to classify our courses in a public XCRI-CAP feed using VRDF so long as we added a special attribution / copyright notice but were not permitted to produce a public XCRI-CAP feed which categorised items of training using both VRDF and our locally devised set of skills. Since we need our public feeds to contain items of training categorised by Oxford skills, we took the decision to remove the VRDF categorisation. We feel this reduces the value of our feeds to the UK researcher community. We would strongly recommend that Jisc start a dialogue with Vitae’s in order to find a way around the dual categorisation problem. 

 

At the Jisc Course Data event in London on 14 November, it was mentioned that HEFCE are doing some research on what information graduates need in order to decide which course to study. Early indications are that it may be recommended that XCRI-CAP is used instead of KIS. We do not think that we will be the only institution who wishes to classify their courses using a lightweight skill set as well as the VRDF. If this is the case then it is of vital importance that the VRDF conditions of use issue is sorted out.

 

Comments on BSI Draft of XCRI

We have some comments on the BSI draft of XCRI, specifically about dcterms:educationLevel which is defined at: <http://dublincore.org/documents/dcmi-terms/#terms-educationLevel>. Our reading of it is that it specifies the educational level one should be at to be able to understand a document/work. The XCRI-CAP 1.2 specification uses it to annotate a qualification that one can expect to attain for having completed the given course. We'd recommend finding something more suitable (not that we know of anything), or creating something bespoke (possibly with a controlled vocabulary, e.g. based on the NQF and FHEQ).

 

The specification says "The venue element SHOULD be used to describe the physical venue or venues of the presentation. This MAY be a sub-organisation of the provider of the presentation". A venue is generally a building, campus or room, but surely cannot also be an organisational unit. We would recommend dropping the references to organisations and replacing venue/provider/* with venue/*

XCRI Validator

The validator at http://validator.xcri.co.uk/ but times out occasionally when given extremely large datasets (i.e. all our courses).  For particular datasets like:

https://course.data.ox.ac.uk/catalogues/?uri=https%3A//course.data.ox.ac.uk/id/daisy/catalogue&format=xcricap

errors are reported which the specification does not say are errors. (for example, the order of elements is not claimed to be important for most elements but the schema language they are using to check this does not allow for this and so raises an error.) There is at least one error which we cannot solve, that of missing phone numbers, since we do not store the phone numbers of providers and it would be misleading to provide phone numbers related to buildings (even if we had them). There seems no way around this other than to allow phone numbers to be absent.

 

XCRI for Course Booking

Keen readers may have noticed that we are using an XML schema designed for advertising courses to drive a course booking system - this approach caused us to use some Oxford-specific tags such as those identifying course administrators and course “super users” within our internal feeds. This has not had any adverse effect on the public feeds as these local elements are simply not included.

 

We also needed to use local XML elements to specify which attributes of a course should not be displayed externally and which courses will not accept non-Oxford attendees: we defined attributes and vocabularies to specify visibility and eligibility and for another Jisc-funded project CoBoMo, where the booking web service endpoint is located.

 

Oxford’s Course Structure

The course structure at Oxford appears to be different to that allowed by XCRI-CAP 1.2. XCRI-CAP 1.2 allows a course to contain multiple presentations – each presentation is an instance of a course. At Oxford these presentations comprise one or more course components which may be shared by multiple courses and presentations. As an example, a component entitled “How not to chop off a finger” may be taught to students taking the “Guillotine Skills” and “Dissection” courses; both sets of students will attend the same session at the same time. We found no obvious way to represent this in XCRI-CAP 1.2 so ended up using custom tags. Towards the end of the project it was suggested we use the <isPartOf> tags but as no examples of use exist doing this would be more or less equivalent to using custom tags.

External Feed Display

We have expressed concern at how feeds will be displayed, what elements of a course/presentation will be used by course aggregators? For example, we have a number of short course summaries – these could be considered to be abstracts but alternatively list learning outcomes. We plumped for calling them learning outcomes as they are sometimes blocks of HTML code and are generally longer than 140 characters permitted by the <abstract> element. What if every other course has an abstract and due to the brief nature, this is used instead of a description? Oxford course will appear next to others sorely lacking in detail. It would have been useful for all projects if there had been a document that prescribes how each field should be used.

Course type categorization

Oxford has been somewhat different to the rest of the course data community in targeting CPD and graduate training courses, not undergraduate degree programmes.

 

To the best of our knowledge, the community hasn't adopted a way of categorizing courses by type (e.g., 'undergraduate degree', 'postgraduate taught', 'skills training'). As a result, we fear that our feeds could be aggregated under the assumption that they contain undergraduate courses, leading to confusion among users.

 

We recommend that Jisc works to standardise or adopt a taxonomy of course types and provide recommendations for its use in XCRI-CAP feeds.

 

Further details: email and contact names etc

Project Director         Adam Marshall; Bridget Taylor; Patrick Baird

Project Manager        Adam Marshall

Contact email             adam.marshall@it.ox.ac.uk

Project Web URL       http://tinyurl.com/oxcap