Funded by the: Jisc e-Learning programme.
Lead Institution: University of Lincoln.
Learner Provider Type: Higher Education
Project Duration: January 2012 - March 2013
Key Words: Course Data
Case study tags: course data, process improvement, kis, hear, course information, data visualisation, university of lincoln
Note: This is an abridged version of this project's final report. The full version is available here.
Course Data: Making the most of course information - ON Course
Project Summary
The ON Course project was aligned with an existing internally driven project at Lincoln to implement a new ‘Academic Programme Management System’ (APMS) for managing curriculum documentation and related marketing information. As well as documenting this ongoing project, ON Course aimed to ensure that a valid XCRI-CAP feed of course information was published and to demonstrate the uses of rich course data when managed well. As such, ON Course went beyond simply publishing an XCRI-CAP feed by developing publicly accessible APIs for all curriculum information held in the APMS and then demonstrated its richness through visualisations and three prototype applications. Visualisations were created for six years of course data and a paper was presented at a conference in Barcelona on the use of statistical analysis of course data for business intelligence purposes.
Programme information was text-minded and semantically tagged to prototype a ‘course-finder’ application and a set of APIs were developed to enable application development using programme information.
In summary, the project saw the successful implementation of a new Academic Programme Management System for all curriculum data at Lincoln and began to examine and re-use that data in a way that should benefit the institution in areas such as Business Intelligence and Marketing.
What could have been improved? What lessons have been learned?
Developer-led
The ON Course project was ‘data-driven’, in the sense that Jamie Mahoney, a full-time developer at Lincoln, worked on it for 12 months with the objectives of understanding and demonstrating re-use of the course data being compiled through the APMS. Jamie’s work relied on and followed the development of the APMS very closely.
Early on, the APMS did not provide any data for research and development, so to familiarise ourselves with the type of data we would be working with, we began by scoping the extent of the data which was required by the KIS (KIS: A summary http://coursedata.blogs.lincoln.ac.uk/2012/02/10/key-information-sets-kis-asummary/).This gave us a good indication of what data the university was obliged to provide and how it measured up against the availability of existing data. Jamie produced a useful summary of the KIS for other institutions to use (Summary of KIS Fields, Descriptions and Data Sources http://lncn.eu/diy3).
We also undertook a focus group with students to better understand what information they looked for when applying to the university. The outcome of this workshop was analysed and written up on the project website (Focus Group http://coursedata.blogs.lincoln.ac.uk/2012/03/15/focus-group-14th-march-2012/).
Learning from visualisations
Next, we spent some time researching data visualisation best practice in order to inform our work on visualising the APMS data (Data Visualisation blog posts http://coursedata.blogs.lincoln.ac.uk/2012/03/12/data-visualization-1/ &
http://coursedata.blogs.lincoln.ac.uk/2012/03/15/data-visualization-2/).) This was very instructive and laid the groundwork for a peer-reviewed conference paper on ‘Data Visualisation and Statistical Analytics Within the Decision Making Process’ (Data Visualisation and Visual Analytics Within the Decision Making Process http://coursedata.blogs.lincoln.ac.uk/2012/11/22/conference-paper-accepted/). While waiting for the APMS to be populated, we had to go back to previous data generated from reports in order to understand the extent of the programme data we would eventually be dealing with. These early reports covered six years of course data and around 50,000 individual award/module relationships (What to do with six years of course data? http://coursedata.blogs.lincoln.ac.uk/2012/03/30/what-todo-with-six-years-of-course-data/ & How Not to Visualize Course Data http://coursedata.blogs.lincoln.ac.uk/2012/03/30/how-not-to-visualize-course-data/). As well as gaining a comprehensive and historical overview of programme, course and module data, it allowed Jamie to learn how to use the visualisation tool, Gephi, which produced some interesting and beautiful visualisation that offered limited business intelligence but pointed towards further work that was both possible and potentially useful (Analysing Network Visualisation Statistics http://coursedata.blogs.lincoln.ac.uk/2012/06/01/analysing-network-visualization-statistics/).
Open Badges
While waiting for the APMS data to be compiled and made available to us, we considered how course data could be used in conjunction with Open Badges (Designing a Badge System for Universities http://coursedata.blogs.lincoln.ac.uk/2012/04/25/designing-an-badge-system-for-universities/). Open Badges are a new, additional form of accreditation intended for any form of learning. They have not yet been widely applied in the HE sector, but given our understanding of course data, we could see how course data could be used to develop a system for awarding Open Badges. We documented and released the source code for our open badges system, offering a prototype application for awarding badges from an HEI using Mozilla’s Open Backpack (Release the badges! http://coursedata.blogs.lincoln.ac.uk/2012/05/23/release-the-badges/).Having done this, we discussed the possible use of Open Badges at Lincoln with staff from Planning and the Registry who thought that it may complement the HEAR (HEAR http://www.hear.ac.uk). However, there were clearly significant implications in integrating the awarding of badges against formal programme learning outcomes and we felt this was beyond the scope of the project. However, it is something we may pick up again at a later date. A demonstrator application for our work on Badges is available online. A member of the university can sign in and is minted a sample badge which is then passed to their Mozilla Open Backpack, completing the process of receiving, accrediting and displaying their open badge (Badge application http://badges.lncd.lincoln.ac.uk/).
Following our work on Badges, Jamie returned to visualisations of course data and negotiating with staff about potential applications for the use of course data. On this latter point, early conversations were not fruitful. Staff didn’t have a full grasp of the aggregate data that the APMS would make available to them until later in the project when the APMS was populated. In contrast, our work on visualising course data gave us early insight into what was possible with the data and this fed into the conference paper as well as producing further visualisation and research which we published on our blog (Back to Visualising Course Data! http://coursedata.blogs.lincoln.ac.uk/2012/05/25/back-tovisualizing-course-data/ & Making Lasagne Not Spaghetti http://coursedata.blogs.lincoln.ac.uk/2012/09/06/making-lasagne-not-spaghetti/).
API-driven development
Around eight months into the project, the APMS project made data available to us that we could warehouse and write our own APIs for. This is a process we have undertaken on a number of projects and allowed us to develop a large relational database of institutional data, including programme information, building data, people data, bibliographic data and event data. This is leading to the release of http://data.lincoln.ac.uk, part of the http://data.ac.uk initiative. With the APMS data being warehoused in our ‘nucleus’ data warehouse, we then began mining it, in response to a request by the university Webmaster for a prototype ‘course finder’ application. The work is documented on our blog, and involved the use of Reuter’s Open Calais semantic tagging API (In Search for Similar Courses http://coursedata.blogs.lincoln.ac.uk/2012/10/12/in-search-of-similarcourses/). In addition to visualising course data, this process was a further method of understanding the complexity of the relationships between programmes, courses and modules. The tags that Open Calais returned were likewise warehoused in our database and used as the basis of the course finder application which makes suggestions for courses of possible interest to the user. A demonstration of this application is online (Course Finder application http://coursefinder.lncd.lincoln.ac.uk/). In addition to the Course Finder application, the Head of Computer Science at Lincoln asked whether we could develop an ‘assessment builder’ application that would make the process of writing assignment documentation quicker, easier and more accurate. By tying the application in with assessment data, the assessment strategy delivered within the module will be identical to the strategy as defined in the validated module documents. The warehousing of APMS data and the development of the APIs was an important exercise in modelling course data for our needs. We learned that although data may be available through APIs, it may not be presented in formats that are usable in the development of other applications. By reading, storing and re-publishing the data through our own APIs, we make the data more useful for both ourselves and, where appropriate, others (We have written about API-driven development for our Jisc-funded Orbital project. ‘Eating your own dog food’ http://eprints.lincoln.ac.uk/5962/). This process also helps to develop an understanding of the sheer scale and complexity of the data being worked with. The APIs for our course data are publicly accessible (the user must request an authentication key) as is the research data we have produced (ON Course research data https://ckan.lincoln.ac.uk/group/on-course). Tony Hirst has produced a number of blog posts which evaluate the usefulness of the APIs and potential applications of the data (OUseful.info http://blog.ouseful.info/tag/oncourse/). Tony’s use of the Lincoln XCRI-CAP feed, ON Course APIs and programme data pointed to the advantage each institution has over third-party services reliant on XCRI feeds. The programme information held by institutions is much richer than provided by XCRI feeds and represents a source of information that could be repurposed to provide value for the organisation (From XCRI-CAP 1.2 to CourseData API? http://blog.ouseful.info/2013/01/24/fragmentary-coursedata-thoughts-from-xcri-cap1-2-to-coursedata-api/). His use of our API demonstrated how freely accessible and powerful third-party tools such as Google Spreadsheets, can be used to view, browse and re-purpose the data with very little additional overhead required by the institution. Tony also provided comments on the design of the API, which we have incorporated and corrected where appropriate.
Future Impact
It is often the case that the impact and benefits of a project are realised after the project has ended and the work is absorbed more fully into the rhythm of the institution and its committee structures. The implementation of the APMS has had an immediate impact on the management of course data at Lincoln as well as the marketing of the information and the production of diploma supplements, too. Earlier methods of creating and managing curriculum information have been wholly replaced by the use of the APMS and we have, for the first time, produced an up-to-date, valid XCRI-CAP feed. From this year, the APMS will be used to generate our KIS submission on an annual basis and we expect it to be the main source of HEAR data, too. The three applications that ON Course produced (Badges, Course Finder and Assessment Wizard) are prototypes which we expect will find future uses as we are able to disseminate them more widely across the university. We would like to see further exploration of how open badges might be used to recognise attributes of student learning that are currently not formally recognised e.g. workshop participation. We would also like to see the Course Finder application embedded within the main university website. The Assessment Wizard is likely to be piloted by the School of Computer Science and following that we will offer it to other programmes within the university. More generally, ON Course builds on past Jisc projects that focus on data-driven development and the role of APIs in developing applications and integrated services within an enterprise environment.
Our work on visualising course data has demonstrated that with the right skills, such data can be mined, analysed and remodelled to provide a more holistic understanding of the impact of institutional change processes. We will continue to use the project blog as a way of reflecting on our work with course data and any future developments that occur.
This report was written shortly after we presented our conference paper and we fully intend to develop the paper into a journal article.
Recommendations/Conclusions
Implementing a programme/course management system is a large, complex project. Before embarking on it, an institution must ensure it has the time, people and funding available to make it a success. Stakeholders at senior management team level must actively support and drive the project.
The development of curricula within HEIs is a primary responsibility of academic staff. The implementation of a course management system should therefore include academics as key stakeholders.
The time needed to populate a new, structured system from existing unstructured information should not be underestimated. It is likely that it will have to be done manually, and will take considerable time and effort.
Extensive and properly structured data about programmes can lead to more innovative uses of the information elsewhere. Without a common, consistent structure and means of access, re-use (internal, public and third-party) is almost impossible.
The provision of an XCRI-CAP feed was, correctly, a minimum requirement for the Jisc programme of funding. However, institutions should recognise the value of their full programme data on a much broader level, understanding that it offers insights into one of its core activities (teaching and learning) and can inform other change management programmes across the campus. (For example, in an earlier project, Lincoln produced tools for enabling closer collaboration between academics and estates staff in the redesign of campuses and the re-use of space. Similarly, visualisations and text mining of course data could be used to inform the collaborative development of curricula. http://learninglandscapes.lincoln.ac.uk/).
Application development around the use of course data has so far tended to be marketing focused. However through the use of visualisations and interactive tools, it could be put to much wider use. The HE community should consider how the use of course data might fundamentally change the design of curricula and provide the basis for discussion between staff and students about the re-production of academic life. Improving the level of oversight and insight into the core activity of teaching and learning can reveal and help question existing constraints in the curriculum design and QA processes, for example. (At Lincoln, the curriculum design process has recently undergone changes recognized by the QAA (http://www.lincoln.ac.uk/news/2013/02/643.asp)). ‘Student as Producer’ is the organizing principle of academic life, encouraging and enabling students “at all levels to view themselves as active producers of knowledge, rather than passive consumers.” http://studentasproducer.lincoln.ac.uk)
Course data, alongside other datasets (e.g. space utilisation, research activity, energy use, facilities, etc.) provides unprecedented insight into the nature of our organisations. Institutional managers should approach their organisations as objects of research and development. The knowledge and skills exist within HEIs, among students and staff, to better understand, critique and develop the form of the organisation.
Data visualisation techniques provide a new and novel oversight of large and rich course datasets. However, due to the complexity of this data, the use of static visualisations quickly reaches their limit. Software applications are needed that help staff and students interact with the data and model new pathways to learning, reveal greater opportunities for collaborative and cross-disciplinary teaching, and critically evaluate the history and future of the institution.
Further details: email and contact names etc
Project Director Joss Winn (PI) Contact email jwinn@lincoln.ac.uk
Project Manager Allister Homes
Project Webpage URL http://coursedata.blogs.lincoln.ac.uk
Project Video - http://www.youtube.com/watch?v=pjOWeTwkIyM&list=PLvsH4KQkUo_lHJgfRsFPRPdlMsuENK6MC&index=4
