Authors: Simon Wilkinson (simon.wilkinson@nottingham.ac.uk), Heather Rai (heather.rai@nottingham.ac.uk)

JISC e-Learning Activity Area: e-assessment

Higher Education Academy Subject Centre: medicine dentistry and veterinary medicine

 

Case study tags: online learning, use of specialist software, an effect on exam results, student satisfaction with e-learning,innovation in learning and teaching, an influence on educational research, staff personal development, a positive effect on recruitment, a positive effect on retention,an influence on policy, use of resources,modifications to learning spaces,management of learning assets, an effect on social equality, tangible benefits of e-learning, university of nottingham, e-assessment, medicine dentistry and veterinary medicine

 

Background & Context

 

Why did you use this e-learning approach?

 

The use of high-stakes summative assessment is important for all stakeholders. From the students' point of view failure at examination time can have serious progression/career implications and can be costly in terms of wasted fees. From the staff perspective being able to consistently and rigorously defend standards is at the heart of any robust assessment process. However, a key interface between these two groups lies in the accommodation of various forms of disabilities. Being able to pass exams is critical to progression through all degrees, but failure to adequately address potentially discriminating features of an assessment can result in students with certain forms of disabilities being unfairly disadvantaged. This case study describes steps taken to enhance an online assessment system TouchStone to afford a wider range of accommodations. The main aim being to focus on the measurement of subject matter understanding rather than a students' ability to interact with a particular assessment format.

 

What was the context in which you used this e-learning approach?

 

A number of new laws have been enacted to protect the interests of individuals who study in higher/further education and have various disabilities: Disability Discrimination Act (DDA) (1992) and Special Educational Needs and Disability Act (SENDA) (2001) in the UK, and Individuals with Disabilities Education Act (1991) in the US. Analysis of applicants awarded places at the University of Nottingham reveals self-reported rates of disability ranging from 3% to 9% across different faculties. This case-study is written from the perspective of the Medical School within the Faculty of Medicine and Health Science. Reported numbers of students with some form of disability are low at 3%.

 

Interest in accommodating individuals with disabilities has grown as the number of online summative exams has increased. As Konur (2007) notes that examinations emerge as a more critical issue for all stakeholders (staff and students). Without proper consideration the move towards increased online assessment could exacerbate any potential disadvantages for individuals with certain forms of disability.

 

What was the design?

 

The priority was to create a design that could be: a) implemented quickly before each exam with no specialist personnel required, and b) could be used on any computer used for examinations rather than being limited to specific machines. Also, because of the uniqueness of examinee requirements the system must permit accommodation at the individual level rather than whole groups. Training staff and the use of client-side cookies to remember settings were all considered but in the end the most effective and robust solution considered was to create an additional database table to hold individual requirements. Utilising an extra table within the main CAA system had the advantage that accommodations can be 'distributed' to any client machine that the candidate sits at for an exam and is easy to centrally administer. A new simple interface was then added to the system (see figure below) to allow non-technical staff to easily set student preferences.

 

 

Staff involved in the design included disability support officers from the Disability Policy Unit regarding the most common types of accommodations that might be required by students. Administrative personnel who timetable exams were also involved to make them aware of the range of customisations that can be performed in the system and finally IT support staff who may be required to actually set up individual examinee profiles.

 

How did you implement and embed this e-learning approach?

 

The first step towards embedding the approach was to talk with experts from the Disability Policy Unit within the University to try and better understand the range of accommodations that might be required. This was a two-way process whereby the unit advised the programming team on what in theory would be needed, and conversely as the design was implemented the programming team confirmed back with the unit what was actually possible with the technology.

 

The design is now formally embedded within an official departmental/institutional policy. Firstly an assessment method is approved at school teaching committee level. The method (i.e. online examination) will then be publicised to students via a VLE. If any students are worried that the format of the exam may unfairly disadvantage them then they are referred to the Disability Policy Unit for assessment. This unit will then report back to a faculty-based administrative unit with recommendations for any exam adjustment and then this will be relayed to an IT person to alter the examinees' profile within the CAA system (see Case Study: University of Nottingham, e-Assessment, Medicine).

 

Importantly the student is then informed that the changes have been made and asked to evaluate these adjustments by accessing any of the many online formative assessments via the VLE.

 

Currently accommodations are only made for individuals which have been properly assessed by the Disability Policy Unit. Thus far this has not caused any problems with the non-disabled student body.

 

Building on the success of personal adaptation within the CAA system, the University of Nottingham Medical School has now embarked upon a re-design programme to allow customisation of the VLE. Although learning through a VLE is not subject to the same time pressures of a summative exam, any adjustment that can result in faster or more readable text for dyslexic or colour blind users must be considered advantageous.

 

Technology Used

 

What technologies and/or e-tools were available to you?

 

For severe forms of disability there are screen readers and Braille tablets but accommodating less severe forms of disability can be challenging. Most modern browsers can alter the size of text and force background colours but these are quite crude devices. Within an assessment system there are a number of different colours that are used: background, foreground, headings, number of marks, and notes. Failure to adequately consider certain forms of adaptation at the design stage can lead to unsightly images whose backgrounds do not change because it was assumed that the background would always be white. Also, the accessibility capabilities of a browser would not know about any extra time permitted for a dyslexic student, for example. It was therefore decided that the assessment system itself should be redesigned to provide more flexibility.

 

Tangible Benefits

 

What tangible benefits did this e-learning approach produce?

 

• Presentation Adjustments. In addition to the importance of UK legislation such as the Disability Discrimination Act 2005 and Special Educational Needs and Disability Act (SENDA), it is widely acknowledged that reading from computer displays is slower and visually more stressful than paper. Individuals with mild disabilities that have gone undiagnosed in paper exams may find new and unfamiliar challenges trying to complete online assessments. TouchStone has been designed to provide support for a full customisation of background/foreground colours and a range of different font sizes. For some users a pale pastel background could be useful in alleviating migraine headaches, others to avoid colour combinations that they may not be able to perceive through colour blindness, through to examinees with Scotopic sensitivity syndrome who are sensitive to different coloured backgrounds and text presentation. All these changes can be controlled through the creation of personal profiles within the system. This saves a lot of staff time as the computer system can handle a large amount of specific alterations to the interface in a way that would be very tedious if completed on paper.
• Response Adjustments. The TouchStone assessment system utilises standard HTML form elements such as radio buttons, check boxes and command buttons. Most browsers, including Internet Explorer and Firefox, will permit keyboard operation of such items using a combination of tab keys and the return button. This affords the inclusion of examinees who may not be able to use a mouse for one reason or another.
• Timing Adjustments. Paper-based exams are usually printed with the exam duration shown on the front cover. Replicating this online is important so that examinees can pace themselves correctly. However, letting students with extra time know precisely how long they have can also be facilitated online. A user profile in TouchStone can be set to include extra time as a percentage. The system will then take the normal duration of an exam, expressed in hours and minutes, and apply this extra time percentage and display this to users with this accommodation in their profile.
• Settings Adjustments. Where possible a separate room is used for candidates with additional time so that they are not disturbed when the normal exam time completes and a potentially large and noisy cohort of students tries to leave.

 

Did implementation of this e-learning approach have any disadvantages or drawbacks?

 

The largest disadvantage to the approach was upfront development time to alter the CAA system to be more flexible. What at first appeared quite a simple task of changing foreground and background colours and resizing text quickly grew to changing the colours of three other text categories and re-editing icons to set transparent colours. However, it must be stressed that the redesign time/cost is a one off expense.

 

An ongoing drawback is the potentially higher level of negotiation that is required with examinees. The Disability Policy Unit, administrative units and the IT support personnel must work closely together to ensure that students are made aware of what adjustments can be made, what would be most suitable for them, and then to ensure that any that are performed work as intended. This approach taken so far has been quite proactive whereby the range of accommodations possible online are highlighted to all users formally identified through the Disability Policy Unit.

 

How did this e-learning approach accord with or differ from any relevant departmental and/or institutional strategies?

 

The University of Nottingham has published a Disability Equality Scheme (DES) document that states at the beginning: "The University of Nottingham is committed to equality of opportunity and is keen to achieve an inclusive environment for staff, students and stakeholders, in which all its provision, policies and procedures, including the curriculum, are accessible." The University of Nottingham Medical School has built and operates its own computer-based assessment system so it was important that the system should be refined in light of this institutional policy.

 

Lessons Learned

 

Summary and Reflection

 

The key advantage of attempting to include as many accommodations as possible within the online system, aside from any legislative requirements, is to ensure that academic standards are maintained. The University of Nottingham utilises several image hotspot and labelling questions in summative examinations which do not directly translate into any paper question types. If an alternative question was written for paper is the academic challenge the same as the interactive computer-based question? The colours used with these interactive question types can be altered within TouchStone, the challenge for the next version is to provide full keyboard support for these questions that use Flash instead of standard HTML form objects.

 

The numbers of students requesting various accommodations has been low, there are only 30 currently, so no formal research has yet been conducted. However, informal discussions with several individuals have highlighted the acceptability of this customisation approach in the students eyes.

 

Konur, O. (2007) Computer-assisted teaching and assessment of disabled students in higher education: the interface between academic standards and disability rights. Journal of Computer Assisted Learning, Vol 23. pp. 207-219.