University of Glamorgan - Phased online summative assessment in undergraduate accounting


Author: Pru Marriott (pmarriott@glam.ac.uk)

JISC e-Learning Activity Area: e-assessment

Higher Education Academy Subject Centre: business management accountancy and finance

 

Case study tags: online learningan effect on learningan effect on exam resultsan effect on student personal development,student satisfaction with e-learning,innovation in learning and teachinguse of resourcestangible benefits of e-learning,university of glamorgane-assessment,business management accountancy and finance

 

Background & Context

 

The use of phased online summative assessment on a first year undergraduate accounting module.

 

Why did you use this e-learning approach?

 

The use of phased online summative assessment is recognised as an important and powerful assessment practice which supports high-quality learning and teaching. The rationale for the change in assessment was in response to students' poor performance, lack of participation in the learning process and superficial engagement in the topic area and focussed on:

 

 

What was the context in which you used this e-learning approach?

 

The students in the case study are first year accounting and finance degree students studying six 20 credit compulsory modules: Financial Accounting, Management Accounting & IT, Quantitative & Research Skills for Accountants, The Economic Environment, Organisational Behaviour and Business & Company Law. The delivery of the undergraduate programme is linear and the module is taught for 24 weeks over the academic year. Weekly class contact consists of a 2-hour workshop and a 1-hour lecture.

Sixty students (33 male and 27 female) were enrolled on the module at the beginning of the academic year and all had achieved at least 220 points at 'A' level, although very few of them had studied accounting to this level.

 

The case study concerns first year accounting and finance degree students enrolled on the Financial Accounting module assessed by an in-class time-constrained case study, weighted 30%, and an end of year 3 hours' time-constrained examination.

Changing the method of assessment in a significant way was challenging in terms of how the students would engage with the revised assessment practice and how effective it would be in improving student learning. Given the need for the multiple delivery of tests, the potential for collusion between students was identified.

 

What was the design?

 

QMP was used to develop 5 phased online summative assessments. At the commencement of the module students were issued with a lecture programme which highlighted the dates of their online summative assessments and the topic areas to be tested (see Table 1 below).

 

Table 1 - Assessment Timetable and Topics Tested

Week Beginning Activity Topics Tested
23/10/06 QMP Test 1 Introduction to Accounting and the Accounting Equation; Double-entry book keeping
27/11/06 QMP Test 2 Preparation of a Trial Balance and the Financial Statements; Books of Original Entry (Sales Day Book and Sales Ledger); Books of Original Entry (Purchase Day Book and Purchase Ledger); Books of Original Entry (Cash Book and The Journal)
08/01/07 QMP Test 3 Accruals and Prepayments - The Matching Concept; Accounting for Depreciation and the Disposal of Fixed Assets; Accounting for Bad Debts and Provision for Doubtful Debts; Bank Reconciliation Statements
29/01/07 Paper Based Test Topics covered in QMP tests 1 - 3 (Designed to consolidate knowledge)
26/02/07 QMP Test 4 Errors and Suspense Accounts; Incomplete Records; Types of Business Organisations Accounting for VAT
26/03/07 QMP Test 5 Company accounts; Company finance; Cashflow statements; Interpretation of accounts; Partnerships

 

A bank of 400 questions was created, separated into topic groupings, and comprised multiple-choice, multiple-response, true/false, yes/no and text match questions. Five online assessments were written from these questions and typically students would receive a test of 30 questions taken from a topic grouping comprising of 60 questions.

 

The feedback to each test included a score for the test, a summary of the questions asked and the student answers to those questions and an explanation of the correct answers.

 

A QMP expert, employed at the university, was instrumental in the initial creation of the question bank, from questions provided by the tutor, and responsible for delivering a series of training seminars on the use of QMP. Ongoing technical support was also provided throughout the duration of the case study.

 

How did you implement and embed this e-learning approach?

 

Sixty students were enrolled on the module and all students were given the option of taking the tests online or paper based. One student within the group had a visual impairment which precluded her from sitting all the tests online and another requested a paper-based copy, out of preference, for the first test only - he voluntarily reverted to online assessment thereafter.

 

Prior to each test, hard copies of test papers were prepared as a backup should there be any technical issues which would prevent online delivery of the assessment. The contingency plan was not needed as all tests were successfully delivered online.

All the tests were time-constrained (50 minutes) and sat under exam conditions in a supervised on-campus computer lab during the students' workshop sessions on pre-determined days during the academic year.

 

The limited size of computer labs required multiple deliveries (3 groups of 20 students) of the tests which were undertaken during the same weeks but at different times. To prevent unauthorised and unscheduled access the tests were password-protected with the name and password being changed for each sitting.

 

To prevent collusion between students during and after each session, the tests comprised questions which were randomly selected from the question bank and then randomly ordered on the assessment papers. Additionally, the answers linked to the multiple-choice and multiple response type questions were 'shuffled'. These safeguards resulted in students receiving a unique set of questions each time and removed the opportunity for unfair practice.

 

An anticipated issue was raised by the external examiner concerning the relative difficulty of each test. This was addressed by the design of the assessments where each combination of test comprised an equal number of questions testing topic specific knowledge and an equal distribution of difficulty levels. The levels included easy, medium and difficult. The chance of any test being significantly more difficult than any other was removed.

 

Technology Used

 

What technologies and/or e-tools were available to you?

 

Question Mark Perception (QMP) is a web-based interactive assessment tool which facilitates the creation, delivery, tracking and management of online assessments, quizzes and surveys. The assessments created in Question Mark Perception may be delivered through any web-based environment including the internet and school or college intranet sites. A wide variety of question formats are supported including multiple-choice, true/false, missing word, ordering, complex matching and hotspot styles. The questions and associated tracking database are stored on a central server and accessed by students through a standard browser window on a PC with internet access. Student performance and progress can then be monitored by tutors with secure online access to the database.

 

It is a mature product which can be blended with traditional or online teaching methods and is compatible with the university's current learning systems and integrates well with our current VLE (BlackBoard).

 

Tangible Benefits

 

What tangible benefits did this e-learning approach produce?

 

In order to collect data on students' attitude towards the introduction of online phased summative assessment a self-administered feedback form was distributed to all students in attendance at the end of the first semester. In addition to the feedback form focus groups were conducted to elicit opinions on the new system of phased online assessment.

 

The results for the academic year 2006-07 showed the mean pass rate for the module to be greater than the mean pass rate for the three accounting related modules for the first time since 2003-04.

 

Pass Rate

 

Also, when comparing the grade point average across each of the three accounting related modules for each year, the performance on Financial Accounting in 2006-07 has seen the most improvement.

 

Financial improvement

 

In addition to these positive improvements in performance, the results for 2007 show a return to the alignment of coursework and examination performance last experienced in 2003-04.

 

Coursework/Examination performance alignment

 

From the analysis of student results, there is prima facie evidence that students' performance on the financial accounting module has improved as a result of changing the method of assessment.

 

From the feedback sheets and focus group interviews it was evident that the students prefer to be assessed online rather than by traditional methods and expressed a preference for phased online assessment testing small areas of the syllabus rather than one test covering many topics. Also the students considered that the phased assessment improved their learning by encouraging them to allocate their study time appropriately throughout the year. Students considered the phased testing and the feedback they receive to be timely and useful in the learning process which facilitated opportunities for self-assessment, evaluation and reflection at an early stage and throughout the year. This provided opportunities to perform better in future assessments by focussing on topic areas where weaknesses exist.

 

The following are a sample of quotes taken from the focus group interviews:

 

'I think it is easier to do a test every now and then because you are revising throughout the year, so when it comes to your last final exams you will find it a lot easier because you would have done the work all through the year.' (Group 1, Student 4)

 

'I think it's good for the first year though, the idea of the first year is to ease you into Uni life and I think these tests help you integrate the work easier than bombarding you with big assessments and exams...' (Group 1, Student 3)

 

'Yeah, the instant feedback is good cos normally you have to wait for weeks for feedback. I mean they give you a deadline to hand in the work but they don't give you the mark back for ages.' (Group 2, Student 7)

 

'With these exams [tests] at least you know how and where you are and what you are capable of doing so by the end of the year, you know you can revise on the stuff that you are no good at.' (Group 2, Student 4)

 

'Say, if you revise a topic and do a test and have a bad mark then you know you have to go back to revise that topic... They make you learn more throughout the year rather than doing it at the end.' (Group 2, Student 7)

 

'If you failed in one test, it pushes you more to do better on the next one.' (Group 1, Student 5)

 

'Yeah, if you take the tests away I don't think I will keep doing the same things to be honest with you. I wouldn't work through the year to the same extent.' (Group 2, Student 2)

 

Did implementation of this e-learning approach have any disadvantages or drawbacks?

 

There were no drawbacks from the tutor's perspective other than the time that was expended in creating the question bank, setting up the tests and inputting the detailed feedback on the system. This initial effort, however, will not be required to be repeated and the benefits will be available for many years. From an institutional perspective, there was a cost associated with the acquisition of the QMP software which was piloted on this case study. As a result of the positive feedback from the case study the system will be used on modules across the university.

 

How did this e-learning approach accord with or differ from any relevant departmental and/or institutional strategies?

 

The university, through its Centre for Excellence in Learning and Teaching (CELT), promotes the development and support of excellent learning experiences for students and staff and works with faculties to promote the embedding of blended learning into modules. Blended learning relates to the enhancing of current learning opportunities using supporting technologies.

 

The introduction of phased online assessment using QMP was fully supported by the faculty and CELT as its adoption accords with the principles of enhancing the student learning experience.

 

The success of the e-assessment introduced in this case study provides evidence to support its integration into other module across the university. This assessment method will be adopted by other first year modules within the Business School in the next academic year.

 

Lessons Learned

 

Summary and reflection

 

Assessment plays a significant role in the teaching/learning process and the positive comments received from students relating to their recognition of the benefits of engagement, self-assessment and reflection, feedback, motivation and time management afforded by the continuous/phased nature of the assessment provides evidence to suggest that a change in assessment practice has been effective.

 

For the tutors, measuring student development and highlighting, at an early stage, those students that were experiencing difficulties in understanding certain topic areas enabled the tutors to provide appropriate support where needed. For the students, the immediate online feedback made a contribution towards developing their motivation and engagement with the subject through self evaluation and reflection. By identifying students' strengths and weaknesses and thereby raising their awareness and expectations, student learning and participation in the learning process improved. The prima facie evidence of an improvement in student performance compared with previous years also supports this.

 

The case study reports how technology has been used to improve students' learning and how technology can be used successfully for assessment purposes. The e-learning approach has worked in terms of meeting the departmental and institutional strategies for improving learning and teaching through the use of technology.

 

Undertaking this case study has provided me with an understanding of how students prefer to learn and how they engage in the teaching and learning process. Summative assessment plays a major part in their university experience and the way in which we assess has major implications on the success or otherwise of our students. Adopting an assessment strategy that students engage with is therefore vital.

 

Further Evidence

 

'Students considered the phased testing and the feedback they receive to be timely and useful in the learning process... throughout the year.'

 

'Yeah, if you take the tests away.. I wouldn't work through the year to the same extent.' [student comment]