Authors: Mel Powell (m.j.powell@derby.ac.uk)
JISC e-Learning Activity Area: e-assessment
Higher Education Academy Subject Centre: business management accountancy and finance
Case study tags: online learning, an effect on exam results, an effect on student personal development, student satisfaction with e-learning, innovation in learning and teaching, an influence on educational research, staff satisfaction with e-learning,staff personal development, a positive effect on retention, tangible benefits of e-learning, university of derby, e-assessment, an effect on learning,business management accountancy and finance
Background & Context
Why did you use this e-learning approach?
A progressive set of computer-based formative assessment materials with extensive learning materials and feedback were created for a cohort of first year full-time students studying economics on business programmes. The materials were used to support private study and did not replace lectures or classes. The aim was to reduce failure rates amongst full-time students through improving self-regulating learning through computer-based formative private study. Recent literature suggests high failure rates may result from inadequate self-regulated learning skills among first year students and the limitations of traditional lecture and class formats in developing these skills. Increasing the degree of formative assessment and self-regulated learning can have positive impacts on learning outcomes for such modules. In particular, practice and learning structure can be reinforced through formative assessment to improve self-regulated learning and learning outcomes.
The formative e-assessment format was chosen because it was relatively easy to apply to the subject material (interactive graphics, calculations and links to Excel files) and because Derby University has an established department and expertise in designing and producing computer-based learning and assessment materials. In addition, prior experience of developing and using summative computer-based assessment on the module had shown that students were keen to use computer-based assessment and learning materials. The project started in response to (and was funded by) a University call for projects to increase student retention. The approach also met general University criteria to develop e-learning within existing modules.
What was the context in which you used this e-learning approach?
Derby University has a wide range of undergraduate business related degrees, some specialist such as Marketing or Business Management, and some joint honours Business and Law. All students on business-related degrees take a micro-economics first year module. This module is the focus for this case study. Student numbers on the module range from 250 to 350 a year. It was a one semester module taken in the Autumn, assessed by 2 phased (invigilated) computer-based summative assessments taken in weeks 6 and 9 of the semester and a final written examination. The assessment starts with simple multiple choice formats (assessment 1), moves on to calculations and model manipulation (assessment 2) and ends with an applied analysis using the economic tools to analyse a real world business case (examination).
The students come from a wide range of backgrounds, including home, EU and overseas students. Most students have either not studied economics before or only studied economics as part of a business A-level or equivalent qualification. They are generally unprepared for the model building approach and use of graphical analysis, and lack basic numeracy skills and the ability to apply a generalised model to a specific context. Failure rates for this module have been high. This type of subject requires regular practice and development of knowledge and skills over short time spans to achieve learning outcomes. The module teaching comprises a set of 11 lectures, supported by 11 tutorial classes (in groups of up to 20 students. Both lectures and tutorials are supported through the University's VLE. Online materials include study guidance, information on assessment and use of computer-based learning materials and formative assessment, lecture notes, tutorial materials (with answer sets), study guidance, recommended reading for each lecture and class. The tutorials are designed around a set of problems and activities which include analysing newspaper articles and using graphical analysis of relevant models. The lecturer is one of the two or three tutorial tutors. This format was used before and after the introduction of the computer-based formative learning and assessment materials. After introducing the materials, we also provided a weekly clinic for students who needed additional support.
The main challenge for the module staff was developing the content for the e-learning and formative assessment. It was important that the materials used were closely tied to the lecture materials and the tutorial materials, whilst developing the appropriate skills and knowledge for the summative assessments. They also needed to be designed to provide extensive feedback loops and developmental learning as the students would be progressing at their own pace. Because the size of the files in use would be large, the materials would only run on computers with sufficient memory. At the time when the materials were first introduced, many of the computers available in student labs had insufficient memory. The University was also unable to mount the materials on the intranet so that students could access it externally. This meant that students would have to access the materials while in the University. This posed more of a problem for part-time than full-time students. We were concerned that students might use the computer-based formative learning and assessment materials and decide not to attend tutorials. Finally, we were aware that students rarely read any of the guidance material provided online and we anticipated that many students would never find out about the computer-based formative materials unless a wide range of information sources and incentives were used.
What was the design?
The computer-based formative assessment system was designed and created by the module leader and one tutor in conjunction with the Derby University's computer assessment department (CIAD). The process began with the creation of e-learning materials and formative assessment activities by the module leader and tutor in Word format. The CIAD team used this material to generate the formative materials using the TRIAD software. We used the first unit as a pilot with current students asking for feedback on ease-of-use, relevance, and suggested improvements. It was refined and developed and then sent to an external for moderation. The remaining units were designed in a similar way. The two tutors compared notes on successful methods of communicating what was required and we quickly created a format that the CIAD team could easily interpret.
The materials contained instructions for progression at all levels. They also provided conceptual knowledge, used both interactive learning activities and interactive formative assessment with extensive feedback loops, automated advice on incorrect responses, advice and activities for improvement, multiple attempts, staged development and continuous formative grades. The materials were mainly interactive activities such as simulations, drag and pull diagrams, recognition exercises, calculation activities, and concept identification activities, leading to staged applied analysis of real world newspaper articles. The aim was to keep the student active and provide continuous feedback whilst building knowledge and skills. Students would require the knowledge and skills for the three computer-based summative assessments for the module. Access was by username and password and was not time-constrained. However, students had to access the materials on a campus computer. Information on date of access, time taken, activities undertaken and scores achieved were recorded with each student access for monitoring.
The formative learning and assessment materials were explicitly related to other learning materials identified for the module and to the three summative assessments. Each lecture was supported by a unit of computer-based materials. Students received information on the screen about the links to lectures and classes as well as to other support materials. Some activities were designed to help students prepare for class activities and discussions, others to prepare for summative assessments and to promote the use of other learning materials such as lecture notes and books. Students were told to use the materials throughout the module and were told the outcomes and usage would be monitored. The overall design was created to follow good practice and theoretical developments in formative assessments and self-regulated learning.
How did you implement and embed this e-learning approach?
The first set of students to use the e-learning and assessment materials were told about the support materials in the first lectures and given details on how to access the system, reasons why it would help them to study and the benefits of using the materials. They were also reminded that the information was available in handbooks and also on the university intranet. They were told that they should start to access the material that week to prepare for the first tutorial in the second week and that access would be monitored. We had created a new weekly clinic time for students on the module to attend for individual or group help. Anyone failing to access by week three would be asked to attend additional clinics. In the second week, when the first tutorial started, tutors reminded students about the e-learning and assessment materials and asked students who had tried it what they thought about it and whether they had any difficulties. Students were disappointed that they could not access the system remotely from home, but soon realised that they could use the system in short bursts. It was a useful activity to fill in gaps in their timetables when they were attending lectures and tutorials.
The teaching team liaised with the CIAD team on a regular basis to monitor any errors or problems with the materials identified by students. These were dealt with contemporaneously. As no other staff were involved in the roll-out, no additional training was required.
The approach was evaluated in two ways. Firstly, students were asked to provide informal feedback in tutorials on the materials and the system of formative assessment and learning. Secondly, the module tutor created a data set on outcome and input variables by student for the module. The data set was based on individual student information. It included information on student attendance at tutorials, student access and outcomes from the formative e-assessment materials, and student achievement in summative assessment for the module. In addition, each student was asked to complete a skills audit questionnaire before and after the module. The scores from this audit were entered into the data set by student. Evidence of the statistical analysis of the data is provided in the linked document.
Students also provided formal end of module feedback in the form of a standardised student questionnaire which could be compared to previous cohorts on the module. The formal feedback responses and the achievement standards in summative assessment were compared for the cohort that used the formative e-learning and assessment materials were compared to previous cohort years. The results are provided in the linked document. In addition, a theoretical model for developing student self-regulated learning through formative assessment was created from the current literature and applied in the analysis of the individual student data. A paper was written on the outcomes for an appropriate conference on student teaching and learning and is provided in related analysis.
In general, the strategy for implementation worked well. We found some students missed out on the early instructions because they started the module late. These students failed to read the available information online and in module handbooks and did not pick up the information because they had not established friendship groups. Some students could not understand that the system could only be accessed through a password and username on a campus computer. They tried to access the system remotely and failed. Some students tried to use their own username and password rather than the one provided. In general, the problems were lack of familiarity with University processes and systems and not taking responsibility to read instructions provided.
Technology Used
What technologies and/or e-tools were available to you?
The computer-based formative learning and assessment system was produced using TRIAD software. Our CIAD department had substantial expertise with the system and created all the materials from our content information. The module team had already used TRIAD software with the CIAD department to create summative assessments for the module. This meant that the module tutors did not need to develop any further expertise in design or technology applications. The CIAD team had also shown us demonstrations of a variety of learning and formative assessment materials which they had developed for other faculties and modules. We could see that they system was ideal for developing interactive graphics, links with data in Excel, presenting learning materials with interactive feedback and activities. It also required very little computing skill from students. Experience of mouse use is all that is required. As the summative assessments for this module were also designed and presented using TRIAD software, students could become used to the interface and types of questions before undertaking a summative assessment.
The TRIAD software allowed us to create a progressive learning system within each unit to allow for individual student differences in understanding, knowledge and skill. When a student enters an answer to a question or undertakes an activity, they enter a pathway of feedback responses. An incorrect answer or activity, depending on its nature, will elicit a feedback explanation and a subsequent loop into another example before taking the student back to the question for another attempt. In addition, the system allowed us to monitor student access in terms of the date of access, the number of times they accessed, the time spent at each access, the level of progression at each access and a summary score. This was important because we could monitor how students were progressing and intervene at an early stage (week 3) when students were failing to engage or struggling to achieve learning outcomes.
Tangible Benefits
What tangible benefits did this e-learning approach produce?
The evaluation data showed improvements in pass rates and grades achieved in the first two computer-based assessments as well as in the final analytical assessment compared with the previous two years. Changes in the distribution of outcome values were examined using the mean scores for summative assessments over a period of 3 years up to 2005 when the new formative assessments were introduced. The pattern shows an increase in mean scores for all assessments relative to both 2003 and 2004, indicating a potential positive impact on student outcomes. The data also showed an increase in the skew of the distribution of assessment grades towards high grades compared with the previous 2 years. There was a rise in the top end performance at A and B grades and a fall in the failure rates in the final assessment (see evaluation document).
The evaluation of formal student feedback showed improvements in most indicators. The feedback results had been good for most factors in prior years but evaluation scores increased compared to the previous year. A higher proportion of students were satisfied with teaching and learning, module workload and the development of their interest in the subject. Informal feedback from students suggested they were very keen to use this type of learning support. They liked being able to access computer-based materials and to test themselves whenever they liked. They liked the feedback within the learning and formative assessment and the clear links to other materials in the module and assessment. Many students said it helped them to gain the confidence to answer questions in tutorials and hence the confidence to attend tutorials in the first semester of their first year.
We found no evidence of a reduction in attendance. There was no evidence to suggest that fewer students bought or accessed an appropriate text book as a result of using formative e-materials. The drop out rate from the module declined relative to previous years, suggesting retention on the module had improved (see drop-out rates). The formative assessment system allowed us to identify weaker students at an early stage in the module and to force non-engaging students to discuss their approach to learning with a tutor. We found that the majority of students who were recalled to a clinic for lack of engagement started to use the computer-based materials as a result with a positive benefit in the first assessment.
The project greatly improved the expertise of the staff involved in its design and implementation and formed an important part of their personal development in e-learning. Our experience has also resulted in wider dissemination of the approach to our colleagues in the School, the University, and the wider academic community. It has resulted in stronger uptake of e-learning approaches within full-time modules within the school. We can identify 3 members of the business teaching team that took up computer-based assessment as a direct result of encouragement by the project team. We strongly believe that the formative materials aid student learning and provide an important source of support for first year first semester students who are unfamiliar with University learning environments. The materials also provide additional support for students whose first language is not English.
The project is part of the School's retention strategy and all the evidence suggests that it has had a positive impact. The project enabled the module staff to move towards a complete computer assessment mode for this module. When the module was introduced, the final assessment was a formal analytical examination using a real world case study. The development of the formative materials lead directly to a new computer-based final assessment. This cut the marking time considerably for the module staff. However, the additional time required to monitor and upgrade the materials have probably outweighed this reduction in marking time.
Did implementation of this e-learning approach have any disadvantages or drawbacks?
Although we applied for and received some funding to reduce our teaching hours in order to develop the materials, we hugely underestimated the amount of time required to create initial materials, test and pilot materials, and to resolve errors. In addition, we underestimated the time required to develop the data set for evaluation. In addition, this project would not have been possible without the expertise of the CIAD department. With this in mind, I do not think this type of development would be appropriate for courses with small numbers of students.
Looking back, we did not anticipate the need to continuously develop the material each year and the time cost that would involve on a regular basis. The original design involved the implementation of additional clinics for weaker students. This was funded by the retention strategy funding. The following year, the clinic system was dropped as the additional staff time was not funded.
How did this e-learning approach accord with or differ from any relevant departmental and/or institutional strategies?
The use of e-learning is quite widespread within the Business programmes as e-learning materials were developed several years ago for an e-learning mode delivery of our Business Studies Degree. Hence this project is consistent with the development of e-learning within the School. At the time of the project, however, the project team were 2 of a small number of staff who used computer-based assessment. What was entirely new in this project was the use of computer-based formative learning and assessment. However, the CIAD support staff are an indication of Derby University's commitment to developing computer-based and e-learning strategies and this project is an example of this. The materials now form the central component of the teaching and learning strategy for first year economics and data analysis within the business school.
Internal dissemination of the project results encouraged a wider group of staff within the business school to develop computer-based assessments and the experience of developing these formative materials provided good experience to support other staff. The module leader for this project also began the development of similar computer-based formative assessment and learning materials for a first year data analysis module within the School. This material has been run successfully using a similar model but is incomplete due to lack of time.
Lessons Learned
Summary and Reflection
The academic literature implies that formative assessments should follow good practice guidelines to be potentially effective. The design of the computer-based formative assessment tool in this project were designed in the light of good practice. The literature also suggests that formative assessment should develop self-regulated learning and be evaluated within the context of the student and the learning environment. This project evaluated the value of formative assessment in developing self-regulated learning against conflicting factors using an Activity model. Thefull analysis can be read in the draft paper on the analysis. The results add to the published evidence base and support the conclusion that computer-based formative learning positively affects achievement rates through self-regulated learning. The analysis suggests the process by which formative assessment affects achievement may differ between computer-based coursework and final written examinations. Improving achievement in examinations may depend on the level of entry self-regulated learning skills and the speed with which these skills can be improved through formative assessment. Improving achievement in computer-based coursework may also be affected by entry-level skills and early acquisition but the evidence suggests continuous development of self-regulated learning through formative assessment can also have a positive impact. The results also suggest that the impact of formative learning can be reduced by individual student characteristics such as a stated need for help with English as a second language. Self-reported study skills were not found to be significant. This does not imply that the level of study skills is irrelevant. Our measure was probably not picking up a difference in actual skills but differences in student perceptions of the level of skills appropriate for university study. The analysis on change in skills suggests that these perceptions may change radically after one semester of study as students upgrade their understanding of skills relevant to university study. Overall, we believe that our formative e-learning approach benefits the majority of students and generates strong positive feedback from students.
Our approach fitted well within the School's strategy but the developmental time constraints may limit its applicability for modules with smaller numbers of students. The experience has taught us that most students are confident about using e-learning materials and will find time to use them if they can see a direct and obvious benefit. Getting students to discuss these benefits in tutorials acts as a good method of promoting and disseminating the information to other students through social networks. Students often discussed how they were progressing in the formative assessment with each other and treated the experience as a social activity. The approach also provided a key method of identifying weak students at an early stage in the module when preventive action could be taken with positive results.
Practice has already changed as a result of the project. A similar set of materials for the first year data analysis module has been started but is not yet complete. However, the first set of materials is available to students and is used to prepare them for a short early computer-aided summative assessment and to identify weaker students who are not engaging in the module. Even at this stage of development, the materials generate student benefits. The CIAD team are developing an approach to provide the materials via the University website for remote access. When this happens, we will attempt to measure the impact on attendance and usage.
Further Evidence
'The aim was to reduce failure rates amongst full-time students through improving self-regulating learning through computer-based formative private study.'
'Overall, we believe that our formative e-learning approach benefits the majority of students and generates strong positive feedback from students.'
