Affordances offered by e-assessment tools > Capabilities of e-assessment

Question 36
To what extent do existing e-assessments provide reliable measures of mathematical understanding, as might otherwise be measured by traditional exams?

• Peter Rowlett

Some express doubts that e-assessment is reliable in this way, and many use e-assessment for quite low-stakes assessment as a result.

What motivates this question?

Writing reliable and valid questions for e-assessment is a difficult task requiring specific expertise. Additionally, there are limitations to the e-assessment format, potentially greater focus on procedural elements, etc. The examination is seen as the ‘gold standard’ for mathematics undergraduate assessment, so it may be useful to investigate whether performance in one correlates with performance in the other.

Broughton et al. (2017) report lecturers believing their e-assessment system was “unreliable as a measure of student performance, knowledge and understanding” and that they “could not rely on the marks that the system offered”. Greenhow (2002) observes that “repeatable-on-demand online objective tests do not appear to rank students correctly” (p. 15).

What might an answer look like?

A study comparing e-assessment marks or rankings with marks or rankings in other assessment methods (on comparable topics) would be useful here. If these only correlate sometimes, this raises difficult questions about which is providing the more accurate measure of mathematical understanding. Say we assume exams are a good tool for assessment and find that e-assessment sometimes doesn’t correlate well with this. Can we then establish in what circumstances taking formal marks from e-assessment is appropriate?

Related questions

• This question is closely related to Q27: How can formative e-assessments improve students’ performance in later assessments?, which refers to performance in exams and this question refers to measuring mathematical understanding via exams.
• This is related to Q29: What are suitable roles for e-assessment in formative and summative assessment?.
• This is related to Q36: To what extent do existing e-assessments provide reliable measures of mathematical understanding, as might otherwise be measured by traditional exams?
• Q24: To what extent does the randomisation of question parameters, which makes sharing answers between students difficult, adequately address plagiarism?, while on a different subject, may also involve comparison of results between e-assessment and other assessment methods.

References

Broughton, S.J., Hernandez-Martinez, P. & Robinson, C.L. (2017). The effectiveness of computer-aided assessment for purposes of a mathematical sciences lecturer. In M. Ramirez-Montoya (Ed.), Handbook of Research on Driving STEM Learning with Educational Technologies (pp. 427-443). Hershey, PA: IGI Global.

Greenhow, M. (2002). Answer Files — What more do they reveal? Maths-CAA Series, January 2002. Retrieved from http://icse.xyz/mathstore/node/61.html