Affordances offered by e-assessment tools > Free-form student input

Question 41
Are there differences in performance on mathematics problems presented on paper versus as e-assessments?

Does the different submission medium of computer based questions and paper based questions affect students performance?

What motivates this question?

Previous research has found systematic differences in performance between paper-based and online versions of the same test (Backes and Cowan, 2019).

There are two main differences between computer-based and paper-based mathematical assessments which could affect the final mark awarded to a student.

1. The medium used to submit answers.

Does the need to enter answers into a computer affect the correctness of that answer?

Possible issues that could affect this include:

The need to enter the answer using some computer syntax or equation editor, which might lead to errors (Sangwin & Ramsden, 2007)
Whether students carry out working on paper before entering their answer into a computer or not. Any lack of working may result in slips due to mental-only processing of the problem, whereas students submitting on paper may be more inclinded to produce more complete paper workings whilst solving a problem.
Opportunities to separate out the process of decision making, e.g expand out this term, from the ability to perform some calculation accurately.

For work carried out on paper, handwriting is a relevant factor. The early studies by James (1927) and Markham (1976) found that quality of handwriting significantly influenced grades given to essays. Many researchers have consistently studied the association of poor handwriting with lower marks from graders (e.g., Chase, 1986; Markham, 1976; Chase, 1968; Soloff, 1973).

2. The method used for marking.

Computer marking may be more consistent than human marking, since human markers may miss small errors in one submission when marking a complete set of submissions, yet human markers are more able to use their intuition to (rightly or wrongly) award marks where the believe they know what a student meant to say, even if they did not fully express it. Automated marking is also in principle objective, fair and reliable, whereas doubts have been expressed about the consistency and quality of human markers (Foster, 2007). However, an e-assessment system should be monitored because systematic marking errors may take place (Ferrão, 2010).

What might an answer look like?

An experiment could investigate the effect of the medium on the students’ answers. For instance, students could be divided into “writing” and “typing online” groups and asked to solve the same problems. This would enable comparison of the performance between the two groups, both in terms of the correctness of the answers, and the features (such as length and quality of argument) of the answers.

This question is intimately related to:
- Q42: How can we automate the assessment of work traditionally done using paper and pen? and
- Q47: How can we assess open-ended tasks using e-assessment?
Differences in performance could be due to a number of related factors beyond simply “paper versus computer”:
- the approach used to mark the students’ work, related to Q43: How can we emulate human marking of students’ working such as follow-on marking and partially correct marking?
- different types of errors students may make, as in Q2: Do the errors students make in e-assessments differ from those they make in paper-based assessments?
- the input mechanisms available in the e-assessment system, as in Q39: What methods are available for student input of mathematics?

References

Backes, B., & Cowan, J. (2019). Is the pen mightier than the keyboard? The effect of online testing on measured student achievement. Economics of Education Review, 68, 89-103.

Butcher and Jordan: A comparsion of human and computer marking of short free-text student reponses: https://doi.org/10.1016/j.compedu.2010.02.012

Chase, C. (1968). The impact of some obvious variables on essay test scores. Journal of Educational Measurement, 5:315–318.

Chase, C. (1986). Essay test scoring: Interaction of relevant variables. Journal of Educational Measurement, 23:33–41.

Ferrão, M. (2010). E-assessment within the Bologna paradigm: evidence from Portugal. Assessment & Evaluation in Higher Education, 35(7), 819-830. https://doi.org/10.1080/02602930903060990

Foster, B. (2007). Using computer based assessment in first year mathematics and statistics degree courses at Newcastle University. MSOR Connections, 7(3), 41-45.

James, A. (1927). The effect of handwriting on grading.English Journal, 16:180–205.

Markham, L. (1976). Influences of handwriting quality on teacher evaluation of written work. American Educational Research Journal, 13:277–283.

Sangwin, C. J., & Ramsden, P. (2007). Linear syntax for communicating elementary mathematics. Journal of Symbolic Computation, 42(9), 920-934. https://doi.org/10.1016/j.jsc.2007.07.002

Soloff, S. (1973). Effect of non-content factors on the grading of essays. Graduate Research in Education and Related Disciplines, 6:44–54.

Question 41Are there differences in performance on mathematics problems presented on paper versus as e-assessments?