My students rated my logistics higher than my teaching

The dataset

In the second half of Spring 2019 I was the Graduate Teaching Fellow for CG 501, Computation and Cognition, a core course for MSc cognitive science students at IIT Gandhinagar. I ran 17 classes: 11 lectures, 2 quizzes, and 4 days of project presentations. I wrote the questions for two assignments, two quizzes, the course project and the end-semester exam, and graded all of them.

At the end, the academic office ran the standard course reaction survey. Ten students responded. Twenty rated items on a five-point scale — 0 "not at all", 1 "below average", 2 "average", 3 "above average", 4 "very much so" — plus five free-text boxes.

Ten respondents is a small sample and I will come back to what that does and does not permit. But I have the item-level data, which is more than most people get back from a teaching evaluation, and the item-level structure turns out to be where the information is.

The number that says nothing

Grand mean across all 198 numeric ratings: 2.80. Overall instructor rating (item 20, "overall the instructor taught well"): 2.30, with a distribution of two 1s, three 2s, five 3s and zero 4s.

That is a slightly-above-"average" course taught by an instructor rated slightly below "above average". As a summary it is almost perfectly uninformative. It is also the only number that typically survives into an administrative record, which is my first complaint about instruments of this kind and not the interesting one.

The number that says something

Group the twenty items by what they actually measure, and the flat aggregate splits into two populations:

The four highest-scoring items in the whole survey are: course policies announced sufficiently early (3.70), text and reference material easily available (3.60), the course website was useful (3.50), and all allotted lectures were held (3.40). Every one of those is a logistics item. Not one of them requires me to have understood anything about teaching.

The lowest-scoring items are: questions raised in class satisfactorily answered (2.20), pace of instruction was even (2.20), grading was consistent (2.20), fundamental principles well emphasised (2.30), assignments helped understanding (2.30). Every one of those is a teaching item.

A 1.0-point gap on a five-point scale between "I was organised" and "I taught well", from the same ten people on the same form on the same day.

Why I believe the gap and not much else

Ten respondents, and a between-item comparison drawn from them. The obvious objection is that with n=10 almost nothing is distinguishable from noise, and for any single item that is correct — the standard error on a mean of ten ordinal ratings is large enough that I would not defend a claim about any one number.

The reason I trust the gap more than I trust its endpoints is that it is a within-respondent contrast. Each of the ten people rated both the logistics items and the teaching items. Whatever makes a respondent generally generous or generally harsh — and respondents 9 and 10 differ by nearly two full points in personal mean, 1.90 against 3.80 — applies to both halves of their own form and subtracts out of the comparison. The design is paired, and paired contrasts are exactly where small samples retain power.

Look at it respondent by respondent and the pattern is not carried by one or two people. Respondent 5, who rated me 1 on overall teaching, gave 4s to policies, lectures held, and reference material. Respondent 9, the harshest overall at a personal mean of 1.90, gave 4 to policies announced early and 3s to material availability and the course website. Even the two most negative evaluators separated my administration from my instruction, and rated the former highly.

That is what makes me believe it. Not the size of the gap — I would not defend "1.0 points" as a quantity — but its consistency across respondents who agreed on nothing else.

The opinion

Course evaluations reliably measure whether an instructor was organised, and measure teaching quality only weakly — which makes the single summary score they produce a good instrument pointed at the wrong construct.

I want to be careful here, because there is a self-serving version of this argument and it is not the one I am making. The self-serving version says student evaluations are unreliable, therefore my 2.30 on overall instruction means nothing. I do not believe that. The 2.30 is consistent with the free-text comments, consistent with the pace item, and consistent with my own account written before I saw the numbers. It means what it says.

The claim is narrower and, I think, more uncomfortable: the instrument has good discriminating power, and most of that power lands on the dimension that is easiest to fake. Announcing the grading policy in week one is a calendar operation. It costs an hour. It is fully visible to students, unambiguous to rate, and it is what the four highest-scoring items on this form are measuring. Explaining backpropagation at a pace that ten people with different backgrounds can all follow is the hard part of the job, is what the low-scoring items are measuring, and is where the noise, the disagreement and the genuine difficulty all live.

An instructor optimising against this instrument would correctly conclude that the cheapest available point comes from logistics. I do not think that is a scandal — being organised is a real service to students and I am glad it is measured. But the aggregate presents the two as interchangeable contributions to one number, and they are not remotely the same kind of achievement.

What the free text added

Five of ten respondents named neural networks as taught too fast — "neural networks", "Neural networks", "some of the neural network topics were covered too fast", "Perceptron and Hopefield network", "Linear Regression kNN". Nobody named anything as too slow; two explicitly said nothing was.

That is the most actionable line in the dataset and it is unanimous in direction. The pace item scored 2.20 with six of ten respondents putting it at exactly "average" and none at "very much so", which reads less as a complaint than as a diffuse sense that something was off — the free text is what localises it. The neural network block was six lectures compressed into the first half of my allocation, and I was writing them while running my own experiments. The students felt precisely where that happened.

Two respondents named game theory as what they liked ("game theory", "Game theory was fun"), which was the module I built from nothing and taught second, with more preparation time behind it. One asked for more board work over slides — "Teaching is better on board than PPT" — which I read as an artefact of the same thing: the neural network lectures leaned on slides I had prepared in a hurry, and the game theory lectures were largely me at a whiteboard drawing payoff matrices.

The two dissenting notes are worth recording because they cut against the narrative. One respondent wanted more neural networks and less general machine learning, on the entirely fair grounds that the published course outline had promised it. One rated grading consistency at 0 and wrote "I don't agree with the grading" — grading consistency scored 2.20 with two zeros, and that is the item I have the least ability to adjudicate, since I graded everything alone and had no second marker to check myself against.

What I cannot tell you

Whether any of this generalises. One course, one instructor, ten respondents, one institution, one semester. The pattern I have described — logistics high, delivery low, gap stable within respondents — is a real feature of this dataset, and I have no evidence at all that it is a real feature of course evaluations in general. There is a substantial literature on student evaluation validity that I had not read at the time and have only skimmed since; it is contested in both directions and I am not going to pretend to summarise it.

There is also a fact about response rate I cannot resolve. The survey went to a class in which I do not know how many students were registered, and I have ten forms. If the non-respondents differed systematically from the respondents, every number above moves and I have no way to check.

What I would say to anyone reading their own evaluation data: do not read the aggregate. Split the items by what they demand of you, and read the gap. Mine told me I had run a well-administered course and taught an uneven one, which was true, and which the 2.80 grand mean had entirely concealed.