Designing and teaching a graduate course from scratch

The assignment

CG 501, Computation and Cognition, is a core course for MSc cognitive science students at IIT Gandhinagar. I taught the second half of it in Spring 2019 as a Graduate Teaching Fellow, alongside Prof. Krishna Prasad Miyapuram, who taught the first half and who took a liberal view of what I was allowed to put in mine.

The stated purpose of the course is to introduce cognitive science students to computational tools for inspecting, modelling and exploring the behaviour of complex systems, from neurons to organisms. The students were not computer scientists. Most had undergraduate backgrounds in psychology, biology or engineering, and for several of them this was the first course requiring them to write code that had to work.

I ran 17 classes: 11 lectures, 2 quizzes, and 4 days of project presentations.

What was taught

Machine learning, six lectures. Vectors and matrices as functions, then the perceptron; multilayer perceptrons and backpropagation, with a scikit-learn demo on the iris dataset; model evaluation and Hopfield networks; linear regression and K-means clustering; k-nearest neighbours; NumPy and a from-scratch kNN implementation.

Game theory, five lectures. This module was not in the syllabus and I added it. The prisoner's dilemma and its paradox, with three separate framings — the canonical story, the hawk-dove money version, and Cournot duopoly — to make the point that the matrix is the object and the story is interchangeable. Then dominance and iterated elimination, best response and Nash equilibrium, coordination games with multiple equilibria, mixed strategies and the indifference principle, extensive-form games and backwards induction. Equilibria were demonstrated live in Gambit as well as on the whiteboard.

The linear algebra was taught instrumentally: a matrix is a function that takes a vector and returns a vector, and the exercises asked students to construct the matrix producing a given transformation. That framing came back on both the quiz and the final exam, which is the kind of repetition I used deliberately.

How it was assessed

Assignments. Two, both requiring working code. The first asked students to implement a perceptron in NumPy and train one for each of AND, OR, NAND and NOR, printing weights before and after learning, with XOR left as the following week's lecture. The second was a fill-in-the-blanks Jupyter notebook distributed through a public repository — K-means clustering or linear regression, with every ?? carrying a nearby comment stating the goal, plus questions at the end to be answered in a markdown cell. Submissions had to be a restart-and-run-all notebook named by roll number.

The design intent was to require students to combine what was discussed in class with information they had to find themselves. My own account at the time: this hands-on approach was aimed at the inertia that makes programming feel alien to first-time users. Whether it worked is the one thing the evaluation data is most critical about, and I discuss that below.

Quizzes. Two, each around ten points. The machine learning quiz asked what a single partial derivative of the cost function represents in plain language, how to decide at run time that K-means has converged, a perceptron forward pass followed by "now find a w such that the output is zero for the same input", matrix construction, and a confusion matrix for a degenerate classifier. The game theory quiz asked for a pure-strategy equilibrium, backwards induction on a game tree, and two modelling questions with no matrix supplied.

Course project. One per student, signed up from a list of twenty topics across four tools. NetLogo agent-based models from the standard library — ants, mammoths, virus on a network, traffic grid, signalling game — assessed on parameter exploration, with students required to state what they expected before changing a parameter and explain any disagreement with the observation, and bonus credit for modifying the model code to test their own idea. scikit-learn projects on Parkinson's disease classification, keystroke biometrics, and regression on Open Psychometrics personality datasets, each requiring a stated minimum set of reported metrics. A NumPy project computing equal error rate from ROC curve points. Gambit projects on nuclear first-strike policy, transition to a circular economy, an attendance game, and a voting-and-friendship game under secret versus open ballot.

Deliverables were a ten-minute presentation and a report, with source notebooks or captioned screenshots depending on the tool.

End-semester exam. Ten questions, answer eight, equal weight, two and a half hours. Roughly balanced between the two modules, with the modelling questions carrying the same weight as the mechanical ones.

The evaluation data

The academic office ran the standard course reaction survey. Ten students responded to twenty rated items on a five-point scale plus five free-text questions. I have the item-level responses, transcribed into a machine-readable form alongside the codebook.

The summary, stated plainly:

• Grand mean across 198 numeric ratings: 2.80 on a 0–4 scale.
• Overall instructor rating: 2.30. Distribution: two 1s, three 2s, five 3s, zero 4s.
• Administrative items (policies announced early, lectures held, materials available, course website): mean 3.55.
• Delivery items (lectures useful, fundamentals emphasised, board work, organisation, visual aids, questions answered, pace): mean 2.53.
• Lowest items: questions answered in class 2.20, pace even 2.20, grading consistent 2.20, fundamentals emphasised 2.30, assignments helped understanding 2.30.
• Five of ten respondents named neural networks as taught too fast. Nobody named anything as too slow.
• Two respondents named game theory as what they liked — the added module, and the only unprompted enthusiasm in the free text.

The most useful structure in the data is the roughly one-point gap between what I scored on organisation and what I scored on instruction, which holds within individual respondents including the two harshest evaluators. I have written that analysis up separately in what the course survey measured.

What the data says against my own account

Three months after the course ended I wrote a teaching statement for an institute teaching award nomination. It describes my approach as deliberately breadth-first — cover a variety of topics, whet the appetite, give students enough to explore further on their own. It also, four paragraphs later, concedes that managing research alongside teaching meant some topics went too fast.

The evaluation data makes those the same sentence. Eleven lectures absorbing both a machine learning course and a game theory course produced exactly the pace complaint that five students independently reported, localised to the block I prepared under the most time pressure. The statement presents the breadth as a pedagogical choice and the pace as a scheduling accident; they were one decision. That reading is developed in breadth-first in the classroom.

The other thing the data says is that my assignments did not do what I intended. "Assignments helped better understand the course material" scored 2.30, with two students rating it 0, and two of the free-text improvement suggestions asked for fewer evaluations and fewer coding assignments. My statement at the time recorded that I had been told I was too demanding with the assignments. I recorded it as feedback received rather than as a conclusion, and the numbers suggest it deserved to be a conclusion.

Grading consistency, at 2.20 with two zeros and one student writing "I don't agree with the grading", is the item I can least defend. I set open modelling questions, which I still believe were the right questions, and I graded every script alone without a rubric written in advance and without a second marker. That combination generates exactly this complaint and I have no counter-evidence.

Artefacts

Lecture slide decks, both quizzes and the end-semester exam, the project brief, the demonstration notebooks, the day-wise schedule, and the transcribed evaluation data with its codebook. The assignment notebooks were distributed through a public repository during the course.

What I would change

Cut one of the two fields and teach the remaining one slowly enough that the fundamentals item clears 3.0. Write the grading rubric before the assessment rather than during the marking. State the capability a student should have at the end, in one sentence, before writing the first lecture — the syllabus I actually wrote was a list of topics, and a list of topics has no stopping condition.

Keep the game theory module, keep the fictional modelling scenarios, and keep the degenerate-classifier question that appeared on both quizzes and the final.