Chapter 2

Item Response Theory (IRT)

Imagine that you’re teaching a fourth grade math class. You’ve just administered a test with 10 questions. Of those 10 questions, two questions are very simple, two are incredibly hard, and the rest are of medium difficulty. Now imagine that two of your students take this test. Both answer nine of the 10 questions correctly. The first student answers an easy question incorrectly, while the second answers a hard question incorrectly. Which student has demonstrated greater mastery of the material?

Under a traditional grading approach, you would assign both students a score of 90 out of 100, grant both of them an A, and move on to the next test. But this approach illustrates a key problem with measuring student ability via testing instruments: test questions do not have uniform characteristics. So how can we measure student ability while accounting for differences in questions?

IRT models student ability using question-level performance instead of aggregate-test-level performance. Instead of assuming all questions contribute equivalently to our understanding of a student’s abilities, IRT provides a more nuanced view on the information each question provides about a student. It is founded on the premise that the probability of a correct response to a test question is a mathematical function of parameters such as a person’s latent traits or abilities and item characteristics (such as difficulty, “guessability,” and specificity to topic).

Figure B shows two item response function curves generated by an IRT model. The curves illustrate how an IRT model relates a student’s ability with the probability of answering a question correctly, given that question’s difficulty, discrimination levels, and “guessability.” While IRT models are atemporal and reliant upon a single measure of ability (and thus reflect only one facet of the science behind Knewton recommendations), they help us better understand how a student’s performance on tests relates to his ability.

Probabilistic Graphical Models (PGMs)

This framework, which encompasses statistical methods such as Bayesian networks and Markov random fields, allows data scientists to code and manipulate probability distributions over multi-dimensional spaces in which there are hundreds or even thousands of variables at play. In other words, PGMs allow Knewton analysts to build complex models one effect at a time, relating the many learning activities they observe to estimations that are useful for recommendation.

One of the ways in which Knewton applies PGMs is by using a student’s known proficiencies to determine which other topics he may be ready to master. For instance, such a model might help the platform discover to what degree a mastery of fractions helps students master decimals and to what degree a mastery of decimals helps students master exponentiation. Knewton data scientists can thus determine the relationship between mastery of fractions and mastery of exponentiation. Ultimately, the discovery of these types of relationships allows the Knewton Adaptive Learning Platform to continually refine its recommendations.  

Hierarchical agglomerative clustering

In data mining, hierarchical clustering is a method of analysis that aims to construct a hierarchy or structure of clusters. At Knewton, the technique is used to detect latent structures within large groups and build algorithms that determine how students should be grouped and what features they should be grouped by. An implementation of this technique is incorporated in Knewton Math Readiness, a web-based developmental math course, which provides a dashboard that allows teachers to group students according to their level of concept mastery of the material they are working on.