One of the things we’re interested in at Knewton is how long a student spends working. How long does it take to solve a practice question? How far do students make it into an instructional video?
Measuring how much time each student spends on each item allows us to help teachers understand better how their students are working and whether they are engaged by the material.
When students are disengaged, their interactions with Knewton can sometimes reflect that. For example, when students move through coursework much faster than they usually do but without better performance, they might not be paying much attention, and are just clicking to get to the next thing. We call such behavior “spamming.”
Everything is relative: If a student is generally a fast worker, a fast response doesn’t count as spamming. Similarly, we take into account how long we expect each item to keep a student’s attention. Spending 28 seconds watching a 30-second video is generally considered working, while 28 seconds on a 5-minute video might suggest something else.
Once Knewton’s algorithm establishes a baseline expectation for each student and each piece of content, we can look at how long each interaction took and, by way of illustration, assign it a working-or-spamming probability.
Since Knewton is integrated into different learning applications aimed at different age groups, we wanted to see whether spamming rates vary by grade level, but users working on elementary school materials (most of them, presumably, elementary school students) and users of college coursework answered unexpectedly quickly at about the same rate. We also looked at whether students were more likely to spam on certain days of the week, but we didn’t see a Friday effect with spamming like we do with performance.
In addition, we examined whether different types of questions affected how likely students were to spam a particular question. The Knewton open platform, for example has some questions that are multiple choice questions and others that require students to type in an answer. Working interactions were almost evenly split between free response and multiple choice, but spamming occurred disproportionately on free response questions. It turns out that students were roughly three times as likely to spam on free-response questions.
Understanding this kind of spamming behavior augments our sense of what keeps each individual student engaged and, more broadly, contributes to our understanding of how students interact with learning applications. Knowing whether students are working productively enables Knewton and its partners to make better applications that help students remain in a learning flow and help spamming students get back on track.