The Knewton Blog

Our monthly newsletter features edtech and product updates, with a healthy dose of fun Knerd news.

Good Students Do Their Homework

Posted in Adaptive Learning on September 13, 2016 by

As the school year begins again, so do the homework debates: Do students have too much homework or too little? How young is too young? Is homework making a difference? Or would kids be better off playing outside and going to bed early, as one Texas teacher told her second-grade class?

Teachers, parents, and students have strong and sometimes contradictory opinions about the value of homework, but answers to these and other questions are not easy to come by. A lot depends on contextual information about individual students, their teachers, and the subject matter, and this information is difficult to gather and analyze.

We do know, however, that some students are using Knewton-powered learning applications to do homework, and the anonymized data that Knewton collects can shed light on its impact on student performance.

More than 100,000 elementary school students using one of our partner applications gave about 18.5 million answers to math questions over the course of the 2015-16 school year. What does this wealth of data tells us about doing homework?

It turns out that good students do, in fact, do their homework.

Math problems answered during the school day — between 8 a.m. and 3 p.m. — are presumably done in class, while answers submitted between 3 p.m. and 9 p.m. are most likely homework.

The graph below shows when during the day students did their math problems over the course of the 2015-2016 school year:

knewton-math-homework-time-of-day

Most of the work in this Knewton-powered partner application happened in school. About one-sixth of the work, however — about 3 million answers to math problems — got done as homework.

When these students did homework, they answered math problems correctly more often than they did at school. Before 3 p.m., about 65% of answers were correct. Between 3 p.m. and 9 p.m., more than 80% of answers are correct. You can see how their performance improves after school:

knewton-math-homework-time-correct-responses

(The y-axis begins at 50, not zero.)

What’s going on here? There are a couple of possible explanations:

  1. Doing work outside of school helped the students get more questions right than they did at school. Maybe they had more time to think, or felt less pressure, or had to contend with fewer distractions, or got help from parents, older siblings, or the Internet.
  2. The other possibility is that stronger students did more homework than lower performing students did. Since the percentage of correct responses is an average of the efforts of 100,000 students, the diligence of the strongest students could lift the entire group’s performance above 80%.If this second possibility is true, a seemingly impressive statistical gain masks the fact that, for many of these students, homework isn’t making a difference. It’s a lot like when you have a group of five people and one of them gets $100: The group’s average wealth goes up $20, but four of them don’t see any benefit.

So which is it: that students do better when they do homework, or that hard-working whiz kids are making everyone look better?

In an attempt to find out which possibility is more likely, we sorted the 100,000 students into five groups of equal size, based on how often they answered correctly — kind of like a teacher grading on a curve.

knewton-math-homework-quintile-groups

The average student in the highest performing group (in orange, let’s call it Group A) answered 94% of math questions correctly. The students in the lowest performing group (light blue, Group E) gave right answers only 24% of the time.

How many math problems were each of these five groups doing, and was it school work or homework?

knewton-math-homework-tim-correct-quintiles

At school, there is a peculiar relationship between practice and proficiency. The lowest-scoring group of students, Group E, does the least amount of work. Group D, which performs better, does more work than Group E. Groups B and C, which perform better still, do the most work at school.

However, the strongest students, Group A, are barely working in school more than Group E, the lowest-scoring group.

The data can’t explain why this relationship exists, but it’s easy to imagine the classrooms they describe. Math teachers know that different concepts will come naturally to some children, while for other students greater effort is required. Some students succeed without trying very hard, others slack off, and still others struggle despite diligent practice.

When it comes to homework, however, we clearly see that the higher a group’s score was, the more homework they did. The result is striking: The five groups are in alphabetical order with Group A, the top performers, doing the most homework, and Group E doing the least. This suggests that stronger students simply do more homework than their peers.

How does each group perform over the course of the day?

knewton-math-homework-time-correct-quintiles

The performance of each group improves during homework time, but these increases are too slight to suggest that homework makes students perform better. And the differences between groups are bigger than differences within each group. Scores for Group E fell off around 7 p.m.: Maybe they were in an afterschool program until their parents picked them up after work? This drop is statistically significant, but you can see that the shaded areas, showing the margin of error, get bigger as fewer students work on math.

We still don’t know whether the stronger students are performing better because they did their homework, or that they did their homework because they’re stronger students. So this finding will not resolve the debates over which approaches to homework are productive and beneficial, or whether it all amounts to busywork.

Still, aggregated student data does contain valuable lessons about how students learn. We’ve seen that student performance after school is much higher because well-performing students tend to do their homework, not because students tend to perform better after school hours.

So your teachers are correct when they say that good students do their homework.

Ruben Naeff is a data scientist at Knewton.