At Knewton, we work very closely with publishers’ editorial teams in the course of integrating their products with our technology.
One thing we’ve heard repeatedly in our conversations with publishers is a desire for a more quantitative means of evaluating how well their particular textbook content is performing for students. Publishers can and do solicit qualitative feedback from students and teachers, but this takes a lot of time and can be difficult to gather at scale. While they can also track observed metrics like student logins or usage times, this surface-level information only goes so far.
We’ve found that publishers really want to understand how well content actually contributes to student learning and helps improve achievement. They want to use this information to build better products, and improve existing ones. Knewton has built the world’s first and only web-based engine to automatically extract statistics comparing the relative quality of content items — enabling us to infer more information about student proficiency and content performance than ever before possible.
With Knewton Content Insights, this information is now available at a glance for publishers and other content creators.
Content Insights: Sample Metrics
Content Insights help publishers improve their content’s quality, quantity, and organization; make informed investments in their development cycle; and better support students using learning products. Here are a few examples of the information a Knewton Content Insights dashboard provides.
How well do individual questions assess a student’s understanding of a topic?
Questions with high assessment quality do a good job predicting students’ answers on other related questions. For these questions, knowing whether a student got the question right reveals a lot about how well she understands the given concept overall.
This information helps publishers figure out which questions to keep, and which to revise or replace — and ultimately improve learning experiences for students.
Where do students run out of content?
Exhaustion looks at how much content students are using in a particular area. For example, if students tend to use all the material in a chapter on cell division, publishers can focus on creating additional questions or instructional content for this topic. This helps publishers use their content development resources more efficiently and make sure students have the materials they need to succeed.
What is the relative difficulty of a given question?
Content creators can see how hard each question is for students, relative to other questions intended to test similar material. Difficulty is calculated by analyzing how often a question is answered incorrectly and how many tries it takes each student to answer correctly, along with other factors. The difficulty metric helps content creators better target their content’s level to students’ needs.
Publishers can also look at difficulty as it relates to exhaustion. If they see that students are using all the content in a certain area, and the topic is particularly hard, they will probably want to prioritize adding more material in that area.
Building Better Products
With digital learning comes a huge improvement in product transparency. Digital learning empowers students, teachers, and schools to see exactly how much a given product impacts learning outcomes. Content Insights provide publishers with actionable data to build more effective products, and enable them to get ahead in the market by basing their sales efforts on proven outcomes.
Knewton Content Insights are currently live with several beta partners. Soon, they will be available to all publishers in the course of a normal Knewton integration. Feedback from current and prospective customers has been overwhelmingly positive.
Partners are using different metrics depending on their needs. One beta partner building K-12 products is particularly focused on content exhaustion — they’re using the metric to drive their content creation efforts, prioritizing topic areas where students tend to run out of content most quickly.
A partner in the ELT space is particularly excited about seeing data about the relative difficulty of their assessment items and groups of items — e.g., do students generally find the present simple tense easier to understand than the present perfect tense? They plan to use Content Insights to rewrite items that prove much harder or easier than other items that assess the same concepts, to provide a more consistent experience for all students.
We’re excited to see how future partners apply Content Insights to improve their editorial processes. Knewton will continue to expand the education insights we offer to partners. Ultimately, our goal is to help the world build better products that drive bigger learning gains for students.
Interested in using Content Insights for your digital product? If you’re a current Knewton partner, talk to your partnership manager. Otherwise, get in touch here.