A Message from Wiley

Greetings from Wiley!

I want to personally express my excitement for Wiley’s acquisition of Knewton, which I will be leading.

Let me reassure you that we are integrating Knewton into our Wiley education business and remain 100% committed to you and your students. We will continue  to support Knewton’s Alta and enterprise solutions fully. In fact, we plan to expand the Alta catalog to support more courses, and continue to invest in the platform as well.

It’s important you hear this directly from me – your prices will not change. You will also continue to receive the excellent support you’ve grown accustomed to from Knewton, with additional support from our Wiley team!

Why are we so excited about the Knewton acquisition?

• We believe in Knewton’s mission to improve learning outcomes and access through affordable learning solutions. Knewton’s mission aligns with Wiley’s beautifully, as we are committed to creating and delivering the highest-quality content and courseware that leads to better skill mastery and learning outcomes.
• Alta enables us to continue to expand our affordable learning solutions portfolio. We’ve already been making progress on this front via inclusive access and print rental programs. In fact, it is a personal mission of mine to expand and extend our affordability initiatives. Our acquisition of Knewton will help accelerate those efforts.
• We are excited by Knewton’s technology, capabilities, and talent. We know we are acquiring world-class technology and will be welcoming incredible Knewton talent into the Wiley family with great excitement!

We thank you for your continued support and look forward to becoming part of the Knerd community.

Renée Altier
Vice President and General Manager, Digital Education
Wiley

Interpreting Knewton’s 2017 Student Mastery Results

This post was developed with Illya Bomash, Knewton’s Managing Data Scientist.

Results. Efficacy. Outcomes.

Student success is the ultimate goal of learning technology. Despite this, there exists a startling lack of credible data available to instructors and administrators that speaks to the impact of ed-tech on learning and academic performance.

To provide instructors and administrators with greater transparency into the effectiveness of alta and the Knewton adaptive technology that powers it, we analyzed the platform results of students using alta. These results represent our effort to validate our measure of mastery (more on that to come) and provide instructors and administrators with much-needed transparency regarding the impact of alta on student achievement.

Here, we hope to provide context and explanation that we hope will leave educators and those in the ed-tech community with a clearer picture of how we arrived at the these results — and why they matter.

Our data set

The findings in this report are drawn from the results of 11,586 students who cumulatively completed more than 130,000 assignments and 17,000 quizzes in alta in 2017.

This data set includes all of alta’s 2017 spring and summer student interactions. Only cases in which the relevant calculations are impossible have been excluded — such as quiz scores for a course in which the instructor chose not to administer quizzes. So while these results aren’t from randomized, controlled trials, they do paint an accurate portrait of student performance across alta users, making use of as much of our student data as possible.

Why mastery?

Our adaptive technology is based on the premise that if a student masters the concepts tied to the learning objectives of their course, that student will succeed in the course and be prepared to succeed in future courses. It’s also based on the premise that Knewton’s mathematical model of student knowledge states — which we frequently refer to as Knewton’s proficiency model — can determine when a student has reached mastery.

This basis in mastery manifests itself in how students experience alta: Every assignment that a student encounters in alta is tied to learning objectives that have been selected by the instructor for their course. A student “completes” an alta assignment when our proficiency model calculates that a student has mastered all of the learning objectives covered in that assignment.

Our 2017 Mastery Results seek to clarify two things: the frequency with which students achieve mastery in alta, and the later performance of students who have (and have not) achieved mastery, as determined by our proficiency model.

Controlling for students’ initial ability level

In this analysis, we wanted to assess the impact of mastery across the full spectrum of student ability levels. To capture a sense of each student’s initial proficiency, we aggregated the first two questions each student answered across all of the concepts he or she encountered in the course. The percentage of those questions the student answered correctly provides a naive but reasonable estimate of how well the student knew the material entering the course.

We looked at the distribution of this score across all of our students, tagging each student’s history with a label corresponding to where they fell among all users.

• Struggling: Students whose initial ability fell into the bottom 25% of our population
• Average: Students whose initial ability fell into the middle 50% of our population
• Advanced: Students whose initial ability fell into the top 25% of our population

Note: Knewton’s proficiency model neither uses this measure nor tags students with any kind of “ability label.” Our adaptive technology calculates a detailed, individualized portrait of each student’s proficiency levels across a wide range of concepts after each student interaction. But for the sake of this comparative impact analysis, we’ve chosen to use these distinctions as a tool to compare students of similar initial abilities.

Our findings

Students of all ability levels achieved mastery with alta at high rates

Analyzing students’ assignment completion revealed that with alta, students achieve mastery at high rates. As seen in Figure 1, across all students, 87% of the time, students working on an assignment in alta achieved mastery. Even among students who struggled to complete a particular assignment, 82% eventually reached mastery.

Achieving mastery with alta makes a positive impact on students’ academic performance

We know that with alta, students are highly likely to achieve mastery. But what is the impact of that mastery? When our model indicates that a student has mastered the material, how well does the student perform on future assignments, quizzes, and tests?

For any given level of initial ability, Knewton’s adaptive learning technology is designed to facilitate reaching mastery effectively for any student willing to put in the time and effort. To validate Knewton’s measure of mastery, we compared the performance of students who mastered prerequisite learning objectives (for adaptive assignments) and target learning objectives (for quizzes) through altawith students of similar initial ability who did not master these concepts.

Mastery improves the quiz scores for students of all ability levels

Figure 2 shows average Knewton quiz scores for students who did/did not reach mastery of the quiz learning objectives on prior adaptive assignments. Quiz takers who mastered at least ¾ of the quiz learning objectives through previous adaptive work went on to achieve substantially higher quiz scores than similarly-skilled peers mastering ¼ or fewer of the learning objectives.

Mastery levels the playing field for struggling students

Putting in the work to reach mastery on the relevant adaptive assignments increased initially struggling students’ average quiz scores by 38 percentage points, boosting scores for these students above the scores of otherwise advanced students who skipped the adaptive work.

Mastery improves future platform performance

Students who master the learning objectives on earlier assignments also tend to perform better on later, more advanced assignments.

Assignment completion

As Figure 3 shows, controlling for overall student skill levels, students who mastered ¾ of the learning objectives prerequisite to any given assignment tended to complete the assignment at much higher rates than students who did not. This is the virtuous cycle of mastery: the more students master, the better prepared they are for future learning.

Work to completion

Mastery of an assignment’s learning objectives also saves students time. When students began an assignment after having mastered most of its prerequisites, they tended to require significantly fewer questions to complete it. For students who mastered at least ¾ of the prerequisites to any given adaptive assignment, completing the assignment took 30-45% fewer questions than for students who did not (see Figure 3). Mastery helps students of all abilities learn faster, and struggling students see the biggest gains: for these students, prerequisite mastery leads to an average postrequisite assignment shortening by more than 40%.

The road ahead

Any self-reported efficacy results will be met with a certain amount of scrutiny. While we’ve attempted to be as transparent as we can be about our data, we understand that some will question the validity of our data or our approach to presenting it.

It’s our hope that, if nothing else, the reporting of our results will inspire others in the ed-tech community to present their own with the same spirit of transparency. In many ways, these results are intended not as a definitive end-point but as the start of a more productive conversation about the impact of technology on learning outcomes.

Lastly, while our 2017 Student Mastery Results are encouraging, we know that they exist in a world that is constantly changing. The challenges in higher education are becoming greater and more complex. The student population is growing increasingly diverse. Our technology and our approach to learning is evolving.

This year, we plan to update these numbers periodically and provide the results of other analyses with the goal of providing greater transparency into the effectiveness of alta and deeper insights into how students learn.

View full mastery results

Easy to use: The design of Knewton’s alta

The what, why, and how behind alta’s ease of use, and why focusing on user experience is a differentiator for Knewton.

The mission of the Knewton UX Team is to represent our user’s interests in the product experience. We do this in a few ways: 1) by listening to and observing our users to better know them and provide solutions to their problems, 2) by providing them a high-quality experience that is as delightful to use as it is effective, and 3) to differentiate our products from the competition. By nature, our processes integrate with every corner of the business as we attempt to design our workflows and experiences to achieve the best results. We want to eliminate bad design.

Better Design

“Bad” design can lead to enormous waste of time and resources and results in lost leads. Be it a technology stack, CRM flow or user experience, poor design choices can all lead to lost opportunity and leave your product at a competitive disadvantage. It might even lead to embarrassingly dangerous errors like, say mistakenly sending out an inbound nuclear missile warning to an entire state!

That’s why Knewton believes that the alta user experience and our business processes must be designed with ease of use and efficiency in mind. And, like many of you, I have observed in software product-focused organizations that these communications, support, management and metrics systems we duct-tape together and call a “business” are severely entropic — as new needs and goals emerge, new people and ideas cycle through the organization, adding to the complexity.

Staying focused on identifying and solving real user problems with a design thinking mindset will help give your product a competitive advantage.

Great product experiences are rarely born of a chaotic set of goals and business processes.  Knewton has made a real investment in user experience and research because we understand the advantage these capabilities bring, especially as a differentiator in education.

Only a focused student can effectively learn, and student performance insight is perhaps one of the most important components of effective teaching through software. Both will remain elusive without a carefully considered UX.

Simple Is Hard

No technology company has time to continually take a step back and reflect on their product experience, and then redesign each time new features are added. It’s easier and faster to bolt on features without considering how they impact the usability and perception of a product for users.

At Knewton, in creating alta we’ve asked ourselves, how can we take all these disparate problems and needs and boil them down into a simple product experience that will scale to accommodate our roadmap of the next 24–36 months, all while the plane is flying? How do we create a scalable, elegant design system that will help us be more efficient as a product team?

You can see how hard it is to deliver a quality UX simply by looking at our competitors and other entrants in EdTech. Most are afflicted with a bad case of featuritis.

By contrast, alta’s ease of use is a differentiator. Instructors and students can see that user experience is important to Knewton and naturally gravitate toward an experience that is content-forward, intuitive, calming, focused and responsive.

There are a few key ways we’ve designed the alta experience to build trust with educators and students.

Focus on the Fundamentals

Why will a higher education instructor choose alta over other options already in the marketplace? We start with the fundamental elements that create the conditions for ease of use.

Consistent Navigation and Context

Our users should always know where they are while using alta, and what they should do next. Most good UX designers will tell you that there should be one main purpose per screen, accessible with a clear call to action. In alta, we’ve reduced visual clutter, and replaced it with more structure to prioritize focus.

Done right, a user interface will essentially disappear for users — they won’t be thinking about how to use it, or spending precious time interpreting choices.

This includes a consistent, scalable navigation, which is critical infrastructure for the usability of any piece of software. Done right, a user interface will essentially disappear for users — they won’t be thinking about how to use it, or spending precious time interpreting choices.

But clear navigation is only part of a successful user experience. Since alta is a adaptive learning technology with assignments that can be of variable length, context is the key to a more relaxed, focused student.

We make sure a students options are always accessible and they know their current level of mastery, with persistent access to a prominent progress bar and mastery view. Similar enhancements that make it easy for instructors to track student mastery and easily aid struggling students are on the way.

Clean User Interface

We’re making alta the most usable, legible, and accessible personal learning experience. For students, a responsive user interface adapts to their web device so they can work how and when they like. We’re developing our alta Design System based on Google’s Material Design, not older bootstrap-like frameworks used by our competitors. That means our experience is more modern and mobile-friendly. alta feels more like other native and web apps students and instructors are accustomed to using in their personal lives.

Less Friction

In user experience, friction is defined as interactions that inhibit people from intuitively and painlessly achieving their goals within a digital interface. Friction is a major problem because it leads to bouncing, reduces conversions, and frustrates would-be customers to the point of abandoning their tasks. — Victoria Young, Telepathy

In alta, we’ve focused on enhancing interactions such as our onboarding flow or when a user encounters an empty state, so that a first time a user understands how to proceed without having to investigate. We’ve revamped our course and assignment cover components to feature key information at the top of the screen, such as status and estimated work remaining and completed, as well as making our calls to action more prominent and informative.

Building a User-Centered Process

Some ways UX design and research help the alta user experience resonate with instructors and students, and deliver real results.

Identify Real Problems

Our sales, marketing and product teams all interact with our users. In fact, we’ve built this as a requirement into our business processes. We all work together to make sure we are coordinating methodically in filtering out the noise, identifying real user problems and addressing them in priority order.

Solving real problems means sometimes going beyond what your users are saying to divine what they actually mean. The result is that alta feels almost like magic to our instructors in higher education, because we’ve succeeded in creating a product more powerful and easier to use than anyone else can due to the baggage and their legacy complexity.

UX Research

The product team is investing in User Experience Research and collaborating with our entire commercial organization. Research in UX informs the decisions we make on how to implement features with real qualitative insights from real users and prospects.

Over the past 6 months we have conducted numerous focus groups and individual research sessions, focused mainly on Grading and Instructor Analytics. The insights from those sessions helped us to iterate quickly on an incredible upgrade to instructor analytics. Meanwhile, we’re planning many more product development focus groups in advance of key roadmap features, such as practice tests and coordinator reports.

With help from our sales team, we’ll connect with our detractors, instructors considering adoption while awaiting specific features, and those curious about alta as an integral part of our workflow. In many cases, Product and UX communicating with potential users will turn skeptics into evangelists.

And there’s more to come in research, such as recurring feature refinement sessions on existing features, student and instructor surveys, market research, and building out our UX Research capabilities to gather more qualitative insights.

World Class Product UX Design Team

In addition to impressive talent in product, engineering, data science, and more, Knewton has built a high-quality product UX Design team comprised mostly of generalists with a high design pedigree and experience in education and a variety of other industries. We are here because we know that a strong UX gives alta a competitive advantage.

Activities such as UX labs and design sprints are baked into the earliest stages of feature discovery, and we continue to refine and try new techniques. We then cross-reference potential solutions with our design system, current product UX/UI, competitive landscape and most importantly, with our users, through research.

This way we help shape how our products are conceptualized and integrated with the overall experience rather than allowing the solution to be pre-ordained when it appears in the early documentation.

Choosing alta Is A No-Brainer

Here’s why all of these overlapping initiatives and processes make alta an easy choice.

Lower Switching Cost, and Less Training

An instructor’s time is valuable, so a more intuitive experience means less headaches when making the switch. alta is not complicated to configure or learn, for students or instructors, and requires far less documentation than our competitors. It just works — both with major LMS systems, and standalone. This quality is by design.

Happier Students = Happier Instructors

We know what students tell their instructors about their experiences with courseware. We uncover these stories in the field, through our research, focus groups, surveys and metrics. We know where and why these products fail to deliver, and of course seek to avoid those same weaknesses in our product. Through reducing student complaints and actually delivering a delightful experience and improved learning outcomes, we’ll achieve nothing short of making teaching and learning easier.

Powerful Insights for Instructors

We’re rolling out new and improved, research-backed analytic tools for instructors, enabling them to easily monitor student performance at each stage of their coursework. We make it easy to identify struggling students and provide granular views into learning objectives and activity. The product team made a real investment in iterating on these features, setting up a framework for an ever evolving set of performance analytics that instructors can rely on.

Transforming Higher Education with Knewton

We’ll continue to focus on creating a superior user experience as we gather new feedback and insights from instructors and students. Perhaps the most critical aspect of our effort to transform higher education through our digital products is the direct connection between our users and our product team, so if you’re an instructor or student interested in speaking with Knewton or participating in future research, please contact us.

Flipped, tipped or traditional: Adaptive technology can support any blended learning model

Introducing alta!