Tag Archives: tech

What Our Interns Did This Summer

This summer we were lucky to have a rockstar set of interns on our tech team. Here’s just a quick look at some of what they accomplished this summer:

Andersen Chen worked on models for estimating the way student proficiencies change over time. Most existing methods are focused on point estimates resulting from tests and surveys, so they don’t concern themselves with how students improve over time — but that’s exactly what we’re focused on!  Andersen worked with our data science team, using both simulated data and anonymized data from our courses, to consider student proficiency as a process changing over time.

Anderson recaps his work in a final presentation to the company.

Zack Newman experimented with Finagle, an open-source remote procedure call (RPC) library written by Twitter. He built a simple HTTP/JSON-to-Thrift proxy server using Finagle. He patched our fork of Apache’s Kafka to add hooks so that our systems could respond appropriately to its rebalances. In addition, he worked with the data science team on modeling student proficiency for concepts on which they might not have been explicitly tested. Zack also had some great slides in his final presentation:

“You may think I’ve spent the whole summer eating,” Zack said in his final presentation.
Actually, we were wondering if you spent it Google image searching.
OK, that looks like a pretty complicated slide. Maybe he did do something after all.

Matthew Suozzo worked on adding configuration management extensions and unit tests to the User and Course Admin services, but his main task was an update to Apache’s thrift project. Matthew also added support for the JSON protocol and Union structures to the Python thrift compiler.

Matt during his final presentation.

Dylan Sherry added system health statistics collectors to the Knewton Recommender System (KRS) using Brightcove’s “diamond” library, and supported instrumentation of KRS via Etsy’s “statsd” library. Dylan also aided the support of new features for Knewton Crab Stacker and considered preliminary steps towards a full-stack dev environment management system, dubbed the Knewton Development Environment. Dylan’s final contribution was to enable diamond to gather information on KRS caches.

Dylan during his final presentation.

Joy Zheng built a web-based data dashboard to quickly aggregate data in a large-scale, distributed, and real-time system for hundreds of thousands of students across the country using Knewtonized courses. The dashboard had to be responsive, smartly display large amounts of data, be searchable and not impact current production services. We solved these problems by using smart optimizations such as threading, lazy caching and lazy display, as well as some efficient in-memory manipulations. From an implementation perspective, the backend for this dashboard interacts with several services deployed in the cloud environment and entailed using Java, JSP’s, Thrift and connecting to services using Cassandra. The front end was built using HTML, CSS and Javascript, and is currently successfully deployed in Production.

Joy shows the Pearson data dashboard during her final presentation.

Tarik Tosun worked on models of student pacing and progress in an adaptive learning environment.  There are several challenges  in modeling this behavior of interest to Knewton, and Tarik, working with the data science team, was able to develop some very intriguing solutions.

Tarik during his final presentation.

If you’re interested in solving interesting world problems, working with exciting data sets, building your professional skills, and learning from the best in the industry, check out our jobs page. We’re always looking for software engineers, data scientists, and other smart people to join our team!

Screen shot 2012-08-30 at 3.56.11 PM

New Post on N Choose K: Web-based, Real-time Data Monitoring

Check out the latest post on the Knewton tech blog, written by one of our star summer interns, Harvard student Joy Zheng.

Here’s a sneak peek:


How do you quickly visualize and validate data in a large-scale, distributed, and real-time system? How do you make it deployable in multiple cloud environments? And how do you track changes to this data over time?

These are questions that I set out to answer in my internship this summer, as I worked to develop a web-based interface that displays how much data the Knewton system is receiving from hundreds of thousands of Pearson students across the country.

At a very high level, this is how things work: Hundreds of thousands of college kids across the country use a recommended textbook published by Pearson. These books have an online component with which the students interact, learn, take tests and quizzes, etc. Information regarding all this activity flows through to us at Knewton. We process it to make recommendations to each student about how to progress through the course, so that the student can achieve the milestones set by the professor.

Read the rest of the post here.

Photo from zacheryjensen on Flickr

Cross-pollination at Knewton: Marketing & Software Engineering

I still remember when I first came to Knewton for my interview over two years ago. I said I was interested in marketing. I was told I could start off in academics drafting verbal coursework. The director assured me that there were plenty of opportunities for “cross-pollination” (at the time, I thought this was an original metaphor :) ).

Within a few days, I started blogging for the marketing team. Within a few months, I was working 50/50 in academics/marketing. I was both building products and marketing them (and sometimes answering customer service tickets!). This was followed by a brief stint in tech, which helped me grasp the science behind recommendation and the intricacies of adaptive learning. It was like my own “rotation program”: I had exposure to customers, and I came to understand the platform from different angles.

There are other benefits related to wearing multiple hats at a company. Working at a startup, you’re exposed to cutting-edge technology and business concepts. This is invaluable from a long-term career perspective because it helps you understand other emerging technologies and business trends. You can be assured that you’re investing your energy and developing expertise in the right areas. In this sense, working at Knewton has been transformative for me personally, professionally, and intellectually. As a result of working here, I feel that I have a more visceral understanding of technology and its impact on business and culture as a whole.

So that’s my story. In this series on cross-pollination, I’ll interview other colleagues of mine who have benefited from the unique career opportunities at Knewton. This first post will feature Jonathan Bethune, one of my old teammates from academics who also started off drafting verbal coursework for our GMAT product.

CY: How did you find your way to Knewton?

Jonathan: Before Knewton I taught humanities in Tokyo and New York City. I left a position with a charter school in the winter of 2009 and decided to look for non-teaching work related to education. Knewton seemed like a natural fit.

CY: What was your first position here?

Jonathan: I was a Content Developer for my first year and a half. At first I wrote test-prep questions and GMAT lessons, then moved on to developing the Knewton Math Readiness course.

CY: Tell us about your transition to tech. What got you interested in coding?

Jonathan: I’d done basic web development and scripting since high school, but really only started at Knewton when a colleague and I worked on a program to automate question creation for one of our projects.

I really enjoyed working with the tech team then and began putting in a lot of time outside of work studying programming and computer science.

I switched to half academics and half tech work in the late summer of 2011, then moved to full-time software engineering in January of the following year.

CY: What are you working on these days?

Jonathan: Currently I am on the Systems Engineering team. I manage software deployment templates and configurations, and help developers structure and launch software stacks. I write a lot of “glue code” in BASH and Python to help automate and validate services before launch.

I also serve as the Deployment Engineer in charge of updating services in production and staging once I get the word from QA.

CY: What advice do you have for someone right out of college who’s interested in making a career at a tech startup?

Jonathan: Being proactive is probably the most important thing if you want to be successful. You cannot sit back and wait for a manager or team lead to hand you an opportunity. You have to look for solutions to problems others do not perceive. You have to anticipate.

CY: Great advice for the working world in general. But how did you initiate that transition?

Jonathan: In my own case no one invited me to join the tech team; I was interested, I studied, and I made my own opportunity by spending weeks annoying a lot of engineers and dev managers. If you want to carve out your own role and rise to a position of substance, you have to have moxy.

How to Build a Robot with Python Scripting on Android

This post was written by Zlata Barshteyn, a former intern here at Knewton. 

Every few months at Knewton, we have something called Hack Day, where team members take a day off from regular work tasks to tackle projects they’ve been itching to develop. These “hacks” typically fall into one of four categories: product, business, culture, or performance.

A Robot for the Office

For my Hack Day project, I worked with my mentor on building a LEGO Mindstorms NXT robot that could serve as a minion around the office. We programmed it to drive around and avoid colliding with obstacles, with help from an infrared sensor. Mounting an Android phone on the robot and having the robot communicate with the phone expanded its capabilities, as the phone had a camera and access to the Web, which the robot did not.

Equipped with a camera and Internet access, the robot was pretty powerful. It could provide virtual tours of the Knewton office, driving around and broadcasting its camera feed to a web page that could be accessed remotely. Viewers could even drive the robot themselves through built-in controls on the page, making the tours extremely flexible and open-ended. With text-to-speech capabilities, the robot could say each of the Knewton team’s SVN commit messages as they came in, and say a phrase typed in by a viewer on the web page. With enough time and Lego bricks, the robot could even be trained to fetch snacks from the kitchen and deliver them personally.

To get the robot to communicate with the phone, we had two options: write something from scratch to send and receive data over Bluetooth, or use the Cellbots app or Python library. The Cellbots team provides software and code libraries to facilitate communication between Android phones and various types of robots, such as the NXT. Using the app would provide many features and controls from the get-go, but we wanted to write and customize code ourselves, so we went with the Python library.

Running Python on Android

Now, the question is, how would Python code run on an Android phone? After all, Android apps are written in Java. That’s where Scripting Layer for Android, a handy Android app, comes in. Once you install it on your phone, you can run files written in a multitude of scripting languages (such as Python) by putting them into the app’s directory on the phone, then opening the app and selecting them. It even comes with “Facades”, which are basically APIs that give you simplified control over things such as Bluetooth and the camera in the scripting language of your choice.

This was our reasoning: If we could run Python scripts on the phone, then we could have the phone host a web server, fetch a video feed using the SL4A Webcam Facade, and display it on the server. Then, whoever wished could navigate to the server’s address and watch a live feed of what the robot “saw” through the Android phone’s camera. Controls could then be placed on the page, allowing someone to turn off the robot’s autonomous navigation and drive it around the office themselves, or even have it speak a given phrase, using the SL4A Text-to-Speech Facade.

Setting Up a Web Server on Your Phone

The first step is to find the IP address of the phone, in order to access the web server from another computer. Check out this blog post (scroll down to “Gleaning your IP address”). Do you want a web page that only lists the contents of some directory on your phone or SD card? If so, you can just use Python’s built-in SimpleHTTPServer, which is the server demonstrated in the previously mentioned blog post. As you can see, it only requires 4 lines of code:

import SimpleHTTPServer
from os import chdir

To run the server and view the page, do the following:

1. Save this code in the SL4A directory
2. Open the SL4A app on your phone
3. Run the code you just wrote
4. Open up a web browser on your computer
5. Navigate to the IP address you saved earlier, port 8000
(so if the IP address was, you’d go to

You will then see a listing of the contents of /sdcard/. As you can see, this is a fairly limited web server, and is only useful for, say, browsing your photos. It definitely wouldn’t be enough for what we wanted to do with the robot (embed a video feed and controls). For that, we’d need a more sophisticated web server.

Further Possibilities

Luckily, using SL4A means that anything you can do with Python on your computer, you can do on your phone. That means that you can even run CherryPy on Android with no hassles, or any other Python web server, really. Once that’s done, you can use the SL4A Facades in your server code, meaning you can use the Webcam Facade to stream a video feed from your phone’s camera, the Text-to-Speech Facade to have the robot say out loud a phrase the user typed in, or any of the other Facades in a variety of creative ways to have your very own office robot/minion.

The bottom line is, if you love developing in Python or any other scripting language, you don’t have to suddenly stop just because you’re working on the Android platform, thanks to Scripting Layer for Android.