Strata Week: The rise of the robot essay graders

Bot graders pass muster, Instagram's small team handles scale, assessing UK open data efforts.

Here are a few of the data stories that caught my attention this week.

Automated essay-scoring software scores as well as humans

Taking a test at the Real Estate Investing College by Casey Serin, on FlickrRobot essay graders: They grade the same as humans. That’s the conclusion of a study conducted by University of Akron’s Dean of the College of Education Mark Shermis and Kaggle data scientist Ben Hamner. The researchers examined some 22,000 essays that were administered to junior and high school students as part of their states’ standardized testing process, comparing the grades given by human graders and those given by automated grading software. They found that “overall, automated essay scoring was capable of producing scores similar to human scores for extended-response writing items with equal performance for both source-based and traditional writing genre” (PDF of the report).

“The demonstration showed conclusively that automated essay scoring systems are fast, accurate, and cost effective,” says Tom Vander Ark, managing partner at the investment firm Learn Capital, in a press release touting the study’s results.

The study coincides with an active competition hosted on Kaggle and sponsored by the Hewlett Foundation, in which data scientists are challenged with developing the best algorithm to automatically grade student essays. “Better tests support better learning,” noted the foundation’s Education Program Director Barbara Chow in the press release. “This demonstration of rapid and accurate automated essay scoring will encourage states to include more writing in their state assessments. And, the more we can use essays to assess what students have learned, the greater the likelihood they’ll master important academic content, critical thinking, and effective communication.”

Personally, I like writing for a human audience. Bots leave really stupid blog comments — but I bet there’s an algorithm for that too.

Velocity 2012: Web Operations & Performance — The smartest minds in web operations and performance are coming together for the Velocity Conference, being held June 25-27 in Santa Clara, Calif.

Save 20% on registration with the code RADAR20

Scaling Instagram

The billion-dollar acquisition of the mobile photo-sharing app Instagram was big news last week. The news coincided with a presentation by co-founder Mike Krieger at an AirBnB Tech Talk about how the startup managed to scale to 30 million users worldwide with a small team of back-end developers (a very small team, in fact). Krieger’s presentation is interesting in its own right, of course, but news of the acquisition by Facebook certainly fueled interest — in the deal and in the tech under the Instagram hood.

Krieger’s slides can be found here. The presentation details some of the early and ongoing challenges of handling the app’s increasing number of users and their photos (including the recent roll-out of an Android app, which added another million new users in just 12 hours). Although Instagram hasn’t suffered any major outages of the likes seen by Twitter and Tumblr, Krieger does note a number of early problems, including a missing favicon.ico that was causing a lot of 404 errors in Django.

Auditing data.gov.uk

The UK’s National Audit Office has just released its look at the government’s open data efforts, reports The Guardian. Although the open data initiative gets good marks for the “tsunami of data” it’s released — 8,300 datasets — there remain questions about cost and usage.

Governmental departments estimate they spend between £53,000 and £500,000 each year on publishing the data, with the police crime maps, for example, costing £300,000 to set up and £150,000 per year to maintain. And it’s not clear that the data is in demand, according to the National Audit Office report: “None of the departments reported significant spontaneous public demand for the standard dataset releases.” This doesn’t account for the ways in which third-party vendors may be using the data, however.

Big Data Week

April 23-29 is “Big Data Week,” an event created by DataSift that will feature meetups and hackathons in several cities around the world. Big Data Week aims to bring together the “core communities” — data scientists, data technologies, data visualization, and data business. A list of events is available on the Big Data Week website.

Got data news?

Feel free to email me.

Photo: Taking a test at the Real Estate Investing College

tags: , ,

Get the O’Reilly Data Newsletter

Stay informed. Receive weekly insight from industry insiders.

  • http://dannyman.toldme.com/ Daniel Howard

    Smart and well-prepared students are efficient, and don’t waste undue effort in pursuit of a grade. We were taught how to write essays that score well. Contemporary students will learn to write for the algorithm . . .

    So . . . can the students dispense with the rhetorical body of the essay and submit an outline of the arguments, seeded with the appropriate keywords?

    (In my school, we were trained to focus on a strong outline, to make the writing process easy and straightforward.)

  • Eric

    No time to read that pdf, but I suspect there’s no mention of the word “insight” in it.

  • http://comparisonessay.org MIchael H.

    Daniel is spot on. I think we use to spend more time on our outline’s then anything.