Drop testing edutech

How teachers use technology rarely matches how it was designed.

A researcher I know has devoted three years to following a group of low-income students in the Baltimore area who have been learning geometry with the help of an innovative online program. Her paper (which isn’t published yet) is a marvel of careful observations and statistical analysis. Its conclusion, however, is poignant: not only did the students who used the computer program not learn more geometry than the ones taught the old-fashioned way–they might have learned less.

The program was thoughtfully designed and took advantage of the latest and greatest learning algorithms. If any program should be able to help students learn geometry, one might be tempted to conclude, it should be this one. That kind of logic could give ammunition to those who declare that computer-assisted learning is bunk.

But there’s more behind the story.

The researcher told me (and is writing in the paper) that she observed even the most well-intentioned teachers really struggled to figure out how to use the technology. The program wasn’t well integrated into the regular classwork. The “protocols” for use, carefully constructed by the developers, weren’t followed because, as every teacher knows, stuff just happens. Students moved out of town; new students showed up. Teachers came; teachers went. The list goes on.

It was, in short, a pretty good reflection of how technology gets implemented in most classes — hardly in the precise and careful way designed by those who have sweated over the program.

The trial was a flop — not because the technology failed but because there was a mismatch between how the designers believed it should be used and how the teachers wound up using it. Was that the teachers’ fault? Nope. Every day, in every class in the world, teachers come up with workarounds to cope with the unexpected. Most technology, however, isn’t yet as resilient.

We drop test hardware before we send it into the field. Seems like it’s time to start drop testing software programs before sending them into the classroom.

tags: , , ,
  • Radar O'Reilly Fan

    This is a very nice article but I was disappointed because I thought that this was a website about the guy from M*A*S*H.

  • Dave Pawson

    And the link to this online geometry program please Elizabeth?

  • Michael Graham

    Technology in the classroom is just another tool for teaching. It takes a skilled techno-teacher to implement researched based technology into any classroom environment. As one of the only tech savvy teachers in my middle school it has become my unofficial title: tech instructional facillitator. Technology is the best thing to happen to education since the pencil. Bit teachers must be trained in the implementation to use it correctly. I describe my ability to find anything on the Internet and display it on my projector in a nanosecond an art. That without it I would be a less effective teacher. I can’t wait till they give me the iDesk or the iWhiteboard. Come on Steve give it to me and I will show increased student achievment!!!

  • Matt Kroger

    My wife worked for an online school. The curriculum wasn’t delivered through a carefully designed program. Instead content was delivered through websites created by the teachers using a WYSIWYG program. Then, students met with the teacher a few times over Elluminate in a conference to have regular class-time.

    I thought that was a pretty good model. The teachers were able to leverage the chat fuctions of the conferencing software for kids that didn’t feel comfortable talking.

    However, content continued to be an issue. For someone like my wife, who is both a perfectionist and relatively computer savvy, it worked well, because she would make websites with inherently good design (multiple pages with sequenced navigation, embedded videos, etc.). Stuff to really engage the kids. However, most of the teachers had black-and-white paragraph delimited pages with all the content in one page.

    For something like this to work, you need dedicated teachers with more control over the way their content is delivered, and more desire to deliver it well that way. This isn’t a “No-Child-Left-Behind” teachers suck polemic. It’s just a statement. For an effective online classroom, you need teacher customized content and directed student interaction.

    Hmm. You’d think Web 2.0 could handle that.

  • Marie Bjerede

    It seems that we could benefit from user experience design when it comes to school technology. One of the great things about web based technologies is the opportunity to iterate educational software on Internet time, much in the way Google senses how people use their apps and tunes them accordingly.

  • Jethro Taylor

    I work in ed-tech, in a public school district (Canada). Here are some of my observations:

    1) The vast majority of the IT workers in schools have zero educational background. I’m not talking about the so-called ‘technical facilitators’ (which is usually just an excuse for schools to save money by tacking more duties onto a teacher), but I mean the geeks, the guys that actually install the networks, purchase the hardware, and make the decisions about what software is allowed on the computers, what websites are permitted, and who has admin privileges.

    2) Funding for technology, especially that coming from governments in the form of technology grants (as opposed to funds taken from Operating budgets by school boards) almost never allow the purchase of professional development. Governments only want to brag about how many computers they put in schools, period.

    3) Administrators are almost always older professionals, teachers who were promoted; and their understanding of information technology is limited. Their understanding of EDUCATION technology is even more limited. This leaves them vulnerable to greedy vendors who simply want the fat sales contract and don’t care about or understand ed-tech themselves. And frankly, this means pretty much every single ‘ed-tech’ vendor out there- oh, they CLAIM to be cutting-edge, hip, ed-tech companies, but in reality, they are only there to sell the latest gadgets to credulous administrators.

    4) Teachers, even the ones we might label as ‘poor’ or even ‘crappy’ are not given the time needed to properly learn how to integrate ed-tech into their classrooms. They are not given the PD time to learn classroom management skills for highly technical classrooms. They are not given the release time to develop lesson plans and migrate their own systems to heavily technology-dependent models. Teachers aren’t given the freedom to develop their own differentiated instruction materials based on that their students need, but rather are forced to use the ICT templates developed by the geeks and other non-teachers (in conjunction with those vendors I mentioned earlier).

    5) Students are commonly treated like criminals (or in the very best cases, like potential criminals) with regard to their interaction with the technology. Everyone is scared of the students: “Oh, the kids will look at pr0n!”; “Oh, the kids will download muzik!”; “Oh, the kids will cyber-bully!”; “Oh, the kids will cyber-sex!”. Rather than giving the students the respect they deserve as human beings, they are treated as the weakest link in the ed-tech system. Nobody plans PD for the STUDENTS. Nobody gives out money for student-oriented leadership seminars or anything like that- the expectation is that if we give them gadgets, they’ll automatically figure out how to best use it. And then, we freak out because they DO find a way to use the tech, but it doesn’t model the ‘traditional’ education system, and we don’t know what to do about it because it’s too far out of our understanding.

    The simple fact is that until these problems are addressed, ed-tech won’t be able to truly succeed in transforming education. Oh, we’ll still have the isolated pockets of brilliance that we see now- but those examples are anomalies, and would have been showing up as incidents of excellence regardless of the technology involved, since they are driven by factors external to the technology: highly dedicated, enthusiastic teachers; supportive administrators; and boards, parents, and funders who give a damn.

  • Mariana

    There is lots of research in Mathematics Education showing similar results than those obtained by the researcher mentioned in this article. It is true that programs are not necessarily designed according to the needs of teachers, but it is also true that the assumption that teachers can use efficiently in their classrooms without specific instruction is not valid. New research in Math Ed is showing interesting perspectives on these issues. It is worth to read it!

  • Carl Malartre

    Maybe the online program had too many protocols and not enough simplicity, but Michael Graham is right: “teachers must be trained in the implementation to use it correctly”.

    Marie is also right when she talks about “the opportunity to iterate educational software on Internet time”. You usually want to develop close to the students and the teachers, and you want to be aligned with their needs. And fast!

    The program is probably genuinely useful if used by trained teachers with more resources and more experience. Or it’s simply a bad design!

    Carl from http://www.BuzzMath.com

  • elizabeth corcoran

    Thanks for great comments: Dave — I’ll post a link to the study when it gets published. The researcher requested that I wait until then. (Sorrry!) I think that MattK & MichaelG are both pointing toward the same observation — that learning in practice (in situ?) is the the way to go. I’d agree!

  • Ann McCarthy

    Jethro Taylor is on target, especially his comments about teachers needing the time and opportunity to learn how to use technology and how to integrate it into the curriculum. Most teachers would love to be better at using technology, but, as Jethro says, school districts are more interested in advertising how many computers are placed in classrooms than in making sure they are used to advantage. Instead of being an asset to instruction, technology usually ends up becoming one more burden added to all the other stuff teachers have to deal with. (“The teachers can take care of it” syndrome.)

    If teachers were allowed to work with technologists and explain what they need to know and what they would like to do, it would be a different story. But school administrators usually are loathe to pay any attention to teacher opinion.

    As for treating children like criminals–Jethro makes a valid point, but the reason teachers are so paranoid about letting students go out on the internet is that it only takes one unpleasant incident for a career to be ruined. Teachers are responsible for everything that goes wrong in a classroom, and are very vulnerable to lawsuits.