Skip to main content


Welcome, the Hub connects all projects

Voices From The Field


MSPnet Blog: “Fast or good? or Who’s in charge of ed tech innovation?”

See All Blog Posts

posted December 9, 2014 – by Brian Drayton

A recent policy brief by Noel Enyedy, posted at the National Education Policy Center, tells an interesting story about research on “computer-mediated learning.”  In reading it, I’m bothered all over again about the strange paradox of “educational technology.”  In his analysis, the evidence is weak for the benefits of computer-mediated learning of several kinds, including games, animations and other video products.

For example, Enyedy reviews literature about the effectiveness of digital systems for “personalized instruction,”  one of the many hopeful arguments in favor of large investments in school tech.  One version of  “personalized instruction”  is linked to “adaptive systems”: with the right systems in place, we can diagnose each child’s location in a learning progression, as she works in a digital environment which tracks her work, evaluates roadblocks and avenues of progress, and makes choices about how to display new tasks, information, or other scaffolding to get her from where she is to the system’s goal for her.    Other versions of personalized learning are less comprehensive in architecture, and leave more to the initiative and choice of the learner or teacher.  In any case, however, Enyedy’s meta-analysis of studies about personalized instruction suggests that only the most hopeful advocate could see much evidence of benefit.  He writes:

Studies conducted from 2004- 2009 showed a .008 increase in effect size when compared to studies that used technologies that were state of the art from 1997-2003. It is important to note as well that outcomes primarily reflected procedural (or how to) knowledge, not increased efficacy for declarative (informational) knowledge or strategic thinking. That is, improvements do not effectively yield the type of conceptual understanding, problem solving and complex thinking that the current economy requires.

Now, Enyedy holds out the possibility that personalized instruction might show more impact if implemented in a way more in line with students’ technology use outside of school — meaning mobile devices, mostly.   There is not much evidence that this strategy is the right way to go, but it is an avenue that he suggests we should explore.  He also points out (his discussion is thoughtful and I will not do it justice here!) that “personalized instruction” is not the same as “personalized learning,” in which the student has more control over pacing and sometimes other elements of the learning process. This can certainly be supported by digital tools, though it is also supported by books, libraries, teachers, study-buddies, and all the other tools that learners have used for “personalized learning” for the past several centuries.   His brief also includes some valuable suggestions to school systems about how to think clearly about their investment in technology if “personalized instruction” is a key benefit being sought.

But Enyedy’s brief points up the paradox I mentioned above: Given all the knowledge and skill that teachers must bring to bear in their difficult and subtle work with growing minds,   the relentless push to incorporate new technologies of indeterminate value into the classroom can’t be at bottom an educational choice.  One can dismiss critics like Larry Cuban as curmudgeons with Luddite tendencies (see his latest reflections on flashes-in-the-panaceas), but in a time of intense fiscal pressure on schools (and parents), and intense exploitation of the educational “market,” the lessons of history, and of research, seem like wisdom.  Teachers and other educators have no help from  anything like an FDA (however imperfect)  to enforce clinical trials of new technologies before they are advocated on the open market, or even mandated by policy; any very roughly analogous agency, like the Dept. of ED’s What Works Clearinghouse,  is all too vulnerable to the ideology du jour.

Ernst Mayr once opined that biology is uniquely complex among the sciences because it has to reckon with almost all the processes of the known universe.  Human society piles on yet more complications, and in education we see all the ingredients of human society in condensed and unstable form — full of potential, and also still full of mystery.  Our tools are far less complex than we are ourselves, and  it takes a long time to figure out what’s happening when we apply them.  (Most educators gratefully — thankfully! —  abandoned the birch rod as an educational technology a few decades ago, but it has to be said that the effects of corporal punishment were complex enough that a few people in our society can still argue that corporal punishment is an necessary expedient in the classroom.  Is that an educational judgment, or something else?)

How are you weighing the technology choices that face you?  What kind of evidence convinces you that a new tool is likely to be worth the investment to buy it and put it to good use?


Blog comments have been archived, commenting is no longer available.
This blog post has 11 comments, showing all.

What's worth it? High school math

posted by: Louise Wilson on 12/10/2014 11:10 am

I have had commercial software thrust upon me. It's mostly useless, because it's made for profit, not education. I ran a free trial from Carnegie Learning (adaptive feedback) which was great for my 9th grade students performing at the 3rd grade level, but it was too expensive for our school district. I checked before and after skills, and willingness to try. (At this level, skills are what we need, sorry people.) Skills and willingness both increased.
I had someone open my eyes to WeBWorK. It's free. You can write what you want, or take questions from the library. Student "growth" as measured by NWEA of 2 years in one year, and willingness to try has also increased.
I'm using WeBWorK again. It's no dollar cost, it is a time commitment, but everyone seems to think teachers have infinite unpaid time available to work anyway.

No substitute

posted by: David Thomas on 12/11/2014 9:35 am

Euclid is said to have told a grumbling Ptolomy I "there is no royal road to geometry". Today, as then, there is no other road to knowledge (and hopefully wisdom) than enlightened teaching and discourse. Technologies like Geogebra and The Geometers Sketchpad support rather than usurp that discourse. So much EdTech does just the opposite. What comes of practicing content you do not understand? You build larger and more vexing misconceptions.

Teachers' role in innovation/adoption

posted by: Andy Zucker on 12/11/2014 12:52 pm

There are two interesting replies to date (Louise and David) to Brians post. Educational technology is fiercely complex because digital tools encompass so much: computer audio, video, online schools, threaded discussions (like this one), email, word processing, various sensors for science labs, graphing calculators, Geometers Sketchpad, and almost infinitely more tools and specialized software. There is high quality research about some technologies (for example, appropriate use of word processors has been shown to help kids write better) but there are hundreds of technologies and not as many excellent studies.

Louise raises an important point about costs to schools and teachers, both money and time. In my opinion, teachers need to be a significant part of decision-making processes for ed. tech. adoptions in schools. In many school systems that may already be true; in others, decisions are more top-down.

Teachers' role and thoughtful innovation

posted by: Brian Drayton on 12/12/2014 9:38 am

Andy,
I think you've captured the key issues here. It seems, sometimes, that school systems (at all levels) have a hard time even applying "consumer wisdom" to their tech acquisitions. A careful cost/benefit calculation for a new technology would, on the one hand, take into account teacher-time and other costs (as well as the learning curve for students students have time issues, too!) and on the other hand would have "benefit" metrics that examined things other than student "outcomes" (that is, test scores). Do you know of systems that have a sort of layered approach, that recognize that full value may not be realizable (much less measurable) for a few years, but that interim benefits might accrue at earlier stages of the innovation which could provide evidence that it's a good investment?
It's a problem that so much educational change (IMHO) is being driven by market (commercial) forces, rather than educational ones, which work at very different tempos, and have different "bottom lines"... (and education doesn't have a bottom line, it's more like a watershed system!)
-- brian

Schools as watershed systems

posted by: Bill Zoellick on 12/18/2014 11:23 am

Brian --

I really liked your suggestion to think of schools as watershed systems. In addition to my work with schools, I am involved as a volunteer with Friends of Acadia and with watershed restoration at Acadia National Park. The park looks at many indicators of watershed health, not just a "bottom line." But the important idea is that resource managers make sense of these indicators in terms of a bigger concept ... in the case of Acadia's watersheds, the bigger concept is "resilience."

Your idea inspired a blog post (see http://bit.ly/1C5JMHt) about the question of what the big idea would be in place of resilience. I suggest "capacity to get better at getting better."

post updated by the author 12/18/2014

School systems as watersheds, indicators of health

posted by: Brian Drayton on 12/19/2014 3:43 pm

I like your elaboration of this idea, and the analogy to the resilience of a system. Another ecological analogue might be "ecosystem health," which includes ecosystem services, resilience (of course), and various indicators of ecosystem function for the various communities and indeed, a little community ecology applies to school systems, too. After all, the condition of a school system is reflected to some extent by qualities of the students and teachers, but also administrators, school boards, technology and support staff...and the exchanges and dependencies among these players.
It's not new to recognize that a systems approach is helpful in thinking about schools, but whether you are interested in the best use of technology, or the best ways to measure outputs we are a long way from using this in the design of policy, I fear.

Educational Technology

posted by: Martha Syed on 1/3/2015 5:25 pm

I'm sorry for replying so late but I like to be kind of relaxed when I'm reading stuff like this instead of being tensed up from teaching, parent /teacher conferences and staff meetings. I just wanted to say I agree with you Mr. Zucker when you posted from someone else that "teachers and parents need to be a significant part of the decision -making processes for ed tech adoptions in schools. For the communities that are already taking that step (kudos). We have to be more conscious of who it will affect more as far as making sure the true results are about making students confident in their basic education for the present and future.

teacher and parent input

posted by: Brian Drayton on 1/4/2015 6:32 pm

Martha,
Your post makes me think that it might be valuable if interested teachers and parents in a school could form a technology council to discuss what they have learned with new technologies, maybe try things out, and critique them from pedagogical and other points of view, and then make recommendations to the faculty/admins for new software or hardware to test out in various classes. People are so immersed in various tools and gadgets, I think it could be exciting for a community to develop criteria by which to reflect about and evaluate new technologies, and take a more thoughtful and conscious approach to innovation. Does anyone have experience with anything like this? Is it utopian, or worth a try?

Computer-mediated learning does not equal ed tech

posted by: Talbot Bielefeldt on 12/12/2014 2:45 pm

Andy Zucker's first point bears repeating. "Educational technology" is not "computer-mediated learning," although that is one example of ET. Computer-mediated/assisted/adaptive applications are basically traditional information-transfer environments that try to capture aspects of effective teaching. Comprehensive frameworks for technology envision making the school learning environment like professional learning environments, often under the rubric of "21st Century Skills". That is, technology is supposed to facilitate the human interactions that promote deeper learning. This has its own set of problems: Because the content of learning may be more diverse, it is hard to assess with large-scale testing. Teachers still need to be involved in the technology selection and implementation. However, it is essential that we clarify exactly what we mean if we use the term "educational technology." Naive decision-makers may cut basic Internet access (necessary for learning fundamental research skills), because they lump all computer-related ideas with a particular ineffective application.

Ed tech as part of MSP innovations

posted by: Brian Drayton on 12/16/2014 10:45 am

The points raised so far are valuable in thinking about how to make good choices about ed. technology.
I am curious, though, if any of the MSPs have made specific technologies an important ingredient of their innovation?
How did the project's innovation (e.g. in probe-ware, social media techniques, video, or anything else) "fit" with the innovations that their participating school districts were planning on already?
what has worked, and does (did)_ it work better for some participants (teachers, students, schools, administrators, IHE faculty, project staff) than others?
How much do you count on technology to maintain the partnership? What do you do in that regard?
In particular, has technology played a role in keeping teachers, or IHE faculty, connected and active in the program?

MSP Project Resiliency

posted by: Betsy Stefany on 12/20/2014 1:26 pm

Resilience is a key concept and thanks for keeping this term in the discussion. The STEM project that I designed and managed focused on specific technologies, and the ability to be flexible was paramount to success.

I truly appreciate reading all of these wise views on varied educational technology integration. The MSP STEM project that I managed had a full watershed of them as we worked to form an ecosystem to include all stakeholders. While the project has completed the good news is that the efforts within the project spawned innovations during those five years that now forms a sustained community of practice. Perhaps even a new strategy toward digitization as a STEM process.

Continuing the watershed concept, theres been unexpected storms in this era and, as we were the first STEM devoted project we were "out there" in the flood.

These events have changed the availability of online content delivery designs, personal devices (computers/handhelds) and digital data collection tools, all which were parts of the project. Also the marketing directions of these varied elements have overflowed out in our area, directly engaging and changing what participating school districts were planning on a year to year basis.
The project of multiple districts had not one administration that remained in tact. Teachers changed grade levels, domains and whole districts. Yet, as of last week, a full six months after the project, the teachers are advancing in their projects and this advancement can be documented by the digital ecosystem.

Resilience means, however that the success of a project may not be in the same form as expected when the evaluation was initially designed.