Teaching and Learning in a Net-Centric World

Past the "No Significant Difference"

The release last month by the US Dept of Education’s  Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies provides another salvo in the simplistic showdown between online and face to face learning. As expected online learning (both at a distance and a classroom continues to out perform unmediated education.

First, let me repeat the standard whine accompanying every educational meta analysis – there are far too few studies, many of the conditions between control and experimental group are not held (and perhaps cannot be held) constant and as always when one says ‘online learning’ the term includes a very wide range of learning activities, modes of learning, types of teacher intervention and divergent focus on collaborative, cooperative or individual work – and many other variables that have long been associated with changes in learning outcomes. So when “online’ learning is conceived of as the independent variable- it really means this is the variable we are going to focus on, make a vague attempt to control those we can and ignore the rest!  This occurs even though we know there there are a lot of potentially confounding variables in play. However this variability applies to the complex face to face (F2F) classroom environment as well as online. To be fair to the researchers on this study, attempt were made to tag studies for differences in ‘practice variables’- those under control of the teacher/designer and ‘conditions’, rather unchanging environmental differences  between experimental and control groups. However, again messiness intrudes – as noted by he the authors “Many of the reviewed studies, for example, did not indicate (a) whether or not the online instructor had received training in the method of instruction, (b) rates of attrition from the contrasting conditions and (c) contamination between conditions.” Retention and completion rates are a concern in all types of distance education, so not documenting the independent variables’ association with successful completion mares many studies.

As an example of the confusion of terms, methods and technologies, the study has a brief anecdotal section on “individualized instruction”. I went right to that section hoping it talked about changing condition of online study from the usual cohort, to the older self-paced, independent study mode. Unfortunately, what the authors meant by the title was interventions involving more machine interactive learning – tailored responses with additional help for incorrect answers (positive effect) and individualized adaptation presenting different environments to different students (positive effect again).I appreciated the study’s review of the literature and notes that “available research evidence suggests that promoting self-reflection, self-regulation and self-monitoring leads to more positive online learning outcomes.” Attempts to guide the online interactions of groups of learners were less successful than the use of mechanisms to prompt reflection and self-assessment on the part of individual learners”. These results suggests the value that arises from effective use of blogs, self awareness exercises (profile comparisons?) and meta cognition training and experience.

I also found two older references to studies showing that when student-student interaction was increased through cooperative learning strategies, the effect of increased teacher moderation (adding teacher presence to rich student-student collaboration context) was not realted directly to the learning outcomes (see Bernard, R. M., & Lundgren-Cayrol, K. (2001). Computer Conferencing: An Environment for Collaborative Project-Based Learning in Distance Education. Educational Research & Evaluation, 7(2/3) ). This seems to confirm my earlier grand “interaction equivalency theory

Despite these limitations and mixing of apples, oranges and bananas and struggling to think about coherence in the midst of a fruit salad, the math manipulations involved in a meta-analysis of the 46 studies (51 effect sizes) produced some interesting results. Of most press value (and ammunition for skeptical colleagues)  is  “The overall finding of the meta-analysis is that classes with online learning (whether taught completely online or blended) on average produce stronger student learning outcomes than do classes with solely face-to-face instruction. The mean effect size for all 51 contrasts was +0.24, p < .001.” This effect size is not huge, but the significant difference shows that something is happening in these complex contexts that is resulting in higher student learning outcomes. The study goes on to examine comparison between online and blended (blended significantly higher) and completely online and Face to face- a smaller, but significant difference in favor of online.

This type of study (under current miniscule research funding models) will never answer the really important questions regarding types and quantity of instructor, social and cognitive presence, nor tell us  the types of learning activities that are more effective in which contexts. However, we have been struggling for 30 years with perceptions that online learning and all other modes of distance education are not as effective as classroom education at enhancing student learning outcomes. This and other recent research (see Bernard et al 2004, Sitzmann, R., Kraiger, K., Stewart, D., & R., W., 2006) is showing that distance education is beginning to show consistent improvements over classroom learning (thus ending the no significant difference debate? – not likely!).

This study shows that the affordances of networked technology enhance student learning whether at a distance or on campus (blended). That really is no surprise, but should be useful ammunition when laggards and late adopters continue to press for ‘evidence that this stuff works’.

The study will also be useful as defense for the most common type of network enhanced education programming – that being blending online learning with face to face classroom activity. As expected learning gains result when students have opportunities for both. But as Kirkpatrick’s four levels of evaluation and Philips important fifth ROI addition tell us, if learning gain happens, but at greater cost or time expense, such as by doubling the infrastructure or time commitment – the gains may not be cost effective. Rather, blended learning teachers have to juggle input costs to arrive at strategies for getting the best mix of asynchronous online, synchronous online, face-to-face and individualized and cooperative learning activities.

Similar posts
  • New Book from AUPress – An Onli... I was pleased to receive in the post a hard copy of a new book in the Issues in Distance Education book series, for which I continue to serve as the series editor. Now of course you can read all of the books in this series as they are available for download  under Creative Commons [...]
  • Qualitative Research Rebooted 2018 For the past two months, I’ve been occupied with a qualitative study of teachers’ use of digital technology in Alberta Schools. The study is sponsored by the Alberta Teachers’ Association.  It has been very useful for me to get down to actually doing a full scale qualitative study after years of teaching grad students research [...]
  • More on Distance Education Journal Ra... Both academics and administrators love to argue about the value (impact) of their academic work.  The old adage of “Publish or Perish” still has currency. Despite the many distribution opportunities besides and beyond publishing in scholarly journals, the bean counters (myself included) love citation indexes. The basic idea is that the more your work is [...]
  • What the FOLC is new in this article? Sorry, but I couldn’t resist spoofing, in the post title,  the unfortunate sound of the acronym for the “new” model proposed in this article. Now,  I’ve got it out of the way and can only suggest that if this “divergent fork of the Community of Inquiry model” is to survive, it needs a new English [...]
  • Quality in Online Learning Presentati... I was asked to do a video conferencing talk to a meeting of three Mexican Universities yesterday. They are attempting to come up with a common set of criteria to define and measure the quality of their online courses. Perhaps I was not the best person to ask, as I have very mixed feelings about [...]

1 Comment

  1. July 7, 2009    

    Hi, Terry. Good comments. I scanned this report as well and I appreciate your insights about meta-analyses. I performed one a few years ago to see if CAI or CAL instruction improved student learning in K-12 settings. I found it didn’t. This type of systematic analysis has its strengths in trying to measure oranges, apples, and bananas, and I think it is wise to review the outcomes versus casting the ‘statistical’ data aside.

    From reading the outcomes of this study, and reflecting on a recent study I have done on online learners in higher education, I suspect one contributing component of the appeal and affect of online learning is the whole educational experience. That is, it might not be that learning is better online (or with blended modes), but that collectively the convenience, access to resources, communication opportunities, and ways to present work using new technologies enriches life for a student.

    I have found teaching styles online vary, but it does not necessarily mean it limits good learning. With that variable removed, I think the technological world provides learners with more experiences, tools, and opportunities to learn. With that said, I think we still need to continue exploring what makes for good learning in this world considering how information is created, provided, and required, and how the affordances of technology affects this.

No Pings Yet

  1. A report on online learning | David's Occasional Blog on August 17, 2009 at 11:35 pm
  2. Online students perform better than face-to-face students | Tony Bates on September 6, 2009 at 7:23 pm

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to Virtual Canuck via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 332 other subscribers

My posts by Category

My Blog Archives


  • Twitter
  • LinkedIn
  • SlideShare
  • RSS Feed for Posts
  • Email

Follow me on Twitter