Actionable data for improving student learning and inter-institutional comparability – Can we have both?
An article titled Searching for the Holy Grail of learning outcomes from Inside Higher Ed (IHE) caught my attention last week. The article discusses the elusive quest for data that illustrate the value add provided by a student’s progression through a degree program at a particular institution.
Because the Collegiate Learning Assessment (CLA) purports to provide this value added score it is fast becoming the market standard and the author of the article goes on to cite a number of reasons why this coalescence should concern us.
First, here’s some background in case you’re not familiar with the CLA. The largest market for higher ed accountability is undergraduate general education where the focus is on things like a students’ ability to think critically, to read and write effectively, or to solve problems. As I summarized back in 2009, “we now have public accountability campaigns including the Voluntary System of Accountability (VSA), the University and College Accountability Network (U-CAN), and Transparency by Design which serve the public, private not for profit, and private for profit institution types respectively” (McKay Epp, 2009b).
Because the focus of the IHE article was on public institutions, the VSA is the accountability campaign that was highlighted. As background for those not familiar with the VSA, it
allows participating schools to choose among three assessment instruments that are administered to students with the goal being to indicate student proficiency in the areas of reading, writing, and critical thinking. One of these tests, The Collegiate Learning Assessment (CLA), actually tests a sampling of entering freshmen and graduating seniors and correlates student scores to incoming student SAT or ACT scores in an attempt to show the value add provided by an institution over the course of a degree program. (McKay Epp, 2009a).
One of the most interesting critiques of the CLA in the article is the high correlation between it and the SAT. Olds states that “the amount of variance in student learning outcomes after controlling for SAT scores is incredibly small” (2012). The author goes on to say that “Most institutions’ value-added will simply be in the expected range and indistinguishable from each other. Hence, why bother with the CLA” (Olds, 2009).
While the author lists several alternatives to the CLA (which are worth reading), what I found most interesting was the discussion about the struggle that institutions have to find data that create actionable insights for improving student learning. For nearly four years I’ve been working with institutions to help them implement Pearson’s Learning Outcome Manager (LOM) which is a technology enhanced outcome management system.
LOM does an excellent job of providing actionable data to faculty and administrators on student performance against established learning outcomes for online and blended courses and programs. Because outcomes are associated to graded course assignments, it helps ensure that evaluators are seeing students’ best effort and when done well it minimizes additional workload for faculty. The challenge is that LOM generated data is so targeted to individual professors or to a particular course that its results can’t easily be used for inter-institutional comparability.
While I believe a majority of educators would agree that the most important reason to work in assessment is the desire to improve student learning, I also recognize that the demand for data on inter-institutional comparability will not go away. This article provides some interesting alternatives to standardized assessments such as the CLA which I think could work in tandem with data generated from systems like Pearson’s LOM to provide a win-win for the assessment community.
Works CitedMcKay Epp, B. (2009a). Implementing a Technology Enhanced Outcome Management Strategy on Campus that Produces Substantive Improvements in Curriculum, Instruction, and Student Learning. EDULEARN09, Barcelona. Retrieved from http://library.iated.org/ view/MCKAYEPP2009IMP
McKay Epp, B. (2009b). Improving Student Learning: Thoughts and Reflections. Educator's Voice, 10 (3). Retrieved from http://www.pearsonecollege.com/Newsletter/EducatorsVoice/EducatorsVoice-Vol10Iss3.learn
Olds, K. (2012). Searching for the Holy Grail of learning outcomes. Inside Higher Ed. Retrieved February 20, 2012 from http://www.insidehighered.com/blogs/globalhighered/searching-holy-grail-learning-outcomes
Brian Epp | Supervisor, Assessment and Analytics Group | Pearson eCollege
I’ve been teaching online or in a hybrid format for about 12 years now. I’ve been teaching in the classroom for even longer. But when I teach online there’s always something that seems quite trivial that I actually miss. I like to refer to this as one-of-those-go-over-the-syllabus-days, and it’s usually the first day of class.
Of course, students find this pointless and boring, generally, but I’ve found that they do tend to pay attention. I don’t read the syllabus word-for-word, but I at least point out each section, what it means to them, and what they are required to know or do.
There’s a real purpose for this. By my way of thinking, a syllabus is like a contract. It’s a contract I make with students that says, “This is what I’m going to provide to you [knowledge and guidance] and in return, this is what you’re going to provide to me [effort, study, work products, etc.].” By doing this in class (in an on-ground course), I can ask if anyone has any questions. I can look for nodding heads or confusion on faces, and I can address any issues in class. And if I really want to be a stickler, I can have students sign a page stating that they have read and understand the syllabus.
Online, I don’t have that same kind of forum. I do use, for example, a Virtual Office. This is just a discussion forum that I’ve renamed, and I ask students to post any questions about the course there. They can then learn from each others’ questions without sending me the same question via email 20 times. (Although, if it’s a question about a grade or other individual work in the course, then I ask them to email me.) So, if students have questions about the syllabus, they can ask them there.
That isn’t good enough, I’ve found. I still have students come to me saying that they didn’t realize such-and-such and that it would affect so-and-so. So, I also like to include a syllabus quiz. I write up 10 or 15 questions based on the information provided in the syllabus. They are specific questions, e.g., “Which of the following are our course objectives?” and I throw some irrelevant answer choices in there to make sure that they’ve read the course objectives before they can answer that question correctly. And, if I have assignments of different point values, I add a matching question: “Match the assignment on the right to its respective point value on the left.” That way, students will realize that, yes, the final research paper does make up one-third of their grade, and therefore, it is important.
In effect, this is like asking the students to sign off that they have read and understand the syllabus. Heck, I could use a syllabus quiz in my on-ground courses, too. In online courses, it’s particularly important.
In Pearson Learning Studio, I can also enable the Path Builder tool (found under Course Admin --> Enable/Disable Tools), and then use the Course Scheduler link to access Path Builder. I can use Path Builder to “gate” the rest of the course content around the syllabus quiz. If students don’t pass the syllabus quiz, they can’t even move forward to the first Unit. No Christmas-treeing that quiz!
Consider using a syllabus quiz in your own course. It might just make the difference between a smooth-running course and a bunch of Hey-I-didn’t-know-that! excuses later.
Rob Kadel, Ph.D.
Supervisor, Academic Training & Consulting
I just got back from performing a keynote address in Berlin at Online Educa. It was an amazing experience. Not only was the conference packed with over 2000 people, but the city of Berlin was quite breathtaking this time of year. Everywhere you look in Berlin there is some kind of Christmas decoration, tradition, or ornamentation. People gather together at the Christmas markets to drink Gluehwein (a spiced, boiled wine drink that smelled delicious) and sales abound in the shopping areas.
So as I was walking through one of the markets with some friends, I thought back to the decorating of my own tree just a few weeks ago, which led to thoughts of…instructional design! (Seriously, I need a break). With a four year old, Christmas came early this year and we had our tree up on Thanksgiving day!
But the lights on the tree, specifically, were quite an ordeal. Actually they still are. See, last year we bought a new tree. We took our daughter down to “St Nick’s” Christmas store (no joke) and asked for a guided tour of the new trees. While the trees look amazingly real, they ALL – 100% - had a major flaw. It was impossible to buy a tree without pre-decorated lights! And not just pre-decorated, but all white lights. Ugh.
Of course, I get why they do it. Most people hate lighting the tree. It’s time consuming, you end up missing spots, and the only thing worse than getting them on is taking them off. But, I knew then what proved to be true this year. Pre-lit trees are not what they appear to be. See, this year, I had happen EXACTLY what I asked the sales-elf about last year:
ME: “What happens if a light goes out?”
ELF: “That hardly ever happens!”
ME: “Okay, but what if it does?”
ELF: “Well, the lights aren’t connected like they used to be. If one goes out, it doesn’t affect the others, it just goes out. You can replace it or leave it, but the rest of the lights will shine.”
You can probably see where I’m going with this. This year, just as I suspected, we got the tree up, plugged it in, and yep, you guessed it – the entire middle of the tree was black. So, I got to spend about an hour, finding, unplugging, and re-plugging new lights into the old sockets, hoping each one would light the strand back up. (I never got more than 4 in a row to light up with any new bulb…)
Alright, enough about my holiday nightmare. So what does this have to do with Instructional Design? Well, as I stood there checking bulb after bulb, I realized that some schools are taking this approach to their online courses. The premise is simple: Most instructors don’t have any education around teaching. Instructional designers know how to design quality courses. So, create a course with a group of designers and let a dozen different faculty teach it. Done and done!
But, of course the analogy then starts to take over. What if you allow instructors to change the course? Some of those new courses will be awesome – amazing even! Others, will be like a darkened bulb bringing down the outcomes average for the department. What if it’s a blinking strand kind of course? In other words, what if it has all kinds of whiz bang media and social interaction? The answer there is that most faculty would need a boat load of instruction just to teach it. (This is why most standardized courses don’t have cool stuff…they just have text, pictures, and some videos. It’s easier to deliver, even though it’s not nearly as engaging for students.) This straight forward approach to design for mass clusters of courses would be the equivalent of an all-white tree. Guess what? I don’t WANT an all-white tree. That’s why last year I spent about 3 hours going through and changing out 4 out of 5 bulbs to a color. I want color. I LIKE color.
Ok you say - so let’s not use instructional designers. Let’s let faculty design all of their own courses! Guess what you get then? You’ll get some lights perched perfectly on the limbs. They will be unobtrusive, casting a healthy glow from the inside of the tree, almost as if the tree itself is on fire. But you’ll also get…well, you’ll get the Griswald tree too. You’ll get lights that look as if they were flung on the tree by a four year old with a slingshot, appearing as if they may fall off at any minute. You will get some bulbs that are significantly dimmer than others. You’ll get 5 reds in a row. You’ll get classes that have nothing but text and no interaction with the professor except for an occasional rant and the final, posted grades at the end of term.
See, I’ve said it before and I’ll say it again. There HAS to be a better way. There has to be a healthy mix of instructional design, subject matter expertise, and personal touches that allow a class to be unique, engaging, and a quality experience in terms of assessment. The school that figures out how to truly mix sound pedagogy with effective delivery and authentic assessment in a media rich, social environment will rule the world.
But until then, we’ll have to take it one light at a time. We’ll have to create the best possible bulb section for our trees or try to create at least tri-color trees that are uniformed. But one day…it will be different.
Oh, by the way, when I landed in Germany my daughter got on the phone. She just HAD to tell me something.
ME: “Hey Peanut!”
ADDIE: “Hi Daddy.”
ME: “What’s going on sweet heart?”
ADDIE: “The middle of the tree is dark again Dad…”
ME: Guttural moaning...
Happy holidays and may your light shine brightly on whatever educational environment in which you teach. Good luck and good teaching.
Dr. Jeff D Borden
Sr Director of Teaching & Learning
Abounding research has confirmed for us the power of instructor presence, immediacy and feedback in online courses. Conceptually, “immediacy” refers to behaviors that lessen the “psychological distance between communicators” (Weiner & Mehrabian, 1968). In practice, and in particular as it applies to online learning and course delivery, the behaviors and practices that generate and sustain instructor immediacy must often occur in scenarios of total or varying degrees of physical distance and separation of course participants. The power to connect with students who we do not physically see or meet with in person can have much to do with the resources available to us to power open lines of communication.
For online instructors, a critical need is to have the capacity for communication, just as we would have in a face-to-face classroom. Instructor-to-student communication around the delivery of content, course information, grades, assessment feedback, etc as well as the need for student-to-student communication around course concepts, discussions or group work illuminates the need for a wide range of tools to facilitate communication exchanges.
How do you foster immediacy and the communication you need to have with your online students?
If you teach with the Pearson’s LearningStudio system, you might use the Announcements and Email tool for asynchronous communication with your students. You might also be using the Threaded discussion tool for targeted areas of communication such as an instructor virtual office, a weekly topical discussion or dedicated group work areas. When synchronous communication is preferred, you might also use Classlive or the Chat tool. All of these measures share the function and outcome of providing means through which communication, interaction and dialogue can occur.
In addition to in-course tools for communication such as those briefly highlighted earlier, the greater Web community offers a wide range of tools you and I can use to generate channels of communication. In this post, I’ll highlight my most recent experience with Google Voice as a communication tool in my teaching.
Google Voice isn’t a new feature from the well-recognized Google family, but it is one that is still unfamiliar to many and is being continually improved. Google Voice offers key functionality you may consider to be helpful to you in communication with your students.
What is Google Voice?
Google Voice can be used to enhance the existing capabilities of your phone, regardless of the phone or carrier you might use. Key features include:
- One Number: Use a single number that rings you anywhere.
- Online voicemail to your inbox like email.
- Transcribed messages.
- Free calls & text messages to the U.S. & Canada.
Directly from the welcome email I received upon signing up, here is a brief highlight of Google Voice and few of its features:
“Welcome to Google Voice. Google Voice gives you a single phone number that rings all of your phones, saves your voicemail online, and transcribes your voicemail to text. Other cool features include the ability to listen in on messages while they're being left, block unwanted callers, and make cheap international calls. We hope you enjoy using Google Voice.”
Your inbox, messages, features and settings can all be accessed and customized from your Google Voice Page (google.com/voice).
Here’s how I am using Google Voice with my students
So, here is a brief overview of my current setup and a few of the ways I use Google Voice to enhance the phone communication channel with my students. First, I did some research on Google Voice before signing up and discovered the ability it would provide me to essentially bridge my current phones and receive voicemails all in one place- my Google Voice Inbox. I decided to give it a try!
During the sign-up process, I was prompted whether I wanted to use Google Voice with my existing mobile phone number or a new phone number from Google. I selected to create a new phone number figuring I could later choose whether I would actually use it. As it turns out, having a Google number, which is free, opened up additional possibilities for me in leveraging Google Voice for phone communication with my students. With a Google number, I was able to now provide my students with 1 phone number where they could call me and I set parameters around what occurs when a student calls that number, such as which of my actual phones (home, mobile, etc) ring when my Google number is called. I was also able to set when my phones can & cannot ring, and the greeting that students hear when I don’t answer their call immediately. Awesome! (More on this later in this post).
Having a Google number has also meant I do not need to provide students with my personal mobile number or home number, but can set Google Voice to route my calls to those personal telephones, if I choose to. When I am not available to answer their call, students hear my personal greeting and are able to leave a voicemail message. That message then arrives in my Google Voice inbox as transcribed text that I can actually read prior to or in place of listening to the voicemail message.
In this example, one of my students called my unique Google number. I wasn’t available to answer at the time of the call, so my student was routed to my personal greeting message and left a voicemail for me. Within seconds, I received the voicemail alert in my Google Voice inbox and was able to read what the student spoke in the voicemail message. The alert in my inbox displays the number from which the voicemail was received, the date/time in which it was received and a transcribed version of the voicemail message. Pretty cool!
A few days after the original setup, I spent a bit of time exploring the Settings area within the Google Voice page and noticed several handy features that could help me even further. As mentioned earlier, I was able to schedule when my phones would be able to ring with a call and when they should not. This means I don’t have to worry about hearing my phones ring when a student calls me in the middle of the night. This is particularly useful when your online students span the country and/or the globe and could be calling you from a wide range of time zones!
Within settings, I also noticed the ability to elect to have text message alerts of new Google Voice activity sent directly to my cell phone. I went ahead and signed up to receive a text message alert on my personal cell phone whenever a student leaves a voicemail message at my Google number. (Keep in mind this is something you can change or cancel at any time). For me, this handy feature has meant I am automatically alerted of a student’s voicemail message even if I am not in my Google Voice inbox or even on my computer. Below is a screen capture of what the text message alert looks like on my cell phone. Notice the transcription of the voicemail message directly within the text message alert on my screen, giving me the ability to ‘see’ the content of my student’s voicemail message without having to be on my computer, log into my Google Voice account, or even ‘listen’ to the voicemail message:
I’ve certainly noticed instances in which the transcription of a voicemail isn’t complete. When this happens, I am able to listen to the actual message to hear my student’s recording directly. I am also able to alert Google of when this occurs and even “donate” the transcript for improvement of Google Voice in the future.
As you’ve probably figured out by now, having a Google number and Voice account means I can also receive text messages from my students, free of charge. Now, I know some of us as instructors are not sold on text messaging with our students. And you’ll find the features of this tool that are helpful to you and you can use those. I’ve found adding a Contacting your Professor announcement in my online course and including within it several parameters for communication and my expectations for students can really help clarify the way I encourage my students to communicate with me, minimizing issues of appropriate methods of contact, etc.
When a student does send me a text message (which I’ve noted in my announcement as an acceptable form of communication), the text message arrives in my Google Voice inbox, just like an email would. It is organized under “Texts” and I have elected to have alerts of new text messages to my Google number sent directly to my cell phone. The alert includes details of the message as well as the text itself, much like the earlier screen capture. And I can choose to respond with a text message, a phone call, or any other method I’d like to use.
Additional thoughts on Google Voice for Communication and Community
I’ve been using Google Voice for some time now and it has become a frequently-used tool in my toolkit as I seek to build effective communication with my students. I’ve also been able to refer my students to sign up for their own Google number to be able to make free nationwide calls to classmates throughout the course and to team members during a group project. All the student messages I receive, including voicemail messages and text messages, are stored within my Google Voice inbox and I can access them from anywhere I can access the internet. I can choose to delete messages once I’ve responded to the student or keep the message stored in my inbox, which means I no longer have to worry about time limits for the storage and retrieval of student’s messages. As a robust phone communication tool to support what I am already doing in my online courses and the ways I seek to foster immediacy, Google Voice is certainly worth a try.
Here are some resources to get you started
If you’re curious about getting started with Google Voice, take a look at this brief introductory video: What is Google Voice? or visit the Google Voice YouTube channel to see more. To sign up, go to google.com/voice. Click “Try it Out” to sign up with a current Google account or to create a new Google Account. Once your Google account has been created and verified, click on and follow the on-screen instructions to get started. Again, you’ll have the option of using Google Voice with an existing mobile phone number or creating a brand new (and free) Google number.
By the way, if on-the-go advanced calling and voicemail functionality is a welcomed addition for your communication toolkit, you can download the Mobile App, which gives you access to your Google Voice inbox and messages right there on your mobile phone. Pretty handy if you like to stay connected while mobile! With the app, I am able to call students or send them text messages that appear as though they are coming from my Google phone number even though I am actually sending them from my personal mobile phone. Pretty cool stuff! Plus, it’s totally free to do within the U.S. and Canada. Currently, the app is available for Blackberry, iPhone and Android powered phones. For more information or to download the app, search for “Google Voice” in your app marketplace.
Until next time, I wish you the best in your courses. Be sure to share a comment with us via this blog if you use Google Voice or another handy tool for communication with your students!
Academic Trainer & Consultant
Assessment & Analytics Group | Academic Training & Consulting Team (ATC)
Wiener, M. & Mehrabian, A. (1968). Language within language: Immediacy, a channel in verbal communication. New York: Appleton-Century-Crofts.
In case you missed it, Pearson made an announcement a few weeks ago followed by some serious marketing during EDUCAUSE about one of our newest products, OpenClass. OpenClass is “breaking down barriers and transforming the learning environment,” says Adrian Sannier, Pearson’s Senior V.P. of Learning Technologies. Why? Because, in short, it’s free. As in really free – it’s hosted in the cloud, so there are no hosting costs. There are no licensing costs. In fact, if you’re a school with Google Apps for Education you can start using it right now. Free. Just click here.
But my post today is not to talk about OpenClass directly. I’m not going to try to sell you on it or demonstrate it or talk about all of its amazing features. Instead, I want to talk about some of the reactions we’ve seen to OpenClass. It’s my contention here that we in the educational world (and in our consumer culture in general) have become so suspicious of the word “free” that we can’t possibly believe that something really could be free. Once bitten, twice shy, right?
The Chronicle of Higher Education and Inside Higher Ed both wrote articles on OpenClass following Pearson’s press release. Fine articles – balanced points of view, a few questions that need to be answered, and so on. And kudos to them for taking on these national discussions on the idea of a free LMS.
The heart of the matter, though, comes down to the comments posted by many of those publications’ readers. I’m going to share a few of those comments here, not to poke fun at the authors or to say that they’re wrong or that there’s anything wrong with what they posted. I’m all for free speech and for potential end users to challenge Pearson to deliver on its promises. What I want to point out is the amount of cynicism we see in the world of Learning Management System adoption. Let’s start with a sample of comments (unedited):
- free-hosting sounds great, ...but at what price? what sorts of idiosyncrasies and limitations will this cloud-based LMS have?
- I can almost picture the pop-up ads in OpenClass--"wouldn't you love to be able to [insert Learning Studio feature not present in OpenClass] ?"
- Nothing is free!
- this may not be as “free” as it looks. For a campus to integrate an LMS into their academic mission, it takes time, money and cooperative relationships with faculty.
- I question how free OpenClass really will be. Pearson is a for-profit publisher and, to use OpenClass, I suspect they will have customers to use their textbooks under the guise of an integrated learning platform…I sense there are many strings attached to this so-called free platform
- Like other people, I’m also wondering how “free” this can really be. LMS adoption is a costly process -- in terms of time and money. Plus, a newer LMS is bound to have more problems than better-established LMS that have been evolving over a decade or more.
- While it is nice that "free" (as in gratis) is referenced, it is certainly not Free (as in libre). Of course, one does have to wonder how long the "free" part will last...
- Can we please define "free?" It seems very limited to think of costs only directly related to hosting the application(s) and maintaining the hardware. Is it free in the sense that open source software is free (e.g. free as in speech vs. free as in beer)?
- Pearson could cancel OpenClass at any time, or not fix bugs or insert ads or just stop adding any features or upgrades, and there is nothing anyone can do about it - you're locked in.
- Yeah..it's good.But would you mind if i ask you a question? well I am 31 years of age from Tanzania East Africa I am looking for a sponsor for my master's degree any where can you help please..
Okay, maybe that last one’s not really on point. But I think you get the picture I’m painting here: people are surprised, suspicious, and even (at times) hostile toward the idea of a free LMS. Several readers/commenters act as if Pearson is a drug dealer, using OpenClass to give people a taste, getting them hooked, and then causing them to take out second mortgages on their universities just to stay in the LMS.
Not true! Look, I’m biased here. I see that. I work for the company that makes OpenClass. But I’m also an academic, have years of teaching experience at the university level, and have years of experience with a variety of LMSs. So I know where these readers are coming from. Nothing is ever as free as it seems, right? There are always hidden costs. We, as a consumer culture, have become desensitized to the word “free” because, as one commenter so astutely wrote above, “Nothing is free!”
For example, there are a lot of other LMS offerings out there that purport to be free, but limit you in terms of the number of courses you can create or the number of students you can enroll. OpenClass is not that. There are other LMSs that provide the backend code for free so that you can essentially create your own LMS using their code. Except that you have to pay for hosting – even if that means just a lousy few thousand dollars on some servers and routers. OpenClass is not that.
In short, these other “free” offerings have brought out the cynic in many of us, that anything that says it’s free can’t possibly really be free. We’ve been burned too many times before.
OpenClass is free. You can get it for free out of the Google Apps Marketplace, create a course, enroll students, and run with it. You can create ten thousand courses with 100 students in each. Let your imagination run wild. It’s free. If you still don’t believe me, try it out. Let me know what you think.
Rob Kadel, Ph.D.
Pedagogy & Training Group Supervisor
Academic Training & Consulting
In the 90′s Russell wrote the first and likely most well read defense of online education. The piece, “No Significant Difference” was well written and well received. But it was Twigg’s follow up, “Beyond No Significant Difference” that was an eye-opener for some. Even back in 2001, Twigg discovered what many now know to be true. Outcomes are more easily tracked and often achieved in online classes than they are in their on-ground counter parts.
Now I know what you’re thinking. Oh, the online guy is going to tell us how great online is…but hold on. I know it’s still not the accepted, common-sense paradigm that many would hope it to be. Just this week I read an article about how Ball State faculty are highly suspicious of online education. While I feel that many faculty are simply uneducated about it and several predispositionary thoughts are actually faulty reasoning, it doesn’t change the fact that online education is still seen, by many traditionalists, as the ugly duckling of academia.
So I get it. Really, I do. I hear it all the time. I don’t agree with it and believe I can vigorously and credibly argue the points, but I get it. So rather than my pushing my own biases about the importance and validity of online education, I wanted to share some other’s insights.
Two weeks ago, I attended our President’s Round Table. It was in an extremely beautiful part of South Carolina – it was one of only 4 states I had never set foot in. The conference itself was quite amazing. Not just the food or setting (although Kiawah Island is quite impressive), but the ideas, innovations, and operational issues discussed were truly inspiring. We had speakers from Harvard, Microsoft, and best selling authors talk about the trends in education, technology, and online learning which created wonderfully rich conversations that will shape the future of our business.
As well, the audience was not only ready to listen, but ready to share. It was inspiring to watch Presidents, Directors of Online Learning, Provosts, and more brainstorm for, listen to, and constructively critique ideas in and around how to best serve students. State institutions collaborated with for-profit schools who communicated with religious colleges who listened to community college leaders…it was fantastic! Again, these leaders are ready to fight the good fight!
But what was actually most amazing to me, in the midst of all of the creativity and innovation, was a simple truth that was stated by several of these school leaders. It started with one simple statement and then was reiterated several times throughout the week. It started during a panel discussion where a Director of Online Learning simply said,
“We’ve found our online numbers to be well above our on-ground counterparts. Not only have we found that the research about online courses producing and measuring better outcomes is true, but our retention and faculty survey numbers blow the face to face classes out of the water! We’re double digit points above them…”
What?!? Is that possible? Someone from the crowd actually asked him to repeat the off-the-cuff remark. But when he did, a few other Presidents expressed the same thing. Online numbers for retention, satisfaction, and test scores were significantly higher than on-ground classes teaching the same materials.
So, over the next two days, I asked people at our meals and during our breaks if they had similar experiences at their schools. Many did. Not all, but of the 30-40 leaders present, I heard at least 15 say that they had better numbers online than on-ground.
So, while some are trying desperately to explain away the research of the past two years as poor studies with bad analysis or poorly constructed tests, there is something they cannot simply dismiss…online learning works better in some contexts, with some students, with some disciplines, with some programs, and with some content, than face to face learning. Period.
Good luck and good teaching.
My friend and colleague, Luke Cable, posted about a year ago, "Thoughts on the iPad in Higher Education." And his post still rings true to me. The iPad and other tablet devices are great canvases for putting one's own thoughts into a particular application. Sketch, draw, brainstorm, create mind maps, videos (now on those tablets with cameras), and so on and so forth.
Recently, I decided to take the plunge and buy an iPad 2. For those of you not in-the-know, it is very similar to the original iPad except for having a faster processor and added video capabilities. Oh, and it comes in white as well as black. That's important.
In all seriousness, it's been a great tool. But I'm not (yet) convinced that it is the greatest tool for fully online education.
Now, full disclosure here: I am an Apple fanatic. I owned one of the first Macintosh computers and have been a Mac user ever since. In my small family of four, we have three Mac laptops as well as Apple TV, iPhones, iPod Touches, and so on, ad nauseum.
But as an online educator, I'm still not convinced of the iPad's usefulness in my work. I'm not talking about issues such as its lack of flash support, which by the way are being solved day-by-day by Web publishers either creating flash-free apps, switching to HTML 5 in Web site creation, or even using the newly announced Adobe Flash Media Server applications, which can deliver flash media across multiple platforms, including iOS.
What I have found are two areas where the iPad fails me in interacting with my students. First, support for locally stored files: I need to be able to download, for example, student papers directly to my tablet, edit them, and then upload them back to the learning management system I use for my teaching. The iPad's cloud-based solutions for document management include the MobileMe iDisk (soon to be replaced by iCloud) and WebDAV access to remote servers, a protocol that not all universities and colleges support.
Second on my wish list: full integration with Microsoft Office. Yes, as an Apple fan, I said the "M" word. But let's look at reality: Microsoft Office has become the de facto solution for creating documents, spreadsheets, and presentations. Maybe that will change someday; but until it does, I need more than just a conversion to Apple's Pages app to read my students' papers, track changes, add comments, and so on. (I have some attorney friends who have said the same thing. They find it impossible to adequately mark up drafts of contracts and briefs on any current tablet device.)
Why, you may ask, don't I just switch my loyalty from the iPad to an Android tablet or wait for a Windows 7 (or 8?) enabled tablet? My logic is simple, and similar to that in the above paragraph: the iPad has become the de facto solution for tablet devices. With thousands of quality-controlled apps in their app store, Apple has done what no other tablet maker has been able to accomplish: deliver quality experiences on superior hardware that people are willing to spend several hundred dollars to acquire. Even more so than the iPhone among smart phones, I contend that the iPad is the ubiquitous tablet of choice. No other tablet comes close to bringing so much to such a large market.
That said, for my online teaching, it still falls short. I'll stick with my trusty MacBook, thanks. At least for now, until the next iteration of tablet operating systems and apps can address my needs. Of course, at the rate that Apple releases these things, that might be next month!
Rob Kadel, Ph.D.
Academic Training & Consulting
Well, here we are. Nearly to the end of our journey. Almost there. We’ve finally made it to Hallmark #9. It feels a bit like we’ve been climbing a Colorado 14ner and we’re nearly at the top. We have discussed the first eight Hallmarks of Quality from the Interregional Guidelines for the Evaluation of Distance Education (Online Learning) so we have only the final Hallmark to get through before our ascent to the peak is complete: Integrity.
When I first saw that integrity was the last hallmark my first thought was “Why list Integrity last?”. Was it an afterthought? Doubtful. Was it just part of a natural progression that landed it at the end? Maybe. Or was it specifically selected to be the end cap of what is seen as important, integral and paradigm shifting set of Hallmarks? This is where I place my bet. If for no other reason than this is the only Hallmark with both an asterisk to further information AND an emboldened note within the (already lengthy) first point of analysis/evidence.
While the official language of the hallmark is simply “The institution assures [ensures?] the integrity of its online learning offerings*”, let us make no mistake: this is not just about integrity in the general sense, meaning whole, undivided, unified, consistent or sound. This is about academic integrity.
So let’s take a look at what’s contained in Hallmark #9. First, the asterisk in the statement refers to the WCET paper Best Practice Strategies to Promote Academic Integrity in Online Education. This paper lists academic integrity best practice strategies in categories concerning the institution, curriculum and assessment as well as faculty and student support. It’s worth the quick read for high level tips that can be used to get you going or to validate what you may already be doing. Three of the hallmark’s analysis/evidence items are short and essentially mention that faculty, online orientations and institutional policies should emphasize and integrate academic integrity into their teachings and practices.
The first, most prominent and most impacting analysis/evidence item is this one:
“The institution has in place effective procedures through which to ensure that the student who registers in a distance education course or program is the same student who participates in and completes the course or program and receives the academic credit. The institution makes clear in writing that these processes protect student privacy and notifies students at the time of registration or enrollment of any projected additional costs associated with the verification procedures. (Note: This is a federal requirement. All institutions that offer distance education programs must demonstrate compliance with this requirement.);”
The statement is composed of two parts: a) verifying the student and b) making the verification process, procedures and costs known. In part a), the key words are “effective” and “ensure”. “Effective” is a word that is open to interpretation by that fact that it doesn’t imply perfection, but only a high level of efficacy. And this is appropriate; pretending perfection is attainable is a way to quickly not get anything done. In contrast, “ensure” does carry the weight of making certain without exception or with guarantee. While these two can seem to be in opposition what we can strive for here is when we believe we’ve determined that the student registered for learning is the student who performed the work, that this is indeed true. No false positives or false negatives. Part b) strikes me as an attempt to included students in the effort of keeping academic integrity. The phrase “makes clear in writing that these processes protect student privacy” implies to me that there is a specific effort to communicate to the students that academic integrity is not solely about catching the “bad” cheaters, but protecting the quality, original work that many students choose to do. And that many times protecting good things has a cost. Most importantly though, it needs to be shown and communicated (and be true) that academic integrity is not first a matter of punishment but rather a matter of having a posture of quality between the students and the institution.
This hallmark can be a tall order, potentially a costly order. (For information on types, costs and thoughts on some student authentication / verification systems, see Jennifer’s blog on the student-centric Hallmark #7.) But this hallmark has merit and it makes sense. If students are not who they say they are and their work is not what they say it is, then where’s the purpose in anything that we’re doing (be it on-ground or on-line or somewhere in between)?
Imagine for a moment that you had just been given mid-field season tickets to your favorite sports team. And there you sit, at the season opener, soaking in the atmosphere: 70,000 excited people coming together to cheer a common cause; an immense venue where little expense was spared; the long tradition of the team and sport displayed; players, coaches, owners, concession workers, and so many other people who have worked so hard in the preceding weeks and days to prepare for this moment in time: The Game.
But now imagine that as the teams take the field and the crowd is roaring, the players do only as they please. They ignore the rules and the referees. They high-jack the scoreboard, steal the ball and generally create helter-skelter. Wouldn’t that be maddening? You might think “What in the world was all the effort and preparation for?”. And, truly, it would make a mockery of the sport, the event and everyone involved.
So too it is when learners choose to (or inadvertently do) cheat; it makes a mockery of the class, their peers, the institution and most importantly, their own learning. Is it the truth that no matter what we do, students are still going to try to (and find ways to) cheat? Definitely. Will students compromise their academic integrity by ignoring what resources are made available to them? Probably some. Do either of these things mean that we should stop (or lessen our efforts in) striving for academic integrity? Absolutely not. I think this line from Martin Luther King Jr.’s book Strength to Love says it well: “The ultimate measure of a man[/woman] is not where he[/she] stands in moments of comfort and convenience, but where he[/she] stands at times of challenge and controversy.” If we were to stop furthering our efforts toward academic integrity, even as it becomes more difficult to do, then we risk making a mockery of it all. Hallmark #9 reminds us that we must continue to make purposeful efforts to do establish cultures of academic integrity on our campuses.
Before we sign off from our ten week foray into The Nine Hallmarks of Quality, it seems only fitting that we should take a quick look back over where we have been and what the big picture looks like.
It was back at the end of June when Jeff first introduced the discussion of the Hallmarks and what they might mean for online education. Jeff talked about the consistency and transparency that these Hallmarks can bring. He asserted that these Hallmarks will give online educators another strong foundation to stand upon in the debate around efficacy of online vs. on-campus. But I think the most foretelling statement of Jeff’s was this: “They will illustrate what many of us have already researched and know: Online learning works when it is strategic, designed effectively, and measured evenly.” And as we’ve gone through the Hallmarks in detail I think we can see how they echo this underlying paradigm. One could say that the essential thought running through the Hallmarks is: Be purposeful and make it known.
When I look at the list of hallmarks as the pieces they are and how they come together as a whole, I see them fitting into four categories:
Hallmark 1, Hallmark 2 and Hallmark 3 fall into this category. Notice some of the words in the definitions of these Hallmarks: incorporated, appropriate, integrated. The theme here is for online education to be a part of who you are, for it to be infused in your character. It’s not an addition nor an appendage.
Do It Well
Hallmark 4 asks for rigor in creating the curriculum for online learning (why would we do it any other way?) and Hallmark 5 asks for continual improvement of the curriculum. If you’ve made the decision to have online learning interwoven into the fabric of your institution, then you need to make the purposeful choice to do it well. Saint Francis de Sales said it succinctly: “Be who you are and be that well.”
Set Your Team Up for Success
Where faculty meets students is where the rubber hits the road of learning. To this end, Hallmark 6 and Hallmark 7 are directed toward the critical subject of supporting our faculty and students with resources, training and information. Give them the things they need; get rid of things that will get in their way.
As we’ve mentioned in the first part of our Hallmark 9 blog today, if all the other Hallmarks are fulfilled (the set up for “game time” is done well), but learning doesn’t happen with integrity, then it can all be for naught.
These categories and these Hallmarks together form a cohesive picture of successful online learning that is strategic, designed effectively and evenly measured.
On behalf of the Academic Training and Consulting team, we hope that this blog series has been beneficial to you, give you some insights, and helped to frame the future of online education as education that will lead the future of learning. Next week we’ll be back to our regularly scheduled blogging, so look for some exciting topics in the world of education and technology in the coming weeks!
Academic Trainer & Consultant
We are nearing the end of our series on the Interregional Guidelines for the Evaluation of Distance Education, and this week we will consider the 8th Hallmark, which concerns resourcing. These Guidelines can be found here as presented by the Middle States Commission on Higher Education (MSCHE).
Hallmark #8 reads: The institution provides sufficient resources to support and, if appropriate, expand its online learning offerings.
I think of this Hallmark as essentially saying: “put your money where your mouth is.” In our discussion of the previous Hallmarks, we’ve seen that online education must be integrated into an institution’s mission and operations, and must have strong student and faculty support. Hallmark #8 supports those previously discussed ideas by stipulating that the institution must actually provide the budget and resources to make their online goals happen. We know that resources are tight everywhere, but dedicating resources to a distance education program is an important way to show that the institution values that program, and also funds the training and services that help set it up for success.
An institution seeking accreditation (for example, by MSCHE), will be asked to include evidence documenting how they are meeting the 9 Hallmarks for their online education program. MSCHE provides two areas of evidence that would allow an institution to demonstrate that they are meeting Hallmark #8:
- The institution prepares a multi-year budget for online learning that includes resources for assessment of program demand, marketing, appropriate levels of faculty and staff, faculty and staff development, library and information resources, and technology infrastructure;
- The institution provides evidence of a multi-year technology plan that addresses its goals for online learning and includes provision for a robust and scalable technical infrastructure.
These seem pretty straightforward, as essentially the institution needs to demonstrate that they have allocated sufficient resources to support their online educational goals. To truly support their online venture, they need strong plans for marketing, faculty and staff support (including items laid out in this blog earlier), student support (as discussed here), a robust online curriculum (see discussion here), etc. The institution must also show commitment to the technological aspects of online course delivery, including the technical infrastructure and a great LMS, such as Pearson LearningStudio (not so subtle hint!).
Working through these items should help the success of an online program and allow the institution to truly “put their money where their mouth is.” Good luck and happy budgeting!
– Gail E. Krovitz, Ph.D. –
Director of Academic Training & Consulting
MSCHE (2011) Interregional Guidelines for the Evaluation of Distance Education Programs (Online Learning). Retrieved online from: http://www.msche.org/publications/Guidelines-for-the-Evaluation-of-Distance-Education.pdf
This is the fifth in a series of posts on the 9 Hallmarks of Quality for Distance Education programs that were developed by the Council of Regional Accrediting Commissions (C-RAC) earlier this year.
The institution evaluates the effectiveness of its online offerings, including the extent to which the online learning goals are achieved, and uses the results of its evaluations to enhance the attainment of the goals (MSCHE, 2011).
As institutions seek to develop a culture of assessment that meets increasingly stringent accreditor requirements, a myth prevails that a pre-defined template exists that elegantly solves this ill-structured problem. The truth is that accreditors defer most of the responsibility to the institution who must set their own mission (Hallmark #1), program goals, and individual course outlines that provide the learning experience required for students to demonstrate mastery of the curriculum. They evaluate the extent to which a school has developed an assessment approach that measures curricular and instructional effectiveness and shows how data is used to further the continuous improvement of student learning.
While this may be frustrating to read, there are definitely patterns and best practices that scholars of teaching and learning have developed which synthesize characteristics of successful accountability programs.
First, institutions must be purposeful in their assessment program which means there is a plan for what data to collect and how it will be used to improve student learning. A holistic assessment approach includes both formative and summative assessment within courses and at the program level so students have the ability to remediate their weaknesses before it’s too late. Programs new to assessment usually begin with evaluation of program level goals and move into course level assessment as they mature. Ideally, most assessment can be embedded within the course so faculty of record can gather the data as part of their ongoing student assessment workflow.
This leads to a second major challenge in that perfection can be the enemy of good – or even the ability to get better. Our partners often tell us they’re not ready for assessment and we see academic leaders go through numerous models in their heads without ever actually implementing anything. Getting started creates informed use which yields better questions and action plans going forward.
As we consult on assessment and methods to integrate technology into the outcome management process, we nearly always expose what seem like obvious gaps in curriculum and instruction. This is part of the continuous improvement process and the important thing is to remedy that gap and to then look for the next most critical issue to resolve.
Finally, I’ve often heard assessment experts encourage academic leaders to actually scale back the volume of data they’re collecting. As mentioned earlier, data is meaningless unless you take the time to analyze what you’ve gathered to diagnose gaps and to implement improvement action plans to address the gaps. So, you might consider assessing random samples of student artifacts instead of trying to assess every student each term or you can assess all students against an outcome but only evaluate the outcome every two years.
Our consultants have developed the following modules to support educators in meeting requirements for Hallmark #5.
- Creating a Culture of Assessment
- Writing Quality SLOs
- Rubric Design
- Curriculum Mapping (Institution > Program > Course)
- SLOs and Impact on Course Design (Curriculum mapping within a course)
- Fostering Faculty Ownership of Campus Assessment Culture
- Closing the Loop - Ensuring that SLO Data Impacts Curriculum & Instruction
In addition to the purposeful management of student learning, Hallmark #5 also requires institutions to monitor and set goals for both in-course retention and student persistence through a degree program along with the effectiveness of an institution’s academic and support services (MSCHE, 2011). Again, our consultants can work with you to develop custom reports to track and monitor progress for retention and persistence with student activity and completion data from the LMS. We can also help to identify at-risk students to support the requirement to measure effectiveness of academic and support services although this component certainly requires additional offline analysis of process and services at the institution.
Let us know if you have recommendations for any additional content area we should develop or if you’d like more information on our consulting services.
Middle States Commission on Higher Education (MSCHE). (2011, February). Interregional Guidelines for the Evaluation of Distance Education Programs (Online Learning). Retrieved July 18, 2011 from http://www.msche.org/publications/Guidelines-for-the-Evaluation-of-Distance-Education.pdf
Brian Epp, M.Ed. | Assessment & Analytics Group, Academic Training & Consulting | Pearson eCollege