Online Blogucation
29Feb/120

Actionable data for improving student learning and inter-institutional comparability – Can we have both?

An article titled Searching for the Holy Grail of learning outcomes from Inside Higher Ed (IHE) caught my attention last week. The article discusses the elusive quest for data that illustrate the value add provided by a student’s progression through a degree program at a particular institution.

Because the Collegiate Learning Assessment (CLA) purports to provide this value added score it is fast becoming the market standard and the author of the article goes on to cite a number of reasons why this coalescence should concern us.

First, here’s some background in case you’re not familiar with the CLA. The largest market for higher ed accountability is undergraduate general education where the focus is on things like a students’ ability to think critically, to read and write effectively, or to solve problems. As I summarized back in 2009, “we now have public accountability campaigns including the Voluntary System of Accountability (VSA), the University and College Accountability Network (U-CAN), and Transparency by Design which serve the public, private not for profit, and private for profit institution types respectively” (McKay Epp, 2009b).

Because the focus of the IHE article was on public institutions, the VSA is the accountability campaign that was highlighted. As background for those not familiar with the VSA, it

allows participating schools to choose among three assessment instruments that are administered to students with the goal being to indicate student proficiency in the areas of reading, writing, and critical thinking. One of these tests, The Collegiate Learning Assessment (CLA), actually tests a sampling of entering freshmen and graduating seniors and correlates student scores to incoming student SAT or ACT scores in an attempt to show the value add provided by an institution over the course of a degree program. (McKay Epp, 2009a).

One of the most interesting critiques of the CLA in the article is the high correlation between it and the SAT. Olds states that “the amount of variance in student learning outcomes after controlling for SAT scores is incredibly small” (2012). The author goes on to say that “Most institutions’ value-added will simply be in the expected range and indistinguishable from each other. Hence, why bother with the CLA” (Olds, 2009).

While the author lists several alternatives to the CLA (which are worth reading), what I found most interesting was the discussion about the struggle that institutions have to find data that create actionable insights for improving student learning. For nearly four years I’ve been working with institutions to help them implement Pearson’s Learning Outcome Manager (LOM) which is a technology enhanced outcome management system.

LOM does an excellent job of providing actionable data to faculty and administrators on student performance against established learning outcomes for online and blended courses and programs. Because outcomes are associated to graded course assignments, it helps ensure that evaluators are seeing students’ best effort and when done well it minimizes additional workload for faculty. The challenge is that LOM generated data is so targeted to individual professors or to a particular course that its results can’t easily be used for inter-institutional comparability.

While I believe a majority of educators would agree that the most important reason to work in assessment is the desire to improve student learning, I also recognize that the demand for data on inter-institutional comparability will not go away. This article provides some interesting alternatives to standardized assessments such as the CLA which I think could work in tandem with data generated from systems like Pearson’s LOM to provide a win-win for the assessment community.

Works CitedMcKay Epp, B. (2009a). Implementing a Technology Enhanced Outcome Management Strategy on Campus that Produces Substantive Improvements in Curriculum, Instruction, and Student Learning. EDULEARN09, Barcelona. Retrieved from http://library.iated.org/ view/MCKAYEPP2009IMP

McKay Epp, B. (2009b). Improving Student Learning: Thoughts and Reflections. Educator's Voice, 10 (3). Retrieved from http://www.pearsonecollege.com/Newsletter/EducatorsVoice/EducatorsVoice-Vol10Iss3.learn

Olds, K. (2012). Searching for the Holy Grail of learning outcomes. Inside Higher Ed. Retrieved February 20, 2012 from http://www.insidehighered.com/blogs/globalhighered/searching-holy-grail-learning-outcomes

Brian Epp | Supervisor, Assessment and Analytics Group | Pearson eCollege

23Feb/120

Creativity – why should a business care?

-

I’ve had creativity on my mind the past few weeks. Maybe because I’m currently offering my students an assignment to create a class content-related sign, inspired by a 2009 article in National Geographic that included the sign shown here. (Who knew that dung beetles have the right of way?) This is a fun assignment that brings out some creative and funny work from my students. Anyway, regardless of the cause, I’ve been thinking about the importance of creativity in education, and I recently watched an interesting version of Ken Robinson’s talk on Changing Educational Paradigms (embedded above; you can also check out his famous TED talk Do Schools Kill Creativity?).

As Ken Robinson discusses, there are many reasons to include creativity in education. But what I want to explore here are some of the “business” reasons that creativity is important. Specifically, I want to point out two interesting studies completed by IBM and the American Management Association (conducted in conjunction with the Partnership for 21st Century Skills, or P21) that demonstrate the value that the executive workforce puts on creativity.

The IBM study included more than 1,500 Chief Executive Officers from 60 countries and 33 industries worldwide, and found that 60% of CEOs cited creativity as the most important leadership quality over the next five years (IBM, 2010: 24). They feel that “creative leaders invite disruptive innovation, encourage others to drop outdated approaches and take balanced risks. They are open-minded and inventive in expanding their management and communication styles, particularly to engage with a new generation of employees, partners and customers” (p. 10).

In another example, a survey of managers and business executives conducted by the American Management Association found that 75.7% of respondents felt that critical thinking, communication, collaboration and creativity (the four Cs) “will become more important to their organizations in the next three to five years, particularly as the economy improves and organizations look to grow” (AMA 2010: 4). Those responding felt that the four Cs will be particularly important in keeping up with global competition and the pace of change.

When looking specifically at creativity and innovation, 61.3% said that this was among the most important skills in helping grow their organization, and 31.8% said it was an important skill (p. 5). In terms of their employees, 46.9% felt their employees had average skills and competencies in the area of creativity and innovation, 14.2% were below average, and 31.6% were above average (p. 5).

These studies should provide food for thought on how creativity benefits business, and that students who are encouraged with creative approaches in education may have an advantage in the future job market. So get creative with the ways that you can include creativity in your own teaching or course design!

– Gail E. Krovitz, Ph.D. –
Director of Academic Training & Consulting

American Management Association and Partnership for 21st Century Skills. (2010). AMA 2010 Critical Skills Survey. Accessed online at http://p21.org/storage/documents/Critical%20Skills%20Survey%20Executive%20Summary.pdf

IBM (2010). Capitalizing on Complexity: Insights from the Global Chief Executive Officer Study. Accessed online at http://public.dhe.ibm.com/common/ssi/ecm/en/gbe03297usen/GBE03297USEN.PDF

14Feb/122

Do As I Say, Not As I Do

Conference attendees sleeping

These people still clapped at the end of the session...

How low is your conference bar set these days?  What does it take to make your experience "worth it" anymore?  Is it 1 good keynote and 1 good session?  Is that enough?  Maybe it's a solid pre-conference workshop and two good sessions.  Or is it even less?

I go to 30-40 conferences (led by educators) each year.  Typically I present a keynote address, a few workshops, or possibly a pre-conference session, but I certainly have plenty of time to see and hear a lot of other presenters.  This also means that I end up eating lunch or an occasional dinner with dozens, if not hundreds of strangers.  So, I've been doing some research around the gambling that takes place at conferences. No, not dice in the back of the kitchen or inviting strangers back to a hotel room...(Those are the tech conferences.)  I'm talking about the conference session roulette that everyone takes part in.  Come on lucky session #4...daddy needs a new educational game!

Some conference attendees "double down" on their bets.  Good move.  I watch as more and more often, session participants sit in the back of the room.  They give the presenter(s) about 3 minutes to "hook" them.  If there is no "hook" then out the door and off to another session they go!  Two for one sessions - nice!  And, most conference presenters are making it hard too.  It seems that the "catchy title" is the order of the day, regardless of whether or not the session will actually provide value.  Sprinkle in Web 2.0, or YouTube, or Serious Game and you've got a session title that will make people do a double take!  Come on Serious Games for YouTube and Facebook via Web 2.0 in the Classroom...Daddy needs a new assessment idea!

In my extremely unscientific research, here is what I looked for.  Great sessions (regardless of the identified mode), meaning keynote addresses, workshops, pre-conference, poster, and panel sessions were all game.  I looked for a few simple indicators to determine a successful presentation.

  1. Great content - this is usually determined by the "buzz" after the session and often corresponds with the number of questioners who stick around to talk.  (My personal research seems to indicate that 3 people will stick around regardless of how good or bad a session is.)  This also includes "buzzing" conversations that follow the session to lunch.
  2. Great presenters - these are definitely harder to find, but my indicator here was pretty simple.  Who, or better, how many (in the audience) was paying attention to the presentation?
  3. Great interaction - this one is tough for me.  A lot of conferences are demanding audience "participation" these days.  My problem?  Often the audience members are not subject matter experts, they are simply professors who enjoy sharing their opinions (which is why we're professors, right?) or worse, they simply want to play devil's advocate throughout the session.  So, in both of those situations, other audience members come away feeling like the session was useless.  However, when interaction with multiple audience members takes place regularly (not simply because an audience member forced a question in), it should be noted.

So, after months of tallying on my iPad or iPhone -I love you Evernote - I have some informal numbers.  This is from 22 conferences, 103 sessions, and includes a lot of conference goers...I have no idea how many.  I should also mention that if I didn't go to the presentation, but simply heard about the presentation after the fact, it was not included here.  (I wonder sometimes if those conversations are legitimate...it's like the guy in high school who was always trying to convince you the swimsuit models showed up to every party JUST after you left...)  Anyway, here you go:

  • 92/103 sessions had poor content, which means 11 sessions had great content.
  • 99/103 sessions had poor presenters, which means 4 sessions had great presenters.
  • 99/103 sessions had no audience interaction, which means 4 session had great interaction.
  • 2 sessions had both a great presenter AND great content (although no interaction).

For those of you scoring at home, that does not even begin to approach an 'F'. Even in aggregate, less than 16% of the presentations I attended were...well, quite frankly they were pretty bad.

Conference attendees paying attention to everything but the speaker

At least I got all of my email answered during this session

Let me give you one fresh example from a conference I attended in December.  There were 75-100 people in the lecture style, tiered room.  I was in the very back, at the top, looking down on the presenters and audience members (I was preparing for my session in that same room, which was next.)  Let me describe for you the middle row of about 25 people.

  • 3 were visibly asleep
  • 4 were checking email on their laptops
  • 6 were checking sports sites - mostly fantasy football on their laptops
  • 10 were using their phones (texting for help perhaps?)
  • 1 was writing on a notepad
  • 2 were passing notes back and forth to each other

It doesn't seem to matter what the topic is, what kind of conference it is, or who the speaker / audience members are, these sessions don't seem to be very helpful.  When I attended my own discipline's Communication conference last year, with people who explain to college students how to effectively communicate a message, there was no difference. When I went to a K-12 conference with teachers who certainly need more energy and enthusiasm to reach younger people, it was no different.  When I went to International conferences, it was no different.  (In fact, it was often worse as many of those conferences are made up of "conference papers" - essentially a person sitting in front of the audience reading a research paper out loud...seriously.)

OK...so, enough of the agonizing landscape.  You get it.  In fact, many of you are probably starting to develop a twitch as I've reminded you of things you would prefer to forget.  But here is my big question.

Why is it a surprise that education is having such trouble reaching students?

Apparently, we (educators) have a difficult time communicating with each other.  How can we possibly expect to communicate effectively with our 1, 2, and sometimes 3 generations younger students?  Why don't we apply what we know to work?  Why don't we use what we know to be helpful?

Tell, Show, Do, Review, and Ask in a multi-modal, multi-nodal way and we'll reach a LOT of people.  Why don't we do that?  Use ethos, pathos, logos, and mythos (if you're dying to think about it old-school) and we'll reach a LOT of people.  Why don't we do that?  Incorporate serious games, focus shifts, multimedia, and interactive strategies and we'll reach a LOT of people.  Why don't we do that?

I truly believe that we are our own enemy here.  I KNOW that there are some really creative, innovative, strategic instructors out there who are doing great things...but when they get to a conference to share it, they get very uptight.  The idea of presenting to peers is quite intimidating for many, so those ideas never really get a chance to shine.

Then, there are the conference submission boards who miss out on great stuff.  They don't seem to read or review survey results from previous conferences, giving preference to people who get super positive comments, having thereby illustrated that they have great content, are a great presenter, or include interaction effectively.  I watched a professor at Online-Educa Berlin present a fantastic workshop on rubrics.  She was poised, dynamic, and her content was top notch.  When I told her that she should give that session at some conferences back in the USA, she explained that she tried over a dozen times and never got accepted.  Something about the presentation just wasn't "sexy" enough for the committees, even though I watched her knock it out of the park in Germany.

So let me finish with this.  Let's change the way conference presentations currently run.  Let's all take a pact.  When we're given the opportunity to share our clever, creative, innovative, effective, or useful ideas from our classes with our colleagues...let's not blow off the performance until the plane rideLet's not forget what goes into a good presentation - effective nonverbals, logical reasoning, and passionate verbals.  Let's include some of the "cool" factor when we can, to illustrate the concept.  Let's not forget the power of storyLet's agree to NEVER, EVER, under ANY circumstances READ our notes or (worse) READ our PowerPoint to the audience again!

We can do this.  It's not like we don't know how audiences respond most effectively.  We know that the lecture is one of the poorest ways we can communicate if we want our audience to retain, comprehend, and be engaged.  We KNOW what it takes.  So, let's just change it.  Yes, that simply, let's change our conference behavior.  Let us never again imply that what we say and what we do are not supposed to be joined at the hip.

Good luck and good teaching...and good conference-going!

(BTW - did anyone notice the ironic metaphor for education here?  Boring lectures, audience members not paying attention, little audience interaction, etc?  Hmmm...I guess that's another blog.)

7Feb/120

The New Leadership Alliance for Student Learning and Accountability (www.newleadershipalliance.org) recently released its report, Committing to Quality: Guidelines for Assessment and Accountability in Higher Education. The report suggests that higher education institutions can use the guidelines to help them answer the question, “Are our students Learning?” and contends this is the fundamental question underlying the work these institutions do to prepare students for success.
The guidelines in and of themselves are not really anything new to those of us involved in student learning outcomes assessment: 1. Set Ambitious Goals, 2. Gather Evidence of Student Learning, 3. Use Evidence to Improve Student Learning, and 4. Report Evidence and Results. Indeed, these guidelines form the foundation of most campus-level assessment activities.
What caught my attention in the report was the following statement included in the description of Guideline #2.
Evidence of how well students are achieving learning outcomes (i.e., “What is good enough?”) against externally informed or benchmarked assessments or against similar colleges and universities, where appropriate and possible, provides useful comparisons. At the same time, it is critical to keep in mind that the objective of comparison is not ranking but improvement.
This seems to be one of the biggest hurdles we face when trying to evaluate the results of assessment on our campuses. I imagine most of us would agree that being able to benchmark our assessment results with those of a group of peer institutions would be the ideal. With the exception of national normative data available to those institutions utilizing one of the several standardized tests such as the CLA, there seems to be very little comparative data available to achieve this benchmarking.
Many institutions now utilize various assessment management systems and/or learning management systems with assessment functions included. I wonder if consortia comprised of institutions similar in role and mission and other key characteristics would be willing to engage in assessment data sharing for purposes of benchmarking their assessment results. And I wonder if the process could be facilitated by the use of common learning and/or assessment management systems. Such organizations could provide an enhanced service to their client institutions by serving as a third-party to collect, aggregate, and then return assessment data to “member” organizations. By using the services of an impartial third-party, individual student data and identity of individual institutions could be kept confidential and thus help to ensure the data are not used for ranking institutions as suggested by the New Leadership Alliance in their report.
Given the increasing microscope post-secondary institutions are being viewed under, such an initiative could prove to be a giant leap in terms of demonstrating accountability and transparency to concerned citizens and other stakeholders. Perhaps more importantly, the availability of this type of benchmarking data would surely be vital to quality improvement processes among our colleges and universities with our students being the primary beneficiaries of such efforts.
Kimberly Thompson
Academic Trainer & Consultant - Assessment & Analytics
Pearson eCollege

Filed under: eLearning No Comments
1Feb/120

Philosophy of Teaching Twitter Challenge!

This post could have been titled “What’s Your Teaching Philosophy in 110 Characters or Less?” because we’re asking you to participate in a challenge related to developing and succinctly crafting a version of your philosophy of teaching!

The Challenge*

Please review this this post and the examples provided below about writing a brief teaching philosophy. Then, we challenge our readers here to try it for yourself! We would like to receive your submissions via our Twitter account using a hashtag and to mention our Twitter name in your post. So, how do you do it? When posting your 110 character philosophy of teaching to twitter, please include the following in your post so we can follow your responses: @atcecollege #teachphilosophy

What is a Philosophy of Teaching? Why Should I Write One?

Though many formal teaching philosophy statements run two or more pages, having even a brief framework of your philosophy can be beneficial. According to Chapnick (2009), “creating a philosophy of teaching and learning statement is ultimately both personally and professionally rewarding, and is therefore well worth the effort” (p. 4). Defining our philosophy of teaching helps to provide a framework for our practice as educators.

Do you believe timeliness and access are important, as Stevens III (2009) does in this example of his principles? “The principles I follow are simple: be accessible to students and treat them with respect. Accessibility means being available not just during class and office hours, but at any reasonable time. I encourage them to call me at home, and I promise them a response to email messages within 24 hours” (p. 11). If yes, for example, your philosophy would feature timeliness and access as important to you and in your practice you would work to achieve these principles.

What the philosophy includes might reflect a diverse set of information and depends on the audience. The Teaching Center (2007) offers these as guiding questions: (1) Why do you teach? (2) What do you teach? (3) How do you teach? and (4) How do you measure your effectiveness? Let’s apply that framework here in our challenge!

Can I See an Example?

Of Course! Following the model described above, here are some examples:

Inspiring humanity social science and education engaging and interactive
authentic experience designs @atcecollege #teachphilosophy

Learning experiencing sharing knowing doing frequent engagement
anywhere anytime @atcecollege #teachphilosophy

Lisa Marie Johnson, Ph.D.
Academic Trainer & Consultant
Pearson eCollege


*Notes

  • Do you want to follow the tweets associated with @atcecollege or the tag #teachphilosophy? You can search without a twitter account by going to the Twitter Search page: http://twitter.com/search/
  • Hashtags on Twitter allow for “tagging” a post to twitter (tweet) that makes it easier to search for on twitter.  When you include the Twitter name preceded by the at-symbol - @ - it is a Mention of the account and your post shows up in a list of tweets that refer to that account.
  • If you do not have a Twitter account, but are on Facebook, you could instead post to our ATC eCollege Facebook account in response to the comment about this post: http://twitter.com/search/
  • Hashtags on Twitter allow for “tagging” a post to twitter (tweet) that makes it easier to search for on twitter.  When you include the Twitter name preceded by the at-symbol - @ - it is a Mention of the account and your post shows up in a list of tweets that refer to that account.
  • If you do not have a Twitter account, but are on Facebook, you could instead post to our ATC eCollege Facebook account in response to the comment about this post: MindShift.

References

Chapnick, A. (2009). How to write a philosophy of teaching and learning statement (pp. 4-5). Faculty Focus Special Report - Philosophy of Teaching Statements: Examples and Tips on How to Write a Teaching Philosophy Statement. Magna Publications. Available from http://www.facultyfocus.com/topic/free-reports/

Stevens III, R. S. (2009). Education as becoming: A philosophy of teaching (pp. 11). Faculty Focus Special Report - Philosophy of Teaching Statements: Examples and Tips on How to Write a Teaching Philosophy Statement. Magna Publications. Available from http://www.facultyfocus.com/topic/free-reports/

The Teaching Center (2007). Writing a teaching philosophy statement. Available from the Washington University in St. Louis: http://teachingcenter.wustl.edu/writing-teaching-philosophy-statement