Marketing Positive Results of Outcome Assessment
I’ve blogged before about the external accountability and internal continuous improvement goals representing two competing perspectives on outcomes assessment in higher ed. In an article posted to Inside Higher Ed on February 22, 2010, Dead Dad articulates yet another thing to consider in this complex dynamic. That is whether or not colleges and universities should tout their achievements in improving the student learning experience as a result of completing a particular degree program.
At first glance it may sound like a great idea for institutions to promote their success in advancing the student learning experience on their campuses, however, as with most things the issue gets more questionable when you look deeper. Dean Dad explains that academics would be motivated to critically dissect underperforming outcomes to figure out where the gaps are when viewing this dilemma from the internal continuous improvement perspective. Conversely, the external accountability perspective wants to make an institution look as good as possible and marketing these results would have a tendency for academic leaders to hide the bad in order to accentuate the positive. Dad explains that “…too much transparency in the early stages of improvement-driven assessment can kill it, leading to CYA behavior rather than candor (Dad 2010).”
Ultimately, I believe that publishing and promoting success will motivate students and parents to look twice at institutions who can prove their students are learning which will pressure the lagging colleges and universities to step up their efforts. What I have seen in my work with colleges and universities is that campus assessment plans typically start with deans and faculty first with the goal of increasing transparency to other stakeholders once the key internal stakeholders have developed a sense of comfort and trust with the process. In the end, everyone agrees that it is in the best interest of all stakeholders to include students, parents, and the broader community in the outcome management and reporting process but it’s a challenge to be the first to take this transparency to the next level.
Could it be that outcome performance statistics is what will eventually start to chip away at the idea that a degree from an Ivy League school is better than that of an online for profit? That may be decades down the road but it will be exciting to watch this issue work itself out.
Dad, D. (2010, February 22). Assessment as Marketing. Retrieved February 22, 2010 from Inside Higher Ed, Web site: http://www.insidehighered.com/blogs/confessions_of_a_community_college_dean/assessment_as_marketing
Brian McKay Epp
Higher Education Assessment Consultant
2009 Conference Reflections
Over the past year I’ve attended academic conferences in the U.S., Mexico, Spain, and Bahrain. Here are a few key takeaways I can offer from my perspective as a higher education assessment consultant.
Academics worldwide are debating the scholarship of teaching and learning quite intensely due largely to the disrupting change of the online for-profits, the ubiquitous acceptance of social networking, and the reality of user created content. An article in last week’s The Chronicle of Higher Education nicely summarized the online for profit sector’s impact on challenging all colleges and universities to do a better job not only of creating and tracking student learning outcomes but also for using the data collected to refine curriculum and instruction with an eye toward improving the student learning experience. Most online programs are able to track all activity in a course including page visits, class discussions, assignment uploads, exams, and grades. They are also able to standardize learning outcomes for all sections of a course to ensure comparability of data. This is the point where traditional academics will raise the academic freedom argument, however, I’ve seen traditional faculty agree on a common set of outcomes and even common assessment rubrics even though the assignments they develop to assess student progress may differ by instructor.
I also just returned from the SACS-Commission on Colleges Annual Meeting in Atlanta. I noticed that many universities were talking about course level assessment of student learning outcomes this year which was new. This is an area where I’ve been focusing for the past 18 months so it was nice to see the academy starting to recognize the importance of getting more granular in the assessment of student learning. Previously nearly everyone was satisfied with program level assessment. Program assessment is still important but it should be triangulated with course level assessment data along with indirect measures such as NSSE, CSSE, or Noel Levitz. Many institutions also participate in either the Voluntary System of Accountability (VSA) or the University and College Accountability Network.
The rise of social networking and user created content is another salient takeaway this year. If Web 2.0 or education gaming was in the session title you could count on a packed room. This was the case worldwide. These technologies are moving beyond the early adaptor stage and more into the mainstream. It is important for digital immigrants (those born before 1995) to recognize that digital natives are used to processing multiple channels at once and having just in time access to information. There are theories circulating that indeed even the structure of a digital native’s brain is different. This means we must adapt our method of teaching to be more of a facilitator as opposed to a lecturer who disseminates knowledge. During a Web 2.0 presentation in Guadalajara, Mexico last week I challenged participants to start using at least one new Web 2.0 application first in their personal lives and then to try to integrate the application into their teaching in the Spring semester. I’d be happy to share my presentation with anyone who’s interested. You can email me at briane@ecollege.com if you’re interested.
It’s truly an exciting time in higher education. The next decade is going to bring about dramatic changes at colleges and universities. I look forward to participating in dialogue with many of you as we do our best to make education more accessible and effective for both learners and employers.
Brian McKay Epp
Academic Trainer and Consultant
State of the Student Learning Outcome in the Academy
|
|
Over the past month I’ve consulted with both K-12 and higher education leaders in the U.S., Mexico, and the Middle East. Tracking student achievement and the value add provided by an academic program is high on nearly everyone’s priority list. In fact, in a time of shrinking resources this is one area that is still receiving budgetary support.
In his October 26 article Assessment vs. Action on the Inside Higher Ed website, Scott Jaschik summarizes the results of a survey sent to senior academic leaders at 2,809 regionally accredited institutions in the U.S. The survey was commissioned by the National Institute for Learning Outcomes Assessment (NILOA) which is a joint project between the University of Illinois and Indiana University.
Academic leaders and assessment experts should read the Jaschik article and also bookmark the NILOA website which is an excellent collection of resources and current thought surrounding learning outcome management. Conclusions from the NILOA survey were just released in a report titled “More Than You Think, Less Than We Need: Learning Outcomes Assessment in American Higher Education”.
Essentially, the survey found that nearly all U.S. institutions are actively measuring student learning outcomes driven primarily by accrediting body requirements. The gap that remains, however, is to actually use this data to improve student achievement.
Forward thinkers are actively developing a culture of assessment on campus. They’re using assessment data to drive decisions about everything from curriculum and instruction to admission standards and to inform the strategic planning process (Kuh and Ikenberry, Jaschik, 2009).
Over the past year I’ve been fortunate to work with progressive thinkers who are leveraging technology to enhance the outcome management process and to maximize the time that faculty spend providing meaningful feedback and support to students. I look forward to continuing this work and to identifying and publishing best practices in outcome management.
References
Jaschik, S. (2009, October 26). 'Assessment vs. Action. Retrieved November 2, 2009 from Inside Higher Ed, Web site: http://www.insidehighered.com/news/2009/10/26/assess
National Institute for Learning Outcomes Assessment (October 2009). More Than You Think, Less Than We Need: Learning Outcomes Assessment in American Higher Education. Retrieved November 2, 2009 from Web site: http://www.learningoutcomeassessment.org/NILOAsurveyresults09.htm
Brian McKay Epp
Academic Trainer and Consultant
“Vaccinating” against academic dishonesty
Flu season is right around the corner and many are contemplating flu vaccines. However, as I’m dealing with yet another situation of academic dishonesty (which unfortunately seems to happen about once a semester… so hopefully this is it for a while!), I’m reminded of a great article from our Educator’s Voice newsletter on “Vaccinating against the plagiarism plague” by Charlotte Redden. I like the idea of a vaccination as something we do to in an attempt to prevent an uncomfortable situation we’d rather avoid. So rather than being reactive to situations of academic dishonesty, why not be proactive and try to create a preventative regimen to prevent it in the first place? The main preventative strategies proposed by Redden include educating students about your expectations, warning them that you do monitor for academic dishonesty and also to create less plagiarism-prone assignments in the first place by giving specific directions and scaffolding stages of the assignment.
Whether or not cheating is more common in online courses is a matter of debate (and see some comments here about a recent study suggesting that students are less likely to cheat online than in a traditional classroom). However, whether we teach online or in a traditional classroom, we should all consider our “vaccination” and take a look at our teaching materials. Are there any of Redden’s suggestions that we can implement?
– Gail E. Krovitz, Ph.D. –
Senior Academic Trainer & Consultant
Redden, CA. May 11, 2005. Vaccinating against the plagiarism plague: design and techniques. Educator’s voice Volume 6, Issue 5. Accessed online at: http://www.ecollege.com/Newsletter/EducatorsVoice/EducatorsVoice-Vol6Iss5.learn
Tapestry is coming!
09/08/09: I’m on a plane on my way to perform my first Project Tapestry training. After in many ways suffering through the curriculum meetings and organized “meet and greet your child’s new teacher(s)” sessions that accompany the beginning of the school year in my own children’s schools, I finally get to say all of the things to teachers in another state that I wish I could say to my own children’s teachers here in my own (Denver Public Schools) district. Things like, “Yes, you can see test results immediately after administering the test (or even just a test question), and here’s how to do it.” Or, “differentiation just got a whole lot easier—here’s how to find and distribute online remediation and enrichment to members of the same class, and then to easily monitor and document their progress so that you have more time for teaching them.” Or “this is how you can leverage those eager parents who want both insight and engagement in their child’s day to day learning-- in less time, and with more parental satisfaction.” This is a tool I wish our school district had.
As a teacher, I’d love the integrated approach. I’ve not had long exposure to the inner workings of a K-12 school district, but in the past few months on Project Tapestry, I’ve heard about more tools that don’t “talk” to each other than I ever knew existed. I’ve heard about the woes of Scantrons, the difficulty of reaching out to struggling and high-achieving students, and the plague of paper. I’ve also heard about the crazy things parents do to teachers in order to try to understand what their kids are learning and not learning and how they can help. (OK, I have had long exposure to this last one…I can totally be THAT parent….)
Project Tapestry facilitates the daily reporting of attendance and lunch count and the use of the ever present but now electronic Gradebook with which we are all familiar. It adds to the up and coming technologies of online and Smartboard/clicker-based instruction and assessment (this is much easier when conducted through one interface), and it allows sharing of assessment materials between schools in the district (like benchmark tests that can be authored by lead teachers throughout the district, then released and administered across all schools). Teachers can see the results of these tests immediately, so that curriculum adjustments that used to wait weeks on collective test results reporting can be made the next day. And, standards based assessment results can be looked at over time through a data analytics tool—if Johnny is struggling with reading, when did that start to happen? What step was missed or where do we need to backtrack and pick up again to get him on course? Having an easy interface in which to get this sort of historical information on a student or a class is the first step in the teacher identifying Johnny and getting him remedial content that will bring him back up to grade level.
What if Johnny not only reads, but exceeds? Rather than allowing his exceptional learning abilities to languish, Tapestry’s enrichment materials repositories can aid in giving Johnny the extra challenge that keeps him in his seat at school. We can reach into materials that allow Johnny to push ahead while not taking a lot of instructional time from his classmates.
At the end of the day, all of this prescription and progress lives in the same place, under the same happy URL. The Assessment Manager talks to the Gradebook and to the online classroom, feeds the Data Analytics tool so that it can help the teacher move in the appropriate direction for each child without rewriting the whole lesson plan. Student work is held in a secure, electronic environment that is secure from dogs and the things that can happen to it on the way to school, as well as on the way home where parents can see the grade. Soon, parents will be able to check in "at will" in a way that a mere portal doesn’t allow. All of this without an email snowstorm for the teachers, without a phone, and on an ongoing basis that takes the “SURPRISE” out of report card day.
Anyway, I know I’m excited. Not just for the parents, the teachers, the school districts, but for the students. I know of so many students who don’t know where their grades come from, who have no idea that their efforts are measured against standards and not just some idea a teacher has of what is “good.” Being able to use and share this tool with our students will help our kids to understand that they have control over their learning, that the measurement of learning has a logical face to it, and that they can get help or move ahead as their learning dictates. They will be able to work from home, from the library, and from school on material that appeals to where they are at in a given subject, and they’ll be able to know when they’ve succeeded and quickly go on, or recognize where and how they are challenged and go back. I can look at my own kids-- one who struggles and one who speeds ahead of the others-- and I see how this can work.
09/09/09: First training completed, and there are now some excited teachers and admins in this district! Yep, this is going to be good.
Data
I just got off the phone with a colleague who has lost 35 pounds in 2 months. How did he do it? Data. Well, data mixed with exercise and technology to be more precise. He tried the Nike / iPod experiment and he’s a believer.
This professor of communications and lover of cheese steaks bought a new pair of running shoes a few months back. Then, he bought the Nike sensor system – a small sensor you put in your shoe somehow. This sensor sends information to your iPod during a run. That data tells you (in real time) how you’re doing, but it also allows you to see any trends in your running after you upload the data to the Nike+ website. Apparently he’s run about 340 miles and his average speed has increased by 1 mile per hour. He can tell you how many calories he’s burned and he’s delighted to tell you how many pounds he has lost.
See, data is changing how we live. And data aggregation, data mining, and data analysis are making our lives better as technology gives us more and more ways to use it quickly and easily. For example, my wife was called a few months back about her credit card. Visa thought she might have lost her card. Why? Because she purchased a dress that was 2 sizes too big! Guess what? Her card had been stolen. (No, she had not gained any weight…that would have been awkward!) The credit card company looks for patterns and found something odd in the behavior of the card. So they checked.
Data is everywhere we look today. New cars will tell you how many miles you have driven on a tank of gas and how many more you are likely to get out of that same tank. There is a website where you can upload a sickness in your family. Then, you can look around your city, state, or the entire country to see where other people are sick too. Data might help you avoid the plague!!!
Data is useful and becoming easier and easier to digest. My phone tells me when my flight is late – a handy little feature when you fly 100,000 miles a year. My refrigerator tells me when the filter is no longer doing any good. Heck, even my daughter’s baby monitor tells us when the battery is low. From weather patterns to traffic patterns, data can make our lives tremendously easier.
So why is it so hard to find data for schools? This is especially true with online schools. Shouldn’t you know where your students spend their time in classes? Don’t you think knowing how often you’re B students post vs your D students post to a discussion would be a good piece of information? Does the first day a student checks into class help determine their probability of dropping? If you don’t know the answers to these questions...it’s time to.
One of my favorite tools I’ve ever gotten to work with is a business intelligence tool, created by IBM, that we overlay classes with in our system. This tool allows me and my team to try and predict success, correlate at-risk behaviors to drops, and find benchmarks to hold students accountable to. Did you know that in most online courses a larger class size (30-35) tends to have a better completion rate than classes with less than 30? It’s been proven time and time again through data. (Mind you – data can also beg lots of questions!)
Data mining is becoming easier and easier as technology evolves. Data analysis is becoming more and more automated. It’s time for your school’s programs to join the party! Trends and operational reports are crucial to making accurate predictions and drawing quality conclusions today. Accreditors are soon going to see this power and demand evidence of data-driven decisions for their schools. But before the ‘stick’ of accreditation swats at you, shouldn’t you look to the carrot of quality? Granted, this power can be abused. (My boss loves to look at my completion rates and give me grief as my public speaking class isn’t the highest completed class on campus…it’s public speaking!) But the data is there whether you mine it or not. The information to help you increase retention is sitting there whether or not it’s analyzed.
We study, analyze, and mine data for everything else today. It’s time to get education up to speed, don’t you think? Now if you’ll pardon me…I need to get to a store to buy a sensor. My pants don’t quite fit like they did last year…
Jeff D Borden, M.A.
Senior Director of Teaching & Learning
Momentum Building for Competency Based Learning
Most of us have heard of the European Union along with the establishment of the Euro as a common currency across the continent. Fewer have heard of the Bologna Process which began in June, 1999 with the goal of creating a more standardized higher education system in EU member nations. One initiative has been a tuning project where academics work to define a common set of learning outcomes by discipline and degree level.
The dialogue continues worldwide today about whether a focus on competencies versus assignment grading leads to an improved student learning experience but most would agree there is a difference. Many students are able to memorize processes or to cram for an exam but the ability to apply knowledge, skills, and concepts to new situations requires a deeper level of learning which is better suited for competency based assessment.
A June 4, 2009 blog post on The Chronicle for Higher Education website summarized a recent report commissioned by the Association of American Medical Colleges and the Howard Hughes Medical Institute calling for institutions to focus on competencies instead of courses as a way to improve curriculum for pre-med and medical schools. The report convened a group of educators, practitioners, and researchers to define a set of competencies both for entrance into and graduation from medical school. NCATE has already defined similar competencies for educators and other accreditation bodies are coming on board as well with efforts to agree on a core set of competencies by discipline.
The Lumina Foundation for Education also recently announced a three state Tuning USA project that seeks to define “the subject-specific knowledge and transferable skills that students in six fields must demonstrate upon completion of a degree program”. This is a bottom up effort involving faculty, students, and employers. Representatives from Indiana, Minnesota, and Utah will each define student learning outcomes for two disciplines while striving to preserve the ability for individual institutions and faculty to retain their academic freedom to teach to a common set of outcomes in the manner of their own choosing.
Pearson eCollege will continue to monitor this trend and seeks input from our partner institutions for best practices in outcome management and competency based learning.
References
Benelux Bologna Secretariat (n.d.). About the Bologna Process. Retrieved June 12, 2009 from Web site: http://www.ond.vlaanderen.be/hogeronderwijs/bologna/about/
Lumina Foundation for Education (2009, April 8). News Release. Retrieved June 12, 2009 from , Web site: http://www.luminafoundation.org/newsroom/news_releases/2009-04-08.html
Mangan, K. (2009, June 4). 'Competencies,' Not Courses, Should Be Focus of Medical-School Curricula, Report Says. Retrieved June12, 2009 from The Chronicle of Higher Education, Web site: http://chronicle.com/news/article/6588/competencies-not-courses-should-be-focus-of-medical-school-curricula-report-says
Implementing Competency-Based Learning
Over the past few months I’ve been reflecting and writing about assessment accountability and its intersection with workplace competencies. I believe today’s post on competency-based learning nicely integrates these two topics and provides academic leaders with a progressive assessment model that dovetails nicely into learning outcome management systems which support the more rigorous demands of accrediting bodies.
The figure below, taken from a 2002 U.S. Department of Education report, is a graphical representation of a pedagogical model showing the progression of learning from the engagement with one’s prior knowledge to the demonstration of acquired skills, abilities, and knowledge following a learning experience.
Source: U.S. Department of Education, 2002.
Many traditional, course-based learning models stop after conducting some form of assessment to verify that basic knowledge transfer from faculty to student has occurred. More progressive programs extend this assessment to a more granular level and actually prove that students can demonstrate performance of required competencies. An additional benefit of the more granular, performance-based learning outcomes is that it forces faculty to write specific, observable, and measurable learning goals.
One of the key barriers faced by institutions who are considering a move to competency based learning is the challenge of moving beyond the course-based assessment model along with the subsequent issue of how competency based learning units transfer into traditional academic programs. Western Governors University (WGU) has moved the furthest along this path by defining competency domains required both for general education and major area studies. While WGU is widely recognized as being at the forefront of this movement, other institutions such as Rutgers University, Alverno College, and Maricopa Community College have chosen their own paths and provide alternate implementation strategies (A. Voorhees, 2001, p. 84-86).
Dr. Richard Voorhees, a published author in the field of accreditation and assessment of both distance education and competency-based learning models, provides a twelve point list of actionable best-practices for those considering a move into this paradigm. It includes suggestions that range from obtaining senior level management support to ensuring that multiple assessment points are embedded within a degree program so that meaningful data analysis can inform curriculum enhancements (2001, p. 12).
While many may be reluctant to embrace a degree that didn’t come with a traditional course-based transcript, I believe employers would be impressed with how these graduates could both demonstrate general knowledge as well as the ability to perform required workplace competencies at a higher level than traditional program graduates. Has anyone been directly involved with implementing a competency-based learning model? I’d be interested in your comments on what you learned and how students responded.
References
U.S. Department of Education, National Center for Education Statistics. Defining and Assessing Learning: Exploring Competency-Based Initiatives, NCES 2002-159, prepared by Elizabeth A. Jones and Richard A. Voorhees, with Karen Paulson, for the Council of the National Postsecondary Education Cooperative Working Group on Competency-Based Initiatives. Washington, DC: 2002.
Voorhees, A. (2001). Creating and Implementing Competency-Based Learning Models. New Directions for Institutional Research. 2001(110), 83.
Voorhees, R. (2001). Competency-Based Learning Models: A Necessary Future. New Directions for Institutional Research. 2001(110), 5.
Brian McKay Epp, M.Ed., Academic Trainer and Consultant
Actualizing Assessment Accountability
The higher education community has been rumbling for several years about whether and to what extent government will pressure accrediting bodies to figure out how to hold institutions more accountable for demonstrating student achievement and growth. Tuition increases have consistently outpaced inflation for years which has led to public pressure to figure out why and subsequent calls for schools to justify their value to students, parents, and employers.
Legislators responding to public discontent began to increase calls for colleges and universities to actually prove their academic programs were meeting learning outcomes and to demonstrate their value add. The 2006 U.S. Department of Education report A Test of Leadership: Charting the Future of U.S. Higher Education created quite a buzz as the committee’s conclusions started to reach campus leaders. What is yet to be determined is whether it was a temporary blip that stirred up emotions and fears or whether it will truly be an impetus for significant reform.
A February 2007 meeting of the federal accrediting panel is an indication that increased accountability will likely come eventually but it will take time and may continue the U.S. tendency toward voluntary compliance. The most likely scenario is that we will gradually move toward standardization of outcomes beginning with general education which will in turn lead to a growing number of institutions who implement standardized achievement tests.
There are three assessments today that are being administered on campuses and have been approved by the Voluntary System of Accountability, a program that is being piloted by 239 early adopters in the public system under the auspices of the National Association of State Universities and Land Grant Colleges and the American Association of State Colleges and Universities. A similar program is underway for over 700 private, non-profit institutions (U-CAN) sponsored by the National Association of Independent Colleges and Universities and the Transparency by Design initiative is yet another effort that includes a range of private for-profit and non-profit institutions.
Schools that achieve top scores on these assessments will tout their success which will lead to new categories among prominent ranking systems as evidence that students should attend their programs because of their proven track record of achievement. Eventually the accreditors will then push for similar accountability within undergraduate professional majors and before long most institutions will be on-board.
eCollege is prepared to collaborate with institutions in the efficient management of learning outcomes which includes reporting of successes and challenges to stakeholders. Our teams are developing tools that will track outcomes and link them to course content so faculty, department chairs, and deans will have the information they need to assess program and course effectiveness both for curriculum enhancement and for reporting out to accrediting bodies and employers. We will also be able to show students how well they’re doing against the learning outcomes set by their institutions, degree programs and courses. Imagine the power of being able to pinpoint which assignments or test questions are most effectively contributing to student learning.
Brian McKay Epp, M.Ed.
Academic Trainer and Consultant
Categories
Blogroll
Recent Updates
| M | T | W | T | F | S | S |
|---|---|---|---|---|---|---|
| « Aug | ||||||
| 1 | 2 | 3 | 4 | 5 | ||
| 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| 13 | 14 | 15 | 16 | 17 | 18 | 19 |
| 20 | 21 | 22 | 23 | 24 | 25 | 26 |
| 27 | 28 | 29 | 30 | |||