For Immediate Release:

Wednesday, May 22, 2019

Hi! I’m Dana Wanzer, doctoral candidate at Claremont Graduate University and an avid #EvalTwitter user!
Many people new to evaluation—students and clients alike—struggle with understanding what evaluation is, and many evaluators struggle with how to communicate evaluation to others. This issue is particularly difficult when evaluation is so similar to related fields like auditing and research.
There are many great resources on what evaluation is and how it differs from research, including John LaVelle’s AEA365 blog post, Sandra Mathison’s book chapter, and Patricia Rogers’ Better Evaluation blog post.  I wanted to examine these findings more in depth, so I conducted a study with AEA and AERA members to see how evaluators and researchers defined program evaluation and differentiated evaluation from research.
In this study, I recruited members of AEA (who were primarily members of the PreK-12 Educational Evaluation and Youth-Focused Evaluation TIGs) and members of Division H (Research, Evaluation, and Assessment in Schools) of the American Educational Research Association (AERA). A total of 522 participants completed the survey which, among some other questions, had them define evaluation, choose which of the below diagrams matches their definition of evaluation, and rate how much evaluation and research differs across a variety of study areas (e.g., purpose, audience, design, methods, drawing conclusions, reporting results).
Lesson Learned: Evaluators and researchers alike mostly define evaluation like Scriven’s definition of determining the merit, significance, or worth of something—essentially, coming to a value judgment. However, many evaluators also think evaluation is about learning, informing decision making, and improving programming, indicating the purpose of evaluation beyond simply the process.
Lesson Learned: Mathison described five ways in which evaluation and research could be related:

Half of participants thought research and evaluation overlap like a Venn diagram, which is similar to the hourglass model from LaVelle’s blog post, and a third thought evaluation is a sub-component of research. However, evaluators were more likely to think research and evaluation intersect whereas researchers were more likely to think evaluation is a sub-component of research. Evaluators are seeing greater distinction between evaluation and research than researchers are!
Lesson Learned: Participants agreed that evaluation most differs from research by the purpose, audience, providing recommendations, disseminating results, and generalizing results and are most similar in study designs, methods, and analyses. However, more evaluators thought evaluation and research differed greatly across a multitude of study-related factors like these compared to researchers.
Hot Tip: If your students or clients already know about research, describing how evaluation is similar to and different from research might be a great approach for teaching what evaluation is!
I believe this study will be useful in helping propel the field of evaluation forward, at least by helping our field better describe evaluation but potentially in situating our field as a distinct discipline and profession.
Rad Resource: Hungry for more information? All the study materials, data, and manuscript are posted on the Open Science Framework, a free and open-source website that allows scientists to collaborate, document, and share research projects, materials, and data.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.