For Immediate Release:

Sunday, July 14, 2019

Hello AEA365! We are Bryan Hall, senior director of evaluation, andDanielle Fontaine, senior manager of evaluation at BellXcel (formerly BELL, Building Educated Leaders for Life). BellXcel offers solutions for under-resourced communities to createevidence-based summer and afterschool learning programs that address the opportunity and achievement gaps.
Historically, the focus of our internal program evaluation approach was measuring student-level academic and social-emotional outcomes.  We engaged teachers through survey research, but often only around program engagement and satisfaction.   After a review of our evaluation data and anecdotal evidence, we discovered that our summer program was possibly having an impact on school year teaching practices.  This lead us to look outside of our theory of change to consider how our pre-program professional development and in-program coaching may be impacting long-term teaching practices.  Through additional survey research and in-depth interviews a few months after the summer program ended, we learned that our summer program can transform and enhance teaching practices, mindset, and ultimately influence their daily school year practices.

Lesson Learned:
Engage educators in impact evaluation, using their feedback to understand teacher-level impact in addition to student-level impact. Viewing educators and instructional strategies as inputs and activities in influencing student outcomes was limiting our understanding of program impact. By moving beyond reflective survey questions about engagement and satisfaction, we developed a broader understanding of exactly how educators feel about the program experience and advanced our knowledge of how the program influences teachers professionally.
Hot Tips:

1. Use an Outcomes Harvesting approach to understand trends outside of the standard outcomes. Looking at the full range of evidence, we found anecdotes of teachers who changed their practices because of the summer experience. With this as an outcome, we developed surveys and in-depth interviews to understand how exactly the summer experience was influencing these practices.
2. Reimagine a theory of change, when the evidence is pointing to an important trend. Traditionally, our theory of change saw educators as inputs and professional development as activities. Based on new evidence, our summer intervention goes beyond a program for students and considers educators as direct participants in the intervention.
3. Start with exploratory research. Beginning with a small-scale, no-cost survey in 2016, we intentionally and over time explored which areas should be further investigated using more complex higher-cost methods.

Rad Resources:
AEA365 blog post, Outcomes Harvesting Week: What is Outcome Harvesting? By blog series hosts, Barbara Klugman, Heather Britt, and Heidi Shaeffer. This was the first in a week-long series included great information about Outcomes Harvesting.
Allison Crean Davis’s article In Building an Evidence Base for Research, Start Small – & Follow the Train of Breadcrumbs Toward Larger Answers.
Lovely Dhillon and Sara Vaca’s paper on Refining Theories of Change in the Journal of MultiDisciplinary Evaluation.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.