For Immediate Release:

Sunday, December 30, 2018

Welcome to Internal Evaluation Week, hosted by the Internal Evaluation TIG! This week’s blogs focus on the theme of improvement: as internal evaluators, how are we ensuring that insights are being used to inform program and organizational improvement? Topics include improvement science, communities of practice, and participatory data interpretation.
Hi, my name is Ali Holstein and I’m a consultant and data translator working with youth programs. In my previous role as a program analyst at a non-profit, I would often bring people together to discuss our outcome data, reflect, and set goals for improvement. On paper, it sounds like a solid performance management plan. In reality, those meetings could fall flat. We once asked staff how they felt about data, and their responses included:  “anxious”, “stressed”, and “annoyed.”  This year, we’re trying a new approach, piloting a continuous improvement fellowship as a different way of learning.
In our Good Shepherd Services Improves Fellowship, we invited staff from Good Shepherd’s school programs to examine a shared problem they had identified as ongoing, hard, and frustrating: chronic absenteeism. Staff are pressed for time, but we still had five programs apply to participate in the 6-month fellowship. They were motivated to find better approaches to solving a problem that many had been dealing with their entire careers, and hoped to do so by sharing ideas among a network of peers and learning to solve problems in a disciplined way.
Lessons Learned:
The fellowship has led us to examine how Good Shepherd uses data to think about improvement:
Old Way of thinking: If we bring people together to look at data, it will spark change.
New Way: Invite staff to use improvement science methods to look at a specific problem. Spend time understanding the problem and researching solutions before creating any change ideas. Going slow is okay.
Old Way: If we present the data in different ways, it will lead to insights.
New Way: There are endless ways to use data to understand a problem. Start with insights and follow with the data to drill deeper. Staff already have instincts about what’s working, what hasn’t worked, and what might be worth trying.
Old Way: Staff are too busy for data stuff, so we should do the analysis and present our findings to them.
New Way: Invest time in building the data skills of your staff. They will feel more empowered to ask key improvement questions like: what is the problem we’re trying to solve? What change should we introduce? How will we know if the change is an improvement? [link]
Rad Resources:

• Don’t leave participants/constituents out of your improvement process. As homework, our fellows were asked to do empathy interviews with students to learn more about chronic absenteeism.
• While specific to the school context, Learning to Improve: How America’s Schools Can Get Better at Getting Better by Bryk, Gomez, Grunow, and LeMahieu is a great resource for anyone looking to learn about improvement science.

The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.