For Immediate Release:

Friday, June 22, 2018

Hi, I’m Nicky Grist of the Cities for Financial Empowerment Fund (CFE Fund). When I read that AEA members are interested in “how evaluators collaborate with stakeholders to apply findings of the evaluation to improving their initiative,” I knew I had to share the story of my most successful evaluation project ever.
In 2016 the CFE Fund evaluated municipal Financial Empowerment Centers (FECs), which provide free, one-on-one professional financial counseling for low-income people, as a public service. Among many findings, the evaluation showed that clients were less likely to increase their savings than make other financial improvements, that counselors were aware of these differences, and that the way that the savings outcome was constructed was potentially obscuring or limiting client success.
In 2017, we funded (!) a yearlong effort to explore savings more deeply and test alternative program outcomes in two cities, giving them moderate grants to cover the extra effort expected of their counselors and managers.
The design phase included:

• reading about how low-income people save and how programs measure savings
• interviewing field leaders (government program managers, think tank researchers, academics, and directors of innovative nonprofits)
• surveying counselors
• Photovoice with FEC clients

As a team, the local program managers, a database consultant, the CFE Fund’s program staff, and I clarified the definition of savings and created many new metrics. We built new data entry screens and reports and retrained the counselors, who then used these new metrics with 305 clients over six months. Although it was more work, counselors were enthusiastic about testing ideas they had helped develop.
After six months, we analyzed the data, creating a comparison group of similar clients who were counseled over the same six-month period the previous year. We also resurveyed the counselors and managers, and repeated Photovoice.
I expected the new outcomes to paint a more complete picture of clients’ savings goals, behaviors, and contributions, but the results went beyond my wildest dreams. Compared to the prior year, more pilot clients saw greater savings increases; the average number of sessions per client increased and more clients returned for multiple sessions. Clients gained greater understanding of and confidence about saving. The data better represented the coaching aspects of financial counseling. The data entry screens provided constructive guidance for counselors.
The counselors and managers helped me present the findings to a sold-out (!) live audience, and we also hosted the best-attended webinar in our organization’s history. Clearly, our field was excited to learn not only the results but also the evaluation-based pilot process.
Rad Resource: AEA365! I read about Photovoice here and reached out to the authors for advice – evaluators are great about sharing what they know.
Hot Tip: using evaluation methods to support program improvement is crucial for internal evaluators, especially in settings where traditional evaluations lack political appeal or where programs are not ripe for impact evaluation.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.