Jan06
Using Metrics to Evaluate
I get asked a lot about the question of how to evaluate nonprofits. I once believed that there was a holy grail of metrics that if we just worked harder to find, we could measure all nonprofits, lay them side by side and figure out which ones where doing more good in the world.
And then, I visited a homeless shelter. This one had been around for over 10 years, located between the train tracks and the bus depot. It gave homeless people free breakfast, a place to hang out and some housing. I asked the director, a Unitarian minister, how he measures his effectiveness. I had expected him to say something about the number of people he’s helped find jobs, or the number of breakfast sandwiches they hand out, etc. Instead, he replied simply, “Last week one of our (homeless) regulars died. And 10 people, who he’d gotten to know here, showed up for his funeral. That’s how I measure our effectiveness.”
That sobering encounter made me think hard. Many nonprofits are trying to make a difference in people’s lives. People are not products. We are complicated - changing attitudes, ideas and behaviors could take years and it’s hard to isolate which one factor contributed to any specific result. It’s much more complicated than running a business, where it comes down to profits, units sold, or stock price. The fact that the shelter provided a place where a homeless person made friends who cared enough to go to his funeral - that's something that is impossible to quantify, and yet is so precious and speaks volumes about the nonprofit's impact on this person's life.
The challenge of evaluating a nonprofit using metrics, is similar to asking a parent - what kind of metric do you use to evaluate your parenting? Whether your kid eats green vegetables? Whether your kid makes it onto the basketball team? Whether your kid gets into an Ivy League college? It seems intuitive that none of these metrics adequately reflect adequately whether we have been good parents or not. We may look to some of these things, but then we also use our judgement - is the kid happy and well-adjusted? And even then, we may not know for years whether we succeeded as a parent until we see our kids grown up.
Sociologists struggled with the evaluation question in the 1960’s when evaluating government social programs and most gave up on idea that you could feasibly use the scientific method to evaluate nonprofit programs. MRDC does some great evaluations using control groups and random selection, but there’s very very few programs that would qualify for the huge expense and time involved (plus overcome the moral dilemma to assign certain people to control category where they don’t get treatment/the program). Medium sized and small nonprofits could not afford nor be suited for such types of evaluation.
A good balance I think is that suggested by Phil Buchanan over at the Center for Effective Philanthropy - every organization should set and state its goals, strategies and indicators. Indicators, in my opinion, can embrace narrative stories of impact, as well some sensible numerical indicators. Stories are just as valuable - sometimes a lot more so because they are able to capture nuances - in information as metrics. And in terms of galvanizing donors and volunteers, the research consistently show that photos and stories are more powerful than just numbers.
by Perla N.
1 comment
As funding entities such as the United Way have moved to an emphasis on outcomes, the metrics issues has become more of a challenge. With such a range of methods to define effectiveness, an organization may or may not be defining it in the way that your potential funders may. As the former coordinator of a small statewide association, I observed how some agencies (family service) met the challenge successfully and others were left in the cold. The success stories were able to adapt to their funders critieria. Sometimes, this required tweaks in programming; more often, it was the data collection or reporting that was tweaked (or begun).
Can’t agree more with the need to be able to communicate the mission. (I know instances where actual board members could not succinctly state their organization’s mission when asked and yes, those agencies had more trouble getting funding).
Beyond that, perhaps one approach is to let your outcomes be known year ‘round, in a variety of ways, both quantitative and anecdotal. This not only ensures that you are understood, but may influence the shape and details of what funders seek.
Out here in Pennsylvania, this is vitally more important. We have an incredible number of nonprofits (but that is an entirely different issue!).
