The ProofPilot Blog - Design, Launch & Participant in Research Studies

The Case for Evaluation Even When Money is Tight


Running a health improvement program without evaluation is like managing a business without a cash register, accounting system or budget. Businesses succeed and fail based on maximizing cash in and minimizing cash out. But when your goals are to improve health and wellness just counting cash doesn’t cut it. Your success goes beyond financial profit.

Evaluation helps you measure the impact of your work — both short term and long term. Are your processes running smoothly and efficiently? Are the short-term changes that are necessary to meet your long-term goal actually happening?

Sounds simple. Anyone who has conducted an evaluation can tell you it’s not. What is your evaluation design? How do you collect the data? Do you have statisticians to analyze findings? What is going on now? What needs to change? What does your final goal look like?

Conventional wisdom says 15% of your overall program budget should be set aside for evaluation. When money is scarce, when there’s no army of graduate students to collect data, nor a sabbatical to analyze findings, it is tempting to skip evaluation. After all, the cash savings will go directly into actual services, right?

Resist that temptation. Consider evaluation from the moment you conceive of your initiative. The investment will be worth it. Evaluation can do more than give you process measures. It can help you make a case for additional funding. It can help you make incremental improvements along the way to reach success. It can help you quantify your full impact.

Make the case for greater funding. Totah Behavioral Health Authority provides mental health services in Farmington, New Mexico. Alcoholism, drug abuse, and homelessness afflict the area’s large Native American population. Kris Carlson and Mike Renaud, the homeless program and regional directors, didn’t have enough beds to serve those in need.

They tracked the number of individuals turned away from services. They worked with contacts in the ER, local detox center and at the county jail. When clients ended up in local jails or prisons (instead of relatively cheap beds in their facilities), Carlon and Renaud computed the cost. It was far higher than the cost of supplying housing and fixing the underlying issues. They took this information to county officials. With it Totah won a multi-million dollar grant to house and serve more people.

Funders want to see some data before making funding decisions. They want to see a little traction before making an investment. Decision-makers respond best to concrete stories supported by hard numbers. When the facts show a fundamental result or a sustainable improvement from the current situation, and there’s evidence that the initiative can scale, it just makes sense to invest more.

Find opportunities to improve. Dr. Richard Crosby created an HIV prevention program in Jackson, Mississippi, for black men. HIV rates are on the rise among this population. And Jackson has some of the highest HIV infection rates in the country. Only one problem: traditional engagement techniques didn’t work. Few guys were enrolling in the program.

Fortunately, Dr. Crosby had a strong evaluation program in place. He noticed that pick-up basketball games attracted potential participants and decided to reduce funding for poor-performing enrollment efforts, like outreach workers in gay bars and advertising on online hook-up sites. Today the pick-up basketball games have helped bring enrollment almost to capacity.

While overused, the old saying, “there’s always room for improvement,” continues to be true. No one has unlimited resources. Not Fortune 500 businesses, not small non-profits. Finding opportunities for improvement can save money and improve outcomes.

Show the full impact of your work. In the early 2000s, pediatric asthma rates increased around the country. Repeated ER visits and average hospital stays of 4 or more days cost millions of dollars. Researchers noted a link between asthma and indoor air pollution. The Kings County (Seattle, Washington) Healthy Homes Initiative designed a program to address the problem. They sent community health workers to visit lower-income children with repeated asthma attacks. They provided education and services to reduce the severity and impact of the attacks.

They could have stopped there. They went beyond. They ran experiments to test what actually reduced asthma attacks. Some children received intensive services, regular counseling and home repairs to improve indoor environments. For the other group of children, community health workers simply provided basic lessons on asthma control.

After four years, the Healthy Homes Initiatives was able to show that the intensive services reduced asthma attacks and hospital admissions significantly more than the basic awareness building. More than that, they also were able to identify that the intensive services actually saved $800 in direct costs per participant over four years, even after the higher costs of the intensive services. With 1.3 million children with asthma in the US that is a significant potential savings. And that’s not even considering the broader non-financial value of improved health to those children and their families and communities.

Summary. Beyond time and money, evaluation can tell us if what we are doing works…and how well it works. It can be tempting to skip evaluation and put all your resources towards your community health programming. Doing so misses many opportunities to improve those programs.

This article first appeared on the Way to Wellville blog.

Matthew Amsden is CEO of ProofPilot. ProofPilot is an online platform that makes it easy to design, launch and manage complex research studies — like the evaluations outlined in this article. ProofPilot users do not need to be evaluation experts or have large budgets. They just need a question. ProofPilot has launched several ongoing initiatives Way to Welville. Esther Dyson, the Way to Wellville founder, is a ProofPilot investor.

You'll Also Be Interested In

Other Stories in the ProofPilot Blog