When to Invest in Health Care Program Evaluation

Site Content

Home

Research Services

Areas of Expertise

Experience

Our Approach to Research

When to Invest in Program Evaluation

Employment

Contact Us

If you are like most health care leaders today, you recognize the importance of measuring outcomes and costs. You probably have some measures of the outcomes, cost, and quality of your services. Yet, you may still have questions about some aspects of the cost-effectiveness of your programs and their real value to your customers.

How much should you invest to obtain better performance measures for your managed care programs? The answer depends on several factors: your programs, your customers, and your organization’s willingness to use research results. While there are no hard and fast rules, knowing the answers to the following questions can help you determine whether an investment in scientific program evaluation is worthwhile for your organization.

What is program evaluation?

Program evaluation is a scientific approach used to assess the value of managed care programs. The methods of formal program evaluation have been applied for years to assess federal government programs including Medicare and Medicaid. Program evaluation involves measuring costs and outcomes. It usually includes a scientific research design and statistical analysis to identify and measure program effects. Analysis may control for factors such as case mix, characteristics of the local health care system, and price differences. It may entail cost-benefit or cost-effectiveness analysis.

Because program evaluation puts its focus on measuring program effects, it goes beyond simple outcomes measurement or benchmarking. It provides information about the effects a program achieves, how it achieves them, and what its value is to the managed care organization and its customers.

Will it help you gain a competitive advantage?

Evaluation can help create a competitive advantage in two ways. It can help you make better strategic decisions and it can help with ongoing improvements in quality and efficiency.

Better strategic decisions. Program evaluation can provide critical information when you face important strategic decisions, such as what services to offer, where to introduce new products, what features to include in new products, or whether to adopt a new care management technique company wide.

Evaluation allows managers to identify the real contributions of a program. It helps strategic decision makers see what program features produce desired results and are worth replicating in future programs.

Improved program performance. Perhaps the greatest return from program evaluation occurs when it helps move an organization more rapidly along the learning curve toward more efficient and cost-effective programs. Rigorous evaluation can show you what works before your competitors have figured it out.

Quantitative measures of program effects on outcomes and costs can validate the evolutionary changes in your programs. Measurement of performance is an important part of the continuous quality improvement loop. Evaluation can help you fine tune your managed care programs to keep them ahead of the competition. More effective and efficient programs then allow you to offer your customers lower costs or superior outcomes.

Will it help with marketing?

Do your customers need to know more about your programs? The more active your customers’ interest in quality and value, the greater is the importance of documenting the effectiveness of your programs. In many markets, large employers or health care purchasing coalitions look carefully at evidence of quality and cost-effectiveness in their local market. If the employers in your market use such information in selecting their managed care providers, you may gain a competitive edge if you can provide credible research-based evidence of the cost-effectiveness of your programs.

Market positioning based on your strengths. By showing the true effects of your programs, evaluation can also help you see how to position your program in the market to take advantage of its real strengths and ability to produce results. Evaluation results can help you make that position clear to your customers.

Costs and benefits of evaluation.

Each evaluation project should be justified by its net benefits to the sponsoring organization. The costs are usually easier to estimate than the benefits. Costs of research design, data collection, and analysis can be estimated once an evaluation project is defined. Benefits, in lower costs or improved outcomes, accrue to the managed care organization, its business customers, and the patients it serves.

Often it is difficult to say just how much of this benefit is due to the evaluation project. Evaluation is an enabling tool. It enables managers to develop cost-effective solutions and make the best strategic decisions. It enables employers to see the value of a managed care program. It enables patients to receive better care that will restore health and return them to work and other activities.

As an enabling tool for decision makers, program evaluation acquires its value from the importance of the program or program feature that is being evaluated. So evaluation is more important when it is applied to large or growing programs and when it will help guide important decisions about those programs.

Do you have a choice? For some organizations, evaluation is not a choice but an imperative. Some form of evaluation is essential if your organization’s long-run success depends on sustaining a competitive advantage based on offering cost-effective care. The real question is how much must you invest in program evaluation to avoid mistakes and make the best decisions about future programs.

Evaluation design is important. It is important to design each evaluation project to provide the most useful information for the dollars spent. Broadly defined evaluation projects can provide a detailed and comprehensive assessment of a program. More limited, focused evaluations can be designed to answer specific questions. It is important to determine research needs and identify the value of addressing each issue. Then an evaluation project can be defined to address the key research questions with an affordable budget.

Estimating the value of research. An acceptable project should make a positive net contribution to an organization’s long-run profitability. This contribution may come through reduced costs of service utilization, reduced cost of operating the managed care program, or improved marketing. The benefits of evaluation will be greater for large or costly programs or when used to evaluate new program features with a large cost saving potential.

How much can I afford to spend on program evaluation? Perhaps the best way to decide about an investment in program evaluation is to estimate its rate of return. A back-of-the-envelope calculation of costs and expected benefits can guide decisions. The key variables are the expected ultimate size of the program, the cost of the evaluation, and the expected contribution of the evaluation to improved program efficiency.

This may involve an assessment of the risks and rewards in the evaluation design process. You will want to consider, for example, what are the dollar risks of going national with a program that won’t work well in other settings. Could an evaluation reduce those risks? You might ask, for example, what are the potential dollar benefits to my organization if evaluation shows how to achieve a 5-percent reduction in overall utilization? What investment in evaluation is justified by these potential benefits?

You may find that the expected rate of return from an investment in program evaluation will be quite high. Even if your organization does not have a budget earmarked for program evaluation, a careful look at the costs and potential benefits of a specific evaluation project may convince decision makers that formal evaluation is a good use of your organization’s resources.

When program evaluation is most important

When programs are new. Are your managed care programs new, or have they changed recently? The first outcomes measures from a new program are sometimes the most important. They provide a first close look at how a program is functioning. They come at a time when they can show where improvement can be made as the program grows and develops. Evaluation can help you select program designs that will keep you at the cutting edge.

When a pilot is tested as a model for broader application. The advantages of formal program evaluation over simple outcome measures or benchmarks are particularly important when the results of a pilot project will be used to predict outcomes of applications in other settings. Evaluation can include economic modeling and analysis that controls for differences in casemix, prices, and environment. The estimated models, then, can be used to predict how a new program will perform in other environments.

When you need to learn how best to implement a new-concept program. Evaluation coupled with systematic experimentation with alternative program forms can be a powerful way to discover the most cost-effective program models.

When you are ready. The list on the right hand column of this page includes some signs that your organization may be ready to benefit from formal program evaluation. If one or more of the listed statements applies to your situation, then you may want to look seriously at using formal, scientific program evaluation to help give your program a competitive advantage in the managed care marketplace.

Thomas Grannemann, Ph.D.  
President, Andover Economic Evaluation