SELF-MANAGEMENT
SUPPORT
  • About Self-Management Support

Evaluation

Evaluation & Maintenance:  Reach, Effectiveness, Value

Reach
The RE-AIM (reach, efficacy, adoption implementation and maintenance) model provides one very useful way of considering how to evaluate the CDSMP.  There is also an excellent website. http://www.re-aim.org.

The following questions will help you whom your program is reaching and how representative this is of your area or your target audience.

1. The most basic reach question is how many people are attending your programs.  You may want to collect information such as gender, age, education, ethnicity, and residence.

2. You may want to compare your data with the data of the area.  For example if 15% of your target population is immigrants, are 15% of the CDSMP participants also immigrants?

3. As your program grows you may have a goal of reaching 10% of the seniors in a specific postal code or who attend a specific Sr. Center.  Then at the end of the year you can check to see how you have done.

4. You may want to know which types of publicity bring in which kind of people.

Effectiveness
When people think about evaluation they usually think about evaluating the positive and negative impacts of the program on such things as behaviors, symptoms, health status and or health care utilization.  

Effectiveness evaluations are usually done by getting information from participants (usually by questionnaire) before the program starts and again some time later.  For CDSMP most effectiveness evaluations have been conducted four to six months after the beginning of the program.   For more information about effectiveness studies and sample questionnaires you can go to the following websites.

http://patienteducation.stanford.edu/research/index.html

http://patienteducation.stanford.edu/research/primer.html

Adoption
Adoption evaluations look at the settings and or organizations that are offering a program and how successful each is.   Unfortunately we do not do enough adoption studies.  Here are some examples that you might consider.

 1. You set off to target health plans in your area and talks to many people in many plans, how many plans offer the program and what are the characteristics of the plans that offer the program and those that don’t.

2. You offer leader training to 20 community organizations and ten actually send people to training.  How do those that send people differ from those that do not.  After a year you find that 6 organizations have offered a program but four have not.   It would be good to know what makes the difference.   At the end of two years, there are only 4 organizations offering programs.  Two of these offer several programs a year while two only offer one program a year.  Can you learn anything by talking to these programs and finding out what makes the difference?

3. You initially train 20 Master Trainers.  Fourteen of these get certified and ten are actually training leaders.  Since training Master Trainers is very expensive you might want to know why you had only a 50% success rate and how you can do a better job of selecting and supporting Master Trainers.

Implementation
At the agency level, fidelity refers to the how closely staff members follow the program that the developers provide. This includes consistency of delivery as intended and the time and cost of the program.

At the individual level, implementation refers to clients’ use of the intervention strategies.

Checks on Program Fidelity
• Leaders should be observed or called during each workshop.

• Monitor and follow up on drop outs

• Monitor effectiveness of recruiting strategies

• Offer refresher to leaders yearly

Maintenance
The extent to which a program or policy becomes institutionalized or part of the routine organizational practices and policies. Within the RE-AIM framework, maintenance also applies at the individual level. At the individual level, maintenance has been defined as the long-term effects of a program on outcomes after 6 or more months after the most recent intervention contact.

Costs
How much it will cost to implement the program varies considerably, depending on your own set up.  There are many factors that depend mostly on how many programs you will give and the volunteer support you can rely on. 

Here is a list of items to consider when thinking about costs:

  • Program Coordinator (may be part time or full time)
  • Master Trainers
  • Training of Master Trainers if you do not have any available
  • Leader’s Training (food, room, materials, recruiting leaders, leaders’ manuals)
  • Publicity
  • Participant Registration and Support
  • Leaders (may be volunteers or may receive a stipend)
  • Sites for Programs  (usually donated but you may have to room rental)
  • Materials (for participants)
  • Charts (used by the leaders)
  • Evaluation
  • License