Part VII: Evaluation

VII. EVALUATION
A. Methodology
B. Venue
C. Participant Profiles
D. Pre-Test and Post-Test Results

VII. EVALUATION

A. EVALUATION METHODS

To evaluate the outcome of the sessions, two evaluation instruments were utilized: the Participant Profile and the Session Evaluation Forms. (See Attachment L)

The Participant Profile Form, which was distributed at the beginning of the session, includes participant information (name, title, degree, organization, and contact information); role/position in the faith community; demographic information (age, gender, race, and ethnicity); level of knowledge of end-of-life care, and how the individual learned about the educational session.

The Session Evaluation Forms were distributed for collection at the end of the session. The evaluation forms were collected prior to the presentation of the Certificate of Attendance to generate a high response rate. The evaluation instrument asks for comments on what was the most/least helpful part of the program, suggestions for future educational sessions, rating of the individual components of the program, perception about being better prepared to minister to their faith community at the end of life, and the level of agreement/disagreement with statements related to modules covered during the educational session. A major feature is the question that asks the participant to rate his/her level of knowledge of end-of-life care both pre-session and at the end of the program.

The evaluation results cover the entire project implementation from January through April 2003. They represent findings based on 22 education sessions and the compilation of results from all participant surveys filled out by the participating clergy, lay leaders and other attendees from faith communities in the six regions across the State of Florida. A total of 613 clergy and lay persons attended the sessions with an average number of 28 people at each program. While nine counties were originally targeted for implementation, a total of 13 were actually included in the project upon its conclusion. In addition, many of the participants attending the 22 sessions resided in surrounding counties, increasing the statewide coverage to a total of 28 counties, or 41.8% of all Florida counties. A summary of the evaluation instruments completed by the participating regions is provided in Attachment M. 

B. VENUE

Of the 22 sessions, 15 were held in church settings representing 68.2% of the total. Another 7 or 31.8% were conducted in secular locations such as hospices, a local library, a community college and a local health council. Average attendance was 10.4% higher in sessions held in church venues. 

C. PARTICIPANT PROFILE RESULTS

Role/Position

A total of 536 attendees completed the participant profile at the beginning of the education session. Of these responses, a majority (54.0%) reported themselves as being clergy. These are inclusive of chaplains, deacons, ministers, rectors, reverends, priests, and rabbis. They also include assistant and associate ministers. The next most frequent participant types included caregiver/patient relations/social worker and bereavement/outreach workers (7.5%); lay workers (4.2%); and church administrators (2.9%). All others rated less than 2.6% each. (See Figure 1.) 

Regions tended to follow a similar pattern with the exception of Northwest Florida/Big Bend which had a high turnout of lay workers (16%) and South Florida with a good turnout of clergy in training (7.9%).

Age

The average age of clergy/caregiver/lay worker participants was generally in the middle ranges. (See Figure 2.) While people of all ages attended, the largest group was between the ages of 50-59 years (28.6%); followed by 60-69 year olds (23.2%) and the 40-49 year olds (16.3%). Together they represented 68% of the participants. A striking 10% were over the age of 70 years, and 4.6% were ages 30-39. All regions showed a clustering in the middle age groups. (See Figure 2.)

Gender

No major gender differences were observed. Both men (45.0%) and women (43.7%) were represented. Only Treasure Coast (54.3%) and South Florida (52.4%) had a majority of female participants.

Race and Ethnicity

Nearly two-thirds of the participants were White (64.6%). About 13% were African American, compared to 14.6% in Florida overall. However, a large segment of this total was from South Florida which had a 25% African American participation rate. Hispanics comprised 9.1% of the respondents, a smaller proportion than that found in the general Florida population (16.8%). It should be noted that over half of the participants did not respond to this question.

Geographic Location

The largest portion of participants reported that they lived in suburbia (31.7%). Another 18.1% reported they resided in an urban area. Only 7.0% reported living in a rural community. Once again, however, many respondents did not answer this particular question. No marked differences appear by region.

Means of Hearing about the Session

Over a third of the participants (34.6%) heard about the session through the mail or received an invitation. A little over 10% received information from a flyer. Another 6.7% heard about the session they attended via a religious publication. Well over a third (35.6%) indicated another means of notification. By region, over 38.7% in the Northwest Florida/Big Bend area reported an "other" source of hearing about the session. This may be due to the television and radio coverage they received which was not listed as one of the possible responses. South Florida also had a large "Other" response (45.1%) as it relied heavily on media, word of mouth and email/personal contacts when traditional means of notification were not as successful.

Participant Summary

Of the 613 participants most were clergy leaders of different faiths and groups across the state. Many also had caregiver responsibilities as part of their professional or lay ministry. The vast majority of the participants were between the ages of 40 and 69 years, and there seemed to be a near even distribution among men and women. While nearly two-thirds of the participants were White, a fairly representative group of African American clergy participated, particularly from South Florida. Many of the participants reported that they resided in a suburban or urban area. While mail and invitations were the means by which over a third of the participants heard about the session, a sizeable portion also learned about the program from other means, which could include media, word of mouth, e-mail and other personal contacts, to name a few. (See Attachment N.)

D. PRE-TEST AND POST-TEST EVALUATION RESULTS

Changes in Knowledge of End-of-Life Issues

Table 2 and Figure 3 show that approximately 70% of those participants who completed both the pre-test and post-test reported a higher level of knowledge of end-of-life care on the post-test than they did on the pre-test. Of those who felt that their knowledge of end-of-life issues had room for improvement (i.e., those who had not already rated their knowledge as "excellent" in the pre-test), 80% reported a positive change. Given that both measures occurred on the same day, it seems difficult to attribute this change to the impact of causes other than the educational session. (See also Attachment O.) 

Table 2

Knowledge Level of End of Life/Care Issue
Following the Workshop

As shown in Figure 4, there were differences among the six regions in the proportion of participants who reported a higher level of knowledge at post-test. For example, roughly 87% of participants in the Treasure Coast regional training reported a higher level of knowledge at post-test, versus only 55% of those in the Suncoast/West Florida region. However, as noted below, these differences in outcomes may reflect the backgrounds of the trainees rather than regional differences in the quality of the educational experience. 

Table 3 and Figure 5 illustrate the change from pre-test to post-test in the average level of knowledge for the entire group and for each region when this knowledge is expressed as the mean (or "average") value of the participants' responses. As the figure shows, the mean score for all participants showed a change from a "good" level of knowledge to a "very good" level of knowledge. 

Table 3

In part, the regional differences in outcomes reflected in Figure 6 may be a function of differences in the baseline level of participants' knowledge across regions at the time of the pre-test. As an example, the Suncoast/West Florida region, which showed the smallest increase in average score, includes those participants who reported the highest average level of knowledge at pre-test. Thus, the apparent differences in the program's impact on knowledge across regions, as shown in the figures, appears to reflect, at least in part, differences in the prior training and background of the participants in different regions.

A final perspective on changes in self-perceived knowledge of end-of-life issues is offered by noting the substantial changes from pre-test to post-test in the percentage of persons who described their knowledge as "fair" or "poor" and as "excellent". At pre-test, roughly one in four participants (25.5%) described their knowledge as only "fair" or "poor" whereas at post-test only five participants (0.8%) described their knowledge at these low levels. Similarly, participants describing their knowledge as "excellent" grew from 10.4% at pre-test to 24.5% at post-test. If non-respondents are excluded, the proportion describing their knowledge as "excellent" among those who completed both questionnaires is even higher (30.3%).

Preparation for End-of-Life Ministry

Following the training, participants were asked whether they felt prepared to minister to their faith community at the end of life. As shown in Table 4, 171 participants did not answer this question, but among the 442 who did answer it virtually all felt prepared or "somewhat" prepared. Only five participants (1.1% of the 442 participants who answered the question and 0.8% of all participants) answered "no". Affirmative responses included "yes" (76.2% of those who answered the question and 55% of all participants) and "somewhat" (22.6% of respondents and 16.3% of participants). 

Feelings of being prepared were constantly high across the regions, ranging between 49.4% and 71.7%. (See Figure 7.) In fact, many expressed during the sessions that once they reviewed all the handout materials including audio tapes and books, their knowledge would increase even more.

Table 4

Evaluation of Program Components

Participants were asked to rate the utility of ten different components of the educational program on a five point scale where one indicated "extremely useful" and five denoted "not useful". Generally, all components were perceived as very useful with average scores between 1.29 and 1.81. (See Figure 8.) At each training, the supplemental materials were rated as the single most useful component. Other components with very high scores on utility were "The Dying Process" and "The Grief Process". Perceived as least useful were components on "Cultural Considerations at the End of Life" and the "Opening Session/Role Play". The ranking of components was highly consistent across the multiple performance sites. (See Attachment P)

When participants were asked on the post-test to name the most useful components of the curriculum, a wide variety of responses were offered with many indicating that "the entire program/everything was helpful" (17.4%). However, the two specific components most often identified were "the dying process" (17.4%) and "the grief process" (13.2%). (See Figure 9.) When asked which component was least useful, the modal response was that all parts were useful (42.5%). However, among those who identified least useful components, the most common responses were "the role playing" (9.0%) and "the role of spiritual care" (7.6%). (See Figure 10.)

When asked for suggestions for improvement of the trainings, the most frequent responses that the participants suggested were: "lengthen the workshop to better cover all of the material" (17.5%), and "would like to see more lectures on the subject" (12.5%). Others suggested breaking up into smaller groups (4%), expanding the cultural diversity segment (3%) and engaging in more role playing/audience participation (3%). (See Figure 11 and Attachment Q.) 

Self-Reported Outcomes/Impacts

Participants were asked to respond to seven Likert-type questions in order to assess their own perceptions of impact and their plan to utilize materials. All responses clustered in an area from "somewhat agree" into the "strongly agree" range, indicating quite positive outcomes. The strongest agreement was with the item that stated "Being able to address my own personal issues concerning end-of-life issues will help me to minister more effectively to others". The vast majority (71.5%) of the 424 persons who responded to this question strongly agreed and not one respondent disagreed, (i.e., no responses of 4, 5, or 6 were recorded on the six point response scale.) Conversely, participants were least likely to agree strongly with the statement "I am better able to discuss end-of-life issues with individuals of diverse ethnic/racial backgrounds." Only 37.3% of those answering the question agreed strongly, although most showed some level of agreement. Eighteen participants who answered this question (4.3%) disagreed to some degree with this statement. It is worthy of note that this response is consistent with the somewhat lower level of approval received by this curriculum component. It may be that participants remain somewhat ill at ease in addressing end-of-life issues in communities culturally distinct from their own.

The large majority of respondents agreed that they were "better informed about issues related to the end of life (spiritual, medical or psychological)" after training, with 57% agreeing strongly and less than 1% disagreeing. However, when questions addressed specific types of knowledge, agreement was not quite as strong. An item measuring increased knowledge about "advance care planning, including do not resuscitate orders, health care surrogates and advance directives" drew responses of strongly agree from only 47.3% with 4.1% disagreeing. Similarly, an item measuring a "better understanding of the process of grief and the intervention approaches available at the time of death and throughout the grief process" drew strongly agree responses from about 50% and disagree responses from only 1.4%. A similar proportion agreed strongly that they had a clearer understanding of "my role in ministering to dying persons and their families." (48.7% agreed strongly and 2.4% disagreed). 

A single item measured intent to use "the curriculum and consumer materials within my local community." Fifty-five percent of respondents agreed strongly with this statement and 100% expressed some degree of agreement. (See Table 5 and Attachment R.)

Table 5

Pre-Test and Post-Test Summary

In summary, a very high proportion of participants reported that they had a higher level of knowledge of end-of-life care issues upon completion of this training. It appears that trainees felt most confident that the training had helped them address personal issues and understand medical and spiritual issues in a manner that would improve their ministry. They also generally felt better informed about technical, procedural and legal issues than they did prior to the training, but their responses did not express quite as high a degree of confidence in their mastery of these more complex and detailed components. The complexity of these issues may also be reflected in the number of participants who asked for expanded time and more training. 

It appears that the training offered at these six locations will affect many communities throughout Florida. All participants (100%) stated that they intend to utilize the curriculum and consumer materials in their communities. This intention appears consistent with the very high utility score awarded to the supplemental materials. This heavy emphasis on useful materials for application in home communities suggests that maintenance of communication and updating of materials may be important post-training support mechanisms for the participants. 

Previous | Next | Table of Contents

Back to Top