Evidence-Informed Evaluation ofStudent Clinical Performance andCultural ProficiencyYuhwa Eva Lu, PhD, LCSW01/18/2012International Symposium at Hong Kong University1Background Social work has long sought for new andbetter ways to evaluate direct practice andprogram outcomes. The ability to accurately assess social workpractice, with evidence based criteria, is ofparamount importance.(Bogo, 2004; Gambrill, 2001; Holden et al., 2002; Baskind et al., 2001; Pithouse etal., 2002 )2

Background (continued)The social work students/workers aregenerally evaluated by paper-pencilmethod (i.e., written assignment,standard tests, process recording, etc.)which measures the knowledge domainof critical thinking ability orcomponents of clinical reasoning,3Background (continued)But, not the actual client-workerinteractions (the action) of the socialwork practice, or the transactionalaspect of clinical competence.(Regher, Yee. et al., 1998, Elstein et al.,1978)4

Background (continued)For the past two decades, at school ofsocial work, the dominated evaluationhas been the Empirical-Research Modelwhich is apt to neglect theinteractional, dynamic, and contextualelements of the relational process.(Ventimiglia, et al., 2000)5Background (continued)Discrepancies have been shown: between self-report and observationalassessment. between self-assessment and peerassessment. between student evaluation andinstructor evaluation.(Ellis, 2001; Regehr et al., 2002; Tousignany, et al., 2002)6

The Aims of This Presentation Definition of Terms: cultural vs. clinicalcompetence Introducing “Objective, StandardizedClinical Evaluation” tool & technique. Discussing SW-OSCE: Social Workadaptation of OSCE Findings of a Two-staged Pilot Study onClinical Competence Behavior Checklist(CCBC)7Definition of Terms Competence Professional/Clinical Competence Cultural Competence Cultural Competence in the ClinicalSetting CLAS Standards8

Definition of Terms (continued)Competence: The state or quality of being adequately orwell qualified; ability. A specific range of skill, knowledge, orability.(American Heritage Dictionary)9Definition of Terms (continued)Clinical/Professional Competence:“The habitual and judicious use ofcommunication, knowledge, technicalskills, clinical reasoning, emotions, values,and reflection in daily practice for thebenefit of the individual and thecommunity being served.(Epstein & Hundert, 2002)10

Definition of Terms (continued)Cultural Competence/Empathy:“A set of congruent behaviors, attitudes,practices, and policies that come togetherin a system, agency, or professional thatenables that system, agency, or professionalto work more effectively in cross-culturalsituations”(Cross, Bazron, Dennis, and Isaacs,1989, p. 7)11Definition of Terms (continued)Cultural Competence in the ProfessionalSettings (Cultural Empathy): Awareness of power differentials and oppression Ability to discern and respect different culturalvalues Ability to elicit and affirm client’s cultural identity Ability to formulate culturally congruentassessment and intervention strategies(Lu, Medina, & Kwong, 2004)12

OSCE: Objective StandardizedClinical Evaluation An Evidence Informed Assessment Tool forClinical Competence Carefully scripted and simulated servicedelivery encounter Students are observed in action with asimulated client, video taped, and debriefed(Harden et al., 1975, 1979)13OSCE (continued)OSCE has also been used in nursing,dentistry, pharmacy, clinical psychology,nutrition, and some other educationalprograms and disciplines(Ali et al., 1999; Anderson, et al., 1991; Finlay et al., 1998; Hill et al.,1997;Humphris et al., 2001; Khattab et al., 2001; Luck & Peabody, 2002; McGraw& O’Conner, 1999; Ross et al., 1998; Sloan, 1997; Shanley et al., 2001)14

OSCE (continued)OSCE has been used as summative andformative tool (for student learning,assessing student clinical performance,instructor teaching effectiveness, etc.)(Townsend, McLlvenny, Miller, & Dunn, 2001)15OSCE (continued)It has also been used for the gatewayassessment for medical licensingExams (in Australia, Canada, USA, etc.)(Reznick, Regehr, Yee, Rothman, Blackmore, & Dauphinee, 1998)16

SW-OSCE: Social Work Adaptationof OSCE To built upon the established foundationof OSCE To include both content and processcomponents of social work practice, andthe unique constellation of knowledge,values, and skills that define social workeducation.(Lu, 2000, 2003)17SW-OSCE (continued) Before SW-OSCE can be implemented andfully employed, a “competence behavioralchecklist” needs to be established on thebases of theories, concepts, and empiricaldata.(Lu & Feng, 2004) A literature review yielded 10 categories ofprofessional competence criteria, A surveyand focus groups were conducted(Lu, Medina, & Kwong, 2004)18

SW-OSCE: A Clinical Competence Behavioral ChecklistCompetence Categories Professional values(roles and ethics) Knowledge Cultural empathy Interviewing skills Intervention skills Empowermentperspectives Critical thinking Professional use ofself Evaluation Legal mandates19SW-OSCE (continued) To enhance the efficiency ofimplementation, the 10 categories wereconsolidated into 7, and then furthercollapsed into 5 categories and a “totalscore” item. The concept of “meta-competence” wasadded to the checklist, which refers to thestudent’s insight into his/her ownperformance20

SW-OSCE (Continued) A 9 point, lickert scale assessment checkinglist with 5 categories Interviewing Skills Cultural Competence Knowledge and Intervention Strategies Evaluation Meta Competence Total Score21CCB ChecklistInterviewing SkillsPoor listening skills; hinderingFair listening; attempt toReflective listening; focus on Reflective Listeningof information gathering;gather information;factual information gathering; Verbal and non-verbal communicationassumptive attitude; giving ofexpression of sympathyexpression of empathy Professional Use of SelfadviceCultural CompetenceInsensitive to client’s culturalRecognition of culturalRespect for client’s cultural values; Self awareness of clinician’s own culturevalues; judgmental attitude;differences; awareness ofability to elicit and affirm cultural Sensitivity to diverse cultural values andimposition of ideaspower differentials andidentity; professional judgmentAPPENDIX 2 Social Work Objective Standardized Clinical Evaluation (SW-OSCE)12345Please Explain the Reason for Your Score6constructsoppression; ability to discern Proficiency in cross culturalclient’s e and Intervention StrategiesUnaware of client’s needs; lackAwareness of client’sAbility to address client’s Assessment of client’s needsof self direction; setting ofexpressed needs; limited useimmediate needs; reframing and Develop mutually agreed goalsimpracticable goalsof practice models; attempt toempowering client through eclecticdevelop goals for clientpractice models; clarification of Set priorities and partialize steps for goalattainmentgoals and partialization of steps for Demonstrate use of different modelsattainment123456789EvaluationFailure to assess client’s qualityAwareness of client’sAcknowledgement of client’s Identify indicators of client’s totalof life; inability to monitorprognosis and the indicatorsprogression of change; ability toquality of lifeprogression of change; failure tofor quality of life;make timely closure, termination Timely closure, termination ofidentify indicators forpostponement of closure,and referraltermination or to make a referraltermination or referralreferral123456789Meta CompetenceInflated assessment of, orAppropriate assessment ofIdentify specific processes; Ability to recognize one’s ownunderestimation of, one’s ownone’s own clinicalrecognize the dynamics ofperformance and level of clinicalperformance and clinicalperformance and competenceinteraction with the client; aware ofcompetencecompetenceOverall Evaluationhow to improve12345678912345678922

A Pilot Study:Clinical Competence BehavioralChecklist (CCBC) for SW-OSCEA combination of Quantitative method – ANOVA, & Qualitative method –Content analysis of videotapes(Lu, 2003, 2004, 2005, 2007)23CCBC Development: Pilot Study (Pt. I)Participant Demographics: Sample: 22 of 29 MSW students Ethno-Cultural Issues class, fall, 2005 Age: ranged from 21 to 46 (mean 27,median 25) Gender: 25 females, 4 males 17 students had no previous social work orrelated experience 5 students had 3-16 years of social service orrelated experience24

(continued) All student-interviewers werevoluntary participants, with signedwritten consent Students were observed “in practice”for 10 minutes with a standardizedclient (actor) and videotaped.25(continued) Develop detailed scripts for case scenarios Develop a brief student instruction sheet(with client background and presentingproblem) Use of trained actors as simulated clients Develop a preliminary Clinical CompetenceBehavioral checklist.26

Case-Scenarios DevelopmentSix scenarios were developed to evaluate aMSW “social work direct practice with diversepopulations” course at NY in Fall 2005.Each case scenario reflected diversity inclients’ age, gender, race, religion, sexualorientation, and social economic status (SES).Each comprised a 10-15 page detailedtranscript of a life story.27(continued) #1 “Mrs. Stein”, a forty-five-year-old OrthodoxJewish woman became anxious, angry, fearful,and panicked after being told that her adolescentdaughter was pregnant. #2 “Ms. Lee”, a thirty-five-year-old immigrantfrom China was notified by her daughter’s schooland learned of her truancy and poor academicperformance.28

(continued) #3 “Mr. Shayan”, a twenty-year-old Iranianinternational student felt hopeless, helpless, andsleepless and unemployment. In the fourthscenario, #4 “Ms. Williams”, a thirty-four-year-old AfricanAmerican woman of Caribbean descent, whowas diagnosed with HIV positive, felt anguished,guilty, and had trouble maintaining relationshipswith her fiancé and family.29(continued) #5 “Ms. Rosita” a fifty-nine-year-old Americanborn Latina from high SES background, reportedthat she was experiencing depression and fearregarding her future. #6 “Mr. Rodriguez”, a married twenty-four- yearold American-born Puerto Rican, felt confused ofhis sexual orientation and was mistrustful of asocial worker.30

ProceduresPreparation/Planning Phase: 6 detailed-scripts developed A brief instruction-sheet of each scenario(with client background and presentingproblems) developed Hire OSCE trained actors as simulatedclients Preparation for all participating parties:Preview and discuss of each category (5), theindicators (52), and scoring criteria of the CCBCChecklist.31Procedures (continued) On the day of the SW-OSCE evaluation, all 5student interviewers waiting outside of theclassroom. Given the one-page instruction sheet. Ushered in for a 10-12 minutes interview one byone. Talked to the instructor for 3-5 minutes afterthe interview. Debriefed with all participants (feedback fromthe interviewer, client, instructor and peers).32

The Raters for Initial CCBC Data Standardizedclient Academic grader Faculty member (4) Student-observers Video recorder (non-expert)33CCBC Development: Pilot Study (Pt. I) (continued)Inter-Rater Reliability All raters scored equal to or within one point ofthe mean score on all categories between 81% and91% on a nine point scale.- 91% for the Overall Category- 82% for the Interviewing Skills Category- 83% for the Cultural Competence Category- 86% for the Knowledge and InterventionStrategies Category- 87% for the Evaluation Category- 81% for the Meta-Competence Category34

Inter-Rater Reliability (Pilot - Pt. I)Table 1 Results of the Inter-Rater Reliability of the SW-OSCE Student Evaluation.Percent of Raters Percent of Raters within Percent of Raters within Percent of Raters withinReliabilityEqual to MeanOne Point of MeanTwo Points of MeanThree Points of MeanMeasure(A)Interviewing SkillsCultural CompetenceKnowledge erall(B) %43%14%16%9%0%3%0%Note: Four raters participated in the rating of 22 students.35Qualitative Data Analysis Faculty Raters: 4 Chinese bilingual,bicultural social work faculty members (3of them were the instructors of “directpractice” and “ethno-cultural issues”courses). First, all raters reviewed and discusseddefinitions of the 5 categories and the 52scoring criteria of CCBC with the firstauthor.36

(continued) After each rater scored the taped-interviews separately, all raters met (atAPM of CSWE) to view the tapestogether and deliberated differentialindividual ratings. The interpretation was drawn whenconsensus reached.37Initial Findings Clinical competence does notguarantee cultural competence Racial/ethnic match does notguarantee, nor is it necessary for,rapport building. Academic achievement does notnecessarily translate into either clinicalor cultural competence and vice versa.38 .

Initial Finding (continued) Incoming students without trainingmay perform with competence, bothclinically and culturally. Prior experience or professionaltraining in social work are notindispensable for both clinical andcultural competence;39Initial Findings (continued) Of all the competencies demonstrated,“cultural empathy” was valued most bythe client (actor) to choose futureworkers.40

CCBC Development: Pilot Study (Pt. II-A)Data collected between 12/2005 to 12/2009Participant Demographics: Age from 31 to 56, Mean 31, Median 28 Voluntarily MSW participation of Ethno-Cultural Issues class Sample: N 510 (interviewers, standardizedclients, instructor, and peer MSW students)41Preparation Phase 6 detailed scripts for case scenariosDeveloped Use of OSCE trained actors as simulatedclients brief student instruction sheets (with clientbackgr