Where Are You? Enter your location in the

Where Are You? Enter your location in the

Where Are You? Enter your location in the chat window (lower left of screen) 1 1/26/20 Procuring and Selecting an Independent Evaluator RESEA Evaluation Technical Assistance

Your Moderator: Click icon to add picture Gloria Salas-Kos Senior Program Analyst and RESEA Evaluation TA Coordinator Office of Policy Development and Research, ETA, U.S. DOL 3 Cycles of learning and doing

Plan Evaluation of Program Refine Program or Evaluation Conduct Evaluation Reflect on & Communicate Evaluation Findings 4

In Our Last Webinar, We Described key steps in planning your evaluation Discussed the length of time each step may take Discussed potential hurdles and strategies for addressing them To find the webinar recording of What Evaluation Details Do I Need to Plan for and How Long Will the Evaluation Take? please visit: https://www.workforcegps.org/events/2019/05/30/13/54/What-Evaluation-Details-Do-I-Need-for-aPlan-and-How-Long-Will-It-Take 5 Todays Objectives:

We will review: Types of potential evaluators Appropriate evaluator qualifications and experience; Request for Proposals (RFP) process; Bid assessment and evaluator selection; and University Partnerships Note: This session covers basic evaluation selection and procurement information 6 Your Presenters:

Siobhan Mills De La Rosa Associate Abt Associates Hannah Engle Senior Analyst Abt Associates Lesley Hirsh Assistant Commissioner, Research & Information New Jersey Department of Labor & Workforce Development 7

Has your RESEA staff ever worked with an independent evaluator? Choose the answer that best reflects you (or your organization) 1. Yes we do this all the time! 2. Yes but only occasionally. 3. No. 4. Im not sure. 8 Does your agency work with an outside organization or university partner to evaluate

its programs? Choose the answer that best reflects you (or your organization) 1. Yes! 2. No. 3. Im not sure how can I find out? 9 Where can I find expert evaluators? Research and Evaluation Teams in your Agency or Another State Agency University Research Centers

Research and Evaluation Firms Individual Research Consultants 10 What can my internal RESEA team do? 11 What are the benefits to working with an experienced independent evaluator?

Experienced evaluators do this all the time! They have special experience and expertise in: Research methods Statistical analysis and data collection Protection of human subjects Their findings are independent and transparent No undue influence Can potentially be replicated 12 Tips for Overseeing Evaluation Efforts

Designate a primary point of contact Establish regular communication with your evaluator Regular calls Progress reports Notify your evaluator about changes to: Program implementation Evaluator procedures 13 Considerations in Selecting an Evaluator

Evaluator Qualifications and Funding 14 What qualifications should we look for? Experience conducting evaluations of the design type you are planning? On workforce programs? Lead staff with advanced (i.e., post-graduate) degree and considerable experience (e.g., 5+ years) Junior and mid-level staff to support study activities, may have less education and/or experience

Capacity and resources Other specialized knowledge 15 Additional qualifications for Impact Studies Random Assignment (RA) Studies Leading RA studies of similar size Incorporating RA into claimant selection algorithm in existing data systems Monitoring RA in complex service delivery environments Quasi-experimental designs

Creating comparison groups using your preferred methods Analyzing large datasets created using multiple data sources 16 Considerations of Evaluation Budget Evaluator Qualifications Writing and Revisions Sample Size

Budget Travel and Other Costs Data Complexity Data Collection 17 RESEA Funds for Evaluation Can use up to 10% of their funds May not be sufficient for

larger, multi-year impact evaluations Discreet tasks that match funding period Optional tasks Pooled evaluations 18 Evaluator Procurement Competitive vs Non-Competitive Processes 19

Competitive Evaluator Procurement Writing your Request for Proposals (RFP) Advertising Your RFP Assessing Bids and Selecting an Evaluator

20 Non-Competitive Evaluator Procurement Existing evaluation partnerships Research organizations State universities Internal evaluation unit in the workforce agency or in another agency in the state Always follow state and local procurement rules when selecting your evaluator!

21 Writing a Request for Proposals Competitive Procurement: Step 1 22 What are Requests for Proposals (RFP)? Formal announcement inviting qualified organizations to bid on the evaluation contract Should provide guidance on: Scope of Work

Evaluator Qualifications Proposed Budget Project and Application Timelines Legal requirements 23 The Scope of Work (SOW) Should Describe Your initial thoughts on research questions and evaluation design Tasks, activities, and deliverables you expect your evaluator to complete Level of effort and timeline associated with each task

and deliverable Desired evaluator qualifications May also be called Performance Work Statement We strongly recommend that procurement and legal staff review the SOW and RFP! 24 Include your Evaluation Timeline and Budget For more information on evaluation timelines, see What Evaluation Details Do I Need to Plan for and How Long Will It Take? 25

Evaluations Take Time Evaluations are often multi-year efforts Implementation & outcomes studies may take 1.5+ years Impact studies may take 4+ years Illustrative Example of Impact Study Timeline Year 1 Year 2 Year 3 Year 4

Evaluation planning, design, and launch Assign claimants to an intervention group or control group until target sample size is reached Track claimant outcomes (2 quarter follow-up period after claim approval) Collect outcomes data as it becomes available Data analysis Complete final report 26 Sample Impact Study Level of Effort Staff Title

Tasks Principal Investigator Refine RQs .15 FTE Design the study and impact models Oversee analysis Project Director Project management .25 FTE

Oversee all tasks Review materials and data collection instruments Mid-level Analyst Junior-level Analyst

Collect and clean data Support Project as necessary Design study procedures \ Serve site point of contact Train RESEA staff on procedures Analyze data Annual Est FTE .3 FTE .25 FTE

27 How can my evaluator demonstrate their qualifications? Descriptions of previous evaluation work Biographies and CVs of proposed staff Descriptions of companys institutional capacity Other information required by your state Request and check

references! 28 Advertising Your RFP Competitive Procurement: Step 2 29 Getting the Word Out You can advertise through a number of avenues: Your state or agency website listing contract opportunities Evaluation websites (e.g., www.eval.org, www.appam.org) Direct communication with evaluators from your evidence base (letters, emails)

University websites Social media (Facebook, LinkedIn, Twitter, etc.) Local, state, and other relevant newsletters or publications Make sure your RFP, advertisement, and procurement process all align with local, state, and federal rules regarding procurement! 30 Assessing Bidder Responses Competitive Procurement: Step 3 31

Assessing Bidder Applications Systematically assess each application, using a rubric or points system Determine and codify the most important application factors Assign points to each factor Rate applications on these factors Selecting evaluators in the top range of scores Cost is important, but not the only factor you should consider! 32

If it sounds too good to be true It probably is! Ask clarifying questions! Check references! Ask for documentation of previous work! 33 Selecting Your Evaluator Once you have made a selection, follow state procurement rules for: Notifying the selected bidder of the award Negotiating any final contract details

Signing the contract and beginning work Make it a high priority to keep the process moving. In some states, these final steps can take several months! 34 University Partners An Alternative to Third-Party Evaluators 35 What are university partnerships?

Established relationships with research universities for the purposes of evaluation Relationship is on-going University partners develop relationships with state workforce agencies University partners come to learn state data and workforce systems in depth State workforce agencies can update evaluation priorities from year to year as needs change 36 Case Study: NJ and Heldrich Center for Workforce Development at Rutgers University

NJ Department of Labor and Workforce Development has an evaluation partnership with the John J. Heldrich Center for Workforce Development at Rutgers University The Heldrich Center is located in the School of Public Policy and Planning 37

Evaluation Partnership Origin John J. Heldrich Center for Workforce Development at the Edward Bloustein School of Public Policy and Planning, Rutgers University developed a baseline data warehouse in the 1990s and early 2000s Wage matching used for Eligible Training Provider Consumer Report Card in 2004 Evaluation Partnership Evolution 2012 WDQI Grant Round 2 SLDS Grant 2014

WDQI Grant Round 4 Mapping workforce system relative to customers UI worker profiling and reemployment service (WPRS) initiative, replicated by GAO in 2018 Balanced scorecard metrics 2015 Evaluated sector-strategy training outcomes Established New Jersey Education to Earnings Data System (NJEEDS) Analyzed post-college earnings by major and distributed results to 39 colleges 2016 Interactive dashboards (higher education and State WIA) WIOA Adult and Dislocated Worker evaluation

2017 CTE employment and earnings outcomes study Interactive data visualization tools for jobseekers 39 2018 to Present WDQI Round VII Build out infrastructure and add data sources Improve agency and research access to linked data

Conduct evaluations (e.g., state-funded apprenticeships) Task Order Memorandum of Understanding supported by state dollars Accelerate learning and capacity building around partnership and expand portfolio of evaluation activity.

NJCareer Network: developing online tools to aid jobseekers on their journey from career exploration through job placement RESEA evaluation planning and execution. Closing Remarks Lawrence Burns Reemployment Coordinator Office of Unemployment Insurance, ETA, U.S. DOL

41 Save the Date: Using CLEAR A Demonstration August 19, 2019 at 2pm Eastern 42 Any Questions? Enter your questions in the chat window (lower left of screen)

43 Contact Information: Gloria Salas-Kos Evaluation Technical Assistance Coordinator U.S. DOL Office of Policy Development and Research [email protected] Lawerence Burns Reemployment Coordinator U.S. DOL Office of Unemployment Insurance [email protected]

Siobhan Mills De La Rosa Associates Abt Associates [email protected] RESEA EvalTA Inbox [email protected] 301.968.4405 Hannah Engle

Senior Analyst Abt Associates [email protected] 301.347.5796 45

Recently Viewed Presentations

  • Department of Environmental Quality Division of Mitigation Services

    Department of Environmental Quality Division of Mitigation Services

    Publishing rates for each HUC (>100 different rates) would likely not be supported by majority of customers who favor a simple rate structure. The "premium rate area" option reasonably addresses regional cost disparities. The actual cost methods increase transparency by...
  • Introduction to Assessment Literacy

    Introduction to Assessment Literacy

    INTRODUCTION & PURPOSE. Define what RIGOR means for the purpose of these modules. Use the . ASSESSMENT BLUEPRINT . to document the level of rigor of each skill. Use the . VERBS. in standards and tools that teachers have available...
  • Syllabus/Unit: code: C2 Chemical Resources Lesson number: 5 ...

    Syllabus/Unit: code: C2 Chemical Resources Lesson number: 5 ...

    metal carbonate carbon dioxide water nitric acid metal nitrate sulfuric acid hydrochloricacid acid metal sulfate metal chloride salt iron chloride Complete the word equations Complete the word equation for each reaction between a metal oxide and an acid: nitric acid...
  • The Stoning of Stephen - Mission Bible Class

    The Stoning of Stephen - Mission Bible Class

    Joseph Saves His Family www.missionbibleclass.org. 3. Egypt had enough food because Joseph was in charge. He had correctly interpreted a dream for the Pharaoh so he knew Egypt had to store grain in the years before the famine had come....
  • City of Winter Park

    City of Winter Park

    City of Winter Park. Historic Preservation Awards 2016. Good afternoon. I'm Genean McKinnon, member of the City of Winter Park Historic Preservation Board. I'm pleased to present the 2016 City of Winter Park Historic Preservation awards this morning. ... She...
  • Issue Y2K The Great War for Talent! - Home - Tom Peters

    Issue Y2K The Great War for Talent! - Home - Tom Peters

    Tom Peters' How New Business Works: Rules for Re-invention 10.10.2002
  • NIGHT by Elie Wiesel - Skyview High School

    NIGHT by Elie Wiesel - Skyview High School

    Beethoven Ludwig von Beethoven; a native of Bonn, Germany, produced dramatic romantic music for piano, strings, orchestra, and ballet.Because of the immediacy and emotional intensity of his works, Nazi laws ban Jewish instrumentalists from playing Beethoven's music. Even more identifiable...
  • Praying For Christian Workers Such As Pastors, Missionaries ...

    Praying For Christian Workers Such As Pastors, Missionaries ...

    Praying For Christian Workers Such As Pastors, Missionaries and Evangelists. Matthew 9:36-38 But seeing the crowds, He was moved with compassion on them, because they were tired and scattered like sheep having no shepherd.(37) Then He said to His disciples,...