The following is adapted from Guarding Minds at Work.
Some evaluation questions to consider are: “How will we know whether these actions achieved our intended outcomes?” Or, “How will we know whether our investment in time, effort and money was worthwhile?”
Effective evaluation is:
- Practical: clearly relevant to your intended outcomes, straightforward, measurable, easy to implement, and cost-effective
- Flexible: adaptable to your workplace and its available resources
- Continuous: uses an ongoing quality improvement approach where employee feedback is provided over time and used to modify/improve intervention(s)
Step 1: Decide the purposes of the evaluation
There are many reasons for doing an evaluation, including a commitment to:
- Accountability – demonstrating to leadership or a funding body that promised results were obtained and resources used appropriately
- Quality improvement – an ongoing process of enhancing the excellence of services or products
- Examination of specific outcomes – tracking the progress of specific changes identified as priorities for the organization
- Cost-effectiveness – conducting a comprehensive analysis of outcomes in relation to the resources needed to accomplish them
- Uptake – determining the extent to which employees or other target groups participate in the planned actions
- Fidelity – determining whether actions were implemented consistent with the plan
- Sustainability – forecasting the degree to which the actions can be maintained over the long term
Identify the purposes and methods of evaluation before initiating the action so you will have a clear and useful answer to the question of whether it worked as planned.
Step 2: Identify key informants
Key informants are those who will provide evaluation data for each action. These key informants might include senior leaders, human resources, benefit providers, supervisory staff with responsibility for carrying out the planned action, union representatives, and occupational health and safety staff. Participants of the planned action will always be included as key informants.
Step 3: Make a list of short-term and long-term outcomes to measure
Change can take time. If you focus only on long-term outcomes, you may become discouraged. In addition to setting your long-term goals, identify desired short-term outcomes. Short-term outcomes provide rapid feedback and are typically easier to measure.
Short-term measurements and approaches could include:
- Participation – simply measuring how many people take part in your action
- Behaviour changes – how many times are participants doing things differently. In this case, be specific in advance about the change you are looking for and how you will have participants or their leaders report this activity to you.
- Absenteeism – tracking the number of days off before and after the action. Note that if the discussion was about mental health, an increase in absenteeism may represent people finally seeking out treatment for their condition. Look for ways to maintain confidentiality while tracking the reason for absence.
- Complaints, conflict, or grievances – tracking the number reported before and after the action. Note that in some cases, these may increase first as employees understand what is appropriate behaviour so the short-term increase can represent a positive as well.
- Awareness – survey employees to determine the level of awareness about whatever you are covering such as resources or policies and do it again after the action has been implemented
- Policy or process change – identify any changes made to existing policy or process as a result of your action
- Participant evaluation – asking specific questions related to the experience of the action and intentions going forward to implement change can be very useful data
- Supervisor evaluation – asking the supervisors of participants direct questions about changes in participant behaviour before the action is implemented and in the days, weeks, or months following the action
- Focus groups – bringing together participants to get feedback about how the action has impacted them
- Stress satisfaction scan – this 6-question survey can be administered before and after the action to measure how employees feel about work
Long-term measurements provide enough time for sustainable change to happen and could include any of the above or either of the following:
- Re-survey – doing a follow-up Guarding Minds at Work survey allows you to track changes in how employees rank each of the psychosocial statements
- Data analysis – several relevant sources of data are suggested for each psychosocial factor in the organizational review. This data could be analyzed before action is taken and 13 or more months later. 13 months allows enough time to compare data from the same time period as when the action was first implemented.
Step 4: Use short-term evaluation results to modify
The short-term outcome evaluation results can be used to make adjustments to the action. You may even want to launch a pilot to determine what is working well and perhaps should be enhanced or expanded. And those elements that don’t seem to be reaching their objectives could be changed or dropped. Identifying challenges to success early on allows you to take corrective action that can positively influence your long-term results.
If the short-term outcomes look positive, communicating this can improve overall morale and commitment to your process of improving psychological health and safety.
Step 5: Collect long-term outcome results
After a few cycles of short-term outcome evaluation and action improvement, it’s a good idea to review your long-term evaluation strategy. Are you still on track to measure what is important to the success of this action or do you need to make some adjustments to the data collected?
Ensure that the data collection process is working.
Step 6: Analyze results
Think critically about the evaluation results you have collected and develop a report. This report can be in the form of a podcast, video, slide presentation or publication, but make it easy for all to access and understand. Be transparent about whether your intended outcomes were achieved and include both challenges and successes.
Step 7: Present your results and decide next steps
The purpose of evaluation is for continual improvement. Present your results to both employer and employees for feedback about what went well and what could be done differently. Make your decision about whether to refine, continue or discontinue the action in collaboration with others.
Change takes time and effort. Good evaluation processes help you measure and improve effectiveness. Remember to celebrate successes along the way!