Wellbeing measurement in practice: 999 Club on asking the right questions

We promote the use of the Warwick-Edinburgh Mental Wellbeing Scales (WEMWBS) as a global measure of mental wellbeing. We have conducted a  review of studies that use the scales to bring together the evidence on what works to improve mental wellbeing in UK projects and pilots. The below practice example is designed to accompany this analysis. Here, The 999 Club’s CEO, Tom Neumark, discusses the charity’s experience of using the Short Warwick-Edinburgh Mental Wellbeing scale (SWEMWBS), their move to focus on purpose as a key aspect of wellbeing, and discovering how to ask the right questions.

The 999 Club is a grassroots charity that has been working to end homelessness in South East London for the past 30 years. They do this through their day centre and advice services, which provide a friendly open door for anyone who is experiencing or at imminent risk of homelessness.

What was the challenge?

To demonstrate our impact, for example to funders, and to improve our practice we have historically captured outcomes relating to housing, employment and accessing services. These include levels of physical health, mental health and substance misuse.

These measures remain important to us, but we wanted to bolster them and measure the longer term impact of our work. Specifically, whether the people who have used our services have found a home, and whether this is sustainable.

We believe homelessness should be fleeting and not repeated. Too often we were seeing examples of repeat homelessness, where we had supported people to find housing only for their new tenancy to fall through and for them to return to the streets.

We want people to have a home and to flourish, not languish. This is why wellbeing is an important driver and outcome of our work. Given this, it seemed sensible to measure it, to demonstrate our impact in this area and to improve our practice.

What did you do?

In our first attempt to reliably and credibly assess our impact on clients’ wellbeing we decided to use the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS). 

The scale is commonly used by charities and the public sector, making it comparable with other data sets. We thought this would also increase credibility with stakeholders such as statutory partners and donors.

How did this approach work?

We captured data using SWEMWBS as part of the initial assessment, when people first use our service. We measured again when clients stopped using our service. We wanted to compare the pre and post scores to assess the effect of our services.

When we looked at the changes in scores we found dramatic results which might be termed “zero to hero”:

• Clients often reported very low subjective wellbeing scores during initial assessments. Support services are often rationed on the basis of need, and clients are aware of this. As a result, they often default to emphasising their support needs, their inability to perform tasks and their lack of assets so that they can meet the eligibility criteria and guarantee access to the service. This can result in unreliable scores.
• During keyworking sessions clients often said that their wellbeing was improving. Clients also know they can be excluded from services if they fail to comply. As a result, they stress their attempts to meet the needs of the service.
• Clients often reported very high wellbeing scores at the end of using our services. Our services are free and, we hope, consistent and compassionate. This type of relationship is not common for people who have experienced homelessness, many of whom will have experienced bureaucratic de-personalised, capricious and seemingly cruel services. When people experience service delivered in this way they are often grateful so tell the service what they think the service wants to hear. If they think we want to hear that their wellbeing has improved, then that is what they’ll say.

These behaviours in self-reporting have the potential to mask the actual impact of our interventions. 

Clients were telling us “thanks to you I got better”, so the cynic might ask “what’s the problem?”. 

For us there were at least three concerns:

1. Zero to hero scores hinder reflective practice. Workers and managers do not have data to help them constructively improve their practice.
2. Zero to hero scores lack credibility with partners who may raise their eyebrows at such high scores. For example, I have heard of providers informing commissioners that the wellbeing scores for a particular service have dramatically improved, coincidentally just before the service was due to be re-commissioned.
3. Encouraging people to report how unhappy they are, rather than what they can do, can inhibit strengths-based working.

What did you do next?

As a team, we reflected on this experience and discussed our findings in relation to our theory of change. 

We believe that if people have a sense of purpose, a good support network and meaningful day time activities (such as work, training or participation in cultural activities) then they are better able to sustain their tenancies. 

We knew we wanted to pivot away from SWEMWBS so we researched what measures and what way of measuring would better line up with our approach.

The excellent resource “What matters for our sense of purpose?” from What Works Wellbeing helped guide our thinking and decision to ask how worthwhile people feel the things they do in life are.

Focusing on sense of purpose is better for us because:

• Sense of purpose is highly correlated with what we do, and our associated health and ability to do it.
• As well as housing, our focus is on work, health, physical activity and engagement in cultural activities.
• We can reflect on how clients are pursuing their goals, building on their interests and developing their sense of purpose.

Wellbeing goes beyond pleasant emotions and absence of ill-health; it encompasses the perception of our lives having value, with a sense of meaning and fulfilment.

Though SWEMWBS contains the statement ‘I’ve been feeling useful’, we felt that the other six statements clouded the issue.

To measure sense of purpose more specifically we used the ONS measure of purpose and developed an additional question to capture contribution:

1. “Overall, to what extent do you feel the things you do in your life are worthwhile?” (Taken from ONS4)
2. What daytime activities are you involved in? (i.e. work, training, volunteering or cultural activities)

We ask all clients these questions at the beginning and end of their use of our service and in a follow-up six months later. We added this third data capture point to gain a longer-term perspective and assess if we have supported people to escape homelessness. This time point is based on research indicating the first six months is an important period of tenancy sustainment. 

Questions are now asked by peer workers, rather than during client support sessions. By separating data collection from the support worker relationship, the aim is to minimise pressure on the client to say what they think their support worker wants to hear and capture a more objective response. 

Some 999 Club staff were cynical about how the information would be used and how reliable it would be, but understood that it was part of our theory of change i.e. that our best hope is that we are supporting people to escape homelessness for good. 

To overcome this we used the information in team reflective practice with staff bringing examples of where people have stayed in employment six months after leaving the service and discussing what they did and why they think it achieved results, rather than assessing the performance of individual workers.

Recommendations for others

Through the process we learned that if the measurement of wellbeing is not properly implemented the results can be literally incredible. 

On the basis of our experience, I would recommend that organisations looking to measure wellbeing:

• assess initial and distance travelled scores independently of the key working relationship;
• use the data to compare outcomes for different services, rather than track individuals’ distance travelled;
• be open to adapting and evolving your process using reflexive practice;
• have a clear theory of change with a defined approach to measuring and evaluating.

Tom Neumark is an experienced charity leader with a background in both housing and community development. Before working at 999 Club Tom was CEO at The Peel, a community development charity in Islington, and a Director at Hestia Housing and Support.