The 5-Step Approach to Evaluation

Designing and Evaluating Behaviour Change Interventions

Guidance for service providers, planning partnerships, funders and commissioners

Authored by
Catherine Bisset and Leah Gilman

The 5-Step Approach to Evaluation

Designing and Evaluating Behaviour Change Interventions

Guidance for service providers, planning partnerships, funders and commissioners

Authored by
Catherine Bisset and Leah Gilman

Table of Contents

Foreward

Introduction

Background: The tricky business of assessing impact in a messy world

The 5-Step approach

Step 1: Identify the problem

Step 2: Review the evidence

Step 3: Draw a logic model

Step 4: Monitor your logic model

Step 5: Evaluate the logic model

Helpful Resources

References

Appendix 1: Example data collection framework

Appendix 2: Evaluation report structure

Foreward

"I am delighted to introduce this easy-to-use 5-step evaluation guidance which is grounded in tried and tested methods. Not only should it help evaluate services of any size, but also to design more effective services from the outset.

This practical guide provides links to the key evidence on 'what works' and is packed with examples making it a valuable resource for anyone who wants to assess their contribution to outcomes including funders, planning partnerships, service providers and service staff".

Nicola Edge
Head of Justice Analytical Services
Scottish Government

Introduction

This evaluation pack is aimed at both service providers and funders who aim to promote behaviour change.

For funders and planning partnerships, it aims to:

Offer a strategic, evidence-based and outcomes-focused planning tool

Offer guidance on how to assess evaluations from service providers and therefore direct funding to greatest effect.

Demonstrate the role you can play in promoting and enabling high quality evaluations from those you fund.

For service providers and policy makers, it aims to:

Provide guidance on planning an evidence-based service with a "built in" evaluation process

Provide guidance and resources for you to effectively assess, understand and demonstrate how well your service is working in relation to your aims.

Offer an alternative to randomised control trials, using a "logic model" approach to evaluation, which any service provider can use to evaluate any intervention, regardless of size.

Encourage continual review and improvement of services.

Other audiences

The pack is primarily aimed at funders, commissioners and service providers with a focus on behaviour change. However, it is likely to be of interest to others with an interest in effective evaluation (such as inspectorates) and the approach can easily be adapted for projects that do not primarily seek behaviour change.

If your interest is in REDUCING CRIME AND REOFFENDING then we have also published a tailored version of the 5-step approach for funders and service providers working in that field.

Background: The tricky business of assessing impact in a messy world

How was this pack developed?

This pack has been developed by Scottish Government researchers in Justice Analytical Services in collaboration with stakeholders in other organisations, with the aim of promoting and supporting effective evaluation. Individuals in the following organisations provided invaluable feedback on multiple drafts of the guidance:

The Robertson Trust

Evaluation Support Scotland

Coalition of Care and Support Providers in Scotland

A Scottish approach to evaluation

Co-production
Our approach to evaluation enables funders and service providers to work together in pursuit of their shared aims - to improve outcomes for service users and communities. The 5-step approach also engages with service users' views as a resource for evaluation rather than seeing users solely as an object to be measured.

Asset-based
The 5-step approach focuses on ways in which evaluation is possible for services of any size, rather than expecting all services to use an experimental evaluation method which may not be appropriate or possible for smaller, community-based organisations. The 5-step approach allows even the smallest service to demonstrate the contribution they are making to change.

An Improvement Culture
Evaluation enables improvement and even the most successful service can always be developed further. Furthermore, with the 5-step approach, evaluation is an on-going process, not something to be saved for last. This means that services can be continually improved in order to best meet the needs of their users.

How do you know if you are making a real difference to users (making an impact)?

It's not easy to find out if you're making a real difference to people, especially in the chaotic real world. There are 100s of variables which can effect people's attitudes, motivations and behaviour. So how can you tell if your project is making any difference?

Researchers and scientists generally agree that BEST way to determine if your project or service has made a difference is to use a randomised control trial (RCT), sometimes referred to as an "impact evaluation" but these are not easy to do in practice, especially in a complex social setting.

What are impact evaluations / RCTs?

What is an impact evaluation or RCT?
An impact evaluation or RCT is a much like a scientific experiment. One group (the 'treatment' group) experience your intervention and one group (the control group) does not. You then compare the outcomes for both groups to see if your intervention made any difference. In other words, if you really want to know if you've made a difference, you need to know what would have happened if the same (or similar) users DIDN'T receive your service. This enables you to ATTRIBUTE changes in users to YOUR service rather than other factors like motivation, another programme or family influences.

The control group must either be selected completely at random or otherwise be very carefully selected to have very similar characteristics. Otherwise, you cannot be sure that any apparent differences in results at the end are not the result of differences that were already there at the start and therefore nothing to do with your intervention.

The difficulty with RCTs...

You need a large sample
RCTs are only meaningful IF there is a large control group with very SIMILAR CHARACTERISTICS to the users (the counterfactual). Scotland is a relatively small nation and behaviour change projects often target small or localised populations, making them hard to carry out.

They can be expensive
Funding may be a barrier since RCTs may be expensive to run and therefore not cost-effective as a means of evaluating small-scale projects

They can't tell you everything
RCTs can't tell you WHY something is effective (or ineffective) so learning anything about HOW a project worked is tricky using this method.

Do impact evaluations even ask the right questions? Contribution not attribution

Example - contribution to achieving outcomes

Behaviour change is complex and you can rarely make a long lasting social change on your own. Say you want to design an intervention to increase the number of families who recycle. You quickly realise that to achieve this long lasting social change in behaviour that you need to work collaboratively with partners - local communities, funders, environmental specialists, a marketing firm, supermarkets and schools. The question then becomes….if we do achieve a change in behaviour which one of us is responsible? The answer is, of course, all of you have a distinctive role in contributing towards achieving the outcome……so shouldn't any evaluation of YOUR service assess the extent of YOUR contribution to achieving the outcomes? Impact evaluations (RCTs) put all the pressure on your service to prove you've improved recycling rather than assess the contribution you are making.

An alternative to RCTs

A "middle ground" approach
Rather than carrying out a small RCT which might be impractical and would only deliver meaningless results, we recommend that small-scale project organisers carry out a 5-step approach to evaluation. This is summarised in the following slides and detailed in the remainder of this pack.

This approach to evaluation is practical for projects of any size but does rely on providers having a clear sense of what they're hoping to achieve and how they're going to get there - a theory of change. For this reason, using the 5-step approach, must begin at the planning stage.

What is evaluation really for?

Although doing evaluation requires the use of techniques and tools, bear in mind that its overall purpose is to help you (re) design services, ask questions, gather evidence, interpret the evidence, communicate important information about your service and take informed decisions. In this sense, the ability to ask relevant questions and clearly communicate the answers at the right time to the right people are key skills in making evaluation useful.

The 5-Step approach

The 5-step approach to evaluation

Identify the problem

If your ultimate aim is to change people's behaviour, you need to be clear what it is you are trying to change and why there is currently a need for this to happen

Review the evidence

What you intend to do should be grounded in the evidence of 'what works' and why. Service providers should review the available evidence in order to plan activities which can be expected to achieve the intended behaviour change. The evidence should guide what you do and help you to understand the process through which it should work.

Draw a logic model

A logic model is a diagram which shows, step-by-step, why the activities you plan should achieve your aims. The logic model forms the basis for evaluating the whole project - you are going to test whether these steps happened as you predicted.

Identify Indicators and monitor your model

Use the logic model to identify indicators (i.e. measurements or observations) that things actually happen as you predicted. You will need to collect data about your project FROM THE START on inputs, activities, users, short, medium and long-term outcomes.

Evaluate logic model

Analyse the data you've collected on your various indictors to evaluate how well your project worked for your various users. Report on whether your data suggests the logic model worked as planned. Be honest about any areas which were less effective. Use this to improve your service.

The 5-step approach: A summary

1. Identify the problem
It is essential that you are clear from the start about the problem you are aiming to address. What kind of behaviours are you aiming to change and why is this is needed at this particular time and place? Perhaps there are local gaps in service provision or recent events which suggest intervention would be timely.

2. Review the evidence
The most effective projects and services build from existing evidence about what works - they learn from previous experiences. Therefore, the 5-step approach puts a deliberate emphasis on using existing evidence and the evaluation should measure the extent to which your service is based on evidence. The first step is therefore to gain an understanding of the existing evidence base in order to plan your service.

3. Draw a logic model of how your service should work
The logic model is a step-by-step diagram which shows the ultimate outcomes you are aiming for and step-by-step how you intend to achieve them. It details inputs (e.g. money, staff, resources) needed to deliver your activities and how they should lead to short, medium and long-term outcomes and ultimately meet your aims.

It should describe how evidence, funds and staff will be used to design and deliver activities and how exactly, based on your review of the existing evidence, these activities are expected to lead to short, medium and long term outcomes.

A template and excellent guidance can be found here: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

4. Identify indictors and collect monitoring data
Using your logic model as a guide, identify indicators that will test whether the project actually worked as the logic model predicted. You should collect data on what activities were delivered to whom, as well as evidence that they led (or didn't lead) to the short-term and longer-term changes you anticipated.

Nb. It is important that you collect 'base-line' (pre-project) information about your users to compare with information you later collect during and after the intervention.

5. Evaluate logic model​
You now need to analyse the data you've collected in order to test whether the project worked in accordance with your logic model. You should assess how well activities were delivered, levels of user engagement and whether users' needs were met or their attitudes changed. Case studies can be used to illustrate examples of who the service worked for and did not work for and why that might be.

WARNING!

Do not leave planning your evaluation until the end of your project

Steps 1-3 should be carried out before the project begins

Step 4 (monitoring) should continue from the very start to the end of your project (and, ideally, beyond).

Step 5 (analysis) should not be left to the end either. Interim and on-going evaluations will enable you to make improvements to your project or service.

Step 1: Identify the problem

Identify the Problem

Before it is possible to design an effective service, it is essential that you are clear what behaviour it is that you are aiming to change and why this should be a priority in the context you're intending to work.

An example:

WHAT is the problem? More than 1 in 5 people in Scotland continue to smoke. Long-term declines in rates of smoking have stalled in recent years and smoking is more common in areas of socio-economic deprivation, such as X.

WHY is this a problem? Smoking is a known cause of cancer and heart disease. Rates of smoking are therefore likely to be one cause of health inequalities.

What is your ultimate AIM? Decrease the numbers of people smoking and frequency of smoking in area X

Step 2: Review the evidence

What is "the evidence base"?

For the purpose of evaluation and planning, "the evidence base" refers to all available information which might guide what you do in pursuit of your particular aims.

Evidence can come in many different forms, including anecdotes or personal experience. However, when we talk about evidence in this context, we are usually talking about empirical evidence - that derived from purposively designed research studies. However, be aware that because the evidence base is derived from multiple studies, is not always obvious what will work. Studies can have contradictory findings or may ask different kinds of questions.

The following short guide, produced by the Centre for Research on Families and Relationships, Inspiring Scotland and Evaluation Support Scotland, explains what it means to say a programme is "evidence-based:"
http://www.crfr.ac.uk/assets/CRFR_ESS_IS_Evidence_base_briefing.pdf

Why review the evidence base?

Crucial for Planning
A well-designed project will be based on the available evidence about 'what works,' and what doesn't, in relation to your aims. Reviewing the evidence base as part of the planning process will give you the best chance of achieving behaviour change.

Crucial for Evaluation
However, following the 5-step process, reviewing the evidence is also a crucial phase in the evaluation process. Assuming that an experimental design (i.e. RCT) has not been possible, the 5-step process allows you to evaluate the project by assessing the quality of evidence behind a project's theory of change. I.e. what reason do you have to believe that the project's activities should lead to the outcomes envisaged? In addition, it is important that you have a clear idea of the causal processes which underlie the logic of your project so you can plan how you will gather evidence about whether or not they actually took place (see step 4).

Sources of evidence

Research Evidence
Including results of randomised control trials (RCTs), surveys and qualitative studies (e.g. interviews or focus groups). Systematic, literature or evidence reviews synthesise research evidence on a particular topic.

Evidence from Prior Evaluation
If your service (or a similar one) has already been running for a period of time, your own previous evaluations may provide evidence as to whether the approach works or not, how and for whom.

Anecdotal Evidence
Over years of working in a particular field, your own experiences and those you hear about from others can be a further source of evidence. However, whilst valuable, it is important to remember that such evidence may be particularly subject to bias since it will not have been collected systematically.

Research and/or evaluation evidence should be used where available. However, there is no a simple answer to what counts as "good evidence." It depends on the question you are trying to answer. For more detail see these short videos from the Alliance for Useful Evidence: http://www.alliance4usefulevidence.org/

For best results, use a range of evidence

To draw the most robust conclusions about 'what works,' and why, you should take account of evidence produced through a range of methods. For example, quantitative studies (including the results of RCTs) might help you to establish what usually works and for whom. Qualitative work (e.g. interviews with users who 'succeed' and 'fail' and/or with practitioners) might help you to understand the processes through which interventions work or don't work and consider why barriers may exist to achieving your aims.

TIP!: If you are short on time and resources, systematic and/or literature reviews are an excellent source of evidence. They often analyse both quantitative and qualitative studies on a particular topic and should do the work of summarising all this evidence for you.

Finding evidence

When time and resources, are limited, evidence reviews (also called systematic reviews or literature reviews) are a realistic solution - enabling an overview of the evidence in a relatively short time.

Online databases and archives are the most convenient means through which to locate evidence reviews. The following slides provide links to topic-specific databases and some examples of individual evidence reviews in health, education, environment and sport behaviour change aims. However, the following databases can be of general help in locating relevant evidence:

Search academic databases:

http://www.mendeley.com/dashboard/

http://scholar.google.co.uk/

Search government archives:

http://www.gov.scot/Publications/Recent

https://www.gov.uk/government/publications

TIP! Try searching for "evidence/literature/systematic review" + your behaviour change aim (i.e. "smoking cessation" or "increase recycling").

Area | A or P* | Topic | Link
Health and Social Care | A | Scottish Government Research | http://www.gov.scot/Topics/Research/by-topic/health-community-care
Cochrane Collaboration | http://summaries.cochrane.org/
NICE (guidance and evidence helpful) | http://www.nice.org.uk/
Health Scotland | http://www.healthscotland.com/resources/publications/search-result.aspx?page=1
Institute for Research and Innovation in Social Sciences (IRISS) | http://www.iriss.org.uk/resources
P | Review of 6 health interventions | http://www.storre.stir.ac.uk/handle/1893/3171#.VEYd1o10zmI
Preventing harmful drinking | http://www.nice.org.uk/guidance/ph24
Smoking cessation services | http://www.nice.org.uk/guidance/ph10
Drug treatment and recovery | http://www.gov.scot/Publications/2010/08/18112230/0
Using cycling helmets | http://summaries.cochrane.org/CD003985/INJ_campaigns-to-encourage-children-to-wear-cycle-helmets
Education | A | Scottish Government Research | http://www.gov.scot/Topics/Research/by-topic/education-and-training
Education Endowment Foundation | http://educationendowmentfoundation.org.uk/toolkit/
Joseph Rowntree Foundation | http://www.jrf.org.uk/publications
P | Attainment in writing | https://www.gov.uk/government/publications/the-research-evidence-on-writing
Raising attainment/changing attitudes | http://www.jrf.org.uk/sites/files/jrf/education-attainment-interventions-full.pdf
Crime and Justice | A | Scottish Government Research | http://www.gov.scot/Topics/Research/by-topic/crime-and-justice
Centre for Youth and Criminal Justice | http://www.cycj.org.uk/resources/
P | Reducing reoffending | http://www.gov.scot/Resource/0038/00385880.pdf
Reducing reoffending | https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243718/evidence-reduce-reoffending.pdf
Sport | A | Scottish Government Research | http://www.gov.scot/Topics/Research/by-topic/sport
P | Examining legacy of major sporting events | http://www.gov.scot/Resource/0044/00449028.pdf
Barriers/enablers to regular exercise | http://www.gov.scot/Publications/2006/09/29134901/0
Environment | A | Scottish Government Research | http://www.gov.scot/Topics/Research/by-topic/environment
P | Reducing climate change | http://www.gov.scot/Resource/Doc/340440/0112767.pdf
All areas | A | Evidence for Policy and Practice Information Coordinating Centre (EPPI) | http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=60

* A = Archive of relevant publications, P = specific publication

Under-researched areas

There might be a wealth of evidence about 'what works' in some areas (e.g. smoking cessation). However, you may find a lack of research in relation to your aims. What should you do if this is the case?

Look at similar or related contexts
If you can identify related areas where a larger evidence base is available, you may be able to make logical inferences about what might work based on what has worked in these areas.

E.g. There may only be limited evidence about how to best support persons who become addicted to online gambling. However, the evidence relating to gambling more generally, or addictions in general, may be useful.

Use a rationale
The above approach may not be appropriate or possible in all cases. However, it is always important that your ideas about what might work are based on some kind of rationale. You should be able to explain why, logically, your suggested intervention should achieve your intended aims.

A fictitious example:

How the evidence base supports an intervention to promote young women's physical activity

Intervention (what are we doing?)
Evidence (why are we doing this?)

This project aims to increase physical activity from childhood into adulthood.

Multiple international systematic reviews, drawing on cross-sectional and longitudinal studies have demonstrated the positive impact of physical activity on physical and mental health (see Scottish Government Literature Review, 2004). Physical activity habits have been shown to become established within childhood.

The project is targeted at girls in the final year of primary school and first two stages of secondary school.

Statistical evidence shows that women are more likely to do little or no physical activity than men and that this divergence from their male counterparts begins around the age of 11 (Scottish Health Survey, 1998. 2003)

A choice of team and individual activities will be offered each week, e.g. dance or dodgeball. An emphasis will be made on enjoyment over competition or skill development. There will be no performances or leagues.

A systematic review of the international literature on promoting physical activity, highlighted a need for greater choice for young people, including non-traditional options. Reviews of quantitative and qualitative research by NICE (2007) demonstrate that competition and fear of having to perform may be barriers to taking part in physical activity, particularly for adolescent girls. However, enjoyment has been shown to be a key factor in overcoming these barriers (NICE 2007, Rees et al. 2006)

Social media will be used to promote activities and encourage network-building between participants.

The same reviews by NICE and case-study analysis by the British Heart Foundation (2011) have shown that peer approval and peer participation in physical activity encourages others to join in.

Step 3: Draw a logic model

Show how the project should work:
Clear links between resources, activities and outcomes

What are logic models?

Logic models are step-by-step diagrams which simply show:

What you're hoping to achieve in the long run (long-term outcomes)

The process (short and medium term outcomes) through which your planned activities can be expected to lead to long-term aims.

What resources will you need to do this (inputs)

Who can use them?

Anyone who is planning activities with particular aims in mind can benefit from using a logic model. This includes funders and commissioners, who might use them to plan how to assess applications and allocate funds in pursuit of their overall aims, as well as organisations and individuals planning behaviour change projects or services.

A very simple,evidence-based logic model

A very simple logic model for curing a headache:

Why use logic models?

Because logic models force you to be specific about what you're hoping to achieve and how, step by step, you're going to get there, they are helpful in a number of ways:

1. Planning: Are we clear what we plan to do and why we're doing it?

2. Monitoring performance: Are activities being delivered as you hoped?

3. Continual improvement: Are these activities working (for everyone)? Could we do things better?

4. Transparency: Does everyone within the organisation, and stakeholders outside of it, understand what we're doing and why we're doing it?

5. Evaluation: How successful were we in achieving our aims? Did our model work as expected?

A logic model template to use

This blank template can be found here
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Logic model column content: A quick guide

Situation/Priorities: | What is the existing need/problem you are aiming to address?
Input: | What you need to invest (money, what evidence was embedded, materials, equipment, venue, technology, partners)
Activities: | What you do (e.g. conduct workshops, meetings, sessions, develop resources, assess, facilitate, provide one to-one support)
Participation: | Who you reach (e.g. users, clients, agencies, decision-makers, customers)
Short term outcomes: | What change happened in the short term? (e.g. awareness, learning, knowledge, skills, attitudes, motivations, aspirations, opinions)
Medium term outcomes: | What change happened in the medium term - ACTION (e.g. Practice and demonstrate new skills, behaviour, decision-making, policy change, social action)
Long term outcomes: | What is the ultimate outcome? (e.g. social change, economic change)
Assumptions: | (Linked to your review of the evidence) what assumptions need to be true in order for your model to work?
External factors: | What other factors will influence whether or not your outcomes are achieved? (e.g. economic conditions, local facilities, family context)

A high level funding logic model

Logic models can be a valuable tool at every stage of planning and delivery of projects and services.

The logic model shows an example of how they can be used by funders to plan their activities. This example is a generic framework developed by the Robertson Trust.

A project-level logic model

The following logic model shows how a fictitious project, aimed to increase young women's physical activity levels is expected to work. It is based on international evidence about 'what works' to promote active lifestyles, particularly for young women (see image below). It shows clear links between activities and the expected outcomes, based on what research studies tells us.

This model is quite general, "real life" service providers should be a bit more detailed about the evidence they have used to design and deliver the intervention and also describe the content of activities in more detail.

Step 4: Monitor your logic model

Use the logic model to identify indicators

Once the logic model is completed, you need to figure out how you will be able to tell if your model works as predicted, or not. To do this, you should:

1. Devise "evaluation questions" - specific questions that you need to answer in order to test whether the model is working as predicted.

2. Identify specific indicators (measures or signals of some kind) that can answer these questions and therefore provide evidence that your model is or isn't working as expected.

Example indicators for Activities and Outcomes

Data Collection Principles

Now you've identified your indicators, you need to decide on a way of measuring or observing these things. There are lots of different methods you can use to collect this data but some basic principles to observe are:

Collect data for every stage of your logic model, including resources and activities as well as outputs

Collect data at a unit level (i.e. about every user of the service) and at an aggregate level (i.e. about the service as a whole). Unit level data can be very useful as it can tell you who the service is working for and who it isn't and you can follow the progress of individuals over time. It can also be combined to give you overall data about your service. But remember, if you only collect aggregate data you will not be able to disaggregate it and therefore collect evidence about particular individuals.

Follow users through the project. You should collect data about users at the very start, throughout and ideally beyond completion of the project. This will enable you to evidence whether users have changed, in terms of their attitudes, behaviour or knowledge.

Make use of numbers and stories. Collect qualitative as well as quantitative evidence. Averages and percentages can help you to assess overall trends and patterns in outcomes for service users. Talking to people, hearing about the views and experience of users and stakeholders will help you to explain these patterns.

Don't reinvent the wheel. Standardised and validated (pre-tested) tools are available to measure such things as self-esteem, wellbeing and employability. Using these will enhance the reliability of your evidence and save you valuable time. Freely available tools are detailed here:

http://www.clinks.org/sites/default/files/UsingOffShelfToolstoMeasureChange.pdf

http://www.evaluationsupportscotland.org.uk/resources/tools/

http://inspiringimpact.org/resources/ (follow link to "List of Measurement Tools and Systems")

Be realistic and proportionate. Expensive and/or experimental projects should collect greater amounts of data than well-evidenced and established, cheaper projects. You might want to give questionnaires to all users but it would usually be sensible to carry out in-depth interviews with just a smaller sample of your users.

Data collection methods

Various methods can be used to collect data in relation to your evaluation questions. Data can be collected from service users, staff or outside agencies. Not all methods will be suitable for all projects. Evaluation Support Scotland have produced excellent guidance on using different approaches.

Using Interviews and Questionnaires
http://www.evaluationsupportscotland.org.uk/resources/129/

Visual Approaches http://www.evaluationsupportscotland.org.uk/resources/130/

Using Qualitative Information http://www.evaluationsupportscotland.org.uk/resources/136/

Using Technology to Evaluate http://www.evaluationsupportscotland.org.uk/resources/131/

More general advice on generating useful evidence can be found in the "Evidence for Success" guide http://www.evaluationsupportscotland.org.uk/resources/270/

TIP! The most rigorous evaluations will be based on data collected using a range of methods

Data capture

You need a way of capturing and storing the data you collect which will make it easy for you to analyse.

Input data into an Excel spread sheet (or any other database that allows the data to be analysed rather than just recorded).

Some data could be simply recorded as raw numbers such as costs, number of staff or age.

Some data might be recorded using drop-down menus. E.g. user characteristics (ethnicity, male/female,) response options in questionnaires or attendance at a particular session.

Qualitative data (e.g. from interviews and focus groups) may need to be transcribed or recorded via note-taking.

Data analysis

Numerical data or "tick box" answers might be analysed and reported using percentages and/or averages. E.g. "the median (average) age of users was 16" or "80% of users rated the sessions as 'enjoyable' or 'very enjoyable'."

BUT remember to also report actual numbers as well as percentages, especially if you have only a small number of users. It can be misleading to say 66% of users attended a session, if there are only 6 users in total.

Where you have collected qualitative data (e.g. answers to open questions or interviews), go through all of the responses and highlight where common responses have been made by different people. These common responses can be reported as 'themes', to summarise the kinds of things people have said in their answers.

A data collection framework

A data collection framework is really useful for evaluators. It is a document, often in the form of a table, clearly setting out:

What data you will collect in relation to each stage of the logic model

From whom or what, will you collect your data

Where and how you will record your data (e.g. on a database)

Appendix 1 shows an example of a fictitious data collection framework which is designed to test our previously described intervention to increase physical activity in young women.

Step 5: Evaluate the logic model

Analysing your data to evaluate the project

Once you've collected some or all of your data you can use it to analyse whether or not your model is working as predicted. Analysis is not just a case of describing your data. You need to address the following questions:

What does the data tell you?

Why are you seeing these results (it could be because of your activities or external factors)?

What are you going do about this? How can you improve the outcomes?

Nb. Although you should definitely carry out this process at the end of your project, earlier interim analysis and evaluation is also highly valuable in order to identify problems and improve your service on an on-going basis.

Who should carry out evaluation?

Don't automatically assume that outside evaluations will be more helpful or reliable, nor that funders will necessarily view them this way.

As the next slide shows, there are advantages and disadvantages to both outside and internal evaluations. You should consider these carefully before deciding which approach is right for your organisation.

You may also want to consider commissioning outside expertise to support with particular stages of the evaluation (e.g. designing a data collection framework or reviewing existing evidence).

Whatever your decision, remember to budget for either internal evaluation or external expertise in your funding proposals. ESS provide further guidance on budgeting for self-evaluation:
http://www.evaluationsupportscotland.org.uk/resources/237/

Outside vs. Internal Evaluation

Self evaluation by staff member(s)

Commissioning outside evaluation

Advantages
Advantages

Cheaper

'In house' evaluators should have a clearer idea of your aims and project

Personal investment in improving the service

Easier to evaluate on an on-going basis and implement improvements continuously

Findings may be perceived as more reliable or less biased by some funders and other stake-holders

Evaluators trained in data collection and analysis

Offer an 'outsider' perspective

Disadvantages
Disadvantages

Staff may lack the skills or time to carry out evaluations

Staff may feel pressured to report positive findings

May be perceived as less reliable by some funders

Outside evaluators are usually brought in at the end of a project, limiting ability to implement on-going improvements.

May lack 'insider' knowledge about the project

May also feel pressured to report positive findings to those commissioning them

Testing the logic model: What does the data tell you?

Did the project work as it should have? The data you've collected will help to tell you whether your model worked as predicted, at each stage of the model. The following are examples of questions you might now be able to answer.

Inputs

Which aspects of the service were/were not evidence based?

How much money was spent on activities? Was it sufficient?

How many staff were employed and at what cost?

What was staff/user ratio?

What did the staff do?

How many staff were trained?

What was the training?

Were there enough staff to deliver the activities as planned?

What other resources were required?

Activities and Users

Who were the target group and was the intended target group reached?

What was the size of the target group/ their characteristics?

What were the activities/content?

How many participants were recruited? How successful were recruitment procedures?

How many of the target group participated, how many completed and how many dropped out?

How many sessions were held?

How long was an average session?

Did staff have the right skillset to deliver the content?

Outcomes

How many improved or made progress/did not improve or make progress?

What were the characteristics of the users who made progress?

What were the characteristics of the users who did not make progress?

What type of progress was make e.g. skills, learning?

Did users achieving short-term outcomes go on to achieve longer-term outcomes?

Analysing outcomes: Evidence of change

Outcomes are usually about CHANGE. You might be interested in changes in participants' knowledge, behaviour needs or attitudes (depending on how your logic model predicted your project would work).

Because you are interested in change, it is not enough simply to observe or measure users after the intervention. Participants might display the desired behavior or attitudes after your intervention but you cannot be sure they didn't already hold these views or behave in this way beforehand.

This is why it is so important that you collect data from the very start of your project. This enables you to compare users' views, behaviour or knowledge before and after the project - giving you evidence of whether or not change has occurred. E.g. you could use a standardised questionnaire to measure users' self-esteem before, during and after the project.

Limitation! Even when making comparisons in this way, you cannot be sure that your project caused these changes. There may have been other factors (see sections Evidence of Change & Subjective Views on Contribution). Be honest about these limitations in your reporting.

Explaining outcomes: Assessing contribution

Given the complexity of the social world, it is very unlikely that any single project can make a difference to people's behaviour on its own. Where change is evidenced in users (both positive and negative), it is likely that there are multiple causes for this and your project will only be a part of this.

Without using a randomised control trial (which as we have said is often impractical), it is very difficult to really measure the contribution of a single project. However, we can get a broad sense of the relative importance of the project and how it might have contributed to change, in conjunction with other influences

There are two key ways of doing this:

Subjective views on contribution

Identifying potential outside influences

Subjective views on contribution

Users, staff and other stakeholders are valuable source s of evidence in order to assess the relative contribution of your project to observed changes in users, in relation to other influences. You can:

1) Ask users whether they received other forms of support or influences on their behaviour?

2) Ask users to rate the extent to which each form of help contributed to their success, for example, did they say it was the project, their family, friends, another intervention or their own desire to succeed?

3) Ask others who know the users (e.g. family, teachers, social workers) to rate the relative influence of the project on observed changes.

Limitation!

Asking users and staff to judge the influence of a project runs the risk of 'self-serving bias'. This is the well-established tendency for people to take the credit for success and underplay external factors. One way to limit this tendency is to tell staff, users and other participants that you will be asking others to also assess the contribution of the project. Be honest about this limitation in your evaluation reports.

Identifying potential outside influences

By thinking about other potential influences, outside of your project, which might also have influenced behaviour change, you can put your own evidence into context.

Having identified potential influences, you may then be able to exclude or acknowledge whether they actually influenced your own users.

For example, in relation to the project on young women's physical activity, potential influences you might consider are:

The weather - Unusually good or poor weather might have encouraged participation in the project and/or other kinds of physical activity.

Local facilities - The opening or closure of sports and leisure facilities might have encouraged or discouraged physical activity.

Economic conditions - Changes in employment or income levels for families could impact on user participation in the project and outside forms of physical activity (even if free - travel costs may impact).

Explaining negative or mixed outcomes

It is extremely unlikely that your data will show that your model worked as predicted for all users. Be honest about this. It is helpful to analyse users with poor outcomes (no change or negative change), as well as those showing positive outcomes. Use the data (and any other relevant information) to consider:

Are there any patterns in terms of who shows positive/poor outcomes?
E.g. Are there better outcomes according to gender, age or socio-economic group?

Can you explain these patterns through reference to the way the project was carried out?
E.g. Were activities better targeted at particular groups or likely to exclude others?

Are there any external factors which explain these patterns
E.g. Do cultural norms or practical factors mean particular groups were always less likely to engage?

Remember! Your project cannot explain everything. You are only ever contributing to change. This is true of both positive and negative outcomes. If your project demonstrate poor outcomes, you should analyse external factors as well as internal processes in order to explain them.

What can you do to improve?

The crucial next step in the evaluation process is to use your explanations of outcomes in order to improve your model.

Can you address any issues at the input stage (e.g. issues with staff training or resources)?

Should you extend activities which appear to have been successful?

Is it best to stop or redesign activities which the data suggests are ineffective?

Can you improve the model to better target groups with negative outcomes?

Can you do anything to address external factors which have negatively impacted? E.g. provide transport

Who needs to know about this?

Don't keep your evaluations to yourself! They are important sources of evidence to various groups.

Funders will usually require an evaluation report in order to assess the contribution of a particular project (and their funding of it) to positive change. Remember, funders will also want to see evidence of a commitment to continual improvement. So be honest about difficulties and clear about future plans. Advice on producing evaluation reports can be found in appendix 2.

Staff should ideally be involved in the production of evaluations (particularly at the stage of explaining outcomes and planning for improvement) and should certainly be informed of their findings. This will ensure everyone has a shared vision of how the project is working and how to improve their practice.

Other organisations, particularly those with similar aims, may be able to benefit from your evaluation findings in planning their own projects. Your evaluation contributes to the evidence base which others should review.

Judging the worth of an intervention

How can the 5-Step Approach help funders to make their decisions?

Assessing an Intervention
Funders can use the 5-step approach as a basis for assessing funding proposals for new interventions or deciding whether to provide continuation funding for existing interventions.

For all interventions, we suggest funders ask themselves:

Does the project have clear aims and a rationale for achieving these?

To what extent is the intervention based on strong and consistent evidence drawn from research studies?

Is there is logic model showing clear, evidence-based links between each activity and the outcomes?

Does the intervention include appropriate targets, including targets around the number of people who will engage with, participate in and complete the intervention?

Have evaluation questions been identified and is a plan in place to collect the necessary data to answer these questions?

To what extent did the evaluation show a) that the resources (inputs) and been spent on evidence-based activities, that b) the target group were obtained c) that most completed the intervention and d) that the anticipated outcomes for users were achieved?

Does the evaluation appear honest and realistic (i.e. are areas for improvement identified, as well as strengths and successes)?

For existing interventions, we suggest funders ask themselves:

To what extent did the evaluation show a) that the resources (inputs) have been spent on evidence-based activities, b) that activities are clearly described and were delivered as intended, and c) that targets and anticipated outcomes were achieved?

Does the evaluation provide learning about 'why' the intervention has worked or not worked

Does the evaluation appear honest and realistic (e.g. does it highlight areas for improvement identified, as well as strengths and successes and does it acknowledge the external factors that may have impacted on any outcomes the intervention achieved)?

Potential checklist for behaviour change projects | Yes, No, To some extent (Comments)
Are there clear aims and a rationale? Why was the project needed? |
Was there a clear rationale for selection of target group? |
Is project content (what they are going to do) described in detail? |
Is there a thorough assessment of published research evidence? |
Is this evidence clearly embedded into the design of the project? |
Are there also evidence-based, or at least logical, links between inputs (costs), activities and short ,medium and long term outcomes? |
Has an appropriate evaluation been carried out? |
Has the logic model been tested through collection of relevant data? |
Did the evaluation show that resources were spent appropriately on activities with users? |
Is there evidence that activities were carried out and to a high standard? |
How many were eligible? What was the attendance/completion rate? |
Were predicted outcomes achieved? |
Is there a compelling case that the project made a contribution towards achieving outcomes?

Advantages and disadvantages of the 5-step approach

Advantages

Inclusive - all interventions of any size should be able to conduct this type of evaluation

Giving credit for evidence-based approach and a sound model of change can offset problems with conducting 'gold standard' impact evaluations

Funders can better assess the quality of proposals for new or existing interventions and make a more informed decision about the types of interventions to fund

A transparent and consistent scoring system would support and enable a process of 'certification' (similar to accreditation of formal programmes) which could raise the quality of interventions which in turn should change behaviour in the long-term.

Encourages on-going evaluation and enables continual improvement

Disadvantages

Not everyone is familiar with logic models, how to embed the evidence or evaluations so evaluators and funders might need support

It falls short of a quantitative and objectively verifiable measures of impact on long term outcomes

In order for service providers to conduct a robust logic model evaluation, they must have sufficient time for medium term outcomes to materialise. Short funding cycles may act against this. Although this approach does allow other aspects of the process to be evidenced sooner, for example evidence-based practice, a clear logic model, sound implementation of activities and short term outcomes.

Helpful Resources

General advice on proportionate evaluation for small-scale projects
http://project-oracle.com/standards-of-evidence/

http://www.clinks.org/community/blog-posts/how-can-we-make-evidence-easier#comment-form (see embedded presentation)

Evaluation Plan Worksheets
http://www.uwex.edu/ces/pdande/planning/pdf/EvaluationPlanWorksheet.doc

http://learningstore.uwex.edu/assets/pdfs/G3658-1W.PDF

http://project-oracle.com/uploads/files/2.3_Project_Oracle_-_Evaluation_plan_example_and_template_-_June_2014.pdf

http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Monitoring%20and%20evaluation%20framework.pdf

Logic model guidance, templates and flowcharts
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html

http://www.evaluationsupportscotland.org.uk/resources/127/

http://www.clinks.org/sites/default/files/TheoryofChangeGuide.pdf

http://www.clinks.org/sites/default/files/SHIFT%20Hereforshire%20ToC.pdf

Writing an evaluation report.
http://www.uic.edu/depts/crwg/cwitguide/05_EvalGuide_STAGE3.pdf

http://www.evaluationsupportscotland.org.uk/resources/135/

An example of commissioning using key elements of the 4 step approach: Reducing Reoffending Change fund guidance
http://scotland.gov.uk/Topics/archive/law-order/offender-management/changefund/changefundguidance

References

British Heart Foundation (2011) Translating the evidence: What works for physical activity? Sharing evidence based practice. Loughborough: British Heart Foundation National Centre for Physical Activity and Health.

NICE (2007) Correlates of physical activity in children: A review of quantitative systematic reviews. London: National Institute for Clinical Excellent. Accessed on 26/09/14 at
http://www.nice.org.uk/guidance/ph17/resources/promoting-physical-activity-for-children-review-2-quantitative-correlates-2

NICE (2007) The views of children on the barriers and facilitators to participation in physical activity: A review of qualitative studies. London: National Institute for Clinical Excellent. Accessed on 26/09/14 at
http://www.nice.org.uk/guidance/ph17/resources/promoting-physical-activity-for-children-consultation-on-the-draft-guidance5

Nutley , S. Powell, A. and Davies, Huw (2013) What counts as good evidence? Provocation paper for the alliance of useful evidence. London: Alliance for Useful Evidence

Petticrew, M. and Roberts, H. (2003) Evidence, hierarchies and typologies: horses for courses. Journal of Epidemiology and Community Health. 57:527-529

Rees, R. Kavanagh, J. Harden, A. Shepherd, J. Brunton, G. Oliver, S. and Oakley, A. (2006) Health Education Research. 21: 6 806-825

Ruiz, J. (2004) A literature review of the evidence for culture, the arts and sports policy. Edinburgh: Scottish Executive

Scottish Executive (2003) The Scottish health survey: Summary of key findings. Edinburgh: Scottish Executive. Accessed on 26/09/2014 at http://www.scotland.gov.uk/Resource/Doc/924/0019811.pdf

Shaw, A. McMunn, A. and Field, J. (2000) The Scottish health survey 1998. NHS Scotland. Accessed on 26/09/14 at http://www.sehd.scot.nhs.uk/scottishhealthsurvey/sh8-00.html

Appendix 1: Example data collection framework

Example data collection framework

Part of logic model analysed
Indicators
Data collected from (data source)
Data recorded in…
Data entered into database as…

INPUTS

Were there sufficient resources to run the intervention and how were they deployed?

The evaluation should show what resources were required to run the intervention and whether they were sufficient to deliver the intervention as intended.

The total cost of the intervention.

Average £ spent on each user.

What were funds spent on? How many staff were required, staff, staff case loads, costs of running sessions, cost of materials, venues etc.

Gather views on whether resources were sufficient.

To what extent was the evidence base embedded into the intervention?

Manager and staff

Annual accounts

Intervention level database

Costs, values and views

Costs can be reviewed periodically (e.g. annually)

Part of logic model analysed

Indicators

Data collected from (source)

Data recorded in…

Data entered into database as…

PARTICIPANTS

Collect information on your users to check that you reach your intended target group.

Set up the database so you can collect data on each user.

Data can then be aggregated to provide important quantitative data on users e.g. percentages, averages etc.

You can also see whether the intervention worked for some users but not others by breaking down outcome data into different types of users (e.g. different ages, offence types). Numbers have to be large for this to be meaningful though.

User ID number

N/A

User level database

Entered as 01, 02, 03 etc

Name

User survey

User level database

Name

Date of birth

User survey

User level database

Date of birth

Age at start of programme

User survey

User level database

Age

Gender

User survey

User level database

Column - Gender

Male= 1, Female= 2

Current levels of physical activity

User survey

User level database

Coded e.g.

1 = none, 2 = minimal… 6 = regular cardio exercise

Previous experience of physical activity

User survey

User level database

Coded e.g.

1= none, 2 = school-based, 3 = sports club…

Attitude towards exercise

User survey

User level database

Coded on scale of 1-5 (very positive to very negative)

Part of logic model analysed

Indicators

Data collected from (data source)

Data recorded in…

Data entered into database as…

ACTIVITIES

What did users experience?

Information on activities is important because if activities didn't happen or were poorly delivered, then it is unlikely that outcomes will occur, if they did, something external to the intervention might be responsible.

Work out the number and % of users who complete and did not complete the project as a whole and which activities they took part in.

Number of potential participants informed by school visits

School records

Intervention level database

Number of participants recruited

Weekly register

User level database

This can be calculated via the total number of participants entered into your database

Number of participants at each session choice of activity

Staff observation and weekly register

User level database and intervention level database

For each user record which sessions they attended and activities undertaken. Can code as:

Week 1: 1= attend 2=did not attend

Week 2: 1= attend, 2= did not attend etc.

Also record total numbers for each session and activity in a separate record of activities. This will enable you to identify patterns in individual behaviour as well as analyse overall attendance and participation each week.

Number of sessions run by outside clubs and classes

Staff observation and weekly register

In your record of sessions, record the sessions which were run by, or had visits from, outside organisers. This will enable you to analyse the impact these visits had on take up of outside clubs and classes.

How did users experience the project?

Gather user accounts of what they actually did.

The extent to which users valued the content of the project and their views on the way the it was delivered are important.

User perspectives on what happened in the sessions, the length of sessions, the format, quality of relationships with organisers and peers, what they learned and skills they developed.

What did they enjoy most and least?

Did they come every week? If not, why not?

Did they engage with social media? Why/why not?

User survey

User level database

Assign codes to closed responses

For example, user views on relationships with organisers:

Very poor = 1, poor = 2…….. Very good = 5

and enter into database.

E.g. 2 - enjoyment:

Very enjoyable = 5, enjoyable = 4…very unenjoyable = 1

Analyse qualitative open questions by theme (not entered into data base)

Part of logic model analysed

Indicators

Data collected from

Data recorded in…

Data entered into database as…

SHORT and MEDIUM-TERM OUTCOMES

Did change happen?

Quantitative measures of change

Obtain a pre-intervention base line and post-project assessment end-line. (see info. collected on participants).

Short term outcomes tend to be changes to attitudes, knowledge, learning, motivation or skills.

Medium term outcomes show evidence of individual behaviour change.

Measure the same outcomes at the start and exit point to see if change occurred.

As well as scales, ask the users, supervisors and family as to whether they think users have changed and in what way.

If there is no control group but you want to explore contribution you could elicit views on the relative impact of the intervention by asking users and family about perceived impact other interventions or support has had.

Could also observe sessions at the exit to see if progress has been achieved.

The difference between the baseline situation and the end situation is the measure of whether change happened.

Record the results of tests and surveys with users at the start and end of the intervention:

Nb. Depending on your aims and approach, it might be appropriate to use psychometrics tests, physical tests (e.g. fitness or BMI), as well as surveys to measure attitudes, behaviour, competencies etc.

Did users friends and family think progress was made?

Professional judgement of progress (i.e. from teachers, doctors)

User and family views on the contribution of external factors to outcomes, relative to intervention.

User survey

Family and friends survey

Survey of relevant professionals

User and family survey

User level database

Create two columns-one for the outcome variable before and one column for after the intervention . For example:

Attitude towards physical activity (before):

Very positive = 5, positive = 4…..Very negative = 1

and attitude towards physical activity (after):

Very positive = 5, positive = 4…..Very negative = 1

Could code answers

e.g. Teacher's views of participant's progress:

Very good = 5,
good =4..
none/minimal = 1

And/or transcribe interviews for more depth information

LONG TERM OUTCOMES

(Difficult to assess without RCT)

Did the intervention increase physical activity into adulthood?

Did the intervention improve long-term health and fitness?

Did users think they had made sustained progress?

Do stakeholders (eg. teachers and parents) perceive long-term changes in the group who engaged with the activity?

Longitudinal surveys of user and stakeholder views

Separate analysis conducted

Appendix 2: Evaluation report structure

Structure and content of the report

Section 1: Executive Summary

Provide a brief overview of the project itself and it's overall aims.

Summarise your main findings and recommendations from the evaluation

Section 2: Intervention description

Explain why the project was required/funded. For example, was there a gap in provision?

Describe the project, including costs, target group and aims.

Describe how the project was intended to work, using your logic model (a diagram may be helpful). You should explain how your plans were informed by evidence of 'what works' elsewhere, show in detail how funds were therefore spent on the content of the project and set out the short, medium and long term outcomes that you expected to materialise.

Section 3: Evaluation questions and methods

First set out what questions you were aiming to answer when you collected your evaluation data. E.g.

Inputs - How much did the intervention cost and how funds were spent?

Activities - Were activities carried out as planned? Was the target group reached? How many of the eligible group completed and what did activities consist of?

Short and Medium term (intermediate) outcomes - How many/what percentage of users changed attitudes or behaviour?

Describe what data was collected (quantitative and/or qualitative) in order to answer each evaluation question and describe HOW the data was collected, for example by questionnaire, observation or through the use of standardised tests.

Describe how the data was analysed (i.e. using EXCEL for numerical data or by identifying key themes in qualitatitive data)

Section 4: Findings /Results

Results should be set out to answer each of your research questions and must AT LEAST include the following results as a MINIMUM

The cost/resources used and whether it was sufficient to run the activities?

Which aspects of the project were evidence-based and which were not?

How were users selected and was this effective at reaching the target group?

Characteristics of the eligible group and eventual users (not just completers)

Throughput - how many of the eligible group started, dropped out and completed and what were their characteristics?

Were activities carried out as planned, what was their specific content and how many participated in them?

How many made progress on different measures? Who did /did not and what were their characteristics?

What were users views and experiences of the project and did they perceive it as contributing to change?

Section 5: Interpretation and recommendations

Use your results to comment on the successes, challenges and lessons learned.

Reflect on the relative contribution of your project in relation to other potential influences.

Reflect on which parts of your logic model did and didn't work as predicted and consider why.

List suggestions for modifying or supplementing the project in the future to better meet its aims (don't be afraid to comment on areas for improvement - this lends credibility to your evaluation)

Conclusions MUST to be backed up by your results

TIP! Short chapter summaries are extremely helpful for readers who don't have time to read the full report or who want to get a sense of the evaluation before reading it in detail.

This summary was drawn from excellent guidance on what to include in an evaluation report which can be found here: http://www.uic.edu/depts/crwg/cwitguide/05_EvalGuide_STAGE3.pdf