Understanding your T2 data: Question & Answers

What is the purpose of our T2 survey?

One of the central elements of the Management Standards process is the concept of continuous improvement. Your T1 survey suggested areas of focus for your first round of improvements to working conditions, and likewise, if you go on to conduct a futher survey at T2, your T2 survey will suggest areas of focus for your next round as part of the ongoing risk assessment and action cycle. The primary purpose of the T2 survey is not to assess the ‘success’ of your interventions since the T1 survey.

Why can’t we use T2 survey results to assess the success or progress of our interventions?

A number of factors will influence the survey results as well as your interventions. For example there may have been organisational change, new pieces of work or changes to management arrangements. Even factors seemingly unrelated to work such as the heightened threat of terrorism or an economic slump may affect people’s perceptions of their lives in general and their working environment. These factors cannot be disentangled from the effect of your interventions.

So should I not compare T1 and T2 results at all?

There is a purpose in comparing results as this will tell you where working conditions have improved and where they have not, even if it does not tell you how and why. As a result, you may decide to focus your next round of improvements not on those areas which score worst but those which have improved the least, for example. Another reason for making the comparison is that where there have been positive changes you may be able to use this to make a case to senior management to continue running the Management Standards approach.

What the comparison will not tell you is whether these changes are due entirely to the work you have done whilst following the Management Standards process. When drawing conclusions about your comparison of T1 and T2 data, it will be critical to think about the context. Talking to your staff may help you understand why despite your best efforts, some working conditions appear to have worsened.

Above all, however, you should remember that every time you collect data it should be with a view to targeting resource over the next period of time, not simply to assess progress to date.

Our T2 results show that working conditions have not improved at all since T1. Does this mean the Management Standards approach does not work in our organisation?

No, this conclusion cannot be drawn. If working conditions have not improved you should look at the whole picture and try and understand the context. This will usually involve talking to your staff. It may be that external factors have contributed or it may be that particular interventions have not been well chosen.

One example of an external factor is organisational change. In our experience working closely with dutyholders, some organisations have reported that while they feel they have improved working conditions they believe that the effect will be masked by the increased stress caused by wholesale changes to staffing and mergers with other organisations.

There is evidence that the Management Standards approach can work and it is likely that within the context of ongoing risk assessment and intervention, any worsening of scores will be a temporary blip.

How quickly should I expect to see changes take effect?

It should be borne in mind that the impact on survey results will not be instantaneous. It takes a while for changes to filter down and make a real impact in the workplace and it may be that working conditions do not shift much at all between T1 and T2 but steadily improve beyond that time.

Further, some staff members may still be thinking about problems they encountered in the less recent past, in which case their results would not change from one time to the next. It is recommended that organisations remind their workers that their answers should relate specifically to their present situation in order that changes (in either direction) are detectable.

Our sample of respondents was different at T1 and T2. Will this affect our results?

It may do, but it depends on the nature of the differences. Differences between two samples can largely be categorised as either systematic or non-systematic.

Systematic differences materially alter the demographics of the sample in a way that means you might expect the average scores to differ between the two samples even if nothing in the workplace had changed. Examples:           

1. At T1, everyone was asked to complete the survey but at T2 only specific groups were asked (e.g. senior managers, or people in one particular building)
2. There has been a change in the make-up of the workforce as a whole, such as a merger with another organisation whose workforce were generally lower paid or in different occupational groups.

Non-systematic differences do not materially alter the sample. The simplest way to define a non-systematic difference is one where there is no reason to suspect there would have been any difference in results if the two samples had been asked the questions at the same time. Examples:

1. At T1, everyone was asked to complete the survey but at T2 we sent the survey only to those workers whose surnames began with the letters A-F, so as to achieve a smaller but random sample.
2. We have a high turnover of staff and have lost 600 members of staff since T1. Some of these have been replaced and some not, but there is no obvious difference between the type of people who left and the type of people brought in.

If you believe the differences are non-systematic, then there is no reason to believe the difference will affect your results. The sample sizes do not need to be similar although the smaller sample size still needs to be reasonable.

If you believe the differences are systematic, then the results will be affected but this does not necessarily mean you cannot still draw comparisons – see below.

How do we interpret T2 data if there have been systematic changes to the sample since T1?

The best solution is most usefully illustrated by an example. Suppose that at T1 you surveyed only people in Building A but at T2 you plan to survey Buildings A and B. Then you should ask respondents to say which Building they are in as part of the T2 survey and then conduct two analyses: T1 vs T2 for everyone; and T1 vs T2 for Building A. If the two analyses yield markedly different results then you know that your overall sample at T2 is not comparable to your overall sample at T1 and you should not draw any conclusions from the comparison. Conclusions can be drawn from the Building A (T1 vs T2) analysis but remembering that the sample size will be smaller.

What if we didn’t plan for the samples to be different and couldn’t use this type of solution?

There may still be other ways to compare some of your data. Essentially you are looking for subsets of your data which are comparable (e.g. a particular group of workers who were well represented both times). You may not be able to generalise what you find within these subsets to the whole workforce but it will give you an idea of how things have changed.

If no such subsets exist, then you should be prepared to accept that your T1 and T2 data are not comparable and analyse T2 alone. Where this is the case and you believe there are significant differences between the two samples, you are not comparing like with like and should regard T2 as the start of a new trend and make provision for T3 to be more comparable with T2.

In any event, it is important to remember that the analysis tool is not the only useful measure of change in the workplace and you should look to use as many pieces of information as possible. Examples include sickness absence data, company turnover and other staff feedback mechanisms.

We have only intervened on a particular working condition and yet others have changed. Why is this happening?

It is important to understand that although the six areas of working conditions (demand; control; support; role; relationships and change) are discrete, there are also clear interactions between them. For example, greater demands could lead to a feeling of lower control and a need for greater support. Changing one area of working conditions is likely to have some effect on others, though the nature of this effect cannot be predicted in advance.