What is a probability sample and why is it so important?
The PULS uses probability sampling. This means that all adults in Victoria living at residential addresses have a chance to be included in our sample – and that we know what that chance is. This sets it apart from non-probability approaches where some in the population have no chance of selection, which means you have only partial information about the relationship between your sample and the population. Probability sampling is typically more difficult and expensive but is important in ensuring the data allow us to generalise our findings across the Victorian adult population.
Sometimes non-probability approaches (like opt-in online panels or those using quota, convenience and purposive sampling) claim to be ‘representative’. Looking like the population of interest (e.g. on the basis of similar demographics) and is not the same as being representative. While people using non-probability approaches often generalise their findings to their population of interest, it is rarely appropriate to do so.
To be truly ‘representative’ and able to generalise findings to Victoria as a whole, you need to use probability sampling.
Why 6,000 respondents?
We chose 6,000 respondents to maintain a balance between the ability to look at individual problems or social and demographic groups in detail and value-for-money. Particularly when we look at everyday legal problems, numbers of respondents need to be high enough to be able to talk about experience with confidence. This is particularly the case for rarer problem types, and 6,000 respondents gives us scope to explore the data in a wide variety of ways.
Why do we want to do a face-to-face survey?
Surveys can be conducted in a range of ways and they all have their pros and cons. Face-to-face, telephone, online, postal surveys, and combinations of these, vary in cost, speed, response rate, burden on respondents, the questions you can ask, and how complex the questionnaire can be.
Face-to-face surveys are generally considered the ‘gold standard’ for a survey like the PULS, yielding higher quality data than other modes of delivery. They typically have higher response rates and allow longer and more complex questionnaires, while the presence of an interviewer can help to engage respondents. Face-to-face surveys also have excellent coverage, with only a small percentage without a chance to participate. Interviewing in person can introduce some bias (e.g. more socially desirable responses when interviewed in person), though interviewers can also clarify and further explain for the respondent and encourage more information.
Face-to-face surveys are however the most expensive, and in large geographic areas costs can be prohibitive. Victoria’s manageable distances mean we can take advantage of the benefits of face-to-face administration without prohibitive fieldwork costs.
Is a face-to-face survey risky if there is another COVID-19 outbreak?
Another COVID-19 outbreak is possible, but we don’t see it as a barrier to a successful project. We have a number of contingency plans in place should public health restrictions mean that face-to-face interviewing can’t proceed.
In the case of a short lockdown (such as a five-day lockdown), we would temporarily suspend fieldwork and restart face-to-face interviewing once it is safe to do so. In the case of longer public health restrictions, we would move to conduct the PULS interviews by telephone.
Regardless of our approach, we are confident of our methods. The fieldwork is to be conducted by Roy Morgan Research, with lengthy experience and comprehensive COVID-safe protocols to protect respondents as well as their own staff.