Using telephone interviews for household surveys: A conversation with Prof. Jim Lepkowski

This article is also available in Russian.

Household surveys play an important role in meeting national data needs.¹ But with the ongoing COVID-19 pandemic, household survey programmes in many countries have been impacted in different ways. While some countries have started to use telephone and web surveys for their national surveys, many still rely on face-to-face interviews. It is these operations that have been affected the most by the pandemic. Is now the right moment for countries that have been relying on face-to-face interviews to make the switch to telephone interviews, given that the mobile phone penetration is already quite high in most of the countries? Professor Jim Lepkowski of the University of Michigan, a leading expert on survey methodology, shares his thoughts on designing and conducting telephone surveys in a conversation with the UN Statistics Division (UNSD).

“We should distinguish telephone interviewing from telephone surveys.”

UNSD: Could I start with a big question, that is, what is the future for telephone interviewing? It has been used often in developed countries. How do you see it being adopted by developing countries? I have seen that establishing a sampling frame of telephone numbers can be challenging, among others.

Jim: First, we need to keep in mind that telephone surveys replaced many face-to-face surveys because, at the time, they were much cheaper to do. And cheaper telephone surveys made it possible for more surveys to be done. Private companies and university researchers who never had enough money for face-to-face data collection could do their own surveys for much lower costs per completed interview.

We should also distinguish telephone interviewing from telephone surveys. There are different circumstances where telephone interviews are used in surveys. In longitudinal surveys, in which the same households or persons are interviewed multiple times, households or persons may be initially interviewed face-to-face, but subsequent interviews can be done by telephone. This kind of mode switching is again done to save money. Telephone numbers recorded during face-to-face interviews are used for later calling, eliminating worry about a telephone frame and sampling from it.

Most telephone interviewing early on in the 1970s was being done in surveys that are only carried out through telephone. They required sampling telephone numbers to start. The frame issue became of major importance. Where is a survey organization going to get a list of sample telephone numbers to call? What kind of materials are available? And really importantly, what are the properties of the available frames? Are there households or persons out there in the population that can’t be reached by telephone? How large a share of the population is that uncovered group? Does the frame have numbers in it that while legitimate telephone numbers are not connected in any way to a household or person? Can a single telephone number connect you to multiple persons? And can an individual in the population be reached by more than one telephone number on the frame?

“Using dual landline and mobile phone frames for telephone surveys is a fun counting game but can be challenging for survey organisations”

UNSD: Great. Can we talk more about the sampling issues? In the book Survey Methodology² that you wrote together with your colleagues, there are details on telephone frames in many countries, and even that there is more than one frame in most cases. You also talk about sampling when there are two or more frames, each of which has something to offer that the others do not. For example, in some countries dual frames exist, like separate lists of landline and mobile phone numbers. And many households and persons can be reached from both at the same time. Could you say more about dual frame issues?

Jim: Dual or multiple frame sampling had been around a while before it was used in telephone surveys. At the time when the book was last edited (2009), landline telephone surveys were widely used, but mobile phones had become more and more common. A considerable proportion of people with landlines started to have mobile phones. We also began to see people who couldn’t be reached by a landline but could be reached by a mobile phone number. Worry arose that telephone surveys would cover less and less of the population if the sampling didn’t adapt to include people who couldn’t be reached by landline.

It turned out that in many cases landline numbers and mobile numbers were in different frames. The greater challenge though was that you are calling mobile phones and landlines some persons could be reached from either device. It's not that they necessarily would be selected from both frames at the same time. It was the potential, the probability that those on both frames at the same time could end up in the sample. And those types of persons were being over-represented in samples drawn from both frames.

One could try to disentangle the two frames, figuring out which persons were in both and eliminating one of their entries from one or the other frame. Then the frames would become like strata – non-overlapping. The “dual frame” sample then would just be a stratified sample.

It was impossible to match the two frames and disentangle the multiple probabilities of selection. However, one could rely on self-reports from interviewed sample persons whether they could be reached by both a landline and mobile phone number. Multiplicity weights could then be calculated to adjust estimates for over-representation. It turned out to be kind of a fun counting game, for those who are mathematically inclined. But the use of two frames, and the weighting, was more complicated, and more challenging for survey organizations to carry out.

High penetration of mobile phones means many are moving to mobile only samples

UNSD: It has been more than 10 years since the book was published. Has anything changed since then regarding telephone interviews?

Jim: Yes, one thing has changed a lot: the penetration of mobile phones as people made the transition from no phone or landline phones to mobile phones. That had three consequences for telephone surveys in many countries. First, there is in many countries virtually 100 percent coverage of the entire population by telephone, between landline and mobile phone access. Address-based samples don’t have the coverage advantage they once had. Second, people in some countries no longer use landline phones at all. For example, in the United States about 60 percent of people have mobile phones only. In Finland it is 100 percent. Mobile phones have to be a part of telephone surveys. And lastly, in some countries the percentage of the population that use landline phones only (that is, they do not have a mobile phone) has shrunk, to less than 10 percent in the United States, for example. That means some agencies and organizations are moving to mobile-only samples, anticipating that noncoverage bias would be small.

These consequences have led to an interesting reversal. It used to be in the 1970’s in the United States that landline frames did not cover 5-6 percent of the population. For the sake of lower costs, many survey organizations just called landline households and didn’t worry about the 5-6 percent noncoverage problem. Now more than 90 percent of a population can be reached by mobile phone. When sampling for a survey, many organizations are not bothering with sampling landlines. They accept a less-than-10 percent noncoverage rate to reduce the complexity of operating with two frames. Dual frame telephone sampling is disappearing--it’s still around, but it is less important today than it used to be.

Through discussion with colleagues in Egypt, particularly with my former student Mahmoud Elkasabi (who is now working with Macro International), it was clear that there are similarities between Egypt and the United States mobile phone penetration is better than landlines in almost all countries.

“Better understanding of the structure and coverage of the mobile phone frame is necessary”

UNSD: How is the experience of the United States informing other countries?

Jim: It’s important to remember that in telephone sampling the sampling method is going to be tailored, or fitted, to the frame available. The sampling techniques we have were practical tools invented in some ways to deal with frame properties that other techniques couldn’t deal with. Cluster sampling, for instance, was developed in part as a response to situations where there were no good population lists to use as a list frame. One needs to understand the frame, and then think about what sampling techniques can be applied to it. That’s what happened in United States telephone sampling

Still, there are common things to keep in mind about telephone frames across countries.

All countries have a telephone numbering system that determines what telephone numbers are being assigned to households and persons. The numbering system is publicly known, allowing the creation of a virtual frame of all possible telephone numbers in a country.

Almost universally the virtual telephone number frames are very “dirty”. A lot of numbers, sometimes even a vast majority of the numbers on the virtual frame, are not assigned to a household or person.

In addition, there are very few indicators in the telephone system that tell whether a number is assigned to a household or person or not. Sometimes the numbers are assigned to businesses, but that’s a small share of all possible telephone numbers. And it’s not too difficult to identify those numbers, and screen them out before sample is selected, or after a sample is selected. Maybe later we can talk a little about trying to get better indicators of telephone number status through telephone companies in a country.

The biggest problem is that often time a majority of those potential numbers are not assigned at all, to households or persons, nor to businesses. When called we only hear a ring-without-answer. We cannot distinguish easily and accurately a number not assigned from a number that is assigned but not being answered. This is a big common headache.

The result of these shared features of phone systems across countries is that random digit dialing, or RDD, is just not going to be efficient. RDD in its simplest form, of generating one of the potential phone number in the virtual frame, at random, means that there is a huge proportion of randomly generated telephone numbers whose status cannot be completely determined. A lot of time could be spent dialing numbers that are simply blanks, not assigned to households or persons.

This part of the system needs to be understood before frames can be explored.

What is important is to understand (a) the coverage of mobile phones before the survey is being carried out; and (b) what does the frame look like. These vary country by country and country-specific research needs to be done.

“Without a list of phone numbers from the phone operators, random digit dialing (RDD) is still needed”

UNSD: In terms of frames, can we talk about a little more about RDD? It is not an easy task – taking the United States as an example. Random phone numbers are generated based a fixed phone number structure in the country and then many steps are needed to identify the assigned numbers, which could be a very small portion of all generated numbers. Do we still need RDD if we have a list of mobile phone numbers from the mobile operators or companies?

Jim: Let’s start with considering why random digit dialing even comes up as a sampling technique in telephone surveys.

If we had a list of phone numbers that are assigned to households or persons, RDD wouldn’t be needed. We could sample randomly from the list, and even stratify the list by things like geography.

But few countries have such lists. Most have multiple telephone carriers providing telephone service. There typically is a national organization that determines which carrier gets to use which telephone numbers in which part of the county. There is no centralized repository for telephone numbers that are actually connected to subscribers though. And it is very difficult in such cases even for government agencies to get the cooperation of private company carriers to provide phone numbers for landline or mobile phones in service. A few countries have statistical agencies that do sampling of subscriber telephone numbers, and they use stratified random sampling. But it’s not unusual to learn that researchers in private companies or universities cannot get access to those numbers.

For those researchers, and for everyone in the other countries without lists, RDD is considered the sampling method to use. It is based on the virtual frame of all possible phone numbers. But as we talked about before, there is typically a big problem dealing with phone numbers in an RDD sample that are not assigned to subscribers.

Response rates for telephone interviews lower than for face-to-face interviews

UNSD: Thank you Jim. The discussion on the telephone frame is quite helpful. Now let’s get back to other aspects related to telephone interview. Is the response rate to telephone interviews lower than face-to-face interviews?

Jim: Telephone interviewing does generally have lower response rates compared with face-to-face interviewing. I don’t know this for sure in developing countries, but I do know that it is true in many. For example, I know that telephone survey response rates are lower in many countries in the Middle East. In Egypt frame problems and response rate issues were very similar to what we experienced in the United States at the time. In Qatar in the Persian Gulf, working with John Lee Holmes and Le Trung Kien, response rate and sampling frame issues, although not necessarily at the same frequency, were not a lot different from the United States.

Collecting data through telephone surveys – is your respondent the best informant for household characteristics?

UNSD: What type of information fits better with telephone surveys?

Jim: It depends on whether you would need information from one person in the household, or multiple persons. Would you allow for proxy responses? It is a lot more complicated when the survey requires responses from multiple members in the household.

UNSD: How do you get information on household structure from an individual if we start our interview from an individual rather than a household? This is the case for telephone surveys (especially based on mobile phones). Household structure is important, for example, to understand the living arrangement situation within households under COVID-19.

Jim: Households have been useful in surveys because it is a convenient sampling point to get to people. In many social sciences and other fields, though, a household itself is usually not what you are interested in. If you are interested in household characteristics such as living arrangement, you can still ask the information from the individual respondent. But there is a question about the reliability of the respondent report. You are talking to someone through a phone – is he or she the best person to report such information? Would there be a better informant in that household?

We have been fortunate with face-to-face interviews. For about 70 to 80 per cent of households in the US surveys, women happen to be the first household respondent. Women are in general more reliable informants about many household composition and other characteristics than men. They may not be the best respondent about finance or some economic issues. The informant may need to be different for different kinds of data.

With mobile telephone interviews in the United States, you get more males answering the phone. If you ask them about household composition, we may get less reliable reports than if we had talked with a female in the household. There could be increased response error for household characteristics in mobile telephone surveys.

Ultimately you would have to see whether you would be willing to accept the fact that the individual answering the phone might not be the best informant for household characteristics. Accuracy of information can be improved by more probing, which we are doing anyway. At the end of a household questionnaire section, additional probing can be added to increase the chances that all household members are re being included.

Quality of data from telephone interviewing can be on a par with face-to-face

UNSD: What about the quality of data collected from a telephone interview, compared with data collected through face-to-face interviewing?

Jim: We have a good deal of evidence that the reliability and validity of data obtained by telephone is similar to that from face-to-face interviews.

“Telephone interviewing can be flexible, using dispersed already trained and experienced staff to call from home”

UNSD: What are other areas that we should be aware of when planning a telephone interview?

Jim: Telephone data collection may be centralized in call centers, changing the labor market for interviewers. In a country with multiple languages, and face-to-face interviewing, agencies or companies can hire interviewers locally where respondents live, and some will speak languages or dialects in the community. Centralized facilities may not be able to handle multiple languages as readily. At the same time, survey organizations transitioning from face-to-face to telephone interviewing may want to retain experienced field staff, and greater language coverage. Telephone interviewing can be flexible, using dispersed already trained and experienced staff to call from home, more locally to the sample respondents. My organization at the University of Michigan has a call center, small, about 50 stations. Most telephone interviewing is done by staff hired locally to sample persons, calling from interviewer homes.

Jim Lepkowski is a Research Professor Emeritus in Survey Methodology at the Institute for Social Research and the Department of Biostatistics at the University of Michigan. He is also a Research Professor in the Joint Program in Survey Methodology at the University of Maryland. His survey expertise includes the design and analysis of survey samples, especially area probability and telephone samples of households. Dr. Lepkowski conducted investigations into the design of telephone samples, imputation to compensate for item missing data, weighting to compensate for unit nonresponse, the interaction between interviewer and respondent in the survey interview, and responsive survey design. He was director of the University’s Institute for Social Research graduate degree Program in Survey Methodology, the Summer Institute in Survey Research Techniques, and the Sampling Program for Survey Statisticians.