Publishing Date: November 2003. © 2003. All rights reserved. Copyright rests with the author. No part of this article may be reproduced without written permission from the author.

Web Evolution

Mail vs. On-line Research: Deciding Between Cost and Quality

by F. Annie Pettit and Leanne Lam


The Internet is truly a viable market research tool that can save time and money. Until recently, market research has relied heavily on paper-based surveys. Consequently, we have many years of experience in how paper survey results calibrate with subsequent purchase behaviour. Now, it is imperative to determine how paper-based survey results compare to on-line survey results. Although on-line surveys are formatted quite differently than mail surveys, the best test of the on-line method is a comparison between a typical paper questionnaire, and a typical on-line questionnaire. This paper will review two case studies. The first is a comparison in incidence rates, while the second is a comparison of efficacy, or test control, research.

Case 1: Incidence Rates
In Spring 2002, survey responses from over 4 million randomly selected American and 550,000 Canadian households were collected using a four to six page survey of branded purchase and use of household packaged goods, direct response questions, and demographic information.

Three months after completing this survey, some responders received a second survey consisting of some of the same questions. Responders were randomly assigned to receive either a mail-out or a web survey. Approximately 2500 on-line and 6000 mail-out surveys were successfully completed and matched back to the original data.

The initial survey was used to determine base rates, i.e., the percentage of households that endorsed a particular item. The next step was to determine the percentage of households that endorsed an item on the second survey (three months post). The match rate was calculated as the percentage of households endorsing the item on the second survey, out of all households endorsing the equivalent item on the first survey. An index was calculated as the mail out match rate divided by the on-line match rate.

Each analysis contains at least 200 responders in each group. The average US index value was approximately 100, indicating that mail and web survey results matched back to the original paper-based survey data in a similar way (Selected variables are shown).

Index>100 = Mail out has
higher Match Rate
Average Index 100
Arthritis ­ Common 122
Arthritis - Rheumatoid 104
Asthma 112
Bladder leakage - Any 104
Bladder leakage - Light 96
Buy allergy remedy 5 or more times per year 101
Cold Sores 97

On the other hand, the average Canadian index value was approximately 109, indicating that the paper-based survey matched back to the base survey better than did the on-line survey (Selected variables are shown).

Index>100 = Mail out has higher Match Rate Canada
Average Index 109
Arthritis 105
Asthma 108
Back pain 105
Body aches 108
Dairy digestive difficulty 110
Migraine 101
Osteoporosis 120

Case 2: Efficacy Research Comparison
Efficacy research is used to determine the effect of sampling and couponing programs on brand awareness and market share. On-line and mail surveys are often used to do this, and in this case, the offers were for a nutritional supplement and a personal care paper product.

Test households received an offer (sample and/or coupon) while control groups did not receive an offer. Test and control groups received the same questionnaire weeks later that included a number of standard items such as branded purchase history, purchase intent, recall of the offer, and demographics.

Table 3. Percentages for each comparison measure


Nutritional Supplement Control
Ratio Control
Recall receiving an offer 23.6 45.4 192 30.9 46.9 152
Recall receiving a sample - 36.1   - 40.9  
Recall brand correctly - 73.5   - 72.7  
Purchase P3M 6.3 10.1 160 9.2 13.5 146
Purchase intent 7.9 9.9 125 11.4 14.1 124
Personal Care Paper Product Control
Ratio Control
Recall receiving an offer 13.0 14.9 115 16.1 19.2 119
Recall brand correctly - 25.8   - 27.6  
Purchase P1M 7.1 9.9 139 11.4 9.6 84
Purchase intent 10.6 11.5 108 9.9 6.7 70
* Numbers have been modified to protect client confidentiality. Overall trends, however, remain intact.

Table 3 contains all of the comparison measures. On-line and mail results can differ in two distinct ways. First, the data could differ in terms of percentages. For instance, 45.4% of the on-line Nutritional Supplement Test group recalled the offer, whereas 46.9% of the off-line group did. It appears that the Offline group recalled the offer similar to the on-line group. Second, the data could differ in terms of relationships between variables. In this case, taking false recall percentages into consideration shows that the on-line group did in fact achieve better results (46.9-30.9=16.0 vs. 45.4-23.6=21.8). As a result, payout calculations would be very different depending on whether on-line or off-line results were obtained.

The trends in the results for the on-line and off-line nutritional supplement research were similar; test percentages were larger than control percentages. This leads to the conclusion that either on-line or off-line research would have generated workable results.

For the personal care product, however, the conclusions that would have been drawn based on only on-line or only off-line are very different. For instance, the past one month purchase ratio for the on-line research suggests that the offer was a success, whereas the off-line research suggests that the offer was not a success.

Table 4. Usage and Attitudes Comparison Measures



Study 1. Nutritional Supplement Control Test Control Test
Female responders 79.3 % 76.6 % 79.6 % 80.9 %
Average age 55.8 56.8 63.7 66.6
Would not try without offer - 4.0 - 4.4
Learned something new - 3.7 - 4.0
Would recommend to someone - 3.5 - 3.8
Coupon value was good - 3.9 - 4.4
Study 2. Personal Care Product Control Test Control Test
Female responders 85.9 % 86.9 % 95.3 % 95.0 %
Average age 36.6 38.8 46.5 46.3
Would recommend - 3.1 - 3.9
Soothing on skin - 3.3 - 4.4
Works better than usual brand - 2.8 - 3.5

Table 4 shows results that relate more to usage and attitude types of data. We can see that for age, gender, and attitude mean scores, the on-line numbers were consistently lower. Although the values are not equal and may result in different significance values, the conclusions that would be drawn are the same. In effect, the on-line and off-line results provide similar learnings.

On-line research is quickly evolving. However, we need to be conscientious about how information is interpreted, as on-line efficacy results do not necessarily match mail efficacy results. It is very clear that base rates can easily be different, although they may be directionally similar. Until more on-line historical data is available, we must be mindful not to draw conclusions that on-line research is necessarily comparable to off-line research.


Annie Pettit is a Research Scientist at ICOM Information & Communications Inc.. She can be reached at 416-297-7887 or

Leanne Lam is a Project Coordinator and can be reached at

© 2003. All rights reserved. Copyright rests with the author. No part of this article may be reproduced without written permission from the author.