| Publishing Date: November 2003. © 2003. All rights reserved. Copyright rests with the author. No part of this article may be reproduced without written permission from the author. |
Mail vs. On-line Research: Deciding Between Cost and Quality
by F. Annie Pettit and Leanne Lam
The Internet is truly a viable market research tool that can save time and money. Until recently, market research has relied heavily on paper-based surveys. Consequently, we have many years of experience in how paper survey results calibrate with subsequent purchase behaviour. Now, it is imperative to determine how paper-based survey results compare to on-line survey results. Although on-line surveys are formatted quite differently than mail surveys, the best test of the on-line method is a comparison between a typical paper questionnaire, and a typical on-line questionnaire. This paper will review two case studies. The first is a comparison in incidence rates, while the second is a comparison of efficacy, or test control, research.
Case 1: Incidence Rates
In Spring 2002, survey responses from over 4 million randomly selected American and 550,000 Canadian households were collected using a four to six page survey of branded purchase and use of household packaged goods, direct response questions, and demographic information.
Three months after completing this survey, some responders received a second survey consisting of some of the same questions. Responders were randomly assigned to receive either a mail-out or a web survey. Approximately 2500 on-line and 6000 mail-out surveys were successfully completed and matched back to the original data.
The initial survey was used to determine base rates, i.e., the percentage of households that endorsed a particular item. The next step was to determine the percentage of households that endorsed an item on the second survey (three months post). The match rate was calculated as the percentage of households endorsing the item on the second survey, out of all households endorsing the equivalent item on the first survey. An index was calculated as the mail out match rate divided by the on-line match rate.
Each analysis contains at least 200 responders in each group. The average US index value was approximately 100, indicating that mail and web survey results matched back to the original paper-based survey data in a similar way (Selected variables are shown).
|Index>100 = Mail out has |
higher Match Rate
|Average Index ||100|
|Arthritis Common ||122 |
|Arthritis - Rheumatoid ||104 |
|Asthma ||112 |
|Bladder leakage - Any ||104 |
|Bladder leakage - Light ||96 |
|Buy allergy remedy 5 or more times per year ||101 |
|Cold Sores ||97 |
On the other hand, the average Canadian index value was approximately 109, indicating that the paper-based survey matched back to the base survey better than did the on-line survey (Selected variables are shown).
|Index>100 = Mail out has higher Match Rate ||Canada|
|Average Index ||109|
|Arthritis ||105 |
|Asthma ||108 |
|Back pain ||105 |
|Body aches ||108 |
|Dairy digestive difficulty ||110 |
|Migraine ||101 |
Case 2: Efficacy Research Comparison
Efficacy research is used to determine the effect of sampling and couponing programs on brand awareness and market share. On-line and mail surveys are often used to do this, and in this case, the offers were for a nutritional supplement and a personal care paper product.
Test households received an offer (sample and/or coupon) while control groups did not receive an offer. Test and control groups received the same questionnaire weeks later that included a number of standard items such as branded purchase history, purchase intent, recall of the offer, and demographics.
|Table 3. Percentages for each comparison measure|
| || |
| || ||Mail || |
|Nutritional Supplement ||Control |
|Ratio ||Control |
|Recall receiving an offer ||23.6 ||45.4 ||192 ||30.9 ||46.9 ||152|
|Recall receiving a sample ||- ||36.1 || ||- ||40.9 || |
|Recall brand correctly ||- ||73.5 || ||- ||72.7 || |
|Purchase P3M ||6.3 ||10.1 ||160 ||9.2 ||13.5 ||146|
|Purchase intent ||7.9 ||9.9 ||125 ||11.4 ||14.1 ||124|
|Personal Care Paper Product ||Control |
|Ratio ||Control |
|Recall receiving an offer ||13.0 ||14.9 ||115 ||16.1 ||19.2 ||119|
|Recall brand correctly ||- ||25.8 || ||- ||27.6 || |
|Purchase P1M ||7.1 ||9.9 ||139 ||11.4 ||9.6 ||84|
|Purchase intent ||10.6 ||11.5 ||108 ||9.9 ||6.7 ||70|
|* Numbers have been modified to protect client confidentiality. Overall trends, however, remain intact.|
Table 3 contains all of the comparison measures. On-line and mail results can differ in two distinct ways. First, the data could differ in terms of percentages. For instance, 45.4% of the on-line Nutritional Supplement Test group recalled the offer, whereas 46.9% of the off-line group did. It appears that the Offline group recalled the offer similar to the on-line group. Second, the data could differ in terms of relationships between variables. In this case, taking false recall percentages into consideration shows that the on-line group did in fact achieve better results (46.9-30.9=16.0 vs. 45.4-23.6=21.8). As a result, payout calculations would be very different depending on whether on-line or off-line results were obtained.
The trends in the results for the on-line and off-line nutritional supplement research were similar; test percentages were larger than control percentages. This leads to the conclusion that either on-line or off-line research would have generated workable results.
For the personal care product, however, the conclusions that would have been drawn based on only on-line or only off-line are very different. For instance, the past one month purchase ratio for the on-line research suggests that the offer was a success, whereas the off-line research suggests that the offer was not a success.
|Table 4. Usage and Attitudes Comparison Measures|
| || |
|Study 1. Nutritional Supplement ||Control ||Test ||Control ||Test |
|Female responders ||79.3 % ||76.6 % ||79.6 % ||80.9 %|
|Average age ||55.8 ||56.8 ||63.7 ||66.6|
|Would not try without offer ||- ||4.0 ||- ||4.4|
|Learned something new ||- ||3.7 ||- ||4.0|
|Would recommend to someone ||- ||3.5 ||- ||3.8|
|Coupon value was good ||- ||3.9 ||- ||4.4 |
|Study 2. Personal Care Product ||Control ||Test ||Control ||Test |
|Female responders ||85.9 % ||86.9 % ||95.3 % ||95.0 %|
|Average age ||36.6 ||38.8 ||46.5 ||46.3 |
|Would recommend ||- ||3.1 ||- ||3.9|
|Soothing on skin ||- ||3.3 ||- ||4.4 |
|Works better than usual brand ||- ||2.8 ||- ||3.5 |
Table 4 shows results that relate more to usage and attitude types of data. We can see that for age, gender, and attitude mean scores, the on-line numbers were consistently lower. Although the values are not equal and may result in different significance values, the conclusions that would be drawn are the same. In effect, the on-line and off-line results provide similar learnings.
On-line research is quickly evolving. However, we need to be conscientious about how information is interpreted, as on-line efficacy results do not necessarily match mail efficacy results. It is very clear that base rates can easily be different, although they may be directionally similar. Until more on-line historical data is available, we must be mindful not to draw conclusions that on-line research is necessarily comparable to off-line research.
Annie Pettit is a Research Scientist at ICOM Information & Communications Inc.. She can be reached at 416-297-7887 or apettit@I-com.com.
Leanne Lam is a Project Coordinator and can be reached at llam@I-com.com.
© 2003. All rights reserved. Copyright rests with the author. No part of this article may be reproduced without written permission from the author.