Anne Crassweller, Judy Rogers and Don Williams, NADBank

Worldwide Readership Research Symposium Valencia 2009 Session 6.1

Setting the Stage

Online panels are becoming the standard choice for researchers today. The temptation to use them is overwhelming as they are a less expensive alternative to traditional telephone and face-to-face research and the results are available sooner. Are they the right choice for all research projects?

There is no question that considerable effort is being devoted to building better panels. Commonly, however, the “virtues” of panels are demonstrated not by their intrinsic measurement superiority but by their speed and low cost vis à vis other sampling and data collection modes and/or by deficiencies in other types of surveys (e.g., falling response rates in telephone studies because of consumer avoidance techniques and new technologies masking geography etc as well as the growth in cell-only households).

Conversely, online panel research limitations are often overlooked (convenience samples, coverage bias etc.) in favour of cost and timing attributes. As broadband Internet access increases, coverage error and potential bias for online surveys will likely decrease but not eliminated.

Market research practitioners and users have long recognized that focus groups or shopping mall intercepts are not suitable tools for answering some types of questions. The same applies to online panels. They have their place in the toolbox but purveyors and users are engaged in an ongoing discussion about which types of questions they are best able to answer. Can they replace studies based on a random sample of respondents within a defined universe in which the probability of selection is known?

There remains considerable support for the position that when quantitative estimates such as market shares are required, they should be derived from studies that rely on random probability sampling rather than online panels (Chakrapani,2007) Nonetheless, like other services, newspapers are under increasing pressure to be “with it” (a.k.a. be on the web), to reduce data collection costs (a.k.a. be on the web), and to increase reporting ease and speed (a.k.a. be on the web).

Since 1986 newspapers in Canada have relied on traditional telephone data capture techniques to estimate readership of daily papers in major markets, to build profiles of readers and to set market parameters and pricing for advertising. Millions of dollars of advertising revenue depend on readership and profile estimates of how many and who reads specific daily newspapers. How are print media and advertising mavens going to determine when or if to move to online panels?

As Canada’s newspaper audience measurement agency, the Newspaper Audience Databank Inc. (NADbank) has been monitoring developments in online panel research for some time. In 2006 the organization embarked on a journey to determine if its annual study could be moved from a modified RDD telephone methodology to a web-based survey. The results of this parallel online survey using the TNS Canadian Facts’ web-access panel in Toronto were reported to this symposium in 2007.

The results from the first study showed that the demographic profiles of respondents in the online panel differed from the population as a whole and the telephone sample. As well as demographic differences, there were variations in general media behaviour as well as the primary metric being investigated: readership of daily newspapers.

As only one online panel was included in the test, there was no way to assess the extent to which profile differences were a function of idiosyncrasies of the single panel and/or were linked to online panel data capture per se.

In the fall of 2007 a second, larger scale study was undertaken to further explore the differences between telephone and online protocols; inter-supplier consistency and the potential use of a web-based survey outside of the Toronto market.

Study Objectives and Design

The overall purposes of the research were as follows:

1. To determine the extent to which online results resemble or differ from the current modified RDD telephone survey methodology used to estimate newspaper audiences in Canada’s urban centres;
2. To examine the extent to which there is inter-supplier consistency using independent online panels; and
3. To explore the feasibility of undertaking online panel surveys in small and medium sized Canadian markets.

NADbank partnered with TNS Canadian Facts, Ipsos Canada, Synovate Canada, and Decima-Harris for this second study. NADbank collected five data sets that purport to measure the same phenomenon: readership of daily newspapers in three Canadian cities (Toronto, Québec City and Halifax).

From more than 2000 completed surveys per supplier among Toronto residents to between 400 and 630 in the smaller centres, each of the four firms implemented online panel studies that mimicked, to the extent possible, the ongoing modified RDD telephone readership study.

The four research companies were provided with identical instructions for project design, implementation, weighting and projection. To the extent possible, the survey instrument, survey scheduling procedures, weighting and projection techniques matched those used in the current NADbank telephone survey methodology. To ensure study-to-study consistency, coding of responses to the occupation measures in the four online surveys was performed by TNS Canadian Facts using the same principles and categories the firm uses to code occupation in NADbank’s traditional telephone survey. More details about the methodology are also available from NADbank (under separate cover).

The three cities were selected to represent different market conditions:

• Toronto: a large metropolitan area with multiple English language daily newspapers;
• Halifax: a small metropolitan area with a limited number of English language daily newspapers;
• Québec City: a medium sized metropolitan area with two French language daily newspapers.

Survey materials for Toronto and Halifax were in English; those for Québec City were in French. Targets for completed surveys differed substantively across the three cities (Toronto, 2000; Halifax, 400; Québec City, 500). A common data file incorporating all suppliers’ data in a consistent format was created. This file was used to generate tabulations with identical specifications from supplier to supplier, thereby ensuring that outputs among the online studies and to the telephone survey were directly comparable.

The identity of the four online study suppliers has been masked. In no particular order, the firms were labelled with the first four letters of the alphabet (Panel A, B, C, D). The NADbank telephone readership study is the benchmark against which each online supplier’s data are assessed.

Key Findings

Single Panel Consistency – 2006 vs. 2007 in Toronto

Stability of year-to-year results is an important consideration for currency measurement and a key issue when evaluating the value of a new protocol. Before looking at the multi-panel data the consistency between the two studies conducted with the same supplier panel; TNS Canadian Facts, was reviewed.

NADbank data are weighted to match age-within-gender estimates of the population 18 years of age and over (by market) based on projected census data.

Table 1 shows comparable distribution for the selected demographics that are not captured in the weighting protocol for both the telephone samples and online panels; all conducted by TNS Canadian facts. The findings presented in Table 1 are not restricted to the selected variables but indicative of other “not-weighted” demographics from the 2006 and 2007 studies in Toronto. With some notable exceptions, the patterns hold constant within the data collection modes from year to year but differ between the telephone and online methodologies.

TABLE 1

COMPARISON OF SELECTED SAMPLE AND PANEL DEMOGRAPHIC COMPOSITION 2006 AND 2007 –

weighted

TELEPHONE ONLINE PANEL 2006 2007 2006 2007
Some Post Secondary Education	31%	32%	46%	44%
University Graduate +	38%	37%	33%	33%
Manager/Professional	20%	22%	18%	18%
Blue Collar	12%	13%	9%	9%
Adult Only Households	60%	60%	66%	45%
English as a Mother Tongue	61%	61%	77%	78%

General media habits provide an overview of the behaviour of the respondents in each group and harbinger of the newspaper readership results. Table 2 shows time spent in hours per week with each medium and the reach of any daily newspaper yesterday, on Saturday and readership online (weekly). As noted with the demographics, there is good measure of consistency for the top-line measure of time spent by medium (as well as reach by medium – not shown here); a seductive outcome early in the evaluation process.

TABLE 2.

COMPARISON OF MEDIA HABITS IN TORONTO 2006 AND 2007 – weighted

TELEPHONE ONLINE PANEL 2006 2007 2006 2007
HOURS/WEEK
Daily Newspapers	4.2	4.5	5.8	5.7
Radio	13.0	12.4	13.8	13.7
Magazines	3.7	3.1	4.9	4.2
TV	13.9	13.0	21.4	20.6
Internet	13.9	13.4	22.3	22.1
REACH
Any Paper Yesterday	47%	46%	56%	48%
Any Paper Saturday	43%	40%	52%	44%
Any Paper Online Past Week	19%	21%	29%	28%

Stability begins to breakdown with the Any Newspaper readership results such as any paper yesterday (Table 2). Table 3 illustrates the further degradation of consistency for specific newspaper readership. It should be noted, though, that the general decrease in Saturday readership and increase in weekly online readership between 2006 and 2007 on the telephone (Table 2), is consistent with results in all other markets measured in the telephone in the study.

TABLE 3

READ YESTERDAY IN TORONTO – TELEPHONE VS. ONLINE PANEL 2006 AND 2007

TELEPHONE ONLINE PANEL 2006 2007 2006 2007
READ YESTERDAY
Toronto Star	23%	22%	31%	24%
The Toronto Sun	12%	10%	18%	14%
The Globe and Mail	9%	9%	11%	8%
National Post	5%	5%	7%	7%
Metro	9%	11%	14%	12%
24 hours	8%	8%	15%	12%

Multi-Panel Results

1. Demographics

As mentioned, once the data had been weighted, each online panel supplier matches “exactly” the telephone benchmark for age and gender distributions in each city. Nonetheless, there are considerable fluctuations in the actual (unweighted) distributions by age and gender from online panel to panel. While it can be argued that “weighting” remedies differences between panel respondents and the actual market, further exploration suggests that this may not be the case. For Toronto, the unweighted results for age and gender are shown for the telephone sample and each of the panels. [Table 4].

TABLE 4

COMPARISON OF GENDER AND AGE IN TORONTO

WEIGHTED ACTUAL – unweighted TELEPHONE TELEPHONE PANEL A PANEL B PANEL C PANEL D
Completed Surveys		2063	2001	2214	2112	2029
GENDER
Male	49%	43%	52%	36%	45%	50%
Female	51%	57%	48%	64%	55%	50%
AGE
18-24	12%	9%	13%	6%	5%	9%
25-34	20%	17%	17%	15%	17%	20%
35-49	32%	37%	28%	37%	31%	33%
50-64	22%	23%	31%	28%	26%	23%
65+	14%	13%	11%	14%	22%	15%
Avg. 18+	45	46	45	47	49	45

Once weights have been applied to age and gender variations in the demographics not captured in the weighting protocol between the benchmark telephone and various online results are evident. However, there are no obvious patterns in the size or direction of the discrepancies [Table 5]. For example:

• In Toronto, Panels A (18%) and B (20%) are close to the telephone benchmark for the incidence of persons in the managers/professionals occupation group (21%). Panel C is well below (15%) and Panel D is well above (32%) the benchmark.
• Since Panel C has the lowest estimate of managers/professionals in Toronto, it might also be expected to have a low estimate of university graduates. It does not. In fact, Panel C’s university graduate proportion (42%) is higher than Panels A (33%) and B (34%) and closest of the four online panels to the telephone benchmark (39%).
• Toronto is a major reception centre for new immigrants, explaining why only 6-in-10 adults name English as their mother tongue (telephone). At about 3-in-4, panel respondents are appreciably more likely to have first learned and still understand English than would be anticipated.
• House dwellers represent 84% of the Halifax market according to the telephone survey, but only 75% of the market according to Panel D, 70% according to Panel B, and as few as 62% according to Panel A.
• In Québec City, all panels produce an identical estimate of persons with “some post secondary education” (ranging from 46% to 49%). In each case, these estimates are substantively higher than the benchmark telephone study (36%).

TABLE 5

DEMOGRAPHIC SAMPLER – TORONTO, HALIFAX AND QUÉBEC CITY

TELEPHONE	PANEL A	PANEL B	PANEL C	PANEL D
TORONTO – WEIGHTED, PROJECTED
EDUCATION
Some Post Secondary 32%	44%	45%	40%	33%
University Graduate 39%	33%	34%	42%	56%
OCCUPATION
Managers/Professionals 21%	18%	20%	15%	32%
Other White Collar 17%	17%	20%	18%	21%
Cler/Admin/Bus. Support 6%	9%	9%	9%	5%
Blue Collar 13%	9%	9%	5%	4%
DWELLING
House 78%	67%	73%	70%	75%
Apartment 21%	31%	26%	29%	24%
HOUSEHOLD COMPOSITION
Adults Only 60%	66%	66%	67%	69%
Adults w. Child 40%	34%	34%	32%	31%
MOTHER TONGUE ENGLISH
61%	78%	77%	77%	74%
HALIFAX – WEIGHTED, PROJECTED
EDUCATION
Some Post Secondary 27%	47%	47%	46%	41%
University Graduate 36%	29%	29%	34%	50%
OCCUPATION
Managers/Professionals 23%	16%	18%	12%	30%
Other White Collar 13%	16%	22%	15%	22%
Cler/Admin/Bus. Support 9%	14%	9%	10%	7%
Blue Collar 16%	10%	10%	9%	5%
DWELLING
House 84%	62%	70%	65%	75%
Apartment 14%	34%	28%	32%	23%
HOUSEHOLD COMPOSITION
Adults Only 62%	73%	73%	74%	75%
Adults w. Child 38%	27%	27%	26%	25%
QUÉBEC CITY – WEIGHTED, PROJECTED
EDUCATION
Some Post Secondary 36%	48%	49%	48%	46%
University Graduate 25%	21%	25%	29%	37%
OCCUPATION
Managers/Professionals 12%	8%	10%	7%	13%
Other White Collar 18%	18%	18%	19%	18%
Cler/Admin/Bus. Support 7%	14%	14%	10%	11%
Blue Collar 16%	13%	14%	8%	8%
DWELLING
House 69%	53%	56%	54%	61%
Apartment 30%	45%	41%	43%	37%
HOUSEHOLD COMPOSITION
Adults Only 71%	77%	78%	75%	76%
Adults w. Child 29%	23%	22%	24%	24%

1. General Media Habits

General media habits of Toronto’s adults (for example) are also highly variable across the online panels. Ranging from a high of about 82 minutes to about 57 minutes, the average number of minutes spent reading a printed version of any daily newspaper yesterday is substantively higher for online panel members than it is for telephone sample respondents, at about 50 minutes, on average [Table 6]. Television viewing time is also consistently higher for online panel respondents than is the case for telephone sample members. Radio listening time, however, hovers around the same level; irrespective of the data capture mode or the online supplier. Not surprisingly, average time spent on the internet yesterday among panel respondents is also higher than is the case among telephone sample respondents.

TABLE 6

GENERAL MEDIA BEHAVIOUR IN TORONTO

TELEPHONE PANEL A PANEL B		PANEL C	PANEL D
AVERAGE NUMBER MINUTES SPENT YESTERDAY….
Reading Printed Newspapers 50 69	63	82	57
Reading Magazines 44 49	45	57	48
On the Internet 140 206	163	196	169
Listening to Radio 138 144	139	147	118
Watching TV 125 185	168	188	144
AVERAGE NUMBER OF PAST 5 WEEKDAY NEWSPAPER ISSUES READ
2.8 2.7	2.7	2.5	2.5

1. Newspaper Readership Read Any Paper Yesterday

As the demographics and general media behaviour vary substantively from panel to panel and between panels and the telephone benchmark, so do readership estimates (see Appendix for complete summary tables).

As one of the most salient media measures for daily newspapers, rea d yesterday estimates across the four panels and the three markets are worthy of scrutiny. Table 7 illustrates that there are no obvious patterns in direction or degree of variation.

Toronto

• For example, compared to the telephone benchmark (22%), three online panels generate higher readership levels for the Toronto Star (B, 30%; C, 31%; D, 35%). At 24%, Panel A is close to the benchmark.
• All panels generate estimates consistent with the telephone benchmark (11%) for Metro Toronto.
• At 21%, Panel D’s read yesterday estimate for The Globe and Mail far surpasses the telephone benchmark (9%) and estimates from all other online panel suppliers.

Halifax

• Did close to two-thirds of adult Halagonians read at least one daily newspaper yesterday (Panels B or D) or was it closer to one half of them (telephone benchmark; Panel A)?

Québec City

• Panel D appears to generate read yesterday estimates for Toronto and Halifax dailies that are substantively higher than the telephone benchmark and, in some cases, than estimates by other online suppliers. Why, then, do we find that Panel D’s estimate for Le Journal de Québec (23%) is lower than every other online supplier’s and than the telephone benchmark (29%)?

Table 7

Read Yesterday Readership in Toronto, Halifax and Québec City

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
TORONTO
Toronto Star	22%	24%	30%	31%	35%
The Toronto Sun	10%	14%	15%	12%	9%
The Globe and Mail	9%	8%	9%	12%	21%
National Post	5%	7%	7%	7%	12%
Metro	11%	12%	12%	11%	12%
24 hours	8%	12%	10%	9%	8%
Any	46%	48%	53%	53%	62%
HALIFAX
The ChronicleHerald	40%	43%	54%	52%	54%
The Daily News	17%	19%	25%	20%	27%
The Globe and Mail	4%	6%	5%	6%	12%
National Post	1%	1%	1%	2%	1%
Any	48%	51%	63%	58%	65%
QUÉBEC CITY
Le Journal de Québec	29%	31%	30%	26%	23%
Le Soleil	23%	23%	25%	25%	32%
Any	48%	47%	48%	45%	48%

Online Past Week Readership

It would be reasonable to assume that members of online panels experience fewer constraints in accessing newspapers online than does the public at large. This research confirms this assumption. Indeed, panel members exhibit appreciably higher levels of past week online readership than would be expected based on the telephone benchmark findings.

The “gaps” between online readership estimates from the telephone survey and the online panels are indicative of a fundamental difference between the data collection modes [Table 8]. In fact, these differences are sufficiently substantive across all online suppliers and the three test markets that they raise concerns about the ability of online panels to “match” the behaviour of all adult residents in Canada’s cities.

If all newspapers in all markets experienced the same level of disparity for online readership, one might conclude that the telephone survey is under-representing the online market for newspapers. Since the differences between the telephone estimates and each of the four online panels vary by paper/market, such a conclusion does not seem warranted:

• • estimates for past week The Globe and Mail online readership are similar for the telephone, Panels A, B, and C but very different for Panel D;
  • all online estimates are appreciably higher than the telephone estimate for online readership of both dailies in Halifax and for Le Soleil in Québec City.

TABLE 8

WEEKLY READERSHIP OF ONLINE EDITIONS

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
TORONTO
Toronto Star	13%	19%	20%	22%	29%
The Toronto Sun	4%	9%	9%	10%	9%
The Globe and Mail	9%	9%	9%	12%	22%
National Post	3%	3%	4%	4%	8%
Metro	1%	4%	3%	4%	3%
24 hours	2%	4%	3%	4%	4%
Any	21%	28%	29%	34%	44%
HALIFAX
The ChronicleHerald	14%	33%	37%	32%	37%
The Daily News	8%	15%	17%	14%	17%
The Globe and Mail	4%	6%	7%	8%	19%
National Post	1%	2%	3%	3%	5%
Any	17%	38%	42%	37%	47%
QUÉBEC CITY
Le Soleil	7%	18%	14%	16%	21%

Different Panels, Same Readers?

For the following analyses, the focus is on Toronto as the base sizes for this city are appreciably larger than those in Halifax and Quebec City.

Even if the incidence of reading various newspapers in their print or online versions differed somewhat from panel to panel and between the panels and the telephone benchmark, it would be reasonable to anticipate that the profiles of readers of specific publications would be similar from study to study. In other words, a person who claims to have read yesterday’s Toronto Star or The Globe and Mail should have a similar demographic and behavioural profile, irrespective of the source of readership information. Is this the case? No, it is not.

Based on the examples displayed in Table 9, none of the online panels produces profiles of newspaper readers that are consistent with those generated by the telephone survey. Furthermore, the online panel profiles differ substantively, one to another.

• • What proportion of Toronto Star readers are university graduates? Is it about two-fifths as estimated in the telephone study (43%), about half as estimated by Panel C (49%) or almost three-fifths as estimated by Panel D (57%)?
  • What proportion of The Toronto Sun readers are over 65 years of age? Is it seven percent as estimated in the telephone survey or twice this proportion as estimated by Panels A (15%), B (14%) or C (16%)?

Similar variations are evident for other characteristics of Toronto’s newspaper readers (e.g., tenure, managerial/professional job status, etc.).

Readers of yesterday’s The ChronicleHerald in Halifax and the Le Journal de Québec also manifest quite different demographic characteristics, depending on whether they were interviewed in the telephone study or completed the survey online by one of the four suppliers.

TABLE 9

DEMOGRAPHIC SAMPLER FOR SELECTED NEWSPAPERS

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
TORONTO
READ YESTERDAY’S ISSUE	n=440	n=505	n=676	n=681	n=728
OF TORONTO STAR
Age 65+	28%	25%	25%	23%	20%
University Graduate +	43%	37%	42%	49%	57%
Home Language English	87%	92%	93%	94%	91%
Manager/Professional	20%	20%	19%	14%	31%
Own Home	79%	73%	81%	77%	84%
READ YESTERDAY’S ISSUE	n=196	n=292	n=329	n=244	n=171
OF THE TORONTO SUN
Age 65+	7%	15%	14%	16%	10%
University Graduate +	19%	22%	27%	29%	44%
Home Language English	89%	94%	87%	93%	94%
Manager/Professional	22%	20%	20%	14%	31%
Own Home	72%	61%	69%	63%	83%
READ YESTERDAY’S ISSUE	n=185	n=159	n=183	n=265	n=454
OF THE GLOBE AND MAIL
Age 65+	17%	27%	24%	24%	21%
University Graduate +	66%	55%	64%	67%	75%
Home Language English	92%	91%	87%	91%	93%
Manager/Professional	39%	27%	33%	25%	42%
Own Home	85%	73%	76%	76%	86%
HALIFAX
READ YESTERDAY’S ISSUE	n=110	n=185	n=276	n=261	n=268
OF THE CHRONICLEHERALD
Age 65+	24%	24%	20%	24%	20%
University Graduate +	41%	32%	32%	31%	51%
Home Language English	99%	99%	95%	98%	98%
Manager/Professional	25%	22%	17%	13%	30%
Own Home	80%	69%	68%	69%	83%
QUÉBEC CITY
READ YESTERDAY’S ISSUE	n=144	n=182	n=183	n=176	n=126
OF LE JOURNAL DE QUÉBEC
Age 65+	20%	18%	27%	21%	18%
University Graduate +	17%	15%	23%	19%	28%
Home Language French	99%	99%	100%	100%	100%
Manager/Professional	9%	9%	11%	6%	11%
Own Home	74%	57%	63%	57%	69%

Demographic and Reader Profiles

Comparisons of profile information within key demographic segments further support the hypothesis that online panel results differ from the benchmark telephone survey in ways that could influence estimates of newspaper readership and other media behaviour.

Blue-Collar Workers

Take, for example, blue-collar workers in Toronto [Table 10]. Depending on which online study is selected, fewer than 1-in-10 of Toronto’s blue collar workers would be between 18 and 24 years (9%, Panel C) or 1-in-3 would be (34%, Panel D). Based on the telephone survey, we would expect about 1-in-7 of these blue-collar workers to be young adults (15%, Telephone). As noted, all coding for occupation was performed by TNS Canadian Facts to ensure consistency of editing.

Of particular note is the propensity of online panellists to be individuals with English as their mother tongue and the language spoken most often in the home. Of Toronto’s blue collar workers identified in the benchmark telephone survey, we would

expect about 6-in-10 to name English as their mother tongue (59%, Telephone). In contrast, each of the test’s online panels produces a profile in which at least 7-in-10 (73%, Panel D) and as many as 9-in-10 blue-collar workers first learned English (89%, Panel A; 80%, Panel B; 87%, Panel C).

Toronto is Canada’s largest settlement region for new immigrants. For many of these immigrants, English is not their mother tongue. Are measurement tools that appear to be so heavily skewed toward the English-speaking/reading market a viable tool for estimating media behaviour for the entire adult population in a city such as Toronto?

TABLE 10

SELECTED DEMOGRAPHICS FOR BLUE COLLAR WORKERS IN TORONTO

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
GENDER
Male	76%	70%	78%	64%	68%
Female	24%	30%	22%	36%	32%
AGE GROUP
18-24	15%	11%	16%	9%	34%
MOTHER TONGUE
English	59%	89%	80%	87%	73%

Online blue-collar workers also differ from those in the benchmark telephone survey in which daily newspapers they claim to have read yesterday, in their “content” readership and in the amount of time they spend with various media [Table 11].

Yesterday’s readership of the Toronto Star is higher among blue collar workers captured in the online panels, and especially Panel D (30%) relative to the telephone survey benchmark (15%). Similarly, The Toronto Sun, a tabloid with historically strong appeal among blue-collar Torontonians (21%, Telephone) ranges from a “read yesterday” readership level of 11% for Panel D to a high of 36% for Panel B. In summary, there is considerable and inexplicable “bounce” in the estimates of readership by blue- collar workers among the online panel studies and between the panel studies and the benchmark telephone survey.

Not surprisingly, the online cohort spends much more time on the Internet but also spends more time watching television but less time listening to the radio than do blue-collar workers in the telephone sample. Using the time spent on the Internet yesterday as the example, average time spent estimates varies dramatically between the telephone survey (93 minutes) and online estimates and among the online estimates, from a low of 123 minutes for Panel B to a high of 190 minutes for Panel C.

TABLE 11

MEDIA BEHAVIOUR OF BLUE COLLAR WORKERS IN TORONTO

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
READ YESTERDAY’S ISSUE:
Toronto Star	15%	24%	21%	19%	30%
The Toronto Sun	21%	24%	36%	24%	11%
The Globe and Mail	3%	3%	4%	6%	8%
National Post	2%	3%	3%	1%	8%
Metro	11%	14%	14%	16%	21%
24 hours	12%	21%	18%	11%	19%
USUALLY READ:
Sports pages	27%	31%	30%	33%	28%
Finance/Business pages	14%	15%	12%	21%	20%
AVG. # MINUTES SPENT
On Internet	93	182	123	190	167
Listening to radio	196	159	137	162	166
Watching TV	120	183	145	215	149

Managers & Professionals

As evident in Tables 12 and 13, managers and professionals in Toronto experience levels of variation on key demographic and media behaviours similar to those experienced by their blue-collar counterparts.

TABLE 12

SELECTED DEMOGRAPHICS FOR MANAGERS/PROFESSIONALS IN TORONTO

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
GENDER
Male	59%	56%	59%	62%	61%
Female	41%	44%	41%	38%	39%
EDUCATION
University Graduate	57%	53%	51%	62%	74%
MOTHER TONGUE
English	64%	71%	75%	77%	72%
HOUSEHOLD TENURE
Own	82%	71%	73%	71%	86%
HOUSEHOLD COMPOSITION
Adults Only	58%	65%	64%	61%	63%
Adults & Teens/Children	42%	35%	36%	39%	37%

Did one-fifth (20% Telephone) or one-third (34%, Panel D) of Toronto’s managers/professionals read yesterday’s issue of the Toronto Star? Do about one-third of them “usually” read the financial/business pages of daily newspapers (Telephone and Panels A, B, C) or do one-half of them (50%, Panel D)?

Their “other media” behaviour is also subject to considerable variation, depending on which online panel is compared to the benchmark telephone study.

TABLE 13

MEDIA BEHAVIOUR FOR MANAGERS/PROFESSIONALS IN TORONTO

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
READ YESTERDAY’S ISSUE:
Toronto Star	20%	27%	29%	30%	34%
The Toronto Sun	10%	16%	16%	11%	8%
The Globe and Mail	17%	13%	15%	20%	27%
National Post	7%	9%	10%	8%	15%
Metro	12%	12%	11%	11%	12%
24 hours	7%	12%	6%	10%	7%
USUALLY READ:
Sports Pages	30%	25%	25%	23%	31%
Finance/Business Pages	36%	30%	32%	32%	50%
AVG. # MINUTES SPENT
On Internet	151	196	161	186	170
Listening to Radio	124	128	144	136	109
Watching TV	108	140	144	159	125

Young People (18 – 24 Years)

Appreciably more online panel youths are “students” than is the case among 18 to 24 year olds contacted in NADbank’s telephone survey. Those included in the online panels, like blue-collar and professional/managerial workers, are also especially likely to have English as their mother tongue. In this regard, however, there is not only variation between telephone and online panel youth, but also among the online panels (72%, A; 82%, B; 74%, C; 64%, D) [Table 14].

TABLE 14

DEMOGRAPHICS AND MEDIA BEHAVIOUR OF 18-24 YEAR OLDS IN TORONTO

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
EMPLOYMENT STATUS
Student	39%	57%	49%	57%	46%
EDUCATION
Some Post Secondary	48%	53%	57%	55%	56%
University Graduate	15%	14%	13%	16%	20%
MOTHER TONGUE
English	57%	72%	82%	74%	64%
READ YESTERDAY’S ISSUE:
Toronto Star	18%	19%	17%	27%	24%
The Toronto Sun	7%	8%	13%	14%	10%
The Globe and Mail	4%	8%	3%	10%	11%
National Post	2%	4%	3%	6%	6%
Metro	13%	15%	16%	17%	19%
24 hours	8%	11%	10%	11%	14%
USUALLY READ:
Sports Pages	29%	15%	19%	22%	16%
Finance/Business Pages	8%	11%	8%	11%	16%
AVG. # MINUTES SPENT
On Internet	187	275	218	296	236
Listening to Radio	129	120	136	120	135
Watching TV	143	160	171	195	148

Conclusions

• There was as much consistency between 2006 and 2007 TNS Canadian Facts panels as there was between the two telephone samples for the top-line demographic and media habits data. Based on specific newspaper readership results, however, this consistency would appear to be a misleading metric for evaluating the validity of the online protocol.
• There is considerable “bounce” among online suppliers’ study findings for key demographic and behavioural subgroups:
  • Critical currency data differ from one online panel supplier to another;
  • The characteristics of a newspaper’s audience differ substantively depending on which online panel is used as the source;
  • Demographic subgroups within the population (e.g., blue-collar workers; young people) differ substantively from one online panel to another in terms of their other demographic characteristics and in terms of their readership/other media behaviour.
• Newspaper readership estimates (read yesterday) vary more extensively between the telephone and online studies in smaller centres (Halifax, Québec City) where sample sizes are smaller and recruitment of sufficient panel members is more challenging than in larger centres such as Toronto.
• Among the online panel suppliers, findings vary in amplitude and direction from measure to measure. In this environment of flux, it would be difficult to develop a set of criteria on which to select an online panel supplier.
• There is no firm basis on which to develop a conversion factor or weight that could bridge telephone and online findings because the directional and amplitude changes from panel to panel and city to city are too extensive.
• Variation from one online panel to another appears to be a harbinger of the variation that might be experienced if a single online panel supplier were taking measurements over multiple points in time (year-to-year changes that are results of turnover in panel membership). However, only two data points were available, using the TNS Canadian Facts panel, and it is likely premature to judge the long term results from using a single web-based panel.
• The extent to which differences between telephone and online results are a function of moving from an administered (telephone) to a self-completion questionnaire (online) cannot be estimated.

Discussion

The nature of the research that NADbank conducts requires a representative sample in order to accurately determine audience sizes and profiles for the daily newspapers being surveyed. The modified RDD telephone protocol has served NADbank well. The long-term expectation is that it will become increasing difficult to collect quality samples by this methodology. This research is part of an ongoing investigation into potential strategies for maintaining the quality of the NADbank Study.

The findings of this investigation confirmed the results of the first test that a web-based panel does not provide a representative sample, and secondly that different panels produce different results. While top level consistency was found in the TNS Canadian Facts panel (2006 to 2007), it is likely a red herring and does not provide a strong case for the suitability of the online panel protocol for media measurement studies. It would not appear to be the time for NADbank to move from its current telephone survey platform to the type of online panel used by commercial market research firms for multiple clients. These panels, regardless of the supplier, produce results that are sufficiently variable and fall short of face validity criteria (e.g., mother tongue in Toronto) as to render them unsatisfactory for estimating newspapers’ market share.

Given that a full shift to a new platform is not warranted it will be critical to continue to explore alternatives and potential transition opportunities to ensure that the platform provides valid, reliable results. It is essential that NADbank maintain data quality and trust in currency estimates. Next steps will be to explore alternative mechanisms for recruiting online panel members so that they more closely resemble a random sample of the population at large. In other words, ruling out existing commercially available “off-the-shelf” online panels for a newspaper audience study is not equivalent to ruling out a transition to any form of online data collection in the future.

NADbank will monitor developments such as the hybrid launched in 2008 for a large-scale financial services study in the U.K. (Cooke 2007) to determine whether a multi-modal approach could resolve some of the coverage and response rate issues associated with the current telephone platform.

Another option will be to consider building a panel from a random and representative sample of households (area frame) in major urban markets that could be accessed according to the preferences of the respondent (mail, online, telephone, face-to- face). Since the cost to develop a panel frame of sufficient size to meet NADbank’s needs and the costs of keeping the panel “refreshed” would be high, consideration might be given to working with a research supplier and/or other media organizations so that multiple users share the development and maintenance costs.

The inability to reach consumers with traditional research tools is growing but technologies are evolving to provide new avenues to reach those consumers. It is still ‘early days’ and a great deal of exploratory work will need to be done to assure ourselves that the new research protocols provide businesses with the data they need to make sound decisions. This test was another step along the way.

Appendices

Additional Summary Tables

TABLE A-1 GENDER AND AGE

TORONTO

WEIGHTED ACTUAL – unweighted

TELEPHONE TELEPHONE PANEL A PANEL B PANEL C PANEL D

COMPLETED SURVEYS 2063 2001 2214 2112 2029

GENDER

Male 49% 43% 52% 36% 45% 50%

Female 51% 57% 48% 64% 55% 50%

AGE

18-24 12% 9% 13% 6% 5% 9%

25-34 20% 17% 17% 15% 17% 20%

35-49 32% 37% 28% 37% 31% 33%

50-64 22% 23% 31% 28% 26% 23%

65+ 14% 13% 11% 14% 22% 15%

AVERAGE 18+ 45 46 45 47 49 45

HALIFAX*

COMPLETED SURVEYS 257 403 476 466 465

GENDER

Male 48% 41% 55% 36% 44% 48%

Female 52% 59% 45% 64% 56% 52%

AGE

18-24 12% 6% 12% 4% 5% 7%

25-34 19% 11% 18% 12% 16% 15%

35-49 30% 31% 30% 32% 29% 30%

50-64 24% 34% 22% 35% 29% 34%

65+ 14% 18% 19% 17% 21% 13%

AVERAGE 18+ 45 50 46 50 49 47

QUÉBEC CITY

COMPLETED SURVEYS 507 578 625 629 526

GENDER

Male 48% 42% 49% 41% 47% 47%

Female 52% 58% 51% 59% 53% 53%

AGE

18-24 11% 9% 15% 8% 5% 5%

25-34 18% 15% 19% 25% 17% 21%

35-49 27% 29% 23% 32% 28% 32%

50-64 26% 29% 31% 27% 30% 33%

65+ 17% 18% 11% 9% 18% 9%

AVERAGE 18+ 46 48 44 44 48 46

*Note that the base size for telephone interviews in Halifax is smaller than the online test because in the traditional telephone survey, two waves are generated and combined to generate annual estimates. The total annual sample for the telephone survey is 400.

TABLE A-2

DAILY NEWSPAPER READERSHIP IN TORONTO – WEIGHTED, PROJECTED

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
TORONTO STAR
Read yesterday	22%	24%	30%	31%	35%
5 day cume	38%	39%	45%	47%	51%
Saturday	28%	32%	39%	38%	44%
Sunday	18%	21%	26%	26%	31%
7 day cume	45%	46%	53%	54%	59%
Online past week	13%	19%	20%	22%	29%
Total weekly	50%	53%	58%	61%	66%
THE TORONTO SUN
Read yesterday	10%	14%	15%	12%	9%
5 day cume	22%	27%	28%	25%	19%
Saturday	8%	13%	15%	12%	9%
Sunday	12%	18%	19%	16%	11%
7 day cume	26%	32%	33%	31%	23%
Online past week	4%	9%	9%	10%	9%
Total weekly	27%	35%	35%	34%	27%
THE GLOBE AND MAIL
Read yesterday	9%	8%	9%	12%	21%
5 day cume	19%	15%	17%	20%	34%
Saturday	10%	8%	9%	10%	19%
6 day cume	20%	16%	18%	22%	36%
Online past week	9%	9%	9%	12%	22%
Total weekly	24%	20%	22%	27%	43%
NATIONAL POST
Read yesterday	5%	7%	7%	7%	12%
5 day cume	11%	11%	12%	12%	20%
Saturday	5%	5%	6%	6%	11%
6 day cume	12%	11%	13%	13%	21%
Online past week	3%	3%	4%	4%	8%
Total weekly	14%	13%	14%	15%	24%
METRO TORONTO
Read yesterday	11%	12%	12%	11%	12%
5 day cume	22%	23%	22%	23%	23%
Online past week	1%	4%	3%	4%	3%
Total weekly	22%	24%	23%	24%	24%
24 hours
Read yesterday	8%	12%	10%	9%	8%
5 day cume	16%	22%	19%	20%	16%
Online past week	2%	4%	3%	4%	4%
Total weekly	17%	23%	23%	21%	18%
ANY PAPER
Read yesterday	46%	48%	53%	53%	62%
5 day cume	68%	67%	69%	71%	77%
Saturday	40%	44%	51%	50%	61%
Sunday	27%	33%	39%	37%	37%
Weekly print cume	73%	73%	76%	77%	82%
Online past week	21%	28%	29%	34%	44%
Total weekly	77%	77%	80%	82%	87%

TABLE A-3

DAILY NEWSPAPER READERSHIP IN HALIFAX – WEIGHTED, PROJECTED

TELEPHONE PANEL A PANEL B PANEL C	PANEL D
THE CHRONICLEHERALD
Read yesterday 40% 43% 54% 52%	54%
5 day cume 63% 63% 71% 71%	73%
Saturday 42% 43% 52% 48%	55%
Sunday 31% 34% 42% 39%	44%
7 day cume 69% 65% 74% 74%	76%
Online past week 14% 33% 37% 32%	37%
Total weekly 72% 75% 83% 80%	83%
THE DAILY NEWS
Read yesterday 17% 19% 25% 20%	27%
5 day cume 32% 38% 43% 41%	44%
Saturday 15% 17% 20% 15%	24%
Sunday 15% 18% 20% 16%	21%
7 day cume 34% 39% 45% 44%	47%
Online past week 8% 15% 17% 14%	17%
Total weekly 37% 44% 50% 49%	52%
ANY PAPER*
Read yesterday 48% 51% 63% 58%	65%
5 day cume 73% 73% 81% 77%	84%
Saturday 48% 51% 63% 53%	66%
Sunday 39% 43% 52% 45%	53%
Weekly print cume 77% 74% 83% 79%	86%
Online past week 17% 38% 42% 37%	47%
Total weekly 81% 81% 89% 86%	92%
*Note Any Paper includes the national newspapers which are not displayed here.

TABLE A-4

DAILY NEWSPAPER READERSHIP IN QUÉBEC CITY – WEIGHTED, PROJECTED

TELEPHONE		PANEL A	PANEL B	PANEL C	PANEL D
LE JOURNAL DE QUÉBEC
Read yesterday	29%	31%	30%	26%	23%
5 day cume	48%	46%	43%	41%	45%
Saturday	34%	37%	34%	37%	30%
Sunday	21%	27%	25%	30%	22%
7 day cume	54%	51%	51%	50%	50%
LE SOLEIL
Read yesterday	23%	23%	25%	25%	32%
5 day cume	35%	35%	33%	31%	46%
Saturday	32%	32%	33%	40%	34%
Sunday	23%	23%	26%	30%	0%
7 day cume	41%	40%	42%	45%	51%
Online past week	7%	18%	14%	16%	21%
Total weekly	43%	45%	47%	51%	56%
ANY PAPER
Read yesterday	48%	47%	48%	45%	48%
5 day cume	68%	63%	59%	58%	67%
Saturday	58%	58%	56%	62%	56%
Sunday	41%	43%	45%	50%	22%
Weekly print cume	77%	71%	69%	72%	75%
Online past week	7%	18%	14%	16%	21%
Total weekly	78%	74%	72%	77%	78%

References

Chakrapani, C. (2007) On the validity of online panels. Canadian Journal of Marketing Research, volume 24, issue 1, p.7-15.

Crassweller, A., Williams, D., Thompson, I. (2007). Online data collection – A solution or a band-aid?, Paper resented at the 2007 WRRS in Vienna.

Cooke, M., Watkins, N., and Moy, C. (2007). Market measurement: the next frontier for panel research. Panel Research 2007, ESOMAR World Research Conference, Orlando.

Rogers, J., Crassweller, A. and Williams, D. Between random Samples and Online Panels: Where is the Next Lily Pad? Panel Research 2008, ESOMAR World Research Conference, Dublin.

The authors

Anne Crassweller is President of the Newspaper Audience Databank Inc. (NADbank) in Toronto, Canada. Judy Rogers is the President of Research Resolutions & Consulting Ltd. in Toronto, Canada.

Donald Williams is Vice-President, Research Director at the Newspaper Audience Databank Inc. (NADbank) in Toronto, Canada.