Home > Participation Criteria > 2021 Participation Criteria

Participation Criterion (iii) Approach for the Leaders' Debates for the 44th General Election

I. Introduction

On June 22, 2021, the Leaders' Debates Commission ("the Commission") set the participation criteria for the next federal leaders' debates in accordance with the Commission's amended mandate.

In order to qualify under criterion (iii) a leader of a political party must meet the following:

"five days after the date the general election is called, the party receives a level of national support of at least 4%, determined by voting intention, and as measured by leading national public opinion polling organizations, using the average of those organizations' most recently publicly-reported results."

The Commission has decided how it will apply criterion (iii), and this document provides the rationale for these determinations. The Commission is driven by its guiding principles for debate participation criteria, and in particular that they be simple, clear, and objective.

The Commission made these decisions following its request for and receipt of advice from the academics organizing the Canadian Elections Study. Convened by Professor Peter Loewen, the experts in this Polling Advisory Group are André Blais, Claire Durand, Allison Harell, Richard Johnston, Daniel Rubenson, and Laura Stephenson, in addition to Dr. Loewen.

In making its determination for a political party under criterion (iii), the Commission will select polls that:

were completed and released in the window from 9 days before to 5 days after the issuance of the writs;
were conducted by public opinion polling organizations that are either members of the Canadian Research Insights Council or are providing public opinion data to major media organizations, using the polling organizations' most recent publicly-reported result;
report explicit information about the support level for the party;
were conducted on a nationally representative sample of at least 1,000 respondents; and
publicly release methodological information sufficient for verification of the integrity of the poll, including question wording for the vote choice question(s), fielding dates, and details on sample size, weighting criteria, and survey mode.

The Commission will not consider internal party polling or polls commissioned by a political party.

In order to average the selected polls and determine a party's level of support, the polls will be averaged using a simple mean for each party included in at least two polls.

The Commission will make its decision with respect to which political parties meet the debate participation criteria no later than six days after the date the election is called.

The Commission's rationale for these determinations, as well as the advice received from the Polling Advisory Group, is provided below.

II. Criterion (iii)

In order to qualify under criterion (iii) a leader of a political party must meet the following:

"five days after the date the general election is called, the party receives a level of national support of at least 4%, determined by voting intention, and as measured by leading national public opinion polling organizations, using the average of those organizations' most recently publicly-reported results."

In its June 22, 2021 decision setting the participation criteria, the Commission stated that it would select public opinion polls based on the quality of the methodology employed, the reputation of the polling organizations, and the frequency and timeliness of the polling conducted.

The Commission also stated it may consider professional advice to assist the Commission in how to select the leading national public opinion polls to be used in applying the criterion.

To this end, the Commission sought the advice of the academics organizing the Canadian Elections Study (CES), a university research project initiated in 1965 to examine various aspects of federal elections. The Commission's initial point of contact was Professor Peter Loewen, who in addition to co-leading the CES study for the 44^th General Election, is a professor in the Department of Political Science and the Munk School of Global Affairs and Public Policy; Associate Director, Global Engagement at the Munk School; and Director of PEARL (Policy, Elections, & Representation Lab).

Dr. Loewen then convened a group of academics to provide advice to the Commission on the application of criterion (iii). The academics convened in this Polling Advisory Group are Peter Loewen, André Blais, Claire Durand, Allison Harell, Richard Johnston, Daniel Rubenson, and Laura Stephenson. Each of these academics is an expert in the field.

The Commission asked the Polling Advisory Group to provide advice to the Commission in order to make a determination under criterion (iii). In particular, the Commission asked the Polling Advisory Group to provide its advice on the following:

The timeframe during which to consider polling results;
Which national public opinion polling organizations to select; and
How to average the polling organizations most recently publicly-reported results.

The Commission received a submission from the Polling Advisory Group, included in full in Annex A. The Commission also obtained an absence of conflict declaration from each member of the group.

The group's recommendations for each of the three items, and the Commission's resulting analysis and conclusions will be considered in turn.

In its consideration of each of these elements, the Commission is driven by its guiding principles for debate participation criteria, and in particular that they be simple, clear, and objective.

Timeframe

The Polling Advisory Group recommended that polls be included provided that the poll fieldwork "be completed in the window from 9 days before to 5 days after the drawing of the writ." This amounts to a timeframe of 14 days in which to consider polling results. The Polling Advisory Group further explained that this timeframe contributes to the generation of the "largest possible number of polls."

The Commission has stated that it will consider the timeliness of the polling conducted. The Commission is satisfied that the recommendation of the Polling Advisory Group achieves this objective. The Commission concludes that a timeframe of 14 days is reasonable because it will provide the Commission with access to as broad a range of polls as possible for the application of criterion (iii), while still ensuring that the polls used remain in close proximity to the date of determination.

The Commission also examined the historical record by reviewing polls in past elections in the period five days after and nine days before the issuance of the writs. Voting intentions for parties did not frequently show significant movement during this time period.

Some public opinion polling organizations may release multiple polls issued during this period; in these instances, criterion (iii) states that the Commission will consider the organizations' most recent publicly-reported result.

Selection of national public opinion polling organizations

The Polling Advisory Group recommended that, in applying criterion (iii), the Commission select polls that:

a) are conducted by public opinion polling organizations that are either members of the Canadian Research Insights Council (CRIC) or are providing public opinion data to major media organizations;
b) report explicit information about the support level for the party;
c) were conducted on a nationally representative sample of at least 1,000 respondents; and
d) publicly release methodological information sufficient for verification of the integrity of the poll, which would include question wording for the vote choice question(s), fielding dates, and details on sample size, weighting criteria, and survey mode.

The Polling Advisory Group's rationale for these criteria states that, in their view, they satisfy "the guiding values articulated by the Commissioner" in that they are "clear and simple" while still requiring polls to pass a "credibility test."

The Commission has stated that it would select public opinion polls based on the quality of the methodology employed and the reputation of the polling organizations. The Commission is satisfied that the recommendation of the Polling Advisory Group takes this position into consideration.

In considering the recommendation made by the Polling Advisory Group, the Commission has reviewed CRIC's Public Opinion Research Standards and Disclosure Requirements that its members are required to follow. Its objectives are:

To support sound and ethical practices in the disclosure of research;
To ensure research is unbiased and supports decision-making in the public, private and not-for-profit sectors;
To enhance public trust and improve the public's understanding of the use of research; and
To ensure the appropriate transparency and disclosure of research results and methods of studies.

The Commission concludes that CRIC's requirements of its members align with the Commission's mandate and its guiding principles for setting participation criteria, and that therefore polls conducted by CRIC members within the 14-day timeframe are appropriate for inclusion in a determination of average voting intention.

The Commission recognizes that CRIC does not represent the entirety of the Canadian public opinion polling industry. However, the Commission agrees with the Polling Advisory Group that CRIC's membership includes a significant proportion of Canada's national public opinion organizations.

The Commission also agrees with the Polling Advisory Group's recommendation to supplement the pool of available polls by including those organizations who have been contracted by the major Canadian media organizations for the purposes of conducting polls to measure Canadians' voting intentions, as long as these polls satisfy methodological and disclosure requirements. This ensures that CRIC membership is not a necessary condition for inclusion in the Commission's decision-making process while maintaining a focus on the reputation and methodology of the organizations to be considered.

The Commission also concludes that it will not consider internal party polling, as recommended by the Polling Advisory Group, because internal party polling would not satisfy the above requirements that polls be publicly reported and undertaken by a CRIC member or a major news organization. The Commission also will not consider polling commissioned by a political party in order to ensure that the data available for the Commission's consideration is objective.

The Commission agrees with the Polling Advisory Group's recommendation that the measurement of a political party's level of support shall be done using polls where that party receives an explicit level of support. This means that respondents directly indicate their intention to vote for that party.

The Commission assessed this recommendation in light of two related considerations. First, not all polls will provide the option to respondents to indicate support for all political parties. Second, some polls will provide respondents with the option to select "Other" as their voting intention, after a subset of the full list of registered and eligible political parties is presented as a choice. While it may be argued that these two considerations mean that the Commission does not have a complete picture of voting intention for each political party, the Commission concludes that there are several factors that weigh against this argument. The Commission explains these factors in the following paragraphs.

The Commission is of the view that public opinion polling organizations have a professional and reputational interest in presenting to respondents as options those parties that have a meaningful presence in daily political conversation at the time. This means that it is likely that parties that would measure 4% or higher in voting intention are presented as options in their polls.

The Commission further notes that some current polls present respondents with the option to select political parties that have historically registered much less than 4% in past electoral results. Some polls also allow respondents to select newly emerging political parties and/or change the list of available responses over time. All of this evidence indicates that national public opinion polling organizations are likely to present respondents with the opportunity to indicate explicit support for parties that are likely to receive a measured voting intention of 4% or higher.

With regards to polls that have responses for "Other" parties, the Commission concludes that it is not in keeping with the guiding principles to undertake any method that would attempt to assign these voting intentions to particular political parties or to otherwise infer what respondents wished to indicate with a "Other" response. The Commission therefore concludes that adopting the approach of requiring an explicit level of support is reasonable and ensures the criteria and their application are simple, clear, and objective.

The Commission asked the Polling Advisory Group to consider whether a minimum sample size for polls should be considered, and the Commission agrees with the Polling Advisory Group's recommendation to only include those polls that have a sample size of 1,000 or greater. The Commission has reviewed the recent history of polls undertaken in Canada and notes that almost all polls conducted by leading national public opinion polling organizations use a sample size of 1,000 or more, and so it is unlikely that credible polls will be excluded as a result of this requirement. Polls with a sample size less than 1,000 are likely to have a greater margin of error and may not be comparable to polls with more respondents, and as a result establishing a minimum size of 1,000 will allow for a better comparison amongst the polls selected to determine the average.

The Commission also asked the Polling Advisory Group to provide expert advice on how to assess the credibility of all potential polls that could be used by the Commission. The Polling Advisory Group recommended to the Commission that it provide this expert advice in two parts.

In order to be clear prior to the issuance of the writs on what the Commission will be looking for, the group recommended that polls be included so as long as its release "is accompanied by the following publicly accessible information: exact wording on questions about vote intention; dates of fieldwork, sample size; weighting criteria, if any; and survey mode." The group also recommended that it provide advice to the Commission after the election is called explaining "the inclusion or exclusion of an individual firms' results" by assessing each poll against the pre-identified requirements.

The Commission is satisfied that the Polling Advisory Group's recommendations on the specific components required of each poll to be included in the Commission's analysis, in terms of method, transparency, and disclosure, will help ensure a credible process that aligns with the Commission's guiding principles.

The Commission recognizes that a fully predictable process would seek to name the specific polling organizations to be used in advance of the issuance of the writs but has concluded that in a minority context with an unknown election call date; such an approach may limit the polls available for use by the Commission. The Commission concludes that that the two-step process recommended by the Polling Advisory Group, which first identifies ahead of time the requirements for a poll to be included and then after the issuance of the writs provide specific advice assessing each potential poll against these requirements meets the Commission's guiding principles.

The Commission also considered a submission it received from The Conservative Party of Canada stating that the Commission, when making its determination for criterion (iii), should exclude polls from entities that have previously demonstrated a bias or animus towards a federally-registered political party. The Conservative Party of Canada submitted that such a bias exists towards them by EKOS Research Associates.

The Commission has considered this submission in its decision-making process. The Commission also provided this information to Dr. Loewen to ensure that the Polling Advisory Group's recommendations on a proposed methodology for applying criterion (iii) took this submission into consideration.

The Polling Advisory Group's submission recommends that polls be included subject to a number of factors discussed above which assess that poll based on organizational reputation, methodological soundness, and transparency and disclosure standards. The Commission agrees with the rationale of the group and concludes that the Commission's selection of national public opinion polling organizations will be based on these factors. No polling organizations will be excluded on the grounds that, in the view of one or more political party, they have previously demonstrated a bias towards a political party.

Averaging the selected polls

The Polling Advisory Group has recommended that, for each political party included in at least two polls, the mean value of that party's level of support as measured by voting intention will be determined using a simple mean across the polls in which it receives an explicit level of support.

The Commission has stated it will use an average of polls in order to make its determination of whether a party receives a level of national support of at least 4%, determined by voting intention. The Commission has considered the Polling Advisory Group's recommendations for how to determine the average of the selected polls. In so doing, the Commission is aware that there are differences amongst the methodologies and sample sizes of the above polls; that each poll involves statistical uncertainty or a 'margin of error'; and that each was conducted at slightly different periods.

The Commission is of the view that taking a simple mean is reasonable given that polls will only be included if they have a representative sample of at least 1,000 respondents and were all conducted in close proximity to the date of determination.

The Commission also concludes that it is reasonable to require a political party to appear in at least two polls that report explicit information about the support level for that party. As stated previously, the Commission is of the view that public opinion polling organizations are likely to present as options to respondents those parties that could measure 4% or higher in voting intention. Criterion (iii) also states that a party's level of national support shall be measured by polling organizations, meaning more than one organization.

With regards to statistical uncertainty and margin of error, the Commission concludes that it has attempted to mitigate the impact of this unavoidable uncertainty by incorporating multiple polls within its analysis. Additionally, the Commission is aware that polls typically report the maximum margin of error, which is the margin of error when the reported value is 50%. For values less than 10%, as is the case for applying criterion (iii), the margin of error is likely to be smaller.^{Footnote 1}

The Commission therefore concludes that each political party's mean value across the polls in which it receives an explicit level of support, as measured by voting intention, will be used to determine whether it receives a level of support of at least 4%.

III. Conclusion

In making its determination for a political party under criterion (iii), the Commission will select polls that:

Were completed and released in the window from 9 days before to 5 days after the issuance of the writs;
Were conducted by public opinion polling organizations that are either members of the Canadian Research Insights Council (CRIC) or are providing public opinion data to major media organizations, using the organizations' most recent publicly-reported result;
report explicit information about the support level for the party;
were conducted on a nationally representative sample of at least 1000 respondents; and
publicly release methodological information sufficient for verification of the integrity of the poll, including question wording for the vote choice question(s), fielding dates, and details on sample size, weighting criteria, and survey mode.

The Commission will not consider internal party polling or polls commissioned by a political party.

In order to average the selected polls and determine a party's level of support, the polls will be averaged using a simple mean for each party included in at least two polls.

IV. Next Steps

The Commission will make its decision with respect to which political parties meet the debate participation criteria no later than six days after the date the election is called. The Commission will then issue invitations to leaders of those political parties that meet debate participation criteria. A party not invited will have an opportunity to ask the Commission to reconsider within two days of the invitation to leaders. The Commission will make its final decision with respect to any such party within three days of that party's application for review. The tight timelines are being set to ensure that both the debates producer has sufficient time to produce a debate of high quality, as required by the OIC, and that the political parties can properly prepare for the debates in order to ensure they are informative for Canadians.

Annex A – Expert Group process for determining party inclusion in the Leaders' Debates (received August 11, 2021)

Prepared by Peter John Loewen, André Blais, Claire Durand, Allison Harell, Richard Johnston, Daniel Rubenson, and Laura Stephenson (the Expert Group)

Summary

The purpose of this report is to propose a method for determining which parties are eligible to participate in Canadian leaders' debates in the next federal election.

This report reviews the three inclusion criteria set out by the Commission, as well as the principles for determining included parties. After reviewing these, we propose a decision rule for inclusion, justify it according to the principles articulated by the Commission, and then lay out the exact method for determining inclusion given this decision rule.

Criteria for inclusion

On June 22, 2021, the Commissioner, Rt. Hon. David Johnston, released criteria for inclusion in the next leaders' debate.^{Footnote 2} These three criteria are:

(i): on the date the general election is called, the party is represented in the House of Commons by a Member of Parliament who was elected as a member of that party; or

(ii): the party's candidates for the most recent general election received at that election at least 4% of the number of valid votes cast; or

(iii): five days after the date the general election is called, the party receives a level of national support of at least 4%, determined by voting intention, and as measured by leading national public opinion polling organizations, using the average of those organizations' most recently publicly-reported results.

Our purpose is to specify a method to determine which party(ies) should be included according to the third criterion, assuming they do not qualify under the first two criteria.

In addition to the criteria specified, the Commissioner further stated that "With regards to criteria (iii), the Commission will select public opinion polls based on the quality of the methodology employed, the reputation of the polling organizations, and the frequency and timeliness of the polling conducted."

Values

Finally, the Commissioner noted that the criteria were "informed by our guiding principles that debates organized by the Commission should use simple, clear and objective criteria."

Decision Rule

The Commission has set a clear polling standard for inclusion in the Debates: an average level of national support of 4%, as measured by leading national public opinion polling organizations, using the most recent results of each organization, provided those results cover the first five days following the drawing of writs (more colloquially, the 'first five days of the campaign').

The decision rule(s) we recommend for the averaging of polls for each party are the following:

For each party, to include polls from all polling organizations who are members of the Canadian Research Insights Council (CRIC) and/or from polling organizations which are providing public opinion data to major media organizations. Provided that:
The polls report explicit information about the support level for the party;
The poll fieldwork must be completed in the window from 9 days before to 5 days after the drawing of the writ;
The results of those polls were publicly available on or before the fifth day of the campaign;
The polls were conducted on a nationally representative sample of at least 1000 respondents (unweighted);
For each poll, its release is accompanied by the following publicly accessible information: exact wording of questions about vote intention; dates of fieldwork; sample size; weighting criteria, if any; and survey mode;
For any polling firm or organization, the most recent poll satisfying criteria 1-6 will be included;
The polls will be averaged using a simple mean for each relevant party included in at least two polls.
Internal party polls will not be included.

The expert group will report which parties have an average support level equal to or exceeding 4% (unrounded) of decided voters across at least two polls and which parties fall below this threshold.

Justification

The decision rules above satisfy the specifics of the third criterion of inclusion, as well as the guiding values articulated by the Commissioner. First, we believe that our condition (1) of membership in CRIC - the leading accreditation body of Canadian public opinion research firms - or partnership with a major media organization satisfies the Commission's standard of leading national organizations. Moreover, it is a clear and simple rule, satisfying the guiding principles. Second, rule 2 satisfies the guiding principles of simplicity, clarity, and objectivity. Rules 3-6 are articulated to generate the largest number of possible polls, conditional upon them passing a credibility test (rule 6). Finally, the use of simple weighting satisfies the guiding principles of clarity and simplicity.

Process

We articulate a process to implement the above decision rules.

Beginning as soon as possible, the polling partners of major media organizations for the purposes of the election should be identified. Combining these firms with those who are in CRIC will generate the complete set of polling organizations to be considered by the Expert Group.
A definitive list of these organizations that identifies where their polls and accompanying information are posted will be created.
The Expert Group will develop a spreadsheet for calculating average party support.
Following the drawing of writs, the Expert Group will review polls released by all listed polling organizations in the 9 days prior to writs being issued.
Accepted results from the most recent poll conducted by a firm will be entered into the spreadsheet, with polling results updated up to and including the fifth day following the drawing of writs. The Expert Group will also compile all background information for each included poll in a single document.
On the sixth day of the campaign, the spreadsheet and identification of which parties equal or exceed 4% will be presented to the Commissioner, along with a memo explaining the results and any inclusion or exclusion of an individual firms' results.
The final decision will rest with the Commissioner.

On the Expert Group

Peter Loewen is a Professor at the University of Toronto. He is a co-investigator of the Canadian Election Study.

André Blais is a Professor at the Université de Montréal. He is a former co-investigator of the Canadian Election Study.

Claire Durand is a Professor at l'Université de Montréal. She is the former President of the World Association for Public Opinion Research.

Allison Harell is a Professor at the Université du Québec à Montréal. She is a co-investigator of the Canadian Election Study.

Richard Johnston is a Professor emeritus at the University of British Columbia. He is a former co-investigator of the Canadian Election Study.

Daniel Rubenson is a Professor at Ryerson University. He is a co-investigator of the Canadian Election Study.

Laura Stephenson is a Professor at Western University. She is a co-investigator of the Canadian Election Study.