By analysing the composition of judging panels and outcomes of rounds, this article demonstrates and discusses an bias toward teams from one’s nation.
I’ve heard people say that the World Universities Debate Championships is “the Olympics of debating.” I know I, for one, have never felt as patriotic watching archery or curling as I have pounding on a desk in an auditorium at some faraway university. Each year, I pack up my grandfather’s enormous flag and fly across the world, chanting patriotically on break night, waving my flag in the final round. And while there is nothing wrong with allowing a thousand nerds a week of patriotic honour, I began to wonder whether we are fully capable of turning off that patriotic allegiance when we enter a round as a judge. Can we ever be truly impartial? Or, subconsciously or not, are we always rooting for the home team?
Judge Nationality Affects the Outcomes of Outrounds
For the purposes of this analysis, I examined three regions: North America, Europe, and Australasia. I defined a team’s home region by which continental championship it would attend (NorthAms, Euros, or Australs), and judges’ home regions by the teams for which they most frequently competed in British Parliamentary debate 1. I looked for whether the regional composition of panels correlated with the outcomes of rounds, and found that in early outrounds, a team has a better chance of advancing if there is a judge from its home region on the panel. Three major patterns emerged.
First, a team is particularly advantaged by the presence of an independent adjudicator 2 from its region on its panel. While most judges in outrounds were selected through the independent adjudicator process, institutional adjudicators were brought by teams or the host university. For example, in partials, octos, and quarters at Berlin 3, 23% of judges were institutional adjudicators. There are a number of reasons why these judges provide less of an advantage than an independent adjudicator: they may be afraid to challenge the views of more established judges, and other judges on the panel may be less likely to listen to them. Whatever the explanation, the data shows that the presence of a regional independent adjudicator (“regional representation”) is especially advantageous: only 21% of teams with no regional independent adjudicator on their panels advanced, compared to 52% of teams with one regional independent adjudicator and 60% of teams with two or more, X 2 (2, N=80) = 6, p = .05 4.
Second, there is a particular advantage to having disproportionate regional representation on a panel. That is, teams are most likely to advance when the panel has judges from their home region but not from their opponents’ region(s). Some rounds, especially when all the teams are from the same region, are judged exclusively by judges from outside that region; that practice doesn’t uniquely disadvantage any team. However, when teams come from different regions and the judges on the panel are disproportionately from one of those regions, their chances of breaking differ drastically. Across Berlin and Manila partials, octos, and quarters, in rooms with at least two judges from one team’s home region and no judges from another team’s region, the “well-represented” team advanced 64% of the time and the “unrepresented” team only 20% of the time, X 2 (2, N=50) = 9.93, p = .002. This discrepancy is even more extreme when there are two independent adjudicators from one team’s home region on the panel. In that situation in Berlin, the “well-represented team” advanced 72% of the time, while the “unrepresented team” advanced only 16% of the time, X 2 (2, N=50) = 15.91, p < .0001.
Third, underdogs benefit particularly from being judged by adjudicators from their region. The odds of upsetting a higher-seeded team are better with regional judges present on the panel, especially regional independent adjudicators. Analysing all outrounds through quarters in Berlin and Manila together, a team not seeded to advance did so 40% of the time when there was at least one judge from its region on the panel, but only 25% of the time when there were none, p = .55 using Fischer’s exact test. This effect becomes marginally significant when examining the presence of independent adjudicators (and thus, only in Berlin, since Manila did not publish a list of independent adjudicators). A team not seeded to advance did so 45% of the time when there was at least one regional independent adjudicator on the panel, but in no cases where there was no regional independent adjudicator, p = .09 using Fischer’s exact test.
Chances of Advancing for Teams with 2+ Regional Independent Adjudicators in Berlin Outrounds, by Opponent Representation
This Bias Primarily Benefits European Teams and Handicaps North Americans.
The benefits of overrepresentation are not evenly shared by teams from all regions. Some regions consistently have many regional judges on panels, and some very few. In total, across all partials, octos, and quarters in Manila and Berlin, the teams were 40% European, 39% Australasian, and 21% North American. In the same period, the judges were 48% European, 35% Australasian, and 17% North American. Although this overall representation is roughly proportional, the allocation of judges benefits teams from certain regions in two important ways.
First, teams from certain regions are much less likely than teams from other regions to be in a room with no regional judge. Across all partials, octos, and quarters in Manila and Berlin, only 5% of European teams were in a room with no European judge; but 18% of Australasians and 31% of North Americans had no regional judge. Judges from certain regions are often allocated to rooms with no teams from that region. This trend was particularly pronounced in Manila: that year, North American independent adjudicators judged just four times in octos and quarters combined, but there was a North American team in the room only once. 5 Yet both North American teams in quarters, and four of the six North American teams in octos, lacked a single North American judge on their panel (let alone a North American independent adjudicator). All told, 75% of North American teams in outrounds in Manila had no regional judge on their panel, as did 45% of Australasians but just 13% of Europeans. Similarly, the one South African team to break in the last two years (in Berlin) lacked a regional judge on its panel even though African judges were judging other rounds. Thus, even when judges from underrepresented regions are breaking in proportion with teams from those regions, those teams are often not judged by those judges.
Second, the problem of disproportionate representation on panels tends to affect some regions much more than others. As discussed above, the effect on the result of the round is most drastic in a room where one team has two or more regional judges and one has none. Of course, this might be less alarming if regions were evenly advantaged and disadvantaged, but they are not. Of the rounds in Berlin and Manila at Berlin that followed this pattern, 57% of the overrepresented teams were European and 43% Australasian – not a single one was North American. Of teams benefiting from two independent adjudicators in Berlin, 78% were European. In contrast, 55% of the teams with no regional judges in these rooms were North American, 9% African, 27% Australasian, and 9% European. 6 A Fischer’s exact probability test of this distribution shows that it is extraordinarily unlikely that such a pattern is due to chance (p = .0001). Thus, although the overall proportion of judges from each region is not egregiously out of line with the proportion of teams, the allocation of these judges works to the benefit and detriment of certain regions much more than others. When European teams lacked a judge from their region, it tended to be when they were in an entirely European room – they were not comparatively worse off than their opponents. But when North American teams lacked a judge from their region, it was never when they were in an entirely North American room, and they were unusually likely to be facing a team that did have substantial representation on the panel. Thus, even if the overall judge break was proportional, the distribution of judges between rounds put the brunt of the impact of disproportionate representation on teams from just a few countries, primarily in North America.
Possible explanations
The data leads to two overall conclusions: first, that teams are more likely to advance when they are judged by people from their home region, and second, that teams from certain places are more likely than others to be judged by people from their home region. What might explain these conclusions?
One possibility is outright bias – that judges consciously try to get a team from their home region through to the next round. This explanation, however, is not fully supported by the data. If judges were voting based on allegiance and not their honest opinion, it would presumably be a significant advantage to have two judges on a panel when an opponent only had one; after all, the two judges could simply outvote the lone judge no matter what was said during adjudication. But this is not what happens. While it is a tremendous advantage to have two independent adjudicators when an opponent has none, it is no advantage at all to have two in comparison to one. In the 26 matchups in Berlin outrounds where one team had two independent adjudicators and the other had just one, each set of teams advanced exactly half the time.
On the opposite extreme, it’s possible that this correlation has nothing to do with bias: perhaps certain regions tend to produce more and/or better debaters, and thus also more judges. By this explanation, there are simply more talented European and Australasian judges and teams in the break, and so it is unsurprising that they tend to encounter each other more often and that their teams justifiably advance more often. However, it seems unlikely on face that debating talent is disproportionately distributed across the world; when examining a group of countries which all have top-notch universities with well-funded debating programs, it seems unlikely that some of these countries are consistently producing better debaters than others. Indeed, the data does not support this explanation. First, this account does not explain why upsets are more likely when judges judge teams from their home regions. If the effect were driven by the disproportionate debating talent in the region, it would presumably not hold for the less talented teams from that region who break as low seeds; those wins cannot be explained by a mere correlation between talent and nationality. Second, it does not explain the greater impact of regional judging when an opponent lacks a regional judge.
Rather, it seems most likely that the truth is somewhere in the middle: judges are neither consciously cheating nor perfectly neutral, and their unconscious preferences may bias them for teams from their home regions. Judges may feel obligated to at least make a case for a team from their own region (especially if they know the debaters personally), even if they are willing to be persuaded during adjudication. Or they may be subconsciously more persuaded by teams from their own regions: they might be more familiar with the arguments, speaking style, and strategic choices of such teams. When there is at least one “representative” for every team, this subtle bias is not too powerful; even if judges initially give more credit to the style they are familiar with, they are forced to think about unfamiliar teams and listen to another judge’s generous construction of their contributions. It is only when there is no advocate for one team that the bias becomes powerful – no longer just an initial intuition, but an intuition that is taken for granted as a team quickly falls out of the discussion as no one champions their cause. This explanation is less overtly sinister but just as dangerous and potentially harder for judges to correct.
Possible solutions
So what can be done to eliminate or reduce this bias? The most obvious answer is to have more judges from underrepresented regions, so that no teams face panels without a single regional adjudicator. The percentage of breaking teams from each region is fairly consistent; it would not be difficult to take representation into account when choosing independent adjudicators, just as adjudication teams typically do with other types of representation (like gender).
Of course, historically successful regions might argue that such a policy unfairly caps their numbers of independent adjudicators, by selecting less “qualified” adjudicators from less represented regions over their more “qualified” adjudicators. But this line of argumentation ignores the cyclical power of unrepresentative judging. A more “qualified” British adjudicator was more likely to have had the benefit of panels with European judges, who were in turn more likely to have had friendly faces on their panels when they won their own accolades. When the data proves that teams are more likely to succeed when judges are from their region, it’s hardly fair to conclude that those successes perfectly reflect the underlying talent of the competitors.
Even if a sort of affirmative action is not warranted, the power of regional representation certainly suggests that adjudication teams should be conscious of it as they pick independent adjudicators. They should be careful to count accomplishments from different regions comparably. And they should not allow the location of the tournament (and differential travel costs) to lead to overrepresentation of the nearest judges.
Even with the same regional distribution of judges, there are other tactics an adjudication team could use to allocate panels most fairly. First, when there are a limited number of judges from a particular region, those judges should be allocated to judge teams from that region. North American judges should not be used as mere tiebreakers between two Australians and two Brits on a five-person panel, at the expense of North American teams in other rooms. It’s much less harmful to have a panel with three judges from one country and two from another than it is to have a panel in another room with no judges from one of the team’s regions.
Second, judges from underrepresented regions should be allocated for multiple rounds in a row if necessary. Many adjudication teams try to give judges rounds off during outrounds, but those breaks should be a lower priority than regional representation. After all, judges handle multiple rounds a day during inrounds. And the use of back-to-back rounds to facilitate representation is nothing radical; in fact, the majority of female independent adjudicators in Berlin judged two (or even three) rounds consecutively, presumably in order to foster representation on the panels. The same policy could easily be applied to judges from underrepresented regions.
Finally, if some teams must be judged by a panel with no judge from their region, the adjudication team should be careful not to put two judges from any other team’s region in the same room. That two-to-none dynamic appears to create an echo chamber, leading to the most biased results. Avoiding that situation should be a priority; if it’s not possible to remedy it by adding a judge from the unrepresented team’s region, at the very least a judge with no ties to the teams’ regions should be used in place of a second judge from one team’s region.
In all of these strategies, adjudication teams should be careful not to rely on “token” representation from less experienced judges. A common strategy is to put one institutional judge from the region on the panel. These judges are frequently inexperienced, and understandably intimidated when they break and are placed on a panel with much more accomplished judges. Whatever the explanation, their presence leaves a team hardly better off than with no regional judge at all. If one team has two or more independent adjudicators from its region on the panel while its opponent has no representation, the addition of an institutional adjudicator raises the underrepresented team’s odds of advancing only slightly, from 20% to 31%. In contrast, the addition of an independent adjudicator raises the odds to a balanced 50%. It’s not enough to break judges for the purpose of “representing” teams on a panel when those judges aren’t qualified – only the addition of a qualified, experienced judge creates a fair chance at advancing.
Although these regions are fairly broad, there are substantial trends even based on these broad groupings. Furthermore, breaking regions down into subgroups (separating continental Europe from IONA and Asia from Oceania) did not lead to a different result in any measure reported in this article.
Independent adjudicators are judges selected for subsidisation by the adjudication team based on their experience speaking and adjudicating. In contrast, institutional adjudicators are judges brought by the competing univiersities.
When analysing the effect of independent adjudicators, I included only data from Berlin, because Manila did not publish a list of the independent adjudicators it selected.
A p-value reflects the probability of obtaining a result in a sample even if it were not true in the greater population, considering the size of the sample and the extremity of the result.
Notably, the one North American independent adjudicator who did judge an American team was also the only one who had previously debated in a community outside North America, before moving to the United States for his undergraduate studies; the country where he grew up also had a team in the room.
It’s worth noting that the few European and Australasian teams that fell into this category were generally not from the most historically successful countries of the region – the one European team was Turkish, and one of the Australasian teams was Singaporean.