3.1 Bonferroni Correction

The BC (Bland and Altman Reference Bland and Altman1995; Dunn Reference Dunn1961) reduces the FWER by using a more stringent threshold as the number of tests increases. To control the FWER below $\alpha $ , the BC tests each hypothesis at the significance level $\alpha / (\text {number of tests})$ . For instance, when five hypotheses are tested at the conventional $5$ % level, each test is conducted at the $1$ % level. The BC is easiest to implement among the methods to control the FWER, since researchers only need to implement the standard test procedure and construct confidence intervals with a new significance level.

One caveat is that the BC can be overly conservative. In many applications, the BC reduces the FWER substantially lower than the level set by the user. Hence, the BC may suffer low statistical power and false-negative findings. We illustrate this point later in our simulation study.

Another critique of the BC is that the total number of tests in a “family” cannot be unambiguously defined and tracked (Sjölander and Vansteelandt Reference Sjölander and Vansteelandt2019). Hochberg and Tamhane define family as “[a]ny collection of inferences for which it is meaningful to take into account some combined measure of errors” (Reference Hochberg and Tamhane1987, 5). While conjoint designs clearly pre-specify the number of attribute levels, researchers often conduct many tests to ensure survey quality such as balance and attention checks. Moreover, many applications include subgroup comparisons (Leeper et al. Reference Leeper, Hobolt and Tilley2020). It may not be obvious which tests should be included in the “family” when using the BC.

While the decision on the number of tests may increase the researchers’ degree of freedom, this problem should be ameliorated by pre-registration, as Bansak et al. (Reference Bansak, Hainmueller, Hopkins, Yamamoto, Druckman and Green2021a) suggest for conjoint experiments in general. What constitutes a family depends on whether the type of research is exploratory or confirmatory. “In purely exploratory research the question of interest (or lines of inquiry) are generated by data-snooping. In purely confirmatory research they are stated in advance. Most empirical studies combine aspects of both types of research” (Hochberg and Tamhane Reference Hochberg and Tamhane1987, 5). Discussing this issue in greater detail is beyond the scope of this paper, but pre-registration will ameliorate this ambiguity to some extent. In the conclusion section, we provide a recommendation checklist for conjoint users.

3.2 Benjamini–Hochberg Procedure

The BH procedure (Benjamini and Hochberg Reference Benjamini and Hochberg1995) controls another measure of false-positive findings, the false discovery rate (FDR), which is defined as

$$ \begin{align*} \mathsf{FDR} \equiv \mathbb{E} \left[ \frac{ \# \text{ false discoveries} }{ \#\text{ total discoveries} } \right]. \end{align*} $$

The FDR indicates the average proportion of false positives among all statistical findings. Therefore, lowering the FDR implies that researchers can be more confident in their findings. The BH is a method for containing the FDR under a pre-set level $\alpha $ . The value of $\alpha $ is commonly set to $.05$ , that is, $5$ % of null hypothesis rejections are false positives on average. The key idea of the BH is to remove some findings after conducting standard hypothesis tests. In other words, it prunes significant estimates so that researchers obtain fewer false findings.

The BH is a rank-based method with four steps. (1) For m hypotheses, an m-vector of p-values is produced. (2) Rank the p-values in the ascending order and index by i. (3) Define $k \equiv \max \{i: p_i \leq \alpha \times i / m, 0 \leq i \leq m\}$ . (4) Reject null hypotheses $H_i$ for $i = 1,2, \ldots , k,$ whose p-values are smaller than or equal to $p_k$ , or reject none if k does not exist.

Although discussing theoretical properties of the BH (e.g., Benjamini and Hochberg Reference Benjamini and Hochberg1995; Benjamini and Yekutieli Reference Benjamini and Yekutieli2001) is beyond the scope of this paper, the BH is less susceptible to false-negative conclusions than the BC, because it accepts all statistically significant findings in its first step. However, the BH eliminates fewer false-positive findings. Moreover, the BH does not offer confidence intervals because it uses the p-values. We illustrate these limitations below by simulations and applications.

3.3 Adaptive Shrinkage

The Ash is a recently-proposed, empirical Bayes approach to controlling the FDR developed by Stephens (Reference Stephens2017) and Gerard and Stephens (Reference Gerard and Stephens2018). Applied researchers can easily incorporate the Ash in conjoint analysis routine using the ashr package in R (Stephens et al. Reference Stephens2020).

The basic idea of the Ash is post hoc regularization of estimated coefficients using a spike-and-slab prior (see Figure 2). Regularization, in general, decreases the sampling variance of an estimator by introducing additional information into estimation. For the Ash, the spike-and-slab prior is such auxiliary information. On the one hand, the spike part reflects the fact that some estimates are false positives, inducing estimates to be zero with a certain probability. On the other hand, the slab part allows estimates to be non-zero with the remaining probability. As a result, the Ash shrinks estimated coefficients and produces narrower confidence intervals and smaller mean squared error. As shown in Section 5, the Ash moves point estimates of small absolute values toward zero and removes their statistical significance. By contrast, large point estimates are preserved and their confidence intervals are shortened.

Figure 2 Example of the spike-and-slap prior distribution. The spike (point mass) is at zero, and the slap (gray curve) follows a normal distribution.

Formally, let ${\boldsymbol \beta } = (\beta _1, \ldots , \beta _J)$ denote estimates for J attribute levels, let $\hat {{\boldsymbol \beta }} = (\hat {\beta }_1, \ldots , \hat {\beta }_J)$ denote point estimates of ${\boldsymbol \beta }$ , and let $\hat {\boldsymbol {s}} = (\hat {s}_1, \ldots , \hat {s}_J)$ be the standard errors of $\hat {{\boldsymbol \beta }}$ . Consider the posterior distribution of ${\boldsymbol \beta }$ given $\hat {{\boldsymbol \beta }}$ and $\hat {\boldsymbol {s}}$ :

(1) $$ \begin{align} p({\boldsymbol \beta} | \hat{{\boldsymbol \beta}}, \hat{\boldsymbol{s}}) \propto p(\hat{{\boldsymbol \beta}} | {\boldsymbol \beta}, \hat{\boldsymbol{s}})p({\boldsymbol \beta} | \hat{\boldsymbol{s}}).\\[-24pt]\notag \end{align} $$

The likelihood in Equation (1) is the sampling distribution of $\hat {{\boldsymbol \beta }}$ approximated by the normal distribution with mean ${\boldsymbol \beta }$ and variance $\hat {\boldsymbol {s}}^2$ . To regularize a large number of estimates, independent spike-and-slab prior distributions are placed. Since the Ash is an empirical Bayes method, the mixture probabilities of the spike-and-slab prior are estimated by maximizing the penalized likelihood and then the posterior parameters are estimated using the prior parameter estimates. The confidence intervals are constructed based on the posterior distribution of ${\boldsymbol \beta }$ . Section C.1 of the Supplementary Material provides a greater detail of the model and estimation.

The Ash delivers an additional benefit because of the shrinkage property. Its regularization leads to smaller mean squared errors of the point estimates. This is attractive because in many social science applications, researchers are interested not only in “whether factor X affect respondents’ choice,” but also in “to what extent.” The classic immigrant conjoint experiment, for instance, found a bonus for some education relative to no formal education. When researchers would like to estimate the amount of the education bonus, the other correction methods do not reduce the sampling error of point estimates. The Ash, however, enables us to get more precise estimates in a principled manner. Section C.2 of the Supplementary Material illustrates this point by simulations.

4.1 Zero AMCEs

The results are summarized in Figure 3. As in Figure 1, the bars represent the number of datasets for each number of statistically significant estimates. Note that the black bars are identical to Figure 1. Figure 3 also shows the results of the BC, the BH, and the Ash with a mixture of uniform components and with a mixture of normal components.

Figure 3 False-positive AMCE estimates when all null hypotheses are true with correction methods. Whereas the standard analysis pipeline correctly accepts all null hypotheses in fewer than 80 datasets, the BC, the BH, and the Ash all correct multiple testing in more than 900 experimental trials with the Ash performing the best.

All three correction methods dramatically reduce the probability of false findings. Because all null hypotheses are true, all simulations should result in zero significant coefficients. As we discussed in Section 2, more than 90% of experimental trials would produce at least one significant estimate without correction. By contrast, both the BC and the BH remove false findings in more than 90% of simulation datasets. The Ash performs even better. It eliminates almost all false-positive findings.

4.2 Non-Zero AMCEs

It is perhaps rare that all AMCEs are zero in applications, because attributes are designed to capture promising theoretical hypotheses. We consider two sets of more realistic simulations where some AMCEs are not zero to see how the correction methods perform in such settings.

In the first scenario, one binary attribute has a non-zero AMCE, and the results are shown in Table 1. In the original profile data, this attribute corresponds to Gender. We vary the noise in simulations by changing the heterogeneity of AMCEs and the error variance of the regression model for latent responses. Since only Gender has an effect, the shaded cells are the target we would like to hit: tests identify only one true-positive finding and no false findings. The pattern is quite consistent with the simulation results shown in Section 4.1. Without correction, about 80% of experimental trials produce at least one false-positive finding. All correction methods improve the situation remarkably, with the Ash has the best performance in all circumstances.

Table 1 Number of datasets for each number of true- and false-positive findings when the AMCE of Gender is non-zero. (a) The effect of male is $-.06$ and the effects of female and all other attributes are drawn independently from $\mathcal {N}(0,.015^2)$ . The error variance of the regression model for continuous responses is $.01^2$ . (b) AMCEs are identical to Table 1a, but the error variance of the regression model is $.1^2$ . (c) The effect of male and the other attributes and the error variance are identical to Table 1b, but the effect of female is independently drawn from $\mathcal {N}(0,.12^2)$ . Empty cells indicate zero.

In the second scenario, all levels of the attributes that correspond to Gender, Education, and English in the original data have non-zero AMCEs, whereas the AMCEs of the others are zero.Footnote ⁴ Table 2 presents the results. Because 10 levels have non-zero AMCEs, the shaded cells indicate the number of datasets in which hypothesis tests are perfectly accurate. All cells to the right (above) are the number of samples where some false positives (negatives) are produced. For example, without correction, 248 experimental trials successfully detect exactly the true non-zero AMCEs; 314 detect those AMCEs, plus one false-positive result; three experiments do not yield any false-positive findings, but missed one non-zero effect.

Table 2 Number of datasets for each number of true- and false-positive findings when the true AMCEs of all levels in Gender, Education, and English are non-zero. Obtaining 10 true positives and zero false positives (shaded) is the ground truth. Empty cells indicate zero.

Table 2 shows the trade-off in using correction methods. On the one hand, the use of a correction method dramatically improves the number in the shaded cells. In contrast to $248$ without correction, almost all correction methods find the truth in more than $600$ samples. On the other hand, as the Sum column indicates, all correction methods produce false negatives more often than the standard approach. Reducing the number of false positives comes at a cost of increasing the number of false negatives. Moreover, the trade-off exists among the correction methods, too. As the most conservative correction method, the BC produces false negatives in about 30% of experimental trials. The BH is least likely to miss the true AMCEs, but it produces more false-positive conclusions than the other two. The Ash takes the middle ground: it produces false-negative findings less likely than the BC, and false-positive results less likely than the BH.

Given this trade-off, should researchers use a correction method? In Figure 3 and Table 1, the answer is clear: any correction method dominates non-correction. When only zero or one attribute level has AMCE, the use of correction methods reduces the risk of false-positive findings at no cost since there is nothing to be missed. However, if many levels have AMCEs as in Table 2, correction methods decrease the number of false positives in exchange for an increase of the number of false negatives. Hence, correction methods may not be uniformly better than not correcting.

Figure 4 presents a measure to evaluate this trade-off. It shows the distribution of the true-positive rate (TPR) minus the false-positive rate (FPR) across samples in the same simulations as Table 2. The TPR is the number of true positives divided by the number of true non-zero AMCEs while the FPR is the number of false positives divided by the number of true zero AMCEs. If a test is perfect, its TPR is one and FPR is zero, because the ideal test finds all non-zero AMCEs and does not falsely reject the null on any zero AMCEs. Therefore, the higher density is concentrated on the right in Figure 4, the better. The figure shows that the BH and the Ash achieve a value larger than $.85$ in almost all simulated samples while the distribution without correction has a longer tail on the left. The figure indicates that researchers are more likely to get the ideal outcome with a correction method than without any.

Figure 4 Density Histogram of the Difference between True Positive Rate (TPR) and False Positive Rate (FPR). A larger value on the x-axis indicates better performance. The figure is based on the same simulations as Table 2.

These simulations demonstrate the promise and pitfalls of multiple testing correction methods. First, researchers should always use some correction method when conducting conjoint survey experiments. Since conjoint analysis inherently requires a large number of hypothesis tests, some, if not all, statistically significant findings are likely to be false positives. Second, the risk of false-positive findings cannot be entirely eliminated, and correction methods differ across the ability and cost of reducing the number of false positives. The BC is most aggressive in avoiding false positives, but its cost of missing true findings may be substantial. The BH is the opposite, and the Ash is in between the two. Although none provides the perfect solution, researchers should choose a correction method that best suits their needs. In particular, the choice should be based on a careful assessment on the relative tolerance of false positives and false negatives.Footnote ⁵ We provide a checklist as an additional guidance in concluding remarks.

5.1 Selecting Immigrants in the United States

In the seminal paper on conjoint designs for causal inference, Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014) employ the conjoint design to explore the AMCEs of immigrants’ attributes on preference for admission to the United States. There are nine attributes: Gender, Education, Language, Origin, Profession, Job experience, Job plans, Application reasons, and Prior trips to U.S. To exclude unrealistic attributes combinations, the randomization for Education, Profession, Country of Origin, and Application reasons are conditionally independent given some constraints, and the randomization for the other five attributes are completely independent. The outcome variable is whether a respondent prefers a given profile in a paired comparison.

We focus on two attributes, Country of origin and Profession, shown in Figure 5.Footnote ⁷ The left panel of Figure 5 shows the estimates of the AMCE of each country of origin relative to India, with no correction, the BC, the BH, and the Ash. The most noticeable pattern is that the BC eliminates the statistical significance of all estimates except for the effect of Iraq. If we believe the BC results, respondents in their survey did not distinguish immigrants from India, Mexico, France, Germany, Sudan, and Somalia. On the other hand, coefficients adjusted by the BH and the Ash largely preserve the original paper’s conclusion that immigrants from Sudan, Somalia, and Iraq are less preferred than those from India.

Figure 5 Effects of the immigrant’s country of origin (left) and profession (right) on the probability of being preferred for admission to the United States. For country of origin, the reference category is India; for profession, the reference category is janitor. The plot shows estimates with no correction, the BC (Bonf), the Ash with a mixture of normal components (ash.Norm), and the Ash with a mixture of uniform components (ash.Unif) for each pair of comparison. BH $\checkmark $ next to a point estimate indicates the BH corrected coefficient is significant for the corresponding attribute level. The estimates are based on regression estimators with clustered standard errors at the respondent level; the bars represent 95% confidence intervals. The estimates with no correction replicate the results for the corresponding attributes in Figure 3 in Hainmueller et al. (Reference Hainmueller, Hopkins and Yamamoto2014, 21).

The right panel of Figure 5 presents the results on the Profession attribute. Janitor is the reference category. The original results suggest that there is a bonus for financial analysts, construction workers, teachers, computer programmers, nurses, research scientists, and doctors. Again, the BC renders more coefficients insignificant: financial analysts and computer programmers are indistinguishable from janitors. While the BH preserves all the original findings, the Ash changes the results for construction workers—the bonus for construction workers is indistinguishable from zero. The Ash result is in fact consistent with the argument of the original paper that high-skilled immigrants are preferred to low-skilled workers.

While we cannot adjudicate on the differences with certainty because the true value is unknown, some correction methods lead to more substantively understandable results over the others. The BC seems overly conservative, and its null findings may require further theoretical justification. The BH results agree with most non-corrected results, including some unexpected significant estimates. The Ash corrects some findings away but not as aggressively as the BC does, and it leads to conclusions that make most substantive sense in this application.

5.2 Selecting Trading Partners in Vietnam

Conjoint experiment is also useful in examining attributes of units other than individuals. Spilker et al. (Reference Spilker, Bernauer and Umaña2016) explore what types of countries are preferred partners for Preferential Trade Agreements (PTAs) by conducting conjoint surveys in Costa Rica, Nicaragua, and Vietnam. They include eight attributes in their design: Distance from the partner country’s capital with three levels; Size of the economy with three levels; Culture, a binary variable indicating similarity in tradition and language of the partner country; Religion, which contains three religions for Costa Rica and Nicaragua and four religions for Vietnam; Political system, three levels of the extent to which citizens democratically elect political leaders; Military ally, a binary variable indicates whether the partner country has a security alliance with respondents’ home country; Environmental protection standards and Worker rights protection standards, each takes three levels. All these attributes are completely randomized, and no profile is excluded in the original surveys. The outcome is binary, whether respondents choose a country profile in a paired comparison.

Figure 6 focuses on the effect of two attributes Military ally and Environmental protection standards on the respondents in Vietnam.Footnote ⁸ Among the three countries, Vietnam is the only one where non-military allies are punished relative to military allies. The original paper justifies the finding by its geopolitical location and military-security rivalries in the region (Spilker et al. Reference Spilker, Bernauer and Umaña2016, 710, 714). However, Vietnam has a “Three Nos” defense policy since 1998: no military alliance, no aligning with one country against another, and no foreign military bases on Vietnamese soil.Footnote ⁹ The context makes it difficult to interpret the significant result, because it is unclear what military allies mean to Vietnam given this defense policy. While the BH preserves the original finding, the BC and the Ash correct it away.

Figure 6 Effects of Military ally (top) and Environmental protection standards (bottom) on the probability of being preferred as trading partners in Vietnam. For Military ally, the reference category is allied; for Environmental Protection Standards, the reference category is lower standards. The plot shows estimates with no correction, the BC (Bonf), the Ash with a mixture of normal components (ash.Norm), and the Ash with a mixture of uniform components (ash.Unif) for each pair of comparison. BH $\checkmark $ next to a point estimate indicates the BH corrected coefficient is significant for the corresponding attribute level. The estimates are based on regression estimators with clustered standard errors at the respondent level; the bars represent 95% confidence intervals. The estimates with no correction replicate the results for the corresponding attributes in Figure 1.3 in Spilker et al. (Reference Spilker, Bernauer and Umaña2016, 715).

For environmental standards, while the preference for higher standards relative to lower standards is robust to different correction results, the bonus for countries with similar standards is not. Again, the BC and the Ash render it a false-positive conclusion. The BH agrees completely with the original conclusion, but we cannot rule out the possibility that this is guaranteed by the design of BH: there are not enough significant discoveries to begin with to control for FDR. A lower FDR may be needed to accommodate the smaller number of significant findings in social science researches.

The replication exercise demonstrates the usefulness of applying correction methods in conjoint design from a substantive perspective. Correction methods could raise alarms of potential limitations in the profile design. Such warnings would be valuable especially in the phase of pilot research or pretesting. Moreover, results that stand the test of correction would help authors make more convincing arguments. In this application, the authors of the original paper needed to justify their finding on the preference for military allies only in Vietnam, but it is difficult to interpret this finding given the fact that Vietnam has not have military allies for a while and will not for the foreseeable future. The authors could have avoided interpreting this result by using the BC or the Ash, even though they included the Military ally attribute, which should have been excluded from the design.