Exemption 5 of the FOIA

Prior to the passage of the 1966 Freedom of Information Act, scholars, politicians, journalists, and lawyers expressed frustration and concern toward the perceived excessive secrecy of executive branch officials, which in turn produced a lack of access to information on internal government activity (Reference Halstuk and ChamberlinHalstuk and Chamberlin 2006). Although the 1946 Administrative Procedure Act directed agencies to share records with interested parties, many argued that the APA legislation obstructed rather than facilitated access (S. Rep. No. 89-813 at 38, 1966). A key goal of FOIA was to enable greater government transparency through a “judicially enforceable public right of access” to executive branch agency documents and information (Reference Halstuk and ChamberlinHalstuk and Chamberlin 512, 2006). Importantly, a number of exemptions recognize the need for agency discretion referenced by Justice Reed in Kaiser and allow agencies to withhold information requested through FOIA. For example, agencies could refuse to release documents on the grounds of national security or personal privacy (Reference Halstuk and ChamberlinHalstuk and Chamberlin 2006; Reference Mart and GinsburgMart and Ginsburg 2014). Exemption 5 of FOIA protects “interagency or intraagency memorandums or letters which would not be available by law to a party other than an agency in litigation with the agency” and incorporates the deliberative process privilege (Reference JensenJensen 1999; Reference KennedyKennedy 2005; Reference MilesMiles 1988).Footnote ⁸ In reference to Exemption 5, FOIA legislative history notes, “agency witnesses argued that a full and frank exchange of opinions would be impossible if all internal communications were made public” and the routine disclosure of internal communication would be akin to working within a “fishbowl” (H. Rep. No. 89-1497 at 31, 1966).

The privilege protects the same type of information in the FOIA context and the common law/civil discovery context. Also, in both contexts, courts must determine whether the withheld information properly reflects an exchange of information, opinions, and recommendations prior to the final adoption of agency policy. However, the identity of the requester, or the requester's specific need for the requested material, is not relevant in FOIA deliberative process privilege claims.Footnote ⁹ If a court finds the agency has properly applied the privilege, requester need does not override the application of the privilege in the FOIA context (Reference JensenJensen 1999). In the common law/civil discovery context, however, “courts engage in an ad hoc balancing of the evidentiary need [emphasis added] for allegedly privileged documents against the harm that may result from disclosure.”Footnote ¹⁰ Therefore, “a demonstrated need for requested material” can override the deliberative process privilege in the common law/civil discovery context (Reference NarayanNarayan 2008: 1196).

In either legal context, when ruling on deliberative process privilege claims, federal courts can make a variety of rulings. The court could rule that the privilege does not apply and that the agency must release all of their documents to the requester. Courts could also rule that the deliberative process privilege protects some of the requested material, but the agency must release some documents because of inaccurate application of the deliberative process privilege. In addition, judges can show complete deference to agency claims of deliberative process privilege and rule that all documents are privileged and that the agency litigant does not have to turn over any of the disputed documents.

Understanding Deliberative Process Privilege Litigation Outcomes

In FOIA disputes, courts exhibit substantial deference to agency withholdings (Reference VerkuilVerkuil 2002), particularly in disputes related to national security/foreign policy information (Reference Mart and GinsburgMart and Ginsburg 2014). However, existing research on FOIA litigation does not examine how factors such as agency structure (i.e., independent compared to executive branch) and agency ideological character affect judicial deference in transparency disputes. Specifically, when weighing executive privilege claims, does an agency's closeness to presidential control affect case outcomes? In addition, similar to political actors (i.e., judges, members of Congress), federal agencies are also classified along a liberal-conservative ideological spectrum (Reference Chen and JohnsonChen and Johnson 2015; Reference Clinton and LewisClinton and Lewis 2008). How does the ideology of agencies affect whether judges require agencies to release information to requesters? Finally, does the legal context (i.e., FOIA vs. Non-FOIA) affect judicial deference to agencies? For example, while the “Fast and Furious” litigation initiated by the House Oversight and Ethics Committee was ongoing, government watchdog organization Judicial Watch filed a FOIA request for similar documents sought by the Ethics Committee (Reference HattemHattem 2016). Given that Exemption 5 is a “permissive” exemption, and not a mandatory exemption, agencies have substantial discretion in applying the privilege.Footnote ¹¹ This discretion in application of the privilege confers judicial discretion in determining whether the privilege legally holds. Examination of privilege claims raised in various contexts across the federal executive branch helps further our current understanding of the ways in which federal judicial actors balance concerns of agency autonomy and government transparency.

Agency Institutional Structure

Judges will exhibit more deference to claims of deliberative process privilege raised by independent agencies and commissions, as compared to executive branch agencies.Footnote ¹² Executive branch agencies typically include cabinet agencies and agencies located within the Executive Office of the President (EOP), whereas independent agencies are those that fall outside of the cabinet structure (Reference Devins and LewisDevins and Lewis 2008, Reference LewisLewis 2003). Generally, the president has greater control over policy, hiring and firing, budgets, rulemaking, and reorganization in executive branch agencies (Reference LewisLewis 2008; Reference MoeMoe 1982; Reference SheehanSheehan 1992; Reference Wood and WatermanWood and Waterman 1991). Independence, as it relates to federal agencies, can refer to independence of agency leadership and/or an independence of policy decisions (Reference SelinSelin 2015). Independent agencies are associated with party-balancing requirements, term limits, and other features that limit the president's ability to appoint and remove agency leadership (Reference Aberbach and RockmanAberbach and Rockman 2009; Reference LewisLewis 2003). Exemption from Office of Management and Budget (OMB) review can also reduce presidential influence over administrative policy (Reference SelinSelin 2015). Although expertise and policy considerations influence agency/commission creation, scholars find that partisanship significantly affects whether agencies are insulated from executive influence. Specifically, when the president's party enjoys majority party control of Congress by larger margins, Congress is more willing to place administrative structures under greater presidential control (Reference LewisLewis 2003: 127).

Courts may apply greater scrutiny to executive branch agencies because of concerns of decisionmaking driven by partisan politics (Reference SheehanSheehan 1992; Reference StephensonStephenson 2004). One may expect that judges would show increased deference to those administrative agencies closest to the president (Reference JohnsonJohnson 2014), given the potential for sanctions or retaliatory response from the executive (Reference HowellHowell 2003; Reference JohnsonJohnson 2003). However, federal judges will apply more scrutiny to executive branch agencies due to concern over the influence of partisan and political considerations in agency decisionmaking (Reference SheehanSheehan 1992), particularly in disputes involving the release of politically sensitive information. In the deliberative process privilege context, parties have accused officials working in agencies in close proximity to the president of withholding information that could lead to negative scrutiny of a presidential administration (Reference Kamen and ItkowitzKamen and Itkowitz 2014). In the dispute over documents pertaining to the “Fast and Furious” operation, Representative Darrell Issa's (R-CA) investigatory panel argued, “the President and the Attorney General attempted to extend the scope of the Executive Privilege well beyond its historical boundaries to avoid disclosing documents that embarrass or otherwise implicate senior Obama Administration officials” (Reference Kamen and ItkowitzKamen and Itkowitz 2014). Although the desire to prevent the release of information that could shine a negative light on administration officials and policy is understandable,Footnote ¹³ it does not represent a valid legal basis for withholding documents under the deliberative process privilege. Given the possibility that political/partisan concerns of agency officials may motivate privilege claims, I expect federal judges to apply heightened scrutiny, and less deference, to executive branch agencies in deliberative process privilege claim disputes.

1. Hypothesis 1: Federal judges will be more likely to defer to claims of deliberative process privilege from independent agencies, and less likely to defer to privilege claims from executive branch agencies.

Judge-Agency Ideological Congruence

Ideological congruence between federal judges and agencies will affect whether judges defer to an agency's privilege claim. The ideological environment consistently influences outcomes in cases involving federal agency litigants (Reference CrowleyCrowley 1987; Reference FixFix 2014; Reference Humphries and SongerHumphries and Songer 1999; Reference Mart and GinsburgMart and Ginsburg 2014; Reference ReveszRevesz 1997; Reference SheehanSheehan 1990). Similar or congruent policy preferences usually produce greater judicial deference to agencies in court. Reference CrowleyCrowley's (1987) analysis of U.S. Supreme Court review of administrative agencies finds that conservative justices show less support for social policy agencies (i.e., OSHA), which promulgate more liberal polices (as compared to economic policy agencies). In addition to the policy jurisdiction of the agency, the ideological direction of the agency decision affects judicial deference in federal courts (Reference CrowleyCrowley 1987; Reference SheehanHumphries and Songer 1999; Kaheny and Rice 2010; Sheehan 1990).

Reference Mart and GinsburgMart and Ginsburg's (2014) analysis of FOIA litigation provides important insight into the nuanced way partisan considerations emerge in transparency disputes. They find that majority-Republican panels in the D.C. Circuit were less likely to grant disclosure in Exemption 1 disputes involving national security or foreign affairs information (Reference Mart and GinsburgMart and Ginsburg 2014). The finding of increased deference was not present outside of the D.C. Circuit. However, Reference Mart and GinsburgMart and Ginsburg's (2014) analysis does not examine how the ideological tenor of the agency affects judicial deference to agency privilege claims.

Withholding documents differs from other policy outputs and decisions aligned with the mission of the agency. For example, an Environmental Protection Agency (EPA) regulation that requires lower carbon emissions from energy producers would be a “liberal” policy decision under most circumstances. However, the decision by an agency to withhold documents under the deliberative process privilege does not easily map upon the liberal-conservative ideological spectrum given the decreased connection to substantive agency policy outcomes. Instead, the ideology of the agency will influence judicial deference to deliberative process privilege claims. As noted by Reference ClintonClinton et al. (2012: 352), “government agencies are fundamentally political.” Agencies reflect factors such as organizational, personnel, policy, and mission-based preferences (Reference Bertelli and GroseBertelli and Grose 2009; Reference CohenCohen 1986; Reference NixonNixon 2004), which many times align upon a liberal-conservative ideological spectrum (Reference Clinton and LewisClinton and Lewis 2008; Reference ClintonClinton et al. 2012). An agency's ideological orientation, in the context of agency's decision to withhold information, provides an important cue, or signal, which affects judicial assessment of the credibility of an agency's claim of privilege. Incongruent policy views between judges and the agency reduce the credibility of agency arguments to withhold documents and should result in judicial rulings requiring greater agency transparency. Specifically, liberal judges should apply more scrutiny to deliberative process privilege claims of conservative agencies; whereas conservative judges should be less deferential to liberal agencies that attempt to withhold documents under the deliberative process privilege.

1. Hypothesis 2: Federal judges will be more likely to defer to agency claims of deliberative process privilege when there is ideological congruence between the judge and the agency, and less likely to defer to privilege claims from ideologically incongruent agencies.

Legal Context

Finally, the legal context of the deliberative process privilege claim will also influence whether a federal judge will defer to a claim of privilege. In addition to the influence of judicial policy preferences on decisionmaking (Reference Segal and SpaethSegal and Spaeth 2002), legal factors such as precedent, the standard of review, and the controlling statute have a significant impact on agency litigation outcomes (Reference Humphries and SongerHumphries and Songer 1999; Reference RasoRaso 2015; Reference Schuck and Donald ElliotSchuck and Donald Elliot 1990). FOIA requires courts to use de novo review when examining agency withholdings and “this more intense level of review means that individuals are more likely to be protected and agency actions are more likely to be reversed” (Reference VerkuilVerkuil 2002: 699). However, in his analysis of FOIA litigation outcomes between 1990 and 1999, Reference VerkuilVerkuil (2002) finds that only 10 percent of agency withholding decisions are reversed, which suggests that the expected correlation between scope of review and case outcome is not always present. Similarly, Mart and Reference Mart and GinsburgGinsburg's (2014) examination of Exemption 1 litigation shows that district courts rule in favor of the federal government in about 74 percent of cases.

Questions remain on the effect of FOIA itself of judicial outcomes. One way to examine the effect of FOIA is to explore judicial outcomes on FOIA and non-FOIA privilege claims. Judges explain that agencies should construe all FOIA exemptions narrowly, and that the government bears the burden of proving that withheld documents are exempt (Reference MilesMiles 1988).Footnote ¹⁴ Given that FOIA dictates de novo review (Reference KwokaKwoka 2013; Reference VerkuilVerkuil 2002) and FOIA's strong “presumption of disclosure,”Footnote ¹⁵ I expect federal judges to exhibit less deference toward claims of deliberative process privilege in the FOIA context, and more deference in the common law/discovery context.

1. Hypothesis 3: Federal judges will be more likely to defer to agency claims of deliberative process privilege raised in the common law/discovery context, and less likely to defer to agency claims of privilege in the FOIA context.

Independent Variables

My three key independent variables are agency institutional structure, judge-agency ideological congruence, and the legal context of the deliberative process privilege claim. For agency structure, I create the indicator variable “Independent Agency”: agencies and commissions outside of the cabinet structure are coded as 1, and cabinet departments and bureaus are coded as 0.Footnote ²¹ I use the Sourcebook of United States Executive Agencies to categorize independent agencies and cabinet agencies. I expect federal district courts to exhibit greater deference (i.e., Partial Privilege and Complete Privilege) to deliberative process privilege claims raised by independent agencies and commissions. Approximately 24 percent of the observations in the data set involve independent administrative agencies and commissions.

Developing an indicator of judge-agency ideological congruence is somewhat complex. I use Judicial Common Space (JCS) scores to measure district judge ideology (Reference BoydBoyd 2010). For the judges in my analysis, the JCS scores range from −0.502 to 0.578 (liberal to conservative).Footnote ²² Although many previous analyses use the party of the president as a proxy for agency ideology, Reference Clinton and LewisClinton and Lewis (2008) use expert ratings and agency characteristics to develop a static measure of agency ideology for administrative agencies in existence between 1988 and 2005. The Reference Clinton and LewisClinton and Lewis (2008) scores are based upon expert assessment of whether the agency in question tends liberal, tends conservative, or neither consistently. Similar to JCS scores, negative values of the Reference Clinton and LewisClinton and Lewis (2008) scores indicate liberal agencies and positive values indicate conservative agencies. For the agencies in my analysis, the ideological scores range from −1.72 (Peace Corps) to 2.21 (Department of Defense).Footnote ²³

I create an indicator variable, “Judge-Agency Ideological Congruence,” to capture the ideological congruence between agencies and federal district judges. Specifically, if the district judge's JCS score and the agency's Reference Clinton and LewisClinton and Lewis (2008) score both fall on the right of 0 of their respective ideological scales, this suggests that the agency and the judge are both somewhat conservative. When the judge and the agency both fall on the same side of their respective ideological scale, I code this as 1. If the agency and the district judges fall on opposite sides of their respective scales, the observation receives a coding of 0. Fifty-three percent of the judges and administrative agencies observations are ideologically congruent. Judges should be more likely to defer to claims of deliberative process privilege when judges are ideologically congruent with federal agencies.

Finally, I create an indicator variable, “FOIA” for my measure of legal context. I code the “FOIA” variable as 1 when deliberative process disputes occur in the context of a FOIA request for agency documents and 0 otherwise (i.e., common law/civil discovery). I expect federal district judges to show greater deference to deliberative process privilege claims raised in the non-FOIA context and less deference in the common law/civil discovery context. Sixty-five percent of cases involve FOIA disputes.

Control Variables

I control for a number of additional factors in my analysis that could influence case outcomes and are correlated with my key variables of interest. The status of the parties (i.e., government vs. nongovernment) can affect litigation outcomes (Reference Black and BoydBlack and Boyd 2012; Reference GalanterGalanter 1974; Reference McGuireMcGuire 1995). I divide the litigants in the deliberative privilege cases into four categories and create four indicator variables: Business (23 percent), Individuals (45 percent), Public Interest (29 percent), and Government (4 percent). Corporations and companies comprise the business categories, whereas advocacy groups, unions, church organizations, interest groups, and so on comprise the public interest category. The government category includes government entities or officials (i.e., cities or legislators) suing for access to executive branch information, and the individual category includes individuals such as taxpayers, individuals in federal custody, and researchers seeking access to privileged documents. Because the public interest category involves many groups with repeat experience before the court (i.e., Judicial Watch and Tax Analysts), I expect administrative agencies to receive the least deference when those litigants appear in court, and the most deference when individuals petition for privilege documents. Because of advantages in resources, the likelihood of success of those in the business and government category should fall in between that of the individual and public interest categories.

Because agency type can affect judicial decisionmaking (Reference CrowleyCrowley 1987), I control for whether the agency's jurisdiction involves Social Regulatory policy (48 percent), Economic Regulatory policy (11 percent), or both Economic and Social Regulatory policy (23 percent). No regulatory policy jurisdiction serves as the base category (Reference Clinton and LewisClinton and Lewis 2008). I also control for whether the Department of Justice is the agency claiming privilege. A comparatively high percentage of cases (19 percent) involve the Department of Justice. Federal judges may be wary of releasing documents held by law enforcement agencies out of deference to ongoing agency investigations/litigation. Cases involving the Department of Justice as a litigant are coded as 1, and cases are coded as 0 otherwise.

Federal Magistrates decide the outcome in a number of cases in my analysis. Federal magistrate judges, who sometimes sit in district courts, are appointed by the federal judges in each district and serve 8-year renewable terms. Given that magistrates do not have life terms, and potentially have progressive ambition in terms of advancing upward in the federal court ladder, I expect magistrate judges to show increased deference to the executive branch in privilege cases and uphold agency withholding decisions. I create an indicator variable, Magistrate, that is coded as 1 for magistrate judges and 0 for non-magistrate judges.Footnote ²⁴ Magistrate judges decide 20 percent of outcomes in the data set.

Presidents can have a significant influence over the transparency of the executive branch (Reference RosenbergRosenberg 2008). Specifically, individual presidents can issue executives orders that can either increase or decrease access to government documents (Reference RosenbergRosenberg 2008); therefore, I control for the identity of the presidential administration. The time-period under investigation involves the Clinton Presidency (1994–2000) and the Bush II Presidency (2001–2004). I include an indicator variable for the Clinton Presidency (60 percent) in all models.

I create an indicator variable to control for potential effects of the D.C. District. The D.C. District handles a substantial number of disputes involving federal administrative agencies, and 43 percent of the disputes in the deliberative process privilege dataset occurred in the D.C. District. Existing research suggests that D.C. Circuit judges may be more deferential to government litigants in transparency cases (Reference Mart and GinsburgMart and Ginsburg 2014).

Finally, I also control for whether the opinion for the decision is Published. Although many studies of judicial decisionmaking understandably focus on published opinions, much of the work of district court judges results in unpublished opinions (Reference BoydBoyd 2015). Circuits develop varying norms and procedures for opinion publishing, and individual, institutional, and political factors influence whether district judges formally publish their decisions.Footnote ²⁵ Examining both published and unpublished opinions provides a more representative analysis of judicial decisionmaking in executive privilege litigation. Forty-eight percent of the observations involve published decisions.Footnote ²⁶ The descriptive statistics for the models used in the analysis appear in Table 2.

Table 2. Descriptive Statistics

Before presenting the results of the multinomial analysis, Table 3 provides a preliminary examination of deliberative process privilege outcomes through difference of means comparisons. To facilitate the analysis, I create a dichotomous variable by collapsing the No Privilege and Partial Privilege categories and comparing them against the Complete Privilege category. The results provide varying levels of support for my hypotheses. Judges are more likely to show independent administrative agencies complete deference in deliberative process privilege claims. In other words, judges are more likely to uphold an independent agency's decision to withhold documents under the deliberative process privilege. As previously stated, concern for partisan influence in transparency decisions may motivate increased judicial scrutiny, and less deference, for executive branch agencies. The results for Agency-Judge ideological congruence and FOIA are less precise and less conclusive. According to the estimates in Table 3, agencies that are ideologically congruent with federal judges and observations involving FOIA litigation are more likely to fall within the Complete Privilege category. However, these results are shy of the upper-boundary of traditionally accepted levels of statistical significance. Below, I discuss the results of the multinominal logit analysis using all three outcome categories.

Table 3. Difference of Means Analyses

^** p < 0.05.

Multivariate Analysis Results

Institutional and ideological factors substantially influence judicial decisionmaking in government transparency cases. In Table 4, the Complete Privilege category serves as the reference category for the Partial Privilege and No Privilege categories. According to the coefficient estimates in Table 4, district judges are less likely to show no deference (No Privilege) to independent agencies, relative to the reference category of Complete Privilege. Viewing the probability estimates in Table 5, district judges are less likely to require independent agencies to release all withheld documents as compared to executive branch agencies. The probability that an independent agency receives complete deference (Complete Privilege) is 0.866, whereas the probability that a district judge exhibits complete deference to an executive branch agency's claim of privilege is 0.633. In other words, an independent agency is +0.233 more likely to have their deliberative privilege claim upheld in full compared to an executive branch agency. For example, a claim of privilege made by an independent commission like the National Labor Relation Board has a greater likelihood of surviving judicial scrutiny when compared to a claim raised by the Department of Labor.Footnote ²⁷ The change in probability for independent and executive branch agencies across privilege outcomes appears in Figure 1.Footnote ²⁸

Table 4. Judicial Decisionmaking on Deliberative Process Privilege Claims: Multinomial Logit Estimates

Standard errors clustered by case citation. Tests are two-tailed. Complete Privilege is reference category.

Note: ^**p < 0.05; ^*p < 0.10.

Table 5. Predicted Probabilities: Probability Estimates for Deliberative Process Privilege Outcomes

Predicted probabilities calculated by holding categorical variables at their mode. ^**p < 0.05, ^*p < 0.1. “+” indicates choice with the highest probability.

Figure 1. Agency Structure and Deliberative Process Privilege Claim Federal District Court Rulings. [Color figure can be viewed at wileyonlinelibrary.com]

In their discussion of FOIA Exemption 1 usage and litigation, Reference Mart and GinsburgMart and Ginsburg (2014: 726) decry not only what they describe as excessive secrecy from agencies, but also “excessive judicial deference.” If agencies assume that federal courts will be more likely to uphold agency withholdings and not disrupt agency decisionmaking, this could potentially lead to an even greater propensity of certain agencies to restrict access to internal information. Specifically, the expectation of greater judicial deference may promote less agency transparency.

The results in Table 4 also show that courts are also less likely to require ideologically congruent agencies to release all withheld documents (No Privilege), relative to the base category of Complete Privilege.Footnote ²⁹ According to the estimates in Table 5, the probability that an incongruent agency receives a ruling allowing the agency to withhold all disputed documents is 0.457; whereas the probability that a court rules an ideologically congruent agency can withhold all disputed deliberative process privilege documents is 0.633. For example, compared to a liberal judge, a conservative judge reviewing a privilege claim from a conservative executive branch agency such as the Department of Defense (DOD) is more likely to defer to the DOD's deliberative process privilege claim. Figure 2 presents the predicted probability changes for agency ideological congruence across all three dependent variable categories.Footnote ³⁰

Figure 2. Ideological Congruence and Deliberative Process Privilege Claim Federal District Court Rulings. [Color figure can be viewed at wileyonlinelibrary.com]

Contrary to my hypothesis, the results in Table 4 suggest that federal judges exhibit greater deference to agencies in the FOIA context; however, the estimates for the FOIA dispute variable are less precise than those for structural and ideological indicator variables. Specifically, according to Table 5, the probability that an agency falls within the Complete Privilege category is 0.663, whereas cases in the common law/civil discovery context have a probability of 0.443 of receiving complete deference. Figure 3 presents the predicted probability changes for the FOIA variable across all dependent variable categories. Although the burden of proof for withholding documents falls on the government in the FOIA context, these results seem to echo the findings by Reference Mart and GinsburgMart and Ginsburg (2014) and Reference VerkuilVerkuil (2002) of substantial deference to the government in FOIA claims.Footnote ³¹ Reference VerkuilVerkuil (2002: 716) speculates that judicial “skepticism” toward FOIA litigation, given the costs to agencies for compliance and the nature of the requesters, could contribute to the deference agencies receive in court. Also, the fact that the common law/civil discovery deliberative privilege can be overcome with a showing of need, unlike the FOIA deliberative privilege (Reference JensenJensen 1999), could also contribute to this result. The estimates for the control variables are imprecise and do not reach the traditional levels of statistical significance.Footnote ³² Specifically, according to the findings in Table 2, factors such as the identity of the requester and whether the case appeared in the D.C. District court do not affect the level of deference districts judges show agencies in deliberative process privilege litigation.

Figure 3. Legal Context and Deliberative Process Privilege Claim Federal District Court Rulings. [Color figure can be viewed at wileyonlinelibrary.com]