Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Open access
  • Cited by 1

Actions:

MathJax
MathJax is a JavaScript display engine for mathematics. For more information see http://www.mathjax.org.
      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Local Rankin–Selberg integrals for Speh representations
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Local Rankin–Selberg integrals for Speh representations
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Local Rankin–Selberg integrals for Speh representations
        Available formats
        ×
Export citation

Abstract

We construct analogues of Rankin–Selberg integrals for Speh representations of the general linear group over a $p$ -adic field. The integrals are in terms of the (extended) Shalika model and are expected to be the local counterparts of (suitably regularized) global integrals involving square-integrable automorphic forms and Eisenstein series on the general linear group over a global field. We relate the local integrals to the classical ones studied by Jacquet, Piatetski-Shapiro and Shalika. We also introduce a unitary structure for Speh representation on the Shalika model, as well as various other models including Zelevinsky’s degenerate Whittaker model.

Footnotes

The second named author is partially supported by NSF grant DMS 1700637.

1 Introduction

The theory of Rankin–Selberg integrals for $\operatorname{GL}_{n}\times \operatorname{GL}_{n^{\prime }}$, studied by Jacquet, Piatetski-Shapiro and Shalika in a series of papers starting in the late 1970s (notably [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83]), is a basic tool in the theory of automorphic forms with an abundance of applications. The theory is based on global zeta integrals (which involve Eisenstein series in the case $n^{\prime }=n$) that unfold to adelic integrals of Whittaker–Fourier coefficients of cuspidal representations. By local multiplicity one, these integrals factorize into a product of local zeta integrals pertaining to generic representations and their Whittaker models.

The purpose of this paper is to study a modification of the local Rankin–Selberg integrals in the equal-rank case for a class of representations $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ where $\unicode[STIX]{x1D70B}$ is an irreducible generic representation of $\operatorname{GL}_{n}$ over a $p$-adic field and $m\geqslant 1$ is an integer. If $\unicode[STIX]{x1D70B}$ is unitarizable (and generic), $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is the Langlands quotient of the parabolic induction of $\unicode[STIX]{x1D70B}|\det |^{(m-1)/2}\otimes \unicode[STIX]{x1D70B}|\det |^{(m-3)/2}\otimes \cdots \otimes \unicode[STIX]{x1D70B}|\det |^{(1-m)/2}$. In particular, if $\unicode[STIX]{x1D70B}$ is discrete series, $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is the usual ‘Speh representation’. These representations (of $\operatorname{GL}_{mn}$) are not generic if $m>1$ (i.e. they do not admit a Whittaker model). Instead, the integrals involve a different model which for simplicity we will call a Shalika model. (We caution, however, that it does not exactly coincide with the standard notion of the Shalika model in the literature.) It is known that any $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ admits a unique Shalika model, a fact that reflects the ‘smallness’ of $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$. Structurally, the new integrals look very much like the classical ones and in fact they can be explicitly related. In particular, the unramified computation reduces to that of the classical Rankin–Selberg integrals [Reference Jacquet and ShalikaJS81] (which in turn uses Cauchy’s identity and Shintani’s formula for the unramified Whittaker function of $\operatorname{GL}_{n}$ [Reference ShintaniShi76]).

Just as the Whittaker model gives rise to the so-called Kirillov model [Reference Gel’fand and KajdanGK75] (by restriction to the mirabolic subgroup, namely, the stabilizer of a vector in $\operatorname{GL}_{n}$ in its standard $n$-dimensional representation) the Shalika model gives rise to a closely related object which we call the Kirillov–Shalika model. The role of the mirabolic subgroup is now played by the joint stabilizer of $m$ linearly independent vectors in $\operatorname{GL}_{mn}$. The argument of Gelfand and Kazhdan shows that the Kirillov–Shalika model contains all functions that are compactly supported modulo the equivariance subgroup.

There are, however, some differences between the classical theory and its suggested analogue. First, in the unramified case, we are unaware of a simple closed formula for the spherical function in the Shalika model, except if $n\leqslant 2$ or if $n=3$ and $m=2$. A related, equally difficult, problem is the asymptotic behavior of a function in the Shalika model. Apart from the above-mentioned cases, both problems go beyond the ‘comfort zone’ of spherical varieties, for which the works of Sakellaridis [Reference SakellaridisSak13] and Sakellaridis and Venkatesh [Reference Sakellaridis and VenkateshSV17] provide a conceptual framework and satisfactory answers to the questions above. Moreover, at this stage it is not clear whether there is an analogue of the Bernstein–Zelevinsky theory of derivatives [Reference Bernšten̆ and ZelevinskyBZ76, Reference Bernshtein and ZelevinskyBZ77] in the case at hand. In particular, we do not know whether the restriction of $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ to a parabolic subgroup of type $((n-1)m,m)$ is of finite length.

Another aspect of the paper is to provide an explicit, manifestly positive, unitary structure for the Speh representation in its Shalika model. (By this we mean that the positive-definiteness is ‘clear and obvious’ from the definition.) Once again, this is modeled on the case of generic unitarizable representations, in which Bernstein gives a unitary structure for their Whittaker models by taking the $L^{2}$-inner product of Whittaker functions restricted to the mirabolic subgroup [Reference BernsteinBer84]. For $m>1$ we use instead the joint stabilizer of $m$ vectors, as before.

Along with the above-mentioned Shalika model, the representations $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ admit various other models, for instance the degenerate Whittaker model considered by Zelevinsky (for any irreducible representation) in [Reference ZelevinskyZel80]. We can think of this as a sequence of models starting from the Zelevinsky model and ending with the Shalika model. They all involve a character on a unipotent subgroup and are covered by the general construction of Mœglin and Waldspurger [Reference Mœglin and WaldspurgerMW87]. The unipotent subgroups in the sequence are decreasing. One can write down explicit isomorphisms (transition maps) between these models. This idea has been used by many authors, most recently and systematically by Gomez, Gourevitch and Sahi [Reference Gomez, Gourevitch and SahiGGS16, Reference Gomez, Gourevitch and SahiGGS17]. It also played a role in the recent work of Cai, Friedberg and Kaplan on new doubling constructions of $L$-functions [Reference Cai, Friedberg, Ginzburg and KaplanCFGK19, Reference Cai, Friedberg and KaplanCFK18]. We write an inner product for each of these models and show that the transition maps are unitary.

As far as we know, this is the first time a purely local, manifestly positive hermitian form for a general Speh representation is explicitly given. Of course, the intertwining operator on the standard module whose image is the Speh representation induces its unitary structure – a fact that is true in general for any unitarizable representation on a reductive group (cf. [Reference Knapp and ZuckermanKZ77, §4]). (In the case at hand we will explicitly relate this unitary structure to that on the Shalika model.) However, the semidefiniteness of the intertwining operator is far from obvious – in fact it is equivalent to unitarizability, which is known to be a difficult problem in general, as is evident from the work of Vogan and many others. Another realization of the inner product is obtained by using global theory to embed Speh representations as local constituents of automorphic forms in the discrete spectrum of $\operatorname{GL}_{mn}$ over the adeles [Reference SpehSpe83]. Finally, in the $m=2$ case one can also realize a Speh representation in the discrete spectrum of $L^{2}(H\backslash \!\operatorname{GL}_{2n})$ where $H$ is the symplectic group of rank $n$ [Reference SmithSmi18, Reference Lapid and OffenLO19]. However, there is no such analogue for $m>2$.

In principle, the new local integrals are the local counterpart of certain global integrals, just as in the classical case. However, in addition to Eisenstein series, these global integrals involve automorphic forms in the discrete spectrum, rather than cusp forms, and they unfortunately do not converge (for any value of $s$). It should be possible (for instance by using the recent work of Zydor [Reference ZydorZyd19]) to carry out a regularization procedure to make sense of these integrals and to justify the unfolding procedure. However, we will not discuss this aspect in the paper. Nor we will discuss the archimedean case, for which we expect many of our results to hold without change.

The main new results of this paper are in §§ 4 and 5. The unitary structure for Speh representations (and more generally, $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ for unitarizable generic $\unicode[STIX]{x1D70B}$), on their various models, is given in Theorem 4.3. The new zeta integrals are defined in § 5. The convergence, unramified computation and local functional equations are stated in Theorem 5.1.

We now give some more details about the contents of the paper. In § 2 we first introduce some notation and recall Zelevinsky’s classification of irreducible representations of the general linear group over a local non-archimedean field $F$. We then introduce the class of $m$-homogeneous representations, which includes the usual Speh representations and which is the main focus of the paper. In terms of Zelevinsky’s classification, they simply correspond to multisegments consisting of segments of length $m$, where $m\geqslant 1$ is a fixed integer parameter. The case $m=1$ exactly corresponds to generic representations (i.e. the classical theory). In § 3 we introduce the models pertaining to $m$-homogeneous representations, following Mœglin–Waldspurger. (In order to use their results, we assume from § 3 onward that $F$ is of characteristic $0$. As was pointed out to us by Dmitry Gourevitch, this assumption can be lifted. Details will appear elsewhere.) We also introduce the transition maps between the models. They are given by integrals which entail no convergence issues. Finally, we introduce the Kirillov–Shalika model which is the analogue of the classical Kirillov model for generic representations. In § 4 we introduce a family of bilinear forms on a pair of models of $m$-homogeneous representations. In the case where the two representations are in duality, these bilinear forms specialize to an invariant pairing, at least under some restrictions. In the unitarizable case this gives rise to a manifestly positive invariant unitary structure. The invariance is proved by induction on $m$ using Bernstein’s theorem on invariant distributions with respect to the mirabolic subgroup. In § 5 we define the local Rankin–Selberg integrals for $m$-homogeneous representations using their Shalika models. Applying the transition maps, we can express these integrals in terms of the Zelevinsky model. Hence, we get their rationality in $q^{s}$, the unramified computation and functional equations. In § 6 we obtain more information about the poles of the zeta functions and relate them to the above-mentioned bilinear forms and, in particular, to the invariant pairing. In § 7 we go back to the Kirillov–Shalika model and analyze in detail the case of Speh representations of $\operatorname{GL}_{4}$ pertaining to supercuspidal representations of $\operatorname{GL}_{2}$. We study the asymptotic behavior of a function in the Kirillov–Shalika model. At this stage, it is hard to tell whether the result is representative of the general case or merely a low-rank fluke. In § 8 we write an informal global expression, modeled after the classical Rankin–Selberg integrals, whose regularization is expected to unfold to the local integrals studied in the paper. The regularization is necessary as the integral does not converge. (It would also eliminate extraneous terms in the unfolding procedure.) However, we do not discuss the regularization procedure and only give a purely heuristic argument. Finally, in Appendix A we relate the pairing of § 4 to that induced by the intertwining operator on the standard module.

2 Preliminaries

2.1 Notation

Throughout the paper, fix a non-archimedean local field $F$ with ring of integers ${\mathcal{O}}$ and absolute value $|\cdot |$. In principle it should be possible to deal with the archimedean case as well with proper adjustments, but we do not consider this case here.

From § 3 onward, $F$ is assumed to be of characteristic $0$.

If $H$ is an algebraic group over $F$, we often also use $H$ to denote $H(F)$.

We will consider complex, smooth representations of finite length of the groups $\operatorname{GL}_{n}(F)$, $n\geqslant 0$. We denote the set of irreducible representations of $\operatorname{GL}_{n}(F)$ (up to equivalence) by $\operatorname{Irr}\operatorname{GL}_{n}$ and set $\operatorname{Irr}=\bigcup _{n\geqslant 0}\operatorname{Irr}\operatorname{GL}_{n}$. We write $\operatorname{Irr}\operatorname{GL}_{0}=\{\mathbf{1}\}$. (In contrast, the one-dimensional trivial character of $\operatorname{GL}_{1}$ will be denoted by $\mathbf{1}_{F^{\ast }}$.) The subset of supercuspidal (respectively, square-integrable, essentially square-integrable, tempered, generic) representations will be denoted by $\operatorname{Irr}_{\operatorname{cusp}}$ (respectively, $\operatorname{Irr}_{\operatorname{sqr}}$, $\operatorname{Irr}_{\operatorname{esqr}}$, $\operatorname{Irr}_{\operatorname{tmp}}$, $\operatorname{Irr}_{\operatorname{gen}}$). Thus,

$$\begin{eqnarray}\operatorname{Irr}_{\operatorname{cusp}}\subset \operatorname{Irr}_{\operatorname{sqr}}\subset \operatorname{Irr}_{\operatorname{esqr}}\quad \text{and}\quad \operatorname{Irr}_{\operatorname{esqr}},\operatorname{Irr}_{\operatorname{tmp}}\subset \operatorname{Irr}_{\operatorname{gen}}\!.\end{eqnarray}$$

By convention $\mathbf{1}\in \operatorname{Irr}_{\operatorname{tmp}}$ but $\mathbf{1}\notin \operatorname{Irr}_{\operatorname{esqr}}$.

Let $\unicode[STIX]{x1D70B}$ be a representation of $\operatorname{GL}_{n}(F)$. We denote by $\unicode[STIX]{x1D70B}^{\vee }$ the contragredient of $\unicode[STIX]{x1D70B}$ and by $\operatorname{soc}(\unicode[STIX]{x1D70B})$ the socle of $\unicode[STIX]{x1D70B}$ (the maximal semisimple subrepresentation of $\unicode[STIX]{x1D70B}$). If $\unicode[STIX]{x1D70B}$ is non-zero, then we write $\deg \unicode[STIX]{x1D70B}=n$, the degree of $\unicode[STIX]{x1D70B}$. For any character $\unicode[STIX]{x1D714}$ of $F^{\ast }$ (i.e. $\unicode[STIX]{x1D714}\in \operatorname{Irr}\operatorname{GL}_{1}$) we denote by $\unicode[STIX]{x1D70B}\unicode[STIX]{x1D714}$ the representation obtained from $\unicode[STIX]{x1D70B}$ by twisting by the character $\unicode[STIX]{x1D714}\circ \det$. For instance, $\unicode[STIX]{x1D70B}|\cdot |$ is the twist of $\unicode[STIX]{x1D70B}$ by $|\det |$. We also write $J_{P}(\unicode[STIX]{x1D70B})$ for the (normalized) Jacquet module of $\unicode[STIX]{x1D70B}$ with respect to a parabolic subgroup $P$ of $\operatorname{GL}_{n}$, defined over $F$. If $\unicode[STIX]{x1D70F}\in \operatorname{Irr}\operatorname{GL}_{n}$, then we write $\unicode[STIX]{x1D70F}\leqslant \unicode[STIX]{x1D70B}$ if $\unicode[STIX]{x1D70F}$ occurs as a subquotient of $\unicode[STIX]{x1D70B}$, that is, if $\unicode[STIX]{x1D70F}$ occurs in the Jordan–Hölder sequence of $\unicode[STIX]{x1D70B}$. If $\unicode[STIX]{x1D70F}$ occurs with multiplicity one in the Jordan–Hölder sequence of $\unicode[STIX]{x1D70B}$, then we write $\unicode[STIX]{x1D70F}\,{\leqslant}_{\operatorname{unq}}\unicode[STIX]{x1D70B}$.

If $\unicode[STIX]{x1D70B}_{1},\ldots ,\unicode[STIX]{x1D70B}_{k}$ are representations of $\operatorname{GL}_{n_{1}}(F),\ldots ,\operatorname{GL}_{n_{k}}(F)$ respectively, then we denote the representation parabolically induced from $\unicode[STIX]{x1D70B}_{1}\otimes \cdots \otimes \unicode[STIX]{x1D70B}_{k}$ (normalized induction), with respect to the standard parabolic subgroup of block upper triangular matrices, by $\unicode[STIX]{x1D70B}_{1}\times \cdots \times \unicode[STIX]{x1D70B}_{k}$ and refer to it as the product representation. We also use the notation $\operatorname{Ind}_{H}^{G}$ and $\operatorname{ind}_{H}^{G}$ to denote induction and induction with compact support (both normalized) from a subgroup $H$ of $G$.

For any $\unicode[STIX]{x1D70F}\in \operatorname{Irr}_{\operatorname{esqr}}$, let $\mathbf{e}(\unicode[STIX]{x1D70F})$ be the unique real number $s$ such that the twisted representation $\unicode[STIX]{x1D70F}|\cdot |^{-s}$ is unitarizable (i.e. has a unitary central character). Note that $\mathbf{e}(\unicode[STIX]{x1D70F}^{\vee })=-\mathbf{e}(\unicode[STIX]{x1D70F})$. Any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}$ can be written uniquely (up to permutation) as $\unicode[STIX]{x1D70B}=\unicode[STIX]{x1D70F}_{1}\times \cdots \times \unicode[STIX]{x1D70F}_{k}$ where $\unicode[STIX]{x1D70F}_{i}$ are essentially square-integrable. Let $\mathbf{e}(\unicode[STIX]{x1D70B})=\min \mathbf{e}(\unicode[STIX]{x1D70F}_{i})$. (For consistency we write $\mathbf{e}(\mathbf{1})=0$.) Then $\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\vee })\leqslant 0$ with equality if and only if $\unicode[STIX]{x1D70B}$ is essentially tempered. Moreover, $\unicode[STIX]{x1D70B}$ is tempered if and only if $\mathbf{e}(\unicode[STIX]{x1D70B})=\mathbf{e}(\unicode[STIX]{x1D70B}^{\vee })=0$. More generally, we will say that $\unicode[STIX]{x1D70B}$ is ‘approximately tempered’ (AT) if $\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\vee })+1>0$. Equivalently, $\mathbf{e}(\unicode[STIX]{x1D70F}_{i})-\mathbf{e}(\unicode[STIX]{x1D70F}_{j})<1$ for all $i,j$. It is known that every unitarizable $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}$ is (AT). (This follows from the classification of the unitary dual of $\operatorname{GL}_{n}(F)$ by Tadić [Reference TadićTad86].) We denote by $\operatorname{Irr}_{(AT)}$ the set of (AT) representations.

For any set $A$ we denote by ${\mathcal{M}}(A)$ the free commutative monoid generated by $A$, considered as an ordered monoid. Thus, an element of ${\mathcal{M}}(A)$ (a multiset of $A$) is a finite (possibly empty) formal sum of element of $A$.

2.2 Zelevinsky classification

We recall the well-known results and terminology of [Reference ZelevinskyZel80].

A segment $\unicode[STIX]{x1D6E5}$ (of length $l>0$ and center $\unicode[STIX]{x1D70C}\in \operatorname{Irr}_{\operatorname{cusp}}$) is a non-empty finite subset of $\operatorname{Irr}_{\operatorname{cusp}}$ of the form

$$\begin{eqnarray}\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(l)}=\{\unicode[STIX]{x1D70C}|\cdot |^{(1-l)/2},\unicode[STIX]{x1D70C}|\cdot |^{(3-l)/2},\ldots ,\unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2}\}.\end{eqnarray}$$

We define $\deg \unicode[STIX]{x1D6E5}=l\deg \unicode[STIX]{x1D70C}$ and write $e(\unicode[STIX]{x1D6E5})=\unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2}\in \operatorname{Irr}_{\operatorname{cusp}}$ (the endpoint of $\unicode[STIX]{x1D6E5}$),

$$\begin{eqnarray}\mathfrak{c}(\unicode[STIX]{x1D6E5})=\unicode[STIX]{x1D70C}|\cdot |^{(1-l)/2}+\unicode[STIX]{x1D70C}|\cdot |^{(3-l)/2}+\cdots +\unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2}\in {\mathcal{M}}(\operatorname{Irr}_{\operatorname{ cusp}})\end{eqnarray}$$

and $\unicode[STIX]{x1D6E5}^{\vee }=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}^{\vee }}^{(l)}$. For compatibility we also write $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(0)}=\emptyset$. Denote by ${\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$ the set of all segments. We extend $\deg$ additively to a function ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})\rightarrow \mathbb{Z}_{{\geqslant}0}$. Similarly, we extend $e$ and $\mathfrak{c}$ additively to functions ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})\rightarrow {\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$.

For any $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(l)}\in {\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$, let

$$\begin{eqnarray}Z(\unicode[STIX]{x1D6E5})=\operatorname{soc}(\unicode[STIX]{x1D70C}|\cdot |^{(1-l)/2}\times \unicode[STIX]{x1D70C}|\cdot |^{(3-l)/2}\times \cdots \times \unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2})\in \operatorname{Irr}\operatorname{GL}_{ deg\unicode[STIX]{x1D6E5}}.\end{eqnarray}$$

(For compatibility we also set $Z(\emptyset )=\mathbf{1}$.) Then $Z(\unicode[STIX]{x1D6E5})^{\vee }=Z(\unicode[STIX]{x1D6E5}^{\vee })$. Given $\unicode[STIX]{x1D6E5}_{1},\unicode[STIX]{x1D6E5}_{2}\in {\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$, we write $\unicode[STIX]{x1D6E5}_{2}\prec \unicode[STIX]{x1D6E5}_{1}$ if $\unicode[STIX]{x1D6E5}_{i}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{i}}^{(l_{i})}$ with $\unicode[STIX]{x1D70C}_{2}|\cdot |^{(1-l_{2})/2+\unicode[STIX]{x1D6FC}}=\unicode[STIX]{x1D70C}_{1}|\cdot |^{(1-l_{1})/2}$ for some $\unicode[STIX]{x1D6FC}\in \mathbb{Z}_{{>}0}$ such that $l_{2}-l_{1}<\unicode[STIX]{x1D6FC}\leqslant l_{2}$. If either $\unicode[STIX]{x1D6E5}_{2}\prec \unicode[STIX]{x1D6E5}_{1}$ or $\unicode[STIX]{x1D6E5}_{1}\prec \unicode[STIX]{x1D6E5}_{2}$, then we say that $\unicode[STIX]{x1D6E5}_{1}$ and $\unicode[STIX]{x1D6E5}_{2}$ are linked. The induced representation $Z(\unicode[STIX]{x1D6E5}_{1})\times Z(\unicode[STIX]{x1D6E5}_{2})$ is reducible if and only if $\unicode[STIX]{x1D6E5}_{1}$ and $\unicode[STIX]{x1D6E5}_{2}$ are linked.

The well-known classification result of Zelevinsky [Reference ZelevinskyZel80, Theorem 6.5] extends the map $\unicode[STIX]{x1D6E5}\mapsto Z(\unicode[STIX]{x1D6E5})$ to a degree-preserving bijection

$$\begin{eqnarray}\mathfrak{m}\mapsto Z(\mathfrak{m})\end{eqnarray}$$

between ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ and $\operatorname{Irr}$. If $\mathfrak{m}=\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k}$ and $\unicode[STIX]{x1D6E5}_{i}\not \prec \unicode[STIX]{x1D6E5}_{j}$ for any $i<j$ (which can always be arranged), then $Z(\mathfrak{m})=\operatorname{soc}(Z(\unicode[STIX]{x1D6E5}_{1})\times \cdots \times Z(\unicode[STIX]{x1D6E5}_{k}))$. An element of ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ is called a multisegment. We have $Z(\mathfrak{m})^{\vee }=Z(\mathfrak{m}^{\vee })$, where we extend $^{\vee }$ from ${\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$ to ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ additively. For any $\mathfrak{m}_{1},\mathfrak{m}_{2}\in {\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ we have $Z(\mathfrak{m}_{1}+\mathfrak{m}_{2})\,{\leqslant}_{\operatorname{unq}}Z(\mathfrak{m}_{1})\times Z(\mathfrak{m}_{2})$ [Reference Lapid and MínguezLM16, Proposition 3.5]. In particular, if $Z(\mathfrak{m}_{1})\times Z(\mathfrak{m}_{2})$ is irreducible, then it is equal to $Z(\mathfrak{m}_{1}+\mathfrak{m}_{2})$.

We note the following fact.

(1)

By identifying an irreducible supercuspidal representation with a singleton segment we view ${\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$ as a submonoid of ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$. The map $Z$ restricts to a bijection

(2)$$\begin{eqnarray}{\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})\rightarrow \operatorname{Irr}_{\operatorname{gen}}\!.\end{eqnarray}$$

An element of ${\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$ is called a cuspidal datum. We write $\mathfrak{c}(Z(\mathfrak{m}))=\mathfrak{c}(\mathfrak{m})$. The resulting map

$$\begin{eqnarray}\mathfrak{c}:\operatorname{Irr}\rightarrow {\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})\end{eqnarray}$$

is the supercuspidal support (which of course can be defined without reference to the Zelevinsky classification). The restriction of $\mathfrak{c}$ to $\operatorname{Irr}_{\operatorname{gen}}$ is the inverse of (2).

For any segment $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(l)}$ let $\unicode[STIX]{x1D6E5}^{-}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}|\cdot |^{-1/2}}^{(l-1)}$ denote either the segment obtained by removing the endpoint $e(\unicode[STIX]{x1D6E5})$ of $\unicode[STIX]{x1D6E5}$ if $l>1$ or the empty set otherwise.

Now let $\unicode[STIX]{x1D70E}=Z(\mathfrak{m})$, where $\mathfrak{m}=\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k}$. Let

$$\begin{eqnarray}\mathfrak{m}^{-}=\unicode[STIX]{x1D6E5}_{1}^{-}+\cdots +\unicode[STIX]{x1D6E5}_{k}^{-}\quad \text{(disregarding empty sets)}.\end{eqnarray}$$

Define recursively $\mathfrak{m}^{(0)}=\mathfrak{m}$ and $\mathfrak{m}^{(k)}=(\mathfrak{m}^{(k-1)})^{-}$, $k>0$, with $\mathfrak{m}^{(l)}=0$, $l$ minimal. Let $n_{k}=\deg e(\mathfrak{m}^{(k-1)})$, $k=1,\ldots ,l$, so that $n_{1}+\cdots +n_{l}=\deg \unicode[STIX]{x1D70E}$ and let $\unicode[STIX]{x1D714}_{k}=Z(e(\mathfrak{m}^{(k-1)}))\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n_{k}}$. Let $P=P_{\unicode[STIX]{x1D70E}}=M_{\unicode[STIX]{x1D70E}}\ltimes U_{\unicode[STIX]{x1D70E}}=M\ltimes U$ be the standard parabolic subgroup of type $(n_{l},\ldots ,n_{1})$. By [Reference ZelevinskyZel80, §8.3] the Jordan–Hölder sequence of $J_{P}(\unicode[STIX]{x1D70E})$ admits a unique generic irreducible representation $\unicode[STIX]{x1D714}$ of $M$ and, moreover, $\unicode[STIX]{x1D714}\,{\leqslant}_{\operatorname{unq}}J_{P}(\unicode[STIX]{x1D70E})$. Equivalently (by the uniqueness of the Whittaker model), this means that

(3)$$\begin{eqnarray}\operatorname{Hom}_{N_{M}}(J_{P}(\unicode[STIX]{x1D70E}),\unicode[STIX]{x1D713}_{P})=\operatorname{Hom}_{N}(\unicode[STIX]{x1D70E},\unicode[STIX]{x1D713}_{P})=\operatorname{Hom}_{G}(\unicode[STIX]{x1D70E},\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D713}_{P})\text{ is one-dimensional},\end{eqnarray}$$

where $N$ is the maximal nilpotent group of upper unitriangular matrices and $\unicode[STIX]{x1D713}_{P}$ is a character of $N$ which is trivial on $U$ and non-degenerate on $N_{M}=N\cap M$. (This property determines $P$ uniquely up to association.) Moreover, $\unicode[STIX]{x1D714}=\unicode[STIX]{x1D714}_{l}\otimes \cdots \otimes \unicode[STIX]{x1D714}_{1}$ (see, for example, [Reference Mínguez and SécherreMS14, Lemma 9.17]). (For an arbitrary $P$, $\operatorname{Hom}_{N}(\unicode[STIX]{x1D70E},\unicode[STIX]{x1D713}_{P})$ is finite-dimensional.) We will call the image of $\unicode[STIX]{x1D70E}$ in $\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D713}_{P}$ the Zelevinsky model of $\unicode[STIX]{x1D70E}$. In general, $\unicode[STIX]{x1D70E}\not \leqslant \unicode[STIX]{x1D714}_{l}\times \cdots \times \unicode[STIX]{x1D714}_{1}$. For example, if $\unicode[STIX]{x1D70E}=Z(\{\mathbf{1}_{F^{\ast }},|\cdot |\}+\{\mathbf{1}_{F^{\ast }}\})$, then $l=2$, $\unicode[STIX]{x1D714}_{2}=\mathbf{1}_{F^{\ast }}$, $\unicode[STIX]{x1D714}_{1}=Z(\{|\cdot |\}+\{\mathbf{1}_{F^{\ast }}\})$ and $\unicode[STIX]{x1D714}_{2}\times \unicode[STIX]{x1D714}_{1}$ is irreducible (and generic).

2.3 Ladder representations

A multisegment $\mathfrak{m}$ is called a (strict) ladder if it can be written as $\mathfrak{m}=\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k}$ where $\unicode[STIX]{x1D6E5}_{i+1}\prec \unicode[STIX]{x1D6E5}_{i}$ for all $i=1,\ldots ,k-1$. The corresponding irreducible representation $Z(\mathfrak{m})$ is called a ladder representation.

Lemma 2.1 [Reference Lapid and MínguezLM16].

The following two statements hold.

  1. (i) [Reference Lapid and MínguezLM16, Lemma 6.17] Let $\unicode[STIX]{x1D70B}_{1},\ldots ,\unicode[STIX]{x1D70B}_{k}$ be ladder representations. Then $\unicode[STIX]{x1D70B}_{1}\times \cdots \times \unicode[STIX]{x1D70B}_{k}$ is irreducible if and only if $\unicode[STIX]{x1D70B}_{i}\times \unicode[STIX]{x1D70B}_{j}$ is irreducible for all $i,j$.Footnote 1

  2. (ii) [Reference Lapid and MínguezLM16, Lemma 6.21] Suppose that $Z(\mathfrak{m}_{1})$ and $Z(\mathfrak{m}_{2})$ are two ladder representations and that each segment of $\mathfrak{m}_{1}$ also occurs in $\mathfrak{m}_{2}$. Then $Z(\mathfrak{m}_{1})\times Z(\mathfrak{m}_{2})$ is irreducible.

The Jacquet module of ladder representations was described in [Reference Kret and LapidKL12]. The following lemma is an immediate consequence.

Lemma 2.2 [Reference Kret and LapidKL12].

Let $\unicode[STIX]{x1D70B}=Z(\mathfrak{m})$ be a ladder representation and $P$ a maximal parabolic subgroup. The following statements hold.

  1. (i) $J_{P}(\unicode[STIX]{x1D70B})$ is a direct sum of irreducible representations of the form $\unicode[STIX]{x1D70F}\otimes \unicode[STIX]{x1D714}$ where both $\unicode[STIX]{x1D70F}$ and $\unicode[STIX]{x1D714}$ are products of ladder representations.

  2. (ii) If $\unicode[STIX]{x1D70F}\otimes \unicode[STIX]{x1D714}\leqslant J_{P}(\unicode[STIX]{x1D70B})$ and $\unicode[STIX]{x1D714}\notin \operatorname{Irr}_{\operatorname{gen}}$, then there exists $\unicode[STIX]{x1D70C}\in \operatorname{Irr}_{\operatorname{cusp}}$ such that $\unicode[STIX]{x1D70C}\leqslant \mathfrak{c}(\unicode[STIX]{x1D714})$, $\unicode[STIX]{x1D70C}|\cdot |\leqslant e(\mathfrak{m})$ but $\unicode[STIX]{x1D70C}\not \leqslant e(\mathfrak{m})$.

  3. (iii) If $\unicode[STIX]{x1D70F}\otimes \unicode[STIX]{x1D714}\leqslant J_{P}(\unicode[STIX]{x1D70B})$ with $\unicode[STIX]{x1D714}\in \operatorname{Irr}_{\operatorname{gen}}$, then $\mathfrak{c}(\unicode[STIX]{x1D714})\leqslant e(\mathfrak{m})$. Moreover, if $\unicode[STIX]{x1D70C}\in \operatorname{Irr}_{\operatorname{cusp}}$ is such that $\unicode[STIX]{x1D70C}\leqslant e(\mathfrak{m})$ and $\unicode[STIX]{x1D70C}|\cdot |\leqslant \mathfrak{c}(\unicode[STIX]{x1D714})$, then $\unicode[STIX]{x1D70C}\leqslant \mathfrak{c}(\unicode[STIX]{x1D714})$.

  4. (iv) If $\unicode[STIX]{x1D70F}\otimes \unicode[STIX]{x1D714}\leqslant J_{P}(\unicode[STIX]{x1D70B})$ and $\unicode[STIX]{x1D70C}\in \operatorname{Irr}_{\operatorname{cusp}}$ is such that $\unicode[STIX]{x1D70C}|\cdot |\not \leqslant \mathfrak{c}(\unicode[STIX]{x1D714})$, then $\unicode[STIX]{x1D70C}$ occurs in $\mathfrak{c}(\unicode[STIX]{x1D714})$ with multiplicity at most one.

Strictly speaking, the results of [Reference Kret and LapidKL12] are stated in terms of the Langlands classification. However, they are also valid in the form above (for the Zelevinsky classification) by either repeating the arguments, or using the Zelevinsky involution.

2.4 $m$-homogeneous representationsFootnote 2

From now on let $m,n\geqslant 1$ be integers and $G=\operatorname{GL}_{mn}$. We say that $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$ is $m$-homogeneous if $\unicode[STIX]{x1D70E}=Z(\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k})$ where each $\unicode[STIX]{x1D6E5}_{i}$ is of length $m$. (If $m=1$ this simply means that $\unicode[STIX]{x1D70E}$ is generic.) We denote by $\operatorname{Irr}_{m\operatorname{-hmgns}}G$ the set of irreducible $m$-homogeneous representations of $G$. For any $\unicode[STIX]{x1D70B}=Z(\{\unicode[STIX]{x1D70C}_{1}\}+\cdots +\{\unicode[STIX]{x1D70C}_{k}\})\in \operatorname{Irr}_{\operatorname{gen}}$, define

$$\begin{eqnarray}\operatorname{Sp}(\unicode[STIX]{x1D70B},m)=Z(\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{1}}^{(m)}+\cdots +\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{k}}^{(m)})\in \operatorname{Irr}.\end{eqnarray}$$

The following result is clear.

Lemma 2.3. The map $\unicode[STIX]{x1D70B}\mapsto \operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ defines a bijection between $\operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$ and $\operatorname{Irr}_{m\operatorname{-hmgns}}G$. We have $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)^{\vee }=\operatorname{Sp}(\unicode[STIX]{x1D70B}^{\vee },m)$ for any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}$.

Remark 2.4. The notion of $m$-homogeneous representations is very close to the concept of ‘representations of type $(n,m)$’ introduced in [Reference Cai, Friedberg, Ginzburg and KaplanCFGK19] and studied further in [Reference Cai, Friedberg and KaplanCFK18]. The difference is that we only consider irreducible representations and emphasize the roles of the Mœglin–Waldspurger models.

Remark 2.5. If $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{sqr}}\operatorname{GL}_{n}$, then $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is known as a ‘Speh representation’. (Strictly speaking, these representations were introduced by Speh in the archimedean case.)

Remark 2.6. In general, if $\unicode[STIX]{x1D70B}$ is unramified (and generic), then $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is not necessarily unramified if $m>1$. More precisely, if $\unicode[STIX]{x1D70B}=Z(\{\unicode[STIX]{x1D70C}_{1}\}+\cdots +\{\unicode[STIX]{x1D70C}_{k}\})$ is unramified (so that $\unicode[STIX]{x1D70C}_{i}$ are unramified characters of $F^{\ast }$ and $\unicode[STIX]{x1D70C}_{i}\neq \unicode[STIX]{x1D70C}_{j}|\cdot |$ for all $i,j$), then $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is unramified if and only if $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{1}}^{(m)},\ldots ,\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{k}}^{(m)}$ are mutually unlinked. For instance, this is the case if $\unicode[STIX]{x1D70B}$ is (AT).

Suppose that $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ with $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$. Then, in the notation of (2), $P_{\unicode[STIX]{x1D70E}}=P_{m,n}=M\ltimes U$ is the standard parabolic subgroup of $G$ of type $(\overbrace{n,\ldots ,n}^{m})$, consisting of the block upper triangular matrices with blocks of size $n\times n$. Thus, $M\simeq \overbrace{\operatorname{GL}_{n}\times \cdots \times \operatorname{GL}_{n}}^{m}$.

Just as in the case $m=1$, there are simple building blocks for $m$-homogeneous representations.

Proposition 2.7. Let $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)\in \operatorname{Irr}G$ be $m$-homogeneous. Then there exist $\unicode[STIX]{x1D70B}_{1},\ldots ,\unicode[STIX]{x1D70B}_{t}\in \operatorname{Irr}_{\operatorname{gen}}$ such that the following statements hold.

  1. (i) $\unicode[STIX]{x1D70E}_{i}:=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{i},m)$ is a ladder representation for all $i$.

  2. (ii) $\operatorname{Sp}(\unicode[STIX]{x1D70B},l)=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{1},l)\times \cdots \times \operatorname{Sp}(\unicode[STIX]{x1D70B}_{t},l)$ for all $l=1,\ldots ,m$. In particular, $\unicode[STIX]{x1D70B}=\unicode[STIX]{x1D70B}_{1}\times \cdots \times \unicode[STIX]{x1D70B}_{t}$ and $\unicode[STIX]{x1D70E}=\unicode[STIX]{x1D70E}_{1}\times \cdots \times \unicode[STIX]{x1D70E}_{t}$.

Moreover, let $Q$ be the maximal standard parabolic subgroup of type $((m-1)n,n)$ and denote by $J_{Q}(\unicode[STIX]{x1D70E})_{;\mathfrak{d}}$ the direct summand of $J_{Q}(\unicode[STIX]{x1D70E})$ pertaining to the supercuspidal data $\mathfrak{d}\in {\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$ in the second ($\operatorname{GL}_{n}$) factor. Then

$$\begin{eqnarray}J_{Q}(\unicode[STIX]{x1D70E})_{;\mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\otimes \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}.\end{eqnarray}$$

Remark 2.8. For $m=1$, $\unicode[STIX]{x1D70B}_{i}$ is essentially a discrete series. This is not the case for $m>1$ in general.

Proof. Write $\unicode[STIX]{x1D70B}=Z(\sum _{i\in I}\{\unicode[STIX]{x1D70C}_{i}\})$ with $\unicode[STIX]{x1D70C}_{i}\in \operatorname{Irr}_{\operatorname{cusp}}$ and let $l\geqslant 1$. We say that a subset $J$ of $I$ is an $l$-chain if it can be written, necessarily uniquely, as $J=\{i_{1},\ldots ,i_{r}\}$ where for all $j=1,\ldots ,r-1$ we have $\unicode[STIX]{x1D70C}_{i_{j}}=\unicode[STIX]{x1D70C}_{i_{j+1}}|\cdot |^{\unicode[STIX]{x1D6FC}_{j}}$ with $\unicode[STIX]{x1D6FC}_{j}\in \{1,\ldots ,l\}$. (For example, for a $1$-chain, $\unicode[STIX]{x1D70C}_{i_{r}},\ldots ,\unicode[STIX]{x1D70C}_{i_{1}}$ is a segment.) Clearly, $J$ is an $l$-chain if and only if $Z(\sum _{j\in J}\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{j}}^{(l)})$ is a ladder representation.

We say that two partitions of $I$ are equivalent if one can be obtained from the other by applying a permutation $\unicode[STIX]{x1D70F}$ of $I$ such that $\unicode[STIX]{x1D70C}_{\unicode[STIX]{x1D70F}(i)}=\unicode[STIX]{x1D70C}_{i}$ for all $i$. It is easy to see that for any $l\geqslant 1$ there exists a partition ${\mathcal{P}}^{(l)}(I)$ of $I$ consisting of $l$-chains, such that for any $J,J^{\prime }\in {\mathcal{P}}^{(l)}(I)$ at least one of the following conditions holds:

  1. (i) $\{\unicode[STIX]{x1D70C}_{j}:j\in J\}\subset \{\unicode[STIX]{x1D70C}_{j}:j\in J^{\prime }\}$;

  2. (ii) $\{\unicode[STIX]{x1D70C}_{j}:j\in J^{\prime }\}\subset \{\unicode[STIX]{x1D70C}_{j}:j\in J\}$;

  3. (iii) for every $j\in J$ and $j^{\prime }\in J^{\prime }$ the segments $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{j}}^{(l)}$ and $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{j^{\prime }}}^{(l)}$ are unlinked.

Moreover, ${\mathcal{P}}^{(l)}(I)$ is unique up to equivalence. Indeed, ${\mathcal{P}}^{(l)}(I)$ can be defined inductively by taking a maximal $l$-chain $J$ of $I$ (with respect to inclusion) together with the partition ${\mathcal{P}}^{(l)}(I\setminus J)$. It follows from this description that if $l\leqslant m$, then up to equivalence, ${\mathcal{P}}^{(l)}(J)=\{J^{\prime }\in {\mathcal{P}}^{(l)}(I):J^{\prime }\subset J\}$ for any $J\in {\mathcal{P}}^{(m)}(I)$ and, in particular, ${\mathcal{P}}^{(l)}(I)$ is a refinement of ${\mathcal{P}}^{(m)}(I)$.

For any $J\subset I$, let $\unicode[STIX]{x1D70B}_{J}=Z(\sum _{j\in J}\{\unicode[STIX]{x1D70C}_{j}\})\in \operatorname{Irr}_{\operatorname{gen}}$ and $\unicode[STIX]{x1D70E}_{J}=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m)$. Then $\unicode[STIX]{x1D70E}_{J}$ is an ($m$-homogeneous) ladder representation for any $J\in {\mathcal{P}}^{(m)}(I)$. It follows from the defining property of ${\mathcal{P}}^{(m)}(I)$, (1) and Lemma 2.1 that 

 is irreducible, hence equals $\unicode[STIX]{x1D70E}$. Likewise, for any $l\leqslant m$, we have 

 Since we may assume that 

 for all $J\in {\mathcal{P}}^{(m)}(I)$, we infer that

In particular, $\unicode[STIX]{x1D70B}=\times _{J\in {\mathcal{P}}^{(m)}(I)}\unicode[STIX]{x1D70B}_{J}$.

By [Reference Kret and LapidKL12], we have

$$\begin{eqnarray}\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m-1)|\cdot |^{-1/2}\otimes \unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}\,{\leqslant}_{\operatorname{unq}}J_{((m-1)n_{J},n_{J})}(\unicode[STIX]{x1D70E}_{J})\end{eqnarray}$$

for all $J\in {\mathcal{P}}^{(m)}(I)$ where $n_{J}=\deg \unicode[STIX]{x1D70B}_{J}$. Therefore, by the geometric lemma of Bernstein and Zelevinsky [Reference Bernshtein and ZelevinskyBZ77],

$$\begin{eqnarray}\displaystyle \operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\otimes \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2} & = & \displaystyle \times _{J\in {\mathcal{P}}^{(m)}(I)}\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m-1)|\cdot |^{-1/2}\otimes \times _{J\in {\mathcal{P}}^{(m)}(I)}\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}\nonumber\\ \displaystyle & {\leqslant} & \displaystyle J_{Q}(\unicode[STIX]{x1D70E}).\nonumber\end{eqnarray}$$

On the other hand, suppose that $\unicode[STIX]{x1D70F}_{J},\unicode[STIX]{x1D714}_{J}\in \operatorname{Irr}$ with $\unicode[STIX]{x1D70F}_{J}\otimes \unicode[STIX]{x1D714}_{J}\leqslant J_{P}(\unicode[STIX]{x1D70E}_{J})$, $J\in {\mathcal{P}}^{(m)}(I)$ and

(4)$$\begin{eqnarray}\mathop{\sum }_{J\in {\mathcal{P}}^{(m)}(I)}\mathfrak{c}(\unicode[STIX]{x1D714}_{J})=\mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}.\end{eqnarray}$$

We claim that this is possible only if $\unicode[STIX]{x1D70F}_{J}=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m-1)|\cdot |^{-1/2}$ and $\unicode[STIX]{x1D714}_{J}=\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}$ for all $J$. We prove it by induction on $\deg \unicode[STIX]{x1D70B}$ using the geometric lemma. The base of the induction is trivial. For the induction step, it is enough to prove that if $J$ is a maximal $m$-chain, then $\unicode[STIX]{x1D714}_{J}=\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}$. We use Lemma 2.2. By part (ii), if $J$ is a maximal $m$-chain, then $\unicode[STIX]{x1D714}_{J}$ is generic. For otherwise, since $\mathfrak{c}(\unicode[STIX]{x1D714}_{J})\leqslant \mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}$, there would exist $i\in I$ such that $\unicode[STIX]{x1D70C}_{i}\notin \{\unicode[STIX]{x1D70C}_{j}:j\in J\}$ but $\unicode[STIX]{x1D70C}_{i}|\cdot |\in \{\unicode[STIX]{x1D70C}_{j}:j\in J\}$ in contradiction to the maximality of $J$. On the other hand, by part (iv), if $\unicode[STIX]{x1D70C}|\cdot |\not \leqslant \mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}$, then $\unicode[STIX]{x1D70C}$ can occur in $\mathfrak{c}(\unicode[STIX]{x1D714}_{J})$ at most once for any $J\in {\mathcal{P}}^{(m)}(I)$. It follows from (4) that if $\unicode[STIX]{x1D70C}|\cdot |\not \leqslant \mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}$, then $\unicode[STIX]{x1D70C}\leqslant \mathfrak{c}(\unicode[STIX]{x1D714}_{J})$ if and only if $\unicode[STIX]{x1D70C}=\unicode[STIX]{x1D70C}_{j}|\cdot |^{(m-1)/2}$ for some $j\in J$. By part (iii), it now follows that if $J$ is a maximal $m$-chain, then $\mathfrak{c}(\unicode[STIX]{x1D714}_{J})=\sum _{j\in J}\{\unicode[STIX]{x1D70C}_{j}|\cdot |^{(m-1)/2}\}$ and hence $\unicode[STIX]{x1D714}_{J}=\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}$ (since $\unicode[STIX]{x1D714}_{J}$ is generic) as required.

This concludes the proof of the proposition. ◻

Remark 2.9. It can be shown that up to permutation, $\unicode[STIX]{x1D70E}_{1},\ldots ,\unicode[STIX]{x1D70E}_{t}$ are the unique ladder representations such that $\unicode[STIX]{x1D70E}=\unicode[STIX]{x1D70E}_{1}\times \cdots \times \unicode[STIX]{x1D70E}_{t}$. We will not need to use this fact.

By Frobenius reciprocity and [Reference Lapid and MínguezLM16, Corollary 4.10], we have the following corollary.

Corollary 2.10. For any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$,

$$\begin{eqnarray}\operatorname{Sp}(\unicode[STIX]{x1D70B},m)=\operatorname{soc}(\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2})\,{\leqslant}_{\operatorname{ unq}}\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}.\end{eqnarray}$$

By induction on $m$, we get the following result.

Corollary 2.11. For any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$, $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is a subrepresentation of

$$\begin{eqnarray}\unicode[STIX]{x1D6F1}:=\unicode[STIX]{x1D70B}|\cdot |^{(1-m)/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(3-m)/2}\times \cdots \times \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}.\end{eqnarray}$$

Equivalently (by passing to the contragredient), $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is a quotient of

$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D6F1}}:=\unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(m-3)/2}\times \cdots \times \unicode[STIX]{x1D70B}|\cdot |^{(1-m)/2}.\end{eqnarray}$$

Remark 2.12. If $\unicode[STIX]{x1D70B}$ is (AT), then $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is the Langlands quotient of $\tilde{\unicode[STIX]{x1D6F1}}$, In particular, in this case $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is the image of the standard intertwining operator from $\tilde{\unicode[STIX]{x1D6F1}}$ to $\unicode[STIX]{x1D6F1}$ and $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)=\operatorname{soc}(\unicode[STIX]{x1D6F1})\,{\leqslant}_{\operatorname{unq}}\,\unicode[STIX]{x1D6F1}$. However, in general for $m>2$ and $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$ it is not true that $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)\,{\leqslant}_{\operatorname{unq}}\,\unicode[STIX]{x1D6F1}$. For instance, if $\unicode[STIX]{x1D70B}=|\cdot |\times |\cdot |^{-1}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{2}$, then $\operatorname{Sp}(\unicode[STIX]{x1D70B},3)$ occurs with multiplicity two in the Jordan–Hölder sequence of $\unicode[STIX]{x1D70B}|\cdot |^{-1}\times \unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70B}|\cdot |$. Note that in this case we still have $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)=\operatorname{soc}(\unicode[STIX]{x1D6F1})$ but we do not know whether this holds in general, that is, whether $\operatorname{soc}(\unicode[STIX]{x1D6F1})$ is always irreducible.

3 The models

3.1 Definition of models

Throughout this section, fix $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$, let $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)\in \operatorname{Irr}G$ and let $P=P_{\unicode[STIX]{x1D70E}}=P_{m,n}=M\ltimes U$ be the standard parabolic subgroup of $G$ of type $\overbrace{(n,\ldots ,n)}^{m}$. Let $\bar{U}=\,^{t}U$ be the opposite of $U$. Fix a non-trivial character $\unicode[STIX]{x1D713}$ of $F$. Let $\unicode[STIX]{x1D6F9}$ be the function on $G$ given by

$$\begin{eqnarray}\unicode[STIX]{x1D6F9}(g)=\unicode[STIX]{x1D713}\biggl(\mathop{\sum }_{1\leqslant i<nm:n\nmid i}g_{i,i+1}\biggr).\end{eqnarray}$$

We denote the restriction of $\unicode[STIX]{x1D6F9}$ to a subset $A$ of $G$ by $\unicode[STIX]{x1D6F9}_{A}$. Let $N=N_{mn}$ (respectively, $\bar{N}=\,^{t}N$) be the group of upper (respectively, lower) unitriangular matrices in $G$. Then $\unicode[STIX]{x1D6F9}_{N}$ is a (degenerate) character on $N$ that is trivial on $U$ and non-degenerate on $N_{M}:=N\cap M$. Recall that by (3), $\operatorname{Hom}_{G}(\unicode[STIX]{x1D70E},\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D6F9}_{N})$ is one-dimensional.

Denote by ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$ the image of $\unicode[STIX]{x1D70E}$ in $\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D6F9}_{N}$, that is, the Zelevinsky model of $\unicode[STIX]{x1D70E}$. By Corollaries 2.10 and 2.11, for any $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$, we have

(5a)$$\begin{eqnarray}(|\det |^{(1-n)/2}\otimes |\det |^{(m-1)(n-1)/2})W_{\operatorname{ Ze}}|_{\operatorname{GL}_{(m-1)n}\times \operatorname{GL}_{n}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N\cap (\operatorname{GL}_{(m-1)n}\times \operatorname{GL}_{n})}}(\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)\otimes \unicode[STIX]{x1D70B}),\end{eqnarray}$$
(5b)$$\begin{eqnarray}\unicode[STIX]{x1D6FF}_{P}^{-1/2}\unicode[STIX]{x1D6FF}^{\prime }W_{\operatorname{ Ze}}|_{M}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{M}}}(\unicode[STIX]{x1D70B}^{\otimes m}),\end{eqnarray}$$
where $\unicode[STIX]{x1D70B}^{\otimes m}=\overbrace{\unicode[STIX]{x1D70B}\otimes \cdots \otimes \unicode[STIX]{x1D70B}}^{m}$, $\unicode[STIX]{x1D6FF}_{P}$ is the modulus character of $P$ and $\unicode[STIX]{x1D6FF}^{\prime }=\unicode[STIX]{x1D6FF}_{P}^{1/2n}$ is the character of $M$ given by

$$\begin{eqnarray}\unicode[STIX]{x1D6FF}^{\prime }(\operatorname{diag}(g_{1},\ldots ,g_{m}))=|\det g_{1}|^{(m-1)/2}|\det g_{2}|^{(m-3)/2}\cdots |\det g_{m}|^{(1-m)/2}.\end{eqnarray}$$

The model ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$ is a particular case of more general models considered in [Reference Mœglin and WaldspurgerMW87] (for any reductive group). Let us recall the setup. Let $\mathfrak{g}=\operatorname{Mat}_{nm,nm}$ be the Lie algebra of $G$ over $F$. For any cocharacter $\unicode[STIX]{x1D711}$ of the diagonal torus $T$, let $\mathfrak{g}=\bigoplus _{j\in \mathbb{Z}}\mathfrak{g}_{j}^{\unicode[STIX]{x1D711}}$ be the corresponding grading

$$\begin{eqnarray}\mathfrak{g}_{j}^{\unicode[STIX]{x1D711}}=\{X\in \mathfrak{g}:\operatorname{Ad}(\unicode[STIX]{x1D711}(s))X=s^{j}X\}\end{eqnarray}$$

and let $\mathfrak{g}_{{\geqslant}j}^{\unicode[STIX]{x1D711}}=\bigoplus _{k\geqslant j}\mathfrak{g}_{k}^{\unicode[STIX]{x1D711}}$, $j\in \mathbb{Z}$, be the corresponding filtration. Let $P_{\unicode[STIX]{x1D711}}$ be the semistandard parabolic subgroup such that $\operatorname{Lie}P_{\unicode[STIX]{x1D711}}=\mathfrak{g}_{{\geqslant}0}^{\unicode[STIX]{x1D711}}$. Then $P_{\unicode[STIX]{x1D711}}=M_{\unicode[STIX]{x1D711}}\ltimes U_{\unicode[STIX]{x1D711}}$ where $M_{\unicode[STIX]{x1D711}}$ is the centralizer of $\unicode[STIX]{x1D711}$, $\operatorname{Lie}M_{\unicode[STIX]{x1D711}}=\mathfrak{g}_{0}^{\unicode[STIX]{x1D711}}$ and $\operatorname{Lie}U_{\unicode[STIX]{x1D711}}=\mathfrak{g}_{{>}0}^{\unicode[STIX]{x1D711}}$. Concretely, if $\unicode[STIX]{x1D711}(s)=\operatorname{diag}(s^{\unicode[STIX]{x1D706}_{1}^{\unicode[STIX]{x1D711}}},\ldots ,s^{\unicode[STIX]{x1D706}_{mn}^{\unicode[STIX]{x1D711}}})$ where $(\unicode[STIX]{x1D706}_{1}^{\unicode[STIX]{x1D711}},\ldots ,\unicode[STIX]{x1D706}_{mn}^{\unicode[STIX]{x1D711}})\in \mathbb{Z}^{mn}$, then

$$\begin{eqnarray}\displaystyle P_{\unicode[STIX]{x1D711}} & = & \displaystyle \{g\in G:g_{i,j}=0\text{ if }\unicode[STIX]{x1D706}_{i}<\unicode[STIX]{x1D706}_{j}\},\nonumber\\ \displaystyle M_{\unicode[STIX]{x1D711}} & = & \displaystyle \{g\in G:g_{i,j}=0\text{ if }\unicode[STIX]{x1D706}_{i}\neq \unicode[STIX]{x1D706}_{j}\},\nonumber\\ \displaystyle U_{\unicode[STIX]{x1D711}} & = & \displaystyle \{g\in G:g_{i,j}=\unicode[STIX]{x1D6FF}_{i,j}\text{ if }\unicode[STIX]{x1D706}_{i}\leqslant \unicode[STIX]{x1D706}_{j}\}.\nonumber\end{eqnarray}$$

Consider the nilpotent $nm\times nm$ matrix $J_{m,n}$ consisting of $m$ lower triangular Jordan blocks of size $n\times n$ each. We say that $\unicode[STIX]{x1D711}$ is of type $(m,n)$ if $\operatorname{Ad}(\unicode[STIX]{x1D711}(s))J_{m,n}=s^{-1}J_{m,n}$, or equivalently, if $\unicode[STIX]{x1D706}_{i}^{\unicode[STIX]{x1D711}}-\unicode[STIX]{x1D706}_{i+1}^{\unicode[STIX]{x1D711}}=1$ for all $i$ not dividing $n$. If $\unicode[STIX]{x1D711}$ is of type $(m,n)$, then $\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}$ is a character of $U_{\unicode[STIX]{x1D711}}$. By [Reference Mœglin and WaldspurgerMW87] (in particular, § II.2) we obtain the following theorem.Footnote 3

Theorem 3.1 [Reference Mœglin and WaldspurgerMW87].

Suppose that $\unicode[STIX]{x1D711}$ is of type $(m,n)$. Then the space

$$\begin{eqnarray}\operatorname{Hom}_{U_{\unicode[STIX]{x1D711}}}(\unicode[STIX]{x1D70E},\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}})=\operatorname{Hom}_{G}(\unicode[STIX]{x1D70E},\operatorname{Ind}_{U_{\unicode[STIX]{x1D711}}}^{G}\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}})\end{eqnarray}$$

is one-dimensional.

In the setting of [Reference Mœglin and WaldspurgerMW87] the data pertaining to Theorem 3.1 is the pair $(\unicode[STIX]{x1D711}^{2},J_{m,n})$. (The more general context of [Reference Mœglin and WaldspurgerMW87] applies to cocharacters which are not necessarily even. However, we will not discuss them here.)

We denote by ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}}(\unicode[STIX]{x1D70E})$ the image of $\unicode[STIX]{x1D70E}$ in $\operatorname{Ind}_{U_{\unicode[STIX]{x1D711}}}^{G}\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}$. It consists of functions that are left equivariant (with respect to some character) under the centralizer of $\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}$ in $P_{\unicode[STIX]{x1D711}}$.

Clearly, any $\unicode[STIX]{x1D711}$ of type $(m,n)$ is determined by the $m$-tuple $(\unicode[STIX]{x1D706}_{n}^{\unicode[STIX]{x1D711}},\unicode[STIX]{x1D706}_{2n}^{\unicode[STIX]{x1D711}},\ldots ,\unicode[STIX]{x1D706}_{mn}^{\unicode[STIX]{x1D711}})$. We consider $(m-1)n+1$ cocharacters $\unicode[STIX]{x1D711}_{0},\ldots ,\unicode[STIX]{x1D711}_{(m-1)n}$ of $T$ of type $(m,n)$ such that $\unicode[STIX]{x1D706}_{nk}^{\unicode[STIX]{x1D711}_{i}}-\unicode[STIX]{x1D706}_{n(k+1)}^{\unicode[STIX]{x1D711}_{i}}=\max (0,nk-i)$, $k=1,\ldots ,m-1$. (Up to a cocharacter of the center of $G$, the cocharacter $\unicode[STIX]{x1D711}_{(m-1)n}^{2}$ corresponds to the $\operatorname{SL}_{2}$-triple pertaining to $J_{m,n}$.) For simplicity we write $P_{i}=P_{\unicode[STIX]{x1D711}_{i}}$, $M_{i}=M_{\unicode[STIX]{x1D711}_{i}}$, $U_{i}=U_{\unicode[STIX]{x1D711}_{i}}$. If $i=nd+r$ where $d=\lfloor i/n\rfloor$ and $0\leqslant r<n$, then $U_{i}$ consists of the matrices whose $n\times n$ blocks $A_{j,k}$ satisfy:

  1. (i) $A_{j,j}$ is upper unitriangular for all $j=1,\ldots ,m$;

  2. (ii) $A_{j,k}$ is strictly upper triangular if $j\neq k$ and $j,k\leqslant d+1$;

  3. (iii) for any $k<d+2$, $(A_{d+2,k})_{a,b}=0$ if $b-a\leqslant n-r$ and $(A_{k,d+2})_{a,b}=0$ if $a-b\geqslant n-r$;

  4. (iv) $A_{j,k}=0$ if $j>k$ and $j>d+2$.

(There is no constraint on $A_{j,k}$ if $j<k$ and $d+2<k$.)

In particular, $U_{0}=N$ while $U_{(m-1)n}$ consists of the matrices whose difference from the identity matrix is strictly upper triangular in each $n\times n$ block. Also, $U_{i+1}\cap N\subset U_{i}\cap N$ and $U_{i}\cap \bar{N}\subset U_{i+1}\cap \bar{N}$ for all $i$.

For brevity we write $P^{\prime }=P_{(m-1)n}$, $M^{\prime }=M_{(m-1)n}\simeq \overbrace{\operatorname{GL}_{m}\times \cdots \times \operatorname{GL}_{m}}^{n}$, $U^{\prime }=U_{(m-1)n}$. In analogy with the case $m=2$ we will refer to ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$ as the Shalika model of $\unicode[STIX]{x1D70E}$. We caution, however, that in the literature, this terminology usually refers to the image of $\unicode[STIX]{x1D70F}\in \operatorname{Irr}\operatorname{GL}_{2n}$ (possibly generic) in $\operatorname{Ind}_{S}^{\operatorname{GL}_{2n}}\unicode[STIX]{x1D713}_{S}$ under a non-trivial intertwining operator, if it exists (in which case it is unique up to a scalar [Reference Jacquet and RallisJR96]), where $S$ is the Shalika group

$$\begin{eqnarray}S=\big\{\!\left(\begin{smallmatrix}g & \\ & g\end{smallmatrix}\right)\left(\begin{smallmatrix}I_{n} & X\\ & I_{n}\end{smallmatrix}\right):g\in \operatorname{GL}_{n},X\in \operatorname{Mat}_{n,n}\!\big\}\end{eqnarray}$$

and $\unicode[STIX]{x1D713}_{S}$ is the character on $S$ given by $\unicode[STIX]{x1D713}(\operatorname{tr}X)$. In the case at hand, any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$ automatically satisfies an equivariance property under the centralizer of $\unicode[STIX]{x1D6F9}_{U^{\prime }}$ in $P^{\prime }$ (which is conjugate to $S$ in the case $m=2$), which justifies our terminology. In general, even for $m=2$, $\operatorname{Hom}_{U^{\prime }}(\unicode[STIX]{x1D70F},\unicode[STIX]{x1D6F9}_{U^{\prime }})$ is infinite-dimensional for $\unicode[STIX]{x1D70F}\in \operatorname{Irr}G$.

Letting $G$ act on right on the vector space $F^{mn}$ of row vectors with standard basis $e_{1},\ldots ,e_{mn}$, $P^{\prime }$ is the stabilizer of the flag

$$\begin{eqnarray}(\operatorname{span}\{e_{nj-k}:j=1,\ldots ,m,~k=0,\ldots ,i-1\})_{i=0,\ldots ,n}.\end{eqnarray}$$

We denote by $\unicode[STIX]{x1D705}:\overbrace{\operatorname{GL}_{m}\times \cdots \times \operatorname{GL}_{m}}^{n}\rightarrow M^{\prime }$ the isomorphism such that the $i$th copy of $\operatorname{GL}_{m}$ acts on $\operatorname{span}\{e_{nj+i}:j=0,\ldots ,m-1\}$.

If $X$ is a matrix over $F$, then we write $\Vert X\Vert$ for the maximum of the absolute value of its entries.

Lemma 3.2. Suppose that $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$. Then there exists $C>0$ with the following property. Suppose that $g\in G$ with $W_{\operatorname{Sh}}(g)\neq 0$. Write $g=u^{\prime }l^{\prime }k$ where $u^{\prime }\in U^{\prime }$, $l^{\prime }=\unicode[STIX]{x1D705}(g_{1},\ldots ,g_{n})\in M^{\prime }$ and $k\in G({\mathcal{O}})$. Then $\Vert g_{i+1}^{-1}g_{i}\Vert \leqslant C$ for all $i<n$.

Proof. It is enough to consider the case $g=\unicode[STIX]{x1D705}(g_{1},\ldots ,g_{n})\in M^{\prime }$. Assume that $W_{\operatorname{Sh}}(g)\neq 0$. Fix $1\leqslant i<n$. For any $X\in \operatorname{Mat}_{m,m}(F)$, let $Y\in U^{\prime }$ be the matrix such that $Y_{nj+i,nk+i+1}=X_{j+1,k+1}$ for all $0\leqslant j,k<m$ and all other non-diagonal entries of $Y$ are zero. Then $W_{\operatorname{Sh}}(gY)=\unicode[STIX]{x1D713}(\operatorname{tr}g_{i}Xg_{i+1}^{-1})W_{\operatorname{Sh}}(g)$. It follows that there exists $C_{1}>0$ depending only on $W_{\operatorname{Sh}}$ such that if $W_{\operatorname{Sh}}(g)\not =0$, then $\unicode[STIX]{x1D713}(\operatorname{tr}g_{i}Xg_{i+1}^{-1})=\unicode[STIX]{x1D713}(\operatorname{tr}g_{i+1}^{-1}g_{i}X)=1$ for all $X\in \operatorname{Mat}_{m,m}(F)$ with $\Vert X\Vert \leqslant C_{1}$. The lemma follows.◻

3.2 Model transition part I

We denote by $M_{i}^{\unicode[STIX]{x1D6F9}}$ (respectively, $P_{i}^{\unicode[STIX]{x1D6F9}}=M_{i}^{\unicode[STIX]{x1D6F9}}\ltimes U_{i}$) the stabilizer of $\unicode[STIX]{x1D6F9}_{U_{i}}$ in $M_{i}$ (respectively, $P_{i}$). (Note that $M_{i}$ determines $i$ so this notation is unambiguous.) We also write ${M^{\prime }}^{\unicode[STIX]{x1D6F9}}=M_{(m-1)n}^{\unicode[STIX]{x1D6F9}}$ and ${P^{\prime }}^{\unicode[STIX]{x1D6F9}}=P_{(m-1)n}^{\unicode[STIX]{x1D6F9}}={M^{\prime }}^{\unicode[STIX]{x1D6F9}}\ltimes U^{\prime }$. Note that ${P^{\prime }}^{\unicode[STIX]{x1D6F9}}$ is unimodular. Explicitly, ${M^{\prime }}^{\unicode[STIX]{x1D6F9}}$ is the image under $\unicode[STIX]{x1D705}$ of $\operatorname{GL}_{m}$ diagonally embedded in $\overbrace{\operatorname{GL}_{m}\times \cdots \times \operatorname{GL}_{m}}^{n}$. It consists of the matrices in $G$ whose $n\times n$ blocks are all scalar matrices. Let $\unicode[STIX]{x1D704}:\operatorname{GL}_{m}\rightarrow {M^{\prime }}^{\unicode[STIX]{x1D6F9}}$ be the resulting identification.

In general, write $i=nd+r$, $0\leqslant r<n$. Then the reductive part of $M_{i}^{\unicode[STIX]{x1D6F9}}$ is the image under $\unicode[STIX]{x1D704}$ of the subgroup

$$\begin{eqnarray}\{\operatorname{diag}(l,t_{d+2},\ldots ,t_{m}):l\in \operatorname{GL}_{d+1},t_{d+2},\ldots ,t_{m}\in F^{\ast }\}.\end{eqnarray}$$

The unipotent radical of $M_{i}^{\unicode[STIX]{x1D6F9}}$ consists of the matrices whose $n\times n$ blocks $A_{j,k}$ satisfy the following requirements.

  1. (i) $A_{j,j}=I_{n}$ for all $j$.

  2. (ii) If $j\neq k$, then $A_{j,k}=0$ unless $k=d+2$ and $j<k$, in which case $(A_{j,k})_{a,b}=0$ unless $a-b=n-r$. Moreover, all the entries of $A_{j,k}$ on the diagonal $a-b=n-r$ coincide.

This group is trivial if $i$ is divisible by $n$, and is of dimension $d+1$ otherwise.

Lemma 3.3. Let $0\leqslant i<(m-1)n$. Then the following statements hold.

  1. (i) The commutator $[U_{i},U_{i+1}]$ is contained in $U_{i}\cap U_{i+1}$. Thus, $U_{i}\cdot U_{i+1}$ is a subgroup of $G$ which contains $U_{i}$ and $U_{i+1}$ as normal subgroups and the quotients $U_{i}U_{i+1}/U_{i}\simeq U_{i+1}/U_{i}\cap U_{i+1}$ and $U_{i}U_{i+1}/U_{i+1}\simeq U_{i}/U_{i}\cap U_{i+1}$ are abelian. Moreover,

    (6)$$\begin{eqnarray}U_{i+1}=(M_{i}\cap U_{i+1})\ltimes (U_{i}\cap U_{i+1})\quad \text{and}\quad U_{i}=(M_{i+1}\cap U_{i})\ltimes (U_{i}\cap U_{i+1}).\end{eqnarray}$$

  2. (ii) We have a short exact sequence

    where $\mathfrak{c}_{i}$ denotes the map $u\mapsto \unicode[STIX]{x1D6F9}([\cdot ,u])$ and $PD$ denotes the Pontryagin dual. Dually,
    where $\mathfrak{c}_{i}^{\prime }$ is defined by the same formula as $\mathfrak{c}_{i}$.

Proof. For any $j,k$, we have $\unicode[STIX]{x1D706}_{j}^{\unicode[STIX]{x1D711}_{i}}-\unicode[STIX]{x1D706}_{k}^{\unicode[STIX]{x1D711}_{i}}-(\unicode[STIX]{x1D706}_{j}^{\unicode[STIX]{x1D711}_{i+1}}-\unicode[STIX]{x1D706}_{k}^{\unicode[STIX]{x1D711}_{i+1}})\in \{-1,0,1\}$. It follows that

(7)$$\begin{eqnarray}\mathfrak{g}_{{\geqslant}j+1}^{\unicode[STIX]{x1D711}_{i+1}}\subset \mathfrak{g}_{{\geqslant}j}^{\unicode[STIX]{x1D711}_{i}}\subset \mathfrak{g}_{{\geqslant}j-1}^{\unicode[STIX]{x1D711}_{i+1}}\quad \text{for all }j.\end{eqnarray}$$

Therefore, $U_{i}\subset P_{i+1}$ and $U_{i+1}\subset P_{i}$. Hence, $U_{i}$ and $U_{i+1}$ normalize each other, so that $U_{i}\cdot U_{i+1}$ is a subgroup of $G$ that contains $U_{i}$ and $U_{i+1}$ as normal subgroups. The equalities (6) are now clear since $T\subset M_{i},M_{i+1}$. By (7), we have $\operatorname{Lie}M_{i+1}\cap U_{i}=\mathfrak{g}_{{>}0}^{\unicode[STIX]{x1D711}_{i}}\cap \mathfrak{g}_{0}^{\unicode[STIX]{x1D711}_{i+1}}\subset \mathfrak{g}_{1}^{\unicode[STIX]{x1D711}_{i}}$. It follows that $M_{i+1}\cap U_{i}$ is abelian since $[\mathfrak{g}_{1}^{\unicode[STIX]{x1D711}_{i}},\mathfrak{g}_{1}^{\unicode[STIX]{x1D711}_{i}}]\subset \mathfrak{g}_{2}^{\unicode[STIX]{x1D711}_{i}}$. Similarly, $M_{i}\cap U_{i+1}$ is abelian. The rest of the lemma follows easily from the fact that $U_{i+1}\cap M_{i}^{\unicode[STIX]{x1D6F9}}=1$.◻

Remark 3.4. If $i=nd+r$, $0\leqslant r<n$, then $U_{i}\cap M_{i+1}^{\unicode[STIX]{x1D6F9}}$ consists of the upper unitriangular matrices whose $n\times n$ blocks $A_{j,k}$ satisfy the following requirements.

  1. (i) $A_{j,j}=I_{n}$ for all $j$.

  2. (ii) If $j<k$, then $A_{j,k}=0$ unless $k=d+2$, in which case $(A_{j,k})_{a,b}=0$ unless $a-b=n-r-1$ and all entries of $A_{j,k}$ along the diagonal $a-b=n-r-1$ are identical.

This group is of dimension $d+1$. (It coincides with the unipotent radical of $M_{i+1}^{\unicode[STIX]{x1D6F9}}$ unless $i+1$ is divisible by $n$.)

In the rest of the section we endow various unipotent subgroups of $G$ with Haar measures. Thanks to the choice of basis $e_{1},\ldots ,e_{mn}$, the Lie algebra of any of these unipotent groups has a natural basis as a vector space over $F$. Our convention will be to take the measure corresponding to the product measure where the Haar measure on $F$ is the one which is self-dual with respect to $\unicode[STIX]{x1D713}$.

The following is a special case of [Reference Gomez, Gourevitch and SahiGGS16] (see also [Reference Gomez, Gourevitch and SahiGGS17]). For future reference and in order to be self-contained we provide the (elementary) proof. We refer the reader to [Reference Gomez, Gourevitch and SahiGGS16, Reference Gomez, Gourevitch and SahiGGS17] for a more thorough discussion about interplay between models.

Proposition 3.5. For any $i=0,\ldots ,(m-1)n-1$, the map

(8)$$\begin{eqnarray}\displaystyle W_{i}\mapsto \int _{U_{i}\cap U_{i+1}\backslash U_{i+1}}W_{i}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U_{i+1}}(u^{\prime })^{-1}\,du^{\prime } & = & \displaystyle \int _{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}W_{i}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U_{i+1}}(u^{\prime })^{-1}\,du^{\prime }\nonumber\\ \displaystyle & = & \displaystyle \int _{U_{i}\cap \bar{N}\backslash U_{i+1}\cap \bar{N}}W_{i}(u^{\prime }\cdot )\,du^{\prime }\end{eqnarray}$$

defines an isomorphism ${\mathcal{T}}_{i}={\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}:{\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})\rightarrow {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i+1}}}(\unicode[STIX]{x1D70E})$. Its inverse is given by

(9)$$\begin{eqnarray}W_{i+1}\mapsto \int _{U_{i}\cap P_{i+1}^{\unicode[STIX]{x1D6F9}}\backslash U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du.\end{eqnarray}$$

In both cases the integrands are compactly supported.

Proof. For any $W_{i}\in \operatorname{Ind}_{U_{i}}^{G}\unicode[STIX]{x1D6F9}_{U_{i}}$, $u\in U_{i}$ and $u^{\prime }\in U_{i+1}$, we have $W_{i}(u^{\prime }u)=\mathfrak{c}_{i}(u)(u^{\prime })\unicode[STIX]{x1D6F9}_{U_{i}}(u)W_{i}(u^{\prime })$. It follows from Lemma 3.3 and the smoothness of $W_{i}$ that $W_{i}|_{U_{i+1}}$ is compactly supported modulo $U_{i}\cap U_{i+1}$ and that, for any $u\in U_{i}$,

(10)

Recall that any $W_{i+1}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i+1}}}(\unicode[STIX]{x1D70E})$ is left-$M_{i+1}^{\unicode[STIX]{x1D6F9}}$-equivariant under a character and, in particular, is left-invariant under any unipotent subgroup of $M_{i+1}^{\unicode[STIX]{x1D6F9}}$. Also, $U_{i}\cap P_{i+1}^{\unicode[STIX]{x1D6F9}}=(U_{i}\cap M_{i+1}^{\unicode[STIX]{x1D6F9}})\ltimes (U_{i}\cap U_{i+1})$. By similar reasoning as before, $W_{i+1}|_{U_{i}}$ is compactly supported modulo $U_{i}\cap P_{i+1}^{\unicode[STIX]{x1D6F9}}$. By Lemma 3.3 and Fourier inversion, the map (9) defines a $G$-equivariant left inverse to ${\mathcal{T}}_{i}$. Since the spaces are irreducible, it is also a right inverse.◻

Remark 3.6. Suppose that $\unicode[STIX]{x1D70E}$ is unramified, $\unicode[STIX]{x1D713}$ has conductor ${\mathcal{O}}$ and $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$ is the unramified vector such that $W_{i}(e)=1$. Then ${\mathcal{T}}_{i}W_{i}(e)=1$. This follows immediately from the proof of Proposition 3.5.

We write

$$\begin{eqnarray}{\mathcal{T}}={\mathcal{T}}^{\unicode[STIX]{x1D713}}={\mathcal{T}}_{(m-1)n-1}\circ \ldots \circ {\mathcal{T}}_{0}:{\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})\rightarrow {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E}).\end{eqnarray}$$

This operator was considered in [Reference Cai, Friedberg and KaplanCFK18, §2.4].

3.3 Model transition part II

We now introduce a subgroup of $G$ that will play an important role in what follows. Let $D=D_{m,n}$ be the joint stabilizer of the vectors $e_{jn}$, $j=1,\ldots ,m$, in $G$ and let $N_{D}=N\cap D\supset N_{M}$. Note that $U^{\prime }\subset D$. (In the case $m=1$, $D$ is the standard mirabolic subgroup.)

The following lemma is straightforward.

Lemma 3.7. We have $M_{i+1}\cap U_{i}=(M_{i+1}\cap U_{i}\cap D)\times (U_{i}\cap M_{i+1}^{\unicode[STIX]{x1D6F9}})$. Hence, the restriction of $\mathfrak{c}_{i}$ to $D\cap U_{i}/D\cap U_{i}\cap U_{i+1}\simeq D\cap M_{i+1}\cap U_{i}$ is an isomorphism. Dually, $\mathfrak{c}_{i}^{\prime }$ defines an isomorphism between $U_{i+1}/U_{i}\cap U_{i+1}\simeq M_{i}\cap U_{i+1}$ and the Pontryagin dual of $D\cap U_{i}/D\cap U_{i}\cap U_{i+1}\simeq D\cap M_{i+1}\cap U_{i}\simeq U_{i}\cap M_{i+1}/U_{i}\cap M_{i+1}^{\unicode[STIX]{x1D6F9}}$.

Hence, we can rewrite (9) as

$$\begin{eqnarray}W_{i+1}\mapsto \int _{D\cap U_{i}\cap U_{i+1}\backslash D\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du=\int _{N_{D}\cap U_{i+1}\backslash N_{D}\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du.\end{eqnarray}$$

Lemma 3.8. Any $W_{\operatorname{Ze}}\in \operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D6F9}_{N}$ is compactly supported on $D\cap \bar{N}$. Hence,

$$\begin{eqnarray}{\mathcal{T}}W_{\operatorname{Ze}}=\int _{U^{\prime }\cap N\backslash U^{\prime }}W_{\operatorname{Ze}}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U^{\prime }}(u^{\prime })^{-1}\,du^{\prime }=\int _{U^{\prime }\cap \bar{N}}W_{\operatorname{Ze}}(u^{\prime }\cdot )\,du^{\prime }=\int _{U^{\prime }\cap \bar{U}}W_{\operatorname{Ze}}(u^{\prime }\cdot )\,du^{\prime }\end{eqnarray}$$

where the integrand is compactly supported.

Proof. Let $g=ank\in G$ with $a=\operatorname{diag}(a_{1},\ldots ,a_{nm})$, $n\in N$ and $k\in G({\mathcal{O}})$. It is well known and easy to prove that if $g\in \bar{N}$, then $\Vert g\Vert \leqslant \max _{i=1,\ldots ,mn}|\prod _{j=i}^{nm}a_{j}|$. On the other hand, it is also easy to see that if $g\in D$, then $|a_{jn}|\leqslant 1$ for $j=1,\ldots ,m$. Thus, if $g\in D$ and $W_{\operatorname{Ze}}(g)\neq 0$, then by the support condition for Whittaker functions we get $|a_{i}|\leqslant C_{1}$ for all $i$ where $C_{1}$ depends only on $W_{\operatorname{Ze}}$. By the above, if moreover $g\in \bar{N}$, then $\Vert g\Vert$ is bounded in terms of $W_{\operatorname{Ze}}$ as required.◻

Recall that any $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$ is $M_{i}^{\unicode[STIX]{x1D6F9}}$-equivariant with respect to some character $\unicode[STIX]{x1D712}_{i}$ of $M_{i}^{\unicode[STIX]{x1D6F9}}$ (depending only on $\unicode[STIX]{x1D70E}$). As in [Reference Cai, Friedberg, Ginzburg and KaplanCFGK19, Reference Cai, Friedberg and KaplanCFK18] we can easily explicate this character.

Lemma 3.9 (Cf. [Reference Cai, Friedberg, Ginzburg and KaplanCFGK19, Proposition 24], [Reference Cai, Friedberg and KaplanCFK18, §2.6]).

For any $i=nd+r$, $0\leqslant r<n$, $l\in \operatorname{GL}_{d+1}$ and $t_{d+2},\ldots ,t_{m}\in F^{\ast }$, we have

$$\begin{eqnarray}\displaystyle & & \displaystyle \unicode[STIX]{x1D712}_{i}(\unicode[STIX]{x1D704}(\operatorname{diag}(l,t_{d+2},\ldots ,t_{m})))\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}(t_{d+2}\ldots t_{m}\det l)|\det l|^{-\binom{r}{2}+\binom{n}{2}(m-d-1)}|t_{d+2}|^{\binom{r}{2}(d+1)}\mathop{\prod }_{j=d+2}^{m}|t_{j}|^{n(n-1)((m+1)/2-j)}\nonumber\end{eqnarray}$$

where $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}$ is the central character of $\unicode[STIX]{x1D70B}$. In particular, for any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$,

$$\begin{eqnarray}W_{\operatorname{Sh}}(\unicode[STIX]{x1D704}(l)g)=\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}(\det l)W_{\operatorname{Sh}}(g)\quad \forall l\in \operatorname{GL}_{m},~g\in G.\end{eqnarray}$$

Proof. It is enough to evaluate $\unicode[STIX]{x1D712}_{i}$ on an element $\unicode[STIX]{x1D704}(t)$ where $t=\operatorname{diag}(t_{1},\ldots ,t_{m})$ is in the diagonal torus of $\operatorname{GL}_{m}$. Note that $\unicode[STIX]{x1D704}(t)$ lies in the center $Z_{M}$ of $M$. Writing $W_{i}=\int _{\bar{N}\cap U_{i}}W_{\operatorname{Ze}}(u\cdot )\,du$ with $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$ (the integrand is compactly supported by Lemma 3.8), the required relation follows from the equality

$$\begin{eqnarray}\displaystyle & & \displaystyle \unicode[STIX]{x1D6FF}_{P}^{1/2}{\unicode[STIX]{x1D6FF}^{\prime }}^{-1}(\unicode[STIX]{x1D704}(t))=\mathop{\prod }_{j=1}^{m}|t_{j}|^{n(n-1)((m+1)/2-j)}\nonumber\\ \displaystyle & & \displaystyle \quad =\bigg(\mathop{\prod }_{j=1}^{d+1}|t_{j}|^{-\binom{r}{2}+\binom{n}{2}(m-d-1)}\bigg)\bigg(\mathop{\prod }_{j=d+2}^{m}|t_{j}|^{n(n-1)((m+1)/2-j)}\bigg)|t_{d+2}|^{\binom{r}{2}(d+1)}\unicode[STIX]{x1D6FF}_{Z_{M}\ltimes (U_{i}\cap \bar{N})}(\unicode[STIX]{x1D704}(t))^{-1}.\nonumber\end{eqnarray}$$

Remark 3.10. Let   (alternating signs on the non-principal diagonal) and $\tilde{w}_{m,n}=\unicode[STIX]{x1D704}(\tilde{w}_{m})$. By Lemma 3.9, we have

$$\begin{eqnarray}W_{\operatorname{Sh}}(\tilde{w}_{m,n}g)=W_{\operatorname{Sh}}(g)\end{eqnarray}$$

for any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$.

Lemma 3.11. The inverse of ${\mathcal{T}}$ is given by

(11)$$\begin{eqnarray}W_{\operatorname{Sh}}\mapsto \int _{N\cap U^{\prime }\backslash N_{D}}W_{\operatorname{Sh}}(u\cdot )\unicode[STIX]{x1D6F9}_{N}(u)^{-1}\,du\end{eqnarray}$$

where the integrand is compactly supported.

Proof. From Proposition 3.5 we only need to check that the integrand is compactly supported. Assume that $W_{\operatorname{Sh}}={\mathcal{T}}W_{\operatorname{Ze}}$. By Remark 3.10, the integral equals

$$\begin{eqnarray}\int _{N\cap U^{\prime }\backslash N_{D}}W_{\operatorname{Sh}}(\tilde{w}_{m,n}u\cdot )\unicode[STIX]{x1D6F9}_{N}(u)^{-1}\,du=\int _{U\cap U^{\prime }\backslash U_{D}}\biggl(\int _{U^{\prime }\cap \bar{U}}W_{\operatorname{Ze}}(v\tilde{w}_{m,n}u\cdot )\unicode[STIX]{x1D6F9}_{U^{\prime }}(v)^{-1}\unicode[STIX]{x1D6F9}_{N}(u)^{-1}\,dv\biggr)du\end{eqnarray}$$

where $U_{D}=U\cap D$. The latter double integral is

$$\begin{eqnarray}\int _{\bar{U}\cap U^{\prime }\backslash \bar{U}_{D}}\biggl(\int _{U^{\prime }\cap \bar{U}}W_{\operatorname{Ze}}(v\bar{u}\tilde{w}_{m,n}\cdot )\unicode[STIX]{x1D6F9}_{U^{\prime }}(v)^{-1}\unicode[STIX]{x1D6F9}_{N}({\tilde{w}_{m,n}}^{-1}\bar{u}\tilde{w}_{m,n})^{-1}\,dv\biggr)\,d\bar{u}.\end{eqnarray}$$

By Lemma 3.8 the integrand is compactly supported in $v$, $\bar{u}$. Thus, the integrand on the right-hand side of (11) is compactly supported.◻

3.4 Kirillov–Shalika model

The following lemma is an analogue of [Reference Gel’fand and KajdanGK75, Proposition 2].

Lemma 3.12. Any non-zero $D$-invariant subspace of $\operatorname{Ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$ contains $\operatorname{ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$. In particular, $\operatorname{ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$ is irreducible.

The proof of [Reference Gel’fand and KajdanGK75, pp. 110–111] (for the case $m=1$) applies verbatim. One only needs to observe that the unipotent radical $V$ of $D$ is abelian, the stabilizer of the character $\unicode[STIX]{x1D6F9}_{V}$ under the action of $D$ modulo $V$ is isomorphic to $D_{m,n-1}$ and the map $p\mapsto \unicode[STIX]{x1D6F9}_{V}(p^{-1}\cdot p)$ defines an open map from $D$ to the Pontryagin dual of $V$.

Let $Q$ be the stabilizer of $\operatorname{span}\{e_{ni}:i=1,\ldots ,m\}$ in $G$ – a maximal parabolic subgroup of $G$ of type $((n-1)m,m)$. Thus, $Q=D\rtimes {M^{\prime }}^{\unicode[STIX]{x1D6F9}}$ and $\unicode[STIX]{x1D6FF}_{Q}|_{{M^{\prime }}^{\unicode[STIX]{x1D6F9}}}\equiv 1$.

Corollary 3.13. For any $m$-homogeneous $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$, the image of the restriction map

(12)$$\begin{eqnarray}W_{\operatorname{Sh}}\mapsto W_{\operatorname{Sh}}|_{D},\quad {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})\rightarrow \operatorname{Ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}\end{eqnarray}$$

contains $\operatorname{ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$. Equivalently, by Lemmas 2.3 and 3.9, if $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$, then the image ${\mathcal{K}}^{\unicode[STIX]{x1D713}}(\unicode[STIX]{x1D70E})$ of the restriction map

$$\begin{eqnarray}W_{\operatorname{Sh}}\mapsto W_{\operatorname{Sh}}|_{Q},\quad {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})\rightarrow \operatorname{Ind}_{{P^{\prime }}^{\unicode[STIX]{x1D6F9}}}^{Q}\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}\end{eqnarray}$$

contains $\operatorname{ind}_{{P^{\prime }}^{\unicode[STIX]{x1D6F9}}}^{Q}\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}$ where $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}$ is the character of ${P^{\prime }}^{\unicode[STIX]{x1D6F9}}$ such that $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}|_{U^{\prime }}=\unicode[STIX]{x1D6F9}_{U^{\prime }}$ and $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}\circ \unicode[STIX]{x1D704}=\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}\circ \det$.

We will call ${\mathcal{K}}^{\unicode[STIX]{x1D713}}(\unicode[STIX]{x1D70E})$ the Kirillov–Shalika model of $\unicode[STIX]{x1D70E}$.

Lemma 3.14. For any $i=0,\ldots ,(m-1)n-1$, the map

$$\begin{eqnarray}\tilde{{\mathcal{T}}}_{i}:W_{i}\mapsto \int _{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}W_{i}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U_{i+1}}(u^{\prime })^{-1}\,du^{\prime }=\int _{U_{i}\cap \bar{N}\backslash U_{i+1}\cap \bar{N}}W_{i}(u^{\prime }\cdot )\,du^{\prime }\end{eqnarray}$$

is an isomorphism between $\operatorname{Ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$ and $\operatorname{Ind}_{D\cap U_{i+1}}^{D}\unicode[STIX]{x1D6F9}_{U_{i+1}}$, whose inverse is given by

$$\begin{eqnarray}W_{i+1}\mapsto \int _{D\cap U_{i}\cap U_{i+1}\backslash D\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du=\int _{N_{D}\cap U_{i+1}\backslash N_{D}\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du.\end{eqnarray}$$

Moreover,

$$\begin{eqnarray}\tilde{{\mathcal{T}}}_{i}(\operatorname{ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}})=\operatorname{ind}_{D\cap U_{i+1}}^{D}\unicode[STIX]{x1D6F9}_{U_{i+1}}.\end{eqnarray}$$

Finally, the $D$-module $\operatorname{ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$ is irreducible.

Proof. Let $W_{i}\in \operatorname{Ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$. As in the proof of Proposition 3.5, by Lemma 3.7 the function

$$\begin{eqnarray}u\in D\cap U_{i}\cap U_{i+1}\backslash D\cap U_{i}\mapsto \unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}{\mathcal{T}}_{i}W_{i}(u)\end{eqnarray}$$

is the Fourier transform of the function $W_{i}\unicode[STIX]{x1D6F9}_{U_{i+1}}^{-1}|_{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}$ at $\mathfrak{c}_{i}(u)$. The first claim follows by Fourier inversion.

Suppose that $W_{i}\in \operatorname{ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$. From the definition (and since $U^{\prime }\subset D$), $\tilde{{\mathcal{T}}}_{i}W_{i}$ is supported on $(D\cap U_{i}\cdot U_{i+1})\unicode[STIX]{x1D6FA}$, where $\unicode[STIX]{x1D6FA}$ is a compact subset of $D$. Fix $g\in \unicode[STIX]{x1D6FA}$. It follows from the above that the function $\tilde{{\mathcal{T}}}_{i}W_{i}(g)$ is compactly supported modulo $D\cap U_{i}\cap U_{i+1}$. Hence, $\tilde{{\mathcal{T}}}_{i}W_{i}$ is compactly supported modulo $D\cap U_{i+1}$.

The last part now follows from the fact that $\operatorname{ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$ is irreducible.◻

From Lemma 3.14, Proposition 3.5 and Corollary 3.13 we obtain the following result.

Corollary 3.15. For any $m$-homogeneous $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$ and for any $i=0,\ldots ,(m-1)n$, the image of the restriction map

(13)$$\begin{eqnarray}W_{i}\mapsto W_{i}|_{D},\quad {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})\rightarrow \operatorname{Ind}_{U_{i}\cap D}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}\end{eqnarray}$$

contains $\operatorname{ind}_{U_{i}\cap D}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$.

Once again, in analogy with the case $m=1$ (conjectured in [Reference Gel’fand and KajdanGK75], proved in [Reference Bernšten̆ and ZelevinskyBZ76, Reference Bernshtein and ZelevinskyBZ77]) it is natural to make the following conjecture.

Conjecture 3.16. For any $m$-homogeneous $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$, the restriction map (12) (or equivalently, (13)) is injective.

We will prove a special case in Corollary 4.4 below.

We do not know whether, in general, the restriction of $\unicode[STIX]{x1D70E}$ to $Q$ is of finite length. (See Proposition 7.1 for a very special case.) Recall that in the case $m=1$ this is known (for any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}$, not necessarily generic) using the theory of derivatives of Bernstein and Zelevinsky [Reference Bernšten̆ and ZelevinskyBZ76, Reference Bernshtein and ZelevinskyBZ77]. It would be very interesting to have an analogous theory for $m>1$.

4 Unitary structure

We take the unnormalized Tamagawa measure on $\operatorname{GL}_{r}$ with respect to $\unicode[STIX]{x1D713}$, that is, the Haar measure associated to the standard gauge form $(\bigwedge _{i,j=1,\ldots ,r}dg_{i,j})/(\det g^{r})$ on $\operatorname{GL}_{r}$ and the self-dual Haar measure on $F$ with respect to $\unicode[STIX]{x1D713}$. Following our convention on Haar measures for unipotent groups (see § 3.2), we obtain a (right) Haar measure on the $F$-points of any algebraic group whose reductive part is a product of general linear groups. This will cover all algebraic groups considered here.

Throughout this section let $\unicode[STIX]{x1D70B},\unicode[STIX]{x1D70B}^{\prime }\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$ and let $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ and $\unicode[STIX]{x1D70E}^{\prime }=\operatorname{Sp}(\unicode[STIX]{x1D70B}^{\prime },m)$. We will work with the models considered in the previous section.

For any $0\leqslant i\leqslant (m-1)n$ and $s\in \mathbb{C}$, we define a bilinear form on ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})\times {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$ by

$$\begin{eqnarray}{\mathcal{B}}_{i}(W_{i},W_{i}^{\prime },s)=\int _{D\cap U_{i}\backslash D}W_{i}(g)W_{i}^{\prime }(g)|\det g|^{s}\,dg\end{eqnarray}$$

(assuming convergence). In particular, for $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$, $W_{\operatorname{Ze}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$,

(14)$$\begin{eqnarray}{\mathcal{B}}_{0}(W_{\operatorname{Ze}},W_{\operatorname{Ze}}^{\prime },s)=\int _{N_{D}\backslash D}W_{\operatorname{Ze}}(g)W_{\operatorname{Ze}}^{\prime }(g)|\det g|^{s}\,dg,\end{eqnarray}$$

and for $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$, $W_{\operatorname{Sh}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$,

$$\begin{eqnarray}{\mathcal{B}}_{\operatorname{Sh}}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },s):={\mathcal{B}}_{(m-1)n}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },s)=\int _{U^{\prime }\backslash D}W_{\operatorname{Sh}}(g)W_{\operatorname{Sh}}^{\prime }(g)|\det g|^{s}\,dg.\end{eqnarray}$$

It follows from Lemma 3.2 that $|\det |$ is bounded above on the support of $W_{\operatorname{Sh}}|_{D}$. Hence,

(15)

A similar statement holds for any ${\mathcal{B}}_{i}$, although we will not use it.

We also write ${\mathcal{B}}_{i}(W_{i},W_{i}^{\prime })={\mathcal{B}}(W_{i},W_{i}^{\prime },0)$ assuming the latter is well defined (either as a convergent integral, or by analytic continuation), in which case it is $D$-invariant.

In general, we do not know whether ${\mathcal{B}}_{i}(\cdot ,\cdot )$ is always defined. (See § 6 and in particular Example 6.5 for further discussion.)

Proposition 4.1. The integral defining ${\mathcal{B}}_{i}(W_{i},W_{i}^{\prime },s)$ converges for $\operatorname{Re}s+\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\prime })+1>0$. Moreover, for all $0\leqslant i<(m-1)n$, $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$, $W_{i}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$, we have

(16)$$\begin{eqnarray}{\mathcal{B}}_{i+1}({\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}W_{i},{\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}^{-1}}W_{i}^{\prime },s)={\mathcal{B}}_{i}(W_{i},W_{i}^{\prime },s).\end{eqnarray}$$

Finally, there exist $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$ and $W_{i}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$ such that ${\mathcal{B}}_{i}(W_{i},W_{i}^{\prime },s)\equiv 1$ for all $s\in \mathbb{C}$.

Remark 4.2. In Proposition 6.2 below we prove that ${\mathcal{B}}_{i}(W_{i},W_{i}^{\prime },s)$ admits meromorphic continuation in $s$ to a rational function in $q^{s}$.

Proof. First note that the last statement follows from Corollary 3.15.

Next, we show the convergence of the integral defining ${\mathcal{B}}_{0}$. Upon twisting $\unicode[STIX]{x1D70B}$ and $\unicode[STIX]{x1D70B}^{\prime }$ by $|\cdot |^{(s+\mathbf{e}(\unicode[STIX]{x1D70B}^{\prime })-\mathbf{e}(\unicode[STIX]{x1D70B}))/2}$ and $|\cdot |^{(s+\mathbf{e}(\unicode[STIX]{x1D70B})-\mathbf{e}(\unicode[STIX]{x1D70B}^{\prime }))/2}$ respectively and using the inequality $|xy|\leqslant (|x|^{2}+|y^{2}|)/2$, we may assume without loss of generality that $\unicode[STIX]{x1D70B}^{\prime }=\overline{\unicode[STIX]{x1D70B}}$, $W_{\operatorname{Ze}}^{\prime }=\overline{W_{\operatorname{Ze}}}$ and $s=0$. Thus, we need to show the convergence of

$$\begin{eqnarray}\int _{N_{D}\backslash D}|W_{\operatorname{Ze}}(g)|^{2}\,dg\end{eqnarray}$$

provided that $\mathbf{e}(\unicode[STIX]{x1D70B})>-\frac{1}{2}$. In fact, we show a slightly stronger assertion, namely the convergence of

(17)$$\begin{eqnarray}\int _{D\backslash G}\int _{N_{D}\backslash D}|W_{\operatorname{Ze}}(lg)|^{2}dl\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{m}\,dg\end{eqnarray}$$

for any $0\leqslant \unicode[STIX]{x1D6F7}\in {\mathcal{S}}(\operatorname{Mat}_{m,nm}(F))$ where $\unicode[STIX]{x1D702}\in \operatorname{Mat}_{m,nm}(F)$ is the matrix whose $i$th row is $e_{ni}$, $i=1,\ldots ,m$. Note that the stabilizer of $\unicode[STIX]{x1D702}$ under the right $G$-action on $\operatorname{Mat}_{m,nm}(F)$ is $D$. Since the modulus character of $D$ is $|\det |^{m}$, (17) is formally well defined and can be rewritten as

(18)$$\begin{eqnarray}\displaystyle & & \displaystyle \int _{N_{D}\backslash G}|W_{\operatorname{Ze}}(g)|^{2}\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{m}\,dg\nonumber\\ \displaystyle & & \displaystyle \quad =\int _{P\backslash G}\int _{N\backslash P}|W_{\operatorname{Ze}}(lg)|^{2}\int _{N_{D}\backslash N}\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}ulg)\,du~|\det l|^{m}\unicode[STIX]{x1D6FF}_{P}(l)^{-1}\,dl~|\det g|^{m}\,dg\nonumber\\ \displaystyle & & \displaystyle \quad =\int _{P\backslash G}\int _{N_{M}\backslash M}|W_{\operatorname{Ze}}(lg)|^{2}\int _{U_{D}\backslash U}\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}ulg)\,du~|\det l|^{m}\unicode[STIX]{x1D6FF}_{P}(l)^{-1}\,dl~|\det g|^{m}\,dg.\end{eqnarray}$$

We may identify the vector space $\operatorname{Mat}_{m,nm}(F)$ with $\operatorname{Mat}_{m,m}(F^{n})$. Observe that, for any $l=\operatorname{diag}(g_{1},\ldots ,g_{m})\in M$, $g\in G$, we have

(19)$$\begin{eqnarray}|\det l|^{(m-1)/2}\int _{U_{D}\backslash U}\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}ulg)\,du=\tilde{\unicode[STIX]{x1D6F7}}_{g}(e_{n}g_{1},\ldots ,e_{n}g_{m})\unicode[STIX]{x1D6FF}^{\prime }(l)\end{eqnarray}$$

where $\tilde{\unicode[STIX]{x1D6F7}}_{g}\in {\mathcal{S}}((F^{n})^{m})$ is the function

(20)$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D6F7}}_{g}(v_{1},\ldots ,v_{m})=\int \unicode[STIX]{x1D6F7}(Xg)\,dX,\quad v_{1},\ldots ,v_{m}\in F^{n},\end{eqnarray}$$

where the integral is taken over the $n\binom{m}{2}$-dimensional affine space of upper triangular $F^{n}$-valued $m\times m$-matrices whose diagonal entries are $v_{1},\ldots ,v_{m}$. Thus, (18) is equal to

$$\begin{eqnarray}\int _{P\backslash G}\int _{(N_{n}\backslash \!\operatorname{GL}_{n})^{m}}|\unicode[STIX]{x1D6FF}_{P}^{-1/2}\unicode[STIX]{x1D6FF}^{\prime }(l)W_{\operatorname{ Ze}}(lg)|^{2}\tilde{\unicode[STIX]{x1D6F7}}_{g}(e_{n}g_{1},\ldots ,e_{n}g_{m})\mathop{\prod }_{i=1}^{m}|\det g_{i}|^{i}~dg_{1}\cdots dg_{m}~|\det g|^{m}\,dg\end{eqnarray}$$

where $l=\operatorname{diag}(g_{1},\ldots ,g_{m})\in M$. Thus, by (5b) the inner integral is a finite linear combination of products of Rankin–Selberg integrals for $\unicode[STIX]{x1D70B}\times \bar{\unicode[STIX]{x1D70B}}$ at $i$, $i=1,\ldots ,m$. The assumption that $\mathbf{e}(\unicode[STIX]{x1D70B})>-\frac{1}{2}$ guarantees that these Rankin–Selberg integrals converge. Since the outer integral is a finite sum, we obtain the convergence of (17).

Now let $0\leqslant i<(m-1)n$, $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$, $W_{i}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$. Recall that $W_{i}|_{U_{i+1}}$ (respectively, ${\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}W_{i}|_{U_{i}\cap D}$) is compactly supported modulo $U_{i}\cap U_{i+1}$ (respectively, $U_{i}\cap U_{i+1}\cap D$).

Moreover, by the unitarity of the Fourier transform and the argument of Proposition 3.5 (cf. (10)) we have

(21)$$\begin{eqnarray}\displaystyle \int _{D\cap U_{i+1}\backslash D\cap U_{i}U_{i+1}}{\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}W_{i}(u){\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}^{-1}}W_{i}^{\prime }(u)\,du & = & \displaystyle \int _{N_{D}\cap U_{i+1}\backslash N_{D}\cap U_{i}}{\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}W_{i}(u){\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}^{-1}}W_{i}^{\prime }(u)\,du\nonumber\\ \displaystyle & = & \displaystyle \int _{U_{i}\backslash U_{i}U_{i+1}}W_{i}(u)W_{i}^{\prime }(u)\,du\nonumber\\ \displaystyle & = & \displaystyle \int _{D\cap U_{i}\backslash D\cap U_{i}U_{i+1}}W_{i}(u)W_{i}^{\prime }(u)\,du\end{eqnarray}$$

where the integrals are absolutely convergent. (We can also write the integrals as

$$\begin{eqnarray}\int _{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}W_{i}(u)W_{i}^{\prime }(u)\,du=\int _{U_{i}\cap \bar{N}\backslash U_{i+1}\cap \bar{N}}W_{i}(u)W_{i}^{\prime }(u)\,du.)\end{eqnarray}$$

It follows that if at least one of integrals

$$\begin{eqnarray}\int _{D\cap U_{i}\backslash D}|W_{i}(g)|^{2}+|W_{i}^{\prime }(g)|^{2}\,dg\quad \text{or}\quad \int _{D\cap U_{i+1}\backslash D}|{\mathcal{T}}_{i}W_{i}(g)|^{2}+|{\mathcal{T}}_{i}W_{i}^{\prime }(g)|^{2}\,dg\end{eqnarray}$$

converges, then so does the other and

$$\begin{eqnarray}\displaystyle {\mathcal{B}}_{i+1}({\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}W_{i},{\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}^{-1}}W_{i}^{\prime }) & = & \displaystyle \int _{D\cap U_{i}U_{i+1}\backslash D}\int _{D\cap U_{i+1}\backslash D\cap U_{i}U_{i+1}}{\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}W_{i}(ug){\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}^{-1}}W_{i}^{\prime }(ug)\,du\,dg\nonumber\\ \displaystyle & = & \displaystyle \int _{D\cap U_{i}U_{i+1}\backslash D}\int _{D\cap U_{i}\backslash D\cap U_{i}U_{i+1}}W_{i}(ug)W_{i}^{\prime }(ug)\,du\,dg={\mathcal{B}}_{i}(W_{i},W_{i}^{\prime }).\nonumber\end{eqnarray}$$

We can now conclude the convergence for all $i$ and the identity (16) since they clearly reduce to the case $i=0$.◻

Theorem 4.3. Suppose that $\unicode[STIX]{x1D70B}^{\prime }=\unicode[STIX]{x1D70B}^{\vee }$ (or equivalently, $\unicode[STIX]{x1D70E}^{\prime }=\unicode[STIX]{x1D70E}^{\vee }$) and $\unicode[STIX]{x1D70B}$ is (AT) (see § 2.1). Then ${\mathcal{B}}_{i}(W_{i},W_{i}^{\prime })$ is a well-defined $G$-invariant pairing on ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})\times {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}^{-1}}(\unicode[STIX]{x1D70E}^{\vee })$. In particular, if $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$ is unitarizable, then ${\mathcal{B}}_{i}$ gives a unitary structure on ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$.

Proof. By Proposition 4.1 ${\mathcal{B}}_{i}(\cdot ,\cdot )$ is well defined and not identically zero. To show invariance it suffices to consider $i=0$. We use induction on $m$. The case $m=1$ (in which $D$ is the standard mirabolic subgroup) is well known and follows from Bernstein’s theorem [Reference BernsteinBer84]. For the induction step, let $m>1$ and let $Q^{\prime }$ be the subgroup of the standard maximal parabolic subgroup of $G$ of type $((m-1)n,n)$ consisting of the matrices whose lower right $n\times n$ corner is upper unitriangular. Write

$$\begin{eqnarray}\displaystyle {\mathcal{B}}_{0}(W_{\operatorname{Ze}},W_{\operatorname{Ze}}^{\prime }) & = & \displaystyle \int _{D\cap Q^{\prime }\backslash D}\int _{N_{D}\backslash D\cap Q^{\prime }}W_{\operatorname{Ze}}(qg)W_{\operatorname{Ze}}^{\prime }(qg)|\det q|^{1-n}\,dq\,dg\nonumber\\ \displaystyle & = & \displaystyle \int _{D\cap Q^{\prime }\backslash D}\int _{D_{m-1,n}\cap N\backslash D_{m-1,n}}W_{\operatorname{Ze}}(qg)W_{\operatorname{Ze}}^{\prime }(qg)|\det q|^{1-n}\,dq\,dg.\nonumber\end{eqnarray}$$

Here we consider $\operatorname{GL}_{(m-1)n}$ (and hence, $D_{m-1,n}$) as a subgroup of $G$. (Note that $\unicode[STIX]{x1D6FF}_{D}=|\det |^{m}$ while $\unicode[STIX]{x1D6FF}_{D\cap Q^{\prime }}=|\det |^{n+m-1}$.) By (5a) and the induction hypothesis, the inner integral is left-$(Q^{\prime },|\det |^{n-1})$-equivariant in $g$. Hence, we can replace the domain of outer integration by $Q^{\prime }\backslash D_{1,mn}$ where $D_{1,mn}$ is the standard mirabolic subgroup of $G$ (the stabilizer of $e_{mn}$). (Note that $\unicode[STIX]{x1D6FF}_{D_{1,mn}}=|\det |$ and $\unicode[STIX]{x1D6FF}_{Q^{\prime }}=|\det |^{n}$.) It follows that ${\mathcal{B}}_{0}(\cdot ,\cdot )$ is $D_{1,mn}$-invariant. By Bernstein’s theorem, it is