BERNSTEIN DEGREE AND ASSOCIATED CYCLES OF HARISH-CHANDRA MODULES – HERMITIAN SYMMETRIC CASE –

by

Kyo Nishiyama, Hiroyuki Ochiai & Kenji Taniguchi

Dedicated to Professor Ryoshi Hotta on his 60th anniversary

Abstract. — Let G̃ be the metaplectic double cover of Sp(2n, ℝ), U(p, q) or O^{∗}(2p). we study the Bernstein degrees and the associated cycles of the irreducible unitary highest weight representations of G̃, by using the theta correspondence of dual pairs. The first part of this article is a summary of fundamental properties and known results of the Bernstein degrees and the associated cycles. Our first result is a comparison theorem between the K-module structures of the following two spaces; one is the theta lift of the trivial representation and the other is the ring of regular functions on its associated variety. Secondarily, we obtain the explicit values of the degrees of some small nilpotent K_{ℂ}-orbits by means of representation theory. The main result of this article is the determination of the asssociated cycles of singular unitary highest weight representations, which are the theta lifts of irreducible representations of certain compact groups. In the proofs of these results, the multiplicity free property of spherical subgroups and the stability of the branching coefficients play important roles.

Résumé. — Soit G̃ la double revêtement métaplectique de Sp(2n, ℝ), U(p, q) ou O^{∗}(2p). Nous étudions les degré de Bernstein et les cycles associés des représentations irréductibles unitaires de G̃ avec plus haut poids, en utilisant la thêta correspondance par paires duales. La première partie de cet article est un abrégé de propriétés fondamentaux et résultats connus des degrés de Bernstein et les cycles associés. Notre premier résultat est un théorème comparatif entre les structures en tant que K-modules de deux espaces suivants; l’un est le thêta relèvement de la représentation évidente, l’autre est l’anneau de fonctions régulières sur sa variété associée. Deuxièmement, nous obtenons les valeurs concrètes des degrés de quelques petites K_{ℂ}-orbites nilpotentes au moyen de théorie de la représentation. Le principal résultat de cet article est la détermination des cyles associés de représentations singuliéres unitaires avec plus haut poids, qui sont les thêta relèvements des représentations irréductibles des certains groupes compacts. Dans les démonstrations de ces résultats, la propriété sans multiplicité de sou-groupes sphériques et la stabilité des coefficients de brnchement jouent rôles importants.

2000 Mathematics Subject Classification. — 2000 Mathematics Subject Classification. Primary 22E46, 32M15, 14L30. Key words and phrases. — Hermitian symmetric space, highest weight representation, nilpotent orbit, associated variety, associated cycle, multiplicity-free action, dual pair correspondence .

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1. Invariants of representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2. Known results and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3. Reductive dual pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4. Fock realization of Weil representation . . . . . . . . . . . . . . . . . . . . . . . . 23

5. Unitary lowest weight representations . . . . . . . . . . . . . . . . . . . . . . . . . . 25

6. Description of K-types of the lowest weight modules . . . . . . . . . . 28

7. Degree of nilpotent orbits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

8. Multiplicity free action and Poincaré series . . . . . . . . . . . . . . . . . . . . 54

9. Associated cycle of unitary lowest weight modules . . . . . . . . . . . . . . 64 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Introduction

Let G be a semisimple (or more generally, reductive) Lie group. For an irreducible admissible representation π of G, there exist several important invariants such as irreducible characters, primitive ideals, associated varieties, asymptotic supports, Bernstein degrees, Gelfand-Kirillov dimensions, etc. They are interrelated with each other, and intimately related to the geometry of coadjoint orbits.

For example, at least if G is compact and π is finite dimensional, the character of π is the Fourier transform of an orbital integral on a semisimple coadjoint orbit ([29]). This is also the case for a general semisimple G and fairly large family of the representations (see [41]). This intimate relation between coadjoint orbits and irreducible representations invokes the philosophy of so-called orbit method, which is exploited by pioneer works of Kirillov and Kostant, and is now being developed by many contributors. However, for a general semisimple Lie group G, it seems that the orbit method still requires much to do. In particular, we should understand some small representations corresponding to nilpotent coadjoint orbits, which are called unipotent.

On the other hand, by definition, most of invariants are directly related to nilpotent coadjoint orbits. In a sense, the corresponding nilpotent orbits represent the leading term of irreducible characters ([1], [44]). The invariants of large representations correspond to the largest nilpotent coadjoint orbit, namely, the principal nilpotent orbit. For large representations, the orbit method seems to behave considerably well. Therefore we are now interested in ‘small’ representations whose invariants are related to smaller nilpotent coadjoint orbits.

One extreme case is the case of finite dimensional representations. In this case, however, the corresponding orbit is zero, and there is not a so much interesting phenomenon. The next to the extreme case is the case of minimal representations, which corresponds to the minimal nilpotent orbit. The minimal nilpotent orbit is unique in the sense that it is the only orbit among non-zero nilpotent ones with the smallest possible dimension. These representations have a simple structure. For example, their K-type structure is in a ladder form and is multiplicity free ([50]). Against its simple structure, though, systematic and thorough study of the minimal representations is still progressing now through the works of Kostant-Brylinski and many other mathematicians. If we turn our attention to the small representations other than minimal ones, it seems that there is relatively less knowledge on them up to now. In this paper, we study small representations which are unitary lowest (or highest) weight representations of G. Such representations exist if and only if G/K enjoys a structure of Hermitian symmetric space, where K denotes a maximal compact subgroup of G.

To be more specific, let us introduce notations. We assume that the symmetric space G/K is irreducible and Hermitian. Moreover, we assume that G is classical other than SO(n, 2), i.e., G = Sp(2n, ℝ), U(p, q) or O^{∗}(2p). Let 𝔤_{0} be the Lie algebra of G and 𝔤_{0} = 𝔨_{0} + 𝔭_{0} the Cartan decomposition with respect to K. We denote the complexified decomposition by 𝔤 = 𝔨 + 𝔭. Since G/K is an irreducible Hermitian symmetric space, the induced adjoint representation of K on 𝔭 breaks up into precisely two irreducible components 𝔭 = 𝔭^{+} ⊕ 𝔭^{−}. Note that, as a representation of K, 𝔭^{−} is contragredient to 𝔭^{+} via the Killing form. We extend this representation of K to the representation of the complexification K_{ℂ} of K holomorphically.

Let L be an irreducible unitary lowest weight module of G. Then it is well-known that the associated variety of L, denoted by AV (L), is the closure of a single nilpotent K_{ℂ}-orbit contained in 𝔭^{−} (we choose an appropriate positive system which is compatible with 𝔭^{+}).

Put r = ℝ-rank G, the real rank of G. Then there exist exactly (r+1) nilpotent K_{ℂ}- orbits {O_{0}, O_{1}, …, O_{r}} in 𝔭^{−}. We choose an indexing of the orbits so that dim O_{i−1} < dim O_{i} holds for 1 ≤ i ≤ r ; in particular, O_{0} = {0} is the trivial one, and O_{r} is the open dense orbit. Most of lowest weight representations L correspond to the largest orbit O_{r}. For example, the associated variety of a holomorphic discrete series (or its limit) is \overline{O_{r}} = 𝔭^{−}. The invariants of the holomorphic discrete series representations are completely understood (see [14], [43], [7]; also see §2.4 below). However, for each orbit O_{m} (0 < m < r), there exists a relatively small family of lowest weight representations whose associated variety is indeed the closure of the orbit O_{m}. Thanks to the theory of reductive dual pairs via the Weil representation of metaplectic groups, we have a complete knowledge of such a family of lowest weight representations (at least for classical groups listed above).

Although we can define a specific ‘small’ representations even for the largest orbit O_{r}, we restrict ourselves to the case O_{m} (m < r) in this introduction. Then there exists a compact group G_{2} corresponding to each m (cf. §3, Table 2) such that (G_{1}, G_{2}) forms a dual pair in a large symplectic group Sp(2N, ℝ). Let Mp(2N, ℝ) be the metaplectic double cover of Sp(2N, ℝ). We denote by H̃ ⊂ Mp(2N, ℝ) the inverse image of a subgroup H ⊂ Sp(2N, ℝ) of the covering map.

The family of unitary irreducible lowest weight representations of G̃ whose associated variety is \overline{O_{m}} is parametrized by Irr(G_{2}), the set of the irreducible finite dimensional representations of G_{2}. We denote the lowest weight representation of G̃ corresponding to σ ∈ Irr(G_{2}) by L(σ) (see §5 for precise description). Roughly, the correspondence σ ↦ L(σ) is the theta lift after twisted by a certain unitary character of G̃_{2}.

Our first observation is the following.

Theorem A. — Let 1_{G_{2}} be the trivial representation of G_{2} and L(1_{G_{2}}) the unitary lowest weight representation of G̃ corresponding to 1_{G_{2}}. The Bernstein degree of L(1_{G_{2}}) coincides with the degree of the closure of the nilpotent orbit \overline{O_{m}} (defined in the sense of algebraic geometry) ;

Deg L(1_{G_{2}}) = deg \overline{O_{m}}.

We also get an explicit and computable formula for Deg L(1_{G_{2}}).

Note that the varieties \overline{O_{m}} are determinantal varieties of various type and an explicit formula of their degree is known as Giambelli-Thom-Porteous formula. Our representation theoretic proof of the formula seems new, and gives an alternative proof.

To prove Theorem A, we construct a K_{ℂ}-equivariant map ψ : V → \overline{O_{m}}, where V is a certain K_{ℂ} × (G_{2})_{ℂ}-module. This map induces an algebra isomorphism

ψ^{∗} : ℂ[\overline{O_{m}}] ⟶^{∼} ℂ[V^{∗}]^{(G_{2})_{ℂ}},

which means that \overline{O_{m}} = V//(G_{2})_{ℂ}. The map ψ is closely related to the dual pair (G, G_{2}), and we call it unfolding of \overline{O_{m}}. By this, the proof of Theorem A reduces to a problem of classical invariant theory.

The ‘smallest’ unipotent representation attached to the orbit O_{m} should be realized on the section of a certain line bundle on O_{m} called half-form bundle ([5], [6], [52]). We investigate such half-form bundles, and get an evidence of strong relationship between the space of global sections of the half-form bundles and L(σ), where σ is a special one-dimensional character of G_{2}.

Next, let us consider a general unitary lowest weight module L(σ) (σ ∈ Irr(G_{2})). We describe its K-type decomposition and the Poincaré series in terms of certain branching coefficient of finite dimensional representations of general linear groups and G_{2}. Such descriptions are well-known among experts. However, references to them are scattered in many places, and sometimes their treatments are ad hoc. Since we need an explicit and unified picture for the K-types of L(σ), we reproduce the decompositions in the sequel.

Now our main theorem says

Theorem B. — Let L(σ) be an irreducible unitary lowest weight module of G̃ corresponding to σ ∈ Irr(G_{2}). Then its Bernstein degree is given by

Deg L(σ) = dim σ · deg \overline{O_{m}}.

There is a notion of associated cycle which is a refinement of the notion of associated variety. Roughly speaking, it expresses associated variety with multiplicity. For a precise definition, see §§1.1 and 1.3. Then the following is an immediate corollary to Theorem B.

Theorem C. — The associated cycle of L(σ) is given by AC (L(σ)) = dim σ · [\overline{O_{m}}].

The proof of Theorem B is based on the theory of multiplicity free action of algebraic groups, which is a subject of §8. The key ingredients of the proof are multiplicity free property of spherical subgroups and Sato’s summation formula of the stable branching coefficients.

Lastly, we would like to comment on several aspects of our results.

First, the Bernstein degree of an irreducible representation π is closely related to the dimension of its “Whittaker vectors”. In fact, for large representations, Matumoto proved that the Bernstein degree and the dimension of algebraic Whittaker vectors coincide ([36]). For ‘small’ representations, we cannot hope the same story, because they do not have any Whittaker vector in a naive sense. However, for complex semisimple Lie groups, Matumoto observed that the finite-dimensionality and non-vanishing of the space of certain degenerate Whittaker vectors determines the wave front set of π ([34], [35]). Recently, Yamashita has found a strong relation between the multiplicity of associated cycles and the dimension of generalized Whittaker vectors in the case of unitary highest weight module ([54]).

Second, let us consider the (twisted) theta correspondence (or Howe correspondence, dual pair correspondence, ...) between L(σ) ∈ Irr(G̃) and σ ∈ Irr(G_{2}). Since G_{2} is compact and σ is finite dimensional, its associated cycle is simply given by AC (σ) = dim σ · [\{0\}]. Recall AC (L(σ)) = dim σ · [\overline{O_{m}}] from Theorem C. These formulas strongly indicate the following; there should be a correspondence between nilpotent orbits of the dual pairs, and it induces certain relation between associated cycles of representations in theta correspondence. An optimistic reflection suggests that, if L(σ) is a theta lift of σ, then their associated varieties are related as

AC (L(σ)) = ∑_{i} m_{i}[\overline{O_{i}}] ⟷ AC (σ) = ∑_{i} m_{i}[\overline{O′_{i}}],

with the same multiplicity, where O_{i} ↔ O′_{i} indicates the orbit correspondence. However we do not have an intuitive evidence of such a kind of correspondence other than the cases treated here.

Third, Theorem A (or K-type decompositions) suggests that we should “quantize” the orbit O_{m} to get an irreducible unitary representation L(1_{G_{2}}), which certainly should be a unipotent representation. For this, it will be helpful to try the similar method exploited by Kostant-Brylinski in the case of the minimal orbit. However, this will require much more than what we have presented in this note.

Now let us explain each section briefly.

In §1, we define the associated cycles and other important invariants of representations in a general setting. After that, we collect their basic properties which will be needed later. In particular, in Lemma 1.1 and Theorem 1.4, we clarify the relationship between the associated cycles and the Bernstein degree (or the degree of the projectivised nilpotent cone); also, we recall the fact that the associated variety is the projection of the characteristic variety under the moment map (Lemma 1.6).

In §2, we briefly summarize known facts and examples of associated cycles of various types of representations. To see what is going on in this paper, §§1.3 and 2.4 will be extremely useful.

In §3, we review the properties of a reductive dual pair which we will need later. After an explicit description of the Fock realization of the Weil representation in §4, §5 is devoted to giving the complete description of the unitary lowest weight representations of G via theta correspondence.

In §6, we give a formula of K-type decomposition of the unitary lowest weight representations, using the branching coefficient of finite dimensional representations of compact groups. These formulas are well-known among experts, however, we need full detailed formulas in the following sections.

In §7, we study the geometry of nilpotent orbits in the case of Hermitian symmetric pair. Take a K_{ℂ}-orbit O_{m} in 𝔭^{−} and the unfolding ψ : V → \overline{O_{m}} as above. We use ψ to study the ring of regular functions ℂ[\overline{O_{m}}] on \overline{O_{m}}, and clarify its K_{ℂ}-module structure. This leads us an identification of deg \overline{O_{m}} and the Bernstein degree of one of the smallest unitary lowest weight module attached to O_{m}. As a result, Theorem A is proved. We also study the global sections of the half-form bundle over O_{m} in the tame cases.

In §8, we study a general theory of multiplicity free actions of a pair of reductive algebraic groups. We define the notions of degree and dimension of the space of covariants. The main result in this section is the formula of the degree and the dimension of covariants (Theorem 8.6).

In §9, we treat general unitary lowest weight representations of G, which are singular. By the results of §8, we prove Theorems B and C in this section.

We thank M. Duflo, M. Vergne, J. Faraut, D. Vogan, J. Adams and H. Yamashita for useful discussions. The first author is grateful to Université Paris VII for the kind hospitality during his stay in 1998. The preliminary note of this paper was prepared at that time.

Notation: We denote the field of real (respectively, complex, quaternionic) numbers by ℝ (respectively, ℂ, ℍ). If 𝕂 is one of these fields, we use the following notation for subsets of matrices:

M(n, m, 𝕂) the set of all n × m matrices,
Sym (n, 𝕂) the set of all symmetric matrices of size n,
Alt (n, 𝕂) the set of all alternating matrices of size n.

These subsets are abbreviated as M_{n,m}, Sym_{n}, Alt_{n} respectively, if there is no confusion on the base fields. For 𝕂 = ℂ or ℍ, we also denote by skew-Her (n, 𝕂) the set of all skew Hermitian matrices of size n. If τ_{λ} is an irreducible finite dimensional representation of GL(m, ℂ) (or U(m)) with highest weight λ, we often write it as τ_{λ}^{(m)}, denoting the rank m of the group explicitly by the superscript.

1. Invariants of representations

1.1. A review on the commutative ring theory. — First of all, we review well-known results in the commutative ring theory, which we need in the subsequent sections. For more details of what are discussed here, we refer the readers to textbooks on the commutative ring theory, for example, [10], [21], [33].

Let V be an n-dimensional vector space over the field ℂ and let A := ℂ[V] be the ring of polynomials on V. For a finitely generated A-module M, the support Supp M of M is defined to be the set of prime ideals 𝔭 with M_{𝔭} ≠ 0. Since M is finitely generated, Supp M coincides with the Zariski closure of Ann M := {a ∈ A | aM = 0}, which is denoted by V (M). We often identify V (M) with the affine variety

V (M) ∩ m-Spec A = {x ∈ V | p(x) = 0 (∀p ∈ Ann M)}.

Let A_{n} be the set of homogeneous polynomials of degree n. By the natural grading A = ⊕_{n=0}^{∞}A_{n}, A is a graded ℂ-algebra. Let M = ⊕_{n=0}^{∞}M_{n} be a finitely generated graded A-module. As usual, we denote the Poincaré series by P (M; t). It is wellknown that there exists a unique polynomial Q(t) and a non-negative integer d such that

P (M; t) = ∑_{n=0}^{∞}(dim M_{n})t^{n} = \frac{Q(t)}{(1 − t)^{d}} , Q(1) ≠ 0.  (1.1)

It turns out that Q(1) is a positive integer. By the expression (1.1), we know that dim M_{n} is a polynomial in n for sufficiently large n, and it is written as

dim M_{n} = \frac{Q(1)}{(d − 1)!}n^{d−1} + (lower order terms of n).

Note that the integer d is the dimension of V (M). The integer Q(1) is called the multiplicity of M, and we denote it by m(M).

A prime ideal 𝔓 ∈ Spec A is called an associated prime of M if 𝔓 is an annihilator of some non-zero element of M. The set of associated primes is denoted by Ass M. It is easy to see that Ass M ⊂ Supp M. The set of minimal elements of Ass M and that of Supp M coincide, and they form a finite set. Let {𝔓_{1}, …, 𝔓_{r}} be the set of minimal primes in Supp M, and let V (M) = ∪_{i=1}^{r}C_{i} be the corresponding irreducible decomposition of the variety V (M).

Choose 𝔔^{1} ∈ Ass M. Then there exists a submodule M^{1} ⊂ M such that M^{1} ≃ A/𝔔^{1}. By induction, there exists a finite sequence 0 = M^{0} ⊂ M^{1} ⊂ ⋯ ⊂ M^{l} = M such that M^{k}/M^{k−1} ≃ A/𝔔^{k} for some 𝔔^{k} ∈ Spec A (k = 1, 2, …, l). It is not hard to check that the integer

mult_{𝔓}(M) := #{𝔔^{k} | 𝔔^{k} = 𝔓}, 𝔓 : minimal prime

is independent of the choice of the sequence {M^{k}}_{k}. This integer is called the multiplicity of M at 𝔓. Note that mult_{𝔓}(M) is reinterpreted as the length of Artinian A_{𝔓}-module M_{𝔓}. By the correspondence of minimal 𝔓 ∈ Ass M and the irreducible component C of V (M), we also denote the multiplicity by mult_{C}(M).

As a refinement of Supp M, we consider the formal linear combination of the minimal primes 𝔓_{i} (or irreducible components C_{i}) with coefficients mult_{𝔓_{i}}(M) = mult_{C_{i}}(M),

\underline{Supp} M := ∑_{i} mult_{𝔓_{i}}(M) [𝔓_{i}] = ∑_{i} mult_{C_{i}}(M) [C_{i}].

More generally, let F be a coherent O_{X}-module on an algebraic variety X. We can refine Supp F analogously. For any irreducible component C of the support of F, the rank of the module F at a generic point of C is a well-defined positive integer mult_{C}(F). This is called the multiplicity of C in the support of F. Then we consider the formal linear combination of the components C of Supp(F) with coefficients mult_{C}(F),

\underline{Supp}(F) = ∑_{C} mult_{C}(F) [C].

The multiplicity of M can be obtained from \underline{Supp} M. Let deg C be the degree of the variety C, i.e. deg C = m(A/𝔓) (see, e.g., [16]). Since the Poincaré series is additive, m(M) is the sum of m(M^{k}/M^{k−1})’s with dim V (M^{k}/M^{k−1}) = dim V (M). By the definition of the sequence {M^{k}}_{k} and the multiplicity mult_{𝔓}(M), we have

Lemma 1.1. —

m(M) = ∑_{{i \atop dim C_{i}=dim V (M)}} mult_{C_{i}}(M) deg C_{i}.

Remark 1.2. — The notion of degree is usually defined for projective varieties. In our case, we can projectivise V (M) and its irreducible components since Ann M is graded. Then deg C_{i} should be interpreted as the degree of the projectivised variety.

1.2. Invariants of U(𝔤)-modules. — In this subsection, we introduce invariants of representations of Lie algebras after [49], [51]. These invariants are main objects of this paper.

Let 𝔤 be a finite dimensional complex Lie algebra and let U(𝔤) be its universal enveloping algebra. We denote by U_{n}(𝔤) the finite dimensional subspace of U(𝔤), spanned by products of at most n-elements of 𝔤. Then {U_{n}(𝔤)}_{n=0}^{∞} is a filtration of U(𝔤), called the standard filtration. By the Poincaré-Birkhoff-Witt (PBW) theorem, the associated graded algebra gr U(𝔤) = ⊕_{n=0}^{∞}U_{n}(𝔤)/U_{n−1}(𝔤) is isomorphic to the symmetric algebra S(𝔤).

Let V be a U(𝔤)-module. A chain 0 = V_{−1} ⊂ V_{0} ⊂ V_{1} ⊂ ⋯ ⊂ V, where the V_{n}’s are subspaces of V, is called a filtration of V if it satisfies the following conditions:

⋃_{n=0}^{∞} V_{n} = V,  U_{n}(𝔤)V_{m} ⊂ V_{n+m},  dim V_{n} < ∞.

By the second condition, the graded object

gr V = ⨁_{n=0}^{∞} gr_{n}V, gr_{n}V := V_{n}/V_{n−1}

has the structure of a graded S(𝔤)-module. A filtration is called good if it also satisfies

U_{n}(𝔤)V_{m} = V_{n+m} (for all m sufficiently large, all n ≥ 0).  (1.2)

In this case, V is a finitely generated U(𝔤)-module and gr V is a finitely generated S(𝔤)-module. Conversely, if V is finitely generated, we can construct a good filtration by choosing a finite dimensional generating subspace V_{0} and by putting V_{n} = U_{n}(𝔤)V_{0}.

Regarding the symmetric algebra A = S(𝔤) as the polynomial ring on the dual space 𝔤^{∗}, we define several invariants of V using those defined via commutative ring theory.

Definition 1.3. — For a finitely generated U(𝔤)-module V, we define the associated variety AV (V), the associated cycle AC (V), the Gelfand-Kirillov dimension Dim V, and the Bernstein degree Deg V by

AV (V) = V (gr V),  AC (V) = \underline{Supp} (gr V),
Dim V = dim AV (V),  Deg V = m(gr V)

respectively. They are independent of the choice of good filtrations of V, and therefore well-defined for V.

For an exact sequence

0 → V_{1} → V_{2} → V_{3} → 0

of finitely generated U(𝔤)-modules, we have Dim V_{2} = max{Dim V_{1}, Dim V_{3}}, and

AV (V_{2}) = AV (V_{1}) ∪ AV (V_{3}).  (1.3)

Note that the associated cycle is not additive in general, i.e., AC (V_{2}) ≠ AC (V_{1}) + AC (V_{3}). If we write

c_{d}(V) = \begin{cases}Deg V & if d = Dim V , \\ 0 & if d > Dim V , \end{cases}

then the Bernstein degree becomes additive in the sense that

Deg V_{2} = c_{Dim V_{2}}(V_{2}) = c_{Dim V_{2}}(V_{1}) + c_{Dim V_{2}}(V_{3}).

The right hand side is equal to Deg V_{1} + Deg V_{3} if Dim V_{1} = Dim V_{3}.

1.3. The structure of invariants of Harish-Chandra modules. — The associated variety of a module (with some assumption, of course) over a reductive Lie algebra 𝔤 is contained in the nilpotent cone in 𝔤^{∗}. Moreover, if it is a Harish-Chandra (𝔤, K)-module, the associated variety has a K_{ℂ}-orbit structure. In this subsection, we shall review these well-known results.

Let G be a connected reductive group over ℝ and 𝔤_{0} its Lie algebra. Take a maximal compact subgroup K ⊂ G and let K_{ℂ} be its complexification. Denote by 𝔤_{0} = 𝔨_{0} + 𝔭_{0} a Cartan decomposition associated to K and by 𝔤 = 𝔨 + 𝔭 its complexification. For a Harish-Chandra (𝔤, K)-module H, we choose a finite dimensional K-invariant generating subspace H_{0} and define a filtration by H_{n} = U_{n}(𝔤)H_{0}. Then the graded object gr H has compatible S(𝔤)- and K_{ℂ}-actions.

By the compatibility of 𝔤- and K_{ℂ}-actions, AV (H) is invariant under the action of K_{ℂ} and 𝔨 acts on gr H trivially. It follows that AV (H) is a K_{ℂ}-invariant subvariety in (𝔤/𝔨)^{∗} ≃ 𝔭.

Fix a connected algebraic group G_{ℂ} with Lie algebra 𝔤. The algebra U(𝔤)^{G_{ℂ}} of Ad (G_{ℂ})-invariants in U(𝔤) is isomorphic to the center Z(𝔤) of U(𝔤), since G_{ℂ} is connected. Filter the algebra U(𝔤)^{G_{ℂ}} by the standard filtration of U(𝔤), then gr U(𝔤)^{G_{ℂ}} is isomorphic to S(𝔤)^{G_{ℂ}}, the algebra of Ad (G_{ℂ})-invariants in S(𝔤). Since any irreducible U(𝔤)-module is annihilated by a maximal ideal in U(𝔤)^{G_{ℂ}}, any U(𝔤)-module of finite length is annihilated by the product of a finite number of maximal ideals in U(𝔤)^{G_{ℂ}}. Such a product is of finite codimension in U(𝔤)^{G_{ℂ}}. Therefore, the radical of the graded object of this product is the ideal S^{+}(𝔤)^{G_{ℂ}}, the set of invariant polynomials without constant term. This argument implies that the associated variety AV (H) is contained in the zero set V (S^{+}(𝔤)^{G_{ℂ}}) of S^{+}(𝔤)^{G_{ℂ}}. Note that V (S^{+}(𝔤)^{G_{ℂ}}) coincides with the set N^{∗} of nilpotent elements in 𝔤^{∗}, since G_{ℂ} is connected.

Consequently, AV (H) is a union of K_{ℂ}-orbits in N^{∗} ∩ (𝔤/𝔨)^{∗} ≃ N_{𝔭}, the set of nilpotent elements in 𝔭. By a theorem of Kostant-Rallis, N_{𝔭} is a finite union of K_{ℂ}- orbits. Summarizing the above discussion and the results of many contributors, we have the following well-known theorem.

Theorem 1.4. — If H is a Harish-Chandra (𝔤, K)-module, then the associated variety AV (H) is a finite union of nilpotent K_{ℂ}-orbits in 𝔭. Moreover, if H is irreducible, we have the following.

(1) There exist nilpotent K_{ℂ}-orbits {C_{i}} ⊂ N_{𝔭} with dimension equal to Dim H such that

AV (H) = ⋃_{i=1}^{l} \overline{C_{i}}.  (1.4)

(2) Denote the associated cycle as AC (H) = ∑_{i} m_{i}[\overline{C_{i}}]. Then the Bernstein degree is given by

Deg H = ∑_{i=1}^{l} m_{i} deg \overline{C_{i}}.  (1.5)

(3) Let I = I_{H} ⊂ U(𝔤) be the associated primitive ideal. Then AV (U(𝔤)/I) is the closure of a single nilpotent G_{ℂ}-orbit C_{H}, and for any i, the G_{ℂ}-orbit through C_{i} coincides with C_{H}. In fact, C_{H} ∩𝔭 decomposes into a finite union of equidimensional nilpotent K_{ℂ}-orbits, and {C_{i}} is a subset of its irreducible components:

C_{H} ∩ 𝔭 ⊃ C_{1}, …, C_{l}.  (1.6)

Remark 1.5. — Take λ ∈ C_{i} and denote by K_{ℂ}(λ) the fixed subgroup of K_{ℂ} at λ. The multiplicity m_{i} in (2) can be interpreted as the dimension of a certain representation of K_{ℂ}(λ). For this, we refer to [51, Definition 2.12].

1.4. Invariants of D_{X}-modules. — The relation between the associated varieties (associated cycles) and the characteristic varieties (characteristic cycles) is discussed in [3]. First, we recall the definition of the characteristic varieties and the characteristic cycles, which is analogous to that of the associated variety and its cycle for 𝔤-module given in § 1.2.

Let X be a smooth algebraic variety over an algebraically closed field ℂ. We denote by D_{X} the sheaf of (algebraic) differential operators on X. On D_{X}, we have a natural increasing filtration by the O_{X}-submodules D_{X}(n), the subsheaf of all differential operators of order ≤ n;

0 = D_{X}(−1) ⊂ D_{X}(0) = O_{X} ⊂ D_{X}(1) ⊂ ⋯ ⊂ D_{X}.

The associated graded sheaf

gr D_{X} = ⨁_{n=0}^{∞} gr_{n}D_{X}, gr_{n}D_{X} = D_{X}(n)/D_{X}(n − 1)

is naturally identified with the direct image sheaf π_{∗}(O_{T^{∗}X}), where π : T^{∗}X → X is the cotangent bundle of X, and O_{T^{∗}X} is the structure sheaf of T^{∗}X.

Let M be a coherent D_{X}-module. Then there is a good filtration

0 = M_{−1} ⊂ M_{0} ⊂ M_{1} ⊂ ⋯ ⊂ M.

The corresponding graded module is defined by

gr M = ⨁_{n=0} gr_{n}M, gr_{n}M = M_{n}/M_{n−1}.

Then gr M is coherent over π_{∗}(O_{T^{∗}X}). The support of gr M as a module on T^{∗}X, more precisely, the support of O_{T^{∗}X} ⊗_{gr D_{X}} gr M, is called the characteristic variety of M. This is a closed conic algebraic subvariety of the cotangent bundle T^{∗}X, which is usually denoted by

Ch(M) = Supp(gr M).

The variety does not depend on the choice of a good filtration. As a refinement of Ch(M), we define the characteristic cycle of a coherent D_{X}-module M by

\underline{Ch}(M) = \underline{Supp}(gr M).

The characteristic cycle is also independent of the choice of a good filtration.

From now on in this subsection, let G be a reductive algebraic group over ℂ and let X be the set of Borel subgroups of G. Then it is known that X is a complete G-homogeneous variety, and the Lie algebra 𝔤 of G acts on X by vector fields on X. This gives a Lie algebra homomorphism

𝔤 → D(X),

where D(X) = Γ(X, D_{X}) denotes the set of all global sections of the sheaf D_{X} on X. This map extends to an algebra homomorphism

ψ : U(𝔤) → D(X),

which is known to be surjective. With the natural filtrations, gr U(𝔤) is canonically isomorphic to the symmetric algebra S(𝔤), while gr D(X) to the set of global sections ℂ[T^{∗}X] of (algebraic) holomorphic functions on the cotangent bundle T^{∗}X. Since the map ψ is compatible with the natural filtrations, we have the associated graded ring homomorphism

ϕ = gr ψ : S(𝔤) → ℂ[T^{∗}X].

The map ϕ gives rise to the moment map

μ : T^{∗}X → 𝔤^{∗},

where 𝔤^{∗} is the dual vector space of 𝔤. It is known that the image of μ is normal and that the map μ is birational onto its image. The moment map is the key to give a relation between the characteristic variety and the associated variety.

Let M be a coherent D_{X}-module. Then the set of all global sections M = Γ(X, M) is a module over D(X) = Γ(X, D_{X}). Using the algebra homomorphism ψ, a D(X)- module is considered as a 𝔤-module. Then M is a finitely generated 𝔤-module with the trivial central character. Conversely, any finitely generated 𝔤-module M with the trivial central character can be obtained in this manner from a coherent D_{X}-module M. Indeed, M is obtained by, so called, the localization such as M = D_{X} ⊗_{U(𝔤)} M using the homomorphism ψ.

Lemma 1.6. — Let M be a coherent D_{X}-module. Consider M = Γ(X, M) as a 𝔤-module.

(1) The associated variety of M is the image of the characteristic variety of M under the moment map:

AV (M) = μ(Ch(M)).

(2) Suppose, moreover, that M has a good filtration {M_{j}}_{j} such that H^{1}(X, M_{j}) = 0. We denote the direct image under the moment map of the O_{T^{∗}X}-module gr M by μ_{∗}(gr M), which is a coherent O_{𝔤^{∗}}-module. Then the associated cycle is described by the cycle of this module

AC (M) = \underline{Supp}(μ_{∗}(gr M)),

where the definition of the cycle of O_{𝔤^{∗}}-module is given in §1.1.

More general statement would be found in Theorem 1.9 and Remark to Lemma 1.6 in [3]. The condition of the vanishing of the first cohomology appearing in (2) of the lemma holds for sufficiently regular infinitesimal characters, due to a result of Serre. See Appendix A of [3], for details.

2. Known results and examples

In this section, we summarize known results and examples of the invariants defined in §1. Some of them are immediately obtained from the definition, others are nontrivial.

2.1. Finite dimensional representation. — For a finite dimensional representation V of a complex Lie algebra 𝔤, we may take V_{0} = V and consequently V_{n} = V for all n ≥ 0. Then the Poincaré series is a constant dim V, and we have

Dim V = 0 and Deg V = dim V.

From this, we conclude that the associated variety of V is {0}, and the associated cycle equals AC (V) = (dim V) · [{0}].

2.2. Generalized Verma module. — Let 𝔮 = 𝔩 + 𝔲̄ be the Levi decomposition of a parabolic subalgebra 𝔮 of a complex reductive Lie algebra 𝔤, where 𝔩 is a Levi subalgebra and 𝔲̄ is the nilpotent radical of 𝔮. Denote by 𝔲 the opposite nilpotent Lie algebra to 𝔲̄. Take an irreducible finite dimensional representation τ_{λ} of 𝔩 with the highest weight λ and extend it to a representation of 𝔮 trivially. The generalized Verma module M(λ) is defined by M(λ) := U(𝔤) ⊗_{U(𝔮)} τ_{λ}.

Proposition 2.1. — The invariants for the generalized Verma module M(λ) are

Dim M(λ) = dim 𝔲 = dim 𝔲̄,  Deg M(λ) = dim τ_{λ},  (2.7)
AV (M(λ)) = 𝔲̄ and  AC (M(λ)) = (dim τ_{λ})[𝔲̄].  (2.8)

Here, we identified 𝔤^{∗} with 𝔤 by the Killing form.

Proof. — By the PBW theorem, M(λ) = U(𝔲) ⊗_{ℂ} τ_{λ} as a vector space and M(λ)_{n} := U_{n}(𝔲) ⊗_{ℂ} τ_{λ} (n = 0, 1, 2, …) defines a good filtration of M(λ). We denote the associated graded module by gr M(λ). Since

dim gr_{n}M(λ) = (dim τ_{λ}) × \binom{n + dim 𝔲 − 1}{dim 𝔲 − 1} ,

we immediately conclude that Dim M(λ) = dim 𝔲 and Deg M(λ) = dim τ_{λ}.

Next, we shall calculate AC (M(λ)). Since 𝔮 is contained in Ann_{S(𝔤)} gr M(λ) and the intersection S(𝔲)∩Ann_{S(𝔤)} gr M(λ) is {0}, Ann_{S(𝔤)} gr M(λ) coincides with S(𝔤)𝔮. Then

AV (M(λ)) = {x ∈ 𝔤^{∗} | ⟨x, 𝔮⟩ = {0}} ≃ 𝔲̄.

Moreover, since 𝔲̄ ≃ ℂ^{dim 𝔲̄} is irreducible and its degree is one, the multiplicity is dim τ_{λ} by Lemma 1.1.

2.3. Lowest weight module. — We use the same notation as in the previous subsection.

Let V be a 𝔮-lowest weight U(𝔤)-module, i.e. there exists an irreducible finite dimensional 𝔩-submodule V_{0} in V such that 𝔲̄ acts trivially on it and V is generated by it. Let λ be the highest weight of V_{0}. By the universality of the generalized Verma module, there exists a unique surjective U(𝔤)-homomorphism

Φ : M(λ) ↠ V.

By this homomorphism, a good filtration on V is induced from that of M(λ). By (1.3) and (2.8), we have

AV (V) ⊂ 𝔲̄.  (2.9)

2.4. Hermitian symmetric case. — Let (G, K) be an irreducible Hermitian symmetric pair. We use the notation in § 1.3. The adjoint representation of K on 𝔭 decomposes into two irreducible components 𝔭^{±}. Since 𝔮 := 𝔨 + 𝔭^{−} is a maximal parabolic subalgebra of 𝔤, we can apply the results in §§ 1.3 and 2.2 for a 𝔮-lowest weight module. By (2.9) and Theorem 1.4, the associated variety of a 𝔮-lowest weight (𝔤, K)-module is a finite union of K_{ℂ}-orbits in 𝔭^{−}.

In particular, since the (𝔤, K)-module of the holomorphic discrete series is a generalized Verma module, the invariants for it are given by (2.7) and (2.8), where τ_{λ} is the minimal K-type and 𝔲̄ = 𝔭^{−}. Namely we have

Proposition 2.2. — Let π_{λ} be a holomorphic discrete series representation of G with the minimal K-type τ_{λ}. Then invariants of π_{λ} are given as

Dim π_{λ} = dim 𝔭^{−} = \frac{1}{2} dim G/K,  Deg π_{λ} = dim τ_{λ},  (2.10)
AV (π_{λ}) = 𝔭^{−} and  AC (π_{λ}) = (dim τ_{λ})[𝔭^{−}].  (2.11)

Let us consider the Poincaré series of a 𝔮-lowest weight module V. Let Z be the center of K and let 𝔷_{0} be its Lie algebra. Under our setting, every element of Z acts on 𝔭^{±} by a non-trivial scalar and it acts on the minimal K-type of V also by a scalar.

Choose a base H of 𝔷_{0} and denote by α the scalar ad (H)|_{𝔭^{+}}. Let h(s) := exp sH ∈ K. The action of h(s) on V gives the Poincaré series of V. More precisely,

Proposition 2.3. — The Poincaré series of a 𝔮-lowest weight module V is

P (gr V; t) = t^{−n_{0}} (trace h(s)|_{V}) ,

where t = e^{αs} and n_{0} is the scalar by which H acts on the minimal K-type of V.

Proof. — First, we consider the generalized Verma module M(λ). The action of h(s) on M(λ)_{n} is a scalar e^{(n+n_{0})αs}. By the definition of the Poincaré series, we have P (gr M(λ); t) = t^{−n_{0}} (trace h(s)|_{M(λ)}). Using the universality of the Verma module, we obtain the Poincaré series of a lowest weight module V in the same way.

2.5. Discrete series of real rank one groups. — For the discrete series representations of real rank one groups, the associated cycles are explicitly obtained by Chang [8].

By many contributors, the associated variety of a discrete series is well-known. Especially, it is a closure of a single K_{ℂ}-orbit in 𝔭, and irreducible (see Theorem 1.4). Then the problem reduces to the determination of the multiplicity. Using the relation between the associated cycle and the characteristic cycle (Lemma 1.6), he calculated it by investigating the fiber of the moment map.

For the explicit value of the multiplicity, we refer to his paper.

2.6. Large representation. — Let G be a real reductive Lie group and let G = KA_{m}N_{m}, 𝔤_{0} = 𝔨_{0} + 𝔞_{m,0} + 𝔫_{m,0} be the Iwasawa decomposition of G and 𝔤_{0} := Lie G, respectively.

For a Harish-Chandra (𝔤, K)-module V, it is known that the Gelfand-Kirillov dimension is at most dim 𝔫_{m,0} ([49]). We call V large if Dim V = dim 𝔫_{m,0}. In this case, V has Whittaker models and the dimension of models coincides with the Bernstein degree of V.

To state more precisely, we need some notation. Let ψ : N_{m} → ℂ^{×} be a unitary character. We denote the differential character of 𝔫_{m,0} by the same symbol ψ. Then ψ is identified with an element of \sqrt{−1}(𝔫_{m,0}/[𝔫_{m,0}, 𝔫_{m,0}])^{∗}. We call ψ admissible if the coadjoint M_{m}A_{m}-orbit of ψ is open in (𝔫_{m,0}/[𝔫_{m,0}, 𝔫_{m,0}])^{∗}. Here, M_{m} is the centralizer of A_{m} in K. For an admissible ψ, we define the space of dual Whittaker vectors Wh_{𝔫_{m,0},ψ}^{∗} (V) by

Wh_{𝔫_{m,0},ψ}^{∗} (V) := {v^{∗} ∈ V^{∗} | Xv^{∗} = ψ(X)v^{∗} (∀X ∈ 𝔫_{m,0})},

where V^{∗} is the dual space of V.

Theorem 2.4 ([36]). — The space Wh_{𝔫_{m,0},ψ}^{∗} (V) is not zero if and only if Dim V = dim 𝔫_{m,0}. In this case, the dimension of Wh_{𝔫_{m,0},ψ}^{∗} (V) equals Deg V.

If V is a principal series representation, the dimension of Wh_{𝔫_{m,0},ψ}^{∗} (V) is obtained by Kostant (quasi-split case, [30]) and Lynch (non-quasi-split case, [32]). Thus by the above theorem, we know the Bernstein degree of V:

Theorem 2.5 ([30], [32]). — The principal series representation Ind _{M_{m}A_{m}N_{m}}^{G}(σ ⊗ e^{ν} ⊗ 1) is large, and the Bernstein degree is #W (𝔤_{0}, 𝔞_{m,0}) · dim σ, where W (𝔤_{0}, 𝔞_{m,0}) is the little Weyl group.

Remark 2.6. — The associated variety of a principal series representation is a finite union of the closure of regular nilpotent K_{ℂ}-orbits in 𝔭. Let {N_{1}, …, N_{l}} ⊂ 𝔭 be the set of all regular nilpotent K_{ℂ}-orbits. Then we have

∑_{i=1}^{l} deg \overline{N_{i}} = #W (𝔤_{0}, 𝔞_{m,0})

(see [31]). Since deg \overline{N_{i}} = deg \overline{N_{j}}, we see that deg \overline{N_{i}} = #W (𝔤_{0}, 𝔞_{m,0})/l.

There are explicit calculations of Whittaker models of some low rank groups. For the following representations, the Whittaker models are explicitly determined.

(1) Large discrete series representations of Sp(2, ℝ) (by Oda [38]).

(2) Large discrete series representations of SU(2, 2) (by Yamashita [53] and Hayata-Oda [19]).

(3) The generalized principal series representation Ind _{P_{J}}^{G} (σ ⊗ e^{ν+ρ_{J}} ⊗ 1) of G = Sp(2, ℝ) (by Hayata [18]). Here, P_{J} = M_{J}A_{J}N_{J} is the Jacobi parabolic subgroup of G and σ is a discrete series representation of M_{J} ≃ ℂ^{×} × SU(1, 1).

(4) Large discrete series representations of SU(n, 1) and Spin(2n, 1) (by Taniguchi [46]).

From these calculations, we know their Bernstein degrees. The Bernstein degrees of (1)–(3) are all four. Those of (4) are twice the multiplicities, which are obtained by Chang (see § 2.5). In other words, the degrees of the associated variety of large discrete series representations of SU(n, 1) and Spin(2n, 1) are two (cf. Lemma 1.1).

2.7. Minimal representation. — In this subsection, we will give Bernstein degrees of so-called minimal representations. Here we only consider non-Hermitian symmetric space G/K, though the arguments below equally works well for general situations.

If G/K is non-Hermitian, G has a minimal representation if and only if G/K is in the following list.

– Classical case : SO(p, q)/SO(p) × SO(q) where p ≥ q ≥ 3, p + q ∈ 2ℤ or p ∈ 2ℤ, q = 3.

– Exceptional case : The following 8 cases.

\begin{matrix}F_{4,4}/Sp(3) × SU(2) & G_{2}/SO(4) & E_{6,4}/SU(2) × SU(6) & E_{6,6}/Sp(4) \\ E_{7,4}/Spin(12) × SU(2) & E_{7,7}/SU(8) & E_{8,4}/E_{7} × SU(2) & E_{8,8}/Spin(16) \end{matrix}

Take the minimal nilpotent G_{ℂ}-orbit O_{min} ⊂ 𝔤. Then in this case O_{min} ∩ 𝔭 =: Y is a single nilpotent K_{ℂ}-orbit, which is minimal among non-zero nilpotent K_{ℂ}-orbits in 𝔭 with respect to the closure relation.

Theorem 2.7 (Vogan). — Let π_{min} be a minimal representation of G. Then there exists some weight ν such that

π_{min}|_{K} ≃ ⨁_{m≥0} τ_{mψ+ν},

where ψ is the highest weight of 𝔭 (= the highest root), and τ_{mψ+ν} is the irreducible representation of K with highest weight mψ + ν.

Remark 2.8. — The weight ν is the highest weight of the minimal K-type of π_{min}. For an explicit description of ν, we refer to Table 1 of [5] and the references cited there.

Put Δ_{c}^{+}(ψ) = {α ∈ Δ_{c}^{+} | ⟨ψ, α⟩ ≠ 0}, where Δ_{c}^{+} denotes the totality of positive compact roots. For α ∈ Δ_{c}^{+}, note that ⟨ψ, α⟩ ≠ 0 if and only if 2⟨ψ, α⟩/⟨ψ, ψ⟩ = 1 ([27, Lemma 2.2]).

Proposition 2.9. — With the above notation, we have

Dim (π_{min}) = #Δ_{c}^{+}(ψ) + 1 = dim_{ℂ} O_{min}/2 = dim_{ℂ} Y,
Deg (π_{min}) = deg \overline{Y} = (#Δ_{c}^{+}(ψ))! ∏_{α∈Δ_{c}^{+}(ψ)} \frac{⟨ψ, ψ⟩}{2⟨ρ_{c}, α⟩},
AC (π_{min}) = [\overline{Y}].

Proof. — From the explicit description of ν (cf. [5, Table 1]), we conclude that ⟨ν, α⟩ = 0 holds for each positive compact root α ∉ Δ_{c}^{+}(ψ). Also we know a good filtration of (π_{min}, V) is given by

V_{n} = ⨁_{m≤n} τ_{mψ+ν}

(see [50]). Put d = #Δ_{c}^{+}(ψ) + 1. By Weyl’s dimension formula, we calculate the dimension of gr_{n}V as

dim gr_{n}V = dim τ_{nψ+ν} = ∏_{α∈Δ_{c}^{+}} \frac{⟨nψ + ν + ρ_{c}, α⟩}{⟨ρ_{c}, α⟩}

= n^{d−1} ∏_{α∈Δ_{c}^{+}(ψ)} \frac{⟨ψ, α⟩}{⟨ρ_{c}, α⟩} ∏_{α∉Δ_{c}^{+}(ψ)} \frac{⟨ν + ρ_{c}, α⟩}{⟨ρ_{c}, α⟩} + (lower order terms of n)

= \frac{1}{(d − 1)!} \{(d − 1)! ∏_{α∈Δ_{c}^{+}(ψ)} \frac{⟨ψ, ψ⟩}{2⟨ρ_{c}, α⟩}\} n^{d−1} + (lower order terms of n)

From the last formula, we can read off the desired formulas of dimension and degree.

On the other hand, since Y is a K_{ℂ}-orbit through a highest weight vector in 𝔭, \overline{Y} is a highest weight variety (see [48]). Then the decomposition of the coordinate ring as a K_{ℂ}-module becomes

ℂ[\overline{Y}] ≃ ⨁_{m≥0} τ_{mψ},

with grading given by m. By the same method as above, we conclude that deg \overline{Y} is equal to Deg π_{min} which proves that AC (π_{min}) = [\overline{Y}].

3. Reductive dual pair

Let W be a real symplectic space of dimension 2N. We put G = Sp(W) = Sp(2N, ℝ) and G̃ = Mp(2N, ℝ), the metaplectic double cover of G (see [47, § I.2] for example). A pair of reductive subgroups (G_{1}, G_{2}) of G is called a reductive dual pair if they are mutually commutant to each other in G (see [22], for example). We denote by (G̃_{1}, G̃_{2}) the inverse image of these subgroups under the covering map G̃ → G. Then they are also commutant to each other in G̃.

Let us assume that the pair (G_{1}, G_{2}) is irreducible (see [25, § 4] for definition). Then there are two possibilities.

(I) The pair (G_{1}, G_{2}) jointly acts on W. This action is irreducible.

(II) There exists a maximally totally isotropic space U of W, such that W = U ⊕U^{∗} gives the irreducible decomposition with respect to the joint action of the pair.

In the following, we only treat the dual pair of type (I), so that we assume that the joint action of G_{1} × G_{2} on W is irreducible. Then, by the irreducibility, there exist a division algebra D over ℝ and vector spaces V_{1}/D and D\V_{2} over D for which the following two properties hold. First, W is the tensor product of V_{1} and V_{2} over D :

W = V_{1} ⊗_{D} V_{2}.

Second, G_{i} (i = 1, 2) acts on V_{i} irreducibly as D-linear transformations. We put

2n = dim_{ℝ} V_{1}, m = dim_{D} V_{2},  (3.12)

hence dim_{ℝ} W = 2N = 2nm. Note that the division algebra is given by D ≃ End_{G_{1}}(V_{1}) ≃ End_{G_{2}}(V_{2}).

Since W carries a symplectic structure (and (G_{1}, G_{2}) is a pair in the symplectic group Sp(W)), it produces some additional structure on the vector spaces V_{1} and V_{2}. Namely, we have the following.

First, there exists an involution ι of D (possibly trivial). Second, V_{i} (i = 1, 2) carries a sesqui-linear form ( , )_{i} which is invariant under G_{i}. One of the forms, say ( , )_{1}, is skew-Hermitian with respect to the involution ι and the other ( , )_{2} is Hermitian; and the original symplectic form ⟨ , ⟩_{W} on W is given by the product of these forms:

⟨ , ⟩_{W} = Re ( , )_{1} ⊗_{D} ( , )_{2}.

Moreover, the group G_{i} is the full isometry group with respect to ( , )_{i}. In the following, we always assume that ( , )_{1} is skew-Hermitian, and ( , )_{2} is Hermitian.

Here is a table (Table 1) of such pairs borrowed from [25, Table 4.1].

In this paper, we only treat the case where one of the pair, say G_{2}, is compact. In fact, we have the following explicit cases in Table 2 in mind. However, we try to keep general situation whenever possible. In any case, G_{2} is always assumed to be compact.

Let us specify an explicit embedding of (G_{1}, G_{2}) into G = Sp(2nm, ℝ). Although our arguments below are fairly general, sometimes it is convenient to use a concrete realization. In each of three cases, we will give a symplectic vector space ℝ^{2nm}

Table 1. Reductive dual pairs of type (I).

(D, ι) G  (G_{1}, G_{2})
(ℝ, 1) Sp(2nm, ℝ) (Sp(2n, ℝ), O(p, q)) m = p + q
(ℂ, 1) Sp(4nm, ℝ) (Sp(2n, ℂ), O(m, ℂ))
(ℂ, ) Sp(2nm, ℝ) (U(p, q), U(r, s))  n = p + q, m = r + s
(ℍ, ) Sp(2nm, ℝ) (O^{∗}(2p), Sp(r, s))  n = 2p, m = r + s

Table 2. Reductive dual pairs (G_{1}, G_{2}) with G_{2} being compact.

(D, ι) G  (G_{1}, G_{2})
Case (Sp, O) (ℝ, 1) Sp(2nm, ℝ) (Sp(2n, ℝ), O(m))
Case (U, U) (ℂ, ) Sp(2nm, ℝ) (U(p, q), U(m))  n = p + q
Case (O^{∗}, Sp) (ℍ, ) Sp(2nm, ℝ) (O^{∗}(2p), Sp(2m)) n = 2p

endowed with an explicit symplectic form in terms of invariant bilinear forms of V_{1} and V_{2}. This will determine the group G = Sp(2nm, ℝ).

Case (Sp, O). Let ℝ^{2n} be a symplectic vector space with a symplectic form

(u, v)_{1} = ^{t}uJ_{n}v (u, v ∈ ℝ^{2n}), J_{n} = [ \begin{matrix}0 & −1_{n} \\ 1_{n} & 0 \end{matrix} ] ,  (3.13)

and consider G_{1} = Sp(2n, ℝ) as the isometry group of (ℝ^{2n}, ( , )_{1}). For G_{2} = O(m), we take the standard Euclidean bilinear form (u, v)_{2} = ^{t}uv (u, v ∈ ℝ^{m}), and consider O(m) = O(ℝ^{m}, ( , )_{2}). Then the tensor product W = ℝ^{2n} ⊗_{ℝ} ℝ^{m} with a symplectic form

⟨ , ⟩_{W} = ( , )_{1} ⊗_{ℝ} ( , )_{2}

gives the embedding (G_{1}, G_{2}) ↪ G = Sp(W, ⟨ , ⟩_{W}).

Let us see this embedding infinitesimally. So, first consider 𝔰𝔭(2n, ℝ):

𝔰𝔭(2n, ℝ) = {Z ∈ 𝔤𝔩(2n, ℝ) | ^{t}ZJ_{n} + J_{n}Z = 0}
= \{( \begin{matrix}X_{11} & X_{12} \\ X_{21} & −^{t}X_{11} \end{matrix} ) | \begin{matrix}X_{11} ∈ 𝔤𝔩(n, ℝ), \\ X_{12}, X_{21} ∈ Sym (n, ℝ) \end{matrix} \} . (3.14)

Then it is embedded into larger 𝔰𝔭(2nm, ℝ) as

𝔰𝔭(2n, ℝ) ∋ ( \begin{matrix}X_{11} & X_{12} \\ X_{21} & −^{t}X_{11} \end{matrix} ) ⟼ ( \begin{matrix}X_{11}^{⊕m} & X_{12}^{⊕m} \\ X_{21}^{⊕m} & −^{t}X_{11}^{⊕m} \end{matrix} ) ∈ 𝔰𝔭(2nm, ℝ), (3.15)

where

X^{⊕m} = diag (X, X, …, X) (m-times).

Similarly, 𝔬(m, ℝ) = Alt (m, ℝ) is embedded into 𝔰𝔭(2nm, ℝ) as

𝔬(m, ℝ) ∋ X ⟼ ( \begin{matrix}X ∗ 1_{n} & 0 \\ 0 & X ∗ 1_{n} \end{matrix} ) ,

where

X ∗ A = ( \begin{matrix}x_{11}A & x_{12}A & ⋯ & x_{1m}A \\ x_{21}A & x_{22}A & ⋯ & x_{2m}A \\ ⋮ & ⋮ & & ⋮ \\ x_{m1}A & x_{m2}A & ⋯ & x_{mm}A \end{matrix} ) .  (3.16)

Case (U, U). Consider an indefinite Hermitian form ( , )_{1} on ℂ^{n} of signature (p, q) (n = p + q):

(u, v)_{1} = ^{t}\overline{u}I_{p,q}v (u, v ∈ ℂ^{n}), I_{p,q} = [ \begin{matrix}1_{p} & 0 \\ 0 & −1_{q} \end{matrix} ] .  (3.17)

Then, G_{1} = U(p, q) is the full isometry group of (ℂ^{n}, ( , )_{1}). Also we take a definite Hermitian form ( , )_{2} on ℂ^{m} as (u, v)_{2} = ^{t}\overline{u}v (u, v ∈ ℂ^{m}). This determines the unitary group G_{2} = U(m). Then the tensor product W = ℂ^{n} ⊗_{ℂ} ℂ^{m} naturally inherits a Hermitian form ( , )_{1} ⊗_{ℂ} ( , )_{2}. We make use of its imaginary part to define a symplectic form on W ≃ ℝ^{2nm}:

⟨ , ⟩_{W} = Re (\sqrt{−1} ( , )_{1} ⊗_{ℂ} ( , )_{2}).

The form ⟨ , ⟩_{W} is clearly non-degenerate and it defines the isometry group G = Sp(W, ⟨ , ⟩_{W}) ≃ Sp(2nm, ℝ).

Under our explicit realization of U(p, q), its Lie algebra is given as

𝔲(p, q) = \{Z ∈ 𝔤𝔩(p + q, ℂ) | ^{t}\overline{Z}I_{p,q} + I_{p,q}Z = 0\}
= \{Z = ( \begin{matrix}Z_{11} & Z_{12} \\ ^{t}\overline{Z_{12}} & Z_{22} \end{matrix}) | \begin{matrix}Z_{11} ∈ skew-Her (p, ℂ) \\ Z_{22} ∈ skew-Her (q, ℂ) \\ Z_{12} ∈ M(p, q, ℂ) \end{matrix}\} .  (3.18)

Let us write Z = X + \sqrt{−1} Y with X, Y ∈ M(n, ℝ). Then an explicit embedding into 𝔰𝔭(2n, ℝ) is given by

𝔲(p, q) ∋ X + \sqrt{−1} Y ↦ (\begin{matrix}X & −Y I_{p,q} \\ I_{p,q}Y & I_{p,q}XI_{p,q} \end{matrix}) ∈ 𝔰𝔭(2n, ℝ).  (3.19)

Now the above embedding composed by the embedding (3.15) will give the desired realization of 𝔲(p, q) in 𝔰𝔭(2nm, ℝ).

On the other hand, the compact companion 𝔲(m) is embedded into 𝔰𝔭(2nm, ℝ) as

𝔲(m) = Alt (m, ℝ) + \sqrt{−1} Sym (m, ℝ) ∋ X + \sqrt{−1} Y
⟼ ( \begin{matrix}X ∗ 1_{n} & −Y ∗ I_{p,q} \\ Y ∗ I_{p,q} & X ∗ 1_{n} \end{matrix} ) ∈ 𝔰𝔭(2nm, ℝ).  (3.20)

Case (O^{∗}, Sp). Let O(2p, ℂ) be the complex orthogonal group with respect to the following bilinear form

(u, v) = ^{t}uS_{p}v (u, v ∈ ℂ^{2p}), S_{p} = (\begin{matrix}0 & 1_{p} \\ 1_{p} & 0 \end{matrix}) .

We realize G_{1} = O^{∗}(2p) as a subgroup of O(2p, ℂ), namely,

O^{∗}(2p) = O(2p, ℂ) ∩ U(p, p) ⊂ M(2p, ℂ),  (3.21)

where U(p, p) is realized in the same way as Case (U, U). Similarly, we realize G_{2} = Sp(2m) as a compact subgroup of Sp(2m, ℂ) :

Sp(2m) = Sp(2m, ℂ) ∩ U(2m) ⊂ M(2m, ℂ).

First we describe embedding of 𝔬^{∗}(2p) into 𝔰𝔭(2n, ℝ) (n = 2p). Our realization of O^{∗}(2p) gives its Lie algebra as

𝔬^{∗}(2p) = \{Z ∈ 𝔤𝔩(2p, ℂ) | ^{t}\overline{Z}I_{p,p} + I_{p,p}Z = 0, ^{t}ZS_{p} + S_{p}Z = 0\}

= \{(\begin{matrix}X & −Y \\ \overline{Y} & \overline{X} \end{matrix} ) | X ∈ skew-Her (p, ℂ), Y ∈ Alt (p, ℂ)\} ,

where I_{p,p} is given by (3.17).

It is subtle to describe a symplectic form of the larger Sp(2nm, ℝ) (n = 2p) in terms of the original (skew-)Hermitian forms over ℍ which define O^{∗}(2p) and Sp(2m) as the full isometry groups. Instead, we give here only an explicit embedding of O^{∗}(2p) infinitesimally. Let us write X = X_{1} + \sqrt{−1} X_{2} and Y = Y_{1} + \sqrt{−1} Y_{2} with real matrices X_{i}, Y_{i} (i = 1, 2). Then, the infinitesimal embedding of 𝔬^{∗}(2p) into 𝔰𝔭(2n, ℝ) is given by

𝔬^{∗}(2p) ∋ (\begin{matrix}X & −Y \\ \overline{Y} & \overline{X} \end{matrix} ) ↦ ( \begin{matrix}X_{1} & −Y_{1} & −X_{2} & −Y_{2} \\ Y_{1} & X_{1} & Y_{2} & −X_{2} \\ X_{2} & −Y_{2} & X_{1} & Y_{1} \\ Y_{2} & X_{2} & −Y_{1} & X_{1} \end{matrix} ) ∈ 𝔰𝔭(2n, ℝ).  (3.22)

This embedding is compatible with the embedding given in Case (U, U), i.e., we have a sequence of subgroups

O^{∗}(2p) ↪ U(p, p) ↪ Sp(2n, ℝ).

The embedding into the larger 𝔰𝔭(2nm, ℝ) is given by the composition of (3.15) and (3.22).

Let us see the embedding of the compact companion 𝔰𝔭(2m). Its Lie algebra becomes

𝔰𝔭(2m) = \{Z ∈ 𝔤𝔩(2m, ℂ) | ^{t}\overline{Z} + Z = 0, ^{t}ZJ_{m} + J_{m}Z = 0\}

= \{Z = (\begin{matrix}X & −Y \\ \overline{Y} & \overline{X} \end{matrix} ) | X ∈ skew-Her (m, ℂ), Y ∈ Sym (m, ℂ)\} .

If we denote Z = A + \sqrt{−1} B with real matrices A and B, then the embedding is given by

𝔰𝔭(2m) ∋ Z = A + \sqrt{−1} B ⟼ (\begin{matrix}A ∗ 1_{p} & −B ∗ 1_{p} \\ B ∗ 1_{p} & A ∗ 1_{p} \end{matrix}) ∈ 𝔰𝔭(2nm, ℝ).

4. Fock realization of Weil representation

Let ω be the Weil representation of Mp(2n, ℝ), the metaplectic double cover of Sp(2n, ℝ). Weil representation plays central roles in many fields, and a large amount of results are known. For example, see [24], [25], [28], [47], [40], etc. We introduce here, among all, explicit realization of Harish-Chandra module of ω on a polynomial ring (e.g., see [25] and [9]). It is called Fock model.

For the time being, we write G = Sp(2n, ℝ) and G̃ = Mp(2n, ℝ). Since we only consider Harish-Chandra modules, in fact we do not need entire Mp(2n, ℝ) but only its complexified Lie algebra 𝔤 = 𝔰𝔭(2n, ℂ) and a maximal compact subgroup K̃ = U(n) . We fix a maximal compact subgroup K ≃ U(n) in Sp(2n, ℝ) as follows. Put

Sp(2n, ℝ) = \{g ∈ GL(2n, ℝ) | ^{t}gJ_{n}g = J_{n}\} , J_{n} = [ \begin{matrix}0 & −1_{n} \\ 1_{n} & 0 \end{matrix} ] .  (4.23)

Then K is given as

K = \{[ \begin{matrix}a & −b \\ b & a \end{matrix} ] | a, b ∈ M(n, ℝ), a + ib ∈ U(n)\} .  (4.24)

We identify K and U(n) as above and sometimes we will write a + ib ∈ K. Let 𝔤_{0} = 𝔨_{0} +𝔭_{0} be the corresponding Cartan decomposition, and 𝔤 = 𝔨+𝔭 its complexification.

Let E_{ij} be the matrix unit, and put

F_{ij} := E_{ij} − E_{ji}, G_{ij} := E_{ij} + E_{ji}.

Then it is easy to see that a basis of 𝔨 is given by

A_{ij} := (\begin{matrix}F_{ij} & O_{n} \\ O_{n} & F_{ij} \end{matrix}) (1 ≤ i < j ≤ n), B_{ij} := (\begin{matrix}O_{n} & −G_{ij} \\ G_{ij} & O_{n} \end{matrix}) (1 ≤ i ≤ j ≤ n), (4.25)

and that of 𝔭 is given by

C_{ij} := (\begin{matrix}G_{ij} & O_{n} \\ O_{n} & −G_{ij} \end{matrix}) (1 ≤ i ≤ j ≤ n), D_{ij} := (\begin{matrix}O_{n} & G_{ij} \\ G_{ij} & O_{n} \end{matrix}) (1 ≤ i ≤ j ≤ n). (4.26)

The representation space of ω in Fock model is a polynomial ring in n variables. Here we only give the explicit action of each basis element on the polynomial ring ℂ[x_{1}, x_{2}, …, x_{n}]:

\begin{matrix}ω(A_{ij}) = x_{i}∂_{x_{j}} − x_{j}∂_{x_{i}}, & ω(B_{ij}) = \sqrt{−1} (x_{i}∂_{x_{j}} + ∂_{x_{i}}x_{j}), \\ ω(C_{ij}) = 2∂_{x_{i}}∂_{x_{j}} − \frac{1}{2}x_{i}x_{j}, & ω(D_{ij}) = −\sqrt{−1} (2∂_{x_{i}}∂_{x_{j}} + \frac{1}{2}x_{i}x_{j}). \end{matrix}  (4.27)

The action of K̃ on ℂ[x_{1}, …, x_{n}] is the symmetric tensor product of the natural representation of U(n) on ℂ^{n} tensored by det^{1/2}, which requires the double cover K̃.

Put H_{i} = −\sqrt{−1} B_{ii}/2 and let

𝔱 = {∑_{i=1}^{n} t_{i}H_{i} | t_{i} ∈ ℂ}

be a Cartan subalgebra of 𝔤 contained in 𝔨. We define ε_{j} ∈ 𝔱^{∗} as ε_{j}(H_{i}) = δ_{ij}. Then the root system Δ(𝔤, 𝔱) is given by

Δ(𝔤, 𝔱) = {ε_{i} − ε_{j} | 1 ≤ i ≠ j ≤ n} ∪ {±(ε_{i} + ε_{j}) | 1 ≤ i ≤ j ≤ n},

where ε_{i} − ε_{j} is a compact root while ±(ε_{i} + ε_{j}) is non-compact. We take a positive system in the standard way :

Δ^{+}(𝔤, 𝔱) = {ε_{i} − ε_{j} | 1 ≤ i < j ≤ n} ∪ {ε_{i} + ε_{j} | 1 ≤ i ≤ j ≤ n}.

Then root vectors X_{α} (α ∈ Δ(𝔤, 𝔱)) and its action on Weil representation are given as

H_{i} = −\frac{\sqrt{−1}}{2} B_{ii},  ω(H_{i}) = x_{i}∂_{x_{i}} + \frac{1}{2},  (4.28)
X_{ε_{i}−ε_{j}} = \frac{1}{2}(A_{ij} − \sqrt{−1} B_{ij}) (i ≠ j),  ω(X_{ε_{i}−ε_{j}}) = x_{i}∂_{x_{j}},  (4.29)
X_{ε_{i}+ε_{j}} = −\frac{1}{2}(C_{ij} − \sqrt{−1} D_{ij}),  ω(X_{ε_{i}+ε_{j}}) = \frac{1}{2}x_{i}x_{j},  (4.30)
X_{−ε_{i}−ε_{j}} = \frac{1}{2}(C_{ij} + \sqrt{−1} D_{ij}),  ω(X_{−ε_{i}−ε_{j}}) = 2∂_{x_{i}}∂_{x_{j}}.  (4.31)

Note that

𝔨 ≃ 𝔤𝔩(n, ℂ) ∋ E_{ij} ↔ \{ \begin{matrix}X_{ε_{i}−ε_{j}} = x_{i}∂_{x_{j}} & (i ≠ j), \\ H_{i} = x_{i}∂_{x_{i}} + \frac{1}{2} & (i = j). \end{matrix}  (4.32)

We write

Δ_{n}^{+} = {ε_{i} + ε_{j} | 1 ≤ i, j ≤ n}, Δ_{n} = Δ_{n}^{+} ⊔ (−Δ_{n}^{+}),

the set of non-compact roots, and

Δ_{k}^{+} = {ε_{i} − ε_{j} | 1 ≤ i < j ≤ n}, Δ_{k} = Δ_{k}^{+} ⊔ (−Δ_{k}^{+}),

the set of compact roots. Then 𝔭 decomposes up into two K-irreducible components 𝔭^{±} given by

𝔭^{±} = ∑_{±α∈Δ_{n}^{+}} 𝔤_{α},

where 𝔤_{α} denotes the roots space corresponding to α. Note that ω(𝔭^{−}) is realized as differential operators of degree two, and that ω(𝔭^{+}) is the multiplication by homogeneous polynomials of degree two. So 𝔭^{±} increases/decreases the degree of the representation space ℂ[x_{1}, x_{2}, …, x_{n}] by 2, while ω(𝔨) keeps the degree stable.

5. Unitary lowest weight representations

Let Ω be the Weil representation of G̃ = Mp(2N, ℝ) (N = nm) and consider reductive dual pair (G_{1}, G_{2}) of compact type in G = Sp(2N, ℝ). In the following, we often write G = G_{1} without the subscription 1. In fact, our main concern is on the irreducible infinite dimensional representations of G = G_{1} which appear in the restriction of Weil representation Ω. Moreover, we assume that G_{2} is contained in the specified maximal compact subgroup K ≃ U(N) of G given in the former section (cf. (4.24)). Each of our three cases (and their realization) clearly satisfies this condition.

For a subgroup H ⊂ G, we denote by H̃ the inverse image of H in G̃ of the covering map G̃ → G, and call it the metaplectic cover of H by abuse of terminology. Since the metaplectic covers G̃_{1} and G̃_{2} commute with each other, we have a natural projection G̃_{1} × G̃_{2} → G̃_{1} · G̃_{2} (product in G̃). By this projection, we consider the restriction Ω|_{G̃_{1}·G̃_{2}} as a representation of G̃_{1} × G̃_{2}. Then we have a discrete and multiplicity free decomposition

Ω ≃ \begin{matrix}∑ & ⊕ \\ σ̃∈Irr(G̃_{2}) \end{matrix} L(σ̃ ⊗ χ^{−1}) ⊠ σ̃  (5.33)

as a representation of G̃_{1} × G̃_{2}. Here we denote an irreducible representation of G̃ = G̃_{1} corresponding to σ̃ ∈ Irr(G̃_{2}) by L(σ̃ ⊗ χ^{−1}), where χ ∈ Irr(G̃_{2}) is the unique one-dimensional character which appears in Ω|G̃_{2} (cf. Theorem 4.3 in [25]).

To be more specific, we argue like this. The representation space of Ω is realized on a polynomial ring of N = nm variables. We consider it as the polynomial ring on the dual space of n × m matrices M_{n,m} over ℂ. Since the one-dimensional space of constant polynomials in ℂ[M_{n,m}^{∗}] is preserved by the action of K̃, it is also preserved by G̃_{2} because of our assumption. Hence it gives the one-dimensional character and we denote it by χ ∈ Irr(G̃_{2}).

The representation L(σ̃ ⊗ χ^{−1}) is possibly zero, and if it is not zero, then the representation σ = σ̃ ⊗χ^{−1} factors through to the representation of G_{2}. Therefore we write L(σ) = L(σ̃ ⊗ χ^{−1}) for σ ∈ Irr(G_{2}). The decomposition (5.33) can be rewritten as

Ω ≃ \begin{matrix}∑ & ⊕ \\ σ∈Irr(G_{2}) \end{matrix} L(σ) ⊠ (σ ⊗ χ).  (5.34)

In the following, as explained above, we always twist the representation σ̃ ∈ Irr(G̃_{2}) by χ, and consider σ = σ̃ ⊗ χ^{−1} as the representation of G_{2}. This twist might be sometimes misleading, but it reduces considerable amount of untwisting. For example, under this convention, we have L(1_{G_{2}}) ≠ 0, where 1_{G_{2}} denotes the trivial representation of G_{2}. This representation turns out to be strongly related to geometric properties of nilpotent orbits.

It is known that L(σ) (σ ∈ Irr(G_{2})) is an irreducible unitary lowest weight module of G̃, if it is not zero (cf. Theorem 4.4 in [25]). Every irreducible unitary lowest weight module of G̃ arises in this manner if G = Sp(2n, ℝ) or U(p, q) and the compact companion G_{2} moves all the possible rank. If G is O^{∗}(2p), there are other unitary lowest weight modules which can not be obtained in this manner ([9], [11]).

In our cases, the compact subgroup G_{2} naturally acts on its defining vector space V_{2} keeping the non-degenerate Hermitian form ( , )_{2} invariant (see § 3). Put 2n = dim_{ℝ} V_{1} and m = dim_{D} V_{2} as in (3.12). Then we can realize G in a smaller symplectic group: G ↪ Sp(2n, ℝ), putting m = 1.

Let us denote the Weil representation of the smaller group Mp(2n, ℝ) by ω. Then, it is easy to see that Ω ≃ ω^{⊗m} as a representation of Mp(2n, ℝ), and the Harish-Chandra (𝔤, K)-module of the Weil representation Ω (resp. ω) is realized on the polynomial ring ℂ[M_{n,m}^{∗}] ≃ ⊗^{m}ℂ[(ℂ^{n})^{∗}] over n × m matrices (resp. the polynomial ring ℂ[(ℂ^{n})^{∗}]). Note that we take a contragredient representation of M_{n,m} rather than M_{n,m} itself.

Let K ⊂ G be a maximal compact subgroup of G which lives in U(n) ⊂ Sp(2n, ℝ), where U(n) is a maximal compact subgroup of Sp(2n, ℝ) (cf. (4.24)). We will explain briefly how we get K-type decomposition of L(σ) for each σ ∈ Irr(G_{2}). Note that the product K · G_{2} is compact and that it is contained in the maximal compact subgroup K ≃ U(nm) of G = Sp(2nm, ℝ). It is well-known that K-types of Ω can be described as

Ω|K̃ = \begin{matrix}∞ \\ ∑ & ⊕ \\ k=0 \end{matrix} τ(kψ + 1/2 𝕀), 𝕀 = (1, 1, …, 1),  (5.35)

where ψ is the highest weight of the natural (or defining) representation of K ≃ U(nm) on ℂ^{nm} and τ(λ) is an irreducible finite-dimensional representation of K with highest weight λ. Note that the representation space of τ(kψ + 1/2 𝕀) coincides with the space of homogeneous polynomials of degree k. Decompose τ(kψ +1/2 𝕀) by the joint action of K̃ × G̃_{2}:

τ(kψ + 1/2 𝕀)|_{K̃×G̃_{2}} = \begin{matrix}∑ & ⊕ \\ τ_{1}∈Irr(K̃), σ∈Irr(G_{2}) \end{matrix} m_{k}(τ_{1}, σ) τ_{1} ⊠ σ̃ (σ̃ = σ ⊗ χ).

Note that we again use the projection map K̃ × G̃_{2} → K̃ · G̃_{2} ⊂ K̃ here. In particular, the one-dimensional space τ(1/2 𝕀) is decomposed as

τ(1/2 𝕀)|_{K̃×G̃_{2}} = χ_{1} ⊠ χ.

In other words, the multiplicity for k = 0 has the property

m_{0}(τ_{1}, σ) = \{ \begin{matrix}1 & τ_{1} = χ_{1}, σ = 1_{G_{2}} \\ 0 & otherwise. \end{matrix}

The explicit form of χ and χ_{1} is given in Section 7 after we fix the embedding K ⊂ K.

Since L(σ) consists of the space of multiplicities of σ̃ in Ω, we get K-type decomposition of L(σ) as

L(σ)|K̃ = \begin{matrix}∑ & ⊕ \\ τ_{1}∈Irr(K̃) \end{matrix} ∑_{k=0}^{∞} m_{k}(τ_{1}, σ) τ_{1},  (5.36)

where the sum ∑_{k=0}^{∞} m_{k}(τ_{1}, σ) is necessarily finite.

Let k_{σ} be the lowest possible degree of σ̃-isotypic component of ℂ[M_{n,m}^{∗}]. We define the Poincaré series of L(σ) in terms of the multiplicity m_{k}(τ_{1}, σ) as

P (L(σ); t^{2}) = t^{−k_{σ}} ∑_{k=0}^{∞} ∑_{τ_{1}∈Irr(K̃)} m_{k}(τ_{1}, σ) dim τ_{1} t^{k}.  (5.37)

Note that the action of 𝔭^{+} increases the degree k by two (cf. (4.30)), so we write P (L(σ); t^{2}) instead of P (L(σ); t). We denote the center of K̃ by Z(K̃). We know that Z(K̃) is isomorphic to U(1) and there exists an element H in the Lie algebra of Z(K̃) such that Ω(H) acts on the space of homogeneous polynomials of degree k by k + nm/2. Indeed, H = ∑_{i=1}^{nm} H_{i} with the notation (4.32). The operator Ω(H) is semisimple, and the decomposition into the H-isotypic components is given by (5.35). Moreover, the natural embedding K̃ ⊂ K̃ induces an isomorphism between the Lie algebra of Z(K̃) and that of Z(K̃). We denote the element in the Lie algebra of Z(K̃) corresponding to H by H′. Then the formal character of L(σ) on the compact Cartan subgroup restricted to the center of K̃ can be expressed by the Poincaré series:

trace_{L(σ)}t^{H′} = t^{k_{σ}+nm/2}P (L(σ); t^{2}).  (5.38)

To get explicit multiplicity formulas, we are involved in case-by-case analysis.

6. Description of K-types of the lowest weight modules

Assume that the pair (G_{1}, G_{2}) is in the stable range where G_{2} is the smaller member. This means that m ≤ ℝ-rank G_{1}, where m = dim_{D} V_{2} (cf. §3). Take σ ∈ Irr(G_{2}) for which L(σ) is not zero, and put σ̃ = σ ⊗ χ as above. Let us describe K-type decomposition of L(σ) in each explicit cases.

6.1. Case (Sp, O). — Assume that m ≤ n = ℝ-rank Sp(2n, ℝ). This means the pair (Sp(2n, ℝ), O(m)) is in the stable range. As before, we shall write G = G_{1} = Sp(2n, ℝ) and G = Sp(2nm, ℝ).

Let K = U(n) be a maximal compact subgroup of G = Sp(2n, ℝ) which is realized in the standard way (cf. (4.24)). Let K = U(n × m) ⊂ G act on M_{n,m} = M(n, m, ℂ) naturally as unitary transformation group. The product group U(n) × O(m) acts on M_{n,m} naturally as

(k, h)X = kX^{t}h ((k, h) ∈ U(n) × O(m), X ∈ M_{n,m}).  (6.39)

Since the action is also unitary, it induces a map U(n) × O(m) → U(nm) = K. The image of the above map coincides with K · G_{2}. Note that the kernel of the map is {(±1_{n}, ±1_{m}) ∈ U(n) × O(m)}.

The metaplectic cover K̃ acts on M_{n,m} as the composition of the projection K̃ → K and the natural action of the unitary group K = U(nm). This action induces the representation of K̃ on the polynomial ring ℂ[M_{n,m}^{∗}], which is isomorphic to the symmetric tensor of M_{n,m}. By the formula (4.32), we conclude that the action of K̃ on ℂ[M_{n,m}^{∗}] via Weil representation Ω is twisted by det^{1/2}. We shall denote this representation by ℂ[M_{n,m}^{∗}] ⊗ det^{1/2}. Therefore K̃ acts on ℂ[M_{n,m}^{∗}] as ℂ[M_{n,m}^{∗}] ⊗ det^{m/2} and G̃_{2} acts as ℂ[M_{n,m}^{∗}] ⊗ det^{n/2}. So the one-dimensional representation χ_{1} of K̃ coincides with det^{m/2}, and the one-dimensional representation χ ∈ Irr(G̃_{2}) coincides with det^{n/2}. However, we should be more precise about χ because G_{2} = O(m) is not connected.

The metaplectic cover K̃ has a realization

K̃ = {(k, z) ∈ K × ℂ^{×} | det k = z^{2}}

and the representation det^{1/2} of K̃ is given by the map (k, z) ↦ z. Then the subgroup G̃_{2} is realized as

G̃_{2} = {(k, z) ∈ G_{2} × ℂ^{×} | det^{n}k = z^{2}}

and its character χ = det^{n/2} is given by χ(k, z) = z. The identity component of G̃_{2} is

(G̃_{2})_{0} = {(k, z) ∈ G_{2} × ℂ^{×} | det k = 1, z = 1} ≃ SO(m).

The map (k, z) ↦ (det k, z) induces the isomorphism of the component group G̃_{2}/(G̃_{2})_{0} onto the group

A(G_{2}) = {(t, z) ∈ ℤ_{2} × ℂ^{×} | t^{n} = z^{2}}

of order four. Since the one-dimensional character χ is trivial on the identity component (G̃_{2})_{0}, it induces the character of the component group A(G_{2}). We denote it by the same letter χ, then χ(t, z) = z. First, we consider the case where n is odd. Then A(G_{2}) = \{(ζ^{2}, ζ) | ζ = ±1, ±\sqrt{−1} \} ≅ ℤ_{4}. If we define ε = ((diag (1_{m−1}, −1), \sqrt{−1} ) ∈ G̃_{2}, then ε ∈ A(G_{2}) generates the component group ℤ_{4}. Then we see that

G̃_{2} ≃ SO(m) ⋊ ℤ_{4} if n is odd.

The character χ satisfies χ(ε) = \sqrt{−1} , which determines the character χ of ℤ_{4}. Second, let us consider the case where n is even. In this case,

G̃_{2} = G_{2} × ℤ_{2}  if n is even.  (6.40)

The character χ is trivial on G_{2} = O(m) and is non-trivial on ℤ_{2}.

By the argument in §5, we will get K-type decomposition of L(σ) if we know the explicit decomposition of ℂ[M_{n,m}^{∗}] ⊗ det^{1/2} as K̃ × G̃_{2}-module. We first consider the space ℂ[M_{n,m}^{∗}] as the usual symmetric tensor of the natural representation of the unitary group K = U(n × m), then afterwards we will twist it by det^{1/2} to fit it to the Weil representation Ω.

We extend the U(n) × O(m)-action (6.39) on M_{n,m} naturally to the U(n) × U(m)- action. It is well known (cf. [23]) that, as U(n)×U(m)-module, ℂ[M_{n,m}^{∗}] decomposes as

ℂ[M_{n,m}^{∗}]|_{U(n)×U(m)} ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} τ_{λ}^{(n)} ⊠ τ_{λ}^{(m)},

where P_{m} denotes the set of all partitions of length less than or equal to m. We make use of this decomposition. We identify K with U(n) above, and consider G_{2} = O(m) in U(m) in the standard manner, i.e., O(m) = U(m) ∩ GL(m, ℝ). Decompose τ_{λ}^{(m)} ∈ Irr(U(m)) as O(m)-module:

τ_{λ}^{(m)}|_{O(m)} ≃ \begin{matrix}∑ & ⊕ \\ σ∈Irr(O(m)) \end{matrix} m(λ, σ) σ.  (6.41)

Then we have a joint decomposition

ℂ[M_{n,m}^{∗}]|_{U(n)×O(m)} = \begin{matrix}∑ & ⊕ \\ σ∈Irr(O(m)) \end{matrix} \begin{cases}∑ & ⊕ \\ λ∈P_{m} \end{cases} m(λ, σ) τ_{λ}^{(n)}\} ⊠ σ.

So we completely know σ-isotypic component of ℂ[M_{n,m}^{∗}] in terms of the multiplicity m(λ, σ). Twist of this representation by det^{1/2} causes the twist by det^{m/2} ⊠ χ as a representation of K̃ × G̃_{2}. Therefore L(σ)|K̃ decomposes as

L(σ)|_{U(n)} ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} m(λ, σ) τ_{λ}^{(n)} ⊗ det^{m/2}.  (6.42)

This formula describes the multiplicities of K-types of L(σ) in the case of Case (Sp, O).

To describe the lowest weight and the lowest K-type of L(σ), we give a classification of Irr(O(m)) briefly. For more detailed discussion, see [23, § 3.6] for example. Let σ(μ) be an irreducible representation of SO(m) with highest weight μ.

Lemma 6.1. — Let σ be an irreducible representation of O(m).

(1) If σ|_{SO(m)} is irreducible, then σ and σ ⊗ det are mutually inequivalent.

(2) If σ|_{SO(m)} is reducible, then σ and σ ⊗ det are equivalent. In this case, m is necessarily even. Moreover, there exist positive integers μ_{1} ≥ μ_{2} ≥ ⋯ ≥ μ_{m/2} > 0 such that

σ|_{SO(m)} ≃ σ(μ^{+}) ⊕ σ(μ^{−}),

where μ^{+} = (μ_{1}, μ_{2}, …, μ_{m/2}) and μ^{−} = (μ_{1}, μ_{2}, …, −μ_{m/2}).

In case (1) in the above lemma, it is subtle to tell the difference between σ and σ ⊗det. However, since the difference causes strong influence on our result, we discuss this point.

Take a Cartan subalgebra 𝔥_{0} in 𝔬(m) as

𝔥_{0} = \{H = diag (a(θ_{1}), a(θ_{2}), …, a(θ_{[m/2]}), 0) | θ_{i} ∈ ℝ\}, a(θ) = (\begin{matrix}0 & −θ \\ θ & 0 \end{matrix}) ,

where the last 0 in the expression of H appears if and only if m is odd. We define ε_{j} ∈ 𝔥^{∗} as ε_{j}(H) = \sqrt{−1} θ_{j} in the above expression. Then, positive roots are given by

Δ^{+} = \{ \begin{matrix}\{ε_{i} ± ε_{j} | 1 ≤ i < j ≤ m/2\} & if m is even, \\ \{ε_{i} ± ε_{j} | 1 ≤ i < j ≤ [m/2]\} ⊔ \{ε_{j} | 1 ≤ j ≤ [m/2]\} & if m is odd. \end{matrix}

Assume that σ|_{SO(m)} be irreducible. Write σ|_{SO(m)} = σ(μ) for some highest

weight μ = ∑_{j=1}^{[m/2]} μ_{j}ε_{j}. Let δ = diag (1_{m−1}, −1) ∈ O(m)\SO(m). Then, σ|_{SO(m)} is irreducible if and only if the twisted representation σ(μ)^{δ} is equivalent to σ(μ). Consequently, the highest weight space of σ(μ) is preserved by the action of δ. In particular, if m is even, we get μ_{m/2} = 0.

Since δ^{2} = 1_{m}, its action on the highest weight space is the multiplication by ±1. If it is 1, we will write σ = σ(μ) by abuse of notation; if it is −1, then we denote σ = σ(μ) ⊗ det. Let k = ℓ(μ) so that μ_{k} > μ_{k+1} = 0. We put μ^{+} = μ if σ = σ(μ). If σ = σ(μ) ⊗ det, we add 1 to μ (m − 2k)-times after μ_{k}, i.e.,

μ^{+} = (μ_{1}, μ_{2}, …, μ_{k}, 1, …, 1) = (μ, 1^{m−2k}).

The following theorem is due to Kashiwara-Vergne [28] (see also [23, § 3.6]).

Theorem 6.2. — Assume that m ≤ n = ℝ-rank Sp(2n, ℝ). Then L(σ) is not zero for any σ ∈ Irr(O(m)) and it gives an irreducible unitary lowest weight module of Sp(2n, ℝ) . Let μ^{+} be as above, and extend μ^{+} to the weight of Sp(2n, ℝ) by adding zero. Then the lowest weight of L(σ) is given by

w_{K} (μ^{+} + \frac{m}{2}𝕀) ,

where w_{K} is the longest element of the Weyl group of K = U(n) and 𝕀 = (1, 1, …, 1). Consequently, the lowest K-type of L(σ) is τ(μ^{+}) ⊗ det^{m/2}.

From this theorem, we get the Poincaré series of L(σ) as

P (L(σ); t^{2}) = t^{−|μ^{+}|} ∑_{λ∈P_{m}} m(λ, σ) dim τ_{λ}^{(n)} t^{|λ|}.  (6.43)

Consider the special case where σ ∈ Irr(O(m)) is the trivial representation 1_{O(m)}.

Corollary 6.3. — We have the K-type decomposition of L(1_{O(m)}) as

L(1_{O(m)})|_{U(n)} ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} τ_{2λ}^{(n)} ⊗ det^{m/2},

where P_{m} is the set of all partitions such that ℓ(λ) ≤ m. The Poincaré series of L(1_{O(m)}) is given by

P (L(1_{O(m)}); t) = ∑_{λ∈P_{m}} dim τ_{2λ}^{(n)} t^{|λ|}.  (6.44)

Proof. — It is well-known that

m(λ, 1_{O(m)}) = \{ \begin{matrix}1 & if λ is an even partition, \\ 0 & otherwise. \end{matrix}

Apply this formula to (6.42) and (6.43).

6.2. Case (U, U). — We consider the pair (G_{1}, G_{2}) = (U(p, q), U(m)). We put G = G_{1} = U(p, q) in this subsection. Assume that m ≤ min(p, q) = ℝ-rank U(p, q). This means that the pair (U(p, q), U(m)) is in the stable range.

A maximal compact subgroup of G is isomorphic to U(p) × U(q), and we realized it as

K = \{(\begin{matrix}A & 0 \\ 0 & B \end{matrix}) | A ∈ U(p), B ∈ U(q)\} ⊂ U(p, q).  (6.45)

Put n = p+q. In this case, K ×G_{2} acts on M_{n,m} in somewhat distorted manner. Let us identify M_{n,m} = M_{p,m} ⊕M_{q,m}. Then the action of diag (a, b)×g ∈ (U(p) × U(q))× U(m) is given by

M_{p,m} ⊕ M_{q,m} ∋ X ⊕ Y ⟼ aX^{t}g ⊕ \overline{b}Y ^{t}\overline{g}.  (6.46)

This action gives the projection K × G_{2} → K · G_{2} ⊂ K = U(nm), where K is the maximal compact subgroup of G = Sp(2nm, ℝ) (cf. (3.19) and (3.20)). The kernel of the projection K × G_{2} → K · G_{2} is given by {((α1_{p}, α1_{q}), α^{−1}1_{m}) | α ∈ ℂ^{×}, |α| = 1}.

Let us consider the Weil representation Ω of G̃ = Mp(2nm, ℝ) on ℂ[M_{n,m}^{∗}]. Then the representation of Ω|K̃ is isomorphic to ℂ[M_{n,m}^{∗}] ⊗ det^{1/2}. If we consider the underlying space ℂ[M_{n,m}^{∗}] as ℂ[M_{n,m}^{∗}] = ℂ[M_{p,m}^{∗}] ⊗ ℂ[M_{q,m}], the above embedding of K × G_{2} into K tells us that the representation of U(p) × U(q) is isomorphic to

(ℂ[M_{p,m}^{∗}] ⊗ det^{m/2}) ⊠ (ℂ[M_{q,m}] ⊗ det^{−m/2});  (6.47)

while that of U(m) is isomorphic to

(ℂ[M_{p,m}^{∗}] ⊗ det^{p/2}) ⊗ (ℂ[M_{q,m}] ⊗ det^{−q/2}) ≃ (ℂ[M_{p,m}^{∗}] ⊗ ℂ[M_{q,m}]) ⊗ det^{(p−q)/2}. (6.48)

Therefore the one-dimensional character χ ∈ Irr(G̃_{2}) is equal to det^{(p−q)/2}, and the one-dimensional character χ_{1} of K̃ is det^{m/2} ⊠ det^{−m/2}.

Let us first consider the untwisted representation ℂ[M_{n,m}^{∗}] of K. To decompose the restriction to K × G_{2}, we make use of U(p) × U(m) (or U(q) × U(m)) duality. We have the decomposition as (U(p) × U(q)) × U(m)-module

ℂ[M_{n,m}^{∗}] = ℂ[M_{p,m}^{∗}] ⊗ ℂ[M_{q,m}]
= (\begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} τ_{λ}^{(p)} ⊠ τ_{λ}^{(m)}) ⊗ (\begin{matrix}∑ & ⊕ \\ μ∈P_{m} \end{matrix}(τ_{μ}^{(q)})^{∗} ⊠ (τ_{μ}^{(m)})^{∗})

= \begin{matrix}∑ & ⊕ \\ λ,μ∈P_{m} \end{matrix} (τ_{λ}^{(p)} ⊠ (τ_{μ}^{(q)})^{∗}) ⊠ (τ_{λ}^{(m)} ⊗ (τ_{μ}^{(m)})^{∗}) .

Therefore, if we define the branching coefficient m(λ, μ; ν) by

τ_{λ}^{(m)} ⊗ (τ_{μ}^{(m)})^{∗} = \begin{matrix}∑ & ⊕ \\ ν \end{matrix} m(λ, μ; ν) τ_{ν}^{(m)},  (6.49)

we get

ℂ[M_{n,m}^{∗}]|_{K×U(m)} = \begin{matrix}∑ & ⊕ \\ ν \end{matrix} \begin{cases}∑ & ⊕ \\ λ,μ∈P_{m} \end{cases} m(λ, μ; ν) τ_{λ}^{(p)} ⊠ (τ_{μ}^{(q)})^{∗}\} ⊠ τ_{ν}^{(m)}.

To get the representation Ω|_{K̃_{1}×G̃_{2}}, we should twist the above decomposition by

(det^{m/2} ⊠ det^{−m/2}) ⊠ det^{(p−q)/2}. After this twisting, for σ = τ_{ν}^{(m)} ∈ Irr(U(m)), we get the K-type decomposition of L(σ):

L(τ_{ν}^{(m)})|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ,μ∈P_{m} \end{matrix} m(λ, μ; ν) (τ_{λ}^{(p)} ⊗ det^{m/2}) ⊠ (τ_{μ}^{(q)} ⊗ det^{m/2})^{∗}.  (6.50)

To determine the lowest weight of L(τ_{ν}^{(m)}), we prove a lemma.

Lemma 6.4. — Take an arbitrary dominant integral weight ν of U(m), and write it as

ν = (a_{1}, a_{2}, …, a_{s}, 0, …, 0, −b_{t}, …, −b_{2}, −b_{1}),

where

\begin{matrix}a_{1} ≥ a_{2} ≥ ⋯ ≥ a_{s} > 0, b_{1} ≥ b_{2} ≥ ⋯ ≥ b_{t} > 0, \\ a_{i}, b_{j} ∈ ℤ; s, t ≥ 0 and s + t ≤ m. \end{matrix}

Consider a set of pairs of partitions {(λ, μ) ∈ P_{m} × P_{m} | m(λ, μ; ν) ≠ 0}. Then partitions

\{ \begin{matrix}λ = α := (a_{1}, a_{2}, …, a_{s}, 0, …, 0) and \\ μ = β := (b_{1}, b_{2}, …, b_{t}, 0, …, 0) \end{matrix}

minimize the degree |λ| + |μ| among such pairs. Moreover, (α, β) is a unique pair which attains the minimal degree. In this case, m(α, β; ν) = 1 holds.

Proof. — Take a sufficiently large l ≥ 0 such that ν′ = ν + (l, …, l) becomes a partition. We have

m(λ, μ; ν) = dim ((τ_{λ} ⊗ τ_{μ}^{∗}) ⊗ τ_{ν}^{∗})^{U(m)}  (6.51)
= dim ((τ_{λ} ⊗ det^{l}) ⊗ (τ_{μ} ⊗ τ_{ν′})^{∗})^{U(m)}  (6.52)
= c_{μ,ν′}^{λ+(l^{m})} ≠ 0,  (6.53)

where c_{γ,δ}^{η} = [τ_{γ} ⊗ τ_{δ} : τ_{η}] denotes the Littlewood-Richardson coefficient. Since c_{γ,δ}^{η} ≠ 0 implies |η| = |γ| + |δ|, we have |λ| + ml = |μ| + |ν′|, or equivalently |λ| = |μ| + |ν|. Therefore, in order to minimize |λ|+|μ|, we only have to make |λ| minimal. However, if ν′ is not contained in λ + (l^{m}), the coefficient c_{μ,ν′}^{λ+(l^{m})} vanishes. Therefore, λ = α is the smallest possible partition (e.g., see [13, § 5.2, Proposition 3]). If we take μ = β, then it is easy to see that m(α, β; ν) = 1 (loc. cit.).

If we denote the highest weight of τ_{ν}^{∗} by

ν^{∗} = (b_{1}, b_{2}, …, b_{t}, 0, …, 0, −a_{s}, …, −a_{2}, −a_{1}),

it holds that m(λ, μ; ν) = m(μ, λ; ν^{∗}). By the same argument as above, we conclude that μ = β is the only possibility for m(α, β; ν) ≠ 0.

Remark 6.5. — If ν is also a partition, the above proof tells us that m(λ, μ; ν) = c_{μ,ν}^{λ}, where c_{μ,ν}^{λ} is the Littlewood-Richardson coefficient.

Theorem 6.6. — Assume that m ≤ min(p, q) = ℝ-rank U(p, q). Then L(σ) is not zero for any σ = τ_{ν}^{(m)} ∈ Irr(U(m)) and it gives an irreducible unitary lowest weight module of U(p, q) . For ν, define α, β as in Lemma 6.4, and put β^{∗} = (0, …, 0, −b_{t}, …, −b_{2}, −b_{1}). Then the lowest weight of L(τ_{ν}^{(m)}) is given by

w_{K} (α + \frac{m}{2}𝕀_{p}, β^{∗} − \frac{m}{2}𝕀_{q}) ,

where w_{K} is the longest element of the Weyl group of K = U(p) × U(q) and 𝕀_{p} = (1, …, 1) = (1^{p}). Consequently, the lowest K-type of L(τ_{ν}^{(m)}) is (τ_{α}^{(p)} ⊗ det^{m/2}) ⊠ (τ_{β}^{(q)} ⊗ det^{m/2})^{∗}.

Proof. — It is known that L(σ) is an irreducible unitary lowest weight module of the metaplectic cover U(p, q) . So we simply have to determine its lowest weight. To do that, we only need to know the lowest K-type (or harmonic K-type) which is unique. By Lemma 6.4, we conclude that τ_{α}^{(p)} ⊠ τ_{β}^{(q)}^{∗} gives such a K-type with a twist by χ_{1} = det^{m/2} ⊠ det^{−m/2}.

By this theorem, we obtain Poincaré series of L(τ_{ν}^{(m)}):

P (L(τ_{ν}^{(m)}); t) = t^{−|β|} ∑_{λ,μ∈P_{m}} m(λ, μ; ν) dim τ_{λ}^{(p)} dim τ_{μ}^{(q)} t^{|μ|}
= t^{−|α|} ∑_{λ,μ∈P_{m}} m(λ, μ; ν) dim τ_{λ}^{(p)} dim τ_{μ}^{(q)} t^{|λ|}.

This formula follows from (5.37) after a reflection on degrees. Note that the summation is taken over (λ, μ) satisfying |λ| − |μ| = |ν| (see the proof of Lemma 6.4). Hence the total degree of τ_{λ}^{(p)} ⊠ (τ_{μ}^{(q)})^{∗} is given by |λ| + |μ| = 2|μ| + |ν| = 2|λ| − |ν|, while |ν| = |α| − |β| and k_{σ} = |α| + |β| for σ = τ_{ν}^{(m)}.

Consider the special case where τ_{ν}^{(m)} is trivial, i.e., ν = 0. Then it is easy to see that

m(λ, μ; 0) = \{ \begin{matrix}1 & if λ = μ, \\ 0 & otherwise. \end{matrix}

Therefore we get

Corollary 6.7. — We have the K-type decomposition of L(1_{U(m)}) as

L(1_{U(m)})|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix}(τ_{λ}^{(p)} ⊗ det^{m/2}) ⊠ (τ_{λ}^{(q)} ⊗ det^{m/2})^{∗}.  (6.54)

Its Poincaré series becomes

P (L(1_{U(m)}); t) = ∑_{λ∈P_{m}} dim τ_{λ}^{(p)} dim τ_{λ}^{(q)} t^{|λ|}.  (6.55)

6.3. Case (O^{∗}, Sp). — We consider the pair (G_{1}, G_{2}) = (O^{∗}(2p), Sp(2m)) in G = Sp(2nm, ℝ) (n = 2p), which is in the stable range, i.e., we assume that m ≤ [p/2] = ℝ-rank O^{∗}(2p).

In this case, a maximal compact subgroup K of O^{∗}(2p) is isomorphic to U(p). We realize the isomorphism as

O^{∗}(2p) ⊃ K = \{(\begin{matrix}X & 0 \\ 0 & \overline{X} \end{matrix}) | X ∈ U(p)\} ↔ X ∈ U(p).  (6.56)

Then K × G_{2} is imbedded into K = U(2pm) canonically. To be more precise, this embedding of K = U(p) and G_{2} = Sp(2m) = Sp(2m, ℂ) ∩ U(2m) is given by the action on M_{p,2m} as

M_{p,2m} ∋ X ⟼ gX^{t}h ((g, h) ∈ K × G_{2}).

The action induces a projection K × G_{2} → K · G_{2} ⊂ K with kernel {(±1_{p}, ±1_{2m}) ∈ U(p) × Sp(2m)}.

Let us consider the Weil representation Ω. As the representation space of Ω, we take the polynomial ring ℂ[M_{p,2m}^{∗}] as before. Then we know that Ω|K̃ is isomorphic to

ℂ[M_{p,2m}^{∗}] ⊗ det^{m},

where ℂ[M_{p,2m}^{∗}] is considered as the symmetric tensor product of the representation M_{p,2m} above. On the other hand, we have G̃_{2} ≃ Sp(2m)×ℤ_{2}, and the one-dimensional character χ arises as the non-trivial character of ℤ_{2} as in (6.40). Therefore, we have Ω|G̃_{2} ≃ ℂ[M_{p,2m}^{∗}] ⊗ χ.

First, let us treat the untwisted symmetric tensor. So we decompose ℂ[M_{p,2m}^{∗}] by using U(p) × U(2m) duality :

ℂ[M_{p,2m}^{∗}]|_{U(p)×U(2m)} ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{2m} \end{matrix} τ_{λ}^{(p)} ⊠ τ_{λ}^{(2m)}.

Take a highest weight λ for U(2m) and μ for Sp(2m). Let us define the branching coefficient m(λ, μ) by

τ_{λ}^{(2m)}|_{Sp(2m)} ≃ \begin{matrix}∑ & ⊕ \\ μ \end{matrix} m(λ, μ)σ_{μ},

where σ_{μ} ∈ Irr(Sp(2m)) is the irreducible representation of Sp(2m) with highest weight μ. We also write m(λ, σ_{μ}) instead of m(λ, μ). With this notation, we can write down the decomposition :

ℂ[M_{p,2m}^{∗}]|_{U(p)×Sp(2m)} = \begin{matrix}∑ & ⊕ \\ σ_{μ}∈Irr(Sp(2m)) \end{matrix} \begin{cases}∑ & ⊕ \\ λ∈P_{2m} \end{cases} m(λ, μ) τ_{λ}^{(p)}\} ⊠ σ_{μ}.

To get the restricted representation Ω|_{K̃×G̃_{2}}, we must twist the above representation by det^{m} ⊠ χ. Therefore L(σ)|K̃ decomposes as

L(σ_{μ})|_{U(p)} ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{2m} \end{matrix} m(λ, μ) τ_{λ}^{(p)} ⊗ det^{m}.  (6.57)

This formula describes the multiplicities of K-types of L(σ_{μ}) in the case of Case (O^{∗}, Sp).

Theorem 6.8. — Assume that m ≤ [p/2] = ℝ-rank O^{∗}(2p). Then L(σ) is not zero for any σ = σ_{μ} ∈ Irr(Sp(2m)) and it gives an irreducible unitary lowest weight module of O^{∗}(2p) . Extend μ to the weight of O^{∗}(2p) by adding zero. Then the lowest weight of L(σ_{μ}) is given by

w_{K} (μ + m𝕀_{p}) ,

where w_{K} is the longest element of the Weyl group of K = U(p) and 𝕀_{p} = (1, …, 1) = (1^{p}). Consequently, the lowest K-type of L(σ_{μ}) is τ_{μ}^{(p)} ⊗ det^{m}.

Proof. — See [23, § 3.8.5].

From the above theorem, we obtain the Poincaré series of L(σ_{μ}):

P (L(σ_{μ}); t^{2}) = t^{−|μ|} ∑_{λ∈P_{2m}} m(λ, μ) dim τ_{λ}^{(p)} t^{|λ|}.

36 K. Nishiyama, H. Ochiai & K. Taniguchi

Consider the special case where σ_{μ} = 1_{Sp(2m)}, i.e., μ = 0. It is well-known that

m(λ, 1_{Sp(2m)}) = \{ \begin{matrix}1 & if λ_{2i−1} = λ_{2i} for 1 ≤ i ≤ m, \\ 0 & otherwise. \end{matrix}

So we get Corollary 6.9. — We have the K-type decomposition of L(1_{Sp(2m)}) as

L(1_{Sp(2m)})|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} τ_{λ^{#}}^{(p)} ⊗ det^{m},  (6.58)

where λ^{#} = (λ_{1}, λ_{1}, λ_{2}, λ_{2}, …) is a transposed even partition which is obtained by doubling each row of λ. Its Poincaré series is given by

P (L(1_{Sp(2m)}); t) = ∑_{λ∈P_{m}} dim τ_{λ^{#}}^{(p)} t^{|λ|}.  (6.59)

7. Degree of nilpotent orbits

7.1. Automorphism groups of Hermitian symmetric spaces. — Let G be one of real reductive Lie groups Sp(2n, ℝ), U(p, q), or O^{∗}(2p). These groups appear as the group G_{1} in Table 2. The division algebra D is specified there. Let K be a maximal compact subgroup of G specified in § 6. In all cases, the corresponding Riemannian symmetric spaces G/K have G-invariant complex structure. In other words, the spaces G/K are Hermitian symmetric spaces. For G = Sp(2n, ℝ), U(p, p), or O^{∗}(4k), the corresponding Hermitian symmetric space G/K is of tube type. For G = U(p, q) with p ≠ q, or O^{∗}(4k + 2), G/K is not of tube type. For definitions and properties of symmetric spaces, see [20].

We fix a complexification G_{ℂ} of the real Lie group G. Let K_{ℂ} be the minimal complex Lie subgroup of G_{ℂ} containing K. We list up here (G_{ℂ}, K_{ℂ}) for the convenience of readers.

Table 3. Complexifications of (G, K).

\begin{matrix}G & G_{ℂ} & K_{ℂ} \\ Sp(2n, ℝ) & Sp(2n, ℂ) & GL(n, ℂ) \\ U(p, q) & GL(p + q, ℂ) & GL(p, ℂ) × GL(q, ℂ) \\ O^{∗}(2p) & O(2p, ℂ) & GL(p, ℂ) \end{matrix}

For real Lie groups such as G and K, we denote the corresponding Lie algebra by 𝔤_{0}, 𝔨_{0}, respectively. Its complexification is denoted by 𝔤 and 𝔨. The corresponding Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔭 is stable under the restriction of the adjoint action to K_{ℂ}. Moreover, in our cases, the subspace 𝔭 breaks up into the sum of two nonisomorphic irreducible representations of K_{ℂ}, say

𝔭 = 𝔭^{+} ⊕ 𝔭^{−}.

The representation 𝔭^{−} is the contragredient representation of 𝔭^{+}.

Let us describe the pair (𝔭^{+}, 𝔭^{−}) and the action of K_{ℂ} on them for each case. Although the action itself is fairly well-known, we need more explicit features in the following.

For G = Sp(2n, ℝ), we realize it as in (4.23) and a maximal compact subgroup K ≃ U(n) is also specified there (4.24). The complexification G_{ℂ} is identified naturally with Sp(2n, ℂ) with respect to the same symplectic form as G (see (3.13) for the symplectic form). Then, the decomposition 𝔭 = 𝔭^{+} ⊕ 𝔭^{−} is given by

𝔭^{±} = \{(\begin{matrix}±\sqrt{−1} A & A \\ A & ∓\sqrt{−1} A \end{matrix}) | A ∈ Sym (n, ℂ)\} .

Therefore, we can identify the both spaces with the space of symmetric matrices of size n. To see the action of K_{ℂ} ≃ GL(n, ℂ), it is more convenient to use the different realization of Sp(2n, ℝ). Let

γ = \frac{1}{\sqrt{2}} (\begin{matrix}1_{n} & −\sqrt{−1} 1_{n} \\ −\sqrt{−1} 1_{n} & 1_{n} \end{matrix}) ,

which is called the Cayley transform. The conjugation of Sp(2n, ℝ) by γ produces a different (but isomorphic) real form of Sp(2n, ℂ), and we denote it by G^{γ} = Sp(2n, ℝ)^{γ}. In G^{γ}, the conjugated maximal compact subgroup K^{γ} has a simple diagonal form:

K^{γ} = \{(\begin{matrix}k & 0 \\ 0 & ^{t}k^{−1} \end{matrix}) | k ∈ U(n)\} .

The complexification K_{ℂ}^{γ} is also expressed similarly as above, but k belonging to GL(n, ℂ). Then 𝔭^{γ} is represented by off diagonal matrices

𝔭^{γ} = \{( \begin{matrix}0 & B \\ C & 0 \end{matrix} ) | B, C ∈ Sym (n, ℂ)\} ,

and

𝔭^{γ+} = \{( \begin{matrix}0 & B \\ 0 & 0 \end{matrix} ) | B ∈ Sym (n, ℂ)\} , 𝔭^{γ−} = \{( \begin{matrix}0 & 0 \\ C & 0 \end{matrix} ) | C ∈ Sym (n, ℂ)\} .

We denote the element ( \begin{matrix}0 & B \\ C & 0 \end{matrix} ) of 𝔭^{γ} by (B, C). Then the adjoint action of an

element k ∈ K_{ℂ}^{γ} on 𝔭^{γ} is given by

k(B, C) = (kB^{t}k, ^{t}k^{−1}Ck^{−1}).

We sometimes identify the K_{ℂ}-module 𝔭^{+} with Sym (n, ℂ).

For G = U(p, q), we realized it as the full isometry group of the indefinite Hermitian form (3.17) (cf. (3.18)), and a maximal compact subgroup K ≃ U(p) × U(q) is given in (6.45). The complexification G_{ℂ} is naturally identified with GL(p + q, ℂ), and K_{ℂ} is given by

K_{ℂ} = \{k = (\begin{matrix}k_{1} & 0 \\ 0 & k_{2} \end{matrix}) | k_{1} ∈ GL(p, ℂ), k_{2} ∈ GL(q, ℂ)\} .

The other member of the Cartan decomposition is expressed by off diagonal matrices

𝔭 = \{( \begin{matrix}0 & B \\ C & 0 \end{matrix} ) | B ∈ M(p, q, ℂ), C ∈ M(q, p, ℂ)\} ,

and such an element is denoted by (B, C). Irreducible subspaces 𝔭^{±} are given as

𝔭^{+} = \{( \begin{matrix}0 & B \\ 0 & 0 \end{matrix} ) | B ∈ M(p, q, ℂ)\} , 𝔭^{−} = \{( \begin{matrix}0 & 0 \\ C & 0 \end{matrix} ) | C ∈ M(q, p, ℂ)\} .

The adjoint action of an element k = (k_{1}, k_{2}) ∈ K_{ℂ} on 𝔭 is given by

(k_{1}, k_{2})(B, C) = (k_{1}Bk_{2}^{−1}, k_{2}Ck_{1}^{−1}).

Therefore, the representation 𝔭^{+} of K_{ℂ} is identified with M(p, q, ℂ).

For G = O^{∗}(2p), we gave a realization in (3.21). A maximal compact subgroup K ≃ U(p) is chosen again as diagonal matrices (6.56). The complexified Lie group G_{ℂ} is identified with

O(2p, ℂ) = \{Z ∈ GL(2p, ℂ) | ^{t}ZS_{p}Z = S_{p}\} , S_{p} = (\begin{matrix}0 & 1_{p} \\ 1_{p} & 0 \end{matrix}) ,  (7.60)

and

K_{ℂ} = \{(\begin{matrix}k & 0 \\ 0 & ^{t}k^{−1} \end{matrix}) | k ∈ GL(p, ℂ)\} .

We identify K_{ℂ} and GL(p, ℂ) in the following, so k ∈ K_{ℂ} denotes a matrix in GL(p, ℂ). Now 𝔭 becomes

𝔭 = \{( \begin{matrix}0 & B \\ C & 0 \end{matrix} ) | B, C ∈ Alt (p, ℂ)\} .

As above, we denote the element ( \begin{matrix}0 & B \\ C & 0 \end{matrix} ) by (B, C). The K_{ℂ} stable decomposition

of 𝔭 is given by

𝔭^{+} = \{( \begin{matrix}0 & B \\ 0 & 0 \end{matrix} ) | B ∈ Alt (p, ℂ)\} , 𝔭^{−} = \{( \begin{matrix}0 & 0 \\ C & 0 \end{matrix} ) | C ∈ Alt (p, ℂ)\} .

The adjoint action of an element k ∈ K_{ℂ} on 𝔭 is

k(B, C) = (kB^{t}k,^{t}k^{−1}Ck^{−1}).

We identify the K_{ℂ}-module 𝔭^{+} with Alt (p, ℂ).

7.2. Kostant-Rallis decomposition. — In this subsection, we summarize the K_{ℂ}-orbit decomposition of 𝔭^{−}. The orbit decomposition of 𝔭^{+} is the same. We denote the real rank of Lie group G by r (cf. § 6). Then there are exactly (r + 1) K_{ℂ}-orbits in 𝔭^{−} in each of the above three cases. We give a parametrization of these orbits by (r + 1) integers, 0, 1, …, r,

𝔭^{−} = ∐_{j=0}^{r} O_{j}.

We know that the dimension of the orbits are distinct (see below). We arrange the numbering of orbits so that an orbit with the larger index has the larger dimension. With this indexing, the set O_{r} is an open dense subset of 𝔭^{−} in the classical topology (or, also in Zariski topology). On the contrary, the orbit O_{0} = {0}. We also know that the closure in classical topology (or, also in Zariski topology),

\overline{O_{m}} = ∐_{j≤m} O_{j}.

In other words, the closure relation of the orbits is linear ordering. Each closure is a Zariski closed subset of the affine space 𝔭^{−}, then \overline{O_{m}} is an affine algebraic variety. We denote the defining ideal of these subset \overline{O_{m}} by

I_{m} := \{p ∈ ℂ[𝔭^{−}] | p|\overline{O_{m}} = 0\}.

This is an ideal of the polynomial ring ℂ[𝔭^{−}] on 𝔭^{−}. Then, by definition, the coordinate ring ℂ[\overline{O_{m}}] is isomorphic to the residual ring ℂ[𝔭^{−}]/I_{m}.

Note that we can identify 𝔭^{+} with the dual vector space of 𝔭^{−} via Killing form. Therefore, by the natural identification, ℂ[𝔭^{−}] = S(𝔭^{+}), where S(𝔭^{+}) denotes the symmetric algebra. Since 𝔭^{+} is an abelian subspace of 𝔤, we also identify S(𝔭^{+}) with the enveloping algebra U(𝔭^{+}). We use these identification freely in the following. In particular, as a K_{ℂ}-module, ℂ[\overline{O_{m}}] is isomorphic to a quotient module of S(𝔭^{+}).

Let L = L(σ) be an irreducible unitary lowest weight module of G treated in § 5. We construct a good filtration of L by taking the lowest K-type as the generating subspace of L (cf. § 1.2). Let M = gr L be the associated graded S(𝔤)-module. Since the generating subspace is preserved by 𝔨 and 𝔭^{−}, the S(𝔤)-module M is annihilated by 𝔨 and 𝔭^{−}. Therefore, its associated variety AV (L) is contained in 𝔭^{−}, by the identification above, and is a K_{ℂ} stable closed subset. Since the K_{ℂ}-orbits in 𝔭^{−} has linear ordering with respect to the closure relation, we can conclude that AV (L) = \overline{O_{m}} for some 0 ≤ m ≤ r.

In the following subsections, we see that there is a strong relationship between ℂ[\overline{O_{m}}] and the K-type decomposition of L(1_{G_{2}}). In fact, they are the same as K_{ℂ}- modules up to some character. This relationship is an example of general phenomenon and is well-known among experts. It is a part of Vogan’s philosophy of orbit method [52].

Our main aim of the following subsections is calculation of the Bernstein degree of L(1_{G_{2}}). Our approach is purely representation theoretic. It turns out that Deg L(1_{G_{2}}) coincides with the classical degree of the corresponding orbit \overline{O_{m}} = AV (L(1_{G_{2}})), which coincides with determinantal variety of various type (see, e.g, [12] or [16, Lecture 9]). Hence our calculation here will give a new proof of the formula of deg \overline{O_{m}} called Giambelli-Thom-Porteous formula ([15], [17]; also see [12, Chapter 14]).

7.3. The case G = Sp(2n, ℝ). — Consider G = Sp(2n, ℝ). In this case, as is given above, K = U(n), K_{ℂ} = GL(n, ℂ), 𝔭^{−} = Sym (n, ℂ). The action of k ∈ K_{ℂ} on A ∈ 𝔭^{−} is given by

k · A = ^{t}k^{−1}Ak^{−1} (k ∈ GL(n, ℂ), A ∈ Sym (n, ℂ)).  (7.61)

We define a locally closed subset of Sym (n, ℂ) by

O_{m} = {A ∈ Sym (n, ℂ) | rank (A) = m}, (m = 0, 1, …, n).

By the definition of the action of K_{ℂ}, it is easy to see that O_{m} is stable under the action of K_{ℂ}. Moreover, they classify all the K_{ℂ}-orbits in 𝔭^{−}. The matrix ∑_{j=1}^{m} E_{jj} belongs to the orbit O_{m}. Here, E_{ij} is the matrix unit, that is, (i, j)-entry of the matrix E_{ij} is one and all other entries are zero. The dimension of the orbit O_{m} is given by

dim O_{m} = rm − (m − 1)m/2.

For subsets I = {i_{1}, i_{2}, …, i_{m+1}} and J = {j_{1}, j_{2}, …, j_{m+1}} of {1, 2, …, n} with the same cardinality (m + 1), we define the minor

D_{IJ}(A) = det(a_{i_{p}j_{q}})_{1≤p,q≤m+1},

where A = (a_{ij})_{1≤i,j≤n} ∈ Sym (n, ℂ). Then the defining ideal I_{m} of \overline{O_{m}} is generated by these minors

{D_{IJ} | I, J ⊂ {1, 2, …, n}, |I| = |J| = m + 1}.

Recall the dual pair (Sp(2n, ℝ), O(m)) in § 3. We define an unfolding of the orbit O_{m} by an extra action of O(m), or more precisely, its complexification O(m, ℂ). Let us consider the space of m × n matrices M_{m,n} = M(m, n, ℂ) and define an action of K_{ℂ} × O(m, ℂ) = GL(n, ℂ) × O(m, ℂ) ∋ (k, h) on M_{m,n} by

(k, h) · X = hXk^{−1} (X ∈ M_{m,n}).

For X ∈ M_{m,n}, we define

ψ(X) = ^{t}XX ∈ Sym (n, ℂ).

This is a polynomial map of degree two. With the trivial action of O(m, ℂ) on \overline{O_{m}}, the map

ψ : M_{m,n} → \overline{O_{m}}

is K_{ℂ} × O(m, ℂ)-equivariant, that is, ψ(hXk^{−1}) = ^{t}k^{−1}ψ(X)k^{−1} for all k ∈ K_{ℂ}, h ∈ O(m, ℂ). We see that the image of ψ coincides with \overline{O_{m}}.

Lemma 7.1. — The map ψ above induces the ℂ-algebra isomorphism

ψ^{∗} : ℂ[\overline{O_{m}}] ∋ f ↦ f ∘ ψ ∈ ℂ[M_{m,n}]^{O(m,ℂ)} = S(M_{n,m})^{O(m,ℂ)},

which means that \overline{O_{m}} is the geometric quotient M_{n,m}//O(m, ℂ). In particular, \overline{O_{m}} is a normal variety. Here we consider M_{m,n} = M_{n,m}^{∗} as the algebraic dual of M_{n,m}.

Proof. — The induced map ψ^{∗} is injective since ψ is surjective. The classical invariant theory, in the modern reformulation [24, § 3.4], says that every O(m, ℂ)-invariants on M_{m,n} is generated by typical invariants of degree two, which implies the map ψ^{∗} is surjective.

Now we come back to the dual pair (Sp(2n, ℝ), O(m)) in G = Sp(2nm, ℝ) and the Weil representation Ω of G̃ (cf. § 3). Let L(1_{O(m)}) be an irreducible unitary lowest weight module of G̃ which corresponds to the trivial representation of O(m). We should clarify the relationship between O_{m} and the representation L(1_{O(m)}).

Since the associated variety of L(1_{O(m)}) is contained in 𝔭^{−}, it is enough to see the annihilator of gr L(1_{O(m)}) in U(𝔭^{+}). Therefore, let us see the action of the noncompact root vector X_{ε_{a}+ε_{b}} ∈ 𝔭^{+} via Ω : 𝔰𝔭(2nm, ℝ) → End_{ℂ}(ℂ[M_{n,m}]),

Ω(X_{ε_{a}+ε_{b}}) = \frac{1}{2} ∑_{j=1}^{m} x_{aj}x_{bj},

(see (4.30)). By this formula, we see Ω(X_{ε_{a}+ε_{b}}) ∈ ℂ[M_{m,n}]^{O(m,ℂ)}. Moreover, we have

2 Ω(X_{ε_{a}+ε_{b}}) = ψ_{ab},

here ψ_{ab} ∈ ℂ[M_{m,n}]^{O(m,ℂ)} is the ab-component of ψ. This means that the subspace spanned by typical invariants ψ_{ab} coincides with the image Ω(𝔭^{+}). Thus, the subalgebra of Ω(U(𝔭^{+})) generated by Ω(𝔭^{+}) is isomorphic to ℂ[M_{m,n}]^{O(m,ℂ)}, which is generated by typical invariants as is explained above. Let us define the natural good filtration of L = L(1_{O(m)}) by L_{k} = U_{k}(𝔭^{+})1, where 1 is the constant polynomial with value 1. Then we have an isomorphism

L(1_{O(m)}) ≅ gr L(1_{O(m)}) ≅ U(𝔭^{+})/I_{m}

as U(𝔭^{+})-modules, K_{ℂ}-modules and filtered modules. The filtration induced by the degree of polynomials coincides with the natural filtration up to a shift. This implies

Lemma 7.2. — There are algebra isomorphisms

Ω(U(𝔭^{+})) ≃ ℂ[M_{m,n}]^{O(m,ℂ)} ≃ ℂ[\overline{O_{m}}] = ℂ[𝔭^{−}]/I_{m}.

We have Ann L(1_{O(m)}) = Ann gr L(1_{O(m)}) = I_{m} in U(𝔭^{+}).

Proof. — As is explained above, we have the desired isomorphisms. For the annihilator, note that the representation space ℂ[M_{n,m}^{∗}]^{O(m,ℂ)} of L(1_{O(m)}) has the natural grading as K-module and 𝔭^{+} acts on ℂ[M_{n,m}^{∗}]^{O(m,ℂ)} = ℂ[M_{m,n}]^{O(m,ℂ)} as a homogeneous operator of degree two. This means that the annihilator in U(𝔭^{+}) does not change after taking gradation as a filtered U(𝔤)-module.

Corollary 7.3. — The representation L(1_{O(m)}) has the following properties.

(1) The associated variety of L(1_{O(m)}) is \overline{O_{m}}.

(2) As a K̃-module, L(1_{O(m)}) is isomorphic to ℂ[\overline{O_{m}}] ⊗ det^{m/2}.

(3) The Bernstein degree of L(1_{O(m)}) coincides with deg \overline{O_{m}}.

Proof. — (1) is a direct consequence of the above lemma.

Let us consider (2). By definition, L(1_{O(m)}) is realized on ℂ[M_{n,m}^{∗}]^{O(m,ℂ)} = ℂ[M_{m,n}]^{O(m,ℂ)} (see § 5). As is explained in § 5, to get K̃-module structure of L(1_{O(m)}), we must twist ℂ[M_{m,n}]^{O(m,ℂ)} by det^{m/2}. Therefore, untwisting of L(1_{O(m)}) produces ℂ[M_{m,n}]^{O(m,ℂ)} itself, and the module structure factors through to that of K.

Since the unfolding map ψ has degree two, it is easy to see the definition of Deg L(1_{O(m)}) and deg \overline{O_{m}} coincides, which proves (3).

Let us calculate Deg L(1_{O(m)}) = deg \overline{O_{m}} explicitly. Recall the good filtration L_{k} = U_{k}(𝔭^{+})1. By (6.44) and the Weyl’s dimension formula, we know

dim L_{k} = ∑_{λ∈P_{m},|λ|≤k} dim τ_{2λ}^{(n)}

= ∑_{λ∈P_{m},|λ|≤k} \frac{∏_{1≤i<j≤n}(2λ_{i} − 2λ_{j} − i + j)}{∏_{1≤i<j≤n}(j − i)}

= \frac{2^{m(m−1)/2+m(n−m)}k^{m(m−1)/2+m(n−m)+m}}{∏_{i=1}^{m}(n − i)!} ×

× ∫_{{0≤xm≤x_{m−1}≤···≤x_{1}, \atop x_{1}+···+xm≤1}} ∏_{1≤i<j≤m}(x_{i} − x_{j}) ∏_{i=1}^{m} x_{i}^{n−m}dx_{1} ⋯ dx_{m}

+ (lower order terms of k)

= \frac{2^{mn−m(m+1)/2}k^{mn−m(m−1)/2}}{m! ∏_{i=1}^{m}(n − i)!} ×

× ∫_{{x_{i}≥0 \atop x_{1}+···+xm≤1}} ∏_{1≤i<j≤m} |x_{i} − x_{j}| ∏_{i=1}^{m} x_{i}^{n−m}dx_{1} ⋯ dx_{m}

+ (lower order terms of k)

for sufficiently large k. Here, in the third equality, we devide the formula by a suitable power of k and interprete the leading term as a Riemann sum for the integral.

Let us generalize the integral above slightly, and denote it as

I^{α}(s, m) = ∫_{{x_{i}≥0, \atop x_{1}+···+xm≤1}} |Δ|^{α} (∏_{i=1}^{m} x_{i})^{s} dx_{1} ⋯ dx_{m},  (7.62)

where Δ = ∏_{1≤i<j≤m}(x_{i} − x_{j}) is the difference product. An explicit formula of this integral is given by using Gamma function of Hermitian symmetric cone ([37]).

Theorem 7.4. — Let I^{α}(s, m) be as in (7.62). For Re s > −1 and α = 1, 2, 4, we have

I^{α}(s, m) = \frac{∏_{j=1}^{m} Γ(jα/2 + 1)Γ(s + 1 + (j − 1)α/2)}{Γ(α/2 + 1)^{m}Γ(sm + N + 1)},  (7.63)

where N = m + \frac{α}{2}m(m − 1).

Summarizing above, we have the following theorem.

Theorem 7.5. — Assume that m ≤ n = ℝ-rank Sp(2n, ℝ), and consider the reductive dual pair (Sp(2n, ℝ), O(m)).

(1) The unitarizable lowest weight module L(1_{O(m)}) of Sp(2n, ℝ) has the lowest weight \frac{m}{2}(1, 1, …, 1) = \frac{m}{2} ∑_{i=1}^{n} ε_{i}. Its associated cycle is multiplicity-free and given by AC (L(1_{O(m)})) = [\overline{O_{m}}].

(2) The Gelfand-Kirillov dimension and the Bernstein degree of L(1_{O(m)}) are

Dim L(1_{O(m)}) = dim \overline{O_{m}} = m (n − \frac{m − 1}{2}) ,

Deg L(1_{O(m)}) = deg \overline{O_{m}} = ∏_{l=0}^{m−1} \frac{l!}{l!!} \frac{(2n − 2m + l)!!}{(n − m + l)!},

where l!! = l(l − 2)(l − 4) ⋯ 2 for an even integer l, and l!! = l(l − 2)(l − 4) ⋯ 1 for odd l.

Proof. — From the top degree term of dim L_{k} above, we get the Gelfand-Kirillov dimension

Dim L(1_{O(m)}) = mn − \frac{m(m − 1)}{2} =: d,

and

Deg L(1_{O(m)}) = \frac{2^{d−m}d!}{m! ∏_{i=1}^{m}(n − i)!}I^{1}(n − m, m)

= \frac{2^{d}}{π^{m/2}m!} ∏_{j=1}^{m} \frac{Γ(j/2 + 1)Γ(d/m − (j − 1)/2)}{Γ(n − j + 1)}

= ∏_{l=0}^{m−1} \frac{l!}{l!!} \frac{(2n − 2m + l)!!}{(n − m + l)!}.

We close this subsection by giving the relation between the lowest weight module L(1_{O(m)}) and the half-form bundle on the orbit O_{m}. We choose a representative

λ = ∑_{j=1}^{m} E_{jj} ∈ O_{m} ⊂ Sym (n, ℂ) ≅ 𝔭^{−}

of the orbit O_{m}. The group K_{ℂ} = GL(n, ℂ) acts on O_{m} transitively by (7.61). The stabilizer (K_{ℂ})_{λ} of λ in K_{ℂ} is

(K_{ℂ})_{λ} = \{k = ( \begin{matrix}g_{1} & 0 \\ ∗ & g_{2} \end{matrix} ) | g_{1} ∈ O(m, ℂ), g_{2} ∈ GL(n − m, ℂ)\} .  (7.64)

We denote the determinant of the isotropy representation by det(Ad |_{T_{λ}O_{m}}) : (K_{ℂ})_{λ} → ℂ^{×}, where T_{λ}O_{m} is the tangent space of O_{m} at λ. It is written by

det(Ad |_{T_{λ}O_{m}}) = (det g_{1})^{n−m}(det g_{2})^{−m} = (det g_{1})^{n}(det k)^{−m},

with the notation (7.64). The cotangent bundle T^{∗}O_{m} is a K_{ℂ}-equivariant vector bundle. The line bundle Λ^{top} = ⋀^{dim O_{m}} T^{∗}O_{m} consisting of volume forms on the orbit O_{m} is a K_{ℂ}-equivariant line bundle. Then it corresponds to the one-dimensional representation of the isotropy subgroup (K_{ℂ})_{λ}. In this case it is given by the coisotropy representation

det(Ad^{∗}|_{T_{λ}^{∗}O_{m}}) : (K_{ℂ})_{λ} ∋ k ↦ (det g_{1})^{−n}(det k)^{m} ∈ ℂ^{×},

with the notation (7.64). We introduce the square root of the line bundle Λ^{top}, denoted by ξ, and consider the set Γ(O_{m}, ξ) of its global sections. We will give the relation between this line bundle on the orbit O_{m} and the lowest weight representation under consideration.

In what follows, we assume that n is even. We define the one-dimensional representation

ξ : (K̃_{ℂ})_{λ} ∋ k ↦ det^{m/2}k ∈ ℂ^{×}.

By the definition, the coisotropy representation is the square of ξ;

det(Ad^{∗}|_{T_{λ}^{∗}O_{m}}) = det(Ad |_{T_{λ}O_{m}})^{−1} = ξ^{2}.

This means that ξ corresponds to the half-form bundle on the orbit O_{m} = K_{ℂ}/(K_{ℂ})_{λ} = K̃_{ℂ}/(K̃_{ℂ})_{λ}. The set of global sections Γ(O_{m}, ξ) has a natural K̃_{ℂ}- module structure.

Proposition 7.6. — For 0 ≤ m < n and n ∈ 2ℤ, the lowest weight module L(1_{O(m)}) is isomorphic to Γ(O_{m}, ξ) as K̃-modules.

Proof. — We denote the complexification of the character χ_{1} : K̃ → ℂ^{×} introduced in Section 6.1 by the same character. To be more explicit, we define the character χ_{1} : K̃_{ℂ} → ℂ^{×} by χ_{1}(k) = det^{m/2}k. The restriction of χ_{1} to the isotropy subgroup coincides with ξ. Then, we see that

Γ(O_{m}, ξ) = Ind K̃_{ℂ} ; (K̃_{ℂ})_{λ}ξ = χ_{1} ⊗ Ind K̃_{ℂ} ; (K̃_{ℂ})_{λ}1_{(K̃_{ℂ})_{λ}} = χ_{1} ⊗ ℂ[O_{m}]

for 0 ≤ m ≤ n as K̃_{ℂ}-modules. On the other hand, we have seen in Corollary 7.3(2) that

L(1_{O(m)}) = ℂ[\overline{O_{m}}] ⊗ det^{m/2}.

Since \overline{O_{m}} is normal (cf. Lemma 7.1), and, for m ≠ n, codim \overline{O_{m}}O_{m} ≥ 2 for m ≠ n, the restriction map gives a natural isomorphism ℂ[\overline{O_{m}}] = ℂ[O_{m}] (cf. [10, Chapter 11, § 11.2]). This shows the proposition.

7.4. The case G = U(p, q). — Let G = U(p, q). In this case, K = U(p) × U(q), K_{ℂ} = GL(p, ℂ) × GL(q, ℂ), 𝔭^{−} = M(q, p, ℂ). The action of (k_{1}, k_{2}) ∈ K_{ℂ} on A ∈ 𝔭^{−} is given by

k_{2}Ak_{1}^{−1}.  (7.65)

Put r = ℝ-rank U(p, q) = min(p, q). We define a subset of M_{q,p} = M(q, p, ℂ) by

O_{m} = {A ∈ M_{q,p} | rank (A) = m}, (m = 0, 1, …, r).

By an argument similar to the case Sp(2n, ℝ), we know that O_{m} is a K_{ℂ}-orbit, and they give a complete classification of K_{ℂ}-orbits in 𝔭^{−}. Note that the matrix ∑_{j≤m} E_{jj} is contained in O_{m}. It is easy to see that

dim O_{m} = (p + q)m − m^{2},

hence all the orbits have different dimensions. The defining ideal I_{m} of \overline{O_{m}} is generated by the minors

{D_{IJ} | I ⊂ {1, 2, …, q}, J ⊂ {1, 2, …, p}, |I| = |J| = m + 1}.

The affine algebraic variety \overline{O_{m}} is called the determinantal variety.

Now recall the dual pair (U(p, q), U(m)). Let GL(m, ℂ) be the complexification of U(m). We consider the natural action of K_{ℂ} × GL(m, ℂ) = (GL(p, ℂ) × GL(q, ℂ)) × GL(m, ℂ) ∋ (k_{1}, k_{2}, h) on (A, B) ∈ M_{m,p} ⊕ M_{m,q} ≃ M_{m,p+q} by

(^{t}h^{−1}Ak_{1}^{−1}, hB^{t}k_{2}),  (7.66)

which comes from (6.46). For (A, B) ∈ M_{m,p} ⊕ M_{m,q}, we define an unfolding map ψ by

ψ(A, B) = ^{t}BA ∈ M_{q,p}.

This is a polynomial map of degree two. Note that ψ(∑_{l≤j} E_{ll}, ∑_{l≤j} E_{ll}) = ∑_{l≤j} E_{ll} ∈ O_{m}. From this, we see that the image of ψ coincides with \overline{O_{m}}. With the trivial action of GL(m, ℂ) on \overline{O_{m}}, the map

ψ : M_{m,p+q} → \overline{O_{m}}

is K_{ℂ} × GL(m, ℂ)-equivariant, that is, ψ(^{t}h^{−1}Ak_{1}^{−1}, hB^{t}k_{2}) = k_{2}ψ(A, B)k_{1}^{−1} for all (k_{1}, k_{2}) ∈ K_{ℂ}, h ∈ GL(m, ℂ). This map induces the ℂ-algebra homomorphism

ψ^{∗} : ℂ[\overline{O_{m}}] ∋ f ↦ f ∘ ψ ∈ ℂ[M_{m,p+q}]^{GL(m,ℂ)}.

As a summary we have

Lemma 7.7. — There exists a ℂ-algebra isomorphism

ψ^{∗} : ℂ[\overline{O_{m}}] → ℂ[M_{m,p+q}]^{GL(m,ℂ)} = S(M_{p+q,m})^{GL(m,ℂ)},

which means that \overline{O_{m}} is the geometric quotient M_{p+q,m}//GL(m, ℂ). In particular, \overline{O_{m}} is a normal variety. Here we consider M_{p+q,m} as the contragredient space to M_{m,p+q}.

Proof. — It is injective since ψ is surjective. The classical invariant theory also says that every GL(m, ℂ)-invariants on M_{m,p+q} is generated by typical invariants of degree two, that is, this map ψ^{∗} is surjective.

For the Weil representation of the dual pair (U(p, q), U(m)) ∈ Sp(2nm, ℝ) and the unitary lowest weight module L(1_{U(m)}), we have expected the same story. Take a Cartan subalgebra 𝔱 in 𝔨 consisting of diagonal matrices

𝔱 = {H = diag (a_{1}, …, a_{p}, b_{1}, …, b_{q}) | a_{i}, b_{j} ∈ ℂ}.

This is also a Cartan subalgebra of 𝔤. We define ε_{i}, δ_{j} ∈ 𝔱^{∗} by ε_{i}(H) = a_{i}, δ_{j}(H) = b_{j} for above H ∈ 𝔱. Then the set of positive non-compact roots is

Δ_{n}^{+} = {ε_{i} − δ_{j} | 1 ≤ i ≤ p, 1 ≤ j ≤ q}.

Put

X_{ε_{a}−δ_{b}} = ( \begin{matrix}0 & E_{ab} \\ 0 & 0 \end{matrix} ) ∈ 𝔤𝔩(p + q, ℂ) = 𝔤.

Bernstein Degree and Associated Cycles 47

Then X_{ε_{a}−δ_{b}} is a non-compact root vector in 𝔭^{+}. From the embedding (3.19) and the Fock realization (4.27) of the Weil representation Ω, we conclude that

Ω(−2X_{ε_{a}−δ_{b}}) = ψ_{ab} = ∑_{j=1}^{m} x_{aj}y_{bj} (1 ≤ a ≤ p, 1 ≤ b ≤ q), (7.67)

where (x_{aj})_{1≤a≤p,1≤j≤m} ∈ M_{p,m} and (y_{bj})_{1≤b≤q,1≤j≤m} ∈ M_{q,m}. Note that these quadratics (7.67) generate the full invariants S(M_{p+q,m})^{GL(m,ℂ)}. From this, we get Lemma 7.8. — There are algebra isomorphisms

Ω(U(𝔭^{+})) ≃ ℂ[M_{m,p+q}]^{GL(m,ℂ)} ≃ ℂ[\overline{O_{m}}] = ℂ[𝔭^{−}]/I_{m}.

We have Ann L(1_{U(m)}) = Ann gr L(1_{U(m)}) = I_{m} in U(𝔭^{+}). Proof. — The proof is similar to that of Lemma 7.2. Corollary 7.9. — (1) The associated variety of L(1_{U(m)}) is \overline{O_{m}}.

(2) As a K̃-module, L(1_{U(m)}) is isomorphic to ℂ[\overline{O_{m}}] ⊗ (det^{m/2} ⊠ det^{−m/2}).

(3) Bernstein degree of L(1_{U(m)}) coincides with deg \overline{O_{m}}. Proof. — The proof is similar to that of Corollary 7.3. For the K-type decomposition of L(1_{U(m)}), see (6.54).

Let us define the natural filtration of L = L(1_{U(m)}) by L_{k} = U_{k}(𝔭^{+})1, where 1 is a constant polynomial. By (6.55), we know

dim L_{k} = ∑_{{λ∈Pm \atop |λ|≤k}} dim τ_{λ}^{(p)} dim τ_{λ}^{(q)}

= ∑_{{λ,l(λ)≤m \atop |λ|≤k}} \frac{∏_{1≤i<j≤m}(λ_{i} − λ_{j} − i + j) ∏_{{1≤i≤m, \atop m+1≤j≤p}}(λ_{i} − i + j) ∏_{m+1≤i<j≤p}(j − i)}{∏_{1≤i<j≤p}(j − i)}

× \frac{∏_{1≤i<j≤m}(λ_{i} − λ_{j} − i + j) ∏_{{1≤i≤m, \atop m+1≤j≤q}} (λ_{i} − i + j) ∏_{m+1≤i<j≤q}(j − i)}{∏_{1≤i<j≤q}(j − i)}

= \frac{k^{m(m−1)/2×2+m(p+q−2m)+m}}{∏_{i=1}^{m}(p − i)!(q − i)!}

× ∫_{{0≤xm≤x_{m−1}≤···≤x_{1}, \atop x_{1}+···+xm≤1}} ∏_{1≤i<j≤m}(x_{i} − x_{j})^{2} ∏_{i=1}^{m} x_{i}^{p+q−2m}dx_{1} … dx_{m}

+ (lower order terms of k)

= \frac{k^{m(p+q−m)}}{m! ∏_{i=1}^{m}(p − i)!(q − i)!} ∫_{{0≤x_{i} \atop x_{1}+···+xm≤1}} ∏_{1≤i<j≤m} |x_{i} − x_{j}|^{2} ∏_{i=1}^{m} x_{i}^{p+q−2m}dx_{1} … dx_{m}

+ (lower order terms of k)

for sufficiently large k.

Theorem 7.10. — Assume that m ≤ min(p, q) = ℝ-rank U(p, q), and consider the reductive dual pair (U(p, q), U(m)).

(1) The unitarizable lowest weight module L(1_{U(m)}) of U(p, q) has the lowest weight

m/2𝕀_{p,q} = m/2 (∑_{i=1}^{p} ε_{i} − ∑_{j=1}^{q} δ_{j}) , where 𝕀_{p,q} = (1, …, 1, −1, …, −1). Its asso-

ciated cycle is given by AC (L(1_{U(m)})) = [\overline{O_{m}}].

(2) The Gelfand-Kirillov dimension and the Bernstein degree of L(1_{U(m)}) is given by

Dim L(1_{U(m)}) = dim \overline{O_{m}} = m(p + q − m),
Deg L(1_{U(m)}) = deg \overline{O_{m}} = ∏_{j=1}^{m} \frac{(j − 1)! (p + q − m − j)!}{(p − j)! (q − j)!}

Proof. — By the formula of dim L_{k} above, we have

Dim L(1_{U(m)}) = m(p + q − m) =: d,
Deg L(1_{U(m)}) = \frac{d!}{m! ∏_{i=1}^{m}(p − i)!(q − i)!}I^{2}(p + q − 2m, m).

Now apply Theorem 7.4.

We show that the half-form bundle on O_{m} is related to some lowest weight representation L(σ). We put

λ = ∑_{j=1}^{m} E_{jj} ∈ O_{m} ⊂ M_{q,p} ≅ 𝔭^{−}.

The group K_{ℂ} = GL(p, ℂ) × GL(q, ℂ) acts on O_{m} by (7.65). The stabilizer (K_{ℂ})_{λ} of λ in K_{ℂ} is

(K_{ℂ})_{λ} = \{(k_{1}, k_{2}) = (( \begin{matrix}g_{1} & 0 \\ ∗ & g_{2} \end{matrix} ) , ( \begin{matrix}g_{1} & ∗ \\ 0 & g_{3} \end{matrix} )) ∈ K_{ℂ} | g_{1} ∈ GL(m, ℂ)\} . (7.68)

The determinant of the isotropy representation is

det(Ad |_{T_{λ}O_{m}}) = (det g_{1})^{p−q}(det k_{1})^{−m}(det k_{2})^{m},

and that of the coisotropy representation

det(Ad^{∗}|_{T_{λ}^{∗}O_{m}}) : (K_{ℂ})_{λ} ∋ (k_{1}, k_{2}) ↦ (det g_{1})^{−(p−q)}(det k_{1})^{m}(det k_{2})^{−m} ∈ ℂ^{×},

with the notation (7.68). We denote the line bundle consisting of volume forms on O_{m} by Λ^{top}, and its square root by ξ. Let us clarify the meaning of the square root ξ of Λ^{top}. We denote the inverse image of the subgroup K_{ℂ} ⊂ K_{ℂ} in K̃_{ℂ} by K̃_{ℂ}. This is a double covering group of K_{ℂ}, which is not necessarily connected, with the covering map K̃_{ℂ} ⟶ K_{ℂ}. We have an realization

K̃_{ℂ} = {(k, z) ∈ K_{ℂ} × ℂ^{×} | k = (k_{1}, k_{2}), (det k_{1})^{m}(det k_{2})^{−m} = z^{2}}.

Through the natural projection, K̃_{ℂ} also acts on O_{m}. We denote the isotropy subgroup at λ ∈ O_{m} by (K̃_{ℂ})_{λ}. This is the inverse image of (K_{ℂ})_{λ}, that is,

(K̃_{ℂ})_{λ} = \{((k_{1}, k_{2}), z) ∈ K̃_{ℂ} | k_{1} = (\begin{matrix}g_{1} & 0 \\ ∗ & g_{2} \end{matrix}) , k_{2} = (\begin{matrix}g_{1} & ∗ \\ 0 & g_{3} \end{matrix}) , g_{1} ∈ GL(m, ℂ)\} . (7.69)

In what follows, we assume that p−q is even and calculate the K-types of Γ(O_{m}, ξ). There exists a well-defined character

ξ : (K̃_{ℂ})_{λ} ∋ (k_{1}, k_{2}, z) ↦ (det g_{1})^{−(p−q)/2}z ∈ ℂ^{×},

with the notation (7.69). By the construction of ξ, the coisotropy representation is the square of ξ:

det(Ad^{∗}|_{T_{λ}^{∗}O_{m}}) = ξ^{2}.

This means that ξ determines the half-form bundle \sqrt{Λ^{top}} on the orbit O_{m}. As a K̃_{ℂ}-module, the set of global sections Γ(O_{m}, ξ) is isomorphic to the induced module

Γ(O_{m}, ξ) = Ind K̃_{ℂ} ; (K̃_{ℂ})_{λ}ξ.

We define a character χ_{1} : K̃_{ℂ} ⟶ ℂ^{×} by χ_{1}(k_{1}, k_{2}, z) = z, and ξ′ : (K_{ℂ})_{λ} → ℂ^{×} by ξ′(k_{1}, k_{2}) = (det g_{1})^{−(p−q)/2} in the notation above. The character ξ′ lifts up to a character of (K̃_{ℂ})_{λ} via projection map, and we denote it by the same letter ξ′ again. Roughly speaking, χ_{1} equals “det^{m/2}k_{1} det^{−m/2}k_{2}”. Then, ξ is the tensor product of ξ′ with the restriction of χ_{1} to the subgroup (K̃_{ℂ})_{λ}. By the reciprocity law,

Ind K̃_{ℂ} ; (K̃_{ℂ})_{λ}ξ = χ_{1} ⊗ Ind K̃_{ℂ} ; (K̃_{ℂ})_{λ}ξ′ = χ_{1} ⊗ Ind _{(K_{ℂ})_{λ}}^{K_{ℂ}}ξ′.

Lemma 7.11. — We assume that m ≤ min(p, q) and p − q ∈ 2ℤ as before, and that max(p, q) ≠ m. Then, as a K_{ℂ}-module, we have an isomorphism

Ind _{(K_{ℂ})_{λ}}^{K_{ℂ}}ξ′ = \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} τ_{λ+l𝕀_{m}} ⊠ τ_{λ}^{∗}

with l = (q − p)/2. Here we denote 𝕀_{m} = (1, …, 1, 0, …, 0), in which 1 appears m-times.

This shows that

Γ(O_{m}, ξ) = \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix}(τ_{λ+l𝕀_{m}} ⊗ det^{m/2}) ⊠ (τ_{λ} ⊗ det^{m/2})^{∗}.

On the other hand, by (6.50), the lowest weight module L(χ^{−1}) also has the same K̃-types. Indeed, the character χ of G_{2} = U(m) is det^{(p−q)/2} = det^{−l} as is shown in (6.48). For ν = l𝕀_{m}, we see that the multiplicity m(λ, μ; ν) defined by (6.49) is

m(λ, μ; ν) = \{ \begin{matrix}1 & if λ = μ + ν \\ 0 & otherwise. \end{matrix}

Summarizing above, we have

Proposition 7.12. — Suppose p − q ∈ 2ℤ and 0 ≤ m < min(p, q). Let χ = det^{(p−q)/2} be the character of G_{2} = U(m) given in (6.48). Then the lowest weight module L(χ^{−1}) is isomorphic to Γ(O_{m}, ξ) as a K̃_{ℂ}-module.

7.5. The case G = O^{∗}(2p). — Let us consider the case G = O^{∗}(2p). In this case K = U(p), K_{ℂ} = GL(p, ℂ), 𝔭^{−} = Alt (p, ℂ). The action of k ∈ K_{ℂ} on A ∈ 𝔭^{−} is given by ^{t}k^{−1}Ak^{−1}.

Put r = ℝ-rank O^{∗}(2p) = [p/2], where [x] is the Gauss symbol. We define a subset of Alt (p, ℂ) by

O_{m} = {A ∈ Alt (p, ℂ) | rank (A) = 2m}, (m = 0, 1, …, r).

Since the rank of alternative matrices is always even, these {O_{0}, O_{1}, …, O_{r}} form the set of all K_{ℂ}-orbits on Alt (p, ℂ). The matrix ∑_{j=1}^{m}(E_{m+j,j} − E_{j,m+j}) is contained in O_{m}. The dimension of the orbit is given by

dim O_{m} = 2pm − m(2m + 1),

and the defining ideal I_{m} of \overline{O_{m}} is generated by

{D_{IJ} | I, J ⊂ {1, 2, …, 2p}, |I| = |J| = 2m + 1}.

Recall the dual pair (O^{∗}(2p), Sp(2m)). Let Sp(2m, ℂ) be the complexification of Sp(2m). We define the action of K_{ℂ} × Sp(2m, ℂ) on A ∈ M_{2m,p} by

(k, h) · A = hAk^{−1}, for k ∈ GL(p, ℂ) = K_{ℂ}, h ∈ Sp(2m, ℂ).

We define an unfolding map ψ by

ψ(A) = ^{t}AJ_{m}A for A ∈ M_{2m,p},

where J_{m} is defined as in (3.13). This is a polynomial map of degree two. Since

ψ( ∑_{j≤2m} E_{jj}) = ∑_{j=1}^{m}(E_{m+j,j} − E_{j,m+j}) ∈ O_{m},

we see that the image of ψ coincides with \overline{O_{m}}. With the trivial action of Sp(2m, ℂ) on \overline{O_{m}}, the map

ψ : M_{2m,p} → \overline{O_{m}}

is K_{ℂ} × Sp(2m, ℂ)-equivariant, that is, ψ(hAk^{−1}) = ^{t}k^{−1}ψ(A)k^{−1} for all k ∈ K_{ℂ} and h ∈ Sp(2m, ℂ). This map induces a ℂ-algebra homomorphism

ψ^{∗} : ℂ[\overline{O_{m}}] ∋ f ↦ f ∘ ψ ∈ ℂ[M_{2m,p}]^{Sp(2m,ℂ)}.

Lemma 7.13. — We have a ℂ-algebra isomorphism

ψ^{∗} : ℂ[\overline{O_{m}}] → ℂ[M_{2m,p}]^{Sp(2m,ℂ)},

which means that \overline{O_{m}} is the geometric quotient M_{2m,p}//Sp(2m, ℂ). In particular, \overline{O_{m}} is a normal variety.

Proof. — The proof is similar to that of Lemma 7.1.

Let us consider the Weil representation of the dual pair (O^{∗}(2p), Sp(2m)) ∈ Sp(2nm, ℝ) (n = 2p) and the unitary lowest weight module L(1_{Sp(2m)}). Take a Cartan subalgebra 𝔱 in 𝔨 consisting of diagonal matrices

𝔱 = {H = diag (a_{1}, …, a_{p}, −a_{1}, …, −a_{p}) | a_{i} ∈ ℂ}.

This is also a Cartan subalgebra of 𝔤. We define ε_{i} ∈ 𝔱^{∗} by ε_{i}(H) = a_{i} for above H ∈ 𝔱. Then the set of positive non-compact roots is

Δ_{n}^{+} = {ε_{i} + ε_{j} | 1 ≤ i < j ≤ p}.

Put

X_{ε_{a}+ε_{b}} = ( \begin{matrix}0 & E_{ab} − E_{ba} \\ 0 & 0 \end{matrix} ) ∈ 𝔬(2p, ℂ) = 𝔤.

Note that the complexification 𝔬(2p, ℂ) is given in (7.60), in which we adopt rather non-standard symmetric bilinear form S_{p}. Then X_{ε_{a}+ε_{b}} is a non-compact root vector in 𝔭^{+}. From the embedding (3.22) and the Fock realization (4.27) of Ω, we get

Ω(−2X_{ε_{a}+ε_{b}}) = ψ_{ab} = ∑_{j=1}^{m}(x_{aj}y_{bj} − x_{bj}y_{aj})  (1 ≤ a < b ≤ p),  (7.70)

where ((x_{aj})_{1≤a≤p,1≤j≤m}, (y_{bj})_{1≤b≤p,1≤j≤m}) ∈ M_{p,2m} = M_{2m,p}^{∗}. These quadratics

(7.70) generate the invariants S(M_{p,2m})^{Sp(2m,ℂ)}.

Lemma 7.14. — There are algebra isomorphisms

Ω(U(𝔭^{+})) ≃ ℂ[M_{2m,p}]^{Sp(2m,ℂ)} ≃ ℂ[\overline{O_{m}}] = ℂ[𝔭^{−}]/I_{m}.

We have Ann L(1_{Sp(2m)}) = Ann gr L(1_{Sp(2m)}) = I_{m} in U(𝔭^{+}).

Proof. — The proof is similar to that of Lemma 7.2.

Corollary 7.15. — (1) The associated variety of L(1_{Sp(2m)}) is \overline{O_{m}}.

(2) As a K-module, ℂ[\overline{O_{m}}] is isomorphic to L(1_{Sp(2m)}).

(3) The Bernstein degree of L(1_{Sp(2m)}) coincides with deg \overline{O_{m}}.

Proof. — The proof is similar to that of Corollary 7.3. For the K-type decomposition of L(1_{Sp(2m)}), see (6.58).

Let us define the natural filtration of L = L(1_{Sp(2m)}) by L_{k} = U_{k}(𝔭^{+})1, where 1 is the constant polynomial. By (6.59), we know

dim L_{k} = ∑_{{λ∈Pm \atop |λ|≤k}} dim τ_{λ^{#}}^{(p)}

= ∑_{{|λ|≤k \atop l(λ)≤m}} \frac{∏_{1≤i<j≤m}(λ_{i} − λ_{j} − 2i + 2j)^{2}((λ_{i} − λ_{j} − 2i + 2j)^{2} − 1)}{∏_{1≤i<j≤p}(j − i)}

× ∏_{{1≤i≤m \atop 2m+1≤j≤p}}(λ_{i} − 2i + j)(λ_{i} − 2i + 1 + j) ∏_{2m+1≤i<j≤p}(j − i)

= \frac{k^{m(m−1)/2×4+2m(p−2m)+m}}{∏_{i=1}^{2m}(p − i)!}

× ∫_{{0≤xm≤···≤x_{1} \atop x_{1}+···+xm≤1}} ∏_{1≤i<j≤m}(x_{i} − x_{j})^{4} ∏_{i=1}^{m} x_{i}^{2(p−2m)}dx_{1} ⋯ dx_{m}

+ (lower order terms of k)

= \frac{k^{m(2p−2m−1)}}{m! ∏_{i=1}^{2m}(p − i)!} ∫_{{0≤x_{i} \atop x_{1}+···+xm≤1}} ∏_{1≤i<j≤m} |x_{i} − x_{j}|^{4} ∏_{i=1}^{m} x_{i}^{2(p−2m)}dx_{1} ⋯ dx_{m}

+ (lower order terms of k)

for sufficiently large k.

Theorem 7.16. — Assume that m ≤ [p/2] = ℝ-rank O^{∗}(2p), and consider the reductive dual pair (O^{∗}(2p), Sp(2m)).

(1) The unitarizable lowest weight module L(1_{Sp(2m)}) of O^{∗}(2p) has the lowest weight m(1, …, 1) = m ∑_{i=1}^{p} ε_{i}, and its associated cycle is given by AC (L(1_{Sp(2m)})) = [\overline{O_{m}}].

(2) The Gelfand-Kirillov dimension and the Bernstein degree of L(1_{Sp(2m)}) is given by

Dim L(1_{Sp(2m)}) = dim \overline{O_{m}} = m(2p − 2m − 1),
Deg L(1_{Sp(2m)}) = deg \overline{O_{m}} = (2m − 1)!! ∏_{j=1}^{m} \frac{(2(j − 1))! (2(p − m − j))!}{(p − j)! (p − m − j)!}.

Proof. — By the formula of dim L_{k}, we get

Dim L(1_{Sp(2m)}) = m(2p − 2m − 1) =: d,
Deg L(1_{Sp(2m)}) = \frac{d!}{m! ∏_{i=1}^{2m}(p − i)!}I^{4}(2p − 4m, m).

Apply Theorem 7.4 to get the desired formula.

We have a relation between the half-form bundle and L(1_{Sp(2m)}) similar to that in

Proposition 7.6. We define λ = ( \begin{matrix}J_{m} & 0 \\ 0 & 0 \end{matrix} ) ∈ Alt (p, ℂ) with J_{m} = ( \begin{matrix}0 & −1_{m} \\ 1_{m} & 0 \end{matrix} ).

The isotropy subgroup of λ in K_{ℂ} is

(K_{ℂ})_{λ} = \{k = ( \begin{matrix}g_{1} & 0 \\ ∗ & g_{2} \end{matrix} ) | g_{1} ∈ Sp(2m, ℂ), g_{2} ∈ GL(p − 2m, ℂ)\} .

Therefore the determinant of the coisotropy representation becomes

det(Ad^{∗}|_{T_{λ}^{∗}O_{m}}) = (det k)^{2m},

and we define its square root by

ξ : (K_{ℂ})_{λ} ∋ k ↦ (det k)^{m} ∈ ℂ^{×}.

Proposition 7.17. — For m < p/2, the set of global section of the half-form bundle Γ(O_{m}, ξ) is isomorphic to L(1_{Sp(2m)}) as a K̃-module.

7.6. A unified formula. — Consider the reductive dual pair (G_{1}, G_{2}) ⊂ G = Sp(2nm, ℝ) of compact type. We put G = G_{1}, which is a non-compact companion. We use the notation in § 3 freely in this subsection. In particular, D = ℝ, ℂ, ℍ is a division algebra over ℝ, and n = 1/2 dim_{ℝ} V_{1}, m = dim_{D} V_{2}. Put r = ℝ-rank G, and α = dim_{ℝ} D = 1, 2, 4.

Summarizing the above three explicit calculations, we have a unified expression of the Gelfand-Kirillov dimension and the Bernstein degree of the unitary lowest weight module L(1_{G_{2}}).

Theorem 7.18. — Assume that the dual pair (G_{1}, G_{2}) is in the stable range, i.e., m ≤ r. We denote by L(1_{G_{2}}) the irreducible lowest weight module of G̃_{1} which is the (twisted) theta lift of the trivial representation of the compact companion G_{2}. Then the associated cycle AC L(1_{G_{2}}) is the closure of the m-th K_{ℂ}-orbit \overline{O_{m}} in 𝔭^{−}. Moreover, we have

Dim L(1_{G_{2}}) = m(n + 1 − \frac{α}{2}(m + 1)) = dim \overline{O_{m}} =: d,

and

Deg L(1_{G_{2}}) = F^{−1} \frac{d!}{m!}I^{α}(n − αm, m) = deg \overline{O_{m}},

where I^{α}(s, m) is the integral (7.63), and the integer F is given by

F = \{ \begin{matrix}∏_{j=1}^{m}(2(n − j))!! = 2^{m−d} ∏_{j=1}^{m}(n − j)! & Case (Sp, O), \\ ∏_{j=1}^{m}(p − j)!(q − j)! & Case (U, U), \\ ∏_{j=1}^{2m}(p − j)! & Case (O^{∗}, Sp). \end{matrix}

Remark 7.19. — If G/K is of tube type, we have

F = \{ \begin{matrix}∏_{j=1}^{m}(2(n − j))!! = 2^{m−d} ∏_{j=1}^{m}(n − j)! & Case (Sp, O), \\ ∏_{j=1}^{m}\{(n/α − j)!\}^{2} & Case (U, U), \\ ∏_{j=1}^{2m}(n/α − j)! & Case (O^{∗}, Sp). \end{matrix}

8. Multiplicity free action and Poincaré series

In this section, we develop a general theory on Poincaré series of graded modules which arise from multiplicity free action of reductive groups. All the groups in this section are complex algebraic groups and irreducible representations are finite dimensional ones.

8.1. Poincaré series of covariants. — Let G_{1} and G_{2} be complex reductive groups and X a vector space on which G_{1} and G_{2} jointly act linearly. We assume that the action of G_{1} ×G_{2} is multiplicity free. This means that the polynomial ring ℂ[X] = Γ(X) decomposes, as a G_{1} × G_{2}-module, into irreducible representations with multiplicity one. Namely, there exists a subset R_{X}(G_{1} × G_{2}) ⊂ Irr(G_{1} × G_{2}) such that

Γ(X) = \begin{matrix}∑ & ⊕ \\ π_{1}⊠π_{2}∈R_{X}(G_{1}×G_{2}) \end{matrix} π_{1} ⊠ π_{2}.

We assume further, in the decomposition, the correspondence π_{1} ↔ π_{2} is one to one. Hence, π_{1} determines π_{2} and vice versa.

We choose suitable positive systems of roots for G_{1} and G_{2}, and fix them in what follows. Let λ be the highest weight of π_{1} = π_{1}(λ) with respect to the positive system we chose. Then we will denote the corresponding highest weight of π_{2} by φ(λ) so that π_{2} = π_{2}(φ(λ)). Let Λ^{+} be a lattice semigroup of the highest weights of π_{1} ∈ Irr(G_{1}) which occur in Γ(X). Then we can write the decomposition as

Γ(X) = \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+} \end{matrix} π_{1}(λ) ⊠ π_{2}(φ(λ)).

Note that the correspondence Λ^{+} ∋ λ ↦ φ(λ) is a semigroup morphism from Λ^{+} into the dominant weight lattice of G_{2}, i.e., φ(λ + η) = φ(λ) + φ(η).

We consider a (reductive) spherical subgroup H of G_{1}. Since H is spherical, for any irreducible representation (π_{1}, V) of G_{1}, V has at most one-dimensional invariants under the action of H : dim V^{H} ≤ 1. We put

Λ^{+}(H) = {λ ∈ Λ^{+} | dim V_{λ}^{H} = 1} ⊂ Λ^{+},

where V_{λ} is a representation space of π_{1}(λ). Let Λ (respectively Λ(H)) be the lattice generated by Λ^{+} (respectively Λ^{+}(H)). Note that it is not necessary to hold that Λ^{+}(H) = Λ^{+} ∩ Λ(H). Since Λ^{+} is a free abelian semigroup generated by finite elements (see the argument in [26, §2]), we can extend the correspondence φ(·) to Λ as a group morphism.

The set of H-invariants of Γ(X) is denoted by Γ(X; 1_{H}) = ℂ[X]^{H}. Then it decomposes multiplicity freely as a G_{2}-module

Γ(X; 1_{H}) = ℂ[X]^{H} ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(H) \end{matrix} π_{2}(φ(λ)).

Since Γ(X; 1_{H}) is a finitely generated graded Noetherian algebra, it has a Poincaré series P (1_{H}; t), where t is an indeterminate. More precisely, we define P (1_{H}; t) in the following way. If the representation π_{1}(λ) ⊠ π_{2}(φ(λ)) occurs in the k-th degree of the polynomial ring Γ(X) = ℂ[X], we write |λ| = k. This degree map is obviously additive |λ + η| = |λ| + |η|. We put

P (1_{H}; t) = ∑_{λ∈Λ^{+}(H)} dim π_{2}(φ(λ)) t^{|λ|} = trace_{Γ(X;1_{H})}(t^{E}),  (8.71)

where E denotes the degree operator. Let {a_{1}, …, a_{d}} ⊂ ℂ[X]^{H} be a set of homogeneous and algebraically independent elements such that ℂ[X]^{H} is integral over a subalgebra ℂ[a_{1}, …, a_{d}] generated by a_{1}, … a_{d}. Put h_{i} = deg a_{i}. Then there exists a polynomial Q(t) such that

P (1_{H}; t) = \frac{Q(t)}{∏_{i=1}^{d}(1 − t^{h_{i}})},  (8.72)

and Q(1) gives a positive integer (see, e.g., [45, Theorem 2.5.6]). The integer Q(1) is independent of the choice of {a_{1}, …, a_{d}} above. We call it the degree of ℂ[X]^{H} and denote Q(1) = Deg Γ(X; 1_{H}). The number d coincides with the transcendental degree of the quotient field of ℂ[X]^{H}, and we denote it by d = Dim Γ(X; 1_{H}), which is the dimension of the geometric quotient X//H.

More generally, for any σ(μ) ∈ Irr(H) with highest weight μ, we denote σ(μ)- covariants of Γ(X) by Γ(X; σ(μ)), i.e.,

Γ(X; σ(μ)) := (σ(μ)^{∗} ⊗ ℂ[X])^{H}.

The space of covariants Γ(X; σ(μ)) is a finitely generated Γ(X; 1_{H}) = ℂ[X]^{H}-module by polynomial multiplication against the second factor (see, e.g., [39]). Note that it carries also a representation of G_{2} on the second factor.

If we decompose the restriction of π_{1}(λ) to H as

π_{1}(λ)|_{H} ≃ \begin{matrix}∑ & ⊕ \\ μ \end{matrix} m(λ, μ) σ(μ)

with multiplicity m(λ, μ), we have the decomposition

Γ(X; σ(μ)) ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+} \end{matrix} m(λ, μ) π_{2}(φ(λ)),

as a G_{2}-module. We define the Poincaré series P (σ(μ); t) of Γ(X; σ(μ)) by

P (σ(μ); t) = ∑_{λ∈Λ^{+}} m(λ, μ) dim π_{2}(φ(λ)) t^{|λ|}.

Since Γ(X; σ(μ)) is a finitely generated graded module over Γ(X; 1_{H}), its Poincaré series has rational expression as

P (σ(μ); t) = \frac{Q(σ(μ); t)}{∏_{i=1}^{d}(1 − t^{h_{i}})}

with the same d and h_{1}, …, h_{d} as in (8.72). Here, Q(σ(μ); t) is a polynomial in t and its value at t = 1 gives a non-negative integer, which is independent of the choice of a_{1}, …, a_{d} again. We call it the degree of covariants Γ(X; σ(μ)) and denotes Deg Γ(X; σ(μ)) = Q(σ(μ); 1).

The purpose of this subsection is to relate the dimension Dim Γ(X; σ(μ)) and the degree Deg Γ(X; σ(μ)) to those of invariants.

For “sufficiently large” λ, the multiplicity m(λ, μ) depends only on the coset [λ] = λ + Λ(H) ∈ Λ^{+}/Λ(H). Here Λ^{+}/Λ(H) is an abbreviation for (Λ^{+} + Λ(H))/Λ(H). To be precise, we have

Lemma 8.1 (Sato). — For any λ ∈ Λ^{+} and σ(μ) ∈ Irr(H), there exists η^{M} ∈ Λ^{+}(H) which satisfies

m(λ + η^{M}, μ) = m(λ + η^{M} + η, μ) (∀η ∈ Λ^{+}(H)).

The integer m(λ+η^{M}, μ) does not depend on the choice of η^{M}. We denote this integer by m([λ], μ) and call it the stable branching coefficient after F. Sato.

Proof. — Our setting here fits into Sato’s assumption [42].

Let Δ_{2}^{+} be a positive root system of G_{2}. We define a subset Δ_{2}^{+}(H) ⊂ Δ_{2}^{+} by

Δ_{2}^{+}(H) = {α ∈ Δ_{2}^{+} | ⟨φ(η), α⟩ = 0 (∀η ∈ Λ(H))},  (8.73)

where ⟨, ⟩ denotes the inner product which is invariant under the Weyl group action. For λ ∈ Λ^{+}, we put

r(λ) = r([λ]) = ∏_{α∈Δ_{2}^{+}(H)} \frac{⟨φ(λ) + ρ, α⟩}{⟨ρ, α⟩},  (8.74)

where ρ is the half sum of positive roots in Δ_{2}^{+}. Note that the right hand side of (8.74) does not depend on individual λ, but depends only on the coset [λ] ∈ Λ^{+}/Λ(H). By definition, r([λ]) is a positive quantity.

Proposition 8.2. — We assume that, for any λ ∈ Λ^{+}, there exists λ^{♭} ∈ Λ^{+} such that

(λ + Λ(H)) ∩ Λ^{+} = λ^{♭} + Λ^{+}(H).  (8.75)

Then, for any μ ∈ Λ^{+}(H), we have

lim_{t↑1} \frac{P (σ(μ); t)}{P (1_{H}; t)} = Deg Γ(X; 1_{H}) ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]).

Remark 8.3. — Condition (8.75) determines λ^{♭} ∈ Λ^{+} uniquely if it exists. Hence, λ^{♭} depends only on the coset [λ] = λ + Λ(H). If we set S = {λ^{♭} | λ ∈ Λ^{+}}, this amounts to

Λ^{+} = S ⊕ Λ^{+}(H);

or, equivalently to say, Λ^{+} is a free Λ^{+}(H)-module over the base set S. From this observation, the map

(·)^{♭} : Λ^{+}/Λ(H) ∋ [λ] ↦ [λ]^{♭} := λ^{♭} ∈ S ⊂ Λ^{+}

is a well-defined section of the projection map Λ^{+} → Λ^{+}/Λ(H).

Corollary 8.4. — Under the same assumption, we have

Deg Γ(X; σ(μ)) = Deg Γ(X; 1_{H}) ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]).

We need a technical lemma to prove the proposition.

Lemma 8.5. — Take arbitrary λ ∈ Λ^{+}.

(1) There exists η_{λ} ∈ Λ^{+}(H) such that

dim π_{2}(φ(λ + η)) ≤ r(λ) · dim π_{2}(φ(η_{λ} + η)) (∀η ∈ Λ^{+}(H)).

(2) We have

dim π_{2}(φ(λ + η)) ≥ r(λ) · dim π_{2}(φ(η)) (∀η ∈ Λ^{+}(H)).

Proof. — By Weyl’s dimension formula, we have

dim π_{2}(φ(λ + η)) = ∏_{α∈Δ_{2}^{+}} \frac{⟨φ(λ + η) + ρ, α⟩}{⟨ρ, α⟩}

= r(λ) ∏_{α∉Δ_{2}^{+}(H)} \frac{⟨φ(λ + η) + ρ, α⟩}{⟨ρ, α⟩}.  (8.76)

To prove (1), it is enough to take η_{λ} ∈ Λ^{+}(H) so that ⟨φ(λ), α⟩ ≤ ⟨φ(η_{λ}), α⟩ holds for any α ∉ Δ_{2}^{+}(H). This is certainly possible. Since φ(·) is a group homomorphism, (8.76) becomes

r(λ) ∏_{α∉Δ_{2}^{+}(H)} \frac{⟨φ(λ + η) + ρ, α⟩}{⟨ρ, α⟩} ≤ r(λ) ∏_{α∉Δ_{2}^{+}(H)} \frac{⟨φ(η_{λ} + η) + ρ, α⟩}{⟨ρ, α⟩}.

Now we are to prove (2). Since ⟨φ(λ), α⟩ ≥ 0, we get

dim π_{2}(φ(λ + η)) ≥ r(λ) ∏_{α∉Δ_{2}^{+}(H)} \frac{⟨φ(η) + ρ, α⟩}{⟨ρ, α⟩} = r(λ) dim π_{2}(φ(η)).

This proves (2).

Proof of Proposition 8.2.. — Let us take arbitrary 0 < t < 1.

First we note that m(λ, μ) ≤ m([λ], μ) for any λ (see [42, Corollary 1.2]). Therefore, we have

P (σ(μ); t) = ∑_{λ∈Λ^{+}} m(λ, μ) dim π_{2}(φ(λ)) t^{|λ|}
≤ ∑_{λ∈Λ^{+}} m([λ], μ) dim π_{2}(φ(λ)) t^{|λ|}
= ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ) ∑_{η∈Λ^{+}(H)} dim π_{2}(φ([λ]^{♭} + η)) t^{|[λ]^{♭}+η|}.(8.77)

For [λ]^{♭}, take η_{[λ]}_{♭} ∈ Λ^{+}(H) as in Lemma 8.5 (1), and recall the definition of P (1_{H}; t) from (8.71). Then we can calculate the above formula as

(8.77) ≤ ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) t^{|[λ]^{♭}|−|η_{_{[λ]}_{♭}} |}

× ∑_{η∈Λ^{+}(H)} dim π_{2}(φ(η_{[λ]}_{♭} + η)) t^{|η_{_{[λ]}_{♭}} +η|}

≤ ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) t^{|[λ]^{♭}|−|η_{_{[λ]}_{♭}} |} P (1_{H}; t).

Note that, for fixed μ, there are only a finite number of cosets [λ] for which m([λ], μ) does not vanish ([42, Corollary 2.5 (iii)]).

On the other hand, if we choose η^{M} ∈ Λ^{+}(H) large enough, we have m(λ + η^{M}, μ) = m([λ], μ) by the definition of the stable branching coefficient. We can take η^{M} uniformly for λ ∈ Λ^{+}, since there are only a finite number of [λ]’s which count. So, by Lemma 8.5 (2), we get the following inequality:

P (σ(μ); t) = ∑_{λ∈Λ^{+}} m(λ, μ) dim π_{2}(φ(λ)) t^{|λ|}
≥ ∑_{λ∈Λ^{+}} m(λ + η^{M}, μ) dim π_{2}(φ(λ + η^{M})) t^{|λ+η^{M} |}
= ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ) ∑_{η∈Λ^{+}(H)} dim π_{2}(φ([λ]^{♭} + η^{M} + η)) t^{|[λ]^{♭}+η^{M} +η|}

≥ ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) t^{|[λ]^{♭}+η^{M} |} ∑_{η∈Λ^{+}(H)} dim π_{2}(φ(η)) t^{|η|}

= ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) t^{|[λ]^{♭}+η^{M} |} P (1_{H}; t).

From these inequalities, we have

∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) t^{|[λ]^{♭}|−|η_{_{[λ]}_{♭}} |}

≥ \frac{P (σ(μ); t)}{P (1_{H}; t)} ≥ ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) t^{|[λ]^{♭}+η^{M} |}.

If we take the limit t ↑ 1, we get

lim_{t↑1} \frac{P (σ(μ); t)}{P (1_{H}; t)} = ∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]).

8.2. Examples of multiplicity free actions and Poincaré series. — We keep the notation in the former subsection §8.1. So G_{1} × G_{2} acts on X multiplicity freely, and H is a spherical subgroup of G_{1}.

In many cases, we have an identity

∑_{[λ]∈Λ^{+}/Λ(H)} m([λ], μ)r([λ]) = dim σ(μ).  (8.78)

It will prove that

Deg Γ(X; σ(μ)) = dim σ(μ) · Deg ℂ[X]^{H},  (8.79)

under the technical condition (8.75). However, at the same time, there also exist exceptions to (8.78). In this subsection, we will give three examples in which (8.78) and hence (8.79) hold. We need these examples later on.

Let B be a Borel subgroup of G_{1} such that HB ⊂ G_{1} is dense. Such a Borel subgroup exists since H is spherical. Define a parabolic subgroup P ⊂ G_{1} as

P = {g ∈ G_{1} | HBg = HB} ⊃ B.

Then L = P ∩ H is a reductive subgroup which contains the derived group of a Levi subgroup of P. The identity component of B ∩ H is a Borel subgroup of the identity component of L. Let B = T U be a Levi decomposition with T being a Cartan subgroup of G_{1}. We will denote by τ_{L}(λ) an irreducible representation of L with highest weight e^{λ}|_{H∩B}.

Let Φ^{+} be the semigroup lattice of dominant weights of G_{1} and Φ the weight lattice. We define

Φ^{+}(H) = {λ ∈ Φ^{+} | dim V_{λ}^{H} = 1},

and denote by Φ(H) a lattice generated by Φ^{+}(H) in Φ. It is known that

Φ(H) = {λ ∈ Φ | e^{λ}|_{H∩T} ≡ 1}.

To get the identity (8.78), we use Sato’s formula ([42, Corollary 2.5])

∑_{[λ]∈Φ^{+}/Φ(H)} m([λ], μ) dim τ_{L}(λ) = dim σ(μ).  (8.80)

However, there are two obstructions to get identity (8.78) by using Sato’s formula (8.80).

One obstruction is in the range of the summation. The representatives [λ] must move all the coset of dominant weight lattice in Sato’s formula. However, in general, Λ^{+}/Λ(H) is a strict subset of Φ^{+}/Φ(H). This obstruction is serious.

The other obstruction is the difference between r(λ) and dim τ_{L}(λ). However, in most cases, they are identical. We do not know an exception up to now.

We summarize here desired conditions which enables us to use Sato’s formula. (S1) Coincidence of coset spaces: Λ^{+}/Λ(H) = Φ^{+}/Φ(H). (S2) Coincidence of dimension functions: r(λ) = dim τ_{L}(λ) (∀λ ∈ Λ^{+}). (S3) Existence of good representatives: for any λ ∈ Λ^{+}, there exists λ^{♭} ∈ Λ^{+} such that

(λ + Λ(H)) ∩ Λ^{+} = λ^{♭} + Λ^{+}(H).  (8.81)

Condition (S3) is equivalent to the following condition (S3′) (see Remark 8.3).

(S3′) There is a subset S ⊂ Λ^{+} which satisfies Λ^{+} = S ⊕ Λ^{+}(H).

If once we check the above conditions, we conclude the formula (8.79).

Theorem 8.6. — If the above three conditions (S1)–(S3) hold, we have

Deg Γ(X; σ(μ)) = dim σ(μ) · Deg ℂ[X]^{H},  (8.82)

for any σ(μ) ∈ Irr(H).

In the following, we examine the above three conditions (S1)–(S3) in each case.

Example A.

Let G_{1} = GL(m, ℂ), G_{2} = GL(n, ℂ) and assume that m ≤ n. This assumption is essential in the following. We take H = SO(m, ℂ). Therefore (G_{1}, H) is a symmetric pair. We put X = M_{m,n}(ℂ) ≃ (ℂ^{m} ⊗ ℂ^{n})^{∗} and let G_{1} × G_{2} act naturally on X as

M_{m,n}(ℂ) ∋ A → ^{t}g_{1}^{−1}Ag_{2}^{−1}, (g_{i} ∈ G_{i}, i = 1, 2).

The decomposition of ℂ[X] is given by

ℂ[X] ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{m} \end{matrix} τ_{GL_{m}}(λ) ⊠ τ_{GL_{n}}(λ),

where P_{m} denotes the set of partitions with length at most m. Therefore the action of G_{1} × G_{2} is multiplicity free, and we have

Λ^{+} = P_{m}, Λ = Φ ≃ ℤ^{m}.

Since we can naturally identify λ ∈ P_{m} with φ(λ) ∈ P_{n}, we will denote φ(λ) simply by the same letter λ. If we denote by P_{m}^{even} the set of even partitions, then it is well-known that

Λ^{+}(H) = P_{m}^{even}, Λ(H) = Φ(H) ≃ (2ℤ)^{m}.

In this case, the coset space Λ^{+}/Λ(H) = Λ/Λ(H) ≃ (ℤ_{2})^{m} is a finite set, and it coincides with Φ^{+}/Φ(H).

We have Δ_{2}^{+}(H) = {ε_{i} − ε_{j} | m < i < j ≤ n} in the standard notation. Using this, one can conclude easily that r(λ) = 1 for any λ ∈ P_{m}. On the other hand, let B be a Borel subgroup consisting of upper triangular matrices. Then HB ⊂ G_{1} is dense and P = B. Since L = H ∩ P = H ∩ B ≃ (ℤ_{2})^{m−1}, L is a finite abelian group. Hence we have r(λ) = dim τ_{L}(λ) = 1.

Next we verify the condition (S3), i.e., (8.81). Put

ϖ_{i} = ∑_{k=1}^{i} ε_{k} = (1, …, 1, 0, …, 0) (1 ≤ i ≤ m),

the fundamental weights for GL(m, ℂ). Note that Λ^{+} has a basis {ϖ_{k} | 1 ≤ k ≤ m} and Λ^{+}(H) has a basis {2ϖ_{k} | 1 ≤ k ≤ m}. Any λ ∈ Λ^{+} can be expressed as

λ = ∑_{k=1}^{m} n_{k}ϖ_{k} (n_{k} ∈ ℤ_{≥0}).

We put

n_{k}^{♭} = \{ \begin{matrix}0 & if n_{k} ∈ 2ℤ, \\ 1 & otherwise \end{matrix}

Under the notation above, we define

λ^{♭} = ∑_{k=1}^{m} n_{k}^{♭}ϖ_{k}.

It is a simple task to verify that λ^{♭} satisfies the condition (8.81), and we have

Λ^{+} = Λ^{+}(H) ⊕ {∑_{k=1}^{m} n_{k}ϖ_{k} | n_{k} = 0, 1} .

From observations above, we conclude Theorem 8.6 holds in this case.

Example B.

Let G_{1} = GL(m, ℂ) × GL(m, ℂ), G_{2} = GL(p, ℂ) × GL(q, ℂ) and assume that m ≤ min(p, q). We take H = ΔGL(m, ℂ) ≃ GL(m, ℂ) the diagonal subgroup. Therefore (G_{1}, H) is a symmetric pair. Let n = p + q. We put

X = M_{m,n}(ℂ) = M_{m,p}(ℂ) ⊕ M_{m,q}(ℂ) ≃ (ℂ^{m} ⊗ ℂ^{p})^{∗} ⊕ (ℂ^{m} ⊗ ℂ^{q}),

and let ((x_{1}, x_{2}), (y_{1}, y_{2})) ∈ G_{1} × G_{2} act naturally on X as

M_{m,n}(ℂ) ∋ (A, B) → (^{t}x_{1}^{−1}Ay_{1}^{−1}, x_{2}B^{t}y_{2}) (A ∈ M_{m,p}(ℂ), B ∈ M_{m,q}(ℂ)).

Then the action is multiplicity free, and the decomposition of ℂ[X] is given by

ℂ[X] ≃ \begin{matrix}∑ & ⊕ \\ (μ,ν)∈P_{m}×P_{m} \end{matrix} (τ_{GL_{m}}(μ) ⊠ τ_{GL_{m}}(ν)^{∗}) ⊠ (τ_{GL_{p}}(μ) ⊠ τ_{GL_{q}}(ν)^{∗}) .

Therefore we have

Λ^{+} = P_{m} × P_{m}, Λ = Φ ≃ ℤ^{m} × ℤ^{m}.

Here, to avoid the confusion, we have twisted the second factor of Λ^{+} by −w_{0}, where w_{0} is the longest element in Weyl group. The correspondence between π_{1}(λ) and π_{2}(φ(λ)) is given by

λ = (μ, ν) ∈ P_{m} × P_{m} ↔ φ(λ) = (μ, ν) ∈ P_{p} × P_{q}

simply extended by zero. Again, we shall identify φ(λ) with λ.

Since τ_{GL_{m}}(μ) ⊠ τ_{GL_{m}}(ν)^{∗} contains non-trivial H-fixed vector if and only if μ = ν, we get

Λ^{+}(H) = ΔP_{m}, Λ(H) = Φ(H) ≃ Δℤ^{m}.

In this case, the coset space Λ^{+}/Λ(H) = Λ/Λ(H) ≃ ℤ^{m} is an infinite set, and it coincides with Φ^{+}/Φ(H).

We have

Δ_{2}^{+}(H) = {ε_{i} − ε_{j} | m < i < j ≤ p} ⊔ {δ_{i} − δ_{j} | m < i < j ≤ q},

in the standard notation, which concludes r(λ) = 1. We take a Borel subgroup B = B_{1} × \overline{B_{1}} ⊂ G_{1}, where B_{1} is the standard Borel subgroup of GL(m, ℂ) consisting of upper triangular matrices and \overline{B_{1}} is its opposite. Then HB ⊂ G_{1} is dense. Again, the parabolic subgroup P coincides with B. Hence L = H ∩ P = H ∩ B = ΔT_{1} is isomorphic to a maximal torus T_{1} in GL(m, ℂ). Therefore, we conclude that r(λ) = 1 = dim τ_{L}(λ).

For λ = (μ, ν) ∈ Λ^{+}, let

μ − ν = ∑_{k=1}^{m} n_{k}ϖ_{k} (n_{k} ∈ ℤ),

where {ϖ_{k}} is the set of fundamental weights of GL(m, ℂ). Put

μ^{♭} = ∑_{k} max(n_{k}, 0)ϖ_{k}, ν^{♭} = ∑_{k} max(−n_{k}, 0)ϖ_{k}.

If we define λ^{♭} = (μ^{♭}, ν^{♭}), it satisfies the condition (8.81). In this case, we get

Λ^{+} = Λ^{+}(H) ⊕ {(∑_{k=1}^{m} n_{k}ϖ_{k}, ∑_{k=1}^{m} n′_{k}ϖ_{k}) | n_{k}n′_{k} = 0, n_{k}, n′_{k} ∈ ℤ_{≥0}} .

Now we conclude that Theorem 8.6 also holds in this case.

Example C.

Let G_{1} = GL(2m, ℂ), G_{2} = GL(p, ℂ) and assume that 2m ≤ p. We take H = Sp(2m, ℂ). Therefore (G_{1}, H) is a symmetric pair. We realize Sp(2m, ℂ) as

Sp(2m, ℂ) = {g ∈ GL(2m, ℂ) | g diag (J_{2}, …, J_{2})^{t}g = diag (J_{2}, …, J_{2})} ,

where

J_{2} = [ \begin{matrix}0 & −1 \\ 1 & 0 \end{matrix} ] .

We put X = M_{2m,p}(ℂ) ≃ (ℂ^{m} ⊗ ℂ^{p})^{∗} and let G_{1} × G_{2} act naturally on X as

M_{2m,p}(ℂ) ∋ A → ^{t}g_{1}^{−1}Ag_{2}^{−1}, (g_{i} ∈ G_{i}, i = 1, 2).

The action of G_{1} × G_{2} is multiplicity free, and we have the decomposition of ℂ[X] as

ℂ[X] ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{2m} \end{matrix} τ_{GL_{2m}}(λ) ⊠ τ_{GL_{p}}(λ).

Therefore

Λ^{+} = P_{2m}, Λ = Φ ≃ ℤ^{2m}.

We shall identify λ ∈ P_{2m} with φ(λ) ∈ P_{p}. The irreducible representation π_{1}(λ) has a non-trivial H-fixed vector if and only if λ_{2i−1} = λ_{2i} (1 ≤ i ≤ m), i.e., λ = ∑_{k=1}^{m} n_{2k}ϖ_{2k} (n_{2k} ∈ ℤ_{≥0}). Therefore, we have

Λ^{+}(H) = ∑_{k=1}^{m} ℤ_{≥0} ϖ_{2k}, Λ(H) = Φ(H) = ∑_{k=1}^{m} ℤ ϖ_{2k} ≃ ℤ^{m}.

Then it is easy to see that

Λ^{+}/Λ(H) = Φ^{+}/Φ(H) ≃ ∑_{k=1}^{m} ℤ_{≥0} ϖ_{2k−1}.

For λ = ∑_{k=1}^{2m} n_{k}ϖ_{k} ∈ Λ^{+}, we define

λ^{♭} = ∑_{k=1}^{m} n_{2k−1}ϖ_{2k−1}.

Then it is clear that λ^{♭} satisfies the condition (8.81), hence we get

Λ^{+} = Λ^{+}(H) ⊕ {∑_{k=1}^{m} n_{k}ϖ_{2k−1} | n_{k} ∈ ℤ_{≥0}} .

Take a Borel subgroup of G_{1} consisting of upper triangular matrices. Then HB ⊂ G_{1} is dense and the parabolic subgroup P is given by

P = {diag (p_{1}, p_{2}, …, p_{m}) | p_{k} ∈ SL(2, ℂ)} · B.

Then we have

L = H ∩ P = {diag (p_{1}, p_{2}, …, p_{m}) | p_{k} ∈ SL(2, ℂ)} ≃ SL(2, ℂ)^{m}.

Therefore,

τ_{L}(λ) = τ_{SL_{2}}(λ_{1} − λ_{2}) ⊠ τ_{SL_{2}}(λ_{3} − λ_{4}) ⊠ ⋯ ⊠ τ_{SL_{2}}(λ_{2m−1} − λ_{2m}),

where τ_{SL_{2}}(μ) is an irreducible representation of SL(2, ℂ) with highest weight μ. Since dim τ_{SL_{2}}(μ) = μ + 1, dim τ_{L}(λ) is given by

dim τ_{L}(λ) = ∏_{k=1}^{m}(λ_{2k−1} − λ_{2k} + 1).

On the other hand, we have

Δ_{2}^{+}(H) = {ε_{2k−1} − ε_{2k} | 1 ≤ k ≤ m} ⊔ {ε_{i} − ε_{j} | 2m < i < j ≤ p}.

Hence we get

r(λ) = ∏_{k=1}^{m}(λ_{2k−1} − λ_{2k} + 1) = dim τ_{L}(λ).

Now we conclude that Theorem 8.6 is also valid in this case.

9. Associated cycle of unitary lowest weight modules

Let (G_{1}, G_{2}) be a reductive dual pair with G_{2} being compact. We often write G = G_{1} without subscription. We treat the three cases given in § 3 ; namely, (G, G_{2}) = (Sp(2n, ℝ), O(m)), (U(p, q), U(m)), or (O^{∗}(2p), Sp(2m)).

In this section, we will prove the following theorem.

Theorem 9.1. — We assume that the pair (G, G_{2}) is in the stable range where G_{2} is a smaller member, i.e., m ≤ ℝ-rank G.

Take a finite dimensional irreducible representation σ ∈ Irr(G_{2}). Then the corresponding representation L(σ) ∈ Irr(G̃) is a unitary lowest weight module of the metaplectic cover G̃ of G. The associated cycle of L(σ) is given by

AC L(σ) = dim σ · [\overline{O_{m}}],  (9.83)

where O_{m} is a nilpotent K_{ℂ}-orbit in 𝔭^{−} given in § 7.

Corollary 9.2. — Let the notation be as above. Then, the Gelfand-Kirillov dimension and the Bernstein degree of L(σ) are given by

Dim L(σ) = dim \overline{O_{m}}, Deg L(σ) = dim σ · deg \overline{O_{m}}.

Explicit formulas for dim \overline{O_{m}} and deg \overline{O_{m}} are given in Theorems 7.5, 7.10 and 7.16.

Let us prove Theorem 9.1 for the pair (Sp(2n, ℝ), O(m)). This pair is the most complicated one, because O(m) is not connected. The other pairs can be treated similarly.

Take an irreducible representation σ ∈ Irr(O(m)) and consider the lowest weight module L(σ) of G̃ = { \atop Sp(2n, ℝ)}. First, let us recall the Poincaré series (6.43) of L(σ)

P (L(σ); t^{2}) = t^{−|μ^{+}|} ∑_{λ∈P_{m}} m(λ, σ) dim τ_{λ}^{(n)} t^{|λ|},

where τ_{λ}^{(n)} is an irreducible finite dimensional representation of K_{ℂ} ≃ GL(n, ℂ) with highest weight λ ∈ P_{m} and P_{m} is the set of all partitions of length less than or equal to m.

We consider two cases, according to σ|_{SO(m)} is irreducible or not (see Lemma 6.1).

1) Let us assume that σ|_{SO(m)} is irreducible. We denote by σ(μ) ∈ Irr(SO(m)) the restriction, where μ is the highest weight. In this case, the branching coefficient m(λ, σ) satisfies

m(λ, σ) + m(λ, σ ⊗ det) = m(λ, μ),

where m(λ, μ) is the branching coefficient with respect to SO(m) , i.e.,

τ_{λ}^{(m)}|_{SO(m)} = \begin{matrix}∑ & ⊕ \\ μ \end{matrix} m(λ, μ)σ(μ).

This means that

t^{|μ|}P (L(σ); t^{2}) + t^{|μ|+m−2k}P (L(σ ⊗ det); t^{2}) = ∑_{λ∈P_{m}} m(λ, μ) dim τ_{λ}^{(n)}t^{|λ|}, (9.84)

where σ = σ(μ) (with the convention after Lemma 6.1) and k = ℓ(μ). The right hand side of (9.84) coincides with the Poincaré series P (σ(μ); t) of covariants of σ(μ) defined in § 8, if we take G_{1} = GL(m, ℂ) ⊃ H = SO(m, ℂ), G_{2} = GL(n, ℂ), and X = M_{m,n} = M_{n,m}^{∗} as in Example A there. To distinguish two types of Poincaré series, we shall write P (Γ(X; σ(μ)); t) instead of P (σ(μ); t) in this section.

Let d = Dim L(1_{O(m)}). Note that the Gelfand-Kirillov dimension of L(σ) and L(σ ⊗ det) also coincides with d. Then we have

lim_{t↑1}(1 − t^{2})^{d}{t^{|μ|}P (L(σ); t^{2}) + t^{|μ|+m−2k}P (L(σ ⊗ det); t^{2})}

= Deg L(σ) + Deg L(σ ⊗ det).  (9.85)

This implies that d = Dim Γ(X; σ(μ)) and

lim_{t↑1}(1 − t^{2})^{d}P (Γ(X; σ(μ)); t) = Deg Γ(X; σ(μ)).  (9.86)

Lemma 9.3. — For any σ ∈ Irr(O(m)), we have

Deg L(σ) = Deg L(σ ⊗ det).

Proof. — We denote a subspace of the symmetric algebra S(M_{n,m}) = ℂ[M_{n,m}^{∗}] on which O(m) acts via σ by V_{σ}. Then the representation space of L(σ) is identified with the σ-covariants (V_{σ} ⊗ σ^{∗})^{O(m)}. In order to get the K̃-action on it, we must twist it by (det k)^{m/2} (k ∈ GL(n, ℂ)), though it does not affect on the gradation itself. Since we only consider the Poincaré series, we simply ignore this twist.

Put

δ = det(E_{ij})_{1≤i,j≤m} ∈ S(M_{n,m}),

where E_{ij} is the matrix unit. Then, clearly δ represents det ∈ Irr(O(m)). The multiplication by δ maps V_{σ} injectively to V_{σ⊗det},

δ : V_{σ} ⟶ V_{σ⊗det}.

This map increases the degree by deg δ = m^{2}, and we conclude that

t^{m^{2}}P (L(σ); t) ≤ P (L(σ ⊗ det); t)

for 0 < t < 1. Since (σ ⊗ det) ⊗ det = σ, we finally get

t^{2m^{2}}P (L(σ); t) ≤ t^{m^{2}}P (L(σ ⊗ det); t) ≤ P (L(σ); t).

If we multiply (1 − t)^{d} (d = Dim L(σ)) and take limit t ↑ 1, we get

Deg L(σ) ≤ Deg L(σ ⊗ det) ≤ Deg L(σ).

By Lemma 9.3, formulas (9.3) and (9.86) imply

Deg L(σ) = Deg L(σ ⊗ det) = 2^{−1} Deg Γ(X; σ(μ)).  (9.87)

Consider a special case where σ = 1_{O(m)}, the trivial representation of O(m). Then the above formula (9.87) becomes

Deg L(1_{O(m)}) = Deg L(det) = 2^{−1} Deg Γ(X; 1_{SO(m)}).  (9.88)

Theorem 8.6 implies that

Deg L(σ) = 2^{−1} Deg Γ(X; σ(μ))
= 2^{−1} dim σ(μ) Deg Γ(X; 1_{SO(m)}) = dim σ(μ) Deg L(1_{O(m)}).

Since the associated cycle of L(σ) is a multiple of \overline{O_{m}}, the multiplicity is given by

Deg L(σ)/ deg \overline{O_{m}} = Deg L(σ)/ Deg L(1_{O(m)}) = dim σ(μ) = dim σ

(cf. Theorems 1.4 and 7.5).

2) Assume that σ|_{SO(m)} = σ(μ^{+}) ⊕ σ(μ^{−}) as in Lemma 6.1 (2). Then it is easy to see that

m(λ, σ) = m(λ, μ^{+}) = m(λ, μ^{−}).

Therefore we have

t^{|μ^{+}|}P (L(σ); t^{2}) = ∑_{λ∈P_{m}} m(λ, μ^{+}) dim τ_{λ}^{(n)}t^{|λ|} = P (Γ(X; σ(μ^{+})); t).

Multiply (1 − t^{2})^{d} both hand sides, and take limit t ↑ 1. Then we get

Deg L(σ) = Deg Γ(X; σ(μ^{+})) = dim σ(μ^{+}) Deg Γ(X; 1_{SO(m)}).

By (9.88), we get

Deg L(σ) = 2 dim σ(μ^{+}) Deg L(1_{O(m)}) = dim σ Deg L(1_{O(m)}),

which proves (9.83) by the same reasoning as 1). This completes the proof of Theorem 9.1 for the pair (Sp(2n, ℝ), O(m)).

For the other pairs, we use Examples B and C in § 8 instead of Example A. In these cases, we have

Deg L(σ) = Deg Γ(X; σ)

for appropriate choice of X. This formula and Theorem 8.6 prove the theorem by almost the same arguments above.

References

[1] D. Barbasch and D. A. Vogan, Jr., The local structure of characters, J. Funct. Anal. 37 (1980), no. 1, 27–55.

[2] W. Borho and J.-L. Brylinski, Differential operators on homogeneous spaces. I. Irreducibility of the associated variety for annihilators of induced modules, Invent. Math. 69 (1982), no. 3, 437–476.

[3] W. Borho and J.-L. Brylinski, Differential operators on homogeneous spaces. III. Characteristic varieties of Harish-Chandra modules and of primitive ideals, Invent. Math. 80 (1985), no. 1, 1–68.

[4] R. Brylinski and B. Kostant, Minimal representations of E_{6}, E_{7}, and E_{8} and the generalized Capelli identity, Proc. Nat. Acad. Sci. U.S.A. 91 (1994), no. 7, 2469–2472.

[5] R. Brylinski and B. Kostant, Minimal representations, geometric quantization, and unitarity, Proc. Nat. Acad. Sci. U.S.A. 91 (1994), no. 13, 6026–6029.

[6] R. Brylinski and B. Kostant, Lagrangian models of minimal representations of E_{6}, E_{7} and E_{8}, in Functional analysis on the eve of the 21st century, Vol. 1 (New Brunswick, NJ, 1993), 13–63, Progr. Math., 131, Birkhäuser, Boston, Boston, MA, 1995.

[7] J.-T. Chang, Characteristic cycles of holomorphic discrete series, Trans. Amer. Math. Soc. 334 (1992), no. 1, 213–227.

[8] J.-T. Chang, Characteristic cycles of discrete series for ℝ-rank one groups, Trans. Amer. Math. Soc. 341 (1994), 603–622.

[9] M. G. Davidson, T. J. Enright and R. J. Stanke, Differential operators and highest weight representations, Mem. Amer. Math. Soc. 94 (1991), no. 455, iv+102 pp.

[10] D. Eisenbud, Commutative Algebra with a View Toward Algebraic Geometry, GTM 150, Springer, New York, 1995.

[11] T. Enright, R. Howe and N. Wallach, A classification of unitary highest weight modules, in Representation theory of reductive groups (Park City, Utah, 1982), 97–143, Progr. Math., 40, Birkhäuser, Boston, Boston, Mass., 1983.

[12] W. Fulton, Intersection theory, Springer, Berlin, 1984.

[13] W. Fulton, Young tableaux, With applications to representation theory and geometry. Cambridge Univ. Press, Cambridge, 1997.

[14] S. Gelbart, Holomorphic discrete series for the real symplectic group, Invent. Math. 19 (1973), 49–58.

[15] Giovanni Zeno Giambelli, Sulle varietà rappresentate coll’annulare determinanti minori contenuti in un determinante simmetrico generico di forme, Atti R. Accad. Sci. Torino 41 (1906), 102 – 125.

[16] J. Harris, Algebraic geometry, A first course. Corrected reprint of the 1992 original, Springer, New York, 1995.

[17] J. Harris and L. W. Tu, On symmetric and skew-symmetric determinantal varieties, Topology 23 (1984), no. 1, 71–84.

[18] T. Hayata, Whittaker functions of generalized principal series on SU(2, 2), J. Math. Kyoto Univ. 37 (1997), no. 3, 531–546.

[19] T. Hayata and T. Oda, An explicit integral representation of Whittaker functions for the representations of the discrete series—the case of SU(2, 2), J. Math. Kyoto Univ. 37 (1997), no. 3, 519–530.

[20] S. Helgason, Differential geometry, Lie groups, and symmetric spaces, Academic Press, New York, 1978.

[21] R. Hotta, Rings and Fields, I., Iwanami Lecture Series, The Foundation of Modern Mathematics 15, Iwanami Shoten 1997.

[22] R. Howe, Reciprocity laws in the theory of dual pairs, in Representation theory of reductive groups (Park City, Utah, 1982), 159–175, Progr. Math., 40, Birkhäuser, Boston, Boston, Mass., 1983.

[23] R. Howe, Perspectives on invariant theory: Schur duality, multiplicity-free actions and beyond, in The Schur lectures (1992) (Tel Aviv), 1–182, Bar-Ilan Univ., Ramat Gan, 1995.

[24] R. Howe, Remarks on classical invariant theory, Trans. Amer. Math. Soc. 313 (1989), no. 2, 539–570. Erratum to: “Remarks on classical invariant theory”, Trans. Amer. Math. Soc. 318 (1990), no. 2, 823.

[25] R. Howe, Dual pairs in physics: harmonic oscillators, photons, electrons, and singletons, in Applications of group theory in physics and mathematical physics (Chicago, 1982), 179–207, Amer. Math. Soc., Providence, R.I., 1985.

[26] R. Howe and T. Umeda, The Capelli identity, the double commutant theorem, and multiplicity-free actions, Math. Ann. 290 (1991), no. 3, 565–619.

[27] A. Joseph, The minimal orbit in a simple Lie algebra and its associated maximal ideal, Ann. Sci. École Norm. Sup. (4) 9 (1976), no. 1, 1–29.

[28] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), no. 1, 1–47.

[29] A. A. Kirillov, Characters of unitary representations of Lie groups, Funkcional. Anal. i Priložen 2 (1968), no. 2 40–55.

[30] B. Kostant, On Whittaker vectors and representation theory, Invent. Math. 48 (1978), no. 2, 101–184.

[31] B. Kostant and S. Rallis, Orbits and representations associated with symmetric spaces, Amer. J. Math. 93 (1971), 753–809.

[32] T. E. Lynch, Generalized Whittaker vectors and representation theory, Thesis, M.I.T., 1979.

[33] H. Matsumura, Commutative ring theory, Translated from the Japanese by M. Reid, Cambridge Univ. Press, Cambridge, 1986.

[34] H. Matumoto, Whittaker vectors and associated varieties, Invent. math. 89(1987), 219 – 224.

[35] H. Matumoto, C^{−∞}-Whittaker vectors for complex semisimple Lie groups, wave front sets, and Goldie rank polynomial representations, Ann. Sci. École Norm. Sup. (4) 23 (1990), no. 2, 311–367.

[36] H. Matumoto, Whittaker vectors and the Goodman-Wallach operators, Acta Math. 161 (1988), no. 3-4, 183–241.

[37] K. Nishiyama and H. Ochiai, Bernstein degree of singular unitary highest weight representations of the metaplectic group. Proc. Japan Acad., 75, Ser. A (1999), 9 – 11.

[38] T. Oda, An explicit integral representation of Whittaker functions on Sp(2; R) for the large discrete series representations, Tôhoku Math. J. (2) 46 (1994), no. 2, 261–279.

[39] V. L. Popov and E. B. Vinberg, Invariant theory. in Algebraic geometry. IV, A translation of Algebraic geometry. 4 (Russian), Akad. Nauk SSSR Vsesoyuz. Inst. Nauchn. i Tekhn. Inform., Moscow, Translation edited by A. N. Parshin and I. R. Shafarevich, Encyclopaedia of Mathematical Sciences, 55. Springer, Berlin, 1994.

[40] S. Rallis and G. Schiffmann, Weil representation. I. Intertwining distributions and discrete spectrum, Mem. Amer. Math. Soc. 25 (1980), no. 231, iii+203 pp.

[41] W. Rossmann, Kirillov’s character formula for reductive Lie groups, Invent. Math. 48 (1978), no. 3, 207–220.

[42] F. Sato, On the stability of branching coefficients of rational representations of reductive groups, Comment. Math. Univ. St. Paul. 42 (1993), no. 2, 189–207.

[43] W. Schmid, On the characters of the discrete series. The Hermitian symmetric case, Invent. Math. 30 (1975), no. 1, 47–144.

[44] W. Schmid and K. Vilonen, Characteristic cycles and wave front cycles of representations of reductive groups, to appear in Annals of Math.

[45] T. A. Springer, Invariant theory, Lecture Notes in Mathematics, Vol. 585. Lecture Notes in Math., 585, Springer, Berlin, 1977.

[46] K. Taniguchi, Discrete series Whittaker functions of SU(n, 1) and Spin(2n, 1), J. Math. Sci. Univ. Tokyo 3 (1996), no. 2, 331–377.

[47] P. Torasso, Sur le caractère de la représentation de Shale-Weil de Mp(n, R) et Sp(n, C), Math. Ann. 252 (1980), no. 1, 53–86.

[48] E. B. Vinberg and V. L. Popov, On a class of quasihomogeneous affine varieties. Math. USSR Izvestja, 6 (1972), 743 – 758.

[49] D. A. Vogan, Jr., Gel’fand-Kirillov dimension for Harish-Chandra modules, Invent. Math. 48 (1978), no. 1, 75–98.

[50] D. Vogan, Singular unitary representations. In Non commutative harmonic analysis and Lie groups, LNM 880(1981), pp. 506 – 535.

[51] D. A. Vogan, Jr., Associated varieties and unipotent representations, in Harmonic analysis on reductive groups (Brunswick, ME, 1989), 315–388, Progr. Math. 101, Birkhäuser, Boston, Boston, MA, 1991.

[52] D. A. Vogan, Jr., The Method of Coadjoint Orbits for Real Reductive Groups, in Representation Theory of Lie Groups, 177–238, IAS/Park City math. ser. 8, Amer. Math. Soc., (2000).

[53] H. Yamashita, Embeddings of discrete series into induced representations of semisimple Lie groups. II. Generalized Whittaker models for SU(2, 2), J. Math. Kyoto Univ. 31 (1991), no. 2, 543–571.

[54] H. Yamashita, private communication (1999). (See Yamashita’s article in this volume.)

Kyo Nishiyama, Faculty of Integrated Human Studies, Kyoto University, Kyoto 606-8501, Japan

E-mail : kyo@math.h.kyoto-u.ac.jp Hiroyuki Ochiai, Department of Mathematics, Kyushu University, Hakozaki, Fukuoka 812-

8581, Japan • E-mail : ochiai@math.kyushu-u.ac.jp Kenji Taniguchi, Department of Mathematics, Aoyama Gakuin University, 6-16-1, Chitose-

dai, Tokyo 157-8572, Japan • E-mail : taniken@gem.aoyama.ac.jp