CLASSIFICATION OF SPHERICAL NILPOTENT ORBITS FOR U(p, p)

KYO NISHIYAMA

To Mai

Abstract. We consider the symmetric pair (G, K) = (U(p, q), U(p) × U(q)). For this pair, we classify spherical nilpotent K_{ℂ}-orbits which are theta lift in the stable range. Moreover, for the pair (G, K) = (U(p, p), U(p) × U(p)) where p = q, we prove that a spherical nilpotent K_{ℂ}-orbit must be a theta lift. As a consequence, we get a complete classification of the spherical nilpotent K_{ℂ}-orbits for the symmetric pair (U(p, p), U(p) × U(p)).

Contents

Introduction 1 1. Theta lifting of nilpotent orbits 3 2. Spherical nilpotent orbits 6 2.1. Spherical lifted orbits 6 2.2. Non-spherical lifted orbits 8 Case (a) 9 Case (b) 9 Case (c) 10 3. Classification of spherical nilpotent orbits 11 References 12

Introduction

Let G be a reductive Lie group and K a maximal compact subgroup of G. We denote by 𝔤 (resp. 𝔨) the complexified Lie algebra of G (resp. K). Then the choice of K determines a Cartan decomposition 𝔤 = 𝔨⊕𝔰, and the complexification K_{ℂ} acts on 𝔰 by the restriction of the adjoint action. Let N (𝔰) be the nilpotent variety consisting of all nilpotent elements

Date: Ver. 1.0 [Fri Jun 28 13:55:38 JST 2002] (compiled on June 28, 2002). 1991 Mathematics Subject Classification. Primary . Key words and phrases. reductive dual pair, theta lifting, nilpotent orbits, spherical orbit, invariant

theory.

in 𝔰. The action of K_{ℂ} preserves N (𝔰) and it has a finite number of K_{ℂ}-orbits ([5]). We call them nilpotent K_{ℂ}-orbits for the symmetric pair (G, K), or just for G.

In this article, we investigate spherical nilpotent K_{ℂ}-orbits for the pair (G, K) = (U(p, q), U(p) × U(q)). A nilpotent K_{ℂ}-orbit 𝕆 is called spherical if a Borel subgroup of K_{ℂ} has an open dense orbit in 𝕆. These orbits are relatively small and play a fundamental role in the representation theory of G (see, for example, [4], [7], [8], etc.). For a complex simple algebraic group G_{ℂ}, there is a complete classification of spherical nilpotent G_{ℂ}-orbits by Panyushev ([12]). However, for the case of symmetric pairs, there seems no such classification of spherical nilpotent orbits. Here, as a first step, we give a complete classification for the pair (U(p, p), U(p) × U(p)).

Our main tool of the classification is the notion of theta lifting of nilpotent orbits in the stable range (see §1 ; we refer [9] for detail). In fact, we classify all the spherical nilpotent K_{ℂ}-orbits for G = U(p, q) which are theta lift in the stable range.

Theorem A (Theorem 2.1). Let (G, K) = (U(p, q), U(p) × U(q)) as above. Let D be a signed Young diagram of signature (p, q) and 𝕆_{D} be the corresponding nilpotent K_{ℂ}-orbit for G = U(p, q). Then 𝕆_{D} is a theta lift in the stable range if and only if ℓ(D) ≥ max{p, q}, where ℓ(D) is the length of D.

If 𝕆_{D} is a theta lift, then it is spherical if and only if the shape of D is of the following form:

[3^{ε} · 2^{k} · 1^{l}] ε = 0, 1; k, l ≥ 0; 3ε + 2k + l = p + q.

Since the length of [3^{ε} · 2^{k} · 1^{l}] is ε + k + l, the above theorem tells us that the orbit corresponding to [3^{ε} · 2^{k} · 1^{l}] with ε + k + l ≥ max{p, q} is spherical. According to the classification by Panyushev, spherical G_{ℂ}-orbits for type A is given by Young diagrams [2^{k} · 1^{l}]. Therefore, Theorem A provides an example of a spherical nilpotent K_{ℂ}-orbit whose G_{ℂ}-hull is not spherical.

Unfortunately, a spherical nilpotent orbit need not be a theta lift in the stable range in general. However, it is so in the case of p = q. We prove the following theorem.

Theorem B (Theorem 3.1). Let (G, K) = (U(p, p), U(p) × U(p)) and D a signed Young diagram of signature (p, p). The corresponding nilpotent K_{ℂ}-orbit 𝕆_{D} is spherical if and only if the shape of D is of the following form:

[3^{ε} · 2^{k} · 1^{l}] ε = 0, 1; k, l ≥ 0; 3ε + 2k + l = 2p and ε + k + l ≥ p.

In particular, spherical nilpotent K_{ℂ}-orbits are obtained by the theta lifting in the stable range.

Acknowledgement. We thank Chengbo Zhu, Roger Howe and David Vogan for useful discussions.

1. Theta lifting of nilpotent orbits

In this section, we review the notion of the theta lift of nilpotent orbits for symmetric pairs in the case of indefinite unitary groups. For general and detailed discussion, see [9].

We denote U(p, q) and its maximal compact subgroup U(p) × U(q) by G and K respectively. Let us begin with the description of the nilpotent orbits for the symmetric pair (G, K) = (U(p, q), U(p) × U(q)). Let 𝔤 be the complexification of the Lie algebra Lie (G), and similarly 𝔨 denotes the complexification of Lie (K). Let 𝔤 = 𝔨 + 𝔰 be the corresponding Cartan decomposition over ℂ. We realize it as

𝔤 = 𝔤𝔩_{p+q}(ℂ) = (\begin{matrix}M_{p}(ℂ) & 0 \\ 0 & M_{q}(ℂ) \end{matrix}) ⊕ (\begin{matrix}0 & M_{p,q}(ℂ) \\ M_{q,p}(ℂ) & 0 \end{matrix}) = 𝔨 ⊕ 𝔰.

For a subset S ⊂ 𝔤, we denote by N (S) the set of all nilpotent elements in S. Then, we call N (𝔰) the nilpotent variety for the symmetric pair (G, K). Let us denote by the complexified algebraic group of K by K_{ℂ}. Then K_{ℂ} acts on 𝔰 via the restriction of the adjoint action, and preserves the nilpotent variety N (𝔰). It is well known that the K_{ℂ}-orbits in N (𝔰) is finite, and that they are classified by the signed Young diagrams. Namely, the K_{ℂ}-orbits in N (𝔰) are classified by the Young diagrams of size p + q, whose boxes are occupied by p plus signs and q minus signs. The signs must appear mutually in rows. For details, see [1, Th. 9.3.3] for example.

Take another unitary group and put G′ = U(m, n). We assume the stable range condition m + n ≤ p, q. The pair (G, G′) forms a reductive dual pair in a large symplectic group 𝔾 = Sp(2(p + q)(m + n), ℝ). In the symplectic group, there are four reductive dual pairs related to (G, G′), which is called diamond pairs [2]. Let us denote:

M = U(p, q)^{2}  ⊃^{Δ}  G = U(p, q)
∪  ∪
L = L^{+} × L^{−} = U(p)^{2} × U(q)^{2} ⊃^{Δ} K = K^{+} × K^{−} = U(p) × U(q),

where Δ denotes the diagonal embedding. Similarly, we denote M′, G′, L′, K′ replacing p and q by m and n respectively. Then, the four dual pairs in 𝔾 are (G, G′), (M, K′), (L, L′), (K, M′). Put

W = M_{p+q,m+n}(ℂ) = \{(\begin{matrix}A & B \\ C & D \end{matrix}) | A ∈ M_{p,m}, B ∈ M_{p,n}, C ∈ M_{q,m}, D ∈ M_{q,n}\}
= W^{+} ⊕ W^{−} = M_{p,m+n}(ℂ) ⊕ M_{q,m+n}(ℂ).

Then K_{ℂ} = GL_{p}(ℂ) × GL_{q}(ℂ) and K′_{ℂ} = GL_{m}(ℂ) × GL_{n}(ℂ) act on W as

((g_{1}, g_{2}), (h_{1}, h_{2})) · (\begin{matrix}A & B \\ C & D \end{matrix}) = ( \begin{matrix}g_{1}A^{t}h_{1} & ^{t}g_{1}^{−1}Bh_{2}^{−1} \\ ^{t}g_{2}^{−1}Ch_{1}^{−1} & g_{2}D^{t}h_{2} \end{matrix}) ,
(g_{1}, g_{2}) ∈ GL_{p}(ℂ) × GL_{q}(ℂ), (h_{1}, h_{2}) ∈ GL_{m}(ℂ) × GL_{n}(ℂ).

We fix a Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔰 as above, and 𝔤′ = 𝔨′ ⊕ 𝔰′ is chosen similarly. We identify

𝔰 = M_{p,q}(ℂ) ⊕ M_{q,p}(ℂ), 𝔰′ = M_{m,n}(ℂ) ⊕ M_{n,m}(ℂ).

Then there is a natural double fibration of W by 𝔰 and 𝔰′, which are explicitly given as follows.

φ : W ∋ (\begin{matrix}A & B \\ C & D \end{matrix}) ⟼ (A^{t}C, D^{t}B) ∈ 𝔰, ψ : W ∋ (\begin{matrix}A & B \\ C & D \end{matrix}) ⟼ (^{t}AB, ^{t}DC) ∈ 𝔰′.

These double fibration maps are called moment maps, and they are K_{ℂ} × K′_{ℂ}-equivariant with the trivial K_{ℂ}-action on 𝔰′, and the trivial K′_{ℂ}-action on 𝔰 respectively. The maps φ and ψ are almost the same. Therefore we will treat only the map ψ in the following.

The map ψ induces an algebra homomorphism ψ^{∗} : ℂ[𝔰′] → ℂ[W] by

ψ^{∗}(x_{i,j}) = ∑_{k=1}^{p} a_{k,i}b_{k,j}, X = (x_{i,j}) ∈ M_{m,n}(ℂ),

ψ^{∗}(y_{i,j}) = ∑_{l=1}^{q} c_{l,j}d_{l,i}, Y = (y_{i,j}) ∈ M_{n,m}(ℂ).

GL_{p}-invariants (resp. GL_{q}-invariants) on ℂ[W^{+}] (resp. ℂ[W^{−}]) are generated by ψ(x_{i,j})^{∗}’s (resp. ψ(y_{i,j})^{∗}’s). Hence ψ : W → 𝔰′ is an affine quotient map by K_{ℂ}, which is surjective under the condition of the stable range. Notice that φ is not surjective in general.

Note that φ(ψ^{−1}(N (𝔰′))) ⊂ N (𝔰), i.e., φ ∘ ψ^{−1} carries nilpotent elements to nilpotent elements. This easily follows from the fact that (A, B) ∈ M_{p,q} ⊕ M_{q,p} = 𝔰 belongs to N (𝔰) if and only if AB is a nilpotent matrix.

Take a nilpotent K′_{ℂ}-orbit 𝕆′. Since φ and ψ are K_{ℂ} × K′_{ℂ}-equivariant, φ(ψ^{−1}(\overline{𝕆′})) ⊂ N (𝔰) is a union of K_{ℂ}-orbits. However, it has much better properties.

Theorem 1.1 ([9]). Assume the stable range condition m + n ≤ p, q. Then the scheme theoretic inverse image ψ^{−1}(\overline{𝕆′}) = \overline{𝕆′} ×_{𝔰′} W is reduced and irreducible. Its image φ(ψ^{−1}(\overline{𝕆′})) is the closure of a single nilpotent K_{ℂ}-orbit 𝕆 : φ(ψ^{−1}(\overline{𝕆′})) = \overline{𝕆}.

Thus we have a correspondence

θ : N (𝔰′)/K′_{ℂ} ∋ 𝕆′ ⟼ 𝕆 ∈ N (𝔰)/K_{ℂ},

which is called the theta lifting. Let D′ be a signed Young diagram corresponding to 𝕆′, and we denote it by 𝕆′ = 𝕆′_{D′}. Then the Young diagram D corresponding to the theta lift 𝕆_{D} = θ(𝕆′_{D′}) is obtained by adding an extra box to the end of each row of D′ ; if the row is empty, we place one box and make the total number of boxes equal to p + q. The signs in the added boxes are automatically determined by those in D′. For more detailed description of the correspondence D′ → D, see [11].

Example 1.2. Here we give an example for the pair U(4, 4) × U(2, 1).

+ ; + ; − ⟶ + − ; + − ; − + ; + ; −  + − ; + ⟶ + − + ; + − ; + ; − ; −  − + ; + ⟶ − + − ; + − ; + ; + ; −  + − + ⟶ + − + − ; + ; + ; − ; −

If we take various G′ = U(m, n) satisfying m + n ≤ p, q, we obtain a number of lifted nilpotent K_{ℂ}-orbits. However, they do not exhaust all the nilpotent orbits. Let 𝕆_{D} ⊂ N (𝔰) be a nilpotent K_{ℂ}-orbit corresponding to the signed Young diagram D. By the length ℓ = ℓ(D) of D, we mean the number of non-empty rows in D (in other words, the number of boxes in the first column). Then we have the following

Lemma 1.3. A nilpotent K_{ℂ}-orbit 𝕆_{D} is a theta lift from certain nilpotent orbit in the stable range if and only if ℓ(D) ≥ max{p, q} holds.

Remark 1.4. We make a convention that the trivial orbit {0} is lifted from the (ideal) trivial orbit of the trivial group U(0, 0). This enables us to state our results in a uniform fashion.

Proof. The original D′ should be the diagram obtainable by deleting the last box in each row of D. So, we delete ℓ(D) boxes. Therefore the condition of the stable range becomes m + n = p + q − ℓ(D) ≤ p, q, which is equivalent to the condition given in the lemma.

Since K_{ℂ} acts on the closure \overline{𝕆}, it naturally acts on the regular function ring ℂ[\overline{𝕆}]. Let us describe the K_{ℂ}-module structure of the regular function ring of the closure of the lifted orbit 𝕆 = θ(𝕆′). To state it, we need some notations.

Let P_{m} be the set of all partitions of length ≤ m and consider α ∈ P_{m} as a dominant integral weight for GL_{m} as usual. Then τ_{α}^{(m)} denotes the irreducible finite dimensional representation of GL_{m} with highest weight α, and τ_{α}^{(m)}^{∗} its contragredient. For α ∈ P_{m} and β ∈ P_{n}, we put

α ⊙_{p} β = (α_{1}, α_{2}, …, α_{m}, 0, …, 0, −β_{n}, …, −β_{1}) = (α, 0, …, 0, β^{∗}) ∈ ℤ^{p}

where there appear (p−(m+n))-times of zeroes between α and β^{∗}. This is also a dominant integral weight for GL_{p}.

The following theorem is proved in [9] (also see [3, Th. 2.5.4] for the structure of harmonics), and it plays a key role in determining the spherical nilpotent orbits among the lifted ones.

Theorem 1.5. Let 𝕆′ ⊂ N (𝔰′) be a nilpotent K′_{ℂ}-orbit and 𝕆 = θ(𝕆′) ⊂ N (𝔰) its theta lift. Then the K_{ℂ}-module structure of the regular function ring of \overline{𝕆} is completely described via that of \overline{𝕆′}. Namely, as a K_{ℂ}-module, we have

ℂ[\overline{𝕆}] = ^{∑⊕} ; \begin{matrix}α,γ∈P_{m} \\ β,δ∈P_{n} \end{matrix} Hom_{K′_{ℂ}}((τ_{α}^{(m)}^{∗} ⊗ τ_{γ}^{(m)}) ⊠ (τ_{β}^{(n)} ⊗ τ_{δ}^{(n)}^{∗}), ℂ[\overline{𝕆′}]) ⊗ (τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)}),

where K_{ℂ} = GL_{p} × GL_{q} acts trivially on the space of multiplicity Hom_{K′_{ℂ}}(⋯).

2. Spherical nilpotent orbits

Let X be a variety on which K_{ℂ} acts regularly. Then X is called a spherical variety (or more specifically, K_{ℂ}-spherical variety), if there exists a Borel subgroup B_{K_{ℂ}} of K_{ℂ} which has an open dense orbit in X. If a nilpotent K_{ℂ}-orbit 𝕆 is a spherical variety, it is called a spherical nilpotent orbit. By definition, 𝕆 is a spherical orbit if and only if \overline{𝕆} is spherical. Since \overline{𝕆} is a closed affine cone, it is spherical if and only if the regular function ring ℂ[\overline{𝕆}] decomposes without multiplicity as a K_{ℂ}-module, i.e., the action of K_{ℂ} on \overline{𝕆} is multiplicity-free.

Let D be a signed Young diagram of signature (p, q). We denote by λ(D) = λ = (λ_{1}, λ_{2}, …, λ_{l}) the partition of p + q corresponding to D, i.e., λ_{i} is the number of boxes in the i-th row of D. We call λ the shape of D.

Theorem 2.1. Let 𝕆_{D} be a nilpotent K_{ℂ}-orbit for G = U(p, q) corresponding to the signed Young diagram D. Assume that ℓ(D) ≥ max{p, q}, where ℓ(D) is the length of D. Then 𝕆_{D} is spherical if and only if its shape λ(D) is given by

λ(D) = [3^{ε} · 2^{k} · 1^{l}] ε = 0, 1; k, l ≥ 0; 3ε + 2k + l = p + q.

Remark 2.2. (1) The shape λ(D) determines the Jordan type of nilpotent elements from 𝕆_{D}. This means that the sphericality does not depend on 𝕆_{D} but on its complex hull G_{ℂ} · 𝕆_{D}.

(2) Panyushev [12] completely classified the spherical nilpotent orbit for complex simple Lie algebra. According to his classification, only the orbits with Jordan types [2^{k} · 1^{l}] are G_{ℂ}-spherical. So, if ε = 1, the complex hull of the orbit above is not spherical under the action of G_{ℂ}.

The rest of this section is devoted to the proof of Theorem 2.1.

2.1. Spherical lifted orbits. Let us check that the nilpotent orbits listed in Theorem 2.1 are in fact spherical.

If the shape of a signed Young diagram is [2^{k} · 1^{l}], then it is a theta lift from the trivial orbit of U(m, n) (m + n = k), where m (resp. n) is the number of rows + − (resp. − +) in D. By Corollary 3.2 in [9], the theta lift from the trivial orbit is spherical.

Next, we consider a signed Young diagram D of shape [3 · 2^{k} · 1^{l}]. Let 𝕆_{D} be the corresponding nilpotent K_{ℂ}-orbit which is lifted from a nilpotent K′_{ℂ}-orbit 𝕆′_{D′}. Without loss of generality, we can assume that D and D′ are of the following form.

D = + − + ; + − ; ⋮ ⋮ ; − + ; ⋮ ⋮ ; + ; ⋮ ; − ; ⋮ D′ = + − ; + ; ⋮ ; − ; ⋮,

where, in D, the row + − appears (m − 1)-times, and the row − + appears (n − 1)-times (m, n ≥ 1). Then, 𝕆′_{D′} is a nilpotent K′_{ℂ}-orbit for G′ = U(m, n). Note that 𝕆′ = 𝕆′_{D′} is also a theta lift from the trivial orbit for U(1) = U(1, 0). Thus we have

ℂ[\overline{𝕆′}] = \begin{matrix}∑ & ⊕ \\ μ∈P_{1} \end{matrix} τ_{(μ,0,... ,0)}^{(m)}^{∗} ⊠ τ_{(μ,0,... ,0)}^{(n)},

where P_{1} = ℤ_{≥0} is the set of partitions of length 1 or 0. Substituting ℂ[\overline{𝕆′}] in Theorem 1.5 by the above formula, we get a decomposition

ℂ[\overline{𝕆_{D}}] = ^{∑⊕} ; \begin{matrix}α,γ∈P_{m} \\ β,δ∈P_{n} \end{matrix}( \begin{matrix}∑ & ⊕ \\ μ≥0 \end{matrix} Hom_{GL_{m}}(τ_{α}^{(m)}^{∗} ⊗ τ_{γ}^{(m)}, τ_{(μ,0,... ,0)}^{(m)}^{∗}) ⊗

Hom_{GL_{n}}(τ_{β}^{(n)} ⊗ τ_{δ}^{(n)}^{∗}, τ_{(μ,0,... ,0)}^{(n)})) ⊗ (τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)}).

If the multiplicity of τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)} does not vanish, we must have |α| − |γ| = μ and |β| − |δ| = μ, where |α| = α_{1} + ⋯ + α_{m} is the size of a partition α. Thus μ is uniquely determined by the pair (α, γ) or (β, δ). Let us assume that μ = |α| − |γ| and consider

Hom_{GL_{m}}(τ_{α}^{(m)}^{∗} ⊗ τ_{γ}^{(m)}, τ_{(μ,0,... ,0)}^{(m)}^{∗}) = Hom_{GL_{m}}(τ_{α}^{(m)}, τ_{γ}^{(m)} ⊗ τ_{(μ,0,... ,0)}^{(m)}).

By Pieri formula, the last expression does not vanish if and only if the skew diagram α − γ is a horizontal μ-strip (see [6, (5.16)] for Pieri formula and the terminologies used here), and in that case τ_{α}^{(m)} appears in τ_{γ}^{(m)} ⊗ τ_{(μ,0,... ,0)}^{(m)} with multiplicity one. The similar assertion holds for β and δ.

Thus we get a multiplicity-free decomposition

ℂ[\overline{𝕆_{D}}] = ^{∑⊕} ; \begin{matrix}α,γ∈P_{m} \\ β,δ∈P_{n} \end{matrix} τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)},

where the summation is taken over α, β, γ, δ satisfying

μ = |α| − |γ| = |β| − |δ| ; and, α − γ and β − δ are horizontal μ-strips.

This proves that 𝕆_{D} is spherical.

2.2. Non-spherical lifted orbits. We are to prove that the rest of lifted orbits are not spherical. We prepare two lemmas.

Lemma 2.3. Let 𝕆 be a theta lift of 𝕆′ in the stable range. if 𝕆 is spherical, then 𝕆′ is spherical too. In other words, theta lifting does not produce any spherical orbit from non-spherical ones.

Proof. We assume that 𝕆′ is not spherical so that ℂ[\overline{𝕆′}] contains some representation τ_{μ}^{(m)} ⊗ τ_{η}^{(n)} with multiplicity ≥ 2. Let us denote the multiplicity by M_{μ,η}. Then we have

Hom_{GL_{m}×GL_{n}}((τ_{α}^{(m)}^{∗} ⊗ τ_{γ}^{(m)}) ⊠ (τ_{β}^{(n)} ⊗ τ_{δ}^{(n)}^{∗}), ℂ[\overline{𝕆′}])

⊃ M_{μ,η} Hom_{GL_{m}}(τ_{α}^{(m)}^{∗} ⊗ τ_{γ}^{(m)}, τ_{μ}^{(m)}) ⊗ Hom_{GL_{n}}(τ_{β}^{(n)} ⊗ τ_{δ}^{(n)}^{∗}, τ_{η}^{(n)}).

Clearly we can choose α, β, γ, δ so that the above tensor product survives. Then, by Theorem 1.5, ℂ[\overline{𝕆}] is not multiplicity-free, hence 𝕆 is not spherical.

Lemma 2.4. Let 𝕆_{1} and 𝕆_{2} be two nilpotent K_{ℂ}-orbits, and suppose that 𝕆_{1} is adherent to the closure \overline{𝕆_{2}}. If 𝕆_{2} is spherical, then 𝕆_{1} is also spherical. In other words, if 𝕆_{1} is not spherical, 𝕆_{2} cannot be spherical.

Proof. Since \overline{𝕆_{1}} ⊂ \overline{𝕆_{2}} is a closed subvariety, the restriction is a K_{ℂ}-equivariant surjection ℂ[\overline{𝕆_{2}}] → ℂ[\overline{𝕆_{1}}]. Thus, if ℂ[\overline{𝕆_{2}}] is multiplicity-free, ℂ[\overline{𝕆_{1}}] is multiplicity-free, which is what we wanted to show.

This lemma tells that the set of spherical nilpotent orbits has a hereditary property with respect to the closure relation.

The closure relation of nilpotent K_{ℂ}-orbits is well known and it is described most conveniently in the language of signed Young diagrams. For this, we refer to [10].

Among the signed Young diagrams which are not listed in Theorem 2.1, the minimal orbits with respect to the closure relation are the following form:

(a) + − + − ; + ; ⋮ ; − ; ⋮  (a′) − + − + ; + ; ⋮ ; − ; ⋮  (b) + − + ; + − + ; + ; ⋮ ; − ; ⋮  (c) + − + ; − + − ; + ; ⋮ ; − ; ⋮  (b′) − + − ; − + − ; + ; ⋮ ; − ; ⋮

Thus it is enough to check the orbits corresponding to (a), (a′), (b), (b′), (c) are not spherical. Since (a), (a′) and (b), (b′) are treated in the similar manner, we only consider cases (a), (b) and (c).

Case (a). Let D be the Young diagram (a). Then 𝕆_{D} is lifted from 𝕆′_{D′} where D′ = + − +. 𝕆′_{D′} is the largest nilpotent K′_{ℂ}-orbit of G′ = U(2, 1), and its closure coincides with the whole nilpotent variety N (𝔰′). Therefore, by Kostant-Rallis theorem, we get

ℂ[\overline{𝕆′_{D′}}] = ℂ[N (𝔰′)] ≃ Ind _{M′_{ℂ}}^{K′_{ℂ}}ℂ (as K′_{ℂ}-module),

where K′_{ℂ} = GL_{2} × GL_{1} and ℂ denotes the trivial representation of

M′_{ℂ} = {diag (t, s, s) | t, s ∈ ℂ^{×}}.

Frobenius reciprocity law tells us that the above formula becomes

ℂ[\overline{𝕆′_{D′}}] ≃ \begin{matrix}∑ & ⊕ \\ μ,ν \end{matrix} Hom_{M′_{ℂ}}(τ_{μ}^{(2)} ⊗ τ_{ν}^{(1)}, ℂ) ⊗ (τ_{μ}^{(2)} ⊠ τ_{ν}^{(1)}) ≃ \begin{matrix}∑ & ⊕ \\ μ_{1}≥0≥μ_{2} \end{matrix} τ_{μ}^{(2)} ⊠ τ_{μ_{1}+μ_{2}}^{(1)}^{∗},

where μ = (μ_{1}, μ_{2}) ∈ ℤ^{2} is a dominant integral weight for GL_{2}. This shows that 𝕆′_{D′} is spherical, and it gives an example of a spherical nilpotent orbit which is not a theta lift in the stable range.

Now by Theorem 1.5, we obtain the decomposition

ℂ[\overline{𝕆_{D}}] = ^{∑⊕} ; \begin{matrix}α,γ∈P_{2} \\ β,δ∈P_{1} \end{matrix}(\begin{matrix}∑ & ⊕ \\ μ_{1}≥0≥μ_{2} \end{matrix} Hom_{GL_{2}×GL_{1}}((τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)})⊠(τ_{β}^{(1)} ⊗ τ_{δ}^{(1)}^{∗}), τ_{μ}^{(2)} ⊠ τ_{μ_{1}+μ_{2}}^{(1)}^{∗}))

⊗ (τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)}).

The multiplicity in the above formula can be rewritten as

\begin{matrix}∑ & ⊕ \\ μ_{1}≥0≥μ_{2} \end{matrix} Hom_{GL_{2}}(τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)}, τ_{μ}^{(2)}) ⊗ Hom_{GL_{1}}(τ_{δ}^{(1)}, τ_{β}^{(1)} ⊗ τ_{μ_{1}+μ_{2}}^{(1)})  (2.1)

Take integers l ≥ k ≥ 1 and put α = (k, 0), γ = (l, 0). Then, by Pieri formula, we get

τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)} ≃ \begin{matrix}∑ & ⊕ \\ k_{1}+k_{2}=k \end{matrix} τ_{(l−k_{1},−k_{2})}^{(2)},

where the summation is taken over non-negative integers k_{1}, k_{2} ≥ 0 such that k = k_{1} + k_{2}. Therefore μ = (l − k_{1}, −k_{2}) gives a non-zero term in (2.1) if and only if δ = β + μ_{1} + μ_{2} = β + l − k. Let us take β = k and δ = l. Thus the multiplicity of τ_{α⊙β}^{(p)}^{∗} ⊠ τ_{γ⊙δ}^{(q)} = τ_{(k,0,... ,0,−k)}^{(p)} ⊠ τ_{(l,0,...0,−l)}^{(q)} is equal to k + 1 ≥ 2. This shows that ℂ[\overline{𝕆_{D}}] is not multiplicityfree.

Case (b). As above, we put D′ = \begin{matrix}+ & − \\ + & − \end{matrix}, which corresponds to the anti-holomorphic orbit

for G′ = U(2, 2). Since 𝕆′_{D′} is a theta lift from the trivial orbit of U(2), we have

ℂ[\overline{𝕆′_{D′}}] ≃ \begin{matrix}∑ & ⊕ \\ μ∈P_{2} \end{matrix} τ_{μ}^{(2)} ⊠ τ_{μ}^{(2)}^{∗}.

By Theorem 1.5, we obtain

ℂ[\overline{𝕆_{D}}] = \begin{matrix}∑ & ⊕ \\ α,β,γ,δ∈P_{2} \end{matrix}( \begin{matrix}∑ & ⊕ \\ μ∈P_{2} \end{matrix} Hom_{GL_{2}×GL_{2}}((τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)})⊠(τ_{β}^{(2)} ⊗ τ_{δ}^{(2)}^{∗}), τ_{μ}^{(2)} ⊠ τ_{μ}^{(2)}^{∗}))

⊗ (τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)}).

Therefore the multiplicity becomes

\begin{matrix}∑ & ⊕ \\ μ∈P_{2} \end{matrix} Hom_{GL_{2}}(τ_{μ}^{(2)}, τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)}) ⊗ Hom_{GL_{2}}(τ_{μ}^{(2)}, τ_{β}^{(2)}^{∗} ⊗ τ_{δ}^{(2)}).  (2.2)

We take integers l ≥ k ≥ 1, and put α = β = (k, 0) and γ = δ = (k + l, k). Pieri formula tells us that

τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)} ≃ τ_{β}^{(2)}^{∗} ⊗ τ_{δ}^{(2)} ≃ \begin{matrix}∑ & ⊕ \\ k_{1}+k_{2}=k \end{matrix} τ_{(l+k_{1},k_{2})}^{(2)}.

Thus, the above multiplicity (2.2) is equal to k + 1 ≥ 2. Hence ℂ[\overline{𝕆_{D}}] is not multiplicityfree.

Case (c). We put D′ = \begin{matrix}+ & − \\ − & + \end{matrix}. Then 𝕆′_{D′} is a theta lift from the trivial orbit of U(1, 1).

So we obtain

ℂ[\overline{𝕆′_{D′}}] ≃ \begin{matrix}∑ & ⊕ \\ μ_{1}≥0≥μ_{2} \end{matrix} τ_{μ}^{(2)}^{∗} ⊠ τ_{μ}^{(2)},

and

ℂ[\overline{𝕆_{D}}] = \begin{matrix}∑ & ⊕ \\ α,β,γ,δ∈P_{2} \end{matrix}(\begin{matrix}∑ & ⊕ \\ μ_{1}≥0≥μ_{2} \end{matrix} Hom_{GL_{2}×GL_{2}}((τ_{α}^{(2)}^{∗} ⊗ τ_{γ}^{(2)})⊠(τ_{β}^{(2)} ⊗ τ_{δ}^{(2)}^{∗}), τ_{μ}^{(2)}^{∗} ⊠ τ_{μ}^{(2)}))

⊗ (τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)}).

The multiplicity becomes

\begin{matrix}∑ & ⊕ \\ μ_{1}≥0≥μ_{2} \end{matrix} Hom_{GL_{2}}(τ_{μ}^{(2)}, τ_{α}^{(2)} ⊗ τ_{γ}^{(2)}^{∗}) ⊗ Hom_{GL_{2}}(τ_{μ}^{(2)}, τ_{β}^{(2)} ⊗ τ_{δ}^{(2)}^{∗}).  (2.3)

For integers l ≥ k ≥ 1, we put α = β = (l, 0) and γ = δ = (k, 0). Then, by Pieri formula, we have

τ_{α}^{(2)} ⊗ τ_{γ}^{(2)}^{∗} ≃ τ_{β}^{(2)} ⊗ τ_{δ}^{(2)}^{∗} ≃ \begin{matrix}∑ & ⊕ \\ k_{1}+k_{2}=k \end{matrix} τ_{(l−k_{1},−k_{2})}^{(2)}.

Thus, the multiplicity (2.3) is equal to k + 1 ≥ 2 for τ_{α⊙_{p}β}^{(p)}^{∗} ⊠ τ_{γ⊙_{q}δ}^{(q)} = τ_{(l,0,... ,0,−l)}^{(p)} ⊠

τ_{(k,0,... ,0,−k)}^{(q)}. Hence 𝕆_{D} is not spherical.

Now Lemma 2.4 tells us that the nilpotent orbit 𝕆 whose closure contains the diagrams (a) – (c) cannot be spherical. This completes the proof of Theorem 2.1.

3. Classification of spherical nilpotent orbits

In §2, we have identified the spherical nilpotent orbits which are theta lifts in the stable range. In this section, we obtain a sufficient condition for non-sphericality. As a consequence, in the case of U(p, p) where p = q, we prove there is no spherical nilpotent orbit other than lifted ones. This gives a complete classification of spherical nilpotent orbits for G = U(p, p).

Theorem 3.1. Let 𝕆 = 𝕆_{D} be a nilpotent K_{ℂ}-orbit for U(p, p). It is spherical if and only if the shape λ(D) of the signed Young diagram D is of form:

λ(D) = [3^{ε} · 2^{k} · 1^{l}] ε = 0, 1; k, l ≥ 0; 3ε + 2k + l = 2p,

and the length satisfies an inequality ℓ(D) = ε + k + l ≥ p.

As explained above, Theorem 3.1 follows from Theorem 2.1 and the following

Lemma 3.2. Let 𝕆 = 𝕆_{D} be a nilpotent K_{ℂ}-orbit for U(p, q). If ℓ(D) < min{p, q}, it is not spherical. In particular, if p = q, a spherical orbit 𝕆 is a theta lift from certain nilpotent K′_{ℂ}-orbit 𝕆′ in the stable range.

The rest of the section is devoted to the proof of the above lemma.

Let B = B_{K_{ℂ}} be a Borel subgroup of K_{ℂ}. The following lemma is almost trivial.

Lemma 3.3. Let 𝕆 be a nilpotent K_{ℂ}-orbit. If dim 𝕆 ≥ dim B_{K_{ℂ}}, it is not spherical.

Proof. Since 𝕆 is a cone and B contains a dilation, the dimension of a B-orbit in 𝕆 cannot exceed dim B − 1.

Take a signed Young diagram D of shape λ = λ(D) with signature (p, q). We put tλ = μ = (μ_{1}, …, μ_{k}), the transposed partition of λ.

Lemma 3.4. With the above notation, dim 𝕆_{D} ≥ dim B_{K_{ℂ}} if and only if

2pq − (p + q) ≥ ∑_{i=1}^{k} μ_{i}^{2}.  (3.1)

Proof. It is well known that the dimension of the G_{ℂ}-hull of 𝕆_{D} is given by

dim G_{ℂ} · 𝕆_{D} = dim 𝔤 − ∑_{i=1}^{k} μ_{i}^{2} = 2 dim 𝕆_{D}.

For this, see [1, Cor. 6.1.4, Remark 9.5.2] for example. Since K_{ℂ} = GL_{p} × GL_{q}, we have dim B_{K_{ℂ}} = p(p + 1)/2 + q(q + 1)/2. Thus we get

dim 𝕆_{D} − dim B_{K_{ℂ}} = \frac{1}{2}\{(p + q)^{2} − ∑_{i=1}^{k} μ_{i}^{2}\} − \{\frac{p(p + 1)}{2} + \frac{q(q + 1)}{2}\}

= \frac{1}{2}\{2pq − (p + q) − ∑_{i=1}^{k} μ_{i}^{2}\},

which proves the lemma.

Let us return to the proof of Lemma 3.2. Without loss of generality, we can assume that μ_{1} = ℓ(D) < p ≤ q. Since μ = (μ_{1}, …, μ_{k}) is a partition of p+q, we have ∑_{i=1}^{k} μ_{i} = p+q. Therefore we have

2pq − (p + q) − ∑_{i=1}^{k} μ_{i}^{2} ≥ (p − 1)(p + q) − ∑_{i=1}^{k} μ_{i}^{2} = (p − 1) ∑_{i=1}^{k} μ_{i} − ∑_{i=1}^{k} μ_{i}^{2}

= ∑_{i=1}^{k} μ_{i}(p − 1 − μ_{i}) ≥ ∑_{i=1}^{k} μ_{i}(p − μ_{1} − 1) ≥ 0.

By Lemma 3.4, the above inequality assures dim 𝕆_{D} ≥ dim B_{K_{ℂ}}, hence 𝕆_{D} cannot be spherical. This completes the proof of Lemma 3.2.

References

[1] David H. Collingwood and William M. McGovern, Nilpotent Orbits in semisimple Lie algebras. Van Nostrand Reinhold, 1993.

[2] R. Howe, Transcending classical invariant theory. J. Amer. Math. Soc. 2 (1989), no. 3, 535–552.

[3] R. Howe, Perspectives on invariant theory: Schur duality, multiplicity-free actions and beyond. in The Schur lectures (1992) (Tel Aviv), 1–182, Bar-Ilan Univ., Ramat Gan, 1995.

[4] J.-S. Huang and J.-S. Li, Unipotent representations attached to spherical nilpotent orbits, Amer. Jour. Math., 121 (1999), no. 3, 497–517.

[5] B. Kostant and S. Rallis, Orbits and representations associated with symmetric spaces. Amer. J. Math. 93 (1971), 753–809.

[6] I. G. Macdonald, Symmetric functions and Hall polynomials. Second edition. Oxford Mathematical Monographs. Oxford Science Publications. The Clarendon Press, Oxford University Press, New York, 1995.

[7] Kyo Nishiyama, Hiroyuki Ochiai, and Kenji Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules—Hermitian symmetric case. Astérisque No. 273, (2001), 13–80.

[8] Kyo Nishiyama and Chen-bo Zhu, Theta lifting of holomorphic discrete series, (The case of U(p, q) × U(n, n)). Trans. AMS. 353 (2001), 3327-3345.

[9] Kyo Nishiyama and Chen-bo Zhu, Theta lifting of nilpotent orbits for symmetric pairs, preprint, 2002.

[10] Takuya Ohta, The closures of nilpotent orbits in the classical symmetric pairs and their singularities. Tohoku Math. J. (2) 43 (1991), 161–211.

[11] Takuya Ohta, Nilpotent orbits of Z_{4}-graded Lie algebra and geometry of the moment maps associated to the dual pair (U(p, q), U(r, s)), preprint.

[12] Dmitri I. Panyushev, On spherical nilpotent orbits and beyond. Ann. Inst. Fourier, Grenoble, 49 (1999), 1453 – 1476.

Faculty of IHS, Kyoto University, Sakyo, Kyoto 606-8501, Japan

 E-mail address: kyo@math.h.kyoto-u.ac.jp