Theta lifting of two-step nilpotent orbits

for the pair O(p, q) × Sp(2n, ℝ) ^{∗}

Kyo Nishiyama

Faculty of Integrated Human Studies Kyoto University Sakyo, Kyoto 606-8501, Japan

                        E-mail: kyo@math.h.kyoto-u.ac.jp

Introduction

Let G be a linear reductive Lie group which is a subgroup in its complexification G_{ℂ}. We denote the Lie algebra of G by 𝔤_{0}, and its complexification by 𝔤 = ℂ ⊗_{ℝ} 𝔤_{0}. We will use the similar notation for any Lie group L; thus, L_{ℂ} denotes its complexification, 𝔩_{0} its Lie algebra, and 𝔩 the complexification of 𝔩_{0}.

Take a maximal compact subgroup K of G. Then K determines a Cartan decomposition 𝔤_{0} = 𝔨_{0} ⊕ 𝔰_{0} and its complexification 𝔤 = 𝔨 ⊕ 𝔰. The adjoint action of K_{ℂ} preserves 𝔰, and the set of all nilpotent elements N_{𝔰} in 𝔰. It is well known that N_{𝔰} is a normal variety and that it has finitely many K_{ℂ}-orbits ([2]).

Now consider a dual pair (G, G′) = (O(p, q), Sp(2n, ℝ)) (see [1] for the properties of dual pairs). In this note, we define certain double fibration maps of nilpotent varieties for O(p, q) and Sp(2n, ℝ). We use the double fibration maps to get a correspondence between nilpotent K_{ℂ}-orbits in 𝔰 and nilpotent K′_{ℂ}-orbits in 𝔰′, which is called a “theta lift”. We describe the theta lifts of two-step nilpotent orbits in N_{𝔰′}, where 𝔤′ = 𝔨′ ⊕ 𝔰′ is a Cartan decomposition for G′ = Sp(2n, ℝ) (Proposition 1.3).

If a nilpotent K_{ℂ}-orbit O ⊂ 𝔰 is the theta lift of a nilpotent K′_{ℂ}-orbit O′ ⊂ 𝔰′, it is interesting to describe the regular function ring ℂ[\overline{O}] by means of ℂ[\overline{O′}]. Our main results are descriptions of the K_{ℂ}-module structure of ℂ[\overline{O}] in terms of the double fibration maps (Theorem 2.4 and Proposition 3.4). In the course of the proof, we realize the closure \overline{O} of the orbit as a geometric quotient of the fiber of \overline{O′} (Proposition 3.3). As an application of these results, we get a formula of branching coefficients between different kind of classical groups (Corollary 3.5).

The K_{ℂ}-module structures of nilpotent orbits may reflect the K-type decompositions of the corresponding admissible representation of G via orbit method (or geometric quantization). Thus we can expect to extract information on the admissible representations from the geometry of nilpotent orbits. This will be treated elsewhere.

∗Manuscript for Proceedings of the Symposium on “Infinite Dimensional Harmonic Analysis”.

Acknowledgments. The author thanks Cheng-Bo Zhu for the discussions on the double fibration maps. Some of the ideas used here are due to him. He also thanks Takuya Ohta and Hiroshi Yamashita for stimulating conversations.

1 Double fibration of nilpotent varieties

Let G = O(p, q) be an orthogonal group of signature (p, q). Then a maximal compact subgroup K is isomorphic to O(p) × O(q). We realize them as follows.

G = O(p, q) = \{g ∈ GL(p + q, ℝ) | ^{t}g1_{p,q}g = 1_{p,q}\} , 1_{p,q} = (\begin{matrix}1_{p} & 0 \\ 0 & −1_{q} \end{matrix}) ,
K = O(p) × O(q) = (\begin{matrix}O(p) & 0 \\ 0 & O(q) \end{matrix}) .

Then the corresponding (complexified) Cartan decomposition is given by

𝔤 = \{( \begin{matrix}α & β \\ ^{t}β & γ \end{matrix}) | \begin{matrix}α ∈ Alt_{p}(ℂ) \\ γ ∈ Alt_{q}(ℂ) \end{matrix} , β ∈ M_{p,q}(ℂ)\}

= (\begin{matrix}Alt_{p}(ℂ) & 0 \\ 0 & Alt_{q}(ℂ) \end{matrix}) ⊕ ( \begin{matrix}0 & M_{p,q}(ℂ) \\ ^{t}M_{p,q}(ℂ) & 0 \end{matrix}) = 𝔨 ⊕ 𝔰.

Hence we identify 𝔰 with M_{p,q}(ℂ) via

M_{p,q}(ℂ) ∋ β ↔ ( \begin{matrix}0 & β \\ ^{t}β & 0 \end{matrix}) ∈ 𝔰.

Denote the set of nilpotent elements in 𝔰 by N_{𝔰}. Then, by the above identification, β ∈ M_{p,q}(ℂ) belongs to N_{𝔰} if and only if ^{t}ββ is a nilpotent matrix, if and only if β^{t}β is so.

Next we consider the symplectic group G′ = Sp(2n, ℝ) of rank n. A maximal compact subgroup K′ is isomorphic to the unitary group U(n) of size n. To realize K′ in a simple way, we define Sp(2n, ℝ) in a slightly different manner from the usual one. Namely, we put

G′ = U(n, n) ∩ Sp(2n, ℂ)
= \{g ∈ GL(2n, ℂ) | ^{t}\overline{g}1_{n,n}g = 1_{n,n}, ^{t}gJg = J\} ,

where

1_{n,n} = (\begin{matrix}1_{n} & 0 \\ 0 & −1_{n} \end{matrix}) , J = (\begin{matrix}0 & 1_{n} \\ −1_{n} & 0 \end{matrix}) .

Then G′ is isomorphic to Sp(2n, ℝ), and

K′ = \{(\begin{matrix}k & 0 \\ 0 & ^{t}k^{−1} \end{matrix}) | k ∈ U(n)\} ⊂ G′

is a maximal compact subgroup. The corresponding Cartan decomposition is given by

𝔤′ = \{(\begin{matrix}H & 0 \\ 0 & −^{t}H \end{matrix}) | H ∈ 𝔲(n)\} ⊕ \{(\begin{matrix}0 & C \\ D & 0 \end{matrix}) | C, D ∈ Sym_{n}(ℂ)\} = 𝔨′ ⊕ 𝔰′.

We identify 𝔰′ with Sym_{n}(ℂ) ⊕ Sym_{n}(ℂ) via

Sym_{n}(ℂ) ⊕ Sym_{n}(ℂ) ∋ (C, D) ↔ (\begin{matrix}0 & C \\ D & 0 \end{matrix}) ∈ 𝔰′.

Then (C, D) belongs to the nilpotent variety N_{𝔰′} if and only if C · D is nilpotent, if and only if D · C is so.

Now we shall define the double fibration maps. Let W = M_{p+q,n}(ℂ) be the space of all the (p + q) × n-matrices. We express a matrix Z in W as

Z = (\begin{matrix}A \\ B \end{matrix}) ∈ W; A ∈ M_{p,n}(ℂ), B ∈ M_{q,n}(ℂ).

We define two maps φ and ψ by

φ :W ∋ Z ⟼ A^{t}B ∈ M_{p,q}(ℂ) = 𝔰,
ψ :W ∋ Z ⟼ (^{t}AA, ^{t}BB) ∈ Sym_{n}(ℂ) ⊕ Sym_{n}(ℂ) = 𝔰′.

Put

M_{ℂ} = GL_{p}(ℂ) × GL_{q}(ℂ) ⊃ O(p, ℂ) × O(q, ℂ) = K_{ℂ},
M′_{ℂ} = GL_{n}(ℂ) × GL_{n}(ℂ) ⊃ ΔGL_{n}(ℂ) = K′_{ℂ},

and define M_{ℂ} × M′_{ℂ}-action on W by

(m, m′) · Z = (\begin{matrix}m_{1}A^{t}m′_{1} \\ m_{2}Bm′_{2}^{−1} \end{matrix}) for Z = (\begin{matrix}A \\ B \end{matrix}) ∈ W,

where

m = (m_{1}, m_{2}) ∈ M_{ℂ} = GL_{p}(ℂ) × GL_{q}(ℂ),
m′ = (m′_{1}, m′_{2}) ∈ M′_{ℂ} = GL_{n}(ℂ) × GL_{n}(ℂ).

We introduce M_{ℂ}-action on 𝔰 (resp. M′_{ℂ}-action on 𝔰′) so that φ : W → 𝔰 is an M_{ℂ} × K′_{ℂ}- equivariant map (resp. ψ : W → 𝔰′ is a K_{ℂ}×M′_{ℂ}-equivariant map). Note that the induced action is compatible with the adjoint K_{ℂ}-action on 𝔰 (resp. K′_{ℂ}-action on 𝔰′). As a GL_{n}(ℂ)- module, the second component Sym_{n}(ℂ) of 𝔰′ is regarded as the contragredient of the first component. By this reason, sometimes we will write 𝔰′ = Sym_{n}(ℂ) ⊕ Sym_{n}(ℂ)^{∗}.

Our first observation is the following.

Lemma 1.1 φ ∘ ψ^{−1} and ψ ∘ φ^{−1} preserves nilpotent elements:

φ(ψ^{−1}(N_{𝔰′})) ⊂ N_{𝔰}, ψ(φ^{−1}(N_{𝔰})) ⊂ N_{𝔰′}.

Proof. This is an easy consequence of direct calculations. Q.E.D.

Definition 1.2 Let O (resp. O′) be a nilpotent K_{ℂ}-orbit in 𝔰 (resp. K′_{ℂ}-orbit in 𝔰′). If \overline{O} = φ(ψ^{−1}(\overline{O′})) holds, we say that O is the theta lift of O′.

Note that φ(ψ^{−1}(\overline{O′})) is an affine closed cone.

Proposition 1.3 Assume that 2n < min(p, q). Let O′_{[r,s]} = O′_{λ_{r,s}} ⊂ N_{𝔰′} be a nilpotent K′_{ℂ}-orbit through

λ_{r,s} = ( \begin{matrix}0 & \begin{aligned}1_{r} \\ & 0 \end{aligned} \\ \begin{aligned}0 \\ & 1_{s} \end{aligned} & 0 \end{matrix} ) (r + s ≤ n).

Then there exists a nilpotent K_{ℂ}-orbit O ⊂ N_{𝔰} for which φ(ψ^{−1}(\overline{O′_{[r,s]}})) = \overline{O} holds, i.e., the theta lift of O′_{[r,s]} exists. We denote O = O_{[n;r,s]}.

Remark 1.4 We allow r = s = 0, which means that O′_{[0,0]} = {0}. Note that O′_{[r,s]} exhausts all the two-step nilpotent orbits in 𝔰′.

Proof. We will specify the nilpotent K_{ℂ}-orbit O = O_{[n;r,s]} in the end of the proof.

To prove the proposition, it suffices to prove that ψ^{−1}(\overline{O′_{[r,s]}}) is irreducible. In fact, if it is irreducible, then φ(ψ^{−1}(\overline{O′_{[r,s]}})) is an irreducible closed set, and is K_{ℂ}-stable in N_{𝔰}. Since N_{𝔰} contains only a finite number of K_{ℂ}-orbits, it must be the closure of a single orbit.

Let us see that ψ^{−1}(\overline{O′_{[r,s]}}) is irreducible. We call

𝔑_{p,k} = {A ∈ M_{p,k}(ℂ) | ^{t}AA = 0}

a null cone of size (p, k). It is known to be irreducible if 2k < p. Thus, if we put

N_{r,s} = \{Z = (\begin{matrix}A \\ B \end{matrix}) ∈ W | A = ( \begin{matrix}1_{r} & 0 \\ 0 & E \end{matrix} ) , B = ( \begin{matrix}0 & 1_{s} \\ F & 0 \end{matrix} ) ,
where E ∈ 𝔑_{p−r,n−r} and F ∈ 𝔑_{q−s,n−s}}
≃ 𝔑_{p−r,n−r} × 𝔑_{q−s,n−s},

then N_{r,s} is irreducible and is contained in the fiber of λ_{r,s}. Moreover, under the condition that 2n < min(p, q), it is easy to check that the exact fiber of λ_{r,s} is given by

K_{ℂ}^{∘} · N_{r,s} = ψ^{−1}(λ_{r,s}),

where K_{ℂ}^{∘} ≃ SO(p, ℂ) × SO(q, ℂ) is the identity component of K_{ℂ}. Now we see that

(K_{ℂ}^{∘} × K′_{ℂ}) · N_{r,s} = ψ^{−1}(O′_{[r,s]}),

is irreducible, and hence ψ^{−1}(\overline{O′_{[r,s]}}) is irreducible.

We can take the following matrix as a representative of a generic K_{ℂ}^{∘} × K′_{ℂ}-orbit in N_{r,s}.

Z = (\begin{matrix}A \\ B \end{matrix}) ∈ W;

A = ( \begin{matrix}1_{r} & 0 \\ 0 & 1_{n−r} \\ 0 & 0 \\ 0 & i1_{n−r} \end{matrix} ) ∈ M_{p,n}(ℂ), B = ( \begin{matrix}1_{n−s} & 0 \\ 0 & 1_{s} \\ 0 & 0 \\ i1_{n−s} & 0 \end{matrix} ) ∈ M_{q,n}(ℂ).  (1.1)

By the above arguments, we know that the theta lift of O′_{[r,s]} should be exactly the K_{ℂ}- orbit through φ(Z), where Z is given in (1.1). Q.E.D.

By the above proof, we conclude that the theta lift O_{[n;r,s]} of O′_{[r,s]} consists of at most three-step nilpotents. It is two-step nilpotent if and only if r = s = 0. Thus, we see that the theta lift of a k-step nilpotent orbit is a (k + 1)-step nilpotent orbit.

2 Regular function ring of nilpotent orbits

In this section, we always assume that 2n < min(p, q).

Let O′_{[r,s]} = O′_{λ_{r,s}} (r + s ≤ n) be a nilpotent K′_{ℂ}-orbit in N_{𝔰′} given in Proposition 1.3. We denote the corresponding theta lift by O_{[n;r,s]}, which is a nilpotent K_{ℂ}-orbit in N_{𝔰}.

We consider the case s = 0 in the following. Then we have

O′_{[r,0]} = {(C, 0) ∈ 𝔰′ | C ∈ Sym_{n}(ℂ), rank C = r},

and it is known that \overline{O′_{[r,0]}} is the associated variety of an irreducible unitary highest weight representation of Sp(2n, ℝ) (or its metaplectic double cover). In particular, \overline{O′_{[n,0]}} ≃ Sym_{n}(ℂ) is the associated variety of a holomorphic discrete series representation of Sp(2n, ℝ).

Since O′_{[r,0]} is a K′_{ℂ}-orbit, the regular function ring ℂ[\overline{O′_{[r,0]}}] carries a natural K′_{ℂ}- module structure. Note that K′_{ℂ} = GL_{n}(ℂ). We denote by P_{k} all the partitions of length ≤ k, i.e., P_{k} = {λ = (λ_{1}, …, λ_{k}) ∈ ℤ^{k} | λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{k} ≥ 0}.

Theorem 2.1 The regular function ring ℂ[\overline{O′_{[r,0]}}] is decomposed as

ℂ[\overline{O′_{[r,0]}}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{r} \end{matrix} τ_{2λ}^{∗} (as a GL_{n}(ℂ)-module),

where τ_{μ} denotes an irreducible finite dimensional representation of GL_{n}(ℂ) with highest weight μ, and τ_{μ}^{∗} is its contragredient.

Proof. See [5], for example. Q.E.D.

Note that the fibration map

ψ : W = M_{p,n}(ℂ) × M_{q,n}(ℂ) ⟶ Sym_{n}(ℂ) × Sym_{n}(ℂ)^{∗} = 𝔰′

is a product of two maps of the same kind,

ψ_{p} : M_{p,n} ∋ A ⟼ ^{t}AA ∈ Sym_{n}(ℂ) and
ψ_{q} : M_{q,n} ∋ B ⟼ ^{t}BB ∈ Sym_{n}(ℂ)^{∗}.

Since Sp(2n, ℝ)/U(n) is a Hermitian symmetric space, 𝔰′ decomposes into two pieces of K′_{ℂ}-stable subspaces 𝔰′ = 𝔰′_{+} ⊕ 𝔰′_{−}, which we can identify with the decomposition 𝔰′ = Sym_{n}(ℂ) ⊕ Sym_{n}(ℂ)^{∗}. Our orbit O′_{[r,0]} lives in 𝔰′_{+} alone. Therefore, if we put Ξ_{[r,0]} = ψ^{−1}(\overline{O′_{[r,0]}}), it is decomposed as a product of closed affine cones

Ξ_{[r,0]} = ψ_{p}^{−1}(\overline{O′_{[r,0]}}) × ψ_{q}^{−1}(\{0\}) = Ξ_{r}^{(p)} × 𝔑_{q,n},

where 𝔑_{q,n} denotes the null cone given in the proof of Proposition 1.3, and

Ξ_{r}^{(p)} = ψ_{p}^{−1}(\overline{O′_{[r,0]}}) = \{A ∈ M_{p,n}(ℂ) | ^{t}AA ∈ \overline{O′_{[r,0]}}\}
= {A ∈ M_{p,n}(ℂ) | rank ^{t}AA ≤ r}.

Recall that

K′_{ℂ} = ΔGL_{n}(ℂ) ⊂ GL_{n}(ℂ) × GL_{n}(ℂ) = M′_{ℂ}.

The following lemma is now clear.

Lemma 2.2 The fiber Ξ_{[r,0]} = ψ^{−1}(\overline{O′_{[r,0]}}) is a product Ξ_{r}^{(p)}×𝔑_{q,n}, and hence it is K_{ℂ}×M′_{ℂ}- stable. The regular function ring breaks up into

ℂ[Ξ_{[r,0]}] ≃ ℂ[Ξ_{r}^{(p)}] ⊠ ℂ[𝔑_{q,n}]

as an (O(p, ℂ) × GL_{n}(ℂ)) × (O(q, ℂ) × GL_{n}(ℂ))-module.

The regular function ring ℂ[𝔑_{q,n}] consists of precisely the O(q, ℂ)-harmonic polynomials in ℂ[M_{q,n}] (see [4], for example). As a consequence, it decomposes in a multiplicityfree manner,

ℂ[𝔑_{q,n}] ≃ \begin{matrix}∑ & ⊕ \\ μ∈P_{n} \end{matrix} σ_{μ}^{(q)} ⊠ τ_{μ} (as an O(q, ℂ) × GL_{n}(ℂ)-module),  (2.1)

where σ_{μ}^{(q)} denotes an irreducible finite dimensional representation of O(q, ℂ) with highest weight μ. Let us decompose ℂ[Ξ_{r}^{(p)}] as an O(p, ℂ) × GL_{n}(ℂ)-module,

ℂ[Ξ_{r}^{(p)}] ≃ \begin{matrix}∑ & ⊕ \\ λ,η \end{matrix} m(λ, η) σ_{η}^{(p)} ⊠ τ_{λ}^{∗} (as an O(p, ℂ) × GL_{n}(ℂ)-module),  (2.2)

where m(λ, η) denotes the multiplicity.

For λ ∈ P_{n}, decompose an irreducible representation τ_{λ}^{(p)} of GL_{p}(ℂ) restricted to O(p, ℂ),

τ_{λ}^{(p)}|_{O(p,ℂ)} ≃ \begin{matrix}∑ & ⊕ \\ η∈P_{n} \end{matrix} b_{η}^{λ} σ_{η}^{(p)},  (2.3)

where b_{η}^{λ} denotes the branching coefficient. Note that η is also a partition of length ≤ n.

Lemma 2.3 The summation in (2.2) is taken over λ, η ∈ P_{n}; and the multiplicity m(λ, η) satisfies the following inequality,

δ_{λ,η} ≤ m(λ, η) ≤ b_{η}^{λ},  (2.4)

where δ_{λ,η} denotes Kronecker’s delta. Moreover, we have a decomposition

ℂ[Ξ_{[r,0]}] ≃ \begin{matrix}∑ & ⊕ \\ λ,μ,η∈P_{n} \end{matrix} m(λ, η) (σ_{η}^{(p)} ⊠ σ_{μ}^{(q)}) ⊠ (τ_{λ}^{∗} ⊠ τ_{μ})  (2.5)

as an (O(p, ℂ) × O(q, ℂ)) × (GL_{n}(ℂ) × GL_{n}(ℂ))-module, where m(λ, η) denotes the multiplicity given above.

Proof. Since Ξ_{r}^{(p)} is a closed subvariety of M_{p,n}, ℂ[Ξ_{r}^{(p)}] is a quotient of ℂ[M_{p,n}]. On the other hand, it is well known that ℂ[M_{p,n}] decomposes as

ℂ[M_{p,n}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} τ_{λ}^{(p)}^{∗} ⊠ τ_{λ}^{(n)}^{∗} (as a GL_{p}(ℂ) × GL_{n}(ℂ)-module).

Therefore, we have

ℂ[M_{p,n}] ≃ \begin{matrix}∑ & ⊕ \\ λ,η∈P_{n} \end{matrix} b_{η}^{λ} σ_{η}^{(p)} ⊠ τ_{λ}^{(n)}^{∗} (as an O(p, ℂ) × GL_{n}(ℂ)-module).

Now the second inequality in (2.4) is clear. The first inequality follows from the fact that 𝔑_{p,n} ⊂ Ξ_{r}^{(p)} (cf. (2.1)). Q.E.D.

Theorem 2.4 We assume that 2n < min(p, q). Then the regular function ring of the theta lift O_{[n;r,0]} decomposes as

ℂ[\overline{O_{[n;r,0]}}] ≃ \begin{matrix}∑ & ⊕ \\ λ,η∈P_{n} \end{matrix} m(λ, η) σ_{η}^{(p)} ⊠ σ_{λ}^{(q)}  (2.6)

as a K_{ℂ} = O(p, ℂ) × O(q, ℂ)-module, where the multiplicity m(λ, η) is given in (2.2) (cf. Lemma 2.3).

We shall prove Theorem 2.4 in the next section.

Corollary 2.5 (1) We have a multiplicity-free decomposition

ℂ[\overline{O_{[n;0,0]}}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} σ_{λ}^{(p)} ⊠ σ_{λ}^{(q)} (cf. [4]).

(2) If we denote the branching coefficient of the restriction GL_{p}(ℂ) ↓ O(p, ℂ) by b_{η}^{λ} (see (2.3)), the following decomposition holds.

ℂ[\overline{O_{[n;n,0]}}] ≃ \begin{matrix}∑ & ⊕ \\ λ,η∈P_{n} \end{matrix} b_{η}^{λ} σ_{η}^{(p)} ⊠ σ_{λ}^{(q)}.

3 Harmonic polynomials and geometric quotient

In this section, we always assume that 2n < min(p, q) as in the former section. To prove Theorem 2.4, we study the induced algebra homomorphisms

φ^{∗} : ℂ[𝔰] ⟶ ℂ[W ], and ψ^{∗} : ℂ[𝔰′] ⟶ ℂ[W ].

Let us introduce a coordinate on 𝔰′. Take (C, D) ∈ 𝔰′_{+} ⊕ 𝔰′_{−} = 𝔰′, where C = (C_{ij}) and D = (D_{ij}) are symmetric matrices. We use {C_{ij} | 1 ≤ i ≤ j ≤ n} ∪ {D_{ij} | 1 ≤ i ≤ j ≤ n} as a coordinate on 𝔰′. Then ψ^{∗} is given explicitly by

ψ^{∗}(C_{ij}) = ∑_{k=1}^{p} A_{ki}A_{kj}; ψ^{∗}(D_{ij}) = ∑_{l=1}^{q} B_{li}B_{lj},

where {A_{ij} = Z_{ij} | 1 ≤ i ≤ p, 1 ≤ j ≤ n} ∪ {B_{ij} = Z_{p+i,j} | 1 ≤ i ≤ q, 1 ≤ j ≤ n} is considered as a system of coodinate functions on W which extracts the (i, j)-th element

of Z = (\begin{matrix}A \\ B \end{matrix}) ∈ M_{p+q,n}(ℂ) = W. Note that the image of the coodinate functions via ψ^{∗} is

precisely the fundamental invariants for K_{ℂ} = O(p, ℂ) × O(q, ℂ), which generate all the K_{ℂ}-invariants in ℂ[W ]. Thus

ψ^{∗} : ℂ[𝔰′] ⟶ ℂ[W ]^{K_{ℂ}}

is surjective. Moreover, we have Lemma 3.1 Assume that 2n < min(p, q). Then the map ψ^{∗} : ℂ[𝔰′] ⟶ ℂ[W ]^{K_{ℂ}} is an isomorphism.

Similarly, if we introduce a coordinate on 𝔰 by the (k, l)-th element of X = (X_{kl}) ∈ M_{p,q}(ℂ) = 𝔰, we see that

φ^{∗}(X_{kl}) = ∑_{i=1}^{n} A_{ki}B_{li},

which is a fundamental invariant for K′_{ℂ} = GL_{n}(ℂ). Thus ψ^{∗} : ℂ[𝔰] → ℂ[W ]^{K′_{ℂ}} is surjective by the similar arguments as above. Let 𝔰_{[n]} = {X ∈ M_{p,q}(ℂ) | rank X ≤ n} be the determinantal variety of rank n.

Lemma 3.2 Assume that 2n < min(p, q). Then the image of φ is precisely the determinantal variety: φ(W) = 𝔰_{[n]}. Thus the induced algebra homomorphism φ^{∗} : ℂ[𝔰_{[n]}] ⟶ ℂ[W ]^{K′_{ℂ}} is an isomorphism.

The proof of the above two lemmas are almost immediate. We omit them.

Proposition 3.3 Let O_{[n;r,s]} be the theta lift of O′_{[r,s]}. Then \overline{O_{[n;r,s]}} is the geometric quotient of the fiber Ξ_{[r,s]} = ψ^{−1}(\overline{O′_{[r,s]}}) by K′_{ℂ}, i.e., \overline{O_{[n;r,s]}} = Ξ_{[r,s]}//K′_{ℂ}. In particular, we have

ℂ[\overline{O_{[n;r,s]}}] ≃ ℂ[Ξ_{[r,s]}]^{K′_{ℂ}}.

Proof. Let J = I(Ξ_{[r,s]}) be the defining ideal of Ξ_{[r,s]} ⊂ W. Then, I = (φ^{∗})^{−1}(J) is the defining ideal of \overline{O_{[n;r,s]}}, since φ(Ξ_{[r,s]}) = \overline{O_{[n;r,s]}}. Recall that φ^{∗} : ℂ[𝔰] → ℂ[W ]^{K′_{ℂ}} is surjective.

ℂ[𝔰] φ^{∗} : surjection ⟶ ℂ[W ]^{K′_{ℂ}}

projection↓  ↓^{projection}
ℂ[\overline{O_{[n;r,s]}}] = ℂ[𝔰]/I  ∼_{⟶} ℂ[W ]^{K′_{ℂ}}/J^{K′_{ℂ}}

Therefore, we get ℂ[𝔰]/I ≃ ℂ[W ]^{K′_{ℂ}}/J^{K′_{ℂ}}. Note that ℂ[Ξ_{[r,s]}]^{K′_{ℂ}} = (ℂ[W ]/J)^{K′_{ℂ}} ≃ ℂ[W ]^{K′_{ℂ}}/J^{K′_{ℂ}}. Thus, the proposition is proved. Q.E.D.

Let us consider the case where s = 0, and assume the decomposition (2.5). By the proposition above, we get

ℂ[\overline{O_{[n;r,0]}}] ≃ ℂ[Ξ_{[r,0]}]^{K′_{ℂ}}

= \begin{matrix}∑ & ⊕ \\ λ,μ,η∈P_{n} \end{matrix} m(λ, η) (σ_{η}^{(p)} ⊠ σ_{μ}^{(q)}) ⊠ (τ_{λ}^{∗} ⊠ τ_{μ})^{ΔGL_{n}(ℂ)}.

By Shur’s lemma, we have

(τ_{λ}^{∗} ⊠ τ_{μ})^{ΔGL_{n}(ℂ)} ≃ \{ \begin{matrix}0 & if λ ≠ μ, \\ ℂ & if λ = μ. \end{matrix}

Therefore, the above formula becomes

ℂ[\overline{O_{[n;r,0]}}] ≃ \begin{matrix}∑ & ⊕ \\ λ,η∈P_{n} \end{matrix} m(λ, η) σ_{η}^{(p)} ⊠ σ_{λ}^{(q)},

which finishes the proof of Theorem 2.4.

Finally, let us assume that r = n, and express the multiplicity m(λ, η) by the Littlewood-Richardson coefficient c_{μ, ν}^{λ}, which is defined by the following formula

τ_{μ} ⊗ τ_{ν} = \begin{matrix}∑ & ⊕ \\ λ \end{matrix} c_{μ, ν}^{λ}τ_{λ}.

Proposition 3.4 Let O_{[n;n,0]} be the theta lift of the open K_{ℂ}-orbit O′_{[n,0]} in 𝔰′_{+}. Then we get a K_{ℂ}-type decomposition

ℂ[\overline{O_{[n;n,0]}}] ≃ \begin{matrix}∑ & ⊕ \\ λ,η∈P_{n} \end{matrix} ( ∑_{μ∈P_{n}} c_{η, 2μ}^{λ}) σ_{η}^{(p)} ⊠ σ_{λ}^{(q)}.

Therefore, the multiplicity m(λ, η) in Theorem 2.4 is given by

m(λ, η) = ∑_{μ∈P_{n}} c_{η, 2μ}^{λ},

for r = n.

Proof. In this case, we have Ξ_{n}^{(p)} = M_{p,n}. Let H be the space of all O(p, ℂ)-harmonics in ℂ[M_{p,n}]. Then we have an isomorphism

H ⊗ ℂ[M_{p,n}]^{O(p,ℂ)} ⟶^{∼} ℂ[M_{p,n}]

given by the multiplication map. Thus we get

ℂ[Ξ_{n}^{(p)}] = ℂ[M_{p,n}] ≃ H ⊗ ℂ[M_{p,n}]^{O(p,ℂ)} ≃ H ⊗ ℂ[𝔰′_{+}].

From the following two decompositions,

H ≃ ℂ[𝔑_{p,n}] ≃ \begin{matrix}∑ & ⊕ \\ η∈P_{n} \end{matrix} σ_{η}^{(p)} ⊠ τ_{η}^{∗} (as an O(p, ℂ) × GL_{n}(ℂ)-module),
ℂ[𝔰′_{+}] ≃ \begin{matrix}∑ & ⊕ \\ μ∈P_{n} \end{matrix} τ_{2μ}^{∗} (as a GL_{n}(ℂ)-module),

we conclude that

ℂ[Ξ_{n}^{(p)}] ≃ H ⊗ ℂ[𝔰′_{+}]

≃ \begin{matrix}∑ & ⊕ \\ η,μ∈P_{n} \end{matrix} σ_{η}^{(p)} ⊠ (τ_{η}^{∗} ⊗ τ_{2μ}^{∗})

≃ \begin{matrix}∑ & ⊕ \\ η,μ∈P_{n} \end{matrix} σ_{η}^{(p)} ⊠ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} c_{η, 2μ}^{λ}τ_{λ}^{∗}

≃ \begin{matrix}∑ & ⊕ \\ λ,η∈P_{n} \end{matrix} ( ∑_{μ∈P_{n}} c_{η, 2μ}^{λ}) σ_{η}^{(p)} ⊠ τ_{λ}^{∗}.

Q.E.D.

As an application of the above proposition, we get an interesting formula for the branching coefficient b_{η}^{λ} (see (2.3) for definition).

Corollary 3.5 If 2n < min(p, q), then we have

b_{η}^{λ} = ∑_{μ∈P_{n}} c_{η, 2μ}^{λ} for λ, η ∈ P_{n}.

Remark 3.6 The branching coefficient b_{η}^{λ} is naturally identified with the multiplicity of the K-type τ_{λ} in the holomorphic discrete series of Sp(2n, ℝ) with the minimal K-type τ_{η}. Thus, it does not depend on the particular value p, but only depends on λ, η ∈ P_{n}.

Proof. This follows from Corollary 2.5 (2). Q.E.D.

4 Further results and comments

Let us briefly discuss generalizations of the results above.

First, we note that we can develop the similar theory interchanging the role of the pair (G, G′), if p + q ≤ n holds. So, if one of the pair is very small (i.e., if the pair is in the stable range), we can define the theta lifting from the smaller member of the pair to the larger one.

Almost all the arguments and results above are also valid for the other type I dual pairs with appropriate modifications. However, we must develop a new, unified language to describe them in general. For example, at present, we have to construct double fibration maps based on the case-by-case analysis. See the arguments in [6] for the pair U(p, q) × U(n, n).

Though the double fibration maps defined here might seem quite ad hoc, we have a natural interpretation for them, using the kernels and the images of nilpotent elements (cf. [7], [3]). Also there may be another interpretation by using moment maps. These interpretations will be useful for a general theory.

Our correspondence of nilpotent orbits is intimately related to the theta lifts of representations of Sp(2n, ℝ) to O(p, q). The orbits O′_{[r,0]} treated in this note are associated to the unitary highest weight representations of Sp(2n, ℝ) (or its metaplectic double cover). In particular, O′_{[n,0]} corresponds to a holomorphic discrete series representation. Therefore, the theta lift O_{[n;n,0]} should be associated to the theta lift of a holomorphic discrete series. See [8] for the theta lift of the trivial representation, which is associated to the trivial orbit O′_{[0,0]} = {0}.

Detailed discussions on the subjects commented above will appear elsewhere.

References

[1] R. Howe, Reciprocity laws in the theory of dual pairs. Progr. Math., 40, pp. 159–175, Birkhäuser Boston, Boston, Mass., 1983.

[2] B. Kostant and S. Rallis, Orbits and representations associated with symmetric spaces. Amer. J. Math., 93 (1971), 753 – 809.

[3] H. Kraft and C. Procesi, Closures of conjugacy classes of matrices are normal. Invent. Math. 53 (1979), no. 3, 227–247.

[4] Kyo Nishiyama, Multiplicity-free actions and the geometry of nilpotent orbits. Preprint, 1999.

[5] Kyo Nishiyama, Hiroyuki Ochiai, and Kenji Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules (Hermitian symmetric case), Kyushu University Preprint Series in Mathematics, 1999-16.

[6] Kyo Nishiyama and Chen-Bo Zhu, Theta lifting of the trivial representation and the associated nilpotent orbit — the case of U(p, q) × U(n, n) —. In “Proceedings of Symposium on Representation Theory 1999” in Tateyama, Chiba, pp. 188 – 206.

[7] Takuya Ohta, The closures of nilpotent orbits in the classical symmetric pairs and their singularities. Tôhoku Math. J. (2) 43 (1991), no. 2, 161–211.

[8] C.-B. Zhu and J.-S. Huang, On certain small representations of indefinite orthogonal groups, Represent. Theory 1 (1997), 190–206 (electronic).

[Compiled on February 3, 2000, Ver. 2.0 (Thu Feb 3 15:10:50 JST 2000)]