THETA LIFTING OF NILPOTENT ORBITS FOR SYMMETRIC PAIRS A long version

KYO NISHIYAMA AND CHEN-BO ZHU

Abstract. We consider a reductive dual pair (G, G′) in the stable range with G′ the smaller member and of Hermitian symmetric type. We study theta lifting of nilpotent K′_{ℂ}-orbits, where K′ is a maximal compact subgroup of G′. The main result is the description of the precise K_{ℂ}-module structure of the regular function ring of the closure of the lifted nilpotent orbit of the symmetric pair (G, K). As an application, we prove sphericality and normality of the closure of certain nilpotent K_{ℂ}-orbits obtained this way.

Contents

Introduction 2 1. Diamond pairs 3 2. Theta lift of a nilpotent orbit 4 3. Lifting of holomorphic nilpotent orbits 11 3.1. Trivial orbit 12 3.2. Singular holomorphic orbits 12 3.3. Regular holomorphic orbit 13 4. Appendix 17 4.1. O(p, q) × Sp(2n, ℝ) (2n < min (p, q)) 17 4.2. U(p, q) × U(m, n) (m + n ≤ min (p, q)) 20 4.3. Sp(p, q) × O^{∗}(2n) (n ≤ min (p, q)) 22 References 25

Date: Ver. 1.0 [Wed Jul 17 10:06:19 JST 2002] (compiled on July 17, 2002). 1991 Mathematics Subject Classification. Primary 22E46, 11F27. Key words and phrases. reductive dual pair, theta lifting, nilpotent orbits, harmonic polynomial,

invariant theory.

Introduction

Let (G, G′) be a reductive dual pair in a symplectic group 𝔾 = Sp(2N, ℝ), where N denotes the rank of 𝔾. Throughout this paper, we will assume that (G, G′) is of type I, and it is in the stable range with G′ the smaller member (cf. [5, 11]). We will also assume that G′/K′ is an irreducible Hermitian symmetric space. By the classification of irreducible dual pairs, our restriction amounts to saying that (G, G′) is from the following list (cf. [7]).

Table 1. The dual pairs treated in this paper

the pair (G, G′)  stable range condition
(O(p, q), Sp(2n, ℝ)) 2n < min(p, q)
(U(p, q), U(m, n)))  m + n ≤ min(p, q)
(Sp(p, q), O^{∗}(2n))  n ≤ min(p, q) + 1

Note that we have excluded the equality 2n = min(p, q) in the first case, to avoid some (small) technicalities.

▶[Extra Remarks] This condition is equivalent to the condition that all the constituents are holomorphic discrete series for the compact dual pairs K^{±} × G′. ◀

We introduce some notations. Let 𝔤 (resp. 𝔤′) be the complexification of the Lie algebra of G (resp. G′) and K (resp. K′) a maximal compact subgroup of G (resp. G′). Let

𝔤 = 𝔨 ⊕ 𝔰, 𝔤′ = 𝔨′ ⊕ 𝔰′,

be respectively Cartan decompositions of 𝔤 and 𝔤′, compatible in certain way (see §2).

Let 𝕆′ ⊂ 𝔰′ be a nilpotent K′_{ℂ}-orbit. Then using certain double fiberation from W ≃ ℂ^{N}, we can define the theta lift of 𝕆′, which is a certain nilpotent K_{ℂ}-orbit 𝕆 in 𝔰 (see §2; cf. [19]). We shall denote 𝕆 = θ(𝕆′). Let ℂ[\overline{𝕆}] (resp. ℂ[\overline{𝕆′}]) be the regular function ring on the closure of 𝕆 (resp. 𝕆′). The purpose of this paper is to investigate the K_{ℂ}- module structure of ℂ[\overline{𝕆}] in terms of the K′_{ℂ}-module structure of ℂ[\overline{𝕆′}]. We will show that the K_{ℂ}-module structure of ℂ[\overline{𝕆}] can in fact be described entirely and explicitly by the regular function ring ℂ[\overline{𝕆′}] and the space of K_{ℂ}-harmonic polynomials H(K_{ℂ}) on W (using certain branching coefficients). The main results are summarized in Theorem 2.9.

In §3, we will give some examples of the lifted orbits and their regular function rings. As an application we prove normality of the closure of some of these nilpotent orbits, which are lifted from the smaller group. Also, we obtain a family of spherical nilpotent orbits. We believe that they occupy a large portion of the set of all the spherical nilpotent K_{ℂ}-orbits, which are not classified yet.

In the Appendix, we describe the explicit form of moment maps and give some basic properties of null cones. Most of the results in the Appendix are known to the experts, but we include them because of lack of appropriate references.

1. Diamond pairs

We review certain structure results related to our dual pairs [7].

Let W_{ℝ} ≃ ℝ^{2N} be a real symplectic space which realizes 𝔾 = Sp(2N, ℝ) as a symplectic group on W_{ℝ}. There is a canonical complex structure on W_{ℝ} and we can view W_{ℝ} as (the underlying real vector space of) a complex vector space W ≃ ℂ^{N}. The symplectic form on W_{ℝ} is then given by the imaginary part of a canonical positive definite Hermitian form on W. By this identification, a maximal compact subgroup 𝕂 of 𝔾 is realized as the unitary group U(W) ≃ U(N) on W ≃ ℂ^{N}.

We may choose maximal compact subgroups K and K′ of G and G′ respectively, in such a way that K · K′ is contained in the standard maximal compact subgroup 𝕂 ≃ U(N) of 𝔾.

In view of Table 1, we sometimes write G = G(p, q), namely G(p, q) will denote one of the groups O(p, q), U(p, q) or Sp(p, q). We denote simply G(k) = G(k, 0), which is compact. Though we are mainly concerned with non-compact cases, results on compact cases will be used heavily.

From the pair (G, G′), we define another three dual pairs, which form so-called diamond dual pairs (see [7, §5]). Namely, take the commutant of K in 𝔾 and denote it by M′. Then M′ is also of Hermitian symmetric type and isomorphic to G′ × G′ containing G′ as the diagonal. The pair (K, M′) is a dual pair of compact type. Also, take the full commutant of K in 𝕂, and denote it by L′. Then L′ is a maximal compact subgroup of M′, and it is isomorphic to K′ × K′ and contains K′ as a diagonal subgroup. Similarly, define M as the commutant of K′ in 𝔾, and L the commutant in 𝕂.

Let us summarize this somewhat complicated situation by the following diagram (Fig. 1).

An explicit description of the diamond pairs for our three cases is given in [7, (5.3)]. In the table there, L (resp. L′) in our notation is written as M^{(1,1)} (resp. M′^{(1,1)}). For convenience of readers, we reproduce the table here (Table 2 below) with some additional remarks.

Note that M/L is a Hermitian symmetric space (not necessarily irreducible), and the containment M ⊃ G (resp. L ⊃ K) may or may not be the diagonal map, but it is always a symmetric pair.

Recall 𝔾 = Sp(W_{ℝ}) and the complex vector space W ≃ ℂ^{N} which is identical to W_{ℝ} as a real vector space. Then there exists a direct sum decomposition W_{ℝ} = W_{ℝ}^{+} ⊕ W_{ℝ}^{−} and correspondingly W = W^{+} ⊕ W^{−}, which are compactible with direct product decompositions L = L^{+} × L^{−} and K = K^{+} × K^{−} in the following way. The subgroups

Figure 1. The diagram of a diamond pair.

[figure: Δ; Δ; L = L^{+} × L^{−}; M; K = K^{+} × K^{−}; G; G′; K′; M′ = G′ × G′; L′ = K′ × K′; ⟵⟶; ⟷; ⟵⟶]

Table 2. Three diamond dual pairs.

\begin{matrix}(G, G′) & M & K = K^{+} × K^{−} & L = L^{+} × L^{−} & K′ \\ (O(p, q), Sp(2n, ℝ)) & U(p, q) & O(p)×O(q) & U(p)×U(q) & U(n) \\ (U(p, q), U(m, n))) & U(p, q)×U(p, q) & U(p)×U(q) & (U(p)×U(p))×(U(q)×U(q)) & U(m)×U(n) \\ (Sp(p, q), O^{∗}(2n)) & U(2p, 2q) & Sp(p)×Sp(q) & U(2p)×U(2q) & U(n) \end{matrix}

M′ = G′ × G′, L′ = K′ × K′

L^{±} and K^{±} are contained in the unitary group U(W^{±}). Moreover the pairs (K^{±}, G′) and (L^{±}, K′) are dual pairs in Sp(W_{ℝ}^{±}). Also note that L^{±} ⊃ K^{±} is a symmetric pair.

For explicit realization of these decompositions, see the Appendix.

2. Theta lift of a nilpotent orbit

Let 𝔊 = Lie (𝔾)_{ℂ} (resp. 𝔎 = Lie (𝕂)_{ℂ}) be the complexified Lie algebra of 𝔾 = Sp(2N, ℝ) (resp. 𝕂 = U(N)). Let 𝔊 = 𝔎⊕𝔓 be the corresponding (complexified) Cartan decomposition. Note that the complexified Lie group 𝕂_{ℂ} acts on the Cartan space 𝔓 by the restriction of the adjoint action, and this action breaks 𝔓 into 𝔓_{±}, each of which can be identified with a copy of the space Sym_{N}(ℂ) of complex symmetric matrices of size N. The action of k ∈ GL(N, ℂ) ≃ 𝕂_{ℂ} on 𝔓 = 𝔓_{+} ⊕ 𝔓_{−} ≃ Sym_{N}(ℂ) ⊕ Sym_{N}(ℂ) is given by

k · (X, Y) = (kX^{t}k, ^{t}k^{−1}Y k^{−1}) (X, Y ∈ Sym_{N}(ℂ)).

Let 𝕆_{min} ⊆ 𝔓_{+} be the minimal nilpotent 𝕂_{ℂ} ≃ GL(N, ℂ) orbit in 𝔓_{+}. We have the identification 𝕆_{min} = {(X, 0) | X ∈ Sym_{N}(ℂ), rank X = 1}. It is also well-known that there is a map

\begin{aligned}j : & W & ⟶ & \overline{𝕆_{min}} = 𝕆_{min} ∪ \{0\}, \\ & w & ⟼ & (^{t}ww, 0), w = (w_{1}, …, w_{N}) \end{aligned}

which gives \overline{𝕆_{min}} as a geometric quotient of the space W by the action of O(1, ℂ) = \{±1\}, and that the quotient map is 𝕂_{ℂ}-equivariant.

Let us consider the dual pair (G, G′) ⊆ 𝔾. The maximal compact subgroup K (resp. K′) determines a Cartan decomposition of the complexified Lie algebra 𝔤 (resp. 𝔤′) of G (resp. G′):

𝔤 = 𝔨 ⊕ 𝔰, 𝔤′ = 𝔨′ ⊕ 𝔰′,

where 𝔨 (resp. 𝔨′) is the complexified Lie algebra of K (resp. K′), and 𝔰 (resp. 𝔰′) is the orthogonal complement of 𝔨 (resp. 𝔨′) with respect to the Killing form. The subspace 𝔰 (resp. 𝔰′) carries a linear action of K_{ℂ} (resp. K′_{ℂ}) via the restriction of the adjoint representation. Since G′/K′ is a Hermitian symmetric space, 𝔰′ breaks up into irreducible pieces 𝔰′_{±} under the action of K′_{ℂ}.

We can arrange the Cartan decompositions so that 𝔰 ⊕ 𝔰′ is contained in 𝔓. Then the inclusion map 𝔰 ⊂ 𝔓 (resp. 𝔰′ ⊂ 𝔓) induces the projection 𝔓^{∗} → 𝔰^{∗} (resp. 𝔓^{∗} → 𝔰′^{∗}). By identifying the complex linear dual 𝔓^{∗} (resp. 𝔰^{∗}, 𝔰′^{∗}) with 𝔓 (resp. 𝔰, 𝔰′) by the invariant trace form of matrices, we obtain two projection maps P_{𝔰} : 𝔓 → 𝔰 and P_{𝔰′} : 𝔓 → 𝔰′. The projections restricted to \overline{𝕆_{min}} will then induce the so-called moment maps:

φ : W ⟶^{geom. quotient by \{±1\}} \overline{𝕆_{min}} ⟶^{projection} 𝔰
ψ : W ⟶ \overline{𝕆_{min}} ⟶ 𝔰′

These maps are K_{ℂ} × K′_{ℂ}-equivariant with the trivial K_{ℂ}-action on 𝔰′, and the trivial K′_{ℂ}-action on 𝔰 respectively. Also we can (and we will) choose the subspaces 𝔰′_{±} ⊂ 𝔰′ and W^{±} ⊂ W so that

ψ(W^{±}) ⊂ 𝔰′_{±}

holds (cf. §1 for the notation W^{±}).

For a set S ⊆ 𝔓, we denote by N (S) the subset of all nilpotent elements in S.

Lemma 2.1. We can choose Cartan decompositions and moment maps so that the following conditions hold.

(1) ψ is an affine quotient map by K_{ℂ} onto 𝔰′, i.e., 𝔰′ ≃ W//K_{ℂ}.

(2) φ is an affine quotient map by K′_{ℂ} onto its closed image.

(3) φ(ψ^{−1}(N (𝔰′))) ⊂ N (𝔰).

Proof. To show that ψ : W → 𝔰′ is an affine quotient map, it suffices to show that ψ induces an algebra isomorphism ℂ[𝔰′] ≃ ℂ[W]^{K_{ℂ}}, where ℂ[W]^{K_{ℂ}} denotes the space of invariants of the polynomial ring ℂ[W]. Clearly it suffices to show that the restrictions ψ|_{W^{±}} induce algebra isomorphisms ℂ[𝔰′_{±}] ≃ ℂ[W^{±}]^{K_{ℂ}^{±}}.

Similarly, statement (2) asserts that the induced algebra homomorphism φ^{∗} : ℂ[𝔰] → ℂ[W] maps ℂ[𝔰] onto the K′_{ℂ}-invariants ℂ[W]^{K′_{ℂ}}. Note that the closed subvariety Im φ ⊂ 𝔰 is then defined by the ideal of relations of generators of invariants, or in other words, the kernel of φ^{∗}.

Explicit constructions of moment maps for each case of the three dual pairs in Table 1 are summarized in the Appendix. From the explicit formulas given there, the statements (1) – (3) follow easily from classical invariant theory [21].

Using the moment maps, we shall define the theta lift of nilpotent orbits from N (𝔰′) to N (𝔰). Before that, we need the following well-known results due to Kostant and Howe. See Theorem 2.5.4, Proposition 3.7.3.7, and Theorem 3.8.6.4 in [8], and references cited there.

As usual denote by H(K_{ℂ}^{±}) the space of harmonic polynomials on W^{±} (under the action of K_{ℂ}^{±}). Similar notations apply throughout this article.

Theorem 2.2. Let (G, G′) be in the stable range (see Table 1) with G′ the smaller member, and consider the dual pairs (K^{±}, G′) in Sp(W_{ℝ}^{±}). Then with respect to the action of K_{ℂ}^{±}, we have a tensor product decomposition of ℂ[W^{±}]:

ℂ[W^{±}] ≃ H(K_{ℂ}^{±}) ⊗ ℂ[W^{±}]^{K_{ℂ}^{±}}.

Moreover as a K_{ℂ}^{±} × K′_{ℂ}-module, the harmonics H(K_{ℂ}^{±}) decompose without multiplicity.

Corollary 2.3. We have

ℂ[W] ≃ H(K_{ℂ}) ⊗ ℂ[W]^{K_{ℂ}} ≃ H(K_{ℂ}) ⊗ ℂ[𝔰′].

Consequently the moment map ψ : W → 𝔰′ is flat.

Proof. Note that the induced map ψ^{∗} of the moment map ψ is such that

ψ^{∗} : ℂ[𝔰′] ≃ ℂ[W]^{K_{ℂ}} ↪ ℂ[W].

Since ℂ[W] ≃ H(K_{ℂ}) ⊗ ℂ[𝔰′], it is free over ℂ[𝔰′]. This implies that ψ is flat.

We introduce some notations. For any complex reductive Lie group E, let Irr(E) be the set of equivalence classes of irreducible finite dimensional representations of E. Given a (completely reducible) representation of E on U, let Irr(E; U) be the subset of Irr(E) which appear in U.

As noted previously, the space H(K_{ℂ}^{+}) is multiplicity-free as a representation of K_{ℂ}^{+} × K′_{ℂ}. Furthermore the decomposition

H(K_{ℂ}^{+})|_{K_{ℂ}^{+}×K′_{ℂ}} ≃ ^{∑⊕}_{σ∈Irr(K_{ℂ}^{+};H^{+}),τ∈Irr(K′_{ℂ};H^{+})} σ ⊠ τ  (2.1)

determines a one-to-one correspondence between σ ∈ Irr(K_{ℂ}^{+}; H^{+}) and τ ∈ Irr(K′_{ℂ}; H^{+}) (see [6]). Similar statements apply for H(K_{ℂ}^{−}).

We shall abbreviate R^{±}(K′_{ℂ}) = Irr(K′_{ℂ}; H(K_{ℂ}^{±})) in the following. Under the assumption of stable range, one can explicitly check that τ^{∗} ∈ R^{+}(K′_{ℂ}) if and only if τ ∈ R^{−}(K′_{ℂ}), where τ denotes the contragredient representation of τ ∈ Irr(K′). So we put

R(K′_{ℂ}) = R^{+}(K′_{ℂ})^{∗} = R^{−}(K′_{ℂ}).  (2.2)

To summarize, for each τ ∈ R^{±}(K′_{ℂ}), there is a unique σ ∈ Irr(K_{ℂ}^{±}) such that

Hom_{K_{ℂ}^{±}×K′_{ℂ}}(σ ⊠ τ, H(K_{ℂ}^{±})) ≠ 0.

We denote this σ by σ^{±}(τ), specifying the dependency of τ and the sign ±. Then, we can rewrite (2.1) and its sister statement as

H(K_{ℂ}^{+}) ≃ \begin{matrix}∑ & ⊕ \\ τ∈R(K′_{ℂ}) \end{matrix} σ^{+}(τ^{∗}) ⊠ τ^{∗}, and H(K_{ℂ}^{−}) ≃ \begin{matrix}∑ & ⊕ \\ τ∈R(K′_{ℂ}) \end{matrix} σ^{−}(τ) ⊠ τ.  (2.3)

Remark 2.4. By the theory of highest weight, we may identify R(K′_{ℂ}) with a certain semigroup in the integral weight lattice of K′_{ℂ}. We note that in an explicit realization, the semigroup R(K′_{ℂ}) may be identified with a set of partitions, and it is saturated. The explicit decompositions in Equation (2.3) are also well-known. See [10] for the orthogonal and unitary cases, or [8] in general.

Theorem 2.5. Take a nilpotent K′_{ℂ}-orbit 𝕆′ ⊂ N (𝔰′). Then the scheme theoretic fiber ψ^{−1}(\overline{𝕆′}) = W ×_{𝔰′} \overline{𝕆′} is a reduced, closed irreducible affine subvariety of W. Moreover, it is the closure of a single K_{ℂ} × K′_{ℂ}-orbit in W.

Proof. Let us prove that the fiber W ×_{𝔰′} \overline{𝕆′} is reduced. For this, it is enough to show that the fiber of each closed point in 𝔰′ is reduced. Namely, we prove the following lemma, which may be of independent interest.

▶[Extra Remarks] We will supply here the reason why it is enough to do it locally.

We are to prove that the algebra A = ℂ[W] ⊗_{ℂ[𝔰′]} ℂ[\overline{𝕆′}] has no nilpotent element. Let us prove it by contradiction. So, assume that f ∈ A, ≠ 0 is nilpotent, i.e., f^{n} = 0 for some n > 0. Then for any x ∈ \overline{𝕆′} we have f^{n}|_{ψ^{−1}(x)} = 0, which implies f|_{ψ^{−1}(x)} = 0 because ψ^{−1}(x) is reduced. Since the defining ideal of ψ^{−1}(x) is 𝕀_{x} = ℂ[W] ⊗_{ℂ[𝔰′]} 𝔪_{x} (𝔪_{x} is the maximal ideal in ℂ[𝔰′] at x), we have

f ∈ ⋂_{x∈\overline{𝕆′}} 𝕀_{x} = ⋂_{x∈\overline{𝕆′}} ℂ[W] ⊗_{ℂ[𝔰′]} 𝔪_{x}

=^{(⋆)} ℂ[W] ⊗_{ℂ[𝔰′]} ( ⋂_{x∈\overline{𝕆′}} 𝔪_{x}) (see below for (⋆))

= ℂ[W] ⊗_{ℂ[𝔰′]} 𝕀(\overline{𝕆′}),

where 𝕀(\overline{𝕆′}) is the defining ideal of \overline{𝕆′}. This means that f = 0 on ψ^{−1}(\overline{𝕆′}), which is a contradiction.

Let us prove the equality (⋆). Since the inclusion ⊃ is obvious, we prove

⋂_{x∈\overline{𝕆′}} ℂ[W] ⊗_{ℂ[𝔰′]} 𝔪_{x} ⊂ ℂ[W] ⊗_{ℂ[𝔰′]} ( ⋂_{x∈\overline{𝕆′}} 𝔪_{x}).

Note that ℂ[W] ⊗_{ℂ[𝔰′]}𝔪_{x} ≃ H ⊗ 𝔪_{x}. Fix a basis {h_{i}}_{i∈I} in H, and write f ∈ ∩_{x}H ⊗ 𝔪_{x} as

f = ∑_{i∈I} h_{i} ⊗ a_{i,x} ∈ H ⊗ 𝔪_{x}.

But this expression means that a_{i,x} = a_{i,x′} ∈ 𝔪_{x} ∩ 𝔪_{x′} for any x, x′ ∈ \overline{𝕆′}, hence we have

f ∈ H ⊗ (∩_{x}𝔪_{x}) = ℂ[W] ⊗_{ℂ[𝔰′]} (∩_{x}𝔪_{x}).

◀

Lemma 2.6. Let x ∈ 𝔰′ be a closed point. Then the scheme theoretic fiber ψ^{−1}(x) = W ×_{𝔰′} {x} is a closed, reduced and irreducible affine subvariety of W. Moreover, it is the closure of a single K_{ℂ}-orbit.

Proof. The scheme theoretic fiber is defined by

W ×_{𝔰′} {x} = Spec (ℂ[W] ⊗_{ℂ[𝔰′]} ℂ_{x}) .

Here ℂ_{x} = ℂ[𝔰′]/𝔪_{x} ≃ ℂ, where 𝔪_{x} denotes the maximal ideal corresponding to the closed point x. Put A_{x} = ℂ[W] ⊗_{ℂ[𝔰′]} ℂ_{x}. We are to show that A_{x} is an integral domain.

Let us recall the tensor product decomposition

ℂ[W] = H(K_{ℂ}) ⊗ ℂ[𝔰′].

Thus, if we abbreviate H = H(K_{ℂ}), we can identify

A_{x} = ℂ[W] ⊗_{ℂ[𝔰′]} ℂ_{x} = H ⊗ ℂ_{x} ≃ H.

However, we should be careful, because H does not have any structure of algebra, but only enjoys a structure of K_{ℂ} × K′_{ℂ}-module. Since H is a graded subspace of ℂ[W], it is naturally graded by the ordinary degree of polynomials. We denote the grading by

H ≃ ⨁_{k≥0} H_{k}.

Then, A_{x} becomes a filtered algebra by putting

(A_{x})_{i} = ∑_{k≤i} H_{k} ⊗ 1 ⊂ H ⊗ ℂ_{x} = A_{x}.

Now assume that A_{x} is not an integral domain. Then there are non-zero elements a, b ∈ A_{x} such that a · b = 0. Let us express a and b as

a = ∑_{i=0}^{d} a_{i} ⊗ 1 ∈ H ⊗ ℂ_{x} ; b = ∑_{j=0}^{d′} b_{j} ⊗ 1 ∈ H ⊗ ℂ_{x},

with a_{d} ≠ 0 and b_{d′} ≠ 0. Then ab = 0 implies

(a_{d}b_{d′}) ⊗ 1 ∈ (A_{x})_{d+d′−1},

but this is impossible. To see it, let J = ℂ[W]_{+}^{K_{ℂ}} be the augmentation ideal generated by homogeneous invariants of positive degree. Then H · J ⊂ ℂ[W] is a prime ideal defining the null cone 𝔑^{+} × 𝔑^{−}, which is known to be irreducible. Note that H · J ≃ H ⊗ J and ℂ[𝔑^{+} × 𝔑^{−}] = ℂ[W]/HJ ≃ H. Since a_{d}b_{d′} ∉ HJ by the irreducibility of the null cone, we have

a_{d} · b_{d′} ⊗ 1 ∉ ∑_{k=0}^{d+d′−1} H_{k} ⊗ ℂ_{x} = (A_{x})_{d+d′−1},

which is a contradiction.

Next, we prove that the fiber ψ^{−1}(x) contains an open dense K_{ℂ}-orbit. Put M = ψ^{−1}(x) and denote by M̂ the asymptotic cone of M (see [18, §5.2] for the definition of asymptotic cone). Then, by the flatness of ψ, the asymptotic cone M̂ coincides with the null cone 𝔑. Let O be a generic K_{ℂ}-orbit in M. Consider the cone ℂM generated by M in W, then it is clear that the dimension of a generic orbit in ℂM is equal to dim O, which in turn coincides with the dimension of the generic orbit in \overline{ℂM} ⊂ W. Since 𝔑 = M̂ ⊂ \overline{ℂM}, the dimension of a generic orbit in 𝔑 cannot exceed that of O. Note that 𝔑 has an open dense orbit (this can be verified by a case-by-case analysis, see the Appendix). This means that dim O ≥ dim 𝔑. On the other hand, we have an equality dim M = dim M̂ = dim 𝔑 of dimensions, hence dim O ≥ dim M. Since O ⊂ M, we conclude that dim O = dim M, and that O is an open dense orbit in M, by the irreducibility of M just proved above.

Let us return to the proof of Theorem 2.5.

By K′_{ℂ}-equivariancy of ψ, we get ψ^{−1}(𝕆′) = K′_{ℂ}·ψ^{−1}({x}) for any x ∈ 𝕆′. Consider the multiplication map K′_{ℂ}× ψ^{−1}({x}) → ψ^{−1}(𝕆′). Since K′_{ℂ}× ψ^{−1}({x}) is irreducible by the above lemma, ψ^{−1}(𝕆′) is also irreducible (as an image of an irreducible set). Therefore its closure \overline{ψ^{−1}(𝕆′)} is irreducible. Since the moment map ψ is flat by Corollary 2.3, it is an open map ([4, Ex. (III.9.1)]). Thus we conclude that ψ^{−1}(\overline{𝕆′}) = \overline{ψ^{−1}(𝕆′)} is irreducible.

By the same lemma, ψ^{−1}(x) contains an open dense K_{ℂ}-orbit O_{x}. Then the union of K′_{ℂ}-translates K′_{ℂ} · O_{x} is dense in ψ^{−1}(𝕆′). Thus \overline{K′_{ℂ} · O_{x}} = \overline{ψ^{−1}(𝕆′)} = ψ^{−1}(\overline{𝕆′}), which means that K′_{ℂ} · O_{x} is open in ψ^{−1}(\overline{𝕆′}).

Remark 2.7. Recently, T. Ohta proved the irreducibility and the existence of an open dense orbit by a totally different (but case-by-case) method ([16], [17]). In fact, his method is applicable beyond the stable range. However, outside the stable range, things get much more complicated and the same statement of the above theorem is no longer true.

From the above theorem, we see that φ(ψ^{−1}(\overline{𝕆′})) is a K_{ℂ}-stable irreducible closed set in 𝔰. Since φ(ψ^{−1}(\overline{𝕆′})) is contained in the nilpotent variety N (𝔰), which has only finite number of K_{ℂ}-orbits, it must be the closure of a single K_{ℂ}-orbit 𝕆.

Definition 2.8. Let 𝕆′ ⊂ 𝔰′ be a nilpotent K′_{ℂ}-orbit. Then ψ^{−1}(\overline{𝕆′}) is a closed, irreducible affine subvariety in W. The image φ(ψ^{−1}(\overline{𝕆′})) is the closure of a single nilpotent K_{ℂ}-orbit in 𝔰, i.e., φ(ψ^{−1}(\overline{𝕆′})) = \overline{𝕆} for a certain nilpotent K_{ℂ}-orbit 𝕆. We call 𝕆 the theta lift of 𝕆′, and denote it by θ(𝕆′).

Recall the subset R(K′_{ℂ}) ⊂ Irr(K′_{ℂ}) defined in Equation (2.2).

Theorem 2.9. Let (G, G′) be in the stable range (see Table 1) with G′ the smaller member. For a nilpotent K′_{ℂ}-orbit 𝕆′ ⊂ N (𝔰′), denote its theta lift by 𝕆 = θ(𝕆′). Put Ξ(𝕆′) = ψ^{−1}(\overline{𝕆′}). Then the closure \overline{𝕆} is an affine quotient of Ξ(𝕆′) by K′_{ℂ}.

\overline{𝕆} ≃ Ξ(𝕆′)//K′_{ℂ}.

Moreover, the K_{ℂ}-module structure of the regular function ring of \overline{𝕆} is given in terms of \overline{𝕆′} as follows:

ℂ[\overline{𝕆}] ≃ (H(K_{ℂ}) ⊗ ℂ[\overline{𝕆′}])^{K′_{ℂ}}
≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} Hom_{K′_{ℂ}}(τ_{1} ⊗ τ_{2}^{∗}, ℂ[\overline{𝕆′}]) ⊗ (σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2})),  (2.4)

where σ^{+}(τ_{1}^{∗})⊠σ^{−}(τ_{2}) is an irreducible representation of K_{ℂ} = K_{ℂ}^{+}×K_{ℂ}^{−} given in Equation (2.3), and K_{ℂ} acts on the space of multiplicities Hom_{K′_{ℂ}}(τ_{1} ⊗ τ_{2}^{∗}, ℂ[\overline{𝕆′}]) trivially.

Proof. The assertion that \overline{𝕆} is an affine quotient follows from general theory of affine quotient maps because φ itself is a quotient map, and Ξ(𝕆′) is a K′_{ℂ}-invariant, affine closed subvariety of W (cf. [13, Prop. 3.3]).

Next we prove the statement on the module structure of ℂ[\overline{𝕆}]. By Theorem 2.5, we have

ℂ[Ξ(𝕆′)] ≃ ℂ[W ×_{𝔰′} \overline{𝕆′}] = ℂ[W] ⊗_{ℂ[𝔰′]} ℂ[\overline{𝕆′}]
≃ (ℂ[W^{+}] ⊗ ℂ[W^{−}]) ⊗_{ℂ[𝔰′_{+}]⊗ℂ[𝔰′_{−}]} ℂ[\overline{𝕆′}]
≃ (ℂ[W^{+}] ⊗_{ℂ[𝔰′_{+}]} ℂ[\overline{𝕆′}]) ⊗_{ℂ[\overline{𝕆′}]} (ℂ[W^{−}] ⊗_{ℂ[𝔰′_{−}]} ℂ[\overline{𝕆′}]) .  (2.5)

Note that ℂ[W^{±}] ≃ H(K_{ℂ}^{±})⊗ℂ[W^{±}]^{K_{ℂ}^{±}} and ℂ[W^{±}]^{K_{ℂ}^{±}} ≃ ℂ[𝔰′_{±}] by Theorem 2.2. Therefore we have

ℂ[W^{+}] ⊗_{ℂ[𝔰′_{+}]} ℂ[\overline{𝕆′}] ≃ (H(K_{ℂ}^{+}) ⊗ ℂ[W^{+}]^{K_{ℂ}^{+}}) ⊗_{ℂ[𝔰′_{+}]} ℂ[\overline{𝕆′}] ≃ H(K_{ℂ}^{+}) ⊗ ℂ[\overline{𝕆′}],

and (2.5) becomes

(2.5) = (H(K_{ℂ}^{+}) ⊗ ℂ[\overline{𝕆′}]) ⊗_{ℂ[\overline{𝕆′}]} (H(K_{ℂ}^{−}) ⊗ ℂ[\overline{𝕆′}])
≃ (H(K_{ℂ}^{+}) ⊗ H(K_{ℂ}^{−})) ⊗ ℂ[\overline{𝕆′}].

Since φ is an affine quotient map by the action of K′_{ℂ}, we have

ℂ[\overline{𝕆}] ≃ ℂ[Ξ(𝕆′)]^{K′_{ℂ}} ≃ ((H(K_{ℂ}^{+}) ⊗ H(K_{ℂ}^{−})) ⊗ ℂ[\overline{𝕆′}])^{K′_{ℂ}}

≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} ((σ^{+}(τ_{1}^{∗}) ⊠ τ_{1}^{∗}) ⊗ (σ^{−}(τ_{2}) ⊠ τ_{2}) ⊗ ℂ[\overline{𝕆′}])^{K′_{ℂ}}

≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} (τ_{1}^{∗} ⊗ τ_{2} ⊗ ℂ[\overline{𝕆′}])^{K′_{ℂ}} ⊗ (σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}))

≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} Hom_{K′_{ℂ}}(τ_{1} ⊗ τ_{2}^{∗}, ℂ[\overline{𝕆′}]) ⊗ (σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2})).

The theorem follows.

3. Lifting of holomorphic nilpotent orbits

We apply Theorem 2.9 to the lifting of the so-called holomorphic nilpotent orbits. In the process, we get a family of spherical orbits and prove normality of the closure of certain lifted orbits. This application reproduces some of our previous results ([12], [13], [15]) as well. In the following, we use the notations in Theorem 2.9 freely.

Let us consider the orbit decomposition of 𝔰′_{+} by the adjoint action of K′_{ℂ}. Any K′_{ℂ}-orbit in 𝔰′_{+} is clearly nilpotent. We call these nilpotent orbit holomorphic. It is well known that 𝔰′_{+} is a prehomogeneous vector space, and there exists a numbering of K′_{ℂ}-orbits 𝕆′_{0}, 𝕆′_{1}, …, 𝕆′_{l} in such a way that 𝕆′_{i−1} ⊂ \overline{𝕆′_{i}} for 1 ≤ i ≤ l. Here l is the real rank of G′. As a consequence, 𝕆′_{0} = \{0\} and \overline{𝕆′_{l}} = 𝔰′_{+}, i.e., 𝕆′_{l} is the open dense orbit in 𝔰′_{+}. The orbit 𝕆′_{l} is called regular, while the orbits {𝕆′_{k}}_{0≤k<l} are called singular.

3.1. Trivial orbit. We first consider the trivial orbit 𝕆′^{1} = 𝕆′_{0} = {0}. Then, the lifted orbit 𝕆^{1} = θ(𝕆′^{1}) is a two-step nilpotent orbit in N (𝔰) (cf. [12]). We have

Theorem 3.1. The group K_{ℂ} acts on \overline{𝕆^{1}} multiplicity-freely. As a K_{ℂ} = K_{ℂ}^{+} × K_{ℂ}^{−}- module, we have

ℂ[\overline{𝕆^{1}}] ≃ \begin{matrix}∑ & ⊕ \\ τ∈R(K′_{ℂ}) \end{matrix} σ^{+}(τ^{∗}) ⊠ σ^{−}(τ).  (3.1)

Proof. Since ℂ[𝕆′^{1}] = ℂ, by Theorem 2.9, we get

ℂ[\overline{𝕆^{1}}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} Hom_{K′_{ℂ}}(τ_{1} ⊗ τ_{2}^{∗}, ℂ) ⊗ (σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2})).

But, by Schur’s lemma, the multiplicity Hom_{K′_{ℂ}}(τ_{1} ⊗ τ_{2}^{∗}, ℂ) is not zero if and only if τ_{1} = τ_{2}, in which case it is ℂ, i.e., the multiplicity is one.

Recall that an orbit 𝕆 is called spherical, or more precisely K_{ℂ}-spherical, if a Borel subgroup of K_{ℂ} has a dense orbit in 𝕆. This is equivalent to saying that K_{ℂ} acts on \overline{𝕆} multiplicity-freely (i.e., ℂ[\overline{𝕆}] decomposes without multiplicity as a K_{ℂ}-module).

Corollary 3.2. The orbit 𝕆^{1} lifted from the trivial orbit is spherical. The closure \overline{𝕆^{1}} is a normal variety.

Proof. The claim that 𝕆^{1} is spherical follows directly from the above theorem. The normality follows from [20, Th. 10].

3.2. Singular holomorphic orbits. In this subsection, we are concerned with the case of singular holomorphic orbits, although the results also hold for the regular holomorphic orbit. We have more to say in the next subsection on the latter case.

Consider the orbit 𝕆′_{k}, where 0 ≤ k ≤ l. In the following, we recall the geometric structure of 𝕆′_{k}, and the decomposition of the regular function ring ℂ[\overline{𝕆′_{k}}] by the action of K′_{ℂ}. For details, see §7 of [14].

There exists a symplectic space W (k)_{ℝ} such that (G(k), G′) forms a compact dual pair in Sp(W (k)_{ℝ}). There is also a complex structure on W (k)_{ℝ}, such that, if we regard W (k) = W (k)_{ℝ} as a complex vector space with respect to the complex structure, then the imaginary part of the Hermitian form provides the original symplectic form on W (k)_{ℝ}. The complex space W (k) carries a natural linear action of G(k)_{ℂ} × K′_{ℂ}, and we have \overline{𝕆′_{k}} = W (k)//G(k)_{ℂ}, an affine quotient of W (k) by the action of G(k)_{ℂ}. Moreover, there is a larger compact group L(k) which contains G(k) as a symmetric subgroup, and the pair (L(k), K′) forms a dual pair in Sp(W (k)_{ℝ}) (cf. §1). From the standard result of Howe [6], we know that L(k)_{ℂ} × K′_{ℂ} acts on W (k) in a multiplicity-free manner:

ℂ[W (k)] ≃ \begin{matrix}∑ & ⊕ \\ τ∈Irr(K′_{ℂ};W (k)) \end{matrix} ρ_{k}(τ) ⊠ τ,  (3.2)

where ρ_{k}(τ) ∈ Irr(L(k)_{ℂ}; W (k)) corresponds to τ ∈ Irr(K′_{ℂ}; W (k)) via the above multiplicity-free action.

The following theorem is proved in [14].

Theorem 3.3. 𝕆′_{k} is a K′_{ℂ}-spherical nilpotent orbit, and we have the following multiplicity-free decomposition:

ℂ[\overline{𝕆′_{k}}] ≃ \begin{matrix}∑ & ⊕ \\ τ∈Irr(K′_{ℂ};𝕆′_{k}) \end{matrix}τ,

where

Irr(K′_{ℂ}; 𝕆′_{k}) = {τ ∈ Irr(K′_{ℂ}; W (k)) | ρ_{k}(τ) has a non-zero G(k)_{ℂ}-fixed vector}.

Now let us denote 𝕆_{k}^{hol} = θ(𝕆′_{k}), the theta lift of the holomorphic orbit 𝕆′_{k}. Then 𝕆_{k}^{hol} is a three-step nilpotent orbit in N (𝔰) (cf. [13]).

As an immediate consequence of Theorems 2.9 and 3.3, we have

Theorem 3.4. For τ, τ_{1}, τ_{2} ∈ Irr(K′_{ℂ}), let us denote by m(τ_{1}, τ_{2} ⊗ τ) the multiplicity of τ_{1} in the tensor product representation τ_{2} ⊗ τ. Then we have the following decomposition.

ℂ[\overline{𝕆_{k}^{hol}}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix}( ∑_{τ∈Irr(K′_{ℂ};𝕆′_{k})}m(τ_{1}, τ_{2} ⊗ τ)) σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}),  (3.3)

where R(K′_{ℂ}) = Irr(K′_{ℂ}; H(K_{ℂ}^{−})), as before.

3.3. Regular holomorphic orbit. Now consider 𝕆_{l}^{hol} lifted from the regular holomorphic orbit 𝕆′_{l} ⊂ 𝔰′_{+}. This example is already covered in the previous subsection. However the orbit 𝕆_{l}^{hol} has a new property which is not shared by the lifts of singular holomorphic orbits. This property allows us to conclude the normality of \overline{𝕆_{l}^{hol}} as well as to have a more convenient description of ℂ[\overline{𝕆_{l}^{hol}}].

For the regular orbit, we have

Ξ(𝕆′_{l}) = ψ^{−1}(\overline{𝕆′_{l}}) = ψ^{−1}(𝔰′_{+}) = W^{+} × 𝔑^{−},

where ψ^{−} = ψ|_{W^{−}} and 𝔑^{−} = (ψ^{−})^{−1}(0) ⊂ W^{−} is a null cone. Thus Theorem 2.9 yields the following

Lemma 3.5. Let 𝕆_{l}^{hol} ⊂ N (𝔰) be the theta lift of the regular holomorphic orbit 𝕆′_{l}, which is open dense in 𝔰′_{+}. Then its closure is an affine quotient of W^{+} × 𝔑^{−} by K′_{ℂ},

\overline{𝕆_{l}^{hol}} ≃ W^{+} × 𝔑^{−}//K′_{ℂ}.

▶[Extra Remarks] On the claim that (vector space)×(normal variety) is again normal:

This can be proved as follows (I have learned this from Makoto Matumoto). What should be proved is the following algebraic statement.

Proposition 3.A. Let A be an integrally closed domain over ℂ. Then the polynomial algebra A[x_{1}, …, x_{n}] over A is integrally closed.

To prove this, it is enough to assume that n = 1. The following lemmas are well known.

Lemma 3.B. If A is a discrete valuation ring, then A[x] is a UFD. In particular, A[x] is integrally closed.

Lemma 3.C. If A is an integrally closed domain, then A = ⋂_{𝔭: height 1} A_{𝔭}.

Let K = Q(A) be the quotient field of A. By the first lemma, we know that A_{𝔭}[x] is integrally closed in Q(A_{𝔭})(x). Since the integral closure of A[x] in K(x) is contained in the integral closure of A_{𝔭}[x] in K(x) = Q(A_{𝔭})(x) for any 𝔭, we have

int. closure(A[x]) ⊂ ⋂_{𝔭: height 1} A_{𝔭}[x] = A[x].

This completes the proof of the proposition. ◀

Let us recall the compact group L^{+} whose action on W^{+} commutes with that of K′. In fact we have the dual pair (L^{+}, K′) ⊂ Sp(W_{ℝ}^{+}). Let

ℂ[W^{+}] ≃ \begin{matrix}∑ & ⊕ \\ τ∈Irr(K′_{ℂ};ℂ[W^{+}]) \end{matrix}ρ^{+}(τ) ⊗ τ  (3.4)

be the decomposition of ℂ[W^{+}] as a L^{+} × K′-module. Here ρ^{+}(τ) ∈ Irr(L^{+}) corresponds to τ ∈ Irr(K′) via the above multiplicity-free decomposition. Note that ℂ[W^{+}] ≃ H(K_{ℂ}^{+}) ⊗ ℂ[W^{+}]^{K_{ℂ}^{+}}, and so Irr(K′_{ℂ}; ℂ[W^{+}]) = Irr(K′_{ℂ}; H(K_{ℂ}^{+})) = R(K′_{ℂ})^{∗}. We may therefore rewrite Equation (3.4) as

ℂ[W^{+}] ≃ \begin{matrix}∑ & ⊕ \\ τ∈R(K′_{ℂ}) \end{matrix} ρ^{+}(τ^{∗}) ⊗ τ^{∗}.  (3.5)

Note that L^{+} is a unitary group containing K^{+} in each of the three cases, hence its complexification L_{ℂ}^{+} is isomorphic to a general linear group. Since the action of L_{ℂ}^{+} commutes with that of K′_{ℂ}, L_{ℂ}^{+} × K_{ℂ}^{−} naturally acts on the affine quotient space

W^{+} × 𝔑^{−}//K′_{ℂ} ≃ \overline{𝕆_{l}^{hol}},

extending the action of K_{ℂ} = K_{ℂ}^{+} × K_{ℂ}^{−}.

Remark 3.6. The orbit 𝕆_{l}^{hol} itself does not admit an action of L_{ℂ}^{+} × K_{ℂ}^{−}, but its closure does.

Theorem 3.7. The group L_{ℂ}^{+} × K_{ℂ}^{−} acts on the closure \overline{𝕆_{l}^{hol}} multiplicity-freely. As a L_{ℂ}^{+} × K_{ℂ}^{−}-module, we have

ℂ[\overline{𝕆_{l}^{hol}}] ≃ \begin{matrix}∑ & ⊕ \\ τ∈R(K′_{ℂ}) \end{matrix} ρ^{+}(τ^{∗}) ⊠ σ^{−}(τ).  (3.6)

Proof. We have

ℂ[\overline{𝕆_{l}^{hol}}] ≃ (ℂ[W^{+}] ⊗ ℂ[𝔑^{−}])^{K′_{ℂ}}

≃ ((\begin{matrix}∑ & ⊕ \\ τ_{1}∈R(K′_{ℂ}) \end{matrix} ρ^{+}(τ_{1}^{∗}) ⊗ τ_{1}^{∗}) ⊗ (\begin{matrix}∑ & ⊕ \\ τ_{2}∈R(K′_{ℂ}) \end{matrix} σ^{−}(τ_{2}) ⊗ τ_{2}))^{K′_{ℂ}}

≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix}(τ_{1}^{∗} ⊗ τ_{2})^{K′_{ℂ}} ⊗ (ρ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}))

≃ \begin{matrix}∑ & ⊕ \\ τ∈R(K′_{ℂ}) \end{matrix} ρ^{+}(τ^{∗}) ⊠ σ^{−}(τ).

Corollary 3.8. The orbit 𝕆_{l}^{hol} is L_{ℂ}^{+} × K_{ℂ}^{−}-spherical. The closure \overline{𝕆_{l}^{hol}} is a normal variety.

Proof. The first statement follows directly from the above theorem. The normality follows from [20, Th. 10], since ℤR(K′_{ℂ}) ∩ ℚ_{+}R(K′_{ℂ}) = R(K′_{ℂ}).

Next we consider the K_{ℂ}-module structure of the regular function ring on \overline{𝕆_{l}^{hol}}. Since \overline{𝕆′_{l}} = 𝔰′_{+}, Theorem 2.9 yields

ℂ[\overline{𝕆_{l}^{hol}}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} Hom_{K′_{ℂ}}(τ_{1}, τ_{2} ⊗ ℂ[𝔰′_{+}]) ⊗ (σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}))

≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} Hom_{K′_{ℂ}}(τ_{2}, τ_{1} ⊗ ℂ[𝔰′_{−}]) ⊗ (σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2})).

For any τ_{1}, τ_{2} ∈ Irr(K′), we define the branching coefficient b_{τ_{2}}^{τ_{1}} by the following formula:

τ_{1} ⊗ ℂ[𝔰′_{−}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{2}∈Irr(K′) \end{matrix} b_{τ_{2}}^{τ_{1}} τ_{2}.  (3.7)

Thus we get

ℂ[\overline{𝕆_{l}^{hol}}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} b_{τ_{2}}^{τ_{1}} σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}).  (3.8)

Remark 3.9. Let us denote by D(τ) a holomorphic discrete series representation of G′ with the extreme K′-type τ. Then it is well known that

D(τ)|_{K′} ≃ τ ⊗ ℂ[𝔰′_{−}]

as K′-modules. For any τ_{1} ∈ Irr(K′), there exists a suitable unitary character χ of K′ such that τ = τ_{1} ⊗ χ is the extreme K′-type of a holomorphic discrete series representation.

Then our branching coefficient b_{τ_{2}}^{τ_{1}} satisfies

D(τ_{1} ⊗ χ)|_{K′} ≃ \begin{matrix}∑ & ⊕ \\ τ_{2}∈Irr(K′) \end{matrix} b_{τ_{2}}^{τ_{1}} (τ_{2} ⊗ χ),

and it does not depend on the choice of the unitary character χ. Hence, Equation (3.8) tells us that the orbit structure of 𝕆_{l}^{hol} encodes the K′-type decomposition of (all) holomorphic discrete series of G′.

We shall show that the branching coefficient b_{τ_{2}}^{τ_{1}} may be expressed in terms of the branching coefficient of finite dimensional representations.

For τ_{1}, τ_{2} ∈ R(K′_{ℂ}), define the branching coefficient c_{τ_{1}}^{τ_{2}} by the following restriction formula:

ρ^{+}(τ_{2}^{∗})|_{K_{ℂ}^{+}} ≃ \begin{matrix}∑ & ⊕ \\ τ_{1}∈R(K′_{ℂ}) \end{matrix} c_{τ_{1}}^{τ_{2}} σ^{+}(τ_{1}^{∗}).  (3.9)

Theorem 3.10. If we define the branching coefficient b_{τ_{2}}^{τ_{1}} and c_{τ_{1}}^{τ_{2}} by (3.7) and (3.9) respectively, then

b_{τ_{2}}^{τ_{1}} = c_{τ_{1}}^{τ_{2}}, ∀τ_{1}, τ_{2} ∈ R(K′_{ℂ}).

Consequently we have

ℂ[\overline{𝕆_{l}^{hol}}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} b_{τ_{2}}^{τ_{1}} σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}) ≃ \begin{matrix}∑ & ⊕ \\ τ_{1},τ_{2}∈R(K′_{ℂ}) \end{matrix} c_{τ_{1}}^{τ_{2}} σ^{+}(τ_{1}^{∗}) ⊠ σ^{−}(τ_{2}).  (3.10)

Proof. From Equation (3.5), we have

ℂ[W^{+}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{2}∈R(K′_{ℂ}) \end{matrix} ρ^{+}(τ_{2}^{∗}) ⊗ τ_{2}^{∗}

≃ \begin{matrix}∑ & ⊕ \\ τ_{1}∈R(K′_{ℂ}) \end{matrix} σ^{+}(τ_{1}^{∗}) ⊠ (\begin{matrix}∑ & ⊕ \\ τ_{2}∈R(K′_{ℂ}) \end{matrix} c_{τ_{1}}^{τ_{2}} τ_{2}^{∗}) (as K_{ℂ}^{+} × K′_{ℂ}-module).

Comparing this with

ℂ[W^{+}] ≃ H(K_{ℂ}^{+}) ⊗ ℂ[𝔰′_{+}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{1}∈R(K′_{ℂ}) \end{matrix} σ^{+}(τ_{1}^{∗}) ⊗ (τ_{1} ⊗ ℂ[𝔰′_{−}])^{∗},

we get

τ_{1} ⊗ ℂ[𝔰′_{−}] ≃ \begin{matrix}∑ & ⊕ \\ τ_{2}∈R(K′_{ℂ}) \end{matrix} c_{τ_{1}}^{τ_{2}} τ_{2}, i.e., b_{τ_{2}}^{τ_{1}} = c_{τ_{1}}^{τ_{2}}.

4. Appendix

In Appendix, we will give an explicit construction of moment maps φ and ψ, and establish some fundamental properties of null cones and harmonics.

4.1. O(p, q) × Sp(2n, ℝ) (2n < min (p, q)). This case is already treated in [12]. For convenience of the readers, we reproduce the synopsis of the arguments, and at the same time we improve some of statements there.

Put G = O(p, q) and G′ = Sp(2n, ℝ), and assume the stable range condition 2n < p, q. We denote:

M = U(p, q)  ⊃  G = O(p, q)
∪  ∪
L = L^{+} × L^{−} = U(p) × U(q) ⊃ K = K^{+} × K^{−} = O(p) × O(q)

M′ = Sp(2n, ℝ)^{2}  ⊃^{Δ} G′ = Sp(2n, ℝ)
∪  ∪
L′ = K′ × K′ = U(n)^{2} ⊃^{Δ} K′ = U(n)

In this diagram, the vertical containment means respectively maximal compact subgroups, and Δ denotes the diagonal embedding. Put

W = M_{p+q,n}(ℂ) = \{(\begin{matrix}A \\ B \end{matrix}) | A ∈ M_{p,n}(ℂ), B ∈ M_{q,n}(ℂ)\}
= W^{+} ⊕ W^{−} = M_{p,n}(ℂ) ⊕ M_{q,n}(ℂ).

The complexification K_{ℂ} = O(p, ℂ) × O(q, ℂ) acts on W via the left multiplication, and K′_{ℂ} = GL_{n}(ℂ) acts on W as

g · (\begin{matrix}A \\ B \end{matrix}) = (\begin{matrix}A^{t}g \\ Bg^{−1} \end{matrix}) , g ∈ GL_{n}(ℂ).

We fix a Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔰 (resp. 𝔤′ = 𝔨′ ⊕ 𝔰′) as

𝔤 = 𝔬(p + q, ℂ) = ( \begin{matrix}Alt_{p}(ℂ) & 0 \\ 0 & Alt_{q}(ℂ) \end{matrix}) ⊕ ( \begin{matrix}0 & M_{p,q}(ℂ) \\ ^{t}M_{p,q}(ℂ) & 0 \end{matrix}) = 𝔨 ⊕ 𝔰,

𝔤′ = 𝔰𝔭(2n, ℂ) = (\begin{matrix}M_{n}(ℂ) & 0 \\ 0 & −^{t}M_{n}(ℂ) \end{matrix}) ⊕ ( \begin{matrix}0 & Sym_{n}(ℂ) \\ Sym_{n}(ℂ) & 0 \end{matrix}) = 𝔨′ ⊕ 𝔰′.

Thus, we can identify 𝔰 = M_{p,q}(ℂ) and 𝔰′ = 𝔰′_{+} ⊕𝔰′_{−} = Sym_{n}(ℂ)⊕Sym_{n}(ℂ). The moment maps are explicitly given by

φ : W ∋ (\begin{matrix}A \\ B \end{matrix}) ⟼ A^{t}B ∈ 𝔰, ψ : W ∋ (\begin{matrix}A \\ B \end{matrix}) ⟼ (^{t}AA, ^{t}BB) ∈ 𝔰′.

These maps are clearly K_{ℂ} × K′_{ℂ}-equivariant if we define the trivial K_{ℂ}-action (resp. K′_{ℂ}-action) on 𝔰′ (resp. 𝔰). In the language of algebra, φ is induced by

φ^{∗}(z_{i,j}) = ∑_{k=1}^{n} a_{i,k}b_{j,k}, Z = (z_{i,j}) ∈ 𝔰, (\begin{matrix}A \\ B \end{matrix}) = (\begin{matrix}(a_{i,j}) \\ (b_{i,j}) \end{matrix}) ∈ W,

and it is well known that φ^{∗}(z_{i,j})’s generate GL_{n}-invariants ℂ[W]^{K′_{ℂ}}. This shows that φ : W → 𝔰 is an affine quotient map onto its closed image:

𝔰 ⊃ φ(W) ≃ W//K′_{ℂ}.

Similarly, ψ is induced by

ψ^{∗}(x_{i,j}) = ∑_{k=1}^{p} a_{k,i}a_{k,j}, X = (x_{i,j}) ∈ 𝔰′_{+}, ψ^{∗}(y_{i,j}) = ∑_{l=1}^{q} b_{l,i}b_{l,j}, Y = (y_{i,j}) ∈ 𝔰′_{−}.

Again, ψ(x_{i,j})^{∗}’s and ψ(y_{i,j})^{∗}’s generate the O(p, ℂ) × O(q, ℂ)-invariants ℂ[W]^{K_{ℂ}}, which proves that ψ : W → 𝔰′ is an affine quotient map. By the assumption of the stable range, 2n < p, q, it is easy to see that ψ : W → 𝔰′ is a surjection, hence

𝔰′ ≃ W//K_{ℂ},
ℂ[𝔰′] = ℂ[Sym_{n}] ⊗ ℂ[Sym_{n}] ≃ ℂ[M_{p+q,n}]^{O(p,ℂ)×O(q,ℂ)} = ℂ[W]^{K_{ℂ}}.

Let us consider the null cones

𝔑 = ψ^{−1}(0) ⊂ W, 𝔑^{±} = (ψ^{±})^{−1}(0) ⊂ W^{±},

where ψ^{±} : W^{±} → 𝔰′_{±} is the restriction of ψ to W^{±}.

Proposition 4.1. Assume the stable range condition 2n < p, q.

(1) The null cone 𝔑 = 𝔑^{+} × 𝔑^{−} is an irreducible normal variety, which is of complete intersection.

(2) There are finitely many K_{ℂ} × K′_{ℂ}-orbits in 𝔑, which are completely classified by the ranks of each component in 𝔑^{±}. Among them, there is an open dense K_{ℂ} × K′_{ℂ}-orbit O_{n,n}, which is a single K_{ℂ}-orbit. Moreover, the singular locus coincides with 𝔑\O_{n,n}, which is of codimension ≥ 2.

(3) The regular function ring ℂ[𝔑] is isomorphic to the harmonics H(K_{ℂ}) as a K_{ℂ}×K′_{ℂ}- module. It is explicitly given as

ℂ[𝔑^{+}] ≃ H^{+}(K_{ℂ}^{+}) ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} σ_{λ}^{(p)}^{∗} ⊗ τ_{λ}^{(n)}^{∗},

ℂ[𝔑^{−}] ≃ H^{−}(K_{ℂ}^{−}) ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} σ_{λ}^{(q)} ⊗ τ_{λ}^{(n)},
ℂ[𝔑] ≃ H(K_{ℂ}) ≃ H^{+}(K_{ℂ}^{+}) ⊗ H^{−}(K_{ℂ}^{−}),

where P_{n} denotes the set of all the partitions of length ≤ n, and τ_{λ}^{(n)} (resp. σ_{λ}^{(p)}) is an irreducible finite dimensional representation of GL_{n} (resp. O(p, ℂ)) of highest weight λ.

Remark 4.2. (1) In [12], the orbit decomposition of the null cone is carried out for K_{ℂ} × L′_{ℂ}, though it is more natural to consider K_{ℂ} × K′_{ℂ}-orbits. In fact, their orbits are the same (under the assumption of stable range). Also, in [12], the normality is proved when 2n + 2 ≤ p, q. Here we improve it, and there is no more restriction other than the stable range condition.

(2) It is rather subtle to specify the representation σ_{λ}^{(p)} because O(p, ℂ) is not connected. For this, see [8, §3.6.2].

Proof. It is straightforward to check the claim for the K_{ℂ} × K′_{ℂ}-orbit decomposition in (2) directly by calculation. Let us denote

O_{r}^{+} = {A ∈ M_{p,n} | ^{t}AA = 0, rank A = r} ⊂ 𝔑^{+},
O_{s}^{−} = {B ∈ M_{q,n} | ^{t}BB = 0, rank B = s} ⊂ 𝔑^{−},
O_{r,s} = O_{r}^{+} × O_{s}^{−} ∈ 𝔑.

Then O_{r}^{+} is a K_{ℂ}^{+}× K′_{ℂ}-orbit, and O_{r,s} is a single K_{ℂ}× K′_{ℂ}-orbit. Since they are classified by rank, the orbit of the largest possible rank O_{n,n} is open dense in 𝔑. This implies that 𝔑 = \overline{O_{n,n}} is irreducible. In fact, O_{n,n} is a single K_{ℂ}-orbit. This follows from the Witt theorem because the orbit is of full rank.

Explicit calculation of the rank of differentials of the defining equations of ψ^{∗}(x_{ij})’s and ψ^{∗}(y_{ij})’s tells us that a point from 𝔑\O_{n,n} is singular. Since

dim 𝔑^{+} = np − \frac{n(n + 1)}{2}, dim 𝔑^{−} = nq − \frac{n(n + 1)}{2},

we have codim 𝔑 = n(n+1), which is equal to the number of the defining equations. This proves that 𝔑 is of complete intersection.

The dimension of the K_{ℂ}^{+} × K′_{ℂ}-orbit in 𝔑^{+} of rank r is given by

dim O_{r}^{+} = r(p + n) − r^{2} − \frac{r(r + 1)}{2}.

From this formula, we can easily show that codim (𝔑\O_{n,n}) ≥ 2.

By Kostant theorem (see [2, Th. 2.2.11], for example), we know that the defining radical ideal of 𝔑 is generated by ψ^{∗}(x_{ij})’s and ψ^{∗}(y_{ij})’s, which are basic invariants of O(p, ℂ) × O(q, ℂ). This means that

ℂ[𝔑] = ℂ[W]/(ψ^{∗}(x_{ij}), ψ^{∗}(y_{ij}))
= ℂ[W]/ℂ[W] · ℂ[W]_{+}^{K_{ℂ}} ≃ H(K_{ℂ}).

The decomposition of H(K_{ℂ}) is given in [8, Th. 3.7.3.1 & Cor. 3.7.3.6]. See also [3] and [10]. By the same theorem of Kostant, the null cone 𝔑 is normal. Also, 𝔑^{±} are normal varieties.

4.2. U(p, q) × U(m, n) (m + n ≤ min (p, q)). Put G = U(p, q) and G′ = U(m, n), and assume the stable range condition m + n ≤ p, q. We denote:

M = U(p, q)^{2}  ⊃^{Δ}  G = U(p, q)
∪  ∪
L = L^{+} × L^{−} = U(p)^{2} × U(q)^{2} ⊃^{Δ} K = K^{+} × K^{−} = U(p) × U(q)

Here Δ denotes the diagonal embedding. Similarly, we denote M′, G′, L′, K′ replacing p and q by m and n respectively.

Put

W = M_{p+q,m+n}(ℂ) = \{(\begin{matrix}A & B \\ C & D \end{matrix}) | A ∈ M_{p,m}, B ∈ M_{p,n}, C ∈ M_{q,m}, D ∈ M_{q,n}\}
= W^{+} ⊕ W^{−} = M_{p,m+n}(ℂ) ⊕ M_{q,m+n}(ℂ).

Then K_{ℂ} = GL_{p}(ℂ) × GL_{q}(ℂ) and K′_{ℂ} = GL_{m}(ℂ) × GL_{n}(ℂ) act on W as

((g_{1}, g_{2}), (h_{1}, h_{2})) · (\begin{matrix}A & B \\ C & D \end{matrix}) = ( \begin{matrix}g_{1}A^{t}h_{1} & ^{t}g_{1}^{−1}Bh_{2}^{−1} \\ ^{t}g_{2}^{−1}Ch_{1}^{−1} & g_{2}D^{t}h_{2} \end{matrix}) ,
(g_{1}, g_{2}) ∈ GL_{p}(ℂ) × GL_{q}(ℂ), (h_{1}, h_{2}) ∈ GL_{m}(ℂ) × GL_{n}(ℂ).

We fix a Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔰 as

𝔤 = 𝔤𝔩_{p+q}(ℂ) = (\begin{matrix}M_{p}(ℂ) & 0 \\ 0 & M_{q}(ℂ) \end{matrix}) ⊕ (\begin{matrix}0 & M_{p,q}(ℂ) \\ M_{q,p}(ℂ) & 0 \end{matrix}) = 𝔨 ⊕ 𝔰.

The Cartan decomposition 𝔤′ = 𝔨′ ⊕ 𝔰′ is chosen similarly. We identify

𝔰 = M_{p,q}(ℂ) ⊕ M_{q,p}(ℂ), 𝔰′ = M_{m,n}(ℂ) ⊕ M_{n,m}(ℂ).

The moment maps φ and ψ are explicitly given by

φ : W ∋ (\begin{matrix}A & B \\ C & D \end{matrix}) ⟼ (A^{t}C, D^{t}B) ∈ 𝔰, ψ : W ∋ (\begin{matrix}A & B \\ C & D \end{matrix}) ⟼ (^{t}AB, ^{t}DC) ∈ 𝔰′.

These maps are K_{ℂ} × K′_{ℂ}-equivariant with the trivial K_{ℂ}-action on 𝔰′, and the trivial K′_{ℂ}-action on 𝔰 respectively. The maps φ and ψ are almost the same. Therefore we will treat only the map ψ in the following.

The map ψ induces an algebra homomorphism ψ : ℂ[𝔰′] → ℂ[W] by

ψ^{∗}(x_{i,j}) = ∑_{k=1}^{p} a_{k,i}b_{k,j}, X = (x_{i,j}) ∈ M_{m,n}(ℂ),

ψ^{∗}(y_{i,j}) = ∑_{l=1}^{q} c_{l,j}d_{l,i}, Y = (y_{i,j}) ∈ M_{n,m}(ℂ).

GL_{p}-invariants (resp. GL_{q}-invariants) on ℂ[W^{+}] (resp. ℂ[W^{−}]) are generated by ψ(x_{i,j})^{∗}’s (resp. ψ(y_{i,j})^{∗}’s). Hence ψ : W → 𝔰′ is an affine quotient map by K_{ℂ}, which is surjective under the condition of the stable range. Note that φ is not surjective in general.

Let us consider the null cones

𝔑 = ψ^{−1}(0) ⊂ W, 𝔑^{±} = (ψ^{±})^{−1}(0) ⊂ W^{±},

where ψ^{±} : W^{±} → 𝔰′_{±} is the restriction of ψ to W^{±}.

Proposition 4.3. Assume the stable range condition m + n ≤ min(p, q).

(1) The null cone 𝔑 = 𝔑^{+} × 𝔑^{−} is an irreducible normal variety, which is of complete intersection.

(2) There are finitely many K_{ℂ} × K′_{ℂ}-orbits in 𝔑, which are completely classified by the ranks of each component in 𝔑^{±}. Among them, there is an open dense K_{ℂ} × K′_{ℂ}-orbit O_{m,n}, which is a single K_{ℂ}-orbit. Moreover, the singular locus is of codimension ≥ 2.

(3) The regular function ring ℂ[𝔑] is isomorphic to the harmonics H(K_{ℂ}) as a K_{ℂ}×K′_{ℂ}- module. It is explicitly given as

ℂ[𝔑^{+}] ≃ H^{+}(K_{ℂ}^{+}) ≃ \begin{matrix}∑ & ⊕ \\ α∈P_{m},β∈P_{n} \end{matrix} τ_{α⊙β}^{(p)}^{∗} ⊗ (τ_{α}^{(m)}^{∗} ⊗ τ_{β}^{(n)}),

ℂ[𝔑^{−}] ≃ H^{−}(K_{ℂ}^{−}) ≃ \begin{matrix}∑ & ⊕ \\ γ∈P_{m},δ∈P_{n} \end{matrix} τ_{γ⊙δ}^{(q)} ⊗ (τ_{γ}^{(m)} ⊗ τ_{δ}^{(n)}^{∗}),
ℂ[𝔑] ≃ H(K_{ℂ}) ≃ H^{+}(K_{ℂ}^{+}) ⊗ H^{−}(K_{ℂ}^{−}),

where, for partitions α ∈ P_{m} and β ∈ P_{n}, we denote

α ⊙ β = α ⊙_{p} β = (α_{1}, α_{2}, …, α_{m}, 0, …, 0, −β_{n}, …, −β_{1}) ∈ ℤ^{p}

and τ_{λ}^{(p)} is an irreducible finite dimensional representation of GL_{p} of highest weight λ.

Proof. Put

O_{r,s}^{+} = {(A, B) ∈ M_{p,m+n} | ^{t}AB = 0, rank A = r, rank B = s} ⊂ 𝔑^{+},

Then O_{r,s}^{+} is a K_{ℂ}^{+} × K′_{ℂ}-orbit, and

dim O_{r,s}^{+} = r(m + p) + s(n + p) + rs − (r + s)^{2}.

It is easy to see that O_{m,n}^{+} ⊂ 𝔑^{+} is an open dense orbit and

dim 𝔑^{+} = dim O_{m,n}^{+} = p(m + n) − mn,

which means 𝔑^{+} is of complete intersection. The singular locus is given by

\overline{O_{m−1,n−1}^{+}} = ∐_{r≤m−1,s≤n−1} O_{r,s}^{+} ⊂ 𝔑^{+},

which is of codimension ≥ 2. The rest of the proof is similar to that of Proposition 4.1. The decomposition of harmonics is given in [8, Th. 2.5.4].

4.3. Sp(p, q) × O^{∗}(2n) (n ≤ min (p, q)). Put G = Sp(p, q) and G′ = O^{∗}(2n), and assume the stable range condition n ≤ p, q. We denote:

M = U(2p, 2q)  ⊃  G = Sp(p, q)
∪  ∪
L = L^{+} × L^{−} = U(2p) × U(2q) ⊃ K = K^{+} × K^{−} = Sp(p) × Sp(q)

M′ = O^{∗}(2n)^{2}  ⊃^{Δ} G′ = O^{∗}(2n)
∪  ∪
L′ = K′ × K′ = U(n)^{2} ⊃^{Δ} K′ = U(n)

Put

W = M_{2p+2q,n}(ℂ) = \{(\begin{matrix}A \\ B \end{matrix}) | A ∈ M_{2p,n}(ℂ), B ∈ M_{2q,n}(ℂ)\}
= W^{+} ⊕ W^{−} = M_{2p,n}(ℂ) ⊕ M_{2q,n}(ℂ).

The complexification K_{ℂ} = Sp(2p, ℂ) × Sp(2q, ℂ) acts on W as

(k_{1}, k_{2}) · (\begin{matrix}A \\ B \end{matrix}) = ( \begin{matrix}k_{1}A \\ ^{t}k_{2}^{−1}B \end{matrix}) , (k_{1}, k_{2}) ∈ Sp(2p, ℂ) × Sp(2q, ℂ),

and K′_{ℂ} = GL_{n}(ℂ) acts on W as

g · (\begin{matrix}A \\ B \end{matrix}) = (\begin{matrix}A^{t}g \\ Bg^{−1} \end{matrix}) , g ∈ GL_{n}(ℂ).

We fix Cartan decompositions 𝔤 = 𝔨 ⊕ 𝔰 and 𝔤′ = 𝔨′ ⊕ 𝔰′ as

𝔤 = 𝔰𝔭(2p + 2q, ℂ) = (\begin{matrix}𝔰𝔭(2p, ℂ) & 0 \\ 0 & 𝔰𝔭(2q, ℂ) \end{matrix}) ⊕ (\begin{matrix}0 & M_{2p,2q} \\ J_{q} ^{t}M_{2p,2q}J_{p} & 0 \end{matrix}) ,

= 𝔨 ⊕ 𝔰, J_{p} = (\begin{matrix}0 & 1_{p} \\ −1_{p} & 0 \end{matrix})

𝔤′ = 𝔬(2n, ℂ) = (\begin{matrix}M_{n}(ℂ) & 0 \\ 0 & −^{t}M_{n}(ℂ) \end{matrix}) ⊕ ( \begin{matrix}0 & Alt_{n}(ℂ) \\ Alt_{n}(ℂ) & 0 \end{matrix}) = 𝔨′ ⊕ 𝔰′.

Thus, we can identify 𝔰 = M_{2p,2q}(ℂ) and 𝔰′ = 𝔰′_{+} ⊕𝔰′_{−} = Alt_{n}(ℂ)⊕Alt_{n}(ℂ). The moment maps are explicitly given by

φ : W ∋ (\begin{matrix}A \\ B \end{matrix}) ⟼ A^{t}B ∈ 𝔰, ψ : W ∋ (\begin{matrix}A \\ B \end{matrix}) ⟼ (^{t}AJ_{p}A, ^{t}BJ_{q}B) ∈ 𝔰′.

These maps are clearly K_{ℂ} × K′_{ℂ}-equivariant with the trivial K_{ℂ}-action on 𝔰′, and the trivial K′_{ℂ}-action on 𝔰 respectively. The moment maps induce algebra morphisms φ^{∗} and ψ^{∗}:

φ^{∗}(z_{i,j}) = ∑_{k=1}^{n} a_{i,k}b_{j,k}, Z = (z_{i,j}) ∈ M_{2p,2q}, A = (a_{i,j}) ∈ M_{2p,n}, B = (b_{i,j}) ∈ M_{2q,n},

ψ^{∗}(x_{i,j}) = − ∑_{k=1}^{p} a_{k+p,i}a_{k,j} + ∑_{k=1}^{p} a_{k,i}a_{k+p,j}, X = (x_{i,j}) ∈ Alt_{n},

ψ^{∗}(y_{i,j}) = − ∑_{l=1}^{q} b_{l+q,i}b_{l,j} + ∑_{l=1}^{q} b_{l,i}b_{l+q,j}, Y = (y_{i,j}) ∈ Alt_{n}.

Classical invariant theory tells us that φ^{∗}(z_{i,j})’s generate the GL_{n}-invariants ℂ[W]^{K′_{ℂ}}, and that ψ(x_{i,j})^{∗}’s (resp. ψ(y_{i,j})^{∗}’s) generate the K_{ℂ}^{+} = Sp(2p, ℂ)-invariants ℂ[W^{+}]^{K_{ℂ}^{+}} (resp. K_{ℂ}^{−} = Sp(2q, ℂ)-invariants ℂ[W^{−}]^{K_{ℂ}^{−}}). This shows that φ and ψ are affine quotient maps. By the assumption of the stable range, n ≤ p, q, the map ψ : W → 𝔰′ is a surjection, and we have

ℂ[𝔰′] ≃ ℂ[W]^{K_{ℂ}}, ℂ[𝔰′_{±}] ≃ ℂ[W^{±}]^{K_{ℂ}^{±}}.

Let us define the null cones by

𝔑 = ψ^{−1}(0) ⊂ W, 𝔑^{±} = (ψ^{±})^{−1}(0) ⊂ W^{±},

where ψ^{±} : W^{±} → 𝔰′_{±} is the restriction of ψ to W^{±}.

Proposition 4.4. Assume the stable range condition n ≤ p, q.

(1) The null cone 𝔑 = 𝔑^{+} × 𝔑^{−} is an irreducible normal variety, which is of complete intersection.

(2) There are finitely many K_{ℂ} × K′_{ℂ}-orbits in 𝔑, which are completely classified by the ranks of each component in 𝔑^{±}. The full rank orbit O_{n,n} is an open dense K_{ℂ}× K′_{ℂ}-orbit in 𝔑, which is also a single K_{ℂ}-orbit. The singular locus of 𝔑 is of codimension ≥ 2.

(3) The regular function ring ℂ[𝔑] is isomorphic to the harmonics H(K_{ℂ}) as a K_{ℂ}×K′_{ℂ}- module, and we have

ℂ[𝔑^{+}] ≃ H^{+}(K_{ℂ}^{+}) ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} σ_{λ}^{(p)}^{∗} ⊗ τ_{λ}^{(n)}^{∗},

ℂ[𝔑^{−}] ≃ H^{−}(K_{ℂ}^{−}) ≃ \begin{matrix}∑ & ⊕ \\ λ∈P_{n} \end{matrix} σ_{λ}^{(q)} ⊗ τ_{λ}^{(n)},
ℂ[𝔑] ≃ H(K_{ℂ}) ≃ H^{+}(K_{ℂ}^{+}) ⊗ H^{−}(K_{ℂ}^{−}),

where σ_{λ}^{(p)} denotes an irreducible finite dimensional representation of Sp(2p, ℂ) with highest weight λ ∈ P_{n}.

Proof. Put

O_{r}^{+} = {A ∈ M_{2p,n} | ^{t}AJ_{p}A = 0, rank A = r} ⊂ 𝔑^{+},
O_{s}^{−} = {B ∈ M_{2q,n} | ^{t}BJ_{q}B = 0, rank B = s} ⊂ 𝔑^{−}.

Then O_{r,s} = O_{r}^{+} × O_{s}^{−} is a single K_{ℂ} × K′_{ℂ}-orbit and

dim O_{r}^{+} = r(2p + n) − r^{2} − \frac{r(r − 1)}{2}, dim O_{s}^{+} = s(2q + n) − s^{2} − \frac{s(s − 1)}{2}.

It is easy to show that O_{n}^{±} ⊂ 𝔑^{±} is an open dense orbit and

dim 𝔑^{+} = dim O_{n}^{+} = 2pn − \frac{n(n − 1)}{2}, dim 𝔑^{−} = dim O_{n}^{−} = 2qn − \frac{n(n − 1)}{2},

which tells us that 𝔑 (resp. 𝔑^{±}) is of complete intersection. The singular locus is given by

∐_{r≤n−2} O_{r}^{±} ⊂ 𝔑^{±},

which is of codimension ≥ 2. The rest of the proof is similar to that of Proposition 4.1. The structure of harmonics is given in [8, Th. 3.8.6.2].

References

[1] Walter Borho and Hanspeter Kraft, Über Bahnen und deren Deformationen bei linearen Aktionen reduktiver Gruppen. Comment. Math. Helv. 54 (1979), no. 1, 61–104.

[2] N. Chriss and V. Ginzburg, Representation theory and complex geometry, Birkhäuser Boston, Boston, MA, 1997 ISBN: 0-8176-3792-3.

[3] Stephen S. Gelbart, A theory of Stiefel harmonics. Trans. Amer. Math. Soc. 192 (1974), 29–50.

[4] Robin Hartshorne, Algebraic geometry. Graduate Texts in Mathematics, No. 52. Springer-Verlag, New York-Heidelberg, 1977.

[5] R. Howe, θ-series and invariant theory, Proc. Symp. Pure Math. 33, Ameri. Math. Soc., Providence, 1979.

[6] R. Howe, Remarks on classical invariant theory. Trans. Amer. Math. Soc. 313 (1989), no. 2, 539– 570. Erratum to: ”Remarks on classical invariant theory”. Trans. Amer. Math. Soc. 318 (1990), no. 2, 823.

[7] R. Howe, Transcending classical invariant theory. J. Amer. Math. Soc. 2 (1989), no. 3, 535–552.

[8] R. Howe, Perspectives on invariant theory: Schur duality, multiplicity-free actions and beyond. in The Schur lectures (1992) (Tel Aviv), 1–182, Bar-Ilan Univ., Ramat Gan, 1995.

[9] J.-S. Huang and J.-S. Li, Unipotent representations attached to spherical nilpotent orbits, Amer. Jour. Math. 121 (1999), no. 3, 497–517.

[10] M. Kashiwara and M. Vergne, On the Segal-Shale-Weil representations and harmonic polynomials, Invent. Math. 44 (1978), no. 1, 1–47.

[11] J.-S. Li, Singular unitary representations of classical groups, Invent. Math., 97 (1989), 237 – 255.

[12] Kyo Nishiyama, Multiplicity-free actions and the geometry of nilpotent orbits. Math. Ann., 318 (2000), 777 – 793.

[13] Kyo Nishiyama, Theta lifting of two-step nilpotent orbits for the pair O(p, q) × Sp(2n, ℝ). In “Infinite Dimensional Harmonic Analysis”, Transaction of a Japanese-German Symposium (Kyoto, September 1999), Kyoto 1999, pp. 278 – 289.

[14] Kyo Nishiyama, Hiroyuki Ochiai, and Kenji Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules—Hermitian symmetric case. Astérisque No. 273, (2001), 13–80.

[15] Kyo Nishiyama and C.-B. Zhu, Theta lifting of holomorphic discrete series, (The case of U(p, q)× U(n, n)). Trans. AMS. 353 (2001), 3327-3345.

[16] Takuya Ohta, Nilpotent orbits of Z_{4}-graded Lie algebra and geometry of the moment maps associated to the dual pair (U(p, q), U(r, s)), preprint.

[17] Takuya Ohta, Nilpotent orbits of Z_{4}-graded Lie algebra and geometry of the moment maps associated to the dual pairs (O(p, q), Sp(2n, ℝ)) and (O^{∗}(2n), Sp(p, q)), in preparation.

[18] V. L. Popov and E. B. Vinberg, Invariant theory, in “Algebraic geometry. IV.”, Encyclopaedia of Mathematical Sciences, 55. Springer-Verlag, Berlin, 1994, pp. 123–284.

[19] T. Przebinda, Characters, dual pairs, and unitary representations, Duke Math. J. 69 (1993), no. 3, 547-594.

[20] E. B. Vinberg and V. L. Popov, A certain class of quasihomogeneous affine varieties. Izv. Akad. Nauk SSSR Ser. Mat. 36 (1972), 749–764.

[21] H. Weyl, The classical groups, Princeton University Press, 1946.

Faculty of IHS, Kyoto University, Sakyo, Kyoto 606-8501, Japan

E-mail address: kyo@math.h.kyoto-u.ac.jp

Department of Mathematics, National University of Singapore, 10 Kent Ridge Crescent, Singapore 119260

E-mail address: matzhucb@nus.edu.sg