Theta lifting of the trivial representation and the associated nilpotent orbit

— the case of U(p, q) × U(n, n) — ∗

Kyo Nishiyama  Chen-bo Zhu

Compiled on December 20, 1999, Ver. 0.3 (Wed Oct 6 17:41:24 JST 1999)

Contents

1 Some small nilpotent orbits in U(n, n) 2 2 Theta lift of the trivial representation of U(p, q) 7 3 Degree of orbits O_{p,q} 9 4 Nullcone and the geometry of nilpotent orbits 11 5 Generalities on Harish-Chandra modules 16 6 Theta lift of a holomorphic discrete series 16 7 Further results 19

∗A note for “Proceedings of Symposium on Representation Theory 1999” in Tateyama, Chiba.

Introduction

In [10], Ochiai, Taniguchi and one of the authors calculate the associated cycles and Bernstein degrees for certain singular unitary representations. Their method uses the theory of dual pairs (or theta correspondence), though it is only applicable to pairs of compact type (i.e., one member of the pair is a compact group). In this note, we consider representations arising from a dual pair of non-compact type, and our purpose is to establish a generalization of the results in [10].

One problem we encounter here is that the theta correspondence for a non-compact pair is not completely understood, in spite of the pioneer work of Howe, and subsequently excellent works of many people such as Adams, Li, Moeglin, Przebinda. Another problem here is the lack of information on the unitary representations of non-compact groups which we try to lift. In the case of compact Lie groups, unitary representations are well-understood and we can do many computations in full details thanks to Cartan-Weyl theory and Frobenius reciprocity. For non-compact groups, such information on unitary representations are not readily available, let alone their classifications.

Therefore, in this note, we shall restrict ourselves to theta lifts of well-known unitary representations such as the trivial representation and holomorphic discrete series representations. To present the ideas clearly, we shall also fix our dual pair as

U(p, q) × U(n, n) (p + q ≤ n).

The condition p+q ≤ n ensures that the pair is in the stable range with U(p, q) as the smaller member. We take an irreducible unitary representation of the smaller group U(p, q) (to be more precise, the metaplectic covering group of U(p, q)), and investigate its theta lift. In the cases we consider, it turns out that the associated variety of the theta lift corresponds to that of the original representation through a “complex moment map”, as expected. Moreover, the associated cycles seem to behave well under this correspondence. These examples strongly suggest that the conjectured formula of associated cycles in [10] is true in general.

The first author would like to thank the National University of Singapore for the kind and warm hospitality during his stay in July 1999, where the first draft of this note was written. We also thank Jian-Shu Li for valuable conversations in NUS.

1 Some small nilpotent orbits in U(n, n)

Let G = U(n, n) ⊃ K = U(n) × U(n), with G_{ℂ} = GL_{2n} and K_{ℂ} = GL_{n} × GL_{n}. We denote by 𝔤 and 𝔨 the complexified Lie algebras of G and K respectively. Let 𝔤 = 𝔨 ⊕ 𝔰 be a Cartan decomposition, and 𝔰 = 𝔰^{+} ⊕ 𝔰^{−} be the K_{ℂ}-stable decomposition of 𝔰.

We begin with some elementary linear algebra. Write a typical element of 𝔰 as

X = [ \begin{matrix}0 & A \\ B & 0 \end{matrix} ] ∈ 𝔰,

where A ∈ 𝔰^{+} = M_{n}(ℂ), B ∈ 𝔰^{−} = M_{n}(ℂ).

Since

X^{2} = [ \frac{AB}{BA} ] ,

we see that X is nilpotent if and only if AB and BA are both nilpotent. In fact, this is equivalent to say that AB or BA is nilpotent.

Note that for (g, h) ∈ K_{ℂ} = GL_{n} × GL_{n}, we have

Ad (diag (g, h)) [ \begin{matrix}0 & A \\ B & 0 \end{matrix} ] = [ \begin{matrix}0 & gAh^{−1} \\ hBg^{−1} & 0 \end{matrix} ] =^{put} [ \begin{matrix}0 & A′ \\ B′ & 0 \end{matrix} ] .

Thus the K_{ℂ}-orbit through X, denoted by O_{X}, has invariants rank A and rank B. Further we have

\{ \begin{matrix}A′B′ = gABg^{−1}, \\ B′A′ = hBAh^{−1}. \end{matrix}

Therefore AB and BA naturally define GL_{n}-conjugacy classes in M_{n}(ℂ).

Let 𝔛_{null} be the cone consisting of those nilpotent elements X in 𝔰 such that X^{2} = 0, namely

𝔛_{null} = \{X = [ \begin{matrix}0 & A \\ B & 0 \end{matrix} ] ∈ 𝔰 | AB = 0, BA = 0\} .

Put

x_{p,q} = [ \frac{\begin{aligned}1_{p} \\ & 0 \end{aligned}}{\begin{aligned}0 \\ & 1_{q} \end{aligned}} ] ∈ 𝔛_{null} (p + q ≤ n).  (1.1)

Denote by O_{p,q} = K_{ℂ} · x_{p,q} the adjoint K_{ℂ}-orbit through x_{p,q}.

Lemma 1.1 (1) We have

\overline{O_{p,q}} ⊂ 𝔛_{null},

where \overline{O_{p,q}} denotes the closure of the orbit O_{p,q}. Moreover, we have the decomposition

𝔛_{null} = ∐_{p+q≤n} O_{p,q}.

(2) O_{p,q} and O_{r,s} generate the same G_{ℂ}-orbit if and only if p + q = r + s. Moreover, we have

dim O_{p,q} = (p + q)(2n − (p + q)), \overline{O_{p,q}} = ∐_{p′≤p, q′≤q} O_{p′,q′}.

Proof. (1) It is clear that O_{p,q} ⊂ 𝔛_{null} and 𝔛_{null} is closed. Hence we have \overline{O_{p,q}} ⊂ 𝔛_{null}. Now take an arbitrary element of 𝔛_{null}. By replacing it by a K_{ℂ}-conjugate, we can assume that the element is of the form

[ \begin{matrix}0 & A \\ B & 0 \end{matrix} ] ∈ 𝔛_{null}, A = [ \begin{matrix}1_{p} \\ & 0 \end{matrix} ] .

By the assumption that AB = 0, BA = 0, we must have

B = [ \begin{matrix}0 \\ & B′ \end{matrix} ] .

Since elements of the form

(g, h) = ([ \begin{matrix}1_{p} & 0 \\ 0 & ∗ \end{matrix} ] , [ \begin{matrix}1_{p} & 0 \\ 0 & ∗ \end{matrix} ]) ∈ K_{ℂ}

fix A, we can conjugate B by some (g, h) to obtain B′ = diag (0, 1_{q}).

(2) It is obvious that the Jordan form of x_{p,q} depends only on p + q. This implies that O_{p,q} and O_{r,s} generate the same G_{ℂ} = GL_{2n}-orbit if and only if p + q = r + s. To calculate the dimension of O_{p,q}, we consider the fixed subgroup of x_{p,q} in K_{ℂ}, denoted by K_{p,q}. We have

K_{p,q} = {(g, h) ∈ K_{ℂ} | g and h are of the following form},  (1.2)

g = [ \begin{matrix}a & ∗ & ∗ \\ 0 & ∗ & ∗ \\ 0 & 0 & c \end{matrix} ] , h = [ \begin{matrix}a & 0 & 0 \\ ∗ & ∗ & 0 \\ ∗ & ∗ & c \end{matrix} ]; a ∈ GL_{p}, c ∈ GL_{q}.

The dimension formula then follows. Finally we observe that an orbit in 𝔛_{null} is determined by rank A and rank B (see the proof of (1)), we therefore conclude that the closure \overline{O_{p,q}} contains all the low rank matrices. Q.E.D.

Proposition 1.2 Assume that p + q < n.

(1) The variety \overline{O_{p,q}} is normal.

(2) The ring of regular functions on O_{p,q} and \overline{O_{p,q}} coincide:

ℂ[O_{p,q}] = ℂ[\overline{O_{p,q}}] (p + q < n).

Remark 1.3 If p + q = n, we will see that ℂ[O_{p,q}] ≠ ℂ[\overline{O_{p,q}}] afterwards (see Lemma 1.4).

Proof. We postpone the proof of the normality of \overline{O_{p,q}} to §4 (Corollary 4.7).

(2) We calculate the codimension of the boundary component \overline{O_{p,q}} \ O_{p,q}. Put r = p + q. Then this is equal to

r(2n − r) − (r − 1)(2n − (r − 1)) = 2(n − r) + 1.

Therefore, if r < n, the codimension is ≥ 2. This implies (2) under the statement (1). Q.E.D.

We denote by Λ_{n}^{+} the set of all dominant integral weights of GL_{n}, namely

Λ_{n}^{+} = {(λ_{1}, …, λ_{n}) ∈ ℤ^{n} | λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{n}},

and

P_{n} = {(λ_{1}, …, λ_{n}) ∈ ℤ^{n} | λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{n} ≥ 0},

the set of all partitions of length ≤ n. For λ ∈ Λ_{n}^{+}, τ_{λ}^{(n)} denotes the irreducible finite dimensional representation of GL_{n} of highest weight λ (with respect to the upper triangular Borel subgroup B of GL_{n}).

For p + q ≤ n, we put

Λ^{+}(p, q) = {(α, 0, …, 0, γ^{∗}) ∈ ℤ^{n} | α ∈ P_{p}, γ ∈ P_{q}},  (1.3)

where γ^{∗} = (−γ_{q}, …, −γ_{2}, −γ_{1}), for γ = (γ_{1}, γ_{2}, …, γ_{q}). We denote λ = (α, 0, …, 0, γ^{∗}) by α ⊙ γ.

Lemma 1.4 The ring of regular functions on O_{p,q} (or \overline{O_{p,q}}), denoted by ℂ[O_{p,q}] (or ℂ[\overline{O_{p,q}}]), inherits the action of K_{ℂ} = GL_{n} × GL_{n}.

(1) If p + q < n, then we have

ℂ[\overline{O_{p,q}}] = ℂ[O_{p,q}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix}(τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)} (as K_{ℂ}-module).

(2) If p + q = n, then we have

ℂ[O_{p,q}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ_{n}^{+} \end{matrix}(τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)},

while

ℂ[\overline{O_{p,q}}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix}(τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)}.

Proof. Recall the fixed subgroup K_{p,q} of x_{p,q} in K_{ℂ} (cf. (1.2)). We have

ℂ[O_{p,q}] ≃ ℂ[K_{ℂ}]^{K_{p,q}} ≃ Ind _{K_{p,q}}^{K_{ℂ}}1_{K_{p,q}}.

By the Frobenius reciprocity of algebraic groups (or from the second term of the above expression), π = τ_{μ}^{(n)} ⊠ τ_{λ}^{(n)} ∈ Irr(K_{ℂ}) occurs in ℂ[O_{p,q}] if and only if

Hom_{K_{p,q}}(π, 1) ≠ 0.

We consider intermediate parabolic P × \overline{P} ⊃ K_{p,q}, where

P = [ \begin{matrix}GL_{p} & ∗ & ∗ \\ 0 & GL_{n−(p+q)} & ∗ \\ 0 & 0 & GL_{q} \end{matrix} ] , \overline{P} = (opposite of P)

with the Levi-decompositions P = MN and \overline{P} = M\overline{N}.

We can write K_{p,q} = L_{p,q}N_{p,q}, where L_{p,q} ≃ GL_{p} × GL_{n′} × GL_{n′} × GL_{q} (n′ = n − (p + q)) and N_{p,q} = N × \overline{N}. Here the subgroup GL_{p} × GL_{q} of L_{p,q} embeds in M × M diagonally. We have

Hom_{K_{p,q}}(π, 1) ≃ Hom_{L_{p,q}}(π̃, 1),

where

π̃ = τ_{λ}^{(n)}/𝔫τ_{μ}^{(n)} ⊠ τ_{λ}^{(n)}/\overline{𝔫}τ_{μ}^{(n)},

and 𝔫, \overline{𝔫} are the Lie algebras of N, \overline{N}, respectively.

Note that π̃ is irreducible under the action of M ×M. since GL_{p}×GL_{q} embeds in M ×M diagonally, we see that

Hom_{L_{p,q}}(π̃, 1) ≠ 0

if and only if π̃ is of the form

(τ_{α}^{(p)} ⊠ 1_{GL_{n′}} ⊠ τ_{β}^{(q)}) ⊠ (τ_{α′}^{(p)} ⊠ 1_{GL_{n′}} ⊠ τ_{β′}^{(q)})

with α′ = α^{∗} and β′ = β^{∗}, and with the obvious notations.

The highest weight of τ_{μ}^{(n)} with respect to B is λ = (α, 0, …, 0, β), and the highest weight of τ_{λ}^{(n)} with respect to \overline{B} is (α^{∗}, 0, …, 0, β^{∗}). Thus λ = μ^{∗}, and it is dominant. If p + q < n, this means that α ∈ P_{p} and β = γ^{∗} for some γ ∈ P_{q}. If p + q = n, λ = (α, β) is a dominant weight if α_{1} ≥ ⋯ ≥ α_{p} ≥ β_{1} ≥ ⋯ ≥ β_{q} (α_{p} ≥ β_{1} is the essential restriction). This proves (1) and the first half of (2) (cf. Proposition 1.2).

Next we assume that p + q = n, and consider the second half of (2). We shall prove this by explicitly constructing a highest weight vector for (τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)} (λ ∈ Λ_{n}^{+}) which should be unique up to a scalar. For

X = [ \begin{matrix}0 & A \\ B & 0 \end{matrix} ] ,

put

\{ \begin{matrix}d_{k}^{+}(X) & = & (upper principal minor of A of size k) \\ d_{k}^{−}(X) & = & (lower principal minor of B of size k) \end{matrix} .

Then a highest weight vector of (τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)} (λ = α ⊙ γ) is given by

d(α, γ) = (∏_{k=1}^{p−1}(d_{k}^{+})^{α_{k}−α_{k+1}}) (d_{p}^{+})^{α_{p}} · (∏_{j=1}^{q−1}(d_{j}^{−})^{γ_{j}−γ_{j+1}}) (d_{q}^{−})^{γ_{q}}.

This function extends to \overline{O_{p,q}} if and only if α_{p} ≥ 0 and γ_{q} ≥ 0. Q.E.D.

We shall give an alternative proof of the above lemma (at least for ℂ[\overline{O_{p,q}}]) in §4 (see Theorem 4.6).

2 Theta lift of the trivial representation of U(p, q)

We reinterpret one of the results of Lee and Zhu [5] in a naive and intuitive way. First consider a seesaw pair:

M′ = U(p, q) × U(p, q)  U(n, n) = G diagonal ∪ ×  ∪ G′ =  U(p, q)  U(n) × U(n) = K

We still keep to the condition p+q ≤ n. We consider the dual pair U(p, q)×U(n) of compact type in Sp(2n(p + q), ℝ). By the U(p, q) × U(n) duality, we have the decomposition of the Weil representation ω of the metaplectic double cover Mp(2n(p + q), ℝ):

ω ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix} L(τ_{λ}^{(n)}) ⊠ (τ_{λ}^{(n)} ⊗ χ_{p,q}) (as a U(p, q) × U(n) -module),  (2.1)

where H denotes the inverse image of H in the metaplectic cover Mp(2n(p + q), ℝ), and χ_{p,q} = det^{(p−q)/2}. If p+q ≤ n, the irreducible representation L(τ_{λ}^{(n)}) is a holomorphic discrete series of U(p, q) with minimal K-type (τ_{α}^{(p)} ⊗ det^{n/2}) ⊠ (τ_{γ}^{(q)} ⊗ det^{n/2})^{∗} where λ = α ⊙ γ ∈ Λ^{+}(p, q).

Now consider two U(p, q) × U(n)-dualities, one holomorphic and the other antiholomorphic (or contragredient), and make a tensor product on them:

ω ⊗ ω^{∗} ≃ (\begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix} L(τ_{λ}^{(n)}) ⊠ (τ_{λ}^{(n)} ⊗ χ_{p,q})) ⊗ (\begin{matrix}∑ & ⊕ \\ μ∈Λ^{+}(p,q) \end{matrix} L(τ_{μ}^{(n)})^{∗} ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})^{∗})

≃ \begin{matrix}∑ & ⊕ \\ λ,μ \end{matrix} (L(τ_{λ}^{(n)}) ⊗ L(τ_{μ}^{(n)})^{∗}) |_{ΔU(p,q)} ⊠ ((τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})^{∗}).  (2.2)

The last decomposition is the one as a representation of U(p, q) × (U(n) × U(n) ). For the above decompositions, we refer the readers to [3] and [4]. Also the reference to [10] will be helpful.

Lemma 2.1 There exists a non-zero quotient map

L(τ_{λ}) ⊗ L(τ_{μ})^{∗} → 1_{U(p,q)} : surjective

if and only if λ = μ. In that case,

dim Hom_{(𝔤′,K̃′)}(L(τ_{λ}) ⊗ L(τ_{λ})^{∗}, 1_{U(p,q)}) = 1,

where 𝔤′ = 𝔲(p, q) and K′ = U(p) × U(q).

Proof. We shall prove this in a more general setting afterwards. See Lemma 5.1. Q.E.D.

Corollary 2.2 Howe’s maximal quotient Ω^{p,q}(1_{U(p,q)}) has the K-type decomposition

Ω^{p,q}(1_{U(p,q)})|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix}(τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{λ}^{(n)} ⊗ χ_{p,q})^{∗}.

Proof. Howe’s maximal quotient is canonically isomorphic to the dual of

Hom_{(𝔤′,K̃′)}(ω ⊗ ω^{∗}, 1_{U(p,q)}).

Then the above Lemma 2.1 and (2.2) give the K-type decomposition. Q.E.D.

In the following, we write Ω^{p,q}(1) instead of Ω^{p,q}(1_{U(p,q)}).

Theorem 2.3 (Lee-Zhu) The maximal quotient Ω^{p,q}(1) is irreducible and gives the thetalift θ^{p,q}(1). Thus we get

θ^{p,q}(1)|K̃ ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix}(τ_{λ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{λ}^{(n)} ⊗ χ_{p,q})^{∗}.

Proof. It is known that, in the stable range, the maximal quotient of the trivial representation is irreducible, and coincides with the theta lift ([13, Proposition 2.1]; see also [6]). Then the theorem follows from the above consideration. This proof is an alternative of that of Lee and Zhu ([5]). Q.E.D.

Recall the identification of nilpotent orbits in 𝔤 with the nilpotent coadjoint orbits in 𝔤^{∗}. Note that the associated variety is by definition an object in the dual space 𝔤^{∗}. Therefore, to consider the associated variety of θ^{p,q}(1), we should take the dual orbit of \overline{O_{p,q}} with respect to the Killing form, which is \overline{O_{q,p}} as a result. Since the K_{ℂ}-module structure on ℂ[\overline{O_{q,p}}] is the contragredient of that of ℂ[\overline{O_{p,q}}], Theorem 2.3 together with Lemma 1.4 strongly suggests that the associated variety of the theta lift θ^{p,q}(1) coincides with \overline{O_{q,p}}. This is in fact true.

Theorem 2.4 The associated cycle of θ^{p,q}(1) is [\overline{O_{q,p}}] without multiplicity. In particular, we have Deg θ^{p,q}(1) = deg \overline{O_{q,p}}, where Deg θ^{p,q}(1) denotes the Bernstein degree of θ^{p,q}(1).

Remark 2.5 The result of Przebinda et al. [2], [11], [12] determines the asymptotic support of θ^{p,q}(1). From the result of Schmid and Vilonen, the asymptotic support and the associated variety correspond to each other through Kostant-Sekiguchi correspondence. Therefore in principle we can compute the associated variety of θ^{p,q}(1) from the known result of Przebinda et al.

We shall give the proof in section 4.

Similar method can be used to compute the K-type decomposition of the theta lift of unitary characters of U(p, q) , and their associated cycles.

3 Degree of orbits O_{p,q}

In this section, we derive an explicit formula for deg \overline{O_{p,q}} = deg \overline{O_{q,p}} = Deg θ^{p,q}(1). Consider λ ∈ Λ^{+}(p, q). Then

λ = (α_{1}, …, α_{p}, 0, …, 0, −γ_{1}, …, −γ_{q}) =^{put} α ⊙ γ (α ∈ P_{p}, γ ∈ P_{q}).

From the explicit description of the highest weight vector for (τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)} (cf. (1)), we conclude that the homogeneous degree of (τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)} is

∥λ∥ = |α| + |γ|.

Note that ∥λ∥ ≠ |α| − |γ|. Therefore, to get the degree of the projective variety \overline{O_{p,q}}, we consider the following Hilbert-Samuel function

H(k) = ∑_{∥λ∥≤k,λ∈Λ^{+}(p,q)} dim(τ_{λ}^{(n)})^{∗} ⊠ τ_{λ}^{(n)} = ∑_{∥λ∥≤k,λ∈Λ^{+}(p,q)}(dim τ_{λ})^{2}.  (3.1)

By Weyl’s dimension formula, we have

dim τ_{λ} = ∏_{1≤i<j≤n} \frac{⟨λ + ρ, ε_{i} − ε_{j}⟩}{⟨ρ, ε_{i} − ε_{j}⟩},

where ⟨λ, ε_{i}⟩ = λ_{i} and

ρ = (\frac{n − 1}{2}, …, \frac{n + 1}{2} − k, …, \frac{1 − n}{2}) .

The result is the following rather complicated formula (here n′ = n − (p + q)):

dim τ_{λ} = \frac{∏_{p+1≤i<j≤p+n′}(j − i)}{∏_{1≤i<j≤n}(j − i)} ·

∏_{1≤i<j≤p}(α_{i} − α_{j} + j − i) ∏_{1≤i≤p<j≤p+n′}(α_{i} + j − i) ×

∏_{1≤i≤p,p+n′<j≤n}(α_{i} + γ_{j−(p+n′)} + j − i) ×

∏_{p+n′<i<j≤n}(γ_{j−(p+n′)} − γ_{i−(p+n′)} + j − i) ∏_{p<i≤p+n′<j≤n}(γ_{j−(p+n′)} + j − i).

From this formula, H(k) becomes, for a sufficiently large k,

H(k) = \frac{k^{d}}{p!q! (∏_{j=n′}^{n−1} j!)^{2}} ∫_{W} (∏_{i=1}^{p} x_{i} ∏_{j=1}^{q} y_{j})^{2n′} ∏_{1≤i≤p,1≤j≤q}(x_{i} + y_{j})^{2} ×

|Δ(x_{1}, …, x_{p})|^{2}|Δ(y_{1}, …, y_{q})|^{2} ∏_{i=1}^{p} dx_{i} ∏_{j=1}^{q} dy_{j} + O(k^{d−1}),

where Δ denotes the difference product, and

W = {(x, y) | x_{i} ≥ 0, y_{j} ≥ 0, ∑_{i=1}^{p} x_{i} + ∑_{j=1}^{q} y_{j} ≤ 1},  (3.2)
d = (p + q)(2n − (p + q)) = n^{2} − (n′)^{2}.

We therefore have the following Theorem 3.1 Put n′ = n − (p + q). Then

deg \overline{O_{p,q}} = \frac{(n^{2} − (n′)^{2})!}{p!q! (∏_{j=n′}^{n−1} j!)^{2}} ∫_{W} (∏_{i=1}^{p} x_{i} ∏_{j=1}^{q} y_{j})^{2n′} ∏_{1≤i≤p,1≤j≤q}(x_{i} + y_{j})^{2} ×

|Δ(x_{1}, …, x_{p})|^{2}|Δ(y_{1}, …, y_{q})|^{2} ∏_{i=1}^{p} dx_{i} ∏_{j=1}^{q} dy_{j},

where W is the region given in (3.2).

We expect that we can calculate this integral by the similar method as in [9]. Example 3.2 Let us consider the simplest case which is non-trivial, i.e., the case where p = q = 1. Then we get

deg \overline{O_{1,1}} = \frac{(4(n − 1))!}{\{(n − 2)!(n − 1)!\}^{2}}  ∫_{x+y≤1, x,y≥0} (xy)^{2(n−2)}(x + y)^{2}dxdy.  (3.3)

Lemma 3.3 We have

∫_{x+y≤1, x,y≥0} x^{α}y^{β}(x + y)^{γ}dxdy = \frac{B(α + 1, β + 1)}{α + β + γ + 2},

where B(s, t) denotes the Beta function.

From this lemma, we get

deg \overline{O_{1,1}} = \frac{(4(n − 1))!}{\{(n − 2)!(n − 1)!\}^{2}} \frac{\{(2(n − 2))!\}^{2}}{4(n − 1)(4(n − 2) + 1)!} = \frac{(4 n − 5) (Γ(2 n − 1))^{2}}{2 (2 n − 3) (Γ(n))^{4}} .

4 Nullcone and the geometry of nilpotent orbits

In this section, we consider a dual pair (G′, G) = (U(p, q), U(r, s)). Afterwards we will take r = s = n and assume that p + q ≤ n; but we do not assume any restriction on the pair for the moment.

Let K′_{ℂ} = GL_{p} × GL_{q} and K_{ℂ} = GL_{r} × GL_{s}, which are the complexifications of maximal compact subgroups K′ = U(p) × U(q) ⊂ G′ and K = U(r) × U(s) ⊂ G. We define a K′_{ℂ} × K_{ℂ}-action on

W = M_{p+q,r+s}(ℂ)

as follows:

\begin{aligned}& r s \\ M_{p+q,r+s} ∋ T = & [ \begin{matrix}X & Z \\ Y & W \end{matrix} ] \begin{matrix}p \\ q \end{matrix} , & k = (a, b) × (g, h) ∈ K′_{ℂ} × K_{ℂ}, \end{aligned}

k · T = [ \begin{matrix}aX^{t}g & ^{t}a^{−1}Zh^{−1} \\ ^{t}b^{−1}Y g^{−1} & bW ^{t}h \end{matrix} ] .

We define two K′_{ℂ} × K_{ℂ}-equivariant maps

\{ \begin{matrix}φ_{1} : & M_{p+q,r+s} & ⟶ & M_{r,s} ⊕ M_{s,r} = 𝔰, \\ φ_{2} : & M_{p+q,r+s} & ⟶ & M_{p,q} ⊕ M_{q,p} = 𝔰′ \end{matrix}

by

\{ \begin{matrix}φ_{1}(T) = (^{t}XZ, ^{t}W Y), \\ φ_{2}(T) = (X^{t}Y, W ^{t}Z). \end{matrix}

Here K_{ℂ} (resp., K′_{ℂ}) acts on 𝔰 (resp., 𝔰′) by the restriction of the Adjoint action, and K′_{ℂ} (resp., K_{ℂ}) acts on 𝔰′ (resp., 𝔰) trivially.

For D ∈ 𝔰 and D′ ∈ 𝔰′, we say that D corresponds to D′ if there exists T ∈ M_{p+q,r+s} such that φ_{1}(T) = D, and φ_{2}(T) = D′. Further we say a K_{ℂ} orbit O corresponds to a K′_{ℂ} orbit O′ if there exists D ∈ O and D′ ∈ O′ such that D and D′ correspond.

Now put r = s = n, and assume that p + q ≤ n.

Lemma 4.1 By the maps, we get

\overline{O_{p,q}} = φ_{1}(φ_{2}^{−1}(0)) ⊂ M_{n} ⊕ M_{n} = 𝔰,

i.e., the nilpotent K_{ℂ}-orbit O_{p,q} ⊂ 𝔰 corresponds to the trivial orbit {0} ⊂ 𝔰′ = M_{p,q} ⊕ M_{q,p}.

Proof. Take T ∈ φ_{2}^{−1}(0). Then we immediately have

φ_{1}(T) = [ \frac{^{t}XZ}{^{t}W Y} ] ∈ 𝔛_{null}

by the definition of φ_{2}. Moreover, it is easy to see that rank ^{t}XZ ≤ p and rank ^{t}W Y ≤ q. Thus we conclude that φ_{1}(φ_{2}^{−1}(0)) ⊂ \overline{O_{p,q}}.

To see the reversed inclusion, let X = Z = [1_{p} 0] and W = Y = [0 1_{q}]. Then φ_{1}(T) = x_{p,q} and φ_{2}(T) = 0. Thus x_{p,q} ∈ φ_{1}(φ_{2}^{−1}(0)). Since φ_{1} and φ_{2} are K′_{ℂ} × K_{ℂ}-equivariant, we have O_{p,q} ⊂ φ_{1}(φ_{2}^{−1}(0)). Taking the closure, we conclude that \overline{O_{p,q}} ⊂ φ_{1}(φ_{2}^{−1}(0)). Q.E.D.

Next we consider the null cone defined by

𝔑 = φ_{2}^{−1}(0) = \{[ \begin{matrix}X & Z \\ Y & W \end{matrix} ] ∈ W | X^{t}Y = 0, W ^{t}Z = 0\} , W = M_{p+q,2n}.

It is K′_{ℂ} × K_{ℂ}-stable.

Let I = ℂ[W]^{K_{ℂ}} be the algebra of K_{ℂ}-invariant polynomials on W. By the classical invariant theory, the matrix elements of X^{t}Y and W ^{t}Z generate I. Thus 𝔑 is the variety of zeros of all the invariants without constant term.

Proposition 4.2 Assume that p + q ≤ n. Then the null cone 𝔑 is an irreducible variety of complete intersections. Its defining ideal is generated by K_{ℂ}-invariants of degree two. Moreover, if p + q < n, it is normal.

Proof. We apply [1, Theorem 2.2.11]. For details, see [8]. Q.E.D.

Let ℌ be the space of K_{ℂ}-harmonic polynomials, and J_{+} the ideal generated by K_{ℂ}- invariants in ℂ[W] which vanish at the origin. The following lemma is well-known.

Lemma 4.3 We have

ℂ[W] = ℌ ⊕ J_{+}.

Remark 4.4 If p + q ≤ n, then ℂ[W] ≃ ℌ ⊗ ℂ[W]^{K_{ℂ}} holds.

Since J_{+} = I(𝔑) is the defining ideal of 𝔑 by Proposition 4.2, the function ring on 𝔑 coincides with ℌ. Note that ℌ carries the action of K′_{ℂ} = GL_{p} × GL_{q} and K_{ℂ} = GL_{n} × GL_{n}. We record this as

Lemma 4.5 The function ring on the null cone 𝔑 = φ_{2}^{−1}(0) coincides with the K_{ℂ}-harmonic polynomials on W = M_{p+q,2n}:

ℂ[𝔑] ≃ ℌ (as a K′_{ℂ} × K_{ℂ}-module).

Now we describe the K′_{ℂ} × K_{ℂ}-module structure of ℌ. Consider the two halves of the matrix T:

T^{+} = [ \begin{matrix}X \\ Y \end{matrix} ] ∈ M_{p+q,n} =: W^{+}, T^{−} = [ \begin{matrix}Z \\ W \end{matrix} ] ∈ M_{p+q,n} =: W^{−}.

Then by U(p, q) × U(n)-duality, we know

ℂ[W^{+}] ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix} L(τ_{λ}^{(n)})^{∗} ⊠ (τ_{λ}^{(n)})^{∗},

as a U(p, q) × U(n)-module. Note that ℂ[W^{+}] is the dual of the symmetric algebra S(W^{+}). Therefore, we take the dual on the right hand side. In this setting, the space of GL_{n}- harmonics of type (τ_{λ}^{(n)})^{∗} has the form ((τ_{α}^{(p)})^{∗} ⊗ τ_{γ}^{(q)}) ⊠ (τ_{λ}^{(n)})^{∗}, where ((τ_{α}^{(p)})^{∗} ⊗ τ_{γ}^{(q)}) is

the lowest K′ = U(p) × U(q)-type of L(τ_{λ})^{∗} tensored χ′_{n,n} ( χ′_{n,n} = det^{n/2} ⊠ det^{−n/2} ) and λ = α ⊙ γ (cf. (1.3)).

Let

ℌ^{+} = ℌ ∩ ℂ[W^{+}], ℌ^{−} = ℌ ∩ ℂ[W^{−}].

Then we have

ℌ^{+} ≃ \begin{matrix}∑ & ⊕ \\ λ∈Λ^{+}(p,q) \end{matrix} ((τ_{α}^{(p)})^{∗} ⊗ τ_{γ}^{(q)}) ⊠ (τ_{λ}^{(n)})^{∗},

where λ = α ⊙ γ (cf. (1.3)). Similarly (but conjugated), we have

ℂ[W^{−}] ≃ \begin{matrix}∑ & ⊕ \\ μ∈Λ^{+}(p,q) \end{matrix} L(τ_{μ}^{(n)}) ⊠ τ_{μ}^{(n)},

and

ℌ^{−} ≃ \begin{matrix}∑ & ⊕ \\ μ∈Λ^{+}(p,q) \end{matrix} (τ_{β}^{(p)} ⊗ (τ_{δ}^{(q)})^{∗}) ⊠ τ_{μ}^{(n)},

where μ = β ⊙ δ.

Theorem 4.6 Put 𝔑 = φ_{2}^{−1}(0), the null cone defined by K_{ℂ}-invariants. Then we have

ℂ[𝔑] ≃ ℌ = ℌ^{+} ⊗ ℌ^{−}

≃ \begin{matrix}∑ & ⊕ \\ λ,μ∈Λ^{+}(p,q) \end{matrix} (((τ_{α}^{(p)})^{∗} ⊗ τ_{β}^{(p)}) ⊠ (τ_{γ}^{(q)} ⊗ (τ_{δ}^{(q)})^{∗})) ⊠ ((τ_{λ}^{(n)})^{∗} ⊠ τ_{μ}^{(n)}) ,

as K′_{ℂ} × K_{ℂ}-module. Furthermore the map φ_{1}^{∗} induces a ring isomorphism:

φ_{1}^{∗} : ℂ[\overline{O_{p,q}}] ⟶^{∼} ℂ[𝔑]^{K′_{ℂ}}.  (4.1)

Proof. We have to show that (4.1) is an isomorphism.

Look at the original map φ_{1} : W ⟶ 𝔰. Then the induced map φ_{1}^{∗} : ℂ[𝔰] ⟶ ℂ[W] is given by

φ_{1}^{∗}(A_{ij})(T) = ∑_{k=1}^{p} X_{ki}Z_{kj}, φ_{1}^{∗}(B_{ij})(T) = ∑_{k=1}^{q} W_{ki}Y_{kj},

where T = [ \begin{matrix}X & Z \\ Y & W \end{matrix} ] ∈ W and \{(A_{ij}, B_{ij}) | 1 ≤ i, j ≤ n\} is the set of coordinates on

𝔰 = M_{n} ⊕ M_{n}. By the classical invariant theory, the right hand side of the above equations generate the ring of K′_{ℂ}-invariants in ℂ[W], so that Image φ_{1}^{∗} = ℂ[W]^{K′_{ℂ}}.

Now consider the restriction φ_{1} : 𝔑 ⟶ 𝔰, which we denote by the same notation. By the above consideration, we know that the induced map actually gives an algebra morphism

φ_{1}^{∗} : ℂ[𝔰] ⟶ ℂ[𝔑]^{K′_{ℂ}} ⊂ ℂ[𝔑] = ℂ[W]/I(𝔑),

and it is surjective. Now, since ℂ[𝔰]/ Ker φ_{1}^{∗} = ℂ[\overline{O_{p,q}}] by Lemma 4.1, we conclude the result. Q.E.D.

Corollary 4.7 If p + q < n, the nilpotent orbit \overline{O_{p,q}} is an irreducible normal variety.

Finally we are ready to prove the following proposition, which implies Theorem 2.4.

Proposition 4.8 The associated variety of θ^{p,q}(1) is \overline{O_{q,p}}.

Proof. We identify the nilpotent orbit O_{q,p} with a coadjoint orbit via Killing form. Then the corresponding orbit is O_{p,q}. In this proof, we always consider O_{p,q} as a coadjoint nilpotent orbit. (See also the remark before Theorem 2.4.)

We take a coordinate system on 𝔰 = M_{n} ⊕ M_{n} ∋ (A, B) as {A_{ij}, B_{ij} | 1 ≤ i, j ≤ n}, as before. Note that we regard A_{ij} as a linear function on 𝔰 which extracts the (i, j)-element of A. So A_{ij}, B_{ij} ∈ ℂ[𝔰]. Take one of the homogeneous defining equations f(A, B) = 0 of \overline{O_{p,q}}. Then we have

φ_{1}^{∗}(f(A, B)) = f(S, T); S_{ij} = φ_{1}^{∗}(A_{ij}) = ∑_{k=1}^{p} X_{ki}Z_{kj}, T_{ij} = φ_{1}^{∗}(B_{ij}) = ∑_{l=1}^{q} W_{li}Y_{lj}.

By Theorem 4.6 and its proof, we know that f(S, T) ∈ I(𝔑)^{K′_{ℂ}}. In particular, there exist polynomials e_{kl}^{±} ∈ ℂ[W] such that

f(S, T) = ∑_{\begin{matrix}1≤k≤p \\ 1≤l≤q \end{matrix}} e_{kl}^{+} · φ_{2}^{∗}(C_{kl}) + ∑_{\begin{matrix}1≤k≤q \\ 1≤l≤p \end{matrix}} e_{kl}^{−} · φ_{2}^{∗}(D_{kl}),

where (C, D) ∈ M_{p,q} ⊕ M_{q,p} = 𝔰′ and {C_{kl}, D_{kl}} is a system of coordinate functions on 𝔰′.

The Weil representation associated to the dual pair U(p, q) × U(n, n) is realized on ℂ[W] by the Fock model. Let Ã_{ij} be the operator of the Weil representation of the matrix

[ \begin{matrix}0 & 0 \\ E_{ji} & 0 \end{matrix} ] ∈ 𝔰.

Define B̃_{ij}, C̃_{kl}, D̃_{kl} similarly. Then we have

Ã_{ij} = 2 ∑_{k=1}^{p} \frac{∂^{2}}{∂X_{ki}∂Z_{kj}} − \frac{1}{2} ∑_{l=1}^{q} W_{li}Y_{lj},

B̃_{ji} = −\frac{1}{2} ∑_{k=1}^{p} X_{ki}Z_{kj} + 2 ∑_{l=1}^{q} \frac{∂^{2}}{∂W_{li}∂Y_{lj}}

up to non-zero constant multiplication.

If we take the K-stable good filtration of θ^{p,q}(1) induced from the total degree filtration of ℂ[W], Ã_{ij} and B̃_{ij} will act on the graded module gr θ^{p,q}(1) as

B̃_{ji} = −\frac{1}{2} ∑_{k=1}^{p} X_{ki}Z_{kj}, Ã_{ij} = −\frac{1}{2} ∑_{l=1}^{q} W_{li}Y_{lj}.

These formulas coincide with S_{ij} and T_{ij} above up to constant multiplication. Therefore we have

(−2)^{− deg f}f(B̃_{ji}, Ã_{ij}) = ∑_{1≤k≤p,1≤l≤q} e_{kl}^{+} · φ_{2}^{∗}(C_{kl}) + ∑_{1≤k≤q,1≤l≤p} e_{kl}^{−} · φ_{2}^{∗}(D_{kl})

on gr θ^{p,q}(1). Note that

φ_{2}^{∗}(C_{kl}) = ∑_{i=1}^{n} X_{ki}Y_{li}, φ_{2}^{∗}(D_{kl}) = ∑_{i=1}^{n} Z_{ki}W_{li}.

On the other hand, the operators C̃_{kl} and D̃_{kl} are given by

C̃_{kl} = 2 ∑_{i=1}^{n} \frac{∂^{2}}{∂X_{ki}∂Y_{li}} − \frac{1}{2} ∑_{j=1}^{n} W_{kj}Z_{lj} ≡ −\frac{1}{2} ∑_{j=1}^{n} W_{kj}Z_{lj},

D̃_{lk} = −\frac{1}{2} ∑_{i=1}^{n} X_{ki}Y_{li} + 2 ∑_{j=1}^{n} \frac{∂^{2}}{∂W_{lj}∂Z_{kj}} ≡ −\frac{1}{2} ∑_{i=1}^{n} X_{ki}Y_{li},

which should vanish on gr θ^{p,q}(1) because θ^{p,q}(1) is associated to the trivial representation of U(p, q) . Hence we conclude φ_{2}^{∗}(C_{kl}) and φ_{2}^{∗}(D_{lk}) act trivially on gr θ^{p,q}(1). From this we see that f(B̃_{ji}, Ã_{ij}) = 0 on gr θ^{p,q}(1). This means that the associated variety of θ^{p,q}(1) is contained in \overline{O_{q,p}}. Comparing dimension, we conclude that they must coincide. Q.E.D.

5 Generalities on Harish-Chandra modules

In this section, we consider general (𝔤, K)-modules, where 𝔤 is the complexification of the Lie algebra of a semisimple Lie group G and K is a maximal compact subgroup of G.

Let H be a (𝔤, K)-module which is K-admissible, i.e., dim Hom_{K}(H, τ) < ∞ for any τ ∈ Irr(K). We also assume that H is locally K-finite, which means that H = H_{K} where H_{K} denotes the space of K-finite vectors in H. Then, H^{∗} = Hom_{ℂ}(H, ℂ)_{K} is also Kadmissible and we have a canonical isomorphism H ≃ (H^{∗})^{∗}. Note that H^{∗} does not denote the algebraic dual of H.

Lemma 5.1 Let H_{1} be a (𝔤, K)-module which is locally K-finite. Then, for any Kadmissible (𝔤, K)-module H_{2} which is locally K-finite, we have

Hom_{(𝔤,K)}(H_{1} ⊗ H_{2}^{∗}, 1) ≃ Hom_{(𝔤,K)}(H_{1}, H_{2}).

We shall omit the proof, as it is straightforward.

Corollary 5.2 Let σ be a finite dimensional (𝔤, K)-module. Then we have

Hom_{(𝔤,K)}(H_{1} ⊗ H_{2}^{∗}, σ) ≃ Hom_{(𝔤,K)}(H_{1}, H_{2} ⊗ σ),

under the same assumption on H_{1} and H_{2} as in Lemma 5.1.

Proof. Since H_{1} ⊗ H_{2}^{∗} is locally K-finite, we can apply Lemma 5.1 and get

Hom_{(𝔤,K)}(H_{1} ⊗ H_{2}^{∗}, σ) ≃ Hom_{(𝔤,K)}(H_{1} ⊗ H_{2}^{∗} ⊗ σ^{∗}, 1).

Note that H_{2}^{∗} ⊗σ^{∗} ≃ (H_{2} ⊗σ)^{∗} because σ is finite dimensional. Now apply Lemma 5.1 again. Q.E.D.

Corollary 5.3 Let ω be the Harish-Chandra module of the Weil representation of Mp(2n(p+ q), ℝ). We consider a dual pair G × G′ = U(p, q) × U(n) in Mp(2n(p + q), ℝ), and regard ω as a (𝔤, K̃)-module. Then for any irreducible Harish-Chandra module L of (𝔤, K̃), we have

Hom_{(𝔤,K̃)}(ω ⊗ ω^{∗}, L^{∗}) ≃ Hom_{(𝔤,K̃)}(L ⊗ ω, ω),

Proof. Note that ω ⊗ ω^{∗} is locally K̃-finite. Also we note that ω is K̃-admissible. Now we can apply Lemma 5.1 twice and get the desired formula. Q.E.D.

6 Theta lift of a holomorphic discrete series

We keep to the notation in §2. In particular L(τ_{λ}) denotes the holomorphic discrete series representation of U(p, q) which corresponds to τ_{λ} = τ_{λ}^{(n)} ∈ Irr(U(n)). Note that λ ∈ Λ^{+}(p, q), since p + q ≤ n. We shall derive a K-type formula for its theta lift.

Lemma 6.1 For any σ ∈ Irr(U(p, q) ), the theta lift θ^{p,q}(σ) is non-zero if and only if there exists a pair (λ, μ) such that

L(τ_{λ}) ⊗ L(τ_{μ})^{∗}|_{ΔU(p,q)} ⟶ σ

is a non-zero (hence surjective) quotient map. In this case, the maximal quotient Ω^{p,q}(σ) contains K-type (τ_{λ} ⊗ χ_{p,q}) ⊠ (τ_{μ} ⊗ χ_{p,q})^{∗} at least once.

Proof. Arguments are the same as in §2, though we use σ instead of 1. Q.E.D.

Now take σ = L(τ_{ν}), a holomorphic discrete series. Then by Lemma 5.1, we have

Hom_{(𝔤′,K̃′)}(L(τ_{λ}) ⊗ L(τ_{μ})^{∗}, L(τ_{ν})) ≃ Hom_{(𝔤′,K̃′)}(L(τ_{λ}) ⊗ L(τ_{μ})^{∗} ⊗ L(τ_{ν})^{∗}, 1_{U(p,q)}). (6.1)

Lemma 6.2 Let c_{μ,ν}^{η} be a branching coefficient defined by

τ_{μ} ⊗ τ_{ν} ≃ \begin{matrix}∑ & ⊕ \\ η \end{matrix} c_{μ,ν}^{η}τ_{η} (as GL_{n}-modules).

If n is even, we have

L(τ_{μ})^{∗} ⊗ L(τ_{ν})^{∗} ≃ \begin{matrix}∑ & ⊕ \\ η∈Λ^{+}(p,q) \end{matrix}c_{μ,ν}^{η}L(τ_{η−n/2δ_{p,q}})^{∗};
δ_{p,q} = 𝕀_{p} ⊙ 𝕀_{q} = (1, …, 1, 0, …, 0, −1, …, −1).

Proof. Consider the Weil representation associated to the dual pair U(p, q) × U(2n). As a (U(p, q) × U(n))^{2}-module, we have

ω ⊗ ω ≃ (\begin{matrix}∑ & ⊕ \\ μ∈Λ^{+}(p,q) \end{matrix} L(τ_{μ}^{(n)}) ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})) ⊗ (\begin{matrix}∑ & ⊕ \\ ν∈Λ^{+}(p,q) \end{matrix} L(τ_{ν}^{(n)}) ⊠ (τ_{ν}^{(n)} ⊗ χ_{p,q}))

≃ \begin{matrix}∑ & ⊕ \\ μ,ν \end{matrix} (L(τ_{μ}^{(n)}) ⊗ L(τ_{ν}^{(n)})) ⊠ ((τ_{μ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{ν}^{(n)} ⊗ χ_{p,q})) ,  (6.2)

while as a U(p, q) × U(2n)-module

ω ⊗ ω ≃ \begin{matrix}∑ & ⊕ \\ η∈Λ^{+}(p,q) \end{matrix} L(τ_{η}^{(2n)}) ⊠ (τ_{η}^{(2n)} ⊗ χ_{p,q}).  (6.3)

Since

τ_{η}^{(2n)}|_{U(n)×U(n)} ≃ \begin{matrix}∑ & ⊕ \\ μ,ν \end{matrix} c_{μ,ν}^{η}τ_{μ}^{(n)} ⊗ τ_{ν}^{(n)},

we have

(6.3) = \begin{matrix}∑ & ⊕ \\ μ,ν \end{matrix} ( \begin{matrix}∑ & ⊕ \\ η \end{matrix} c_{μ,ν}^{η}L(τ_{η}^{(2n)})) ⊠ ((τ_{μ}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{ν}^{(n)} ⊗ χ_{p,q})) .  (6.4)

Comparing (6.2) and (6.4), we get

L(τ_{μ}^{(n)}) ⊗ L(τ_{ν}^{(n)}) ≃ \begin{matrix}∑ & ⊕ \\ η \end{matrix} c_{μ,ν}^{η}L(τ_{η}^{(2n)}).  (6.5)

Since p + q is smaller than n and hence 2n,

L(τ_{η}^{(2n)}) ≃ L(τ_{η−n/2δ_{p,q}}^{(n)}), δ_{p,q} = 𝕀_{p} ⊙ 𝕀_{q}

holds if we shift η by n/2δ_{p,q}. Here we use the assumption that n is even. Also we identify η = (α, 0, …, 0, γ^{∗}) (α ∈ P_{p}, γ ∈ P_{q}) of length 2n (zero occurs 2n − (p + q)-times between α and γ^{∗}) and η = (α, 0, …, 0, γ^{∗}) (zero occurs n − (p + q)-times between α and γ^{∗}) of length n.

Take the K-finite part of the dual of (6.5). Since the K-types of L(τ_{μ}^{(n)}) ⊗ L(τ_{ν}^{(n)}) has finite multiplicities, we get the formula in the lemma. (Or, we can consider ω^{∗} ⊗ ω^{∗} from the beginning.) Q.E.D.

We apply the above lemma to (6.1). Then we get

Hom_{(𝔤′,K̃′)}(L(τ_{λ}) ⊗ L(τ_{μ})^{∗}, L(τ_{ν})) ≃ ∏_{η} c_{μ,ν}^{η} Hom_{(𝔤′,K̃′)}(L(τ_{λ}) ⊗ L(τ_{η−n/2δ_{p,q}})^{∗}, 1_{U(p,q)}).

The last expression is already studied by Lee-Zhu (see Lemma 2.1 and Corollary 2.2), which says that

dim Hom_{(𝔤′,K̃′)}(L(τ_{λ}) ⊗ L(τ_{η−n/2δ_{p,q}})^{∗}, 1_{U(p,q)}) ≤ 1,

and the above mentioned dimension is one if and only if λ = η − n/2δ_{p,q} holds. So we get

Theorem 6.3 Assume that n is even. Let L(τ_{ν}) be a holomorphic discrete series and consider the theta lift θ^{p,q}(L(τ_{ν})) ∈ Irr(U(n, n) ). Then we have

θ^{p,q}(L(τ_{ν}))|K̃ ≃ \begin{matrix}∑ & ⊕ \\ η,μ \end{matrix} c_{μ,ν}^{η}(τ_{η−n/2δ_{p,q}}^{(n)} ⊗ χ_{p,q}) ⊠ (τ_{μ}^{(n)} ⊗ χ_{p,q})^{∗}.

Proof. Put σ = L(τ_{ν}). By the arguments around [7, Lemma 2.2], we know that Ω^{p,q}(σ) is already irreducible for a (holomorphic) discrete series representation σ. Note that p + q ≤ n. By the calculation above, we know that Ω(σ)|K̃ is exactly equal to the right hand side of the desired formula. Q.E.D.

We shall omit the result for odd n, which is similar.

Problem 6.4 Consider the same thing for a finite dimensional irreducible representation σ ∈ Irr(U(p, q) ), which is not unitary.

7 Further results

Here we would like to indicate several generalizations of the results of this note.

• We can consider the associated cycles of the theta lift of holomorphic discrete series representations. In particular, if we take a spherical holomorphic discrete series, which means that the minimal K-type is one-dimensional, then it has a multiplicity-free associated cycle with an irreducible associated variety.

• We have a generalization of the results here to the pair U(p, q) × U(r, s) in the stable range. However, the description of the parameters gets more complicated.

• We have similar results for the other type I dual pairs in stable range (cf. [8]).

These generalizations will be treated in a forthcoming paper.

References

[1] N. Chriss and V. Ginzburg, Representation theory and complex geometry, Birkhäuser Boston, Boston, MA, 1997.

[2] A. Daszkiewicz, W. Kraśkiewicz and T. Przebinda, Nilpotent orbits and complex dual pairs, J. Algebra 190 (1997), no. 2, 518–539.

[3] R. Howe, Remarks on classical invariant theory, Trans. Amer. Math. Soc. 313 (1989), no. 2, 539–570. Erratum to: “Remarks on classical invariant theory”, Trans. Amer. Math. Soc. 318 (1990), no. 2, 823.

[4] R. Howe, Transcending classical invariant theory. J. Amer. Math. Soc. 2 (1989), no. 3, 535–552.

[5] S. T. Lee and C.-B. Zhu, Degenerate principal series and local theta correspondence, Trans. Amer. Math. Soc. 350 (1998), no. 12, 5017–5046.

[6] J.-S. Li, Singular unitary representations of classical groups. Invent. Math. 97 (1989), no. 2, 237–255.

[7] J.-S. Li, Theta lifting for unitary representations with nonzero cohomology. Duke Math. J. 61 (1990), no. 3, 913–937.

[8] K. Nishiyama, Multiplicity-free action on a null cone and the geometry of nilpotent orbits, to appear in Proceedings of JSPS-DFG workshop on Infinite Dimensional Harmonic Analysis in Kyoto, September 1999.

[9] Kyo Nishiyama and Hiroyuki Ochiai, Bernstein degree of singular unitary highest weight representations of the metaplectic group. Proc. Japan Acad., 75, Ser. A (1999), 9 – 11.

[10] Kyo Nishiyama, Hiroyuki Ochiai, and Kenji Taniguchi, Bernstein degree and associated cycles of Harish-Chandra modules (Hermitian symmetric case), Kyushu University Preprint Series in Mathematics, 1999-16. Also available at the following URL:

            http://w3rep.math.h.kyoto-u.ac.jp/papere.html#NOT

[11] T. Przebinda, Characters, dual pairs, and unitary representations, Duke Math. J. 69 (1993), no. 3, 547–592.

[12] T. Przebinda, Characters, dual pairs, and unipotent representations, J. Funct. Anal. 98 (1991), no. 1, 59–96.

[13] C.-B. Zhu and J.-S. Huang, On certain small representations of indefinite orthogonal groups, Represent. Theory 1 (1997), 190–206 (electronic).

Kyo Nishiyama

Faculty of Integrated Human Studies Kyoto University Kyoto 606-8501 Japan

E-mail address: kyo@math.h.kyoto-u.ac.jp

Chen-bo Zhu

Department of Mathematics National University of Singapore Kent Ridge Singapore 119260

E-mail address: matzhucb@nus.edu.sg