Journal of Lie Theory Volume 5 (1995) 241–273 C ◯1995 Heldermann Verlag

Cauchy–Szegö kernels for Hardy spaces on simple Lie groups

Grigori Olshanski

Communicated by J. Hilgert

Résumé. Pour tout groupe simple réel G possédant des représentations unitaires de plus haut poids on peut definir l’espace de Hardy H^{2}(G). C’est un espace hilbertien formé des fonctions holomorphes dans un tube ‘non– commutatif’ Γ^{0} satisfaisant une condition de type Hardy ( Γ^{0} est l’intérieur d’un semi–groupe complexe Γ contenant le groupe G). L’espace H^{2}(G) s’identifie au sous–espace bi–invariant de L^{2}(G) portant la série discrète holomorphe. Le noyau reproduisant K de l’espace de Hardy est appelé le noyau de Cauchy–Szegö. Le résultat principal de l’article est un calcul explicite de ce noyau pour 3 familles de groupes G: groupes symplectiques Sp(n,ℝ), groupes métaplectiques Mp(n,ℝ), et groupes pseudo–unitaires SU(p,q).

Abstract. For any simple real group G possessing unitary highest weight representations one can define the Hardy space H^{2}(G). This is a Hilbert space formed by holomorphic functions in a ‘non–commutative’ tube domain Γ^{0} satisfying a Hardy–type condition ( Γ^{0} is the interior of a non– commutative complex semigroup Γ containing the group G). The space H^{2}(G) is identified with the bi–invariant subspace of L^{2}(G) carrying the holomorphic discrete series. The reproducing kernel K of the Hardy space is called the Cauchy–Szegö kernel. The main result of the paper is an explicit calculation of this kernel for 3 families of groups G: the symplectic groups Sp(n,ℝ), the metaplectic groups Mp(n,ℝ), and the pseudo–unitary groups SU(p,q).

Introduction

Let H be a Hilbert space whose elements f are holomorphic functions in a complex domain D^{1} . Then one can associate to H a kernel function K: if

1 More exactly, one should assume that H is continuously embedded into O(D),

the space of all holomorphic functions, equipped with the topology of uniform convergence on compact subsets.

ISSN 0940–2268 / $2.50 ^{_{C} ◯} Heldermann Verlag f_{1}, f_{2}, … is an arbitrary orthonormal basis in H then

K(z, w) = f_{1}(z)\overline{f_{1}(w)} + f_{2}(z)\overline{f_{2}(w)} + …, z, w ∈ D,

and the definition does not depend on the choice of the basis. The kernel K possesses the following properties: K is holomorphic in z and anti–holomorphic in w; K(z, w) = \overline{K(w, z)}; K is a positive definite kernel on D × D; for any w ∈ D the function K(·, w) lies in H; for any function f ∈ H and any w ∈ D

f(w) = (f, K(·, w))_{H}

(the reproducing property); in particular, for any z, w ∈ D

(K(·, w), K(·, z))_{H} = K(z, w);

finally, the initial Hilbert space H is uniquely determined by its kernel function K.

Classical examples are the Bergman and Hardy (H^{2} ) spaces in the unit disc |z| < 1 and the (say, right) half–plane Re z > 0.

For the Bergman space, the square of norm ∥f∥^{2} is obtained by integrating |f(z)|^{2} over the domain with respect to the Lebesgue measure, and the corresponding kernel function has the form

(1 − z\overline{w})^{−2} (the disc)  or  (z + \overline{w})^{−2} (the half–plane).

For the Hardy space, ∥f∥^{2} is defined as the integral of |f(z)|^{2} over the boundary,^{2} and the corresponding kernel function is

(1 − z\overline{w})^{−1} (the disc)  or  (z + \overline{w})^{−1} (the half–plane).

It is well–known that these examples can be generalized to multidimensional bounded symmetric domains and to tube domains ℝ^{n} +iC (where C is an open convex cone in ℝ^{n}); further generalizations involve interpolation between the Bergman and Hardy cases, vector–valued holomorphic functions etc. Note that various Hilbert spaces of holomorphic functions and their kernel functions naturally arise in connection with unitary highest weight representations. (See, e.g., Faraut and Korányi [3], Inoue [10], Stein and Weiss [22], Vergne and Rossi [23].)

The present paper deals with multidimensional complex domains of another kind, which may be viewed as ‘noncommutative’ tube domains.

Let D be an irreducible bounded symmetric domain and G be a connected group, locally isomorphic to the automorphism group of D. Then G is (locally isomorphic to) one of the groups SU(p, q), Sp(n, ℂ), SO^{∗}(2n), SO_{0}(2, n) or certain 2 exceptional groups of type E. Let G_{ℂ} be the complexification of G.^{3} Then in G_{ℂ} there exist closed subsemigroups Γ with nonempty interior Γ^{0} such that Γ ⊃ G and Γ ≠ G_{ℂ} (in particular, Γ and Γ^{0} are invariant with respect to the two–sided action of the group G). The existence of such semigroups is closely related to the existence of proper closed invariant convex cones C in 𝔤, the Lie algebra of G: there is a bijective correspondence Γ ↔ C between semigroups and cones, and each open semigroup Γ^{0} has the form G exp iC^{0} , where C^{0} is the interior of the cone C. (See Olshanski [19], Hilgert and Neeb [6].) Thus, Γ^{0} indeed looks as a ‘noncommutative’ tube domain with skeleton G.

2 The boundary value of f(z) must be defined in a suitable way.

3 To simplify the discussion we tacitly assume here that G is linear, i.e., admits a global complexification; however, this assumption is not essential.

It turns out that for any semigroup Γ one can define a Hilbert space H^{2}(Γ) ⊂ O(Γ^{0}), which is an analogue of the Hardy H^{2} space. The space H^{2}(Γ) admits a canonical isometric embedding into L^{2}(G) as a two–sided Ginvariant subspace. When Γ is the so–called minimal semigroup, H^{2}(Γ) just carries the holomorphic discrete series of G (in general case H^{2}(Γ) carries a part of the holomorphic discrete series). Let K(z, w) be the kernel function of the space H^{2}(Γ) (it is also called the Cauchy–Szegö kernel). Then the kernel K(z, g), where z ∈ Γ, g ∈ G,^{4} defines the orthoprojection L^{2}(G) → H^{2}(Γ). In particular, if the semigroup is minimal then this is the orthoprojection onto the holomorphic discrete series.

The idea of Hardy spaces carrying the holomorphic discrete series is due to Gelfand and Gindikin [4]. A construction of the spaces H^{2}(Γ) was given in author’s paper [20]. Further works in this direction are Hilgert and Ólafsson [7], Hilgert, Ólafsson, and Ørsted [8].

A natural problem is to compute the Cauchy–Szegö kernel explicitly, at least for the minimal semigroups. To this end, one can try to start from the following presentation of the Cauchy–Szegö kernel:

Assume the center of G is finite. Let λ range over the set of the highest weights of the holomorphic discrete series representations T_{λ} occuring in the decomposition of H^{2}(Γ) (recall that if Γ is minimal, the whole holomorphic discrete series occurs). Let fdim λ stand for the formal dimension of T_{λ}, let χ_{λ} denote the character of T_{λ}, which admits a canonical holomorphic extension to Γ^{0} , and let w ↦ w^{#} be the antilinear antiautomorphism of the semigroup that extends the antiautomorhism g ↦ g^{−1} of the group G. Then we have

(0.1)  K(z, w) = ∑_{λ} fdim(λ) · χ_{λ}(w^{#}z),  z, w ∈ Γ^{0}.

Since for both fdim(λ) and χ_{λ} there exist nice formulas, the problem consists in evaluating the series

(0.2)  ∑_{λ} fdim(λ) · χ_{λ}(γ),  γ ∈ Γ^{0}.

The idea of this approach to computing the Cauchy–Szegö kernel was indicated in [4]; there it was illustrated on the simplest example of the group SU(1, 1) = Sp(1, ℝ). In the present paper, the sum (0.2) (and hence the Cauchy– Szegö kernel) is computed for three families of groups G: the symplectic groups Sp(n, ℝ), their two–sheeted coverings (the metaplectic groups) Mp(n, ℝ), and the pseudo–unitary groups SU(p, q) (note that for the first and second families the semigroup Γ is unique, and for the third family only the minimal semigroup is considered). These are the main results of the paper.

4 Note that the kernel is still well–defined if one of the argument lies in the interior Γ^{0} of the semigroup and another argument – on the boundary Γ\Γ^{0} .

Two methods to sum the series (0.2) are presented. The first method is based on a certain preliminary transformation of (0.2), which is effectued in general terms of root data and holds for any G. The second method consists in reducing the problem to a certain combinatorial identity involving finite– dimensional characters of classical groups (a Littlewood–type formula).

The case of G = Sp(n, ℝ) seems to be the simplest one. It can be easily handled by both methods. Moreover, in this case the result also can be obtained by a direct elementary computation: in fact, this computation, performed by my student V. Ivanov, was the starting point of the present work.

The other two cases require more efforts. For G = Mp(n, ℝ) I apply the first method, and for G = SU(p, q) — the second one.

Note that an analogue of the Hardy space H^{2}(Γ) also can be defined when the groups G are replaced by certain pseudo–Riemannian symmetric spaces G/H; then the role of Γ^{0} ⊂ G_{ℂ} is played by a domain in G_{ℂ}/H_{ℂ} (see [8]). In the particular case of the hyperboloids G/H = SO_{0}(1, n)/SO_{0}(1, n − 1) the Cauchy–Szegö kernel was found by Molchanov . The paper [13] by Koufany and Ørsted contains (among other things) a calculation of the Cauchy–Szegö kernel for the group U(1, 1), and in the next papers [14], [15] these authors computed the ‘odd’ part of the kernel for the groups G = Mp(n, ℝ) and G = Spin^{∗}(2n) and also the whole kernel for G = U(p, q). Note that the approach of [13]–[15] is quite different from that of the present paper (it does not use summation of characters).

It would be interesting to continue the study of Cauchy–Szegö kernels for groups and symmetric spaces and to single out the cases when the kernel admits a closed expression.

A related open problem, raised by J. Faraut, is to study (for the same groups and symmetric spaces) Bergman–type spaces of holomorphic functions and the corresponding kernels (as was observed by J. Faraut, the Bergman space on a semigroup Γ can be defined by integrating |f(z)|^{2} with respect to the Haar measure of G_{ℂ}, restricted to Γ^{0} , and a similar definition also holds when G_{ℂ} is replaced by G_{ℂ}/H_{ℂ}).

The paper is organized as follows.

– Sections 1–2 contain basic notations and definitions.

– In Sections 3, I use Harish–Chandra’s result [5] about the formal dimensions of the holomorphic discrete series representations to write down the Cauchy–Szegö kernel as a series involving finite–dimensional characters only (Corollary 3.5).

– Section 4 is devoted to a useful transformation of the series (0.2) (Theorem 4.2). As a corollary, on obtains that the series is essentially a rational function (Theorem 4.3): this is the only result that I could prove for general groups G.

– Sections 5–7 contain proofs of the main results (Theorems 5.1, 6.1, and 7.1).

– In Section 8, I prove two character identities (Lemma 5.2 and Lemma 7.4), which are used in the second proof of Theorem 5.1 and in Theorem 7.1, respectively.

Acknowledgements: I am deeply grateful to Jacques Faraut for his interest to the present work and numerous stimulating discussions on the subject. I am also thankful to Vladimir Ivanov for his direct computation in the symplectic case; to Masao Ishikawa and Masato Wakayama from which I learned a way to prove character identities; and to Khalid Koufany and Bent Ørsted for informing me about the results of the papers [13], [14], [15] before their publication. I would like to emphasize that the idea to examine the metaplectic group was suggested to me by their result in [14]. Special thanks are to Khalid Koufany for his kind assistance in preparing the manuscript.

The present work was supported by the Russian Foundation for Basic Research under grant 95–01–00814.

1. Basic notation

Let 𝔤 be a simple real Lie algebra, let 𝔨 ⊂ 𝔤 be a maximal compact subalgebra, and let 𝔥 ⊂ 𝔨 be a Cartan subalgebra. One supposes that 𝔨 is not semisimple: 𝔨 ≠ [𝔨, 𝔨]; then 𝔨 has a one–dimensional center and 𝔥 also is a Cartan subalgebra for 𝔤.

The Lie algebras 𝔤 with 𝔨 ≠ [𝔨, 𝔨] are called Hermitian Lie algebras^{5} ; recall that there exist four infinite series of classical Hermitian algebras (𝔰𝔲(p, q), 𝔰𝔭(n, ℝ), 𝔬^{∗}(2n), 𝔬(2, n)) and two exceptional ones.

Let the symbol (·)_{ℂ} denote complexification. Fix a system Δ^{+} of positive roots for (𝔤_{ℂ}, 𝔥_{ℂ}); then Δ^{+} is disjoint union of Δ_{c}^{+} and Δ_{n}^{+} , the compact and noncompact roots, respectively.^{6} Let ρ (respectively, ρ_{c}, ρ_{n}) be the half– sum of the roots in Δ^{+} (respectively, Δ_{c}^{+} , Δ_{n}^{+} ); then ρ = ρ_{c} + ρ_{n}. Let W and W_{c} be the Weyl groups for (𝔤_{ℂ}, 𝔥_{ℂ}) and (𝔨_{ℂ}, 𝔥_{ℂ}), respectively.

Put 𝔥_{Re} = i𝔥 ⊂ 𝔥_{ℂ}, let 𝔥_{Re}^{∗} be the dual space, and let ⟨·, ·⟩ be the pairing between 𝔥_{Re}^{∗} and 𝔥_{Re} . Note that Δ^{+} ⊂ 𝔥_{Re}^{∗}. For a root α ∈ Δ^{+} , let α^{∨} ∈ 𝔥_{Re} be the dual root. Let P ⊂ 𝔥_{Re}^{∗} be the lattice of weights for (𝔤_{ℂ}, 𝔥_{ℂ}); recall that ⟨λ, α^{∨}⟩ ∈ ℤ for λ ∈ P and α ∈ Δ^{+} .

Let P^{+} ⊂ P (resp., P_{+} ⊂ P) be the set of dominant weights relative to Δ^{+} (resp., Δ_{c}^{+} ). I.e., these are weights λ ∈ P satisfying the condition ⟨λ, α^{∨}⟩ ≥ 0 for α ∈ Δ^{+} (for α ∈ Δ_{c}^{+} , respectively). In particular, P^{+} is contained in P_{+} . Further, let P_{+−} consists of those weights λ ∈ P_{+} that satisfy the condition ⟨λ + ρ, β^{∨}⟩ < 0 for each β ∈ Δ_{n}^{+} .

Let G_{ℂ} be the connected simply connected complex Lie group corresponding to 𝔤_{ℂ}, and let G, K, H, K_{ℂ}, and H_{ℂ} stand for the (connected) subgroups in G_{ℂ} corresponding to the subalgebras 𝔤, 𝔨, 𝔥, 𝔨_{ℂ}, and 𝔥_{ℂ}, respectively. Note that the group G is not simply connected: its fundamental group is isomorphic to ℤ.

5 Because these are algebras corresponding to Hermitian symmetric spaces.

6 Recall that a root is said to be compact if the corresponding root vector lies in 𝔨_{ℂ}, and noncompact, otherwise.

2. The Cauchy–Szegö kernel

As main references to this section one may use Vinberg [24] and Olshanski [19], [20]. Also see the introductory paper Faraut [2].

We maintain the notation of section 1. The property 𝔨 ≠ [𝔨, 𝔨] implies that in the vector space i𝔤 ⊂ 𝔤_{ℂ} there exist (nontrivial) closed convex Ginvariant cones. Among them there is a unique (within multiplication by −1) minimal cone C_{min} . Let c_{min} ⊂ 𝔥_{Re} denote the closed convex cone spanned by the dual roots β^{∨}, β ∈ Δ_{n}^{+} ; then C_{min} ∩ 𝔥_{Re} is either c_{min} or −c_{min} (depending on agreement), and we may choose the first variant.

Put Γ = G exp C_{min} ⊂ G_{ℂ}; this is a complex semigroup in G_{ℂ}. Let C_{min}^{0} be the interior of the cone C_{min} ⊂ i𝔤; then Γ^{0} := G exp C_{min}^{0} is the interior of Γ ⊂ G_{ℂ}; hence Γ^{0} is a complex variety.

Let O(Γ^{0}) be the space of holomorphic functions on Γ^{0} ; O(Γ^{0}) is equipped with the topology of uniform convergence on compact subsets of Γ^{0} . The Hardy space H^{2}(G) is a certain Hilbert space, continuously embedded into O(Γ^{0}). It consists of those functions f ∈ O(Γ^{0}) for which

∥f∥^{2} := sup_{γ∈Γ^{0}} ∫ |f(gγ)|^{2}dg < ∞.

The reproducing kernel for this Hilbert space is called the Cauchy–Szegö kernel for the group G (or the semigroup Γ). This is a certain function K(γ_{1}, γ_{2}) on Γ^{0} × Γ^{0} , holomorphic in γ_{1} and antiholomorphic in γ_{2} .

Note that K(γ_{1}, γ_{2}) is still well–defined when one of the arguments belongs to the boundary Γ \ Γ^{0} of Γ^{0} . This allows one to put

L(γ) = K(γ, e), γ ∈ Γ^{0},

where e denotes the unity of the group G_{ℂ}. The function L(γ) is a holomorphic function in Γ^{0} , invariant under transformations γ ↦ gγg^{−1} , where g ranges over G. Note that the kernel K(γ_{1}, γ_{2}) can be recovered from the function L(γ):

(2.1)  K(γ_{1}, γ_{2}) = L(γ_{2}^{#}γ_{1}),

where γ ↦ γ^{#} denotes the involutive antiholomorphic antiautomorphism of the semigroup Γ, defined by

(g exp X)^{#} = (exp X)g^{−1} = g^{−1} exp(Ad(g)X).

The objective of the present paper is to calculate, for certain groups G, the Cauchy–Szegö kernel K (or, that is the same, the corresponding function L in one variable).

Let c_{min}^{0} be the interior of the cone c_{min} ⊂ 𝔥_{Re} , let H_{ℂ}^{+} = H exp c_{min}^{0} , and let H_{Re}^{+} = exp c_{min}^{0} . Note that H_{ℂ}^{+} = H_{ℂ} ∩ Γ^{0} .

Lemma 2.1. The function L is uniquely determined by its restriction to H_{ℂ}^{+} , even to H_{Re}^{+} .

Proof. Since L is a holomorphic function on Γ^{0} , it is uniquely determined by its restriction to exp C_{min}^{0} ⊂ Γ^{0} . Now let us regard L as a function on the open cone C_{min}^{0} . Since L is G-invariant and each G-orbit in C_{min}^{0} has a nonempty intersection with c_{min}^{0} ⊂ 𝔥_{Re} (see Vinberg [24], Theorem 5), L is uniquely determined by its restriction to c_{min}^{0} , which completes the proof.

3. The holomorphic discrete series and the Harish–Chandra correspondence

The main reference for this section is Harish–Chandra [5]. See also Faraut [2].

Recall that the holomorphic discrete series for the group G consists of those irreducible unitary representations of G that are square–integrable and possess a highest weight λ. The possible values for λ are exactly the set P_{+−} defined above in section 1. Given λ ∈ P_{+−}, let us denote by T_{λ} the corresponding representation of G, which is uniquely determined by the weight λ. Each T_{λ} admits a (unique) holomorphic extension to a representation T_{λ} of the semigroup Γ, operating in the Hilbert space of T_{λ}. For γ ∈ Γ^{0} , the operator T_{λ}(γ) is contractive (i.e., ∥T_{λ}(γ)∥ ≤ 1) and depends holomorphically on γ. Moreover, T_{λ}(γ) is of trace class. (About holomorphic extensions of highest weight unitary representations see [19].^{7} )

Lemma 3.1. Let fdim T_{λ} denote the formal dimension of T_{λ} (as a square– integrable representation). Then

(3.1)  L(γ) = ∑_{λ∈P_{+−}} fdim T_{λ} · tr T_{λ}(γ),  γ ∈ Γ^{0}.

Note that both fdim T_{λ} and L(γ) depend on the normalization of the Haar measure dg of the group G.

Proof. For each λ ∈ P_{+−}, choose an orthonormal basis ξ_{λ1}, ξ_{λ2}, … in the Hilbert space of T_{λ}. Recall that our Hardy space carries the whole holomorphic discrete series. Therefore, it follows from the orthogonality relations that the (normalized) matrix coefficients

f_{λij}(γ) := (fdim T_{λ})^{1/2}(T_{λ}(γ)ξ_{λi}, ξ_{λj}), γ ∈ Γ^{0},

where λ ∈ P_{+−} and i, j = 1, 2, …, constitute an orthonormal basis in the Hardy space.

It follows that

L(γ) = K(γ, e) = ∑_{λ,i,j} f_{λij}(γ)\overline{f_{λij}(e)}

7 For general Lie groups these topics are discussed in Neeb’s papers [17], [18].

= ∑_{λ,i}(fdim T_{λ})^{1/2}f_{λii}(γ)

= ∑_{λ} fdim T_{λ} ∑_{i}(T_{λ}(γ)ξ_{λi}, ξ_{λi})

= ∑_{λ} fdim T_{λ} · tr T_{λ}(γ).

Formula (3.1) is the starting point for our computation of the function L.

Since P_{+−} ⊂ P_{+} , for each λ ∈ P_{+−} there exists an irreducible finite– dimensional representation π_{λ} of the complex group K_{ℂ} with highest weight λ. For x ∈ H_{ℂ} and λ ∈ P, let us write x^{λ} for the value of λ, viewed as a one– dimensional character of the torus H_{ℂ}, at x. That is to say, x^{λ} = exp⟨λ, log x⟩.

Lemma 3.2. For each λ ∈ P_{+−}

tr T_{λ}(x) = tr π_{λ}(x)θ(x), x ∈ H_{ℂ},

where

(3.2)  θ(x) = ∏_{β∈Δ_{n}^{+}}(1 − x^{−β})^{−1}.

Proof. Assume first T_{λ} is an arbitrary irreducible unitary highest weight representation of G (not necessarily square integrable). Consider the corresponding irreducible unitarizable Harish–Chandra (𝔤, K)-module that is realized in the space of K -finite vectors of T_{λ}. Then this module is either the generalized Verma module M_{λ} (induced from an irreducible finite–dimensional module π_{λ} over a maximal parabolic subalgebra of 𝔤) or the minimal proper quotient L_{λ} of M_{λ}. In the former case T_{λ} is called nondegenerate, and in the latter case it is called degenerate. It is easily verified that for a nondegenerate T_{λ}, the trace of its holomorhic extension T_{λ} is given by the above formula. Finally one applies a Harish–Chandra’s result [5] stating (in our terms) that if T_{λ} is square–integrable then it is nondegenerate.

Let w_{0} ∈ W_{c} stand for the (unique) element of maximal length; note that w_{0}^{2} = 1. Then

w_{0}(Δ_{c}^{+}) = −Δ_{c}^{+}, w_{0}(Δ_{n}^{+}) = Δ_{n}^{+},

whence

w_{0}(ρ_{c}) = −ρ_{c}, w_{0}(ρ_{n}) = ρ_{n}.

Define φ : P → P by

φ(λ) = −w_{0}(λ + ρ) − ρ = −w_{0}(λ) − 2ρ_{n}.

Note that

−w_{0}(λ + ρ) = φ(λ) + ρ,

so φ is an involutive map of P.

Lemma 3.3. One has φ(P_{+−}) = P^{+} and φ(P^{+}) = P_{+−}, so that φ defines a bijective correspondence P_{+−} ↔ P^{+} .

Proof. Remark that P^{+} and P_{+−} may be described as follows:

P^{+} = {μ ∈ P | ⟨μ + ρ, α^{∨}⟩ > 0 for each α ∈ Δ^{+} }

P_{+−} = {λ ∈ P | ⟨λ + ρ, α^{∨}⟩ > 0 for α ∈ Δ_{c}^{+} and ⟨λ + ρ, β^{∨}⟩ < 0 for β ∈ Δ_{n}^{+} }

Now the claim of the Lemma follows from the fact that

−w_{0}(Δ_{c}^{+}) = Δ_{c}^{+}, −w_{0}(Δ_{n}^{+}) = −Δ_{n}^{+}.

For μ ∈ P^{+} , let V_{μ} denote the irreducible finite–dimensional (complex– analytic) representation of the group G_{ℂ} with highest weight μ, and let dim V_{μ} denote the dimension of V_{μ}.

Lemma 3.4. For an appropriate normalization of the Haar measure dg on the group G,

(3.3)  fdim T_{λ} = dim V_{φ(λ)}  for each λ ∈ P_{+−} .

Proof. For a proof, see Harish–Chandra [5], paper VI, Theorem 4.

It is natural to call the correspondence P_{+−} ↔ P^{+} or {T_{λ}} ↔ {V_{φ(λ)}} the Harish–Chandra correspondence.

Throughout the paper we assume that the normalization of the Haar measure is chosen so that relation (3.3) does hold.

Lemmas 3.1–3.4 imply the following

Corollary 3.5. For each x ∈ H_{ℂ}^{+}

(3.4)  L(x) = θ(x) ∑_{μ∈P^{+}} dim V_{μ} · tr π_{φ(μ)}(x),

where θ(x) is defined by (3.2).

Example 3.6. Consider the simplest case G = SU(1, 1). Then each element x ∈ H_{ℂ}^{+} may be viewed as the 2 × 2 diagonal matrix with the diagonal entries x^{−1} , x, where x now stands for a complex number, |x| < 1. Then the series (3.4) turns into

(3.5)  L(x) = (1 − x^{2})^{−1} ∑_{m=0}^{∞}(m + 1)\frac{x^{m+2}}{1 − x^{2}} = \frac{x^{2}}{(1 − x)^{3}(1 + x)}.

An equivalent expression can be found in Gelfand and Gindikin [4] (see also Faraut [2]).

4. An expression for the function L

Let ε : W → {±1} denote the standard multiplicative function,

ε(w) = (−1)^{length(w)}.

We shall need Weyl’s character formulas for the groups G_{ℂ} and K_{ℂ}:

(4.1)  tr V_{μ}(z) = \frac{∑_{w∈W} ε(w)w(z)^{μ+ρ}}{∑_{w∈W} ε(w)w(z)^{ρ}} ,  z ∈ H_{ℂ}, μ ∈ P^{+},

(4.2)  tr π_{λ}(z) = \frac{∑_{w′∈W_{c}} ε(w′)w′(z)^{λ+ρ}}{∑_{w′∈W_{c}} ε(w′)w′(z)^{ρ}} ,  z ∈ H_{ℂ}, λ ∈ P_{+}.

(It should be noted that there is a small subtlety in formula (4.2), because the group K_{ℂ} is not semisimple and ρ is distinct from ρ_{c}, the object associated with the semisimple group [K_{ℂ}, K_{ℂ}]. But it is not difficult to derive (4.2) from Weyl’s formula for the semisimple group [K_{ℂ}, K_{ℂ}].)

Let D_{G} and D_{K} denote the denominators in (4.1) and (4.2), respectively:

(4.3)  D_{G}(z) = ∑_{w∈W} ε(w)w(z)^{ρ} = ∏_{α∈Δ^{+}}(z^{α/2} − z^{−α/2}),

(4.4)  D_{K}(z) = ∑_{w′∈W_{c}} ε(w′)w′(z)^{ρ} = z^{ρ_{n}} ∏_{α∈Δ_{c}^{+}}(z^{α/2} − z^{−α/2}).

These are regular functions on the torus H_{ℂ}.

(Strictly speaking, in (4.1) and (4.2) one should assume that D_{G}(z) ≠ 0 and D_{K}(z) ≠ 0, respectively.)

Let Ω ⊂ 𝔥_{Re}^{∗} be the dual cone of the cone c_{min} , i.e.,

(4.5)  Ω = {ν ∈ 𝔥_{Re}^{∗} | ⟨ν, β^{∨}⟩ ≥ 0  for each β ∈ Δ_{n}^{+} }.

Since P is a lattice in the vector space 𝔥_{Re}^{∗}, the intersection Ω∩P is an additive monoid.

Lemma 4.1. Suppose z ∈ H_{ℂ} is such that z^{−1} ∈ H_{ℂ}^{+} . Then the sum

(4.6)  F (z) := ∑_{ν∈Ω∩P} z^{ν}

is absolutely convergent. (One could call F (z) the characteristic function of the monoid Ω ∩ P.) Proof. Write z = ux where u ∈ H, x = exp(−X), and X ∈ c_{min}^{0} . Then |z^{ν}| = exp(−⟨ν, X⟩). Since the linear functional ⟨·, X⟩ is strictly positive on Ω \ {0}, the claim follows from obvious estimates.

Let x ↦ x̂ be the involutive map of the torus H_{ℂ} defined by

x̂ = w_{0}(x)^{−1}.

Note that if x ∈ H_{ℂ}^{+} then the function F (z) is well–defined in a neighborhood of the element x̂ in H_{ℂ}.

Theorem 4.2. Let the group G satisfy the assumption of section 1 and let L(x) be the function on H_{ℂ}^{+} , associated with the Cauchy–Szegö kernel of the Hardy space H^{2}(G). Let x ∈ H_{ℂ}^{+} be such that D_{K}(x) ≠ 0, and let t range over the subset of elements of H_{ℂ} satisfying D_{G}(t) ≠ 0. Then

(4.7)  L(x) = \frac{θ(x)}{D_{K}(x)} lim_{t→e} \frac{1}{D_{G}(t)} ∑_{w∈W} ε(w)F (w(t)x̂),

where F is the function on Ω ∩ P, defined in (4.6).

Proof. We shall transform formula (3.4) of Corollary 3.5. From

dim V_{μ} = tr V_{μ}(e) = lim_{t→e} tr V_{μ}(t),

it follows

(4.8)  dim V_{μ} = lim_{t→e} \frac{1}{D_{G}(t)} ∑_{w∈W} ε(w)(w(t))^{μ+ρ},

by virtue of (4.1) and (4.3).

Further,

(4.9)  tr π_{φ(μ)}(x) = \frac{1}{D_{K}(x)} ∑_{w′∈W_{c}} ε(w′)(w′(x))^{φ(μ)+ρ},

by virtue of (4.2) and (4.4).

Let us rewrite the right–hand side of (4.9) by making use of the definition of φ and x̂:

(w′(x))^{φ(μ)+ρ} = (w′(x))^{−w_{0}(μ+ρ)}
= ((w_{0}w′)(x^{−1}))^{μ+ρ}
= ((w_{0}w′w_{0})(w_{0}(x)^{−1}))^{μ+ρ}
= ((w_{0}w′w_{0})(x̂))^{μ+ρ}.

Since ε(w′) = ε(w_{0}w′w_{0}), one may replace w_{0}w′w_{0} by w′, and finally we obtain

(4.10)  tr π_{φ(μ)}(x) = \frac{1}{D_{K}(x)} ∑_{w′∈W_{c}} ε(w′)(w′(x̂))^{μ+ρ}.

By substituting the expressions (4.8) and (4.10) into formula (3.4) of Corollary 3.5 we obtain

L(x) = \frac{θ(x)}{D_{K}(x)} lim_{t→e} \frac{1}{D_{G}(t)} ∑_{μ∈P^{+}} ∑_{w∈W} ∑_{w′∈W_{c}} ε(w)ε(w′)(w(t)w′(x̂))^{μ+ρ}.

After a change of variables, (w, w′) ↦ (w′w, w′), this turns into

(4.11)  L(x) = \frac{θ(x)}{D_{K}(x)} lim_{t→e} \frac{1}{D_{G}(t)} ∑_{μ∈P^{+}} ∑_{w∈W} ∑_{w′∈W_{c}} ε(w)(w(t)x̂)^{w′(μ+ρ)}.

Let P_{reg} denote the set of regular weights,

P_{reg} = {ν ∈ P | ⟨ν, α^{∨}⟩ ≠ 0 for each α ∈ Δ^{+} }.

When μ runs through P^{+} , the weight μ + ρ runs through P^{+} ∩ P_{reg} . Since the sets of the form w′(P^{+} ∩ P_{reg}) (where w′ ∈ W_{c}) are pairwise disjoint, we have

∑_{w′∈W_{c}} ∑_{μ∈P^{+}}(w(t)x̂)^{w′(μ+ρ)} = ∑_{{ν}}(w(t)x̂)^{ν},

where

{ν} := ⋃_{w′∈W_{c}} w′(P^{+} ∩ P_{reg}).

The set {ν} can be written as

{ν ∈ P_{reg} | ⟨ν, β^{∨}⟩ ≥ 0 for each β ∈ Δ_{n}^{+} },

from which it follows that

{ν} = Ω ∩ P_{reg}.

Therefore,

L(x) = \frac{θ(x)}{D_{K}(x)} lim_{t→e} \frac{1}{D_{G}(t)} ∑_{w∈W} ε(w) ∑_{ν∈Ω∩P_{reg}}(w(t)x̂)^{ν}.

Now we remark that

∑_{w∈W} ε(w)w(t)^{ν} = 0 if ν ∈ P \ P_{reg} .

Hence summation over ν ∈ Ω ∩ P_{reg} can be replaced by that over ν ∈ Ω ∩ P. By virtue of the definition (4.6) of the function F we obtain the desired formula.

The argument of Theorem 4.2 implies the following corollary:

Theorem 4.3. Let the group G satisfy the assumption of section 1, and let L(x) be the function on H_{ℂ}^{+} = H_{ℂ} ∩Γ^{0} , associated with the Cauchy–Szegö kernel of the Hardy space H^{2}(G). Then L(x) is restriction to H_{ℂ}^{+} of a rational function on the complex torus H_{ℂ}.

Proof. Probably, the first idea which occurs when looking at formula (4.7) of Theorem 4.2 is to show that the ‘characteristic function’ (4.6) of the additive monoid Ω ∩ P is rational. However, the structure of this monoid seems to be cumbersome, so it is better to use, instead of (4.7), formula (4.11) (an advantage of the latter formula is that the structure of the monoid P^{+} is very simple).

It is immediate from the definition of θ, D_{G}, and D_{K} (see (3.2), (4.3), and (4.4)) that these are rational functions. Since both w and w′ in (4.11) range over finite sets, it suffices to prove that the series

(4.12)  ∑_{μ∈P^{+}} z^{μ+ρ},  ẑ ∈ H_{ℂ}^{+},

is a rational function.

Let ω_{1}, …, ω_{r} stand for the fundamental weights of the system Δ^{+} , where r is the rank of 𝔤_{ℂ}. Then we have

P^{+} = ℤ_{+}ω_{1} + … + ℤ_{+}ω_{r},

whence

∑_{μ∈P^{+}} z^{μ+ρ} = z^{ρ} ∑_{μ∈P^{+}} z^{μ} = z^{ρ} ∏_{i=1}^{r} \frac{1}{1 − z^{ω_{i}}} ,

which is a rational function.

5. The case of Sp(n, ℝ)

Throughout this section one assumes G = Sp(n, ℝ) and G_{ℂ} = Sp(n, ℂ). Let

ξ = (ξ_{1}, …, ξ_{2n})  and  η = (η_{1}, …, η_{2n})

be arbitrary vectors in ℂ^{2n}. Let us regard G_{ℂ} = Sp(n, ℂ) as the group of automorphisms of the alternating form

ξ_{1}η_{2n} + … + ξ_{n}η_{n+1} − ξ_{n+1}η_{n} − … − ξ_{2n}η_{1},

and realize the group G = Sp(n, ℝ) as the intersection U(n, n) ∩ Sp(n, ℂ) where U(n, n) is realized as the group preserving the Hermitian form

(5.1)  [ξ, η] = −ξ_{1}η̄_{1} − … − ξ_{n}η̄_{n} + ξ_{n+1}η̄_{n+1} + … + ξ_{2n}η̄_{2n}.

The subgroup K, which is isomorphic to U(n), is realized as the subgroup of G that fixes the direct sum decomposition ℂ^{2n} = ℂ^{n} ⊕ ℂ^{n}.

Given complex numbers z_{1}, …, z_{2n}, let us write diag(z_{1}, …, z_{2n}) for the diagonal matrix with diagonal entries z_{1}, …, z_{2n}.

The space 𝔥_{Re} consists of the real matrices of the form

X = diag(X_{1}, …, X_{n}, −X_{n}, …, −X_{1});

it will be identified with ℝ^{n} via the map X ↦ (X_{1}, …, X_{n}) ∈ ℝ^{n}. The dual space 𝔥_{Re}^{∗} may be identified with 𝔥_{Re} and hence with ℝ^{n}, too.

Let ϵ_{1}, …, ϵ_{n} be the canonical basis of 𝔥_{Re}^{∗} = ℝ^{n}. Then

Δ_{c}^{+} = {ϵ_{i} − ϵ_{j} (1 ≤ i < j ≤ n)},
Δ_{n}^{+} = {ϵ_{i} + ϵ_{j} (1 ≤ i ≤ j ≤ n)},
Δ^{+} = {ϵ_{i} ± ϵ_{j} (1 ≤ i < j ≤ n), 2ϵ_{i} (1 ≤ i ≤ n)},
ρ = (n, n − 1, …, 1),
ρ_{c} = (\frac{n−1}{2}, \frac{n−3}{2}, …, −\frac{n−3}{2}, −\frac{n−1}{2}),
ρ_{n} = (\frac{n+1}{2}, …, \frac{n+1}{2}).

The weight lattice P ⊂ 𝔥_{Re}^{∗} = ℝ^{n} is simply the integer lattice ℤ^{n} ⊂ ℝ^{n}, and the set P^{+} ⊂ P is identified with the set of partitions of length ≤ n,

P^{+} = {μ ∈ ℤ^{n} | μ_{1} ≥ … μ_{n} ≥ 0}.

The Weyl group W_{c} is isomorphic to the symmetric group 𝔖_{n} of degree n; it operates on 𝔥_{Re} = 𝔥_{Re}^{∗} = ℝ^{n} by permuting the basis vectors ϵ_{1}, …, ϵ_{n}. The Weyl group W is isomorphic to the hyperoctahedral group ℌ_{n}, the semidirect product of 𝔖_{n} with the Abelian group 𝔄_{n} = ℤ_{2}^{n}; the i-th generator of 𝔄_{n} multiplies the i-th basis vector ϵ_{i} by −1 and fixes all remaining vectors ϵ_{j}, j ≠ i.

The cone c_{min} ⊂ 𝔥_{Re} = ℝ^{n} is the ‘hyperoctant’ ℝ_{+}^{n} ⊂ ℝ^{n}, and the dual cone Ω also coincides with ℝ_{+}^{n}.

The complex torus H_{ℂ} consists of the diagonal matrices of the form

diag(z_{1}, …, z_{n}, z_{n}^{−1}, …, z_{1}^{−1})

with nonzero complex z_{1}, …, z_{n}. The subset H_{ℂ}^{+} ⊂ H_{ℂ} is described by the inequalities |z_{1}| > 1, …, |z_{n}| > 1.

The transformation z → ẑ = w_{0}(z)^{−1} of the torus H_{ℂ} takes the form

diag(z_{1}, …, z_{n}, z_{n}^{−1}, …, z_{1}^{−1}) ↦ diag(z_{n}^{−1}, …, z_{1}^{−1}, z_{1}, …, z_{n}).

A complex 2n × 2n matrix γ is called weakly J -contractive if [γξ, γξ] ≤ [ξ, ξ] for each ξ ∈ ℂ^{2n}, where [·, ·] is the indefinite inner product (5.1). If the above inequality is strict for each ξ ≠ 0, then γ is called strictly J -contractive. The semigroup Γ is formed by all weakly J -contractive matrices in Sp(n, ℂ), and the interior Γ^{0} ⊂ Γ consists of strictly J -contractive symplectic matrices.

Note that the eigenvalues of each matrix γ ∈ Γ^{0} are of the form

(5.2)  x_{1}, x_{1}^{−1}, …, x_{n}, x_{n}^{−1},  where |x_{1}|, …, |x_{n}| < 1.

Theorem 5.1. Let K(γ_{1}, γ_{2}) be the Cauchy–Szegö kernel for the Hardy space H^{2}(Sp(n, ℝ)) ⊂ O(Γ^{0}), where Γ^{0} is the open semigroup of strictly J -contractive matrices in Sp(n, ℂ). Let L(γ) = K(γ, e) be the corresponding function in one variable γ ∈ Γ^{0} . Given γ ∈ Γ^{0} , write its eigenvalues as in (5.2). Finally, recall that the Haar measure on the group is normalized so that the relation (3.3) holds.

Then

(5.3)  L(γ) = ∏_{i=1}^{n} \frac{x_{i}^{n+1}}{(1 + x_{i})(1 − x_{i})^{2n+1}} .

(Note that in the case n = 1 the expression (5.3) agrees with formula (3.5).)

This result can be derived in several ways. First this was done by Vladimir Ivanov (a student of the Moscow State University) by a direct calculation. Namely, I suggested him to evaluate the sum in the left–hand side of (5.13) (see below), where the dimension of an irreducible Sp(n, ℂ)-module is expressed by classical Weyl’s formula. When he obtained formula (5.13) I realized that this is in fact a corollary of the well–known character identity (5.12).

I give below two proofs of the theorem: the first proof is based on Theorem 4.2, while the second proof is simply reduction to the identity (5.12).

First Proof. Remark that the right–hand side of (5.3) is a holomorphic function on Γ^{0} , invariant under conjugation by elements of the group G. Then the argument of Lemma 2.1 shows that it suffices to check (5.3) for diagonal matrices γ ∈ H_{ℂ} ∩ Γ^{0} = H_{ℂ}^{+} . Hence one may assume

γ = x = diag(x_{n}^{−1}, …, x_{1}^{−1}, x_{1}, …, x_{n}), |x_{1}|, …, |x_{n}| < 1,

whence

x̂ = diag(x_{1}, …, x_{n}, x_{n}^{−1}, …, x_{1}^{−1}).

Further, by (3.2),

(5.4)  θ(x) = ∏_{i≤j}(1 − x_{i}x_{j})^{−1} = ∏_{i<j}(1 − x_{i}x_{j})^{−1} ∏_{i}(1 − x_{i}^{2})^{−1}

and, by (4.4),

\begin{aligned}D_{K}(x) & = (x_{1} … x_{n})^{−(n+1)/2} ∏_{i<j}(x_{j}^{−1/2}x_{i}^{1/2} − x_{j}^{1/2}x_{i}^{−1/2}) \\ & = (x_{1} … x_{n})^{−n} ∏_{i<j}(x_{i} − x_{j}), \end{aligned}

whence

(5.5)  \frac{θ(x)}{D_{K}(x)} = \frac{(x_{1} … x_{n})^{n}}{∏_{i<j}(x_{i} − x_{j}) ∏_{i<j}(1 − x_{i}x_{j}) ∏_{i}(1 − x_{i}^{2})} .

Let

t = diag(t_{1}, …, t_{n}, t_{n}^{−1}, …, t_{1}^{−1}) ∈ H_{ℂ}.

Then, by (4.3),

(5.6)  \begin{aligned}D_{G}(t) & = ∏_{i<j}(t_{i}^{1/2}t_{j}^{−1/2} − t_{i}^{−1/2}t_{j}^{1/2}) ∏_{i≤j}(t_{i}^{1/2}t_{j}^{1/2} − t_{i}^{−1/2}t_{j}^{−1/2}) \\ & = ∏_{i<j}(t_{i} + t_{i}^{−1} − t_{j} − t_{j}^{−1}) ∏_{i}(t_{i} − t_{i}^{−1}) \end{aligned} .

As the monoid Ω∩P coincides with ℤ_{+}^{n} ⊂ ℝ^{n}, its ‘characteristic function’ F (z) is given by

(5.7)  F (diag(z_{1}, …, z_{n}, z_{n}^{−1}, …, z_{1}^{−1})) = ∏_{i}(1 − z_{i})^{−1}.

For w ∈ W = ℌ_{n} and t ∈ H_{ℂ} as above, let w(t)_{i} be the i-th diagonal entry of the matrix w(t), i = 1, …, n. By Theorem 4.2 and formulas (5.5)–(5.7),

(5.8)  \begin{aligned}L(x) & = \frac{(x_{1} … x_{n})^{n}}{∏_{i<j}(x_{i} − x_{j}) ∏_{i<j}(1 − x_{i}x_{j}) ∏_{i}(1 − x_{i}^{2})} \\ & · lim_{t→e} \frac{1}{∏_{i<j}(t_{i} + t_{i}^{−1} − t_{j} − t_{j}^{−1}) ∏_{i}(t_{i} − t_{i}^{−1})} \\ & · ∑_{w∈ℌ_{n}} ε(w) ∏_{i} \frac{1}{1 − w(t)_{i}x_{i}} . \end{aligned}

To evaluate the latter sum, let us write it as

(5.9)  ∑_{s∈𝔖_{n}} ε(s) ∑_{a∈𝔄_{n}} ε(a) ∏_{i} \frac{1}{1 − (a(s(t)))_{i}x_{i}} .

First, let us fix s ∈ 𝔖_{n} and denote

u_{1} = s(t)_{1}, …, u_{n} = s(t)_{n} ,

so that (u_{1}, …, u_{n}) is a permutation of (t_{1}, …, t_{n}), determined by s. Then the interior sum in (5.9) equals

∏_{i} (\frac{1}{1 − u_{i}x_{i}} − \frac{1}{1 − u_{i}^{−1}x_{i}} )

= ∏_{i} x_{i} ∏_{i}(u_{i} − u_{i}^{−1}) ∏_{i} \frac{1}{(1 − u_{i}x_{i})(1 − u_{i}^{−1}x_{i})}

= ∏_{i} x_{i} ∏_{i}(t_{i} − t_{i}^{−1}) ∏_{i} \frac{1}{(1 − u_{i}x_{i})(1 − u_{i}^{−1}x_{i})} .

Let us substitute this expression into (5.9) and recall that u_{1}, …, u_{n} are a permutation of t_{1}, …, t_{n}. It follows that alternating over s ∈ 𝔖_{n} gives an n × n determinant, so that (5.9) turns into

(5.10)  ∏_{i} x_{i} ∏_{i}(t_{i} − t_{i}^{−1}) det [\frac{1}{(1 − t_{j}x_{i})(1 − t_{j}^{−1}x_{i})}]_{1≤i,j≤n} .

The determinant in (5.10) can be easily computed by reduction to the well–known Cauchy determinant (see Littlewood [16], Chap. XI, 11.7, Lemma III). The result is as follows

(5.11)  \begin{aligned}det & [\frac{1}{(1 − t_{j}x_{i})(1 − t_{j}^{−1}x_{i})}]_{1≤i,j≤n} \\ & = \frac{∏_{i<j}(t_{i} + t_{i}^{−1} − t_{j} − t_{j}^{−1}) ∏_{i<j}(x_{i} − x_{j}) ∏_{i<j}(1 − x_{i}x_{j})}{∏_{i} ∏_{j}(1 − t_{j}x_{i})(1 − t_{j}^{−1}x_{i})}. \end{aligned}

Let us substitute this into (5.10) and next substitute the resulting expression into (5.8), instead of the sum. Then we obtain, after cancellations,

\begin{aligned}L(x) & = \frac{(x_{1} … x_{n})^{n+1}}{∏_{i}(1 − x_{i}^{2})} lim_{t_{1},...,t_{n}→1} ∏_{i} ∏_{j} \frac{1}{(1 − t_{j}x_{i})(1 − t_{j}^{−1}x_{i})} \\ & = \frac{(x_{1} … x_{n})^{n+1}}{∏_{i}(1 − x_{i}^{2})(1 − x_{i})^{2n}} , \end{aligned}

which coincides with (5.3).

Another way to establish formula (5.3) is to derive it from the following well–known character identity (which is one of the the so–called Littlewood formulas):

Lemma 5.2. Let M = (M_{1} ≥ M_{2} ≥ … ≥ M_{n} ≥ 0) range over the set of partitions of length ≤ n; let s_{M}(x_{1}, …, x_{M}) be the Schur function in n variables, indexed by M; and let sp(n)_{M}(t_{1}^{±1}, …, t_{n}^{±1}) be the character of the irreducible finite–dimensional Sp(n, ℂ)-module V_{M} that is indexed by M (one regards the character as a function on the complex torus H_{ℂ} ⊂ Sp(n, ℂ) of diagonal matrices diag(t_{1}, …, t_{n}, t_{n}^{−1}, …, t_{1}^{−1})). Then

(5.12) ∑_{M} sp(n)_{M}(t_{1}^{±1}, …, t_{n}^{±1})s_{M}(x_{1}, …, x_{n}) = \frac{∏_{i<j}(1 − x_{i}x_{j})}{∏_{i} ∏_{j}(1 − t_{j}x_{i})(1 − t_{j}^{−1}x_{i})} .

Proof. See Koike and Terada [12], Lemma 1.5.1, or Sundaram [21]. Another argument, due to M. Ishikawa and M. Wakayama (personal communication) is presented below in section 8. Finally, note that the identity (5.12) also can be derived from the explicit weight correspondence in Howe’s duality between the compact group Sp(k) and noncompact group SO^{∗}(2n), for the special case k = n. (See Howe [9] and Enright, Howe and Wallach [1].)

Corollary 5.3. Let M, s_{M}, and V_{M} be as in Lemma 5.2. Then

(5.13)  ∑_{M} dim V_{M} s_{M}(x_{1}, …, x_{n}) = \frac{∏_{i<j}(1 − x_{i}x_{j})}{∏_{i}(1 − x_{i})^{2n}} .

Proof. Take t_{1} = … = t_{n} = 1 in (5.12).

Second Proof of Theorem 5.1. Look at formula (3.4) of Corollary 3.5. We shall see that its right–hand side coincides (up to certain simple factors) with the left–hand side of (5.13), so that formula (5.3) is equivalent to formula (5.13).

Indeed, one may view each dominant weight μ ∈ P^{+} as a partition M of length ≤ n: the correspondence μ ↔ M is given by

μ = diag(M_{1}, …, M_{n}, −M_{n}, …, −M_{1}) ∈ 𝔥_{Re}^{∗}.

In this notation,

φ(μ) = diag(−M_{n}, …, −M_{1}, M_{1}, …, M_{n}) + (n + 1) diag(−1, …, −1, 1, …, 1).

Let as above

γ = x = diag(x_{n}^{−1}, …, x_{1}^{−1}, x_{1}, …, x_{n}).

Then

tr π_{φ(μ)}(x) = (x_{1} … x_{n})^{n+1}s_{M}(x_{1}, …, x_{n}).

Therefore, formula (3.4) turns into

L(x) = θ(x)(x_{1} … x_{n})^{n+1} ∑_{M} dim V_{M} s_{M}(x_{1}, …, x_{n}).

By using the expression (5.4) for θ(x) we obtain

L(x) = \frac{(x_{1} … x_{n})^{n+1}}{∏_{i<j}(1 − x_{i}x_{j}) ∏_{i}(1 − x_{i}^{2})} ∑_{M} dim V_{M}s_{M}(x_{1}, …, x_{n}).

It follows that (5.3) and (5.13) are equivalent.

6. The case of G̃ = Mp(n, ℝ)

Consider the metaplectic group G̃ = Mp(n, ℝ), which is the two–sheeted covering over the symplectic group G = Sp(n, ℝ). Let Γ̃ and Γ̃^{0} be the two–sheeted coverings over Γ and Γ^{0} , respectively (recall that Γ is the semigroup of J - contractive matrices in Sp(n, ℂ)). The Hardy space H^{2}(G̃) ⊂ O(Γ̃^{0}) can be defined in exactly the same manner as for the group G (or other linear groups; the lack of a global complexification for G̃ plays no role here). Let K̃(γ̃_{1}, γ̃_{2}) be the Cauchy–Szegö kernel on Γ̃^{0} × Γ̃^{0} corresponding to that Hardy space, and let L̃(γ̃) = K̃(γ̃, e) be the associated function in one variable γ̃ ∈ Γ̃^{0} .

The following theorem was obtained after I learned about the idea of K. Koufany and B. Ørsted to consider the ‘odd’ part of the Hardy space on the metaplectic group.

Theorem 6.1. Let γ̃ ∈ Γ̃^{0} , let γ be the image of γ̃ in Γ^{0} , and let x_{1}^{±1}, …, x_{n}^{±1} be the eigenvalues of γ, where one assumes that |x_{1}|, …, |x_{n}| < 1. Then

(6.1)  L̃(γ̃) = L(γ) + L(γ)\frac{∏_{i=1}^{n}(1 + x_{i})}{2^{n}(x_{1} … x_{n})^{1/2}} ,

where L(γ) denotes the function computed in Theorem 5.1.

Note that the function (x_{1} ⋯ x_{n})^{1/2} is well–defined on the covering Γ̃^{0} . Note also that formula (6.1) may be written as follows

(6.2)  L̃(γ̃) = \frac{(x_{1} ⋯ x_{n})^{n+1}}{∏_{i=1}^{n}(1 + x_{i})(1 − x_{i})^{2n+1}} + \frac{(x_{1} ⋯ x_{n})^{n+1/2}}{2^{n} ∏_{i=1}^{n}(1 − x_{i})^{2n+1}} .

When n = 1, the latter expression turns into

L̃(γ̃) = \frac{x^{2}}{(1 + x)(1 − x)^{3}} + \frac{x^{3/2}}{2(1 − x)^{3}} (n = 1, x = x_{1}, |x| < 1),

which can be verified by summing the series

∑_{m≥0}(m + 1)\frac{x^{m+2}}{1 − x^{2}} + ∑_{m≥0}(m + 1/2)\frac{x^{m+3/2}}{1 − x^{2}} .

Proof. We shall slightly modify the basic notation of section 1 and the preliminary results of sections 3 and 4; then we shall argue as in the first proof of Theorem 5.1.

Let K̃ ⊂ G̃ be the inverse image of the maximal compact subgroup K ⊂ G; when K is identified with U(n), the group K̃ is identified with the double covering over U(n) making the function det(·)^{1/2} single–valued. Let H̃ ⊂ K̃ be the corresponding covering over H ⊂ K, and let H̃_{ℂ} ⊃ H̃ be the complex torus complexifying the compact torus H̃. The (additive) group of characters of H̃ (or H̃_{ℂ}) is identified with the lattice

P̃ := P ∪ (P + ϵ) ⊂ 𝔥_{Re}^{∗} = ℝ^{n},

where ϵ = (\frac{1}{2}, ⋯, \frac{1}{2}) ∈ ℝ^{n}.

Let P̃_{+−} be the set of those weights λ ∈ P̃ that satisfy two conditions: first, ⟨λ, α^{∨}⟩ ∈ ℤ_{+} for each α ∈ Δ_{c}^{+} and, second, ⟨λ + ρ, β^{∨}⟩ < 0 for each β ∈ Δ_{n}^{+} . This can be restated as follows:

P̃_{+−} = {λ ∈ P̃ | ⟨λ + ρ, α^{∨}⟩ > 0 ∀α ∈ Δ_{c}^{+} ; ⟨λ + ρ, β^{∨}⟩ < 0 ∀β ∈ Δ_{n}^{+}},

from which it becomes clear that the set P̃^{+} := φ(P̃_{+−}) can be described as follows:

P̃^{+} = {μ ∈ P̃ | ⟨μ + ρ, α^{∨}⟩ > 0 ∀α ∈ Δ^{+}}.

The holomorphic discrete series {T_{λ}} for the metaplectic group G̃ is parametrized by the weights λ ∈ P̃_{+−}. Each T_{λ} can be extended to a holomorphic representation T_{λ} of the covering semigroup Γ̃, and the analogue of Lemma 3.1 reads as follows:

L̃(γ̃) = ∑_{λ∈_{P_{+−}}} fdim T_{λ} · tr T_{λ}(γ̃),  γ̃ ∈ Γ̃^{0}.

It is convenient to split this sum into two parts:

L̃(γ̃) = L̃_{even}(γ̃) + L̃_{odd}(γ̃),

where

L̃_{even}(γ̃) = ∑_{λ∈P_{+−}} fdim T_{λ} tr T_{λ}(γ̃),
L̃_{odd}(γ̃) = ∑_{λ∈_{P_{+−}}\P_{+−}} fdim T_{λ} tr T_{λ}(γ̃).

Note that this splitting exactly corresponds to the splitting

H^{2}(G̃) = H_{even}^{2}(G̃) ⊕ H_{odd}^{2}(G̃) ⊂ O(Γ̃^{0}) = O_{even}(Γ̃^{0}) ⊕ O_{odd}(Γ̃^{0}),

where ‘even’ functions are assumed to be constant on fibers of canonical projection Γ̃^{0} → Γ^{0} whereas ‘odd’ functions take opposite values on each fiber. Thus, L̃_{even} and L̃_{odd} determine the Cauchy-Szegö kernels for H_{even}^{2}(G̃) and H_{odd}^{2}(G̃), respectively.

Clearly L̃_{even}(γ̃) = L(γ), so that formula (6.1) is equivalent to the following one:

(6.3)  L̃_{odd}(γ̃) = L(γ)\frac{∏_{i=1}^{n}(1 + x_{i})}{2^{n}(x_{1} ⋯ x_{n})^{1/2}} .

Let μ range over the set

P̃^{+} \ P^{+} = {μ ∈ P + ϵ | ⟨μ + ρ, α^{∨}⟩ > 0 ∀α ∈ Δ^{+}}.

We cannot write fdim T_{φ(μ)} = dim V_{μ} as before, because there is no finite– dimensional representation with highest weight μ (for there is no double covering over the group Sp(n, ℂ)!), so a claim similar to that of Lemma 3.4 literally fails. Nevertheless, by Harish-Chandra’s result cited above ([5], paper VI, Theorem 4), the formal dimension fdim T_{φ(μ)} is still given by the same analytic expression as for weights μ ∈ P^{+} :

fdim T_{φ(μ)} = \frac{∏_{α∈Δ^{+}}⟨μ + ρ, α^{∨}⟩}{∏_{α∈Δ^{+}}⟨ρ, α^{∨}⟩}

= lim_{t→e} \frac{∑_{w∈W} ε(w)(w(t))^{μ+ρ}}{D_{G}(t)} .

As for Lemma 3.2, it needs no change.

Now one can repeat the argument of Theorem 4.2, which leads to the following result:

L̃_{odd}(γ̃) = \frac{θ(x)}{D_{K}(x)} lim_{t→e} \frac{1}{D_{G}(t)} ∑_{w∈W} ε(w)F̃_{odd}(w(t)x̂),

where, for z = diag(z_{1}, …, z_{n}, z_{n}^{−1}, …, z_{1}^{−1}),

F̃_{odd}(z) = ∑_{ν∈Ω∩(P +ϵ)} z^{ν} = (z_{1} ⋯ z_{n})^{1/2}F (z) = ∏_{i=1}^{n} \frac{z_{i}^{1/2}}{1 − z_{i}} .

Next we repeat the argument of the first proof of Theorem 5.1. The only new point is that the sum in (5.8) (which is rewritten in (5.9)) must be replaced by the following expression:

(6.4)  ∑_{ν∈𝔖_{n}} ε(s) ∑_{a∈𝔄_{n}} ε(a)\frac{(a(u_{i})x_{i})^{1/2}}{1 − a(u_{i})x_{i}} ,

where, as before, (u_{1}, …, u_{n}) stands for the permutation of (t_{1}, …, t_{n}) determined by s ∈ 𝔖_{n}. The interior sum in (6.4) is equal to

\begin{aligned}∏_{i=1}^{n} & (\frac{u_{i}^{1/2}x_{i}^{1/2}}{1 − u_{i}x_{i}} − \frac{u_{i}^{−1/2}x_{i}^{1/2}}{1 − u_{i}^{−1}x_{i}} ) \\ & = ∏_{i} x_{i}^{1/2}(1 + x_{i}) & ∏_{i}(u_{i}^{1/2} − u_{i}^{−1/2}) ∏_{i} \frac{1}{(1 − u_{i}x_{i})(1 − u_{i}^{−1}x_{i})} \\ & = ∏_{i} x_{i}^{1/2}(1 + x_{i}) & ∏_{i}(t_{i}^{1/2} − t_{i}^{−1/2}) ∏_{i} \frac{1}{(1 − u_{i}x_{i})(1 − u_{i}^{−1}x_{i})} , \end{aligned}

which differs by the factor

\frac{∏_{i}(1 + x_{i})}{(x_{1} ⋯ x_{n})^{1/2} ∏_{i}(t_{i}^{1/2} + t_{i}^{−1/2})}

from its counterpart (5.10) in section 5. Since this factor does not vary under alternation over the t_{i} ’s, we see that L̃_{odd}(γ̃) differs from L(γ) by the factor

lim_{t_{1}→1,...,t_{n}→1} \frac{∏_{i}(1 + x_{i})}{(x_{1} ⋯ x_{n})^{1/2} ∏_{i}(t_{i}^{1/2} + t_{i}^{−1/2})} = \frac{∏_{i}(1 + x_{i})}{2^{n}(x_{1} ⋯ x_{n})^{1/2}} ,

which completes the proof.

7. The case of G = SU(p, q)

In this section we fix numbers p, q ∈ {1, 2, 3, …} and we put n = p + q. Equip the space ℂ^{n} = ℂ^{p+q} with the indefinite inner product

[ξ, η] = −ξ_{1}η̄_{1} − ξ_{2}η̄_{2} − … − ξ_{p}η̄_{p} + ξ_{p+1}η̄_{p+1} + … + ξ_{n}η̄_{n}, ξ, η ∈ ℂ^{n},

and realize the group G := SU(p, q) as the group of unimodular complex matrices preserving this inner product:

G = SU(p, q) = {g ∈ SL(n, ℂ) | [gξ, gη] = [ξ, η], ξ, η ∈ ℂ^{n}}.

The group G_{ℂ} is SL(n, ℂ). The group K is the subgroup of G that preserves the direct sum decomposition ℂ^{n} = ℂ^{p} ⊕ ℂ^{q}; K is isomorphic to S (U(p) × U(q)).

Let ϵ_{1}, …, ϵ_{n} be the canonical basis of ℝ^{n} and put ϵ = (1, …, 1).

Let ℝ_{0}^{n} be the hyperplane in ℝ^{n}, orthogonal to ϵ:

ℝ_{0}^{n} = {(x_{1}, …, x_{n}) ∈ ℝ^{n} | x_{1} + … + x_{n} = 0}.

We have

𝔥_{Re} = {x = diag(x_{1}, …, x_{n}) | (x_{1}, …, x_{n}) ∈ ℝ_{0}^{n}},

so that 𝔥_{Re} may be identified with ℝ_{0}^{n} ⊂ ℝ^{n}. The dual space 𝔥_{Re}^{∗} will be identified with the factor space ℝ^{n}/ℝϵ; for brevity, elements of 𝔥_{Re}^{∗} often will be written simply as vectors of ℝ^{n}. In this notation we have

\begin{aligned}Δ_{c}^{+} & = \{ϵ_{i} − ϵ_{j} | 1 ≤ i < j ≤ p or p + 1 ≤ i < j ≤ n\} \\ Δ_{n}^{+} & = \{ϵ_{i} − ϵ_{j} | 1 ≤ i ≤ p, p + 1 ≤ j ≤ n\} \\ Δ^{+} & = \{ϵ_{i} − ϵ_{j} | 1 ≤ i < j ≤ n\}. \end{aligned}

The weight lattice P ⊂ 𝔥_{Re}^{∗} can be identified with ℤ^{n}/ℤϵ, and its subset P^{+} ⊂ P of dominant weights is described as

(7.1)  P^{+} = {μ ∈ ℤ^{n}/ℤϵ | μ_{1} ≥ μ_{2} ≥ … ≥ μ_{n}},

and the transformation φ : P^{+} → P_{+−} takes the form

(7.2) φ(μ) = (−μ_{p} − q, …, −μ_{1} − q, −μ_{n} + p, …, −μ_{p+1} + p)  (mod ℤϵ).

The Weyl group W is isomorphic to the symmetric group 𝔖_{n}; its action on 𝔥_{Re} = ℝ_{0}^{n} and 𝔥_{Re}^{∗} = ℝ^{n}/ℝϵ consists in permuting the coordinates. The Weyl group W_{c} ⊂ W is isomorphic to 𝔖_{p} × 𝔖_{q}, naturally embedded into 𝔖_{n}.

The cone c_{min} ⊂ 𝔥_{Re} = ℝ_{0}^{n} is described as follows:

c_{min} = {x ∈ ℝ_{0}^{n} | x_{1}, …, x_{p} ≥ 0 ; x_{p+1}, …, x_{n} ≤ 0}.

The complex torus H_{ℂ} consists of the complex diagonal matrices of the form

diag(z_{1}, …, z_{n}) , z_{1} ⋯ z_{n} = 1.

The subset H_{ℂ}^{+} ⊂ H_{ℂ} is described by the inequalities

|z_{1}| > 1, …, |z_{p}| > 1 ; |z_{p+1}| < 1, … |z_{n}| < 1.

The semigroup Γ is formed by matrices in SL(n, ℂ) which are weakly J -contractive in the sense of being ‘contractive with respect to the inner product [·, ·].’

The interior Γ^{0} ⊂ Γ is formed by strictly J -contractive matrices.

Given a matrix γ ∈ Γ^{0} , its eigenvalues can be written as

(7.3)  x_{1}^{−1}, …, x_{p}^{−1}, y_{1}, …, y_{q},

where

(7.4)  |x_{1}| < 1, …, |x_{p}| < 1, |y_{1}| < 1, …, |y_{q}| < 1

and

(7.5)  x_{1} ⋯ x_{p} = y_{1} ⋯ y_{q}.

Theorem 7.1. Let K(γ_{1}, γ_{2}) be the Cauchy-Szegö kernel for the Hardy space H^{2} (SU(p, q)) ⊂ O(Γ^{0}), where Γ^{0} is the semigroup of strictly J -contractive matrices in SL(n, ℂ), n = p + q. Let L(γ) = K(γ, e) be the corresponding function in one variable γ ∈ Γ^{0} . Given a matrix γ ∈ Γ^{0} , write its eigenvalues as above (see (7.3)–(7.5)) and put

(7.6)  u = x_{1} ⋯ x_{p} = y_{1} ⋯ y_{q}.

(i) The function L is given by the formula

(7.7)  \begin{aligned}L(γ) & = \frac{u^{n}}{∏_{1≤i≤p}(1 − x_{i})^{n} ∏_{1≤j≤q}(1 − y_{j})^{n}} \\ & · ∑_{r=1}^{p}(∏_{k≠r} \frac{x_{k}}{x_{k} − x_{r}} )\frac{(1 − x_{r})^{n}}{∏_{1≤j≤q}(1 − x_{r}y_{j})} . \end{aligned}

(ii) This expression also can be written as

(7.8)  \frac{u^{n}}{∏_{i}(1 − x_{i})^{n} ∏_{j}(1 − y_{j})^{n}} · (1 − \frac{1}{2πi} ∮_{|ξ|=1} F (ξ; x, y)\frac{dξ}{ξ} ) ,

where

(7.9)  F (ξ; x, y) = \frac{(ξ − 1)^{n}}{(ξ − x_{1}) ⋯ (ξ − x_{p})(ξ − y_{1}^{−1}) ⋯ (ξ − y_{q}^{−1})} .

(Recall that we have agreed to normalize the Haar measure of the group as in Lemma 3.4.)

The proof is presented below, after Corollary 7.5. It is based on a character identity (Lemma 7.4), which is an analogue of the identity of Lemma 5.2.

Remark 7.2. Since the groups SU(p, q) and SU(q, p) are isomorphic, the function L(γ) must be invariant under the transformation

(x_{1}, …, x_{p}; y_{1}, …, y_{q}) ⟼ (y_{1}, …, y_{q}; x_{1}, …, x_{p}).

This invariance property is not evident from (7.7) but can be easily obtained from the second claim of Theorem 7.1. Indeed, we remark that

F (ξ; x, y) = F (ξ^{−1}; y, x)

(because of (7.5)) and replace ξ by ξ^{−1} in the integral (7.8).

The expression (7.7) simplifies when p = 1:

Corollary 7.3. Let K be the Cauchy-Szegö kernel for the group SU(1, q), q = 1, 2, …, and let L(γ) = K(γ, e). Write u^{−1}, y_{1}, …, y_{q} for the eigenvalues of a given matrix γ ∈ Γ^{0} (u = y_{1} ⋯ y_{q}). Then

(7.10)  L(γ) = \frac{u^{q+1}}{∏_{1≤j≤q}(1 − y_{j})^{q+1}(1 − uy_{j})} .

Note that in the case q = 1 the group SU(1, q) = SU(1, 1) is isomorphic to Sp(1, ℝ)), and formula (7.10) agrees with formula (5.3) of Theorem 5.1.

Lemma 7.4. Let Λ = (Λ_{1} ≥ … ≥ Λ_{p} ≥ 0) and M = (M_{1} ≥ … ≥ M_{q} ≥ 0) be arbitrary partitions of length ≤ p and ≤ q, respectively; let s_{Λ}(x_{1}, …, x_{p}) and s_{Λ}(y_{1}, …, y_{q}) be the corresponding Schur functions in p and q variables; let V_{Λ,M} be the irreducible finite-dimensional GL(n, ℂ)-module (where n = p + q) with highest weight

(7.11)  (Λ_{1}, …, Λ_{p}, −M_{q}, …, −M_{1}),

and let gl(n)_{Λ,M}(t_{1}, …, t_{n}) be the character of V_{Λ,M}, viewed as a function on the maximal torus of GL(n, ℂ) formed by diagonal matrices diag(t_{1}, …, t_{n}). Then

(7.12)  \begin{aligned}∑_{Λ,M} & gl(n)_{Λ,M}(t_{1}, …, t_{n})s_{Λ}(x_{1}, …, x_{p})s_{M}(y_{1}, …, y_{q}) \\ & = \frac{∏_{i=1}^{p} ∏_{j=1}^{q}(1 − x_{i}y_{j})}{∏_{k=1}^{n} ∏_{i=1}^{p}(1 − t_{k}x_{i}) ∏_{k=1}^{n} ∏_{j=1}^{q}(1 − t_{k}^{−1}y_{j})} . \end{aligned}

Proof. This character identity can be verified by the same methods as the identity of Lemma 5.2. In particular, one can derive (7.12) from the explicit weight correspondence in Howe’s duality between the groups U(k) and U(p, q) (for the special case k = p + q), described in Kashiwara and Vergne [11]. In section 8 we will prove the identity (7.12) by Ishikawa–Wakayama’s method.

Corollary 7.5. Let Λ, M and V_{Λ,M} be as in Lemma 7.4, and let dim V_{Λ,M} denote the dimension of V_{Λ,M}. Then

(7.13)  \begin{aligned}∑_{Λ,M} & dim V_{Λ,M} · s_{Λ}(x_{1}, …, x_{p})s_{M}(y_{1}, …, y_{q}) \\ & = \frac{∏_{i=1}^{p} ∏_{j=1}^{q}(1 − x_{i}y_{j})}{∏_{i=1}^{p}(1 − x_{i})^{n} ∏_{j=1}^{q}(1 − y_{j})^{n}} . \end{aligned}

Proof. Take t_{1} = … = t_{n} = 1 in (7.12).

Proof of Theorem 7.1. (i) As in the case of the group Sp(n, ℝ) (see the beginning of the first proof of Theorem 5.1), application of Lemma 2.1 allows one to assume that γ is a diagonal matrix contained in H_{ℂ}^{+} . Then we write γ as

γ = diag(x_{p}^{−1}, …, x_{1}^{−1}, y_{1}, …, y_{q}),

where

|x_{1}| < 1, …, |x_{p}| < 1, |y_{1}| < 1, …, |y_{q}| < 1.

By Corollary 3.5,

(7.14)  L(γ) = θ(γ) ∑_{μ∈P^{+}} dim V_{μ} · tr π_{φ(μ)}(γ).

In the present situation

θ(γ) = ∏_{i=1}^{p} ∏_{j=1}^{q} \frac{1}{1 − x_{i}y_{j}}

and, by virtue of (7.2) and (7.6),

tr π_{φ(μ)}(γ) = u^{n}s_{(μ_{1},...,μ_{p})}(x_{1}, …, x_{p})s_{(−μ_{n},...,−μ_{p+1})}(y_{1}, …, y_{q}).

Recall that each dominant weight μ = (μ_{1}, …, μ_{n}) ∈ P^{+} is an element of the quotient lattice ℤ^{n}/ℤϵ (not of the lattice ℤ^{n}!). It will be convenient for us to fix a representative of μ in ℤ^{n} by putting μ_{p} = 0. Then μ will be indexed by a couple (Λ, M) of partitions such that l(Λ) ≤ p − 1 (hence Λ_{p} = 0) and l(M) ≤ q, i.e.,

μ = (Λ_{1}, …, Λ_{p−1}, 0, −M_{q}, …, −M_{1}).

Thus, (7.14) can be rewritten as follows:

(7.15)  \begin{matrix}L(γ) = & u^{n}(∏_{i=1}^{p} ∏_{j=1}^{q} \frac{1}{1 − x_{i}y_{j}} ) \\ · ∑_{Λ,M_{Λp=0}} dim V_{Λ,M}s_{Λ}(x_{1}, …, x_{p})s_{M}(y_{1}, …, y_{q}). \end{matrix}

The restriction Λ_{p} = 0 does not allow us to apply directly the character identity of Corollary 7.5. However, this difficulty can be surmounted with help of the following claim:

Lemma 7.6. Let

Φ(x_{1}, …, x_{p}) = ∑_{Λ} c(Λ)s_{Λ}(x_{1}, …, x_{p})

be a series on Schur functions in p variables, and define

Φ_{0}(x_{1}, …, x_{p}) := ∑_{Λ, Λ_{p}=0} c(Λ)s_{Λ}(x_{1}, …, x_{p}).

Then

(7.16)  Φ_{0}(x_{1}, …, x_{p}) = ∑_{r=1}^{p}(∏_{k≠r} \frac{x_{k}}{x_{k} − x_{r}} )(Φ(x_{1}, …, x_{p})|_{x_{r}=0}) .

Proof of the Lemma. By linearity, it suffices to check (7.16) for Φ = s_{Λ} . That is to say, we must prove that

(7.17)

∑_{r=1}^{p}( ∏_{k≠r} \frac{x_{k}}{x_{k} − x_{r}} )(s_{Λ}(x_{1}, …, x_{p})|_{x_{r}=0})

= \{ \begin{matrix}s_{Λ}(x_{1}, …, x_{p}), & if Λ_{p} = 0 \\ 0, & otherwise. \end{matrix}

Write

s_{Λ}(x_{1}, …, x_{p}) = \frac{A_{Λ}(x)}{V (x)} ,

where

(7.18)

A_{Λ}(x) = det[x_{j}^{Λ_{i}+p−i}]_{1≤i,j≤p}

= ∑_{(j_{1},...,j_{p})} ε(j_{1}, …, j_{p}) x_{j_{1}}^{Λ_{1}+p−1}x_{j_{2}}^{Λ_{2}+p−2} ⋯ x_{j_{p}}^{Λ_{p}},

summed over all permutations (j_{1}, …, j_{p}) of (1, …, n) (here ε(j_{1}, …, j_{p}) stands for the sign of the permutation (j_{1}, …, j_{p})), and

(7.19)  V (x) = A_{(0,...,0)}(x) = ∏_{1≤i<j≤p}(x_{i} − x_{j})

is the Vandermonde determinant.

Assume first Λ_{p} > 0. Then (7.18) implies that s_{Λ}(x_{1}, …, x_{p}) vanishes as x_{r} = 0 for a certain r, which confirms (7.16). So one may assume Λ_{p} = 0. Then (7.18) implies

A_{Λ}(x)|_{x_{r}=0} = ∑_{{(j_{1},...,j_{p}) \atop j_{p}=r}} ε(j_{1}, …, j_{p}) x_{j_{1}}^{Λ_{1}+p−1} x_{j_{2}}^{Λ_{2}+p−2} ⋯ x_{j_{p}}^{Λ_{p}},

whence

(7.20)  ∑_{r=1}^{p}(A_{Λ}|_{x_{r}=0}) = A_{Λ}(x).

Further,

V (x)|_{x_{r}=0} = (∏_{k≠r} \frac{x_{k}}{x_{k} − x_{r}} )V (x), r = 1, …, p.

Along with (7.20) this yields

∑_{r=1}^{r}(∏_{k≠r} \frac{x_{k}}{x_{k} − x_{r}} )(s_{Λ}(x_{1}, …, x_{p})|_{x_{r}=0}) = ∑_{r=1}^{p}(∏_{k≠r} \frac{x_{k}}{x_{k} − x_{r}} )\frac{A_{Λ}(x)|_{x_{r}=0}}{V (x)|_{x_{r}=0}}

= \frac{∑_{r=1}^{p} A_{Λ}(x)|_{x_{r}=0}}{V (x)}

= \frac{A_{Λ}(x)}{V (x)}

= s_{Λ}(x_{1}, …, x_{p}),

which completes the proof of the Lemma.

Let us return to the proof of the Theorem. By Corollary 7.5 and Lemma 7.6,

(7.21)  \begin{aligned}L(γ) = & \frac{u^{n}}{∏_{i,j}(1 − x_{i}y_{j})} ∑_{r=1}^{p}(∏_{k≠r} \frac{x^{k}}{x_{k} − x_{r}} ) \\ & · \{\frac{∏_{i,j}(1 − x_{i}y_{j})}{∏_{i}(1 − x_{i})^{n} ∏_{j}(1 − y_{j})^{n})}\}_{x_{r}=0} . \end{aligned}

Further,

\{\frac{∏_{i,j}(1 − x_{i}y_{j})}{∏_{i}(1 − x_{i})^{n} ∏_{j}(1 − y_{j})^{n})}\}_{x_{r}=0} = \frac{(1 − x_{r})^{n}}{∏_{j}(1 − x_{r}y_{j})} · \frac{∏_{i,j}(1 − x_{i}y_{j})}{∏_{i}(1 − x_{i})^{n} ∏_{j}(1 − y_{j})^{n})} .

Substituting the latter expression into (7.21) we obtain the desired formula (7.7) for L(γ). This completes proof of claim (i) of the Theorem.

(ii) Let us transform the sum in formula (7.7):

(7.22)  \begin{matrix}∑_{r=1}^{p} & ∏_{k≠r} & \frac{x_{k}}{x_{k} − x_{r}} & \frac{(1 − x_{r})^{n}}{∏_{j}(1 − x_{r}y_{j})} \\ = (−1)^{p−1} ∑_{r=1}^{p} \frac{(1 − x_{r})^{n}}{x_{r}} · \frac{x_{1} … x_{p}}{∏_{j}(1 − x_{r}y_{j})} ∏_{k≠r} \frac{1}{x_{r} − x_{k}} . \end{matrix}

By (7.5),

\begin{aligned}\frac{x_{1} … x_{p}}{∏_{j}(1 − x_{r}y_{j})} & = \frac{y_{1} … y_{q}}{∏_{j}(1 − x_{r}y_{j})} = ∏_{j} \frac{y_{j}}{1 − x_{r}y_{j}} \\ & = (−1)^{q} ∏_{j} \frac{1}{x_{r} − y_{j}^{−1}} . \end{aligned}

Therefore, the expression (7.22) is equal to

(−1)^{n−1} ∑_{r=1}^{p} \frac{(1 − x_{r})^{n}}{x_{r}} ∏_{k≠r} \frac{1}{x_{r} − x_{k}} ∏_{j} \frac{1}{x_{r} − y_{j}^{−1}}

= − ∑_{r=1}^{p} \frac{(x_{r} − 1)^{n}}{x_{r}} ∏_{k≠r} \frac{1}{x_{r} − x_{k}} ∏_{j} \frac{1}{x_{r} − y_{j}^{−1}}

= − ∑_{r=1}^{p} Res_{ξ=x_{r}} \frac{(ξ − 1)^{n}}{ξ(ξ − x_{1}) … (ξ − x_{p})(ξ − y_{1}^{−1}) … (ξ − y_{q}^{−1})} ,

where ‘Res_{ξ=a} ’ means ‘residue at the point ξ = a’.

Remark that the poles of the function

ξ ↦ \frac{(ξ − 1)^{n}}{ξ(ξ − x_{1}) … (ξ − x_{p})(ξ − y_{1}^{−1}) … (ξ − y_{q}^{−1})}

in the unit disk |ξ| < 1 occur at the points 0, x_{1}, …, x_{p}, because |y_{j}^{−1}| > 1, 1 ≤ j ≤ q. Next, remark that the residue at ξ = 0 equals 1, because

\frac{(−1)^{n}}{(−x_{1}) … (−x_{p})(−y_{1}^{−1}) … (−y_{q}^{−1})} = \frac{1}{x_{1} … x_{p}y_{1}^{−1} … y_{q}^{−1}} = 1.

Therefore, (7.22) equals

(7.23)  1 − \frac{1}{2πi} ∮_{|ξ|=1} F (ξ; x, y)\frac{dξ}{ξ} ,

where F (ξ; x, y) is defined by (7.9). By substituting (7.23) into (7.7) (instead of the sum) we obtain the desired formula (7.8). This completes the proof of the Theorem.

8. Proof of character identities

In this section we prove Lemma 5.2 and Lemma 7.4 using an elegant method due to M. Ishikawa and M. Wakayama. Proof of Lemma 5.2. Recall the identity that has to be proved:

(8.1) ∑_{M} sp(n)_{M}(t_{1}^{±1}, …, t_{n}^{±1})s_{M}(x_{1}, …, x_{n}) = \frac{∏_{i<j}(1 − x_{i}x_{j})}{∏_{k} ∏_{i}(1 − t_{k}x_{i})(1 − t_{k}^{−1}x_{i})} ,

where M ranges over the set of partitions of length ≤ n, sp(n)_{M} stands for the character of Sp(n, ℂ), indexed by M, and s_{M} is the Schur function.

Also recall that by Weyl’s ‘first character formula’ (see [25], Theorem VII.8.C),

(8.2)  sp(n)_{M}(t_{1}^{±1}, …, t_{n}^{±1}) = \frac{det [\frac{t_{k}^{l_{i}} −t_{k}^{−l_{i}}}{t_{k}−t_{k}^{−1}} ]_{1≤k,i≤n}}{V (t_{1} + t_{1}^{−1}, …, t_{n} + t_{n}^{−1})},

where

(8.3)  (l_{1}, …, l_{n}) = (M_{1} + n, …, M_{n} + 1)

and V (a_{1}, …, a_{n}) stands for the Vandermonde determinant,

V (a_{1}, …, a_{n}) = ∏_{r<s}(a_{r} − a_{s}).

Consider two matrices, T and S, with n rows and countably many columns that are labelled by the positive integers, written in reverse order …, 2, 1,

T = ( \begin{matrix}… & \frac{t_{1}^{3}−t_{1}^{−3}}{t_{1}−t_{1}^{−1}} & … & \frac{t_{1}^{2}−t_{1}^{−2}}{t_{1}−t_{1}^{−1}} & … & \frac{t_{1}^{1}−t_{1}^{−1}}{t_{1}−t_{1}^{−1}} \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ … & \frac{t_{n}^{3}−t_{n}^{−3}}{t_{n}−t_{n}^{−1}} & … & \frac{t_{n}^{2}−t_{n}^{−2}}{t_{n}−t_{n}^{−1}} & … & \frac{t_{n}^{1}−t_{n}^{−1}}{t_{n}−t_{n}^{−1}} \end{matrix} )

S = ( \begin{matrix}… & x_{1}^{2} & x_{1} & 1 \\ ⋮ & ⋮ & ⋮ & ⋮ \\ … & x_{n}^{2} & x_{n} & 1 \end{matrix} ) .

Denoting by S′ the transposed matrix we will compute the n × n determinant det T S′ in two ways.

On the one hand, we have

(T S′)_{ki} = ∑_{r=1}^{∞} \frac{t_{k}^{r} − t_{k}^{−r}}{t_{k} − t_{k}^{−1}} x_{i}^{r−1}

= \frac{1}{t_{k} − t_{k}^{−1}} (\frac{t_{k}}{1 − t_{k}x_{i}} − \frac{t_{k}^{−1}}{1 − t_{k}^{−1}x_{i}} )

= (1 − t_{k}x_{i})^{−1}(1 − t_{k}^{−1}x_{i})^{−1}.

It follows

(8.4)  \begin{aligned}det T S′ & = det[(1 − t_{k}x_{i})^{−1}(1 − t_{k}^{−1}x_{i})^{−1}]_{1≤k,i≤n} \\ & = \frac{V (t_{1} + t_{1}^{−1}, …, t_{n} + t_{n}^{−1})V (x_{1}, …, x_{n}) ∏_{i<j}(1 − x_{i}x_{j})}{∏_{i} ∏_{k}(1 − t_{k}x_{i})(1 − t_{k}^{−1}x_{i})} \end{aligned}

(see (5.11)).

On the other hand, given arbitrary integers l_{1} > … > l_{n} ≥ 1, let us denote by det T_{l_{1}...l_{n}} and det S_{l_{1}...l_{n}} the n-th order minors formed by the columns l_{1}, …, l_{n} of the matrix T and S, respectively. Then we have

(8.5)  det T S′ = ∑_{l_{1}>...>l_{n}≥1} det T_{l_{1}...l_{n}} det S_{l_{1}...l_{n}}.

Let M = (M_{1}, …, M_{n}) be related with l = (l_{1}, …, l_{n}) by (8.3). Then M is a partition and, by Weyl’s character formula (8.2),

det T_{l_{1}...l_{n}} = sp(n)_{M}(t_{1}^{±1}, …, t_{n}^{±1})V (t_{1} + t_{1}^{−1}, …, t_{n} + t_{n}^{−1}),

and we also have

det S_{l_{1}...l_{n}} = s_{M}(x_{1}, …, x_{n})V (x_{1}, …, x_{n}).

Therefore,

(8.6)  \begin{matrix}det T S′ = & V (t_{1} + t_{1}^{−1}, …, t_{n} + t_{n}^{−1})V (x_{1}, …, x_{n}) \\ · ∑_{M} sp(n)_{M}(t_{1}^{±1}, …, t_{n}^{±1})s_{M}(x_{1}, …, x_{n}). \end{matrix}

By comparing both expressions for det T S′, (8.4) and (8.6), we arrive to the desired formula (8.1).

Now we will apply a similar argument to prove the second character identity.

Proof of Lemma 7.4. Recall the identity in question:

(8.7)  \begin{matrix}∑_{Λ,M} gl_{Λ,M}(t_{1}, …, t_{n})s_{Λ}(x_{1}, …, x_{p})s_{M}(y_{1}, …, y_{q}) \\ = \frac{∏_{i=1}^{p} ∏_{j=1}^{q}(1 − x_{i}y_{j})}{∏_{k=1}^{n} ∏_{i=1}^{p}(1 − t_{k}x_{i}) ∏_{k=1}^{n} ∏_{j=1}^{q}(1 − t_{k}^{−1}y_{j})} , \end{matrix}

where n = p + q; Λ and M are arbitrary partitions of length ≤ p and ≤ q, respectively; gl(n)_{Λ,M} is the character of GL(n, ℂ) corresponding to the dominant weight

(Λ_{1}, …, Λ_{p}, −M_{q}, …, −M_{1});

s_{Λ} and s_{M} are Schur functions indexed by Λ and M.

Consider two matrices, T and S, whose rows are labelled by 1, …, n (n = p + q) and columns are labelled by all the integers written in reverse order …, 2, 1, 0, −1, −2, …,

T = ( \begin{matrix}… & t_{1}^{2} & t_{1} & 1, & t_{1}^{−1} & t_{1}^{−2} & … \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ … & t_{n}^{2} & t_{n} & 1, & t_{n}^{−1} & t_{n}^{−2} & … \end{matrix} )

S = ( \begin{matrix}… & x_{1}^{2} & x_{1} & 1, & 0 & 0 & 0 & … \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ … & x_{p}^{2} & x_{p} & 1, & 0 & 0 & 0 & … \\ … & 0 & 0 & 0, & 1 & y_{1} & y_{1}^{2} & … \\ ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ & ⋮ \\ … & 0 & 0 & 0, & 1 & y_{q} & y_{q}^{2} & … \end{matrix} )

where commas in the matrices were used to separate the 0-th and (-1)-th columns. Let us again compute det T S′ in two ways.

On the one hand, we have for 1 ≤ k ≤ n, 1 ≤ i ≤ p, 1 ≤ j ≤ q,

\begin{aligned}(T S′)_{ki} & = (1 − t_{k}x_{i})^{−1} = −x_{i}^{−1}(t_{k} − x_{i}^{−1}), \\ (T S′)_{k,p+j} & = (t_{k} − y_{j})^{−1}, \end{aligned}

whence

(8.8)  det T S′ = (−1)^{p}(x_{1} … x_{p})^{−1} det[(t_{k} − z_{r})^{−1}]_{1≤k,r≤n},

where

(z_{1}, …, z_{n}) = (x_{1}^{−1}, …, x_{p}^{−1}, y_{1}, …, y_{q}).

The determinant in (8.8) is readily reduced to Cauchy’s determinant, and after simple transformations we obtain

(8.9) \begin{matrix}det T S′ = & (−1)^{q(q−1)/2}(t_{1} … t_{n})^{−q} \\ · \frac{V (t_{1}, …, t_{n})V (x_{1}, …, x_{p})V (y_{1}, …, y_{q}) ∏_{i} ∏_{j}(1 − x_{i}y_{j})}{∏_{k} ∏_{i}(1 − t_{k}x_{i}) ∏_{k} ∏_{j}(1 − t_{k}^{−1}y_{j})}. \end{matrix}

On the other hand,

(8.10)  det T S′ = ∑_{+∞>l_{1}>...>l_{n}>−∞} det T_{l_{1}...l_{n}} det S_{l_{1}...l_{n}}.

¿From the form of the matrix S it is clear that det S_{l_{1}...l_{n}} vanishes unless l_{p} ≥ 0 and l_{p+1} ≤ −1, so that we may assume

\begin{aligned}(l_{1}, …, l_{n}) & = (Λ_{1}, …, Λ_{p}, −M_{q}, …, −M_{1}) \\ & + (p − 1, …, 0, −1, …, −q), \end{aligned}

where Λ and M are partitions. Further, we have

det T_{l_{1}...l_{n}} = gl(n)_{Λ,M}(t_{1} … t_{n})V (t_{1}, …, t_{n})(t_{1}, …, t_{n})^{−q},

\begin{matrix}det S_{l_{1}...l_{n}} = & s_{Λ}(x_{1}, …, x_{p})s_{M}(y_{1}, …, y_{q}) \\ · V (x_{1}, …, x_{p})V (y_{1}, …, y_{q})(−1)^{q(q−1)/2}, \end{matrix}

whence

(8.11)  \begin{matrix}det T S′ = (−1)^{q(q−1)/2}(t_{1} … t_{n})^{−q}V (t_{1}, …, t_{n})V (x_{1}, …, x_{p}) \\ · V (y_{1}, …, y_{q}) ∑_{Λ,M} gl_{Λ,M}(t_{1}, …, t_{n})s_{Λ}(x_{1}, …, x_{p})s_{M}(y_{1}, …, y_{q}). \end{matrix}

By comparing (8.9) and (8.11) we obtain the desired identity (8.7).

References

[1] Enright, T. J., R. Howe, and N. Wallach, A classification of unitary highest weight modules, in: Representation theory of reductive groups, Birkhäuser, Boston, 1983, 97–143.

[2] Faraut, J., “Hardy spaces in non–commutative harmonic analysis,” Notes of a course in Summer School on Harmonic Analysis and Geometry, Tuczno, 1994.

[3] Faraut, J., and A. Korányi, “Analysis on symmetric cones,” Oxford Mathematical Monographs, Clarendon Press, Oxford, 1994.

[4] Gelfand, I. M., and S. G. Gindikin, Complex manifolds whose skeletons are semi–simple Lie groups, and analytic discrete series of representations, Funkts. Analiz i Pril. 11 (1977), 19–27 (Russian); English translation: Funct. Anal. Appl. 11 (1977), 258–265.

[5] Harish–Chandra, Representations of semisimple Lie groups, IV–VI, Amer. J. Math. 77 (1955), 743–777; 78 (1956), 1–41; 78 (1956), 564–628.

[6] Hilgert, J., and K.-H. Neeb, “Lie semigroups and their applications”, Springer Lecture Notes in Math. 1552 (1993).

[7] Hilgert, J., G. Ólafsson, Analytic continuation of representations, the solvable case, Japanese J. Math. 18 (1992), 213–290.

[8] Hilgert, J., G. Ólafsson, and B. Ørsted, Hardy spaces on affine symmetric spaces, J. reine angew. Math. 415 (1991), 189–218.

[9] Howe, R., Remarks on classical invariant theory, Trans. Amer. Math. Soc. 313 (1989), 539–570.

[10] Inoue, T., Unitary representations and kernel functions associated with boundaries of a bounded symmetric domain, Hiroshima Math. J. 10 (1980).

[11] Kashiwara, M., and M. Vergne, On the Segal–Shale–Weil representation and harmonic polynomials, Invent. Math. 44 (1978), 1–47.

[12] Koike, K., and I. Terada, Young–diagrammatic methods for the representation theory of the classical groups of type B_{n}, C_{n}, D_{n}, J. Algebra 107 (1987), 466–511.

[13] Koufany, K., and B. Ørsted, Functions spaces on the Ol’shanskiĭ semigroup and the Gel’fand-Gindikin program, Preprint, Odense University, 1995 (submitted).

[14] Koufany, K., et B. Ørsted, Espace de Hardy sur le semi–groupe métaplectique, Odense, 12 septembre 1995, à paraitre dans Comptes Rendus Acad. Sci. Paris.

[15] Koufany, K., and B. Ørsted, The Hardy space on the Oscillator semigroup, Preprint, Odense University, 1995.

[16] Littlewood, D. E., “The theory of group characters and matrix representations of groups,” 2nd edition, Clarendon Press, Oxford, 1958.

[17] Neeb, K.-H., Holomorphic representation theory I, Math. Annalen 301 (1995), 155–181.

[18] —, Holomorphic representation theory II, Acta Math. 173 (1994), 103– 133.

[19] Ol’shanskiĭ, G. I., Invariant cones in Lie algebras, Lie semigroups and the holomorphic discrete series, Funkts. Analiz. i Pril. 15, (1981), 53–66 (Russian); English translation.: Funct. Anal. Appl. 15 (1982), 275–285.

[20] —, Complex Lie semigroups, Hardy spaces, and the Gelfand-Gindikin program. In: “Topics in group theory and homological algebra,” Yaroslavl University Press, 1982, 85–98 (Russian). English translation: Differential Geometry and its Applications, 1 (1991), 297–308.

[21] Sundaram, S., in: D. Stanton, Editor, “Invariant Theory and Young Tableaux,” IMA Volumes in Mathematics and its Applications, 19, Springer–Verlag, 1990.

[22] Stein, E. M., and G. Weiss, “Introduction to Fourier analysis on Euclidean spaces,” Princeton University Press, 1971.

[23] Vergne, M., and G. Rossi, Analytic continuation of the holomorphic discrete series of a semi–simple Lie group, Acta Math. 136 (1976), 1–59.

[24] Vinberg, E. B., Invariant cones and orderings in Lie groups, Funkts. Analiz i Pril. 14 (1980), 1–13 (Russian); English translation in Funct. Anal. Appl. 14 (1980).

[25] Weyl, H., “The classical groups. Their invariants and representations,” Princeton, 1939.

Institute for Problems of Information Transmission Ermolovoy 19 101447 Moscow GSP–4 Russia olsh@ippi.ac.msk.su

Received November 3, 1995 and in final form November 27, 1995