A Geometric Quantization of the

Kostant–Sekiguchi Correspondence for Scalar Type

Unitary Highest Weight Representations

Jan Möllers

Received: February 10, 2013

Communicated by Patrick Delorme

Abstract. For any Hermitian Lie group G of tube type we give a geometric quantization procedure of certain K_{ℂ}-orbits in 𝔭_{ℂ}^{∗} to obtain all scalar type highest weight representations. Here K_{ℂ} is the complexification of a maximal compact subgroup K ⊆ G with corresponding Cartan decomposition 𝔤 = 𝔨 + 𝔭 of the Lie algebra of G. We explicitly realize every such representation π on a Fock space consisting of square integrable holomorphic functions on its associated variety Ass(π) ⊆ 𝔭_{ℂ}^{∗}.

The associated variety Ass(π) is the closure of a single nilpotent K_{ℂ}- orbit O^{K_{ℂ}} ⊆ 𝔭_{ℂ}^{∗} which corresponds by the Kostant–Sekiguchi correspondence to a nilpotent coadjoint G-orbit O^{G} ⊆ 𝔤^{∗}. The known Schrödinger model of π is a realization on L^{2}(O), where O ⊆ O^{G} is a Lagrangian submanifold. We construct an intertwining operator from the Schrödinger model to the new Fock model, the generalized Segal–Bargmann transform, which gives a geometric quantization of the Kostant–Sekiguchi correspondence (a notion invented by Hilgert, Kobayashi, Ørsted and the author).

The main tool in our construction are multivariable I- and K-Bessel functions on Jordan algebras which appear in the measure of O^{K_{ℂ}}, as reproducing kernel of the Fock space and as integral kernel of the Segal–Bargmann transform. As a corollary to our construction we also obtain the integral kernel of the unitary inversion operator in the Schrödinger model in terms of a multivariable J-Bessel function as well as explicit Whittaker vectors.

2010 Mathematics Subject Classification: Primary 22E45; Secondary 17C30, 30H20, 33C70, 44A15, 46E22.

Keywords and Phrases: unitary highest weight representation, orbit method, Schrödinger model, Fock model, Jordan algebra, Segal-Bargmann transform, Bessel function, Bessel operator, unitary inversion operator, Whittaker vectors, branching law.

Introduction

The unitary highest weight representations of a semisimple Lie group G were classified by Enright–Howe–Wallach [13] and Jakobsen [29] independently (see also [14]). Among them are the scalar type representations which were first completely described by Berezin [6] for the classical groups and by Wallach [51] in the general case and are parameterized by the so-called Wallach set (sometimes also referred to as the Berezin–Wallach set). There are various different realizations of these representations, e.g. on spaces of holomorphic functions on a bounded symmetric domain. Rossi–Vergne [48] gave a realization which can by the work of Hilgert–Neeb–Ørsted [25, 26, 27] be understood as the geometric quantization of a nilpotent coadjoint G-orbit. More precisely, their model lives on the space L^{2}(O) of square integrable functions on a Lagrangian submanifold O ⊆ O^{G} of a nilpotent coadjoint G-orbit O^{G} ⊆ 𝔤^{∗} in the dual of the Lie algebra 𝔤 of G. We refer to this realization as the Schrödinger model. The Kostant–Sekiguchi correspondence asserts a bijection between the set of nilpotent coadjoint G-orbits in 𝔤^{∗} and the set of nilpotent K_{ℂ}-orbits in 𝔭_{ℂ}^{∗}, where K_{ℂ} is the complexification of a maximal compact subgroup K ⊆ G of G with corresponding Cartan decomposition 𝔤 = 𝔨 + 𝔭. Under this bijection O^{G} corresponds to a nilpotent K_{ℂ}-orbit O^{K_{ℂ}} in 𝔭_{ℂ}^{∗}. Harris–Jakobsen [19] obtained a realization of each scalar type unitary highest weight representation of 𝔤 on regular functions on the corresponding associated variety in 𝔭_{ℂ}^{∗} (see also [30]). However, their construction is not explicit and of purely algebraic nature. They neither give a geometric construction of the invariant inner product nor do they analytically describe the representation space for the group action. However, for the special case of the even part of the metaplectic representation, which is a highest weight representation of scalar type of 𝔤 = 𝔰𝔭(k, ℝ), a geometric quantization of the corresponding K_{ℂ}-orbit is well-known. It gives the even part F_{even}(ℂ^{k}) of the classical Fock space, consisting of even holomorphic functions on ℂ^{k} which are square integrable with respect to the Gaussian measure e^{−|z|^{2}} dz. In this case also the intertwining operator between the Schrödinger model on L_{even}^{2}(ℝ^{k}), the space of even L^{2}-functions on ℝ^{k}, and the Fock model F_{even}(ℂ^{k}) is explicit. It is given by the classical Segal–Bargmann transform which is the unitary isomorphism 𝔹 : L_{even}^{2}(ℝ^{k}) →^{∼} F_{even}(ℂ^{k}) given by

𝔹ψ(z) = e^{− \frac{1}{2} z^{2}} ∫_{ℝ^{k}} e^{2z·x}e^{−x^{2}}ψ(x) dx.  (0.1)

For Hermitian groups of tube type, we construct in a completely analytic and geometric way analogues of the Fock space and the Segal–Bargmann transform for any unitary highest weight representation of scalar type. The generalized Fock space consists of holomorphic functions on O^{K_{ℂ}} which are square integrable with respect to a certain measure. This establishes a geometric quantization of the nilpotent K_{ℂ}-orbit O^{K_{ℂ}}. We further find explicitly the intertwining operator between this new Fock model and the Schrödinger model, the generalized Segal–Bargmann transform. This integral transform can be understood as a geometric quantization of the Kostant–Sekiguchi correspondence, a notion invented in [23]. We remark that the special case of the minimal nilpotent K_{ℂ}-orbit was treated in [23] which was in fact the starting point of our investigations. Although some of our proofs are obvious generalizations of the proofs in [23] for the rank 1 case (the minimal scalar type highest weight representation), the general case requires new techniques. Whereas in [23] it was possible to work with classical one-variable Bessel functions, we have to use multivariable J-, I- and K-Bessel functions on Jordan algebras in the general case. These were studied before in [8, 10, 11, 12, 16], partly in a different context. We systematically investigate them further, proving asymptotic expansions, growth estimates, invariance properties, differential equations, integral formulas and their restrictions to the K_{ℂ}-orbits O^{K_{ℂ}}. We then use the K-Bessel functions in the construction of the measures on the orbits O^{K_{ℂ}} and the I-Bessel functions as reproducing kernels of the Fock spaces and as integral kernels of the Segal– Bargmann transforms. Additional results not studied in [23] involve a detailed analysis of corresponding K_{ℂ}-orbits and rings of differential operators on them (Section 4.5), the intertwining operator to the bounded symmetric domain model (Section 5.2), Whittaker vectors in the Schrödinger model (Section 6.2) and applications to branching laws (Section 7). We further believe that our more general construction extends the known realizations of unitary highest weight representations in a natural and organic way.

We now explain our results in more detail. Let G be a simple connected Hermitian Lie group of tube type with trivial center. Then G occurs as the identity component of the conformal group of a simple Euclidean Jordan algebra V. Its Lie algebra 𝔤 acts on V by vector fields up to order 2. More precisely, we have a decomposition

𝔤 = 𝔫 + 𝔩 + \overline{𝔫},  (0.2)

where 𝔫 ≅ V acts on V by translations, 𝔩 = 𝔰𝔱𝔯(V) is the structure algebra of V acting by linear transformations and \overline{𝔫} acts by quadratic vector fields. There is a natural Cartan involution ϑ on G given in terms of the inversion in the Jordan algebra. Denote by K = G^{ϑ} the corresponding maximal compact subgroup of G and by 𝔤 = 𝔨 + 𝔭 the associated Cartan decomposition. Then 𝔨 has a one-dimensional center Z(𝔨) = ℝZ_{0}. Let K_{ℂ} be a complexification of K, acting on 𝔭_{ℂ}. Then 𝔭_{ℂ} decomposes into two irreducible K_{ℂ}-modules 𝔭^{+} and 𝔭^{−} and Z_{0} ∈ Z(𝔨) can be chosen such that 𝔭^{±} is the ±i eigenspace of ad(Z_{0}) on 𝔭_{ℂ}. We have the well-known decomposition

𝔤_{ℂ} = 𝔭^{−} + 𝔨_{ℂ} + 𝔭^{+}.  (0.3)

There is a Cayley type transform C ∈ Int(𝔤_{ℂ}) (see (1.5) for the precise definition) which interchanges the decompositions (0.2) and (0.3) in the sense that

C(𝔭^{−}) = 𝔫_{ℂ},  C(𝔨_{ℂ}) = 𝔩_{ℂ},  C(𝔭^{+}) = \overline{𝔫}_{ℂ}.

Unitary highest weight representations

Let (π, H) be an irreducible unitary representation of the universal cover G̃ of G and denote by ( dπ, HK̃) its underlying (𝔤, K̃)-module, K̃ ⊆ G̃ being the universal cover of K. We say that (π, H) or ( dπ, HK̃) is a highest weight representation if

H𝔭^{+} ; K̃ := \{v ∈ H\overline{K} : dπ(𝔭^{+})v = 0\} ≠ 0.

The space H𝔭^{+} ; K̃ is an irreducible 𝔨-representation. If further dim H𝔭^{+} ; K̃ = 1 we say that (π, H) is of scalar type. In this case only the center Z(𝔨) = ℝZ_{0} of 𝔨 can act nontrivially on H𝔭^{+} ; K̃ and the scalar type highest weight representations

of G̃ are parameterized by this scalar. More precisely, we define the Wallach set

W = \{0, \frac{d}{2}, …, (r − 1)\frac{d}{2}\} ∪ ((r − 1)\frac{d}{2}, ∞) ,

where d is a certain root multiplicity of 𝔤 and r denotes the rank of the Hermitian symmetric space G/K. Then for each λ ∈ W there is exactly one scalar type highest weight representation H of G̃ in which Z_{0} acts on H𝔭^{+} ; K̃ by the scalar −i\frac{rλ}{2}.

Nilpotent orbits

We identify 𝔤^{∗} with 𝔤 and 𝔭_{ℂ}^{∗} with 𝔭_{ℂ} via the Killing form of 𝔤_{ℂ}. By the Kostant–Sekiguchi correspondence the nilpotent adjoint G-orbits in 𝔤 are in bijection with the nilpotent K_{ℂ}-orbits in 𝔭_{ℂ}. For a certain class of orbits we make this correspondence more precise. The analytic subgroup L of G with Lie algebra 𝔩 acts on V ≅ 𝔫 via the adjoint representation. It has finitely many orbits, among them an open orbit Ω ⊆ V (the orbit of the identity element in the Jordan algebra) which is a symmetric cone. Its boundary has a stratification into lower dimensional orbits O_{0}, …, O_{r−1} with O_{k} ⊆ \overline{O_{k+1}}. Put O_{r} := Ω. These orbits in 𝔫 generate nilpotent adjoint G-orbits

O_{k}^{G} := G · O_{k} ⊆ 𝔤,  k = 0, …, r.

Via the Kostant–Sekiguchi correspondence these G-orbits correspond to nilpotent K_{ℂ}-orbits O_{0}^{K_{ℂ}}, …, O_{r}^{K_{ℂ}} in 𝔭^{+} ⊆ 𝔭_{ℂ}. Under the Cayley type transform C ∈ Int(𝔤_{ℂ}) these K_{ℂ}-orbits map to

X_{k} := C(O_{k}^{K_{ℂ}}) ⊆ \overline{𝔫}_{ℂ} ≅ V_{ℂ},  k = 0, …, r.

Since C(𝔨_{ℂ}) = 𝔩_{ℂ} each X_{k} is an L_{ℂ} orbit in V_{ℂ}, where L_{ℂ} is a complexification of L, acting linearly on V_{ℂ}. Under the identifications described above O_{k} ⊆ X_{k} is a totally real submanifold.

The Schrödinger model

Rossi–Vergne [48] showed that the scalar type unitary highest weight representation corresponding to the Wallach point λ ∈ W can be realized on a space of L^{2}-functions on the orbit O_{k} for λ = k \frac{d}{2}, k = 0, …, r − 1, and on the orbit O_{r} = Ω for λ > (r − 1)\frac{d}{2}. For convenience we put O_{λ} := O_{k} where either λ = k \frac{d}{2}, 0 ≤ k ≤ r − 1, or k = r for λ > (r − 1)\frac{d}{2}. For each λ ∈ W there is a certain L-equivariant measure dμ_{λ} on O_{λ} (see Section 1.2 for details). Then the natural irreducible unitary representation of L ⋉ exp(𝔫) on L^{2}(O_{λ}, dμ_{λ}) extends to an irreducible unitary representation π_{λ} of G̃ which is the scalar type highest weight representation belonging to the Wallach point λ ∈ W. The differential representation dπ_{λ} of 𝔤 in this realization was computed in [22] and is given by differential operators on O_{λ} up to order 2 with polynomial coefficients. A special role is played by the second order vector-valued Bessel operator B_{λ} which gives the differential action of \overline{𝔫} (see Section 1.6 for its definition). This operator was first introduced by Dib [10] and studied further in [22].

Bessel functions on Jordan algebras

For each λ ∈ W we construct Bessel functions J_{λ}(z, w) and I_{λ}(z, w) for z, w ∈ X_{λ} := L_{ℂ} · O_{λ} (see Sections 3.1 and 3.2). Both J_{λ}(z, w) and I_{λ}(z, w) can be extended to functions on V_{ℂ} × \overline{X_{λ}} which are holomorphic in z and antiholomorphic in w. They solve the differential equations

(B_{λ})_{z}J_{λ}(z, w) = −\overline{w}J_{λ}(z, w),  (B_{λ})_{z}I_{λ}(z, w) = \overline{w}I_{λ}(z, w).

For w = e the identity element and λ > (r−1)\frac{d}{2} the corresponding one-variable functions J_{λ}(x) = J_{λ}(x, e) and I_{λ}(x) = I_{λ}(x, e) on Ω were first studied by Herz [21] for real symmetric matrices and later for the more general case in [10, 11, 12, 15, 16]. We further define a K-Bessel function K_{λ}(x) on O_{λ} for every λ ∈ W which solves the same differential equation as I_{λ}(x), but has a different asymptotic behaviour (see Section 3.3). These functions were introduced by Dib [10] and studied further in [8, 15].

The Fock space

The closure of X_{k} ≅ O_{k}^{K_{ℂ}} is an affine algebraic variety and hence the space ℂ[\overline{X_{k}}] of regular functions on \overline{X_{k}} equals the space P(X_{k}) of restrictions of holomorphic polynomials on V_{ℂ} to X_{k}. We construct measures on X_{k} in two steps. Fix λ ∈ W. First the L-equivariant measure dμ_{λ} on O_{λ} ⊆ X_{λ} gives rise to an L_{ℂ}-equivariant measure dν_{λ} on X_{λ} for every λ ∈ W. In the second step we extend the function

ω_{λ}(x) := K_{λ} ((\frac{x}{2})^{2}) ,  x ∈ O_{λ},

uniquely to a positive density on X_{λ} = L_{ℂ}·O_{λ} which, under the identification of X_{λ} with a K_{ℂ}-orbit, is invariant under the action of K. Then every polynomial F ∈ P(X_{λ}) is square integrable on X_{λ} with respect to the measure ω_{λ} dν_{λ} and we let F_{λ} be the closure of P(X_{λ}) with respect to the inner product

⟨F, G⟩_{F_{λ}} := const · ∫_{X_{λ}} F (z)\overline{G(z)}ω_{λ}(z) dν_{λ}(z),  F, G ∈ P(X_{λ}).

Theorem A (see Proposition 4.4 & Theorems 4.15 & 5.7). For each λ ∈ W the space F_{λ} is a Hilbert space of holomorphic functions on X_{λ} with reproducing kernel

𝕂_{λ}(z, w) = I_{λ} (\frac{z}{2}, \frac{w}{2} ) ,  z, w ∈ X_{λ}.

We further have the following intrinsic description:

F_{λ} = {F |_{X_{λ}} : F : V_{ℂ} → ℂ holomorphic and ∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) < ∞} .

In Section 4.2 we prove that the inner product on F_{λ} can also be expressed as a Fischer type inner product involving the Bessel operator B_{λ}.

Unitary action on the Fock space

Complexifying the differential operators dπ_{λ}(X), X ∈ 𝔤, in the differential representation of the Schrödinger model one obtains a Lie algebra representation dπ_{λ}^{ℂ} of 𝔤 on holomorphic functions on X_{λ} by polynomial differential operators (see Section 2.5 for details). We define a 𝔤-module structure on P(X_{λ}) by composing this action with the Cayley type transform C ∈ Int(𝔤_{ℂ}):

dρ_{λ}(X) := dπ_{λ}^{ℂ}(C(X)),  X ∈ 𝔤.

Theorem B (see Propositions 4.9, 4.10 & 4.17 & Theorem 4.11). For each λ ∈ W the 𝔤-module ( dρ_{λ}, P(X_{λ})) is an irreducible (𝔤, K̃)-module which is infinitesimally unitary with respect to the inner product on F_{λ}. It integrates to an irreducible unitary representation ρ_{λ} of G̃ on F_{λ} with associated variety

Ass(ρ_{λ}) = \overline{O_{k}^{K_{ℂ}}},

where k ∈ {0, …, r} is such that X_{λ} = X_{k}.

The group action of K̃ in the Fock model is induced by the geometric action of K_{ℂ} on the orbits X_{k} ≅ O_{k}^{K_{ℂ}} up to multiplication with a character (see Proposition 4.12). Therefore the K̃-type decomposition of ρ_{λ} equals the decomposition of the space of polynomials on O_{k}^{K_{ℂ}} ⊆ 𝔭^{+} with respect to the natural K_{ℂ}-action which is essentially the Hua–Kostant–Schmid decomposition (see Proposition 4.8). In Section 4.5 we use the Fock model to give explicit generators for the rings 𝔻(\overline{X_{k}}) of differential operators on the singular affine algebraic varieties \overline{X_{k}} ⊆ V_{ℂ}, k = 0, …, r − 1, in terms of the Bessel operator B_{λ}. Although there are already known explicit realizations for highest weight representations on spaces of holomorphic functions our construction has a certain advantage. In the realization on holomorphic functions on the corresponding bounded symmetric domain G/K the group action is quite explicit, but the representation space as well as the inner product are defined in a rather abstract way, at least for the singular highest weight representations. In our Fock model the representation space is by Theorem A explicitly given by holomorphic functions on V_{ℂ} which are square integrable with respect to a certain measure on the orbit X_{λ}. Although our group action is less explicit we still have a useful expression of the Lie algebra action in terms of the Bessel operators which allows e.g. the study of discrete branching laws as done in Theorem E.

The Segal–Bargmann transform

The unitary representations (π_{λ}, L^{2}(O_{λ}, dμ_{λ})) and (ρ_{λ}, F_{λ}) are isomorphic and we explicitly find the integral kernel of the intertwining operator which resembles the classical Segal–Bargmann transform (0.1). For this let tr ∈ V_{ℂ}^{∗} denote the Jordan trace of the complex Jordan algebra V_{ℂ} and recall the I-Bessel function I_{λ}(z, w).

Theorem C (see Proposition 5.1 & Theorem 5.4). Let λ ∈ W, then for ψ ∈ L^{2}(O_{λ}, dμ_{λ}) the formula

𝔹_{λ}ψ(z) = e^{− \frac{1}{2} tr(z)} ∫_{O_{λ}} I_{λ}(z, x)e^{− tr(x)}ψ(x) dμ_{λ}(x),  z ∈ X_{λ},

defines a function 𝔹_{λ}ψ ∈ F_{λ}. This gives a unitary isomorphism 𝔹_{λ} : L^{2}(O_{λ}, dμ_{λ}) → F_{λ} which intertwines the representations π_{λ} and ρ_{λ}.

The 𝔨-finite vectors in the Fock model and in the bounded symmetric domain model have the same realization as polynomials on V_{ℂ} and therefore also the intertwiner between these two models is worth studying. In Theorem 5.10 we find an explicit formula for it in terms of its integral kernel. We remark that Brylinski–Kostant [7] construct a Fock space realization for minimal representations of non-Hermitian Lie groups as geometric quantization of the minimal nilpotent K_{ℂ}-orbit in 𝔭_{ℂ}^{∗} (see also [2]). Apart from the fact that their cases are disjoint to ours they do not construct an intertwining operator between the known Schrödinger model and their Fock model such as our generalized Segal–Bargmann transform. Further their measure on the K_{ℂ}-orbit is not positive whereas our measure is explicitly given in terms of the K-Bessel function and hence positive. Moreover, the Lie algebra action in their picture is given by pseudodifferential operators while in our model the Lie algebra acts by differential operators up to second order. Recently Achab [1] also constructed a Fock space realization for the minimal representation of Hermitian groups. Her construction looks different from ours and it would be interesting to find a connection between her model and our model.

The unitary inversion operator

The explicit integral formula for the Segal–Bargmann transform can be used to find the integral kernel of the unitary inversion operator U_{λ} in the Schrödinger model. This operator U_{λ} is a unitary automorphism on L^{2}(O_{λ}, dμ_{λ}) of order 2 (see Section 6 for its precise definition). The action of U_{λ} together with the relatively simple action of a maximal parabolic subgroup generates the whole group action in the Schrödinger model. To describe its integral kernel recall the J-Bessel function J_{λ}(z, w).

Theorem D (see Theorem 6.3). For each λ ∈ W the unitary inversion operator U_{λ} is given by

U_{λ}ψ(x) = 2^{−rλ} ∫_{O_{λ}} J_{λ}(x, y)ψ(y) dμ_{λ}(y),  ψ ∈ C_{c}^{∞}(O_{λ}).

In various special cases Theorem D was proved earlier. Ding–Gross [11, Theorem 4.10 (iii)] and Faraut–Koranyi [15, Theorem XV.4.1] established Theorem D for the relative holomorphic discrete series, i.e. λ is contained in the continuous part of the Wallach set and is ‘large enough’. (In fact Ding–Gross proved an analogous version of Theorem D for all holomorphic discrete series representations.) For 𝔤 = 𝔰𝔲(k, k) and λ in the discrete part of the Wallach set Theorem D was proved by Ding–Gross–Kunze–Richards [12, Theorem 5.7]. For 𝔤 = 𝔰𝔬(2, n) and λ the minimal non-zero discrete Wallach point Kobayashi– Mano gave two different proofs for this result (see [33, Theorem 6.1.1] and [34, Theorem 5.1.1]). The case of the minimal non-zero discrete Wallach point was systematically treated in [23]. The proof we present for the general case is quite simple, once the Bargmann transform is established, and works in the same way for all scalar type unitary highest weight representations. In Section 6.2 we use the integral kernel of U_{λ} to construct explicit algebraic and smooth Whittaker vectors in the Schrödinger model.

Application to branching laws

We demonstrate the use of our Fock model in one specific example, the branching 𝔰𝔬(2, n) ↘ 𝔰𝔬(2, m) ⊕ 𝔰𝔬(n − m) of the minimal unitary highest weight representation of 𝔰𝔬(2, n). Note that the subalgebra 𝔥 := 𝔰𝔬(2, m) ⊕ 𝔰𝔬(n − m) of 𝔤 = 𝔰𝔬(2, n) is symmetric and of holomorphic type, i.e. the corresponding involution of 𝔤 acts as the identity on the center of 𝔨. Kobayashi [32] proved that any highest weight representation of scalar type if restricted to a holomorphic type subalgebra is discretely decomposable and the decomposition is multiplicity free. He further determined the explicit decomposition for representations in the relative holomorphic discrete series. For arbitrary highest weight representation he gives a general algorithm to find the explicit branching law. We find the explicit decomposition for the non-zero discrete Wallach point by an easy computation in the Fock model. For j = m, n we denote by dπ_{λ}^{𝔰𝔬(2,j)} the unitary highest weight representation of 𝔰𝔬(2, j) of scalar type with Wallach parameter λ as constructed above. Further we let H^{k}(ℝ^{n−m}) be the irreducible representation of 𝔰𝔬(n − m) on the space of spherical harmonics on ℝ^{n−m} of degree k.

Theorem E (see Theorem 7.2). The unitary highest weight representation of 𝔤 = 𝔰𝔬(2, n) belonging to the minimal non-zero discrete Wallach point λ = \frac{n−2}{2} decomposes under restriction to the subalgebra 𝔥 = 𝔰𝔬(2, m) ⊕ 𝔰𝔬(n − m) as follows:

dπ_{λ}^{𝔰𝔬(2,n)} = ⨁_{k=0}^{∞} dπ_{λ+k}^{𝔰𝔬(2,m)} ⊠ H^{k}(ℝ^{n−m}).

We remark that for odd n the representation dπ_{λ}^{𝔰𝔬(2,n)} cannot be obtained via the theta correspondence and hence also the branching law cannot be obtained in this way. For even n this branching law was proved by Kobayashi–Ørsted in [35, Theorem A] and, as pointed out to the author by B. Ørsted, their proof actually carries over also to the case of odd n, although it is not explicitly stated in their paper. However, the techniques they use are more involved than the computation we do in Section 7 to prove Theorem E. In the general case of a Hermitian Lie algebra 𝔤 Kobayashi–Oshima [36] recently classified all symmetric subalgebras of 𝔤 such that the restriction of the minimal scalar type unitary highest weight representation of 𝔤 is discretely decomposable. In [42] the explicit decompositions in these cases are determined.

Organization of the paper

This paper is organized as follows: In Section 1 we recall the necessary background of Euclidean Jordan algebras, their corresponding groups and nilpotent orbits. We also study polynomials and differential operators such as the Bessel operators on Jordan algebras. Section 2 describes three known models for unitary highest weight representations of scalar type, the bounded symmetric domain model, the tube domain model and the Schrödinger model. We further investigate a natural complexification of the latter one. Section 3 deals with Bessel functions on Jordan algebras. We introduce J-, I- and K-Bessel functions and discuss their properties. In Section 4 the Fock model for unitary highest weight representations of scalar type is constructed. We calculate the reproducing kernel of the Fock space and investigate rings of differential operators on the associated varieties. The Segal–Bargmann transform intertwining the Fock model and the Schrödinger model is introduced in Section 5. Here we also give a formula for the intertwiner between Fock model and bounded symmetric domain model. In Section 6 we use the Bargmann transform to obtain the integral kernel of the unitary inversion operator. We then describe Whittaker vectors in the Schrödinger model in terms of this integral kernel. Finally we use the Fock model in Section 7 to obtain in a very simple way the branching law for the smallest non-zero highest weight representation of 𝔰𝔬(2, n) restricted to 𝔰𝔬(2, m) ⊕ 𝔰𝔬(n − m).

Acknowledgements: We thank T. Kobayashi and B. Ørsted for helpful discussions on the topic of this paper.

Notation: ℕ = {1, 2, 3, …}, ℕ_{0} = ℕ ∪ {0}, ℝ^{×} = ℝ \ {0}, ℝ_{+} = {x ∈ ℝ : x > 0}, ℂ^{×} = ℂ \ {0}, 𝕋 = {z ∈ ℂ : |z| = 1}, V^{∗}: algebraic dual of a vector space V, V′: topological dual of a topological vector space V.

1 Preliminaries

In this section we set up the notation and recall the construction of three known models for unitary highest weight representations of Hermitian Lie groups of tube type. For this we use the theory of Euclidean Jordan algebras which formalizes various aspects in a simple fashion. We mostly follow the notation of [15].

1.1 Simple Euclidean Jordan algebras

Let V be a simple Euclidean Jordan algebra of dimension n = dim V with unit element e. For x ∈ V we denote by L(x) ∈ End(V) the multiplication by x and by P (x) = 2L(x)^{2} − L(x^{2}) the quadratic representation. Its polarized version is given by

P (x, y) = L(x)L(y) + L(y)L(x) − L(xy),  x, y ∈ V.

For x, y ∈ V we also define the box operator x□y ∈ End(V) by

x□y := L(xy) + [L(x), L(y)]

and note the identity (x□y)z = P (x, z)y for x, y, z ∈ V. The generic minimal polynomial of V provides the Jordan trace tr ∈ V^{∗} and the Jordan determinant Δ, a homogeneous polynomial of degree r = rk V, the rank of V. Both can be expressed in terms of L(x) and P (x) (see [15, Proposition III.4.2]):

Tr L(x) = \frac{n}{r} tr(x),  Det P (x) = Δ(x)^{\frac{2n}{r}} ,

for x ∈ V (\frac{n}{r} is always an integer). Here we use Tr and Det for the trace and determinant of a linear operator. Using the Jordan trace we define the trace form

(x|y) := tr(xy),  x, y ∈ V,

which is positive definite since V is Euclidean. Both L(x) and P (x) are symmetric operators with respect to the trace form (see [15, Proposition II.4.3]). For any orthonormal basis (e_{α})_{α} ⊆ V with respect to the trace form we have the identity (see [15, Exercise III.6])

∑_{α} e_{α}^{2} = \frac{n}{r}e.  (1.1)

We use the trace form to normalize the Lebesgue measure dx on V. The trace form also defines a norm on V by

|x| := \sqrt{(x|x)},  x ∈ V.

Let c ∈ V be an idempotent, i.e. c^{2} = c. The only possible eigenvalues of the symmetric operator L(c) are 1, \frac{1}{2} and 0. Denote by V (c, 1), V (c, \frac{1}{2}) and V (c, 0) the corresponding eigenspaces. The decomposition

V = V (c, 1) ⊕ V (c, \frac{1}{2}) ⊕ V (c, 0)

is called Peirce decomposition. The subspaces V (c, 1) and V (c, 0) are themselves simple Euclidean Jordan algebras with identity elements c and e − c, respectively. An idempotent c ∈ V is called primitive if it is non-zero and cannot be written as a nontrivial sum of two idempotents. Further, two idempotents c_{1}, c_{2} ∈ V are called orthogonal if c_{1}c_{2} = 0. (This implies (c_{1}|c_{2}) = 0.) A set of primitive orthogonal idempotents which add up to the identity e is called a Jordan frame. The cardinality of every Jordan frame is equal to the rank r of V. Further we say that an idempotent c ∈ V has rank k if it can be written as the sum of k primitive orthogonal idempotents. From now on we fix a Jordan frame c_{1}, …, c_{r}. The symmetric operators L(c_{1}), …, L(c_{r}) commute pairwise and therefore we obtain the mutual Peirce decomposition

V = ⨁_{1≤i≤j≤r} V_{ij},

where

V_{ii} = V (c_{i}, 1), (1 ≤ i ≤ r), V_{ij} = V (c_{i}, \frac{1}{2}) ∩ V (c_{j}, \frac{1}{2}), (1 ≤ i < j ≤ r).

The spaces V_{ii} are one-dimensional and spanned by c_{i} whereas the spaces V_{ij}, 1 ≤ i < j ≤ r have a common dimension d. In particular, this implies the dimension formula

\frac{n}{r} = 1 + (r − 1)\frac{d}{2}.

1.2 The structure group and its orbits

The structure group Str(V) of V is defined to be the group of g ∈ GL(V) such that

P (gx) = gP (x)g^{#},  x ∈ V,

where g^{#} denotes the adjoint with respect to the trace form. This is by [15, Exercise VIII.5] equivalent to the existence of a scalar χ(g) ∈ ℝ^{×} such that

Δ(gx) = χ(g)Δ(x),  x ∈ V.

The map χ : Str(V) → ℝ^{×} defines a character of Str(V) which is on the identity component L := Str(V)_{0} given by (see [15, Proposition III.4.3])

χ(g) = Det(g)^{\frac{r}{n}} ,  g ∈ L.

Let O(V) denote the orthogonal group of V with respect to the trace form. Then K^{L} := L ∩ O(V) is a maximal compact subgroup of L, the corresponding Cartan involution being ϑ(g) = (g^{−1})^{#}. The group K^{L} coincides with the identity component of the group of Jordan algebra automorphisms, i.e.

K^{L} = {g ∈ L : g(x · y) = gx · gy ∀x, y ∈ V }.

At the same time K^{L} is the stabilizer subgroup of the unit element e, i.e.

K^{L} = {g ∈ L : ge = e}.

The Lie algebra 𝔩 = 𝔰𝔱𝔯(V) of L has the Cartan decomposition (see [15, Proposition VIII.2.6])

𝔩 = 𝔨^{𝔩} ⊕ 𝔭^{𝔩},

where 𝔨^{𝔩} is the Lie algebra of K^{L} consisting of all derivations of V and 𝔭^{𝔩} is the space of multiplication operators:

𝔨^{𝔩} = {D ∈ 𝔤𝔩(V) : D(x · y) = Dx · y + x · Dy, ∀x, y ∈ V },
𝔭^{𝔩} = L(V) = {L(x) : x ∈ V }.

The group L acts on V with finitely many orbits. The orbit Ω := L · e is a symmetric cone and is isomorphic to the Riemannian symmetric space L/K^{L}. The Lebesgue measure dx on V clearly restricts to Ω. The following integral formula due to J.-L. Clerc [8, Proposition 2.7] is used in Section 3.3 to calculate K-Bessel functions on boundary orbits:

Lemma 1.1. Let c ∈ V be an idempotent of rank k and let Ω_{1} and Ω_{0} denote the symmetric cones in the Euclidean subalgebras V (c, 1) and V (c, 0), respectively. Further we let Δ^{0}(x) be the determinant function of the Euclidean Jordan algebra V (c, 0). Then the following integral formula holds:

∫_{Ω} f(x) dx = 2^{−k(r−k)d} ∫_{Ω_{1}} ∫_{V (c, \frac{1}{2} )} ∫_{Ω_{0}} f (exp(c□x_{\frac{1}{2}} )(x_{1} + x_{0}))

Δ^{0}(x_{0})^{kd} dx_{0} dx_{\frac{1}{2}} dx_{1}.

The boundary ∂Ω of the symmetric cone Ω has a stratification into lower dimensional orbits. In fact

\overline{Ω} = O_{0} ∪ O_{1} ∪ … ∪ O_{r},  (1.2)

where

O_{j} := L · e_{j}  with  e_{j} = c_{1} + ⋯ + c_{j}.

(We use the convention e_{0} = 0 here.) Note that O_{r} = Ω. Each x ∈ O_{j} has a polar decomposition x = ka with k ∈ K^{L} and a = ∑_{i=1}^{j} a_{i}c_{i}, a_{1}, …, a_{j} > 0 (see [15, Chapter VI.2]). Assuming a_{1} ≥ … ≥ a_{j} > 0 the element a is unique. We define complex powers of x = ka ∈ O_{k} by x^{s} := ka^{s}, a^{s} = ∑_{i=1}^{j} a_{i}^{s}c_{i} for s ∈ ℂ. Note that for s ∈ ℕ this definition agrees with the usual definition of powers by the Jordan algebra multiplication. Since every k ∈ K^{L} is a Jordan algebra automorphism the identity x^{s} · x^{t} = x^{s+t} holds for s, t ∈ ℂ. The Gamma function of Ω is for Re λ > (r − 1)\frac{d}{2} defined by the absolutely converging integral

Γ_{Ω}(λ) := ∫_{Ω} e^{− tr(x)}Δ(x)^{λ− \frac{n}{r}} dx

and is extended meromorphically to the whole complex plane by the identity (see [15, Theorem VII.1.1])

Γ_{Ω}(λ) = (2π)^{\frac{n−r}{2}} ∏_{j=1}^{r} Γ (λ − (j − 1)\frac{d}{2}).

Using the Gamma function we define the Riesz distributions R_{λ} ∈ S′(V) by

⟨R_{λ}, φ⟩ := \frac{2^{rλ}}{Γ_{Ω}(λ)} ∫_{Ω} φ(x)Δ(x)^{λ− \frac{n}{r}} dx,  φ ∈ S(V).

This integral converges for λ > (r − 1)\frac{d}{2} and defines a tempered distribution R_{λ} ∈ S′(V) which has an analytic continuation to all λ ∈ ℂ (see [15, Proposition VII.2.1 & Theorem VII.2.2]). The distribution R_{λ} is positive if and only if λ belongs to the Wallach set (sometimes referred to as the Berezin–Wallach set, see [15, Theorem VII.3.1])

W = \{0, \frac{d}{2}, …, (r − 1)\frac{d}{2}\} ∪ ((r − 1)\frac{d}{2}, ∞) .

Note that the Wallach set consists of a discrete part W_{disc} and a continuous part W_{cont} given by

W_{disc} = \{0, \frac{d}{2}, …, (r − 1)\frac{d}{2}\} ,  W_{cont} = ((r − 1)\frac{d}{2}, ∞) .

For λ ∈ W_{cont} in the continuous part the distribution R_{λ} is an L-equivariant measure dμ_{λ} supported on \overline{Ω} which is absolutely continuous with respect to the Lebesgue measure dx. Hence the boundary ∂Ω = \overline{O_{r−1}} is a set of measure zero. For λ = k \frac{d}{2} ∈ W_{disc} in the discrete part the distribution R_{λ} is an Lequivariant measure dμ_{λ} supported on \overline{O_{k}} for which ∂O_{k} = \overline{O_{k−1}} is a set of measure zero (see [15, Proposition VII.2.3]). If we put

O_{λ} := \begin{cases}O_{k} & for λ = k \frac{d}{2}, 0 ≤ k ≤ r − 1, \\ Ω & for λ > (r − 1)\frac{d}{2}, \end{cases}

then we obtain measure spaces (O_{λ}, dμ_{λ}) for λ ∈ W. The L-equivariant measures dμ_{λ} transform by

dμ_{λ}(gx) = χ(g)^{λ} dμ(x),  g ∈ L.

Note that with this normalization of the measures dμ_{λ} the function ψ_{0}(x) = e^{− tr(x)} always has norm 1 in L^{2}(O_{λ}, dμ_{λ}). In fact, for λ > (r − 1)\frac{d}{2} we find

∫_{O_{λ}} |ψ_{0}(x)|^{2} dμ_{λ}(x) = \frac{2^{rλ}}{Γ_{Ω}(λ)} ∫_{Ω} e^{−2 tr(x)}Δ(x)^{λ− \frac{n}{r}} dx

= \frac{1}{Γ_{Ω}(λ)} ∫_{Ω} e^{− tr(x)}Δ(x)^{λ− \frac{n}{r}} dx = 1,

and the general case follows by analytic continuation. In terms of the polar decomposition of each orbit O_{k} we have the following integral formula (cf. [22, Proposition 1.8])

∫_{O_{λ}} f(x) dμ_{λ}(x) = const · ∫_{K^{L}} ∫_{t_{1}>...>t_{k}} f(ke^{t})J_{λ}(t) dt dk,  (1.3)

where we use the notation t = (t_{1}, …, t_{k}),

e^{t} = ∑_{i=1}^{k} e^{t_{i}}c_{i},  J_{λ}(t) = e^{^{\frac{rλ}{k}} ∑{k \atop j=1} t_{j}} ∏_{1≤i<j≤k} sinh^{d} (\frac{t_{i} − t_{j}}{2} ),  (1.4)

and k ∈ {0, …, r} such that O_{λ} = O_{k}.

1.3 Conformal group and KKT algebra

For a ∈ V let n_{a} be the translation by a, i.e.

n_{a}(x) = x + a,  x ∈ V.

Denote by N := {n_{a} : a ∈ V } the group of all translations. Further define the conformal inversion j by

j(x) := −x^{−1},  for x ∈ V invertible.

The map j defines a rational transformation of V of order 2. The conformal group Co(V) of V is defined as the subgroup of all rational transformations of V generated by N, Str(V) and j:

Co(V) := ⟨N, Str(V), j⟩_{grp}.

We let G := Co(V)_{0} be its identity component. G is a connected simple Hermitian Lie group of tube type with trivial center and every such group occurs in this fashion (see Table 1 for a classification). Conjugation with j defines a Cartan involution ϑ of G by

ϑ(g) = j ∘ g ∘ j,  g ∈ G.

Its restriction to L agrees with the previously introduced Cartan involution ϑ(g) = (g^{−1})^{#}, g ∈ L. Let K := G^{ϑ} be the corresponding maximal compact subgroup of K. The Lie algebra 𝔤 of G is called Kantor–Koecher–Tits algebra. It acts on V by quadratic vector fields of the form

X(z) = u + T z − P (z)v,  z ∈ V,

with u, v ∈ V and T ∈ 𝔩. For convenience we write X = (u, T, v) ∈ 𝔤 for short. In this notation the Lie bracket of X_{j} = (u_{j}, T_{j}, v_{j}), j = 1, 2, is given by (see [15, Proposition X.5.8])

[X_{1}, X_{2}] = (T_{1}u_{2} − T_{2}u_{1}, [T_{1}, T_{2}] + 2(u_{1}□v_{2}) − 2(u_{2}□v_{1}), −T_{1}^{#}v_{2} + T_{2}^{#}v_{1}).

This yields the grading

𝔤 = 𝔫 ⊕ 𝔩 ⊕ \overline{𝔫},

where 𝔫 ≅ V is the Lie algebra of N and \overline{𝔫} = ϑ𝔫. Thus,

𝔫 = {(u, 0, 0) : u ∈ V }

acts via constant vector fields, 𝔩 via linear vector fields and

\overline{𝔫} = \{(0, 0, v) : v ∈ V \}

by quadratic vector fields. Note that the abelian subalgebras 𝔫 and \overline{𝔫} together generate 𝔤 as a Lie algebra. On 𝔤 the Cartan involution ϑ acts via

ϑ(u, D + L(a), v) = (−v, D − L(a), −u)

and hence gives the Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔭 with

𝔨 = {(u, D, −u) : u ∈ V, D ∈ 𝔨^{𝔩}}, 𝔭 = {(u, L(a), u) : u, a ∈ V }.

The Lie algebra 𝔨 of K has a one-dimensional center (see [41, Lemma 1.6.2])

Z(𝔨) = ℝZ_{0},  Z_{0} = −\frac{1}{2}(e, 0, −e),

and hence the universal cover K̃ of K is the direct product of ℝ with a compact connected simply-connected semisimple group. Therefore the deck transformation group of the universal cover G̃ of G is a finite extension of ℤ. Under the action of K the space 𝔭_{ℂ} decomposes into two irreducible components

𝔭_{ℂ} = 𝔭^{+} ⊕ 𝔭^{−}.

In fact, 𝔭^{±} is the ±i eigenspace of ad(Z_{0}) on 𝔭_{ℂ} and is explicitly given by

𝔭^{±} = {(u, ±2iL(u), u) : u ∈ V_{ℂ}}.

We set

E := (e, 0, 0),  H := (0, 2 id, 0),  F := (0, 0, e).

Then (E, F, H) forms an 𝔰𝔩_{2}-triple in 𝔤. We define a Cayley type transform C ∈ Int(𝔤_{ℂ}) by the formula

C := exp(−\frac{1}{2}i ad(E)) exp(−i ad(F)).  (1.5)

It is then routine to check the following formulas:

C(a, 0, 0) = (\frac{a}{4}, iL(a), a),  (1.6)
C(0, L(a) + D, 0) = (i\frac{a}{4}, D, −ia),  (1.7)
C(0, 0, a) = (\frac{a}{4}, −iL(a), a).  (1.8)

From these formulas it is easy to see that the transform C induces isomorphisms

𝔨_{ℂ} →^{∼} 𝔩_{ℂ}, (u, D, −u) ↦ D + 2iL(u),
𝔭^{+} →^{∼} \overline{𝔫}_{ℂ}, (u, 2iL(u), u) ↦ (0, 0, 4u),
𝔭^{−} →^{∼} 𝔫_{ℂ}, (u, −2iL(u), u) ↦ (u, 0, 0).

Since 𝔫 and \overline{𝔫} together generate 𝔤 as a Lie algebra it is now immediate that 𝔭^{+} + 𝔭^{−} generates 𝔤_{ℂ} as a Lie algebra. Hence the real form {(a, L(b), a) : a, b ∈ V } of 𝔭^{+} + 𝔭^{−} generates 𝔤.

V  𝔠𝔬(V)  𝔰𝔱𝔯(V)  n  r  d
ℝ  𝔰𝔩(2, ℝ)  ℝ  1  1  0
Sym(k, ℝ) (k ≥ 2) 𝔰𝔭(k, ℝ)  𝔰𝔩(k, ℝ) ⊕ ℝ  \frac{1}{2}k(k + 1) k  1
Herm(k, ℂ) (k ≥ 2) 𝔰𝔲(k, k)  𝔰𝔩(k, ℂ) ⊕ ℝ  k^{2}  k  2
Herm(k, ℍ) (k ≥ 2) 𝔰𝔬^{∗}(4k)  𝔰𝔲^{∗}(2k) ⊕ ℝ  k(2k − 1) k  4
ℝ^{1,k−1} (k ≥ 3)  𝔰𝔬(2, k) 𝔰𝔬(1, k − 1) ⊕ ℝ  k  2 k − 2
Herm(3, 𝕆)  𝔢_{7(−25)}  𝔢_{6(−26)} ⊕ ℝ  27  3  8

Table 1: Simple Euclidean Jordan algebras, corresponding groups and structure constants

1.4 Complexifications

The complexification V_{ℂ} of V is a complex Jordan algebra. The Jordan trace of V_{ℂ} is given by the ℂ-linear extension of the Jordan trace tr of V. The same holds for the trace form which is the ℂ-bilinear extension of the trace form of V. By abuse of notation we use the same notation tr and (−|−) for the complex objects. A norm on V_{ℂ} is given by

|z| := \sqrt{(z|\overline{z})},  z ∈ V_{ℂ}.

V_{ℂ} can also be considered as a real Jordan algebra and we write W = V_{ℂ} for it. As real Jordan algebra W is simple since V is simple Euclidean. The trace form (−|−)_{W} of W is of signature (n, n) and explicitly given by (see [41, Lemma 1.2.3 (1)])

(z|w)_{W} = 2 ((Re(z)| Re(w)) − (Im(z)| Im(w))) = 2 Re(z|w), z, w ∈ W.

Note that (−|−)_{W} is not ℂ-linear. The identity component L_{ℂ} := Str(V_{ℂ})_{0} of the structure group of the complex Jordan algebra V_{ℂ} is a natural complexification of L (see [15, Proposition VIII.2.6]). As a real Lie group it is the same as Str(W)_{0}, the identity component of the structure group of the real Jordan algebra W. Let U ⊆ L_{ℂ} denote the analytic subgroup of L_{ℂ} with Lie algebra

𝔲 := 𝔨^{𝔩} + i𝔭^{𝔩},

then U is a maximal compact subgroup of L_{ℂ}. The isomorphism C : 𝔨_{ℂ} → 𝔩_{ℂ} introduced in Section 1.3 restricts to an isomorphism 𝔨 →^{∼} 𝔲 which integrates to a covering map η : K̃ → U ⊆ L_{ℂ} with differential

dη(u, D, −u) = D + 2iL(u).

Since 𝔲_{ℂ} = 𝔩_{ℂ} the subgroup U is totally real in L_{ℂ}. For g ∈ L_{ℂ} we denote by g^{#} the adjoint of g with respect to the ℂ-bilinear trace form (−|−) of V_{ℂ}. Since (−|−)_{W} = 2 Re(−|−) this is also the adjoint with respect to the trace form (−|−)_{W} of W. Further put

g^{∗} := \overline{g}^{#},  g ∈ L_{ℂ}.

Then g ↦ (g^{−1})^{∗} defines a Cartan involution of L_{ℂ} with corresponding maximal compact subgroup U. The determinant function Δ_{W} of the real Jordan algebra W is a polynomial of degree 2r. As in the Euclidean case an element g ∈ Str(W) is characterized by the property that there exists a scalar χ_{W} (g) ∈ ℝ^{×} such that

Δ_{W} (gz) = χ_{W} (g)Δ_{W} (z),  z ∈ W.

This defines a real character χ_{W} : L_{ℂ} → ℝ_{+}. The group L_{ℂ} has only one open orbit L_{ℂ} · e (see [31, Theorem 4.2]). Its boundary again has a stratification by lower-dimensional orbits. More precisely we have

\overline{L_{ℂ} · e} = X_{0} ∪ X_{1} ∪ … ∪ X_{r},

where X_{k} = L_{ℂ} · e_{k} (see [31, Theorem 4.2]). Note that O_{k} ⊆ X_{k} is totally real and X_{k} is closed under conjugation. The closure \overline{X_{k}} of each L_{ℂ}-orbit is an affine algebraic subvariety of V_{ℂ} (see [17, Theorem 2.9]). From [41, Section 1.5.2] it is easy to deduce the following formula for the dimension of X_{k}:

dim_{ℂ} X_{k} = dim_{ℝ} O_{k} = k + k(2r − k − 1)\frac{d}{2}.  (1.9)

As in the real case every z ∈ X_{k} has a polar decomposition z = ua with u ∈ U and a = ∑_{i=1}^{k} a_{i}c_{i}, a_{1} ≥ … ≥ a_{k} > 0, the element a being unique with this property (see [15, Proposition X.3.2]). For λ ∈ W and k ∈ {0, …, r} such that O_{λ} = O_{k} we similarly put X_{λ} := X_{k}.

Proposition 1.2. For every λ ∈ W the measure dν_{λ} on X_{λ} defined by

∫_{X_{λ}} f(z) dν_{λ}(z) = ∫_{U} ∫_{O_{λ}} f(ux^{\frac{1}{2}} ) dμ_{λ}(x) du  (1.10)

is the unique (up to scalar multiples) L_{ℂ}-equivariant measure on X_{λ} which transforms by χ_{W}^{λ}.

Proof. Except for the integral formula (1.10) this is essentially [22, Proposition 1.8]. By [22, Proposition 1.8] a χ_{W}^{λ} -equivariant measure on X_{λ} is given by

f ↦ ∫_{U} ∫_{t_{1}>...>t_{k}} f(ue^{t})J_{λ}^{W} (t) dt du,

where e^{t} is as in (1.4),

J_{λ}^{W} (t) = e^{^{\frac{2rλ}{k}} ∑{k \atop j=1} t_{j}} ∏_{1≤i<j≤k} sinh^{d} (\frac{t_{i} − t_{j}}{2} ) cosh^{d} (\frac{t_{i} − t_{j}}{2} ),

and k ∈ \{0, …, r\} such that X_{k} = X_{λ}. Using sinh x · cosh x = \frac{1}{2} sinh 2x we find

J_{λ}^{W} (t) = 2^{−k(k−1) \frac{d}{2}} e^{^{\frac{rλ}{k}} ∑{k \atop j=1} 2t_{j}} ∏_{1≤i<j≤k} sinh^{d} (t_{i} − t_{j}) = 2^{−k(k−1) \frac{d}{2}} J_{λ}(2t)

with J_{λ}(s) as in (1.4). Now note that K^{L} ⊆ U and (ke^{t})^{\frac{1}{2}} = ke^{\frac{1}{2} t} for k ∈ K^{L} and hence

∫_{U} ∫_{t_{1}>...>t_{k}} f(ue^{t})J_{λ}^{W} (t) dt du
= const · ∫_{U} ∫_{K^{L}} ∫_{t_{1}>...>t_{k}} f(uke^{t})J_{λ}(2t) dt dk du
= const · ∫_{U} ∫_{K^{L}} ∫_{s_{1}>...>s_{k}} f(uke^{\frac{1}{2} s})J_{λ}(s) ds dk du
= const · ∫_{U} ∫_{O_{λ}} f(ux^{\frac{1}{2}} ) dμ_{λ}(x) du,

where we have used the integral formula (1.3) for the last equality.

1.5 Nilpotent orbits and the Kostant–Sekiguchi correspondence

Nilpotent coadjoint G-orbits

We identify 𝔤^{∗} with 𝔤 by means of the Killing form and view coadjoint G-orbits on 𝔤^{∗} as adjoint G-orbits on 𝔤. Further we use the embedding

V ↪ 𝔤, u ↦ (u, 0, 0)

to identify the L-orbits O_{k} ⊆ V, k = 0, …, r, with L-orbits in 𝔫 ≅ V. Since 𝔫 is nilpotent, the G-orbits

O_{k}^{G} := G · (e_{k}, 0, 0)

are nilpotent adjoint orbits in 𝔤. Since further O_{k} = L · (e_{k}, 0, 0) we clearly have O_{k} ⊆ O_{k}^{G}. As usual we endow O_{k}^{G} with the Kirillov–Kostant–Souriau symplectic form. The following result was proved in [22, Theorem 2.9 (4)] for the minimal orbit O_{1}. For the general case the author could not find the statement in the existing literature.

Proposition 1.3. O_{k} ⊆ O_{k}^{G} is a Lagrangian subvariety.

Proof. Since O_{k} ⊆ 𝔫 and 𝔫 is abelian the symplectic form vanishes on O and it remains to show that dim O_{k}^{G} = 2 dim O_{k}. By (1.9) we have

dim O_{k} = k + k(2r − k − 1)\frac{d}{2}.

To determine dim O_{k}^{G} we note that

0 = [(u, T, v), (e_{k}, 0, 0)] = (T e_{k}, −2e_{k}□v, 0)

if and only if T e_{k} = 0 and v ∈ V (e_{k}, 0). Hence

dim O_{k}^{G} = dim O_{k} + dim(V/V (e_{k}, 0))
= dim O_{k} + dim V (e_{k}, 1) + dim V (e_{k}, \frac{1}{2})
= dim O_{k} + (k + k(k − 1)\frac{d}{2}) + k(r − k)d
= 2 dim O_{k}.

Nilpotent K_{ℂ}-orbits

Via the Kostant–Sekiguchi correspondence the nilpotent adjoint G-orbits O_{k}^{G} ⊆ 𝔤^{∗} correspond to nilpotent K_{ℂ}-orbits O_{k}^{K_{ℂ}} ⊆ 𝔭_{ℂ}^{∗}. Identifying 𝔭^{∗} with 𝔭 by means of the Killing form we view O_{k}^{K_{ℂ}} as K_{ℂ}-orbits in 𝔭_{ℂ} = 𝔭^{+} + 𝔭^{−}. Following [47] we let (E_{k}, F_{k}, H_{k}) be the strictly normal 𝔰𝔩_{2}-triple in 𝔤 given by

E_{k} := (e_{k}, 0, 0),  H_{k} := (0, 2L(e_{k}), 0),  F_{k} := (0, 0, e_{k}).

We form a new 𝔰𝔩_{2}-triple (E_{k}^{d}, F_{k}^{d}, H_{k}^{d}) by putting

E_{k}^{d} := \frac{1}{2}(E_{k} + F_{k} + iH_{k}) = \frac{1}{2}(e_{k}, 2iL(e_{k}), e_{k}),
H_{k}^{d} := i(E_{k} − F_{k}) = i(e_{k}, 0, −e_{k}),
F_{k}^{d} := \frac{1}{2}(E_{k} + F_{k} − iH_{k}) = \frac{1}{2}(e_{k}, −2iL(e_{k}), e_{k}).

Then E_{k}^{d} ∈ 𝔭^{+} and the K_{ℂ} orbit O_{k}^{K_{ℂ}} corresponding to O_{k}^{G} is given by

O_{k}^{K_{ℂ}} = K_{ℂ} · E_{k}^{d}.

Since 𝔭^{+} is K_{ℂ}-stable we have O_{k}^{K_{ℂ}} ⊆ 𝔭^{+}. If we use the embedding V_{ℂ} ↪ 𝔤_{ℂ}, u ↦ (0, 0, u) to identify the L_{ℂ}-orbits X_{k} ⊆ V_{ℂ}, k = 0, …, r, with L_{ℂ}-orbits in \overline{𝔫}_{ℂ} ≅ V_{ℂ} we obtain the following result:

Proposition 1.4. The Cayley type transform C ∈ Int(𝔤_{ℂ}) is a bijection from the K_{ℂ}-orbit O_{k}^{K_{ℂ}} ⊆ 𝔭^{+} onto the L_{ℂ}-orbit X_{k} ⊆ \overline{𝔫}_{ℂ} for every k ∈ \{0, …, r\}.

Proof. Observe that C(E_{k}^{d}) = (0, 0, 4e_{k}). Since the Lie algebra homomorphism C is a bijection from 𝔨_{ℂ} onto 𝔩_{ℂ} it gives a bijection between the orbits O_{k}^{K_{ℂ}} = K_{ℂ} · E_{k}^{d} and X_{k} = L_{ℂ} · (0, 0, 4e_{k}) which shows the claim.

1.6 Differential operators

General notation

For a map f : X → X on a (real or complex) vector space X we write Df(x) : X → X for its (real or complex) Jacobian at the point x ∈ X. For a complex-valued function g : X → ℂ we denote the directional derivative in the direction of a ∈ X by ∂_{a}g:

∂_{a}g(x) = \frac{d}{dt}|_{t=0} g(x + ta).

Bessel operators - the real case

Let J be a real semisimple Jordan algebra with trace form (−|−)_{J}. The gradient with respect to (−|−)_{J} will be denoted by \frac{∂}{∂x}. If (e_{α})_{α} ⊆ J is a basis of J with dual basis (ê_{α})_{α} ⊆ J with respect to the trace form then the gradient is in coordinates x = ∑_{α} x_{α}e_{α} ∈ J expressed as

\frac{∂}{∂x} = ∑_{α} \frac{∂}{∂x_{α}} ê_{α}.

In terms of the directional derivative the gradient is characterized by the identity (a |\frac{∂}{∂x} )_{J} = ∂_{a} for a ∈ J. The Bessel operator B_{λ}^{J} with parameter λ ∈ ℂ is a vector-valued second order differential operator on J given by

B_{λ}^{J} := P (\frac{∂}{∂x}) x + λ\frac{∂}{∂x}.

In coordinates it is given by

B_{λ}^{J} = ∑_{α,β} \frac{∂^{2}}{∂x_{α}∂x_{β}} P (ê_{α}, ê_{β})x + λ ∑_{α} \frac{∂}{∂x_{α}} ê_{α}.

Denote by ℓ the left-action of Str(J) on functions on J given by ℓ(g)f(x) = f(g^{−1}x). Then the Bessel operator satisfies the following equivariance property:

ℓ(g)B_{λ}^{J}ℓ(g^{−1}) = g^{#}B_{λ}^{J},  g ∈ Str(J),  (1.11)

where g^{#} denotes the adjoint with respect to the trace form of J. We further have the following product rule:

B_{λ}^{J} [f(x)g(x)] = B_{λ}^{J}f(x) · g(x) + 2P (\frac{∂f}{∂x} , \frac{∂g}{∂x}) x + f(x) · B_{λ}^{J}g(x). (1.12)

For either J = V or J = W = V_{ℂ} the operator B_{λ}^{J} is tangential to the orbit Str(J)_{0} · e_{k} if λ = k \frac{d}{2}, 0 ≤ k ≤ r − 1, and hence defines a differential operator on this orbit (see [41, Theorem 1.7.5]). On K^{L}-invariant (resp. U-invariant) functions on V (resp. W) we have the following formulas:

Proposition 1.5. (1) ([15, Theorem XV.2.7]) Let f ∈ C^{∞}(V) be K^{L}- invariant. Then with F (a_{1}, …, a_{r}) = f(a), a = ∑_{i=1}^{r} a_{i}c_{i}, we have

B_{λ}^{V} f(a) = ∑_{i=1}^{r} (B_{λ}^{V})^{i}F (a_{1}, …, a_{r})c_{i},

where

(B_{λ}^{V})^{i} = a_{i} \frac{∂^{2}}{∂a_{i}^{2}} +(λ − (r − 1)\frac{d}{2}) \frac{∂}{∂a_{i}} + \frac{d}{2} ∑_{j≠i} \frac{1}{a_{i} − a_{j}} (a_{i} \frac{∂}{∂a_{i}} − a_{j} \frac{∂}{∂a_{j}} ).

(2) ([43]) Let f ∈ C^{∞}(W) be U-invariant. Then with F (a_{1}, …, a_{r}) = f(a), a = ∑_{i=1}^{r} a_{i}c_{i}, we have

B_{λ}^{W} f(a) = ∑_{i=1}^{r} (B_{λ}^{W})^{i}F (a_{1}, …, a_{r})c_{i},

where

(B_{λ}^{W})^{i} = \frac{1}{4}(a_{i} \frac{∂^{2}}{∂a_{i}^{2}} + (2λ − 1 − (r − 1)d) \frac{∂}{∂a_{i}}

+ \frac{d}{2} ∑_{j≠i} (\frac{1}{a_{i} − a_{j}} + \frac{1}{a_{i} + a_{j}} ) (a_{i} \frac{∂}{∂a_{i}} − a_{j} \frac{∂}{∂a_{j}} )).

For J = V we will write B_{λ} := B_{λ}^{V} for short.

Bessel operators - the complex case

We also need the Bessel operator of the complex Jordan algebra V_{ℂ}. It is a holomorphic vector-valued differential operator on V_{ℂ} which is defined in the same way as the Bessel operator of V. More precisely, if (e_{α})_{α} ⊆ V_{ℂ} is a ℂ-basis of V_{ℂ} with dual basis (ê_{α})_{α} ⊆ V_{ℂ} with respect to the trace form (−|−) (which is ℂ-bilinear) we define the gradient of V_{ℂ} by

\frac{∂}{∂z} = ∑_{α} \frac{∂}{∂z_{α}} ê_{α},  where  \frac{∂}{∂z_{α}} = \frac{1}{2} (\frac{∂}{∂x_{α}} − i\frac{∂}{∂y_{α}} )

is the Wirtinger derivative and z = ∑_{α} z_{α}e_{α} with z_{α} = x_{α} + iy_{α}. The Bessel operator B_{λ}^{V_{ℂ}} of V_{ℂ} is then defined by

B_{λ}^{V_{ℂ}} = P (\frac{∂}{∂z} ) z + λ\frac{∂}{∂z} = ∑_{α,β} \frac{∂^{2}}{∂z_{α}∂z_{β}} P (ê_{α}, ê_{β})z + λ ∑_{α} \frac{∂}{∂z_{α}} ê_{α}.

For a holomorphic function f on V_{ℂ} we have

(B_{λ}^{V_{ℂ}}f)|_{V} = B_{λ}^{V} (f|_{V})

and hence B_{λ}^{V_{ℂ}} is a natural complexification of the Bessel operator B_{λ} of V. Therefore, by abuse of notation, we also abbreviate B_{λ} = B_{λ}^{V_{ℂ}}.

1.7 Bounded symmetric domains of tube type

Let T_{Ω} := V + iΩ ⊆ V_{ℂ} be the tube domain associated with the symmetric cone Ω. Each rational transformation g ∈ G extends to a holomorphic automorphism of T_{Ω}. This establishes an isomorphism between G and the identity component Aut(T_{Ω})_{0} of the group of all holomorphic automorphisms of T_{Ω} (see [15, Theorem X.5.6]). Under this isomorphism the maximal compact subgroup K ⊆ G corresponds to the stabilizer subgroup of the element ie ∈ T_{Ω} so that T_{Ω} ≅ G/K. We identify an element g ∈ G with its holomorphic extension to T_{Ω}. Let

p : T_{Ω} → V_{ℂ}, p(z) := (z − ie)(z + ie)^{−1},

and define D := p(T_{Ω}). Then p restricts to a holomorphic isomorphism p : T_{Ω} →^{∼} D with inverse the Cayley transform (see [15, Theorem X.4.3])

c : D →^{∼} T_{Ω}, c(w) = i(e + w)(e − w)^{−1}.  (1.13)

The open set D ⊆ V_{ℂ} is a bounded symmetric domain of tube type and every such domain occurs in this fashion. Clearly G is also isomorphic to the identity component Aut(D)_{0} of the group of all holomorphic automorphisms of D via the conjugation map

α : G → Aut(D)_{0}, g ↦ p ∘ g ∘ p^{−1} = c^{−1} ∘ g ∘ c.  (1.14)

The stabilizer subgroup of the origin 0 ∈ D corresponds to the maximal compact subgroup K ⊆ G and also D ≅ G/K. Denote by Σ ⊆ ∂D the Shilov boundary of D. The group U acts transitively on Σ and the stabilizer of e ∈ Σ is equal to K^{L} ⊆ U (see [15, Proposition X.3.1 & Theorem X.4.6]). Hence Σ ≅ U/K^{L} is a compact symmetric space. Denote by dσ the normalized U-invariant measure on Σ ≅ U/K^{L}. We write L^{2}(Σ) for L^{2}(Σ, dσ) and denote the corresponding L^{2}-inner product by ⟨−, −⟩_{Σ}.

1.8 Polynomials

In this Section we recall known properties for the space of polynomials on V_{ℂ}.

Principal minors and decompositions

Let P(V_{ℂ}) denote the space of holomorphic polynomials on V_{ℂ}. The space P(V_{ℂ}) carries a natural representation ℓ of L_{ℂ} given by

ℓ(g)p(z) = p(g^{−1}z),  g ∈ L_{ℂ}, p ∈ P(V_{ℂ}).

Since L and U are both real forms of the complex group L_{ℂ} the decompositions of P(V_{ℂ}) into irreducible L- and U-representations are the same. This decomposition can be described as follows:

For j ∈ {1, …, r} we denote by P_{j} the orthogonal projection V → V (e_{j}, 1). Let Δ_{V (e_{j},1)} be the Jordan determinant of the Euclidean Jordan algebra V (e_{j}, 1). We define a polynomial Δ_{j} on V by the formula

Δ_{j}(x) := Δ_{V (e_{j},1)}(P_{j}x),  x ∈ V,

and extend it to a holomorphic polynomial on V_{ℂ}. The polynomials Δ_{1}, …, Δ_{r} are called principal minors of V. Note that Δ_{r} = Δ is the Jordan determinant of V. For m ∈ ℕ_{0}^{r} we write m ≥ 0 if m_{1} ≥ … ≥ m_{r} ≥ 0. If m ≥ 0 we define a polynomial Δ_{m} on V_{ℂ} by

Δ_{m}(z) := Δ_{1}(z)^{m_{1}−m_{2}} ⋯ Δ_{r−1}(z)^{m_{r−1}−m_{r}}Δ_{r}(z)^{m_{r}},  z ∈ V_{ℂ}.

The polynomials Δ_{m}(z) are called generalized power functions. Define P_{m}(V_{ℂ}) to be the subspace of P(V_{ℂ}) spanned by ℓ(g)Δ_{m} for g ∈ L. (Equivalently one can let g ∈ U or g ∈ L_{ℂ}.) We write d_{m} := dim P_{m}(V_{ℂ}) for its dimension.

Theorem 1.6 (Hua–Kostant–Schmid, see e.g. [15, Theorem XI.2.4]). For each m ≥ 0 the space P_{m}(V_{ℂ}) is an irreducible L-module and the space P(V_{ℂ}) decomposes into irreducible L-modules as

P(V_{ℂ}) = ⨁_{m≥0} P_{m}(V_{ℂ}).

The same spaces P_{m}(V_{ℂ}) occur in the decomposition of L^{2}(Σ) into irreducible U-representations. To make this precise let ℤ_{+}^{r} be the set of all m ∈ ℤ^{r} with m_{1} ≥ … ≥ m_{r}. For m ∈ ℤ_{+}^{r} we define another tuple m′ := (m_{1} − m_{r}, …, m_{r−1} − m_{r}, 0). Then m′ ≥ 0 and we define

P_{m}(Σ) := {Δ^{m_{r}}p|_{Σ} : p ∈ P_{m′}(V_{ℂ})}.

If m ≥ 0 then P_{m}(Σ) coincides with the space of restrictions of polynomials in P_{m}(V_{ℂ}) to Σ.

Theorem 1.7 (Cartan–Helgason, see e.g. [15, Theorem XII.2.2]). For each m ∈ ℤ_{+}^{r} the space P_{m}(Σ) is an irreducible unitary U-representation and the space L^{2}(Σ) decomposes into the direct Hilbert space sum

L^{2}(Σ) = ˆ⨁_{m∈ℤ_{+}^{r}} P_{m}(Σ).

We now study the restriction of polynomials to the orbits X_{k} ⊆ V_{ℂ} and O_{k} ⊆ V. Note that for m ≥ 0 and k ∈ {0, …, r − 1} the condition m_{k+1} = 0 means m_{k+1} = … = m_{r} = 0. For convenience we also use this notation for k = r. In this case m_{k+1} = 0 should mean no restriction on m.

Proposition 1.8 ([24, Proposition I.7 (vi) (a)]). For x ∈ X_{k} we have Δ_{k+1}(x) = … = Δ_{r}(x) = 0. In particular, Δ_{m} vanishes on X_{k} iff m_{k+1} ≠ 0, m ≥ 0.

For 0 ≤ k ≤ r let P(X_{k}) and P_{m}(X_{k}) be the images of P(V_{ℂ}) and P_{m}(V_{ℂ}) under the restriction map C^{∞}(V_{ℂ}) → C^{∞}(X_{k}). Similar we define P(O_{k}) and P_{m}(O_{k}) to be the images of P(V_{ℂ}) and P_{m}(V_{ℂ}) under the restriction map C^{∞}(V_{ℂ}) → C^{∞}(O_{k}).

Corollary 1.9. Let k ∈ {0, …, r} and m ≥ 0. Then P_{m}(X_{k}) (resp. P_{m}(O_{k})) is non-trivial if and only if m_{k+1} = 0. In this case the restriction from V_{ℂ} to X_{k} (resp. O_{k}) induces an isomorphism P_{m}(V_{ℂ}) ≅ P_{m}(X_{k}) (resp. P_{m}(V_{ℂ}) ≅ P_{m}(O_{k})) of L-modules (resp. L_{ℂ}-modules). In particular we have the following decompositions:

P(X_{k}) = ⨁_{m≥0, m_{k+1}=0} P_{m}(X_{k}),  P(O_{k}) = ⨁_{m≥0, m_{k+1}=0} P_{m}(O_{k}).

For λ ∈ ℂ and m ≥ 0 we define the Pochhammer symbol (λ)_{m} by

(λ)_{m} := ∏_{i=1}^{r} (λ − (i − 1)\frac{d}{2})_{m_{i}},

where (a)_{n} = a(a + 1) ⋯ (a + n − 1) denotes the classical Pochhammer symbol for a ∈ ℂ, n ∈ ℕ_{0}.

Lemma 1.10 ([15, Lemma XI.2.3]). For p ∈ P_{m}(V_{ℂ}) and λ ∈ W we have

∫_{O_{λ}} e^{−(x|y)}p(x) dμ_{λ}(x) = 2^{rλ}(λ)_{m}Δ(y)^{−λ}p(y^{−1}),  y ∈ Ω.

Spherical polynomials

The representations P_{m}(V_{ℂ}) of L (resp. U) are K^{L}-spherical. The K^{L}- spherical vectors in P_{m}(V_{ℂ}) are spanned by (see [15, Proposition XI.3.1])

Φ_{m}(z) = ∫_{K^{L}} Δ_{m}(kz) dk,  z ∈ V_{ℂ}.

The L^{2}(Σ)-norm of these functions are given by (see [15, Proposition XI.4.1 (i)])

∥Φ_{m}∥_{Σ}^{2} = \frac{1}{d_{m}} ,  m ≥ 0.  (1.15)

By [15, Corollary XI.3.4] there exists a unique polynomial Φ_{m}(z, w) on V_{ℂ} ×V_{ℂ} holomorphic in z and antiholomorphic in w such that

Φ_{m}(gz, w) = Φ_{m}(z, g^{∗}w),  ∀ g ∈ L_{ℂ},
Φ_{m}(x, x) = Φ_{m}(x^{2}),  ∀ x ∈ V.

Lemma 1.11. (1) For all z ∈ V_{ℂ} we have

Φ_{m}(z, e) = Φ_{m}(z).

(2) For x ∈ V and y ∈ \overline{Ω} we have

Φ_{m}(x, y) = Φ_{m}(P (y^{\frac{1}{2}} )x).

(3) For z, w ∈ V_{ℂ} we have

\overline{Φ_{m}(z, w)} = Φ_{m}(w, z).

Proof. (1) First let x ∈ Ω. Recall the complex powers x^{s}, s ∈ ℂ, defined in Section 1.2. Then

Φ_{m}(x, e) = Φ_{m}(P (x^{\frac{1}{4}} )x^{\frac{1}{2}} , e) = Φ_{m}(x^{\frac{1}{2}} , P (x^{\frac{1}{4}} )e) = Φ_{m}(x^{\frac{1}{2}} , x^{\frac{1}{2}} ) =Φ_{m}(x).

Since both sides are holomorphic in x and Ω ⊆ V_{ℂ} is totally real, this also holds for x ∈ V_{ℂ}.

(2) Let x ∈ V and y ∈ Ω. Then by (1) we obtain

Φ_{m}(x, y) = Φ_{m}(x, P (y^{\frac{1}{2}} )e) = Φ_{m}(P (y^{\frac{1}{2}} )x, e) = Φ_{m}(P (y^{\frac{1}{2}} )x).

Now both sides are continuous in y ∈ \overline{Ω} and the claim follows.

(3) First let x, y ∈ Ω. Then by [15, Lemma XIV.1.2] there exists k ∈ K^{L} such that P (y^{\frac{1}{2}} )x = kP (x^{\frac{1}{2}} )y. Using (2) and the K^{L}-invariance of Φ_{m} we find that

Φ_{m}(x, y) = Φ_{m}(P (y^{\frac{1}{2}} )x) = Φ_{m}(P (x^{\frac{1}{2}} )y) = Φ_{m}(y, x).

Since Φ_{m}(x, y) ∈ ℝ for x, y ∈ V, we obtain

\overline{Φ_{m}(z, w)} = Φ_{m}(w, z)  ∀ z, w ∈ Ω.

Both sides are holomorphic in z and antiholomorphic in w and Ω ⊆ V_{ℂ} is totally real. Hence, the formula also holds for z, w ∈ V_{ℂ}.

Lemma 1.12. If m ≥ 0 with m_{k+1} ≠ 0, then Φ_{m}(z, w) = 0 for all z ∈ V_{ℂ} and w ∈ \overline{X_{k}}.

Proof. Write w = he_{k} with g ∈ L_{ℂ}. Then

Φ_{m}(z, w) = Φ_{m}(z, ge_{k}) = Φ_{m}(g^{∗}z, e_{k}).

Therefore, it suffices to show that Φ_{m}(−, e_{k}) = 0 as a polynomial on V_{ℂ}. Since Φ_{m}(z, w) is holomorphic in the first variable it suffices to show that Φ_{m}(−, e_{k}) = 0 as a polynomial on V and since Ω ⊆ V is open it is enough to prove Φ_{m}(x, e_{k}) = 0 for x ∈ Ω. But for x ∈ Ω we have by Lemma 1.11 (2) and (3)

Φ_{m}(x, e_{k}) = Φ_{m}(P (x^{\frac{1}{2}} )e_{k}).

Now write x^{\frac{1}{2}} = ge with g ∈ L, then

P (x^{\frac{1}{2}} )e_{k} = P (ge)e_{k} = gP (e)g^{#}e_{k} = gg^{#}e_{k}

and hence P (x^{\frac{1}{2}} )e_{k} ∈ O_{k} = L · e_{k}. But by Corollary 1.9 we have Φ_{m}|_{O_{k}} = 0 since m_{k+1} ≠ 0 and the proof is complete.

Example 1.13. On the rank 1 orbit X_{1} the spherical polynomials Φ_{m}(z, w) are non-zero if and only if m = (m, 0, …, 0), m ∈ ℕ_{0}, and in this case

Φ_{m}(z, w) = \frac{(\frac{n}{r})_{m}}{d_{m}m!}(z|\overline{w})^{m},  z, w ∈ X_{1}.

(This can e.g. be derived from the expansion of tr(x)^{k} into the polynomials Φ_{m}(x), see [15, Chapter XI.5].)

The Fischer inner product

We equip P(V_{ℂ}) with the Fischer inner product

[p, q] := p (\frac{∂}{∂z} ) \overline{q}(z)|_{z=0} ,  p, q ∈ P(V_{ℂ}),

where \overline{q}(z) := \overline{q(\overline{z})}, z ∈ V_{ℂ}. The action of U on P(V_{ℂ}) is unitary with respect to this inner product and hence the irreducible constituents P_{m}(V_{ℂ}) are pairwise orthogonal. Since U also acts unitarily on P(V_{ℂ}) with respect to the inner product on L^{2}(Σ) the two inner products are proportional on each irreducible constituent P_{m}(V_{ℂ}) (see [15, Corollary XI.4.2]):

[p, q] = (\frac{n}{r})_{m} ⟨p, q⟩_{Σ},  p, q ∈ P_{m}(V_{ℂ}).  (1.16)

Denote by K^{m}(z, w) the reproducing kernel of P_{m}(V_{ℂ}) with respect to the Fischer inner product. By [15, Propositions XI.3.3 & XI.4.1] we have

K^{m}(z, w) = \frac{d_{m}}{(\frac{n}{r})_{m}} Φ_{m}(z, w),  z, w ∈ V_{ℂ}.

Since by [15, Proposition XI.1.1] the completion of P(V_{ℂ}) with respect to the Fischer inner product is a Hilbert space with reproducing kernel K(z, w) = e^{(z|\overline{w})} we obtain the following expansion:

e^{(z|\overline{w})} = ∑_{m≥0} \frac{d_{m}}{(\frac{n}{r})_{m}} Φ_{m}(z, w),  z, w ∈ V_{ℂ}.  (1.17)

Lemma 1.14. For m, n ≥ 0 and p ∈ P_{m}(V_{ℂ}), q ∈ P_{n}(V_{ℂ}):

∫_{U} p(uz)\overline{q(uw)} du = Φ_{m}(z, w)⟨p, q⟩_{Σ}  ∀ z, w ∈ V_{ℂ}.

Proof. Using the reproducing kernel property and the Schur orthogonality relations we obtain

∫_{U} p(uz)\overline{q(uw)} du = ∫_{U} [p, K^{m}(−, uz)]\overline{[q, K^{n}(−, uw)]} du

= ∫_{U} [ℓ(u^{−1})p, K^{m}(−, z)]\overline{[ℓ(u^{−1})q, K^{n}(−, w)]} du

= \frac{1}{d_{m}} [p, q]\overline{[K^{m}(−, z), K^{n}(−, w)]}.

With (1.16) and

\overline{[K^{m}(−, z), K^{n}(−, w)]} = δ_{mn}\overline{K^{m}(w, z)} = δ_{mn} \frac{d_{m}}{(\frac{n}{r})_{m}} Φ_{m}(z, w)

the claimed identity follows.

Laguerre functions

For m ≥ 0 the polynomial Φ_{m}(e + x) is K^{L}-invariant of degree |m| and can hence be written as a linear combination of the K^{L}-invariant polynomials Φ_{n}(x) for |n| ≤ |m|. Following [15, Chapter XV.4] we define the generalized binomial coefficients \binom{m}{n} by the formula

Φ_{m}(e + x) = ∑_{|n|≤|m|} \binom{m}{n}Φ_{n}(x),  x ∈ V.

Using the generalized binomial coefficients we define the generalized Laguerre polynomials L_{m}^{λ}(x) by

L_{m}^{λ}(x) := (λ)_{m} ∑_{|n|≤|m|} \binom{m}{n}\frac{1}{(λ)_{n}} Φ_{n}(−x),  x ∈ V,  (1.18)

and the generalized Laguerre functions ℓ_{m}^{λ}(x) by

ℓ_{m}^{λ}(x) := e^{− tr(x)}L_{m}^{λ}(2x),  x ∈ V.  (1.19)

Both L_{m}^{λ}(x) and ℓ_{m}^{λ}(x) are K^{L}-invariant. Note that for λ > (r − 1)\frac{d}{2} we have (λ)_{n} ≠ 0 for all n ≥ 0 and hence L_{m}^{λ}(x) and ℓ_{m}^{λ}(x) are defined on V. For x ∈ \overline{O_{k}}, k = 0, …, r − 1, we further have Φ_{n}(x) = 0 if n_{k+1} ≠ 0. Therefore the sum in (1.18) reduces to a sum over n with n_{k+1} = … = n_{r} = 0. For such n we have (λ)_{n} ≠ 0 for λ = k \frac{d}{2} and hence the expression (1.18) is defined. This gives Laguerre polynomials L_{m}^{λ}(x) and Laguerre functions ℓ_{m}^{λ}(x) on \overline{O_{λ}} for each λ ∈ W. Properties of the generalized Laguerre functions have been studied in [4, 9].

2 Three different realizations of unitary highest weight representations of scalar type

We describe three known realizations of highest weight representations of scalar type: the bounded symmetric domain model, the tube domain model and the Schrödinger model. We further discuss a complexification of the Schrödinger model which we use in Section 4.3 to construct yet another model, the Fock model.

2.1 The Schrödinger model

The highest weight representation of the universal cover G̃ belonging to the Wallach point λ ∈ W can be realized on L^{2}(O_{λ}, dμ_{λ}). We sketch the construction here (see e.g. [41] for details). First, there is a Lie algebra representation dπ_{λ} of 𝔤 on C^{∞}(O_{λ}) for every λ ∈ W given by

dπ_{λ}(u, 0, 0) = i(u|x),
dπ_{λ}(0, T, 0) = ∂_{T^{#}x} + \frac{rλ}{2n} Tr(T^{#}),
dπ_{λ}(0, 0, v) = i(v|B_{λ}).

Note that for λ = k \frac{d}{2}, k = 0, …, r − 1, the Bessel operator B_{λ} is tangential to the orbit O_{k} and hence defines a differential operator acting on C^{∞}(O_{k}) (see Section 1.6). The representation dπ_{λ} is further infinitesimally unitary with respect to the L^{2}-inner product on L^{2}(O_{λ}, dμ_{λ}). The subrepresentation of ( dπ_{λ}, C^{∞}(O_{λ})) generated by the function

ψ_{0}(x) = e^{− tr(x)},  x ∈ O_{λ},

defines a (𝔤, K̃)-module ( dπ_{λ}, W^{λ}) whose underlying vector space turns out to be (see e.g. [15, Proposition XIII.3.2])

W^{λ} = P(O_{λ})e^{− tr(x)}.

The (𝔤, K̃)-module ( dπ_{λ}, W^{λ}) integrates to an irreducible unitary representation (π_{λ}, L^{2}(O_{λ}, dμ_{λ})) of G̃. The minimal K̃-type is spanned by the function ψ_{0} which is K̃-equivariant:

π_{λ}(k)ψ_{0} = ξ_{λ}(k)ψ_{0},  k ∈ K̃,

where ξ_{λ} : K̃ → 𝕋 is the character of K̃ with differential

dξ_{λ}(u, T, −u) = iλ tr(u).

This implies that the representation π_{λ} descends to a finite cover of G if and only if λ ∈ ℚ (which holds in particular for λ ∈ W_{disc}). For λ > \frac{2n}{r} − 1 the representation π_{λ} belongs to the relative holomorphic discrete series of G̃.

2.2 The tube domain model

For λ ∈ ℂ consider the function

T_{Ω} × T_{Ω} → ℂ, (z, w) ↦ Δ (\frac{z − \overline{w}}{2i} )^{−λ} .

It is of positive type if and only if λ ∈ W (see [15, Proposition XIII.1.2 & Theorem XIII.2.4]). Denote by H_{λ}^{2}(T_{Ω}) the Hilbert space of holomorphic functions on T_{Ω} which has the function Δ(\frac{z−\overline{w}}{2i})^{−λ} as reproducing kernel. For λ > 1 + (r − 1)d this space coincides with the space of holomorphic functions F on T_{Ω} such that

∫_{T_{Ω}} |F (z)|^{2}Δ(y)^{λ− \frac{2n}{r}} dx dy < ∞,

where z = x + iy ∈ T_{Ω} (see [15, Chapter XIII.1]). For every λ ∈ W there is an irreducible unitary representation π_{λ}^{T_{Ω}} of G̃ on H_{λ}^{2}(T_{Ω}). Note that G̃ acts on T_{Ω} by composition of the action of G = Aut(T_{Ω})_{0} on T_{Ω} with the covering map G̃ → G. Then the representation π_{λ}^{T_{Ω}} is given by

π_{λ}^{T_{Ω}}(g)F (z) = μ_{λ}^{T_{Ω}}(g^{−1}, z)F (g^{−1}z),  g ∈ G̃, z ∈ T_{Ω},

where the cocycle μ_{λ}^{T_{Ω}}(g, z) is given by

μ_{λ}^{T_{Ω}}(g, z) = Det(Dg(z))^{\frac{rλ}{2n}} ,  g ∈ G̃, z ∈ T_{Ω},

the powers being well-defined on the universal cover G̃. The representations H_{λ}^{2}(T_{Ω}) and L^{2}(O_{λ}, dμ_{λ}) are isomorphic, the intertwining operator being the Laplace transform (see [15, Theorems XIII.1.1 & XIII.3.4])

L_{λ} : L^{2}(O_{λ}, dμ_{λ}) → H_{λ}^{2}(T_{Ω}), L_{λ}ψ(z) := ∫_{O_{λ}} e^{i(z|x)}ψ(x) dμ_{λ}(x).  (2.1)

The Laplace transform L_{λ} is a unitary isomorphism (up to a scalar) intertwining the group actions.

2.3 The bounded symmetric domain model

The polynomial h(x) := Δ(e − x^{2}), x ∈ V, is K^{L}-invariant and therefore by [15, Corollary XI.3.4] there exists a unique polynomial h(z, w) holomorphic in z ∈ V_{ℂ} and antiholomorphic in w ∈ V_{ℂ} such that

h(gz, w) = h(z, g^{∗}w),  ∀ g ∈ L_{ℂ},
h(x, x) = h(x),  ∀ x ∈ V.

We also write h(z) := h(z, z) for z ∈ V_{ℂ}. Consider the powers h(z, w)^{−λ} for λ ∈ ℂ as functions on D × D. Then h(z, w)^{−λ} is positive definite on D × D if and only if λ ∈ W (see [15, Theorem XIII.2.7]). The corresponding Hilbert space of holomorphic functions on D with reproducing kernel h(z, w)^{−λ} will be denoted by H_{λ}^{2}(D). For λ > 1 + (r − 1)d this space coincides with the space of holomorphic functions f on D such that

∫_{D} |f(w)|^{2}h(w)^{λ− \frac{2n}{r}} du dv < ∞,

where w = u + iv ∈ D (see [15, Proposition XIII.1.4]). For each λ ∈ W there is an irreducible unitary representation π_{λ}^{D} of G̃ on H_{λ}^{2}(D). To give an explicit formula recall the isomorphism α : G → Aut(D)_{0} defined in (1.14) and view it as a covering map α : G̃ → Aut(D)_{0}. Then the representation π_{λ}^{D} is given by

π_{λ}^{D}(g)f(w) = μ_{λ}^{D}(g^{−1}, w)f(α(g)^{−1}w),  g ∈ G̃, w ∈ D,

where the cocycle μ_{λ}^{D}(g, w) is given by

μ_{λ}^{D}(g, w) = Det(D(α(g))(w))^{\frac{rλ}{2n}} ,  g ∈ G̃, w ∈ D,

the powers being well-defined on the universal cover G̃. The representations H_{λ}^{2}(D) and H_{λ}^{2}(T_{Ω}) are isomorphic, the intertwining operator being given by

γ_{λ} : H_{λ}^{2}(T_{Ω}) → H_{λ}^{2}(D), γ_{λ}F (w) := Δ(e − w)^{−λ}F (c(w)),

where c(w) is the Cayley transform defined in (1.13) (see [15, Proposition XIII.1.3 & Theorem XIII.3.4]). The operator γ_{λ} is unitary (up to a scalar) and intertwines the group actions.

2.4 𝔨-type decompositions

In the bounded symmetric domain model the K̃-type decomposition is very explicit. Let λ ∈ W and 0 ≤ k ≤ r such that O_{λ} = O_{k}. Then H_{λ}^{2}(D) decomposes into the direct Hilbert space sum

H_{λ}^{2}(D) = ˆ⨁_{m≥0, m_{k+1}=0} P_{m}(V_{ℂ}),

each summand P_{m}(V_{ℂ}) is irreducible under the action π_{λ}^{D}(K̃) and on it the norm is given by (see [15, Theorem XIII.2.7])

∥p∥_{H_{λ}^{2}(D)}^{2} = \frac{(\frac{n}{r})m}{(λ)_{m}} ∥p∥_{Σ}^{2},  p ∈ P_{m}(V_{ℂ}).  (2.2)

Further, the 𝔨^{𝔩}-spherical vector in each K̃-type P_{m}(V_{ℂ}) is the spherical polynomial Φ_{m}(z) which has norm

∥Φ_{m}∥_{H_{λ}^{2}(D)}^{2} = \frac{(\frac{n}{r})_{m}}{d_{m}(λ)_{m}} .

Correspondingly the underlying (𝔤, K̃)-module W^{λ} of the Schrödinger model L^{2}(O_{λ}, dμ_{λ}) decomposes into K̃-types

W^{λ} = ⨁_{m≥0, m_{k+1}=0} W_{m}^{λ},

where W_{m}^{λ} = L_{λ}^{−1}∘γ_{λ}^{−1}(P_{m}(V_{ℂ})). The 𝔨^{𝔩}-spherical vector in W_{m}^{λ} is given by the Laguerre function ℓ_{m}^{λ}(x) defined in Section 1.8 (see [15, Proposition XV.4.2]) which has norm (see [15, Corollary XV.4.3 (i)])

∥ℓ_{m}^{λ}∥_{L^{2}(O_{λ}, dμ_{λ})}^{2} = \frac{(\frac{n}{r})_{m}(λ)_{m}}{d_{m}} .  (2.3)

2.5 Complexification of the Schrödinger model

The infinitesimal action dπ_{λ} in the Schrödinger model is given by second order differential operators on O_{λ} with polynomial coefficients. Hence the action can be extended to an action dπ_{λ}^{ℂ} of 𝔤 on C^{∞}(X_{λ}) by holomorphic differential operators. More precisely, let D be a differential operator on V with polynomial coefficients. Choose some basis e_{1}, …, e_{n} of V and write x = ∑_{j=1}^{n} x_{j}e_{j} ∈ V. Then D is in coordinates given by

D = ∑_{α∈ℕ_{0}^{n}} c_{α}(x)\frac{∂^{|α|}}{∂x_{1}^{α_{1}} ⋯ ∂x_{n}^{α_{n}}}

with polynomials c_{α}(x) of which only finitely many are non-zero. The coefficients extend uniquely to holomorphic polynomials c_{α}(z) on V_{ℂ}. We define the complexification D^{ℂ} of D to a holomorphic differential operator on V_{ℂ} by

D^{ℂ} := ∑_{α∈ℕ_{0}^{n}} c_{α}(z)\frac{∂^{|α|}}{∂z_{1}^{α_{1}} ⋯ ∂z_{n}^{α_{n}}} ,  where  \frac{∂}{∂z_{j}} = \frac{1}{2} (\frac{∂}{∂x_{j}} − i\frac{∂}{∂y_{j}} )

denotes the Wirtinger derivative for z_{j} = x_{j} + iy_{j} in coordinates z = ∑_{j=1}^{n} z_{j}e_{j} ∈ V_{ℂ}. The operator D^{ℂ} is a complexification of D in the sense that for a holomorphic function f on V_{ℂ} we have

(D^{ℂ}f)|_{V} = D(f|_{V}).

Applying this procedure to dπ_{λ}(X) for X ∈ 𝔤 we put

dπ_{λ}^{ℂ}(X) := dπ_{λ}(X)^{ℂ}.

It remains to show that these holomorphic differential operators are actually tangential to the orbit X_{λ} and hence define an action dπ_{λ}^{ℂ} of 𝔤 on C^{∞}(X_{λ}). For this we use the Schrödinger model of certain unipotent representations of the complex group G_{ℂ} = Co(V_{ℂ}) (the conformal group of the complex Jordan algebra V_{ℂ}), viewed as a real Lie group.

In [22, Proposition 2.14] the authors construct a representation dτ_{λ} of 𝔤_{ℂ}, viewed as a real Lie algebra, on C^{∞}(X_{λ}) for each λ ∈ W. It is explicitly given by

dτ_{λ}(u, 0, 0) = i(u|z)_{W} ,
dτ_{λ}(0, T, 0) = ∂_{T^{#}z} + \frac{rλ}{2n} Tr_{W} (T^{#}),
dτ_{λ}(0, 0, v) = i(v|B_{λ}^{W})_{W} .

By Tr_{W} we mean the real trace of an operator on the real vector space W. Note that dτ_{λ} does not act via holomorphic differential operators, but via real differential operators up to second order on X_{λ}. Further, its is shown in [22, Theorem 1.12] that the representation dτ_{λ} is infinitesimally unitary with respect to the inner product of L^{2}(X_{λ}, dν_{λ}). We have the following result relating dτ_{λ} to the complexification dπ_{λ}^{ℂ}:

Proposition 2.1. For X ∈ 𝔤 we have

dπ_{λ}^{ℂ}(X) = \frac{1}{2} ( dτ_{λ}(X) − i dτ_{λ}(iX)) .  (2.4)

In particular, for every X = (u, T, v) ∈ 𝔤 and all F, G ∈ C^{∞}(X_{λ}) we have

∫_{X_{λ}} dπ_{λ}^{ℂ}(u, T, v)F (z) · G(z) dν_{λ}(z) = ∫_{X_{λ}} F (z) · dπ_{λ}^{ℂ}(u, −T, v)G(z) dν_{λ}(z).

Remark 2.2. The formula (2.4) can be understood as an analog of the Wirtinger derivative

\frac{∂}{∂z} = \frac{1}{2} (\frac{∂}{∂x} − i\frac{∂}{∂y} ) .

Proof of Proposition 2.1. First note that since 𝔫 and \overline{𝔫} together generate 𝔤 as a Lie algebra it suffices to show (2.4) for X ∈ 𝔫 and X ∈ \overline{𝔫}. Further note that for z ∈ V_{ℂ} we have

(a|z) = \frac{1}{2} ((a|z)_{W} − i(ia|z)_{W}) ,  a ∈ V.  (2.5)

Then (2.4) is immediate for X ∈ 𝔫. Now let X = (0, 0, a) ∈ \overline{𝔫}. Let (e_{α}) be any orthonormal basis of V with respect to the trace form (−|−). Write x = ∑_{α} x_{α}e_{α}. We view W = V_{ℂ} as a real Jordan algebra. The vectors

f_{α} := \frac{1}{\sqrt{2}}e_{α}  and  g_{α} := \frac{1}{\sqrt{2}}ie_{α}

constitute an ℝ-basis of W. Its dual basis with respect to the trace form (−|−)_{W} is given by (f̂_{α} := f_{α})_{α} ∪ (ĝ_{α} := −g_{α})_{α}. We write z = ∑_{α} z_{α}e_{α} = ∑_{α} (a_{α}f_{α} + b_{α}g_{α}) with a_{α}, b_{α} ∈ ℝ and z_{α} = \frac{1}{\sqrt{2}}(a_{α} + ib_{α}). Hence,

\frac{∂}{∂a_{α}} = \frac{1}{\sqrt{2}} \frac{∂}{∂x_{α}}  and  \frac{∂}{∂b_{α}} = \frac{1}{\sqrt{2}} \frac{∂}{∂y_{α}}

with z_{α} = x_{α} + iy_{α}, x_{α}, y_{α} ∈ ℝ. With (2.5) we have

\frac{1}{2} ( dτ_{λ}(X) − i dτ_{λ}(iX)) = \frac{i}{2} ((a|B_{λ}^{W}) − i(ia|B_{λ}^{W}))

= i ∑_{α,β} (\frac{∂^{2}}{∂a_{α}∂a_{β}} (a|P (f̂_{α}, f̂_{β})z) + 2\frac{∂^{2}}{∂a_{α}∂b_{β}} (a|P (f̂_{α}, ĝ_{β})z)

+\frac{∂^{2}}{∂b_{α}∂b_{β}} (a|P (ĝ_{α}, ĝ_{β})z)) + iλ ∑_{α} (\frac{∂}{∂a_{α}} f̂_{α} + \frac{∂}{∂b_{α}} ĝ_{α})

= \frac{i}{4} ∑_{α,β} (\frac{∂^{2}}{∂x_{α}∂x_{β}} (a|P (e_{α}, e_{β})z) − 2i\frac{∂^{2}}{∂x_{α}∂y_{β}} (a|P (e_{α}, e_{β})z)

−\frac{∂^{2}}{∂y_{α}∂y_{β}} (a|P (e_{α}, e_{β})z)) + \frac{iλ}{2} ∑_{α} (\frac{∂}{∂x_{α}} e_{α} − i\frac{∂}{∂y_{α}} e_{α})

= i ∑_{α,β} \frac{∂^{2}}{∂z_{α}∂z_{β}} (a|P (e_{α}, e_{β})z) + iλ ∑_{α} \frac{∂}{∂z_{α}} e_{α} = dπ_{λ}^{ℂ}(X).

This shows (2.4) for X ∈ \overline{𝔫} and hence it follows for all X ∈ 𝔤. The stated integral formula now follows from (2.4) using the fact that dτ_{λ}(X) is given by skew-adjoint real differential operators operators on L^{2}(X_{λ}, dν_{λ}) with real coefficients if X = (0, T, 0) ∈ 𝔤 and purely imaginary coefficients if X = (u, 0, v) ∈ 𝔤.

Since dτ_{λ} restricts to an action on C^{∞}(X_{λ}) by differential operators, the same is true for dπ_{λ}^{ℂ} by the previous proposition. Therefore, dπ_{λ}^{ℂ} is a representation of 𝔤 on C^{∞}(X_{λ}) by holomorphic polynomial differential operators of order at most 2.

3 Bessel functions on Jordan algebras

In this section we study J-, I- and K-Bessel functions on symmetric cones and their boundary orbits. These functions play a fundamental role in the study of Schrödinger and Fock models and the intertwining operators between them.

3.1 J-Bessel function

For λ ∈ ℂ with (λ)_{m} ≠ 0 for all m ≥ 0 and z, w ∈ V_{ℂ} we put

J_{λ}(z, w) := ∑_{m≥0} (−1)^{|m|} \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} Φ_{m}(z, w).

(This notation agrees with the one used in [15, Chapter XV.2].) One problem is that for a discrete Wallach point λ = k \frac{d}{2}, k = 0, …, r − 1, we have (λ)_{m} = 0 for all m ≥ 0 with m_{k+1} ≠ 0. However, by Lemma 1.12 we find that for z ∈ \overline{X_{k}} or w ∈ \overline{X_{k}}:

J_{λ}(z, w) = ∑_{m≥0,m_{k+1}=0} (−1)^{|m|} \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} Φ_{m}(z, w).

In this expression the coefficients are non-singular at λ = k \frac{d}{2} since (λ)_{m} ≠ 0 for all m ≥ 0 with m_{k+1} = 0. Therefore we obtain for each λ ∈ W a J-Bessel function J_{λ}(z, w) on V_{ℂ} × \overline{X_{λ}}. It only remains to show convergence of the defining series.

Lemma 3.1. For λ ∈ W the series for J_{λ}(z, w) converges absolutely for all z ∈ V_{ℂ} and w ∈ \overline{X_{λ}} and the following estimate holds:

|J_{λ}(z, w)| ≤ C(1 + |z| · |w|)^{\frac{r(2n−1)}{4}} e^{2r\sqrt{|z|·|w|}}  ∀ z ∈ V_{ℂ}, w ∈ \overline{X_{λ}}

for some constant C > 0 which depends only on the structure constants of V and λ.

Proof. Let z, w ∈ V_{ℂ}. Then by [15, Proposition X.3.2] there exist u ∈ U and a = ∑_{j=1}^{r} a_{j}c_{j} with a_{1} ≥ … ≥ a_{r} ≥ 0 such that w = ua = uP (a^{\frac{1}{2}} )e. With Lemma 1.11 (2) we find

Φ_{m}(z, w) = Φ_{m}(z, uP (a^{\frac{1}{2}} )e) = Φ_{m}(P (a^{\frac{1}{2}} )u^{−1}z, e) = Φ_{m}(P (a^{\frac{1}{2}} )u^{−1}z).

Now suppose further that w ∈ \overline{X_{k}}, 0 ≤ k ≤ r, and m_{k+1} = 0. Then a_{k+1} = … = a_{r} = 0 and hence P (a^{\frac{1}{2}} ) projects onto V (e_{k}, 1) ⊆ \overline{X_{k}}. Thus we find that P (a^{\frac{1}{2}} )u^{−1}z ∈ \overline{X_{k}} and again by [15, Proposition X.3.2] we can write

P (a^{\frac{1}{2}} )u^{−1}z = u′b,

where u′ ∈ U and b = ∑_{j=1}^{k} b_{j}c_{j}, b_{1} ≥ … ≥ b_{k} ≥ 0. Now, by [15, Theorem XII.1.1 (i)] we obtain

|Φ_{m}(z, w)| = |Φ_{m}(u′b)| ≤ b_{1}^{m_{1}} ⋯ b_{k}^{m_{k}}.

We further have the following obvious inequalities (assuming m_{k+1} = 0 for λ = k \frac{d}{2})

d_{m} ≤ dim P_{|m|}(V_{ℂ}) = \binom{n + |m| − 1}{n − 1} ≤ C_{1}(1 + |m|)^{n−1}

≤ C_{1}(1 + m_{1})^{n−1} ⋯ (1 + m_{k})^{n−1},

(\frac{n}{r})_{m} = ∏_{j=1}^{r} (\frac{n}{r} − (j − 1)\frac{d}{2})_{m_{j}} ≥ ∏_{j=1}^{r} (1)_{m_{j}} = m!,

(λ)_{m} = ∏_{j=1}^{k} (λ − (j − 1)\frac{d}{2})_{m_{j}} ≥ ∏_{j=1}^{r} (λ − (k − 1)\frac{d}{2})_{m_{j}}.

We abbreviate λ′ := λ − (k − 1)\frac{d}{2} > 0. Putting things together gives

|J_{λ}(z, w)| ≤ ∑_{m≥0,m_{k+1}=0} \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} |Φ_{m}(z, w)|

≤ C_{1} ∑_{m∈ℕ_{0}^{k}} \frac{(1 + m_{1})^{n−1} ⋯ (1 + m_{k})^{n−1}}{m!(λ′)_{m_{1}} ⋯ (λ′)_{m_{k}}} b_{1}^{m_{1}} ⋯ b_{k}^{m_{k}}

= C_{1} ∏_{j=1}^{k} ( ∑_{m=0}^{∞} \frac{(1 + m)^{n−1}}{m!(λ′)_{m}} b_{j}^{m}) .

Now note that mb^{m} = (b \frac{d}{db})b^{m}, so

∑_{m=0}^{∞} \frac{(1 + m)^{n−1}}{m!(λ′)_{m}} b^{m} = (1 + b \frac{d}{db})^{n−1} ∑_{m=0}^{∞} \frac{1}{m!(λ′)_{m}} b^{m}

= (1 + b \frac{d}{db})^{n−1} _{0}F_{1}(λ′; b),

where _{0}F_{1}(β; z) denotes the hypergeometric function. For _{0}F_{1}(β; z) we have the obvious identity

\frac{d}{dz}_{0}F_{1}(β; z) = \frac{1}{β}_{0}F_{1}(β + 1; z)

and hence (1 + b \frac{d}{db})^{n−1} _{0}F_{1}(λ′; b) is a linear combination of functions of the type

b^{k}_{0}F_{1}(λ′ + k; b),  k = 0, …, n − 1.

Now by [3, equations (4.5.2) & (4.8.5)] the asymptotic behaviour of the hypergeometric function as z → ∞ can be estimated by

|_{0}F_{1}(β; z)| ≲ |z|^{\frac{1−2β}{4}} e^{2|z|^{\frac{1}{2}}}

and hence we obtain

|(1 + b \frac{d}{db})^{n−1} _{0}F_{1}(λ′; b)| ≤ C_{2}(1 + |b|)^{\frac{2n−2λ′−1}{4}} e^{2\sqrt{b}}  ∀ b ∈ [0, ∞),

the constant C_{2} > 0 only depending on n and λ′. Inserting this into the estimate for J_{λ}(z, w) gives

|J_{λ}(z, w)| ≤ C_{1}C_{2}^{k} ∏_{j=1}^{k} ((1 + |b_{j}|)^{\frac{2n−2λ′−1}{4}} e^{2\sqrt{b_{j}}}) .

Now note that

|b| = |u′b| = |P (a^{\frac{1}{2}} )u^{−1}z| ≤ ∥P (a^{\frac{1}{2}} )∥ · |u^{−1}z| = ∥P (a^{\frac{1}{2}} )∥ · |z|,

where ∥P (a^{\frac{1}{2}} )∥^{2} is the largest eigenvalue of P (a^{\frac{1}{2}} )^{#}P (a^{\frac{1}{2}} ) = P (a^{\frac{1}{2}} )^{2}. Since P (a^{\frac{1}{2}} )^{2} acts on V_{ij} by a_{i}a_{j}, its largest eigenvalue is a_{1}^{2}. Hence

|b| ≤ a_{1}|z| ≤ |a| · |z| = |w| · |z|.

Altogether we finally obtain

|J_{λ}(z, w)| ≤ C_{1}C_{2}^{k} (∏_{j=1}^{k} (1 + |b_{j}|))^{\frac{2n−2λ′−1}{4}} e^{2(\sqrt{b_{1}}+···+\sqrt{b_{r}})}

≤ C_{1}C_{2}^{k}(1 + b_{1} + ⋯ + b_{r})^{\frac{r(2n−2λ′−1)}{4}} e^{2\sqrt{r}\sqrt{b_{1}+···+b_{r}}}
≤ C_{1}C_{2}^{k}r^{\frac{r(2n−2λ′−1)}{8}} (1 + |b|)^{\frac{r(2n−2λ′−1)}{4}} e^{2r\sqrt{|b|}}
≤ C(1 + |z| · |w|)^{\frac{r(2n−1)}{4}} e^{2r\sqrt{|z|·|w|}}

with C = C_{1}C_{2}^{k}r^{\frac{r(2n−1)}{8}} > 0 which gives the claim.

The estimate obtained in Lemma 3.1 is not sharp, but suffices for most of our purposes. Recently, Nakahama [44] obtained a sharper estimate which we use in Section 6.2 to find explicit Whittaker vectors:

Proposition 3.2 ([44, Corollary 1.2]). For λ ∈ W and k ∈ ℕ_{0} with Re λ + k > \frac{2n}{r} − 1 there exists a constant C_{λ,k} > 0 such that

|J_{λ}(z^{2}, e)| ≤ C_{λ,k}(1 + |z|_{1}^{k})e^{2| Im z|_{1}},  z ∈ \overline{X_{λ}},

where |z|_{1} = ∑_{j=1}^{r} |a_{j}|, z = u ∑_{j=1}^{r} a_{j}c_{j}, u ∈ U, a_{j} ∈ ℝ.

Example 3.3. On the rank 1 orbit we have, using Lemma 1.12 and Example 1.13

J_{λ}(z, w) = ∑_{m=0}^{∞} \frac{(−1)^{m}}{m!(λ)_{m}} (z|\overline{w})^{m} = Γ(λ)J̃_{λ−1}(2\sqrt{(z|\overline{w})}), z, w ∈ X_{1},

where J̃_{α}(z) = (\frac{z}{2})^{−α}J_{α}(z) is the classical renormalized J-Bessel function which is an even entire function on ℂ.

The following proposition is clear with the results of Section 1.8:

Proposition 3.4. The J-Bessel function J_{λ}(z, w) has the following properties:

(1) J_{λ}(z, w) = \overline{J_{λ}(w, z)} for z ∈ V_{ℂ}, w ∈ \overline{X_{λ}}, (2) J_{λ}(gz, w) = J_{λ}(z, g^{∗}w) for z ∈ V_{ℂ}, w ∈ \overline{X_{λ}}, g ∈ L_{ℂ}.

To prove the differential equation for J_{λ}(z, w) we use the same method as Faraut–Koranyi [15, Theorem XV.2.6] for the one-variable J-Bessel function J (x, e). The first step is to calculate the Laplace transform as defined in (2.1).

Lemma 3.5. Fix w ∈ \overline{X_{λ}} and consider the function J_{λ}(−, w) on V given by x ↦ J_{λ}(x, w). The Laplace transform of J_{λ}(−, w) is given by

(L_{λ}J_{λ}(−, w)) (z) = 2^{rλ}Δ(−iz)^{−λ}e^{−i(z^{−1}|\overline{w})},  z ∈ T_{Ω}.

Proof. Let 0 ≤ k ≤ r be such that O_{λ} = O_{k}. The we find, using Lemma 1.10 and (1.17):

(L_{λ}J_{λ}(−, w)) (z) = ∫_{O_{λ}} e^{i(z|x)}J_{λ}(x, w) dμ_{λ}(x)
= ∑_{m≥0,m_{k+1}=0} (−1)^{|m|} \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} ∫_{O_{λ}} e^{i(z|x)}Φ_{m}(x, w) dμ_{λ}(x)

= 2^{rλ} ∑_{m≥0} (−1)^{|m|} \frac{d_{m}}{(\frac{n}{r})_{m}} Δ(−iz)^{−λ}Φ_{m}(iz^{−1}, w) = 2^{rλ}Δ(−iz)^{−λ}e^{−i(z^{−1}|\overline{w})}.

Proposition 3.6. For λ ∈ W the function J_{λ}(z, w) solves the following differential equation:

(B_{λ})_{z}J_{λ}(z, w) = −\overline{w}J_{λ}(z, w),  z ∈ V_{ℂ}, w ∈ \overline{X_{λ}}.

Proof. First note that it suffices to show the differential equation for λ > (r − 1)\frac{d}{2}, then the general case follows by analytic continuation. Further, it suffices to show the differential equation for z ∈ Ω as J_{λ}(z, w) is holomorphic in z ∈ V_{ℂ}. Since the Laplace transform L_{λ} is injective on functions on Ω the differential equation is equivalent to the identity

L_{λ}(B_{λ}J_{λ}(−, w)) = −\overline{w}L_{λ}J_{λ}(−, w).  (3.1)

Let z ∈ T_{Ω}, then using the symmetry of the Bessel operator B_{λ} we find

(L_{λ}(B_{λ}J_{λ}(−, w))) (z) = ∫_{O_{λ}} e^{i(z|x)}(B_{λ})_{x}J_{λ}(x, w) dμ_{λ}(x)

= ∫_{O_{λ}} (B_{λ})_{x}e^{i(z|x)}J_{λ}(x, w) dμ_{λ}(x) = ∫_{O_{λ}} (P (iz)x + iλz)e^{i(z|x)}J_{λ}(x, w) dμ_{λ}(x)
= i (P (z)\frac{∂}{∂z} + λz) ((L_{λ}J_{λ}(−, w)) (z).

Now by Lemma 3.5 we have (L_{λ}J_{λ}(−, w)) (z) = 2^{rλ}Δ(−iz)^{−λ}e^{−i(z^{−1}|\overline{w})}. Using D(x^{−1}) = −P (x)^{−1} and ∂_{y}Δ(x) = Δ(x)(y|x^{−1}) we find

i (P (z)\frac{∂}{∂z} + λz) [Δ(−iz)^{−λ}e^{−i(z^{−1}|\overline{w})}]

= i (−λP (z)z^{−1} + iP (z)P (z)^{−1}\overline{w} + λz) Δ(−iz)^{−λ}e^{−i(z^{−1}|\overline{w})}
= − \overline{w}Δ(−iz)^{−λ}e^{−i(z^{−1}|\overline{w})}

and (3.1) follows.

3.2 I-Bessel function

For λ ∈ W, z, w ∈ V_{ℂ} and z ∈ \overline{X_{λ}} or w ∈ \overline{X_{λ}} we put

I_{λ}(z, w) := J_{λ}(−z, w) = J_{λ}(z, −w)

= ∑_{m≥0} \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} Φ_{m}(z, w).

By definition the I-Bessel function I_{λ}(z, w) also satisfies the estimates in Proposition 3.1 and has the same properties as in Proposition 3.4

Example 3.7. By Example 3.3 it is immediate that on the rank 1 orbit we have

I_{λ}(z, w) = Γ(λ)Ĩ_{λ−1}(2\sqrt{(z|\overline{w})}),  z, w ∈ X_{1},

where Ĩ_{α}(z) = (\frac{z}{2})^{−α}I_{α}(z) is the classical renormalized I-Bessel function which is an even entire function on ℂ.

Lemma 3.8. For λ ∈ W and y ∈ Ω, z ∈ V_{ℂ}

∫_{O_{λ}} e^{−(x|y)}I_{λ}(x, z) dμ_{λ}(x) = 2^{rλ}Δ(y)^{−λ}e^{(y^{−1}|\overline{z})}.

Proof. This follows immediately from Lemma 3.5.

Proposition 3.9. For λ ∈ W the function I_{λ}(z, w) solves the following differential equation:

(B_{λ})_{z}I_{λ}(z, w) = \overline{w}I_{λ}(z, w),  z, w ∈ X_{λ}.

Proof. Since I_{λ}(z, w) = J_{λ}(z, −w) this is equivalent to Proposition 3.6.

3.3 K-Bessel function

For λ ∈ ℂ and x ∈ Ω we put

K_{λ}(x) := ∫_{Ω} e^{− tr(u^{−1})−(x|u)}Δ(u)^{λ− \frac{2n}{r}} du = ∫_{Ω} e^{− tr(v)−(x|v^{−1})}Δ(v)^{−λ} dv.

Note that our normalization of the parameter λ differs from the one used in [8] and [15, Chapter XVI.3]. By [15, Proposition XVI.3.1] these integrals converge for all λ ∈ ℂ and x ∈ Ω. Since the integrand is positive on Ω we have K_{λ}(x) > 0 for x ∈ Ω. To extend the K-Bessel function also to lower rank orbits we need the following result due to J.-L. Clerc [8, Théorème 4.1]:

Proposition 3.10. Let c ∈ V be an idempotent of rank k. Let Ω_{1} and Ω_{0} be the symmetric cones in the Euclidean Jordan algebras V (c, 1) and V (c, 0), respectively. Further, let K_{λ}^{1} be the K-Bessel function of Ω_{1}, Γ_{Ω_{0}} the Gamma function of Ω_{0} and n_{0} and r_{0} the dimension and rank of V (c, 0). Then for x_{1} ∈ Ω_{1}

K_{λ}(x_{1}) = (2π)^{k(r−k) \frac{d}{2}} Γ_{Ω_{0}} (\frac{n_{0}}{r_{0}} + k \frac{d}{2} − λ) K_{λ}^{1}(x_{1}).  (3.2)

This shows that for λ near k \frac{d}{2} the Bessel function K_{λ}(x) is defined for x ∈ O_{k} and hence we obtain Bessel functions K_{λ} on O_{λ} for λ ∈ W. Note that by (3.2) the function K_{λ} is positive on O_{λ}.

Example 3.11. (1) For V = ℝ we have by [18, formula 3.471 (9)]

K_{λ}(x) = 2K̃_{λ−1}(2\sqrt{x}),  x ∈ Ω = ℝ_{+},

where K̃_{α}(z) = (\frac{z}{2})^{−α} K_{α}(z) is the classical renormalized K-Bessel function.

(2) In the general case the Bessel function K_{λ} is by Proposition 3.10 on the rank 1 orbit O_{1} given by

K_{λ}(x) = const ·K_{λ}^{1}(|x|c_{1}) = const ·K̃_{λ−1}(2\sqrt{|x|}),  x ∈ O_{1}.

Lemma 3.12. For λ ∈ W and m ≥ 0 we have

∫_{O_{λ}} p(x)K_{λ}(x) dμ_{λ}(x) = 2^{rλ}Γ_{Ω}(\frac{n}{r})(\frac{n}{r})_{m}(λ)_{m}p(e),  p ∈ P_{m}(V_{ℂ}).

In particular, for every N > 0 we have

∫_{O_{λ}} (1 + |x|)^{N}K_{λ}(x) dμ_{λ}(x) < ∞.

Proof. It suffices to show the claim for λ > (r − 1)\frac{d}{2}. The general case then follows by analytic continuation. For λ > (r − 1)\frac{d}{2} we have, using Lemma 1.10

∫_{O_{λ}} p(x)K_{λ}(x) dμ_{λ}(x) = \frac{2^{rλ}}{Γ_{Ω}(λ)} ∫_{Ω} p(x)K_{λ}(x)Δ(x)^{λ− \frac{n}{r}} dx

= \frac{2^{rλ}}{Γ_{Ω}(λ)} ∫_{Ω} ∫_{Ω} p(x)e^{− tr(u^{−1})−(x|u)}Δ(u)^{λ− \frac{2n}{r}} Δ(x)^{λ− \frac{n}{r}} du dx
= 2^{rλ}(λ)_{m} ∫_{Ω} e^{− tr(u^{−1})}p(u^{−1})Δ(u)^{− \frac{2n}{r}} du

and under the coordinate change v = u^{−1}, dv = Δ(u)^{− \frac{2n}{r}} du, this is

= 2^{rλ}(λ)_{m} ∫_{Ω} e^{− tr(v)}p(v) dv = 2^{rλ}(λ)_{m}Γ_{Ω}(\frac{n}{r})(\frac{n}{r})_{m}p(e),

where we have again used Lemma 1.10 for the last equality. This shows the desired integral formula. For the second claim we observe that by the previous calculations every polynomial can be integrated against the positive measure K_{λ}(x) dμ_{λ}(x) since P(V_{ℂ}) = ⨁_{m≥0} P_{m}(V_{ℂ}) and hence the claim follows.

Proposition 3.13. The function K_{λ}(x) solves the following differential equation:

B_{λ}K_{λ}(x) = eK_{λ}(x).

Proof. Differentiating under the integral we obtain

B_{λ}K_{λ}(x) = ∫_{Ω} (P (−u)x + λ(−u))e^{− tr(u^{−1})−(x|u)}Δ(u)^{λ− \frac{2n}{r}} du
= ∫_{Ω} − (P (u)\frac{∂}{∂u} + λu) e^{−(x|u)} · e^{− tr(u^{−1})}Δ(u)^{λ− \frac{2n}{r}} du
= ∑_{α} ∫_{Ω} − (P (u)e_{α} \frac{∂}{∂u_{α}} + λu) e^{−(x|u)} · e^{− tr(u^{−1})}Δ(u)^{λ− \frac{2n}{r}} du

= ∑_{α} ∫_{Ω} e^{−(x|u)} · (\frac{∂}{∂u_{α}} P (u)e_{α} − λu) [e^{− tr(u^{−1})}Δ(u)^{λ− \frac{2n}{r}} ] du.

Using ∂_{y}P (x) = 2P (x, y), D(x^{−1}) = −P (x)^{−1} and ∂_{y}Δ(x) = Δ(x)(y|x^{−1}) we obtain

= ∑_{α} ∫_{Ω} e^{−(x|u)}(2P (e_{α}, u)e_{α} + P (u)e_{α} tr(P (u)^{−1}e_{α})
+ (λ − \frac{2n}{r})(e_{α}|u^{−1})P (u)e_{α} − λu)e^{− tr(u^{−1})}Δ(u)^{λ− \frac{2n}{r}} du
= ∫_{Ω} e^{−(x|u)}(2( ∑_{α} e_{α}^{2})u + P (u)P (u)^{−1}e
+ (λ − \frac{2n}{r})P (u)u^{−1} − λu)e^{− tr(u^{−1})}Δ(u)^{λ− \frac{2n}{r}} du.

By (1.1) this is

= ∫_{Ω} e^{−(x|u)}(\frac{2n}{r}u + e + (λ − \frac{2n}{r})u − λu)e^{− tr(u^{−1})}Δ(u)^{λ− \frac{2n}{r}} du
= eK_{λ}(x).

Now let λ ∈ W. For z ∈ X_{λ}, z = ua with u ∈ U, a = ∑_{i=1}^{r} t_{i}c_{i}, t_{i} ≥ 0, we put

ω_{λ}(z) := K_{λ} ((\frac{a}{2})^{2}) .

We note that ω_{λ} is positive on X_{λ}.

Proposition 3.14. The function ω_{λ}(z) solves the following differential equation:

B_{λ}ω_{λ}(z) = \frac{\overline{z}}{4}ω_{λ}(z),  z ∈ X_{λ}.

Proof. Recall the operators B_{λ}^{V} and B_{λ}^{W} acting on functions of r variables (see Proposition 1.5). Let F (a_{1}, …, a_{r}) = K_{λ}(a), a = ∑_{i=1}^{r} a_{i}c_{i}, then F solves the system (B_{λ}^{V})^{i}F = F, i = 1, …, r by Propositions 1.5 and 3.13. Put G(a_{1}, …, a_{r}) := ω_{λ}(a) = F ((\frac{a_{1}}{2} )^{2}, …, (\frac{a_{r}}{2} )^{2}). Then

(B_{λ}^{W})^{i}G(a_{1}, …, a_{r})

= \frac{1}{4}(a_{i} \frac{∂^{2}}{∂a_{i}^{2}} + (2λ − 1 − (r − 1)d) \frac{∂}{∂a_{i}}

+ \frac{d}{2} ∑_{j≠i} (\frac{1}{a_{i} − a_{j}} + \frac{1}{a_{i} + a_{j}} ) (a_{i} \frac{∂}{∂a_{i}} − a_{j} \frac{∂}{∂a_{j}} ))G(a_{1}, …, a_{r})

= \frac{1}{4}(a_{i} (\frac{a_{i}^{2}}{4} \frac{∂^{2}}{∂a_{i}^{2}} + \frac{1}{2} \frac{∂}{∂a_{i}} ) + \frac{a_{i}}{2} (2λ − 1 − (r − 1)d) \frac{∂}{∂a_{i}}

+ \frac{d}{2} ∑_{j≠i} (\frac{1}{a_{i} − a_{j}} + \frac{1}{a_{i} + a_{j}} ) (\frac{a_{i}^{2}}{2} \frac{∂}{∂a_{i}} − \frac{a_{j}^{2}}{2} \frac{∂}{∂a_{j}} ))F((\frac{a_{1}}{2} )^{2} , …, (\frac{a_{r}}{2} )^{2})

= \frac{a_{i}}{4} ( (\frac{a_{i}}{2} )^{2} \frac{∂^{2}}{∂a_{i}^{2}} + (λ − (r − 1)\frac{d}{2}) \frac{∂}{∂a_{i}}

+ \frac{d}{2} ∑_{j≠i} \frac{1}{(\frac{a_{i}}{2} )^{2} − (\frac{a_{j}}{2} )^{2}} ((\frac{a_{i}}{2} )^{2} \frac{∂}{∂a_{i}} − (\frac{a_{j}}{2} )^{2} \frac{∂}{∂a_{j}} ))F((\frac{a_{1}}{2} )^{2} , …, (\frac{a_{r}}{2} )^{2})

= \frac{a_{i}}{4} (B_{λ}^{V})^{i}F ((\frac{a_{1}}{2} )^{2} , …, (\frac{a_{r}}{2} )^{2}) = \frac{a_{i}}{4} F ((\frac{a_{1}}{2} )^{2} , …, (\frac{a_{r}}{2} )^{2})
= \frac{a_{i}}{4} G(a_{1}, …, a_{r}).

Hence we obtain B_{λ}^{W} ω_{λ}(z) = \frac{\overline{z}}{4}ω_{λ}(z) for z = a. Since ω_{λ}(z) is further Uinvariant, we obtain with (1.11) for z = ua with u ∈ U and a = ∑_{i=1}^{r} a_{i}c_{i}:

B_{λ}^{W} ω_{λ}(z) = (u^{−1})^{#}B_{λ}^{W} ω_{λ}(a) = \overline{u}\frac{a}{4}ω_{λ}(a) = \frac{\overline{z}}{4}ω_{λ}(z)

since \overline{u}^{#} = u^{−1} for u ∈ U. Finally we use Proposition 2.1 to find for every a ∈ V:

i(a|B_{λ})ω_{λ}(z) = dπ_{λ}^{ℂ}(0, 0, a)ω_{λ}(z)

= \frac{1}{2} ( dτ_{λ}(0, 0, a)ω_{λ}(z) − i dτ_{λ}(0, 0, ia)ω_{λ}(z))

= \frac{1}{2}i ((a|B_{λ}^{W})_{W} ω_{λ}(z) − i(ia|B_{λ}^{W})_{W} ω_{λ}(z))

= \frac{1}{2}i ((a|\frac{\overline{z}}{4})_{W} − i(ia|\frac{\overline{z}}{4})_{W} ) ω_{λ}(z)

= i(a|\frac{\overline{z}}{4})ω_{λ}(z).

Since this holds for any a ∈ V we find B_{λ}ω_{λ}(z) = \frac{\overline{z}}{4}ω_{λ}(z) and the proof is complete.

Example 3.15. On the rank 1 orbit the function ω_{λ} takes by Example 3.11 (2) the form

ω_{λ}(z) = ω_{λ}(|z|c_{1}) = K_{λ} (\frac{|z|^{2}}{4} c_{1}) = const ·K̃_{λ−1}(|z|).

4 A Fock model for unitary highest weight representations of scalar type

In this section we construct a Fock space F_{λ} = F(X_{λ}, ω_{λ} dν_{λ}) of holomorphic functions on the orbit X_{λ} for every λ ∈ W, calculate its reproducing kernel and find a realization on F_{λ} of the unitary highest weight representation corresponding to the Wallach point λ.

4.1 Construction of the Fock space

Let λ ∈ W. Recall the positive function ω_{λ} ∈ C^{∞}(X_{λ}) from Section 3.3. We endow the space P(X_{λ}) of polynomials on X_{λ} with the L^{2}-inner product of L^{2}(X_{λ}, ω_{λ} dν_{λ}):

⟨F, G⟩_{F_{λ}} := \frac{1}{c_{λ}} ∫_{X_{λ}} F (z)\overline{G(z)}ω_{λ}(z) dν_{λ}(z),  F, G ∈ P(X_{λ})  (4.1)

with c_{λ} = 2^{3rλ}Γ_{Ω}(\frac{n}{r}). This turns P(X_{λ}) into a pre-Hilbert space. Its completion F_{λ} := F(X_{λ}, ω_{λ} dν_{λ}) will be called the Fock space on X_{λ}. It remains to show that the integral in (4.1) converges. Using the polarization principle the following lemma suffices:

Lemma 4.1. For F ∈ P(V_{ℂ}) we have

∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) < ∞.

Proof. Using the integral formula (1.10) we obtain

∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) = ∫_{U} ∫_{O_{λ}} |F (ux^{\frac{1}{2}} )|^{2}K_{λ}(x) dμ_{λ}(x) du.

Now put

p(x) := ∫_{U} |F (ux)|^{2} du.

Clearly p is a polynomial on V, so there are constants C_{1} > 0 and N ∈ ℕ such that |p(x)| ≤ C_{1}(1 + |x|)^{N}. Now, every x ∈ \overline{Ω} has a decomposition x = ka with k ∈ K^{L} and a = ∑_{j=1}^{r} a_{j}c_{j}, a_{j} ≥ 0. In this decomposition the square root is of the form x^{\frac{1}{2}} = ka^{\frac{1}{2}} with a^{\frac{1}{2}} = ∑_{j=1}^{r} aj^{\frac{1}{2}} c_{j}. Since the norm | − | on V is K^{L}-invariant we obtain

|p(x^{\frac{1}{2}} )| ≤ C_{1} (1 + (a_{1} + ⋯ + a_{r})^{\frac{1}{2}} )^{N} ≤ C_{1} (1 + \sqrt{r}(a_{1}^{2} + ⋯ + a_{r}^{2})^{\frac{1}{4}} )^{N}
≤ C_{1}\sqrt{r} (1 + |x|^{\frac{1}{2}} )^{N} ≤ C_{1}\sqrt{r} (\sqrt{2}(1 + |x|)^{\frac{1}{2}} )^{N} = C_{2} (1 + |x|)^{\frac{N}{2}}

with C_{2} = C_{1}\sqrt{r}2^{\frac{N}{2}} . Hence, we find

∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) = ∫_{O_{λ}} p(x)K_{λ}(x) dμ_{λ}(x)
≤ C_{2} ∫_{O_{λ}} (1 + |x|)^{\frac{N}{2}} K_{λ}(x) dμ_{λ}(x).

The latter integral is finite by Lemma 3.12 and the proof is complete.

We explicitly calculate the norms on the finite-dimensional subspaces P_{m}(X_{λ}).

Proposition 4.2. Let m, n ≥ 0 and F ∈ P_{m}(X_{λ}), G ∈ P_{n}(X_{λ}). Then

⟨F, G⟩_{F_{λ}} = 4^{|m|}(\frac{n}{r})_{m}(λ)_{m}⟨F, G⟩_{Σ}.

In particular the subspaces P_{m}(X_{λ}) ⊆ P(X_{λ}) are pairwise orthogonal with respect to the inner product ⟨−, −⟩_{F_{λ}} and for F = Φ_{m} we have

∥Φ_{m}∥_{F_{λ}}^{2} = \frac{4^{|m|}(\frac{n}{r})_{m}(λ)_{m}}{d_{m}} .

Proof. Using the integral formula (1.10), Lemma 1.14 and Lemma 3.12 we obtain

⟨F, G⟩_{F_{λ}} = \frac{1}{c_{λ}} ∫_{X_{λ}} F (z)\overline{G(z)}ω_{λ}(z) dν_{λ}(z)

= \frac{1}{c_{λ}} ∫_{O_{λ}} ∫_{U} F (ux^{\frac{1}{2}} )\overline{G(ux^{\frac{1}{2}} )}ω_{λ}(x^{\frac{1}{2}} ) dμ_{λ}(x) du

= \frac{1}{c_{λ}} ⟨F, G⟩_{Σ} ∫_{O_{λ}} Φ_{m}(x)K_{λ}(\frac{x}{4}) dμ_{λ}(x)

= \frac{4^{rλ+|m|}}{c_{λ}} ⟨F, G⟩_{Σ} ∫_{O_{λ}} Φ_{m}(y)K_{λ}(y) dμ_{λ}(y)

= 4^{|m|}(\frac{n}{r})_{m}(λ)_{m}⟨F, G⟩_{Σ}.

Since ∥Φ_{m}∥_{Σ}^{2} = \frac{1}{d_{m}} by (1.15) this finishes the proof.

Remark 4.3. Comparing the norm on F_{λ} with the norm on the space H_{λ}^{2}(D) gives by (2.2)

∥F ∥_{F_{λ}}^{2} = 4^{|m|}(λ)_{m}^{2}∥F ∥_{H_{λ}^{2}(D)}^{2}  ∀ F ∈ P_{m}(V_{ℂ}).

If we denote by O(X_{λ}) the space of holomorphic functions on the complex manifold X_{λ}, we obtain the following result:

Proposition 4.4. O(X_{λ}) ∩ L^{2}(X_{λ}, ω_{λ} dν_{λ}) is a closed subspace of L^{2}(X_{λ}, ω_{λ} dν_{λ}) and the point evaluation O(X_{λ}) ∩ L^{2}(X_{λ}, ω_{λ} dν_{λ}) → ℂ, F ↦ F (z), is continuous for every z ∈ X_{λ}. In particular, F_{λ} ⊆ O(X_{λ}) ∩ L^{2}(X_{λ}, ω_{λ} dν_{λ}) and the point evaluation F_{λ} → ℂ, F ↦ F (z) is continuous for every z ∈ X_{λ}.

Proof. This is a local statement and hence, we may transfer it with a chart map to an open domain U ⊆ ℂ^{k}. Here the measure ω_{λ} dν_{λ} is absolutely continuous with respect to the Lebesgue measure dz and hence it suffices to show that O(ℂ^{k}) ∩ L^{2}(ℂ^{k}, dz) ⊆ L^{2}(ℂ^{k}, dz) is a closed subspace with continuous point evaluations. This is done e.g. in [20, Proposition 3.1 and Corollary 3.2].

The particular choice of the density ω_{λ} yields the following result:

Proposition 4.5. The adjoint of B_{λ} on F_{λ} is \frac{z}{4}.

Proof. Let F and G be holomorphic functions on X_{λ}. Then by Proposition 2.1 we know that

∫_{X_{λ}} B_{λ}F (z)\overline{G(z)}ω_{λ}(z) dν_{λ}(z) = ∫_{X_{λ}} F (z)B_{λ} [\overline{G(z)}ω_{λ}(z)] dν_{λ}(z).

The function \overline{G(z)} is antiholomorphic and hence

= ∫_{X_{λ}} F (z)\overline{G(z)}B_{λ}ω_{λ}(z) dν_{λ}(z).

By Proposition 3.14 we have B_{λ}ω_{λ}(z) = \frac{\overline{z}}{4}ω_{λ}(z) and therefore

= ∫_{X_{λ}} F (z)\overline{\frac{z}{4}G(z)}ω_{λ}(z) dν_{λ}(z).

4.2 The Bessel–Fischer inner product

We introduce another inner product on the space P(X_{λ}) of polynomials, the Bessel–Fischer inner product. For two polynomials p and q it is defined by

[p, q]_{λ} := p(B_{λ})\overline{q}(4z)|_{z=0} ,

where \overline{q}(z) = \overline{q(\overline{z})} is obtained by conjugating the coefficients of the polynomial q. A priori it is not even clear that this sesquilinear form is positive definite.

Theorem 4.6. For p, q ∈ P(X_{λ}) we have

[p, q]_{λ} = ⟨p, q⟩_{F_{λ}}.  (4.2)

The proof is similar to the proof of [5, Proposition 3.8]

Proof. First note that for all p, q ∈ P(X_{λ})

[(a|\frac{z}{4})p, q]_{λ} = [p, (\overline{a}|B_{λ})q]_{λ}  for a ∈ V_{ℂ},
⟨(a|\frac{z}{4})p, q⟩_{F_{λ}} = ⟨p, (\overline{a}|B_{λ})q⟩_{F_{λ}}  for a ∈ V_{ℂ}.

In fact, the second equation follows from Proposition 4.5. The first equation is immediate since the components (a|B_{λ}), a ∈ V_{ℂ}, of the Bessel operator form a commuting family of differential operators on X_{λ}. Therefore, (a|B_{λ})p(B_{λ})\overline{q}(4z) = 4p(B_{λ})\overline{(\overline{a}|B_{λ})q}(4z) and the claim follows. To prove (4.2) we proceed by induction on deg(q). First, if p = q = 1, the constant polynomial with value 1, it is clear that [p, q]_{λ} = 1 and by Proposition 4.2 we also have ⟨p, q⟩_{F_{λ}} = 1. Thus, (4.2) holds for deg(p) = deg(q) = 0. If now deg(p) is arbitrary and deg(q) = 0 then (\overline{a}|B_{λ})q = 0 and hence

[(a|\frac{z}{4})p, q]_{λ} = [p, (\overline{a}|B_{λ})q]_{λ} = 0  and
⟨(a|\frac{z}{4})p, q⟩_{F_{λ}} = ⟨p, (\overline{a}|B_{λ})q⟩_{F_{λ}} = 0.

Therefore, (4.2) holds if deg(q) = 0. We note that (4.2) also holds if deg(p) = 0 and deg(q) is arbitrary. In fact,

[p, q]_{λ} = p(0)\overline{q(0)} = \overline{[q, p]_{λ}}  and  ⟨p, q⟩_{F_{λ}} = \overline{⟨q, p⟩_{F_{λ}}}

and (4.2) follows from the previous considerations. Now assume (4.2) holds for deg(q) ≤ k. For deg(q) ≤ k + 1 we then have deg((\overline{a}|B_{λ})q) ≤ k and hence, by the assumption

[(a|\frac{z}{4})p, q]_{λ} = [p, (\overline{a}|B_{λ})q]_{λ} = ⟨p, (\overline{a}|B_{λ})q⟩_{F_{λ}} = ⟨(a, \frac{z}{4})p, q⟩_{F_{λ}}.

This shows (4.2) for deg(q) ≤ k + 1 and p(0) = 0, i.e. without constant term. But for constant p, i.e. deg(p) = 0, we have already seen that (4.2) holds and therefore the proof is complete.

4.3 Unitary action on the Fock space

In Section 2.5 we verified that for every λ ∈ W the complexification dπ_{λ}^{ℂ} of the action dπ_{λ} defines a Lie algebra representation of 𝔤 on C^{∞}(X_{λ}) by holomorphic polynomial differential operators in z. It is clear that this action preserves the subspace P(X_{λ}) of holomorphic polynomials. Using this we now construct an action of 𝔤 on P(X_{λ}) by composing the action dπ_{λ}^{ℂ} with the Cayley type transform C ∈ Int(𝔤_{ℂ}) introduced in Section 1.3.

Definition 4.7. Let λ ∈ W. On P(X_{λ}) we define a 𝔤-action dρ_{λ} by

dρ_{λ}(X) := dπ_{λ}^{ℂ}(C(X)),  X ∈ 𝔤.

Proposition 4.8. Let λ ∈ W and k ∈ {0, …, r} such that X_{λ} = X_{k}. The 𝔨-type decomposition of ( dρ_{λ}, P(X_{λ})) is given by

P(X_{λ}) = ⨁_{m≥0,m_{k+1}=0}^{∞} P_{m}(X_{λ})

and in every 𝔨-type P_{m}(X_{λ}) the space of 𝔨^{𝔩}-fixed vectors is one-dimensional and spanned by the polynomial Φ_{m}(z). In particular ( dρ_{λ}, P(X_{λ})) is an admissible (𝔤, 𝔨)-module.

Proof. The Cayley type transform C : 𝔤_{ℂ} → 𝔤_{ℂ} maps 𝔨_{ℂ} to 𝔩_{ℂ} and 𝔨^{𝔩} to itself. The action of 𝔩_{ℂ} in the complexified Schrödinger model dπ_{λ}^{ℂ} is induced by the action of L_{ℂ} on the orbit X_{λ} = L_{ℂ} · e_{k} up to multiplication by a character. Hence the 𝔨-type decomposition of P(X_{λ}) is the same as the decomposition into L_{ℂ}-representations under the natural action ℓ of L_{ℂ}. Therefore the claimed decomposition is clear by Corollary 1.9. The action of 𝔨^{𝔩} is induced by the natural action of K^{L} and hence the unique (up to scalar) 𝔨^{𝔩}-invariant vector in the 𝔨-type P_{m}(X_{λ}) is the unique (up to scalar) K^{L}-invariant vector in P_{m}(X_{λ}) under the natural action ℓ. This finishes the proof.

Proposition 4.9. For each λ ∈ W the representation ( dρ_{λ}, P(X_{λ})) is an irreducible (𝔤, 𝔨)-module.

Proof. Let m, n ≥ 0 be arbitrary and denote by U(𝔤) the universal enveloping algebra of 𝔤. Then it suffices to show that there exists an element of dρ_{λ}(U(𝔤)) which maps Φ_{m} to Φ_{n}. Using Theorem 4.6 we have

0 ≠ ∥Φ_{m}∥_{F_{λ}}^{2} = Φ_{m}(B_{λ})Φ_{m}(4z)|_{z=0} = 4^{|m|} Φ_{m}(B_{λ})Φ_{m}(z)|_{z=0} .

Since B_{λ} is a differential operator of Euler degree −1 the polynomial Φ_{m}(B_{λ})Φ_{m}(z) is constant and by the previous observation it is non-zero. Note that dρ_{λ}(U(𝔤)) = dπ_{λ}^{ℂ}(U(𝔤)) contains multiplication by arbitrary polynomials and differential operators which are polynomials in the Bessel operator B_{λ}. Hence, the operator Φ_{n}(z)Φ_{m}(B_{λ}) ∈ dρ_{λ}(U(𝔤)) maps Φ_{m}(z) to a non-zero multiple of Φ_{n}(z) and the claim follows.

Proposition 4.10. The (𝔤, 𝔨)-module ( dρ_{λ}, P(X_{λ})) is infinitesimally unitary with respect to the L^{2}-inner product ⟨−, −⟩_{F_{λ}}.

Proof. As remarked in Section 1.3 the subspace {(a, L(b), a) : a, b ∈ V } ⊆ 𝔤 generates 𝔤 as a Lie algebra and therefore it suffices to show that the operators

dρ_{λ}(a, 0, a) = \frac{1}{2}i(a|4B_{λ} + z) and dρ_{λ}(0, 2L(a), 0) = \frac{1}{2}(a|4B_{λ} − z)

are skew-adjoint on P(X_{λ}) with respect to the L^{2}-inner product ⟨−, −⟩_{F_{λ}}. But this is clear by Proposition 4.5.

Theorem 4.11. The (𝔤, 𝔨)-module ( dρ_{λ}, P(X_{λ})) integrates to an irreducible unitary representation ρ_{λ} of the universal cover G̃ of G on F_{λ}. This representation factors to a finite cover of G if and only if λ ∈ ℚ. In particular it factors to a finite cover of G if λ ∈ W_{disc}.

Proof. By the previous results it only remains to check in which cases the minimal 𝔨-type P_{0}(X_{λ}) = ℂ1 integrates to a finite cover. The 𝔨-action on 1 is given by

dρ_{λ}(a, D, −a)1 = dπ_{λ}^{ℂ}(0, D + 2iL(a), 0)1 = \frac{rλ}{2n} Tr(2iL(a))1 = iλ tr(a)1.

Therefore, the center Z(𝔨) = ℝ(e, 0, −e) acts by

dρ_{λ}(e, 0, −e)1 = irλ1.

In K we have e^{π(e,0,−e)} = 1 and hence, the claim follows.

In the Fock model the action of the maximal compact subgroup is quite explicit. For this recall the group homomorphism η : K̃ → U ⊆ L_{ℂ} with differential dη(u, D, −u) = D +2iL(u) defined in Section 1.4 and the character ξ_{λ} : K̃ → 𝕋 of K̃ with differential dξ_{λ}(u, T, −u) = iλ tr(u) defined in Section 2.1.

Proposition 4.12. For k ∈ K̃ we have

ρ_{λ}(k)F (z) = ξ_{λ}(k)F (η(k)^{#}z),  z ∈ X_{λ}.

Proof. The action of X = (u, D, −u) ∈ 𝔨 is given by

dρ_{λ}(X) = dπ_{λ}^{ℂ}(0, dη(X), 0)

= ∂_{dη(X)^{#}} + \frac{rλ}{2n} Tr( dη(X)^{#}) = ∂_{dη(X)^{#}} + iλ tr(u)

which implies the claim.

4.4 The reproducing kernel

We now calculate the reproducing kernel 𝕂_{λ}(z, w) of the Hilbert space F_{λ}. For this we first calculate the reproducing kernels 𝕂_{λ}^{m}(z, w) on the finitedimensional subspaces P_{m}(X_{λ}) endowed with the inner product of F_{λ}.

Lemma 4.13. Let λ ∈ W and m ≥ 0. If λ = k \frac{d}{2}, k = 0, …, r − 1, we additionally assume that m_{k+1} = … = m_{r} = 0. Then the following invariance property holds:

𝕂_{λ}^{m}(gz, w) = 𝕂_{λ}^{m}(z, g^{∗}w),  z, w ∈ X_{λ}, g ∈ L_{ℂ}.

Proof. Using Proposition 4.12 we find that for k ∈ K̃ and F ∈ P_{m}(X_{λ}) we have

ρ_{λ}(k)F (z) = ⟨ρ_{λ}(k)F, 𝕂_{λ}^{m}(−, z)⟩_{F_{λ}} = ⟨F, ρ_{λ}(k^{−1})𝕂_{λ}^{m}(−, z)⟩_{F_{λ}}
= \overline{ξ_{λ}(k^{−1})}⟨F, 𝕂_{λ}^{m}((η(k)^{−1})^{#}−, z)⟩_{F_{λ}}
= ξ_{λ}(k)⟨F, 𝕂_{λ}^{m}((η(k)^{−1})^{#}−, z)⟩_{F_{λ}}

and on the other hand

ρ_{λ}(k)F (z) = ξ_{λ}(k)F (η(k)^{#}z) = ξ_{λ}(k)⟨F, 𝕂_{λ}^{m}(−, η(k)^{#}z)⟩_{F_{λ}}.

Since η : K̃ → U is surjective and u^{−1} = u^{∗} = \overline{u}^{#} as well as \overline{u} ∈ U for u ∈ U we obtain

𝕂_{λ}^{m}(uz, w) = 𝕂_{λ}^{m}(z, u^{∗}w),  z, w ∈ X_{λ}, u ∈ U.

Now both sides are holomorphic in u ∈ L_{ℂ} and U ⊆ L_{ℂ} is totally real. Hence the claim follows.

Proposition 4.14. Let λ ∈ W and m ≥ 0. If λ = k \frac{d}{2}, k = 0, …, r − 1, we additionally assume that m_{k+1} = … = m_{r} = 0. Then

𝕂_{λ}^{m}(z, w) = \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} Φ_{m} (\frac{z}{2}, \frac{w}{2} ) ,  z, w ∈ X_{λ}.  (4.3)

Proof. First let λ > (r−1)\frac{d}{2}. By Lemma 4.13 the function K_{λ}^{m}(−, e) ∈ P_{m}(X_{λ}) is K^{L}-invariant and hence there is a constant c_{λ}^{m} such that K_{λ}^{m}(z, e) = c_{λ}^{m}Φ_{m}(z). Since

1 = Φ_{m}(e) = ⟨Φ_{m}, K_{λ}^{m}(−, e)⟩_{F_{λ}} = c_{λ}^{m}∥Φ_{m}∥_{F_{λ}}^{2},

formula (4.3) now follows with Proposition 4.2. Now, for λ > (r − 1)\frac{d}{2} this implies the following identity:

p(z) = [p, 𝕂_{λ}^{m}(−, z)]_{λ} = \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} · p(B_{λ})_{w}Φ_{m}(w, \overline{z})|_{w=0}  (4.4)

for z ∈ V_{ℂ} and p ∈ P_{m}(V_{ℂ}). The right hand side is clearly meromorphic in λ with possible poles at the points where (λ)_{m} = 0. For λ = k \frac{d}{2}, k ∈ {0, …, r − 1}, and m ≥ 0 with m_{k+1} = … = m_{r} = 0 we have (λ)_{m} ≠ 0 and hence the identity (4.4) holds for such λ and m by analytic continuation. Since the reproducing kernel of P_{m}(X_{λ}) is uniquely determined by (4.4) the claim now follows also for λ = k \frac{d}{2}.

Theorem 4.15. The reproducing kernel 𝕂_{λ}(z, w) of the Hilbert space F_{λ} is given by

𝕂_{λ}(z, w) = I_{λ} (\frac{z}{2}, \frac{w}{2} ) ,  z, w ∈ X_{λ}.

Proof. Let 0 ≤ k ≤ r be such that X_{λ} = X_{k}. By the previous result 𝕂_{λ}^{m}(z, w) is the reproducing kernel for the subspace P_{m}(X_{λ}). Further we know by Proposition 4.2 that the spaces P_{m}(X_{λ}) are pairwise orthogonal. Therefore, by [45, Proposition I.1.8], the sum

∑_{m≥0,m_{k+1}=0} 𝕂_{λ}^{m}(z, w) = ∑_{m≥0,m_{k+1}=0} \frac{d_{m}}{(\frac{n}{r})_{m}(λ)_{m}} Φ_{m} (\frac{z}{2}, \frac{w}{2} ) = I_{λ} (\frac{z}{2}, \frac{w}{2} )

converges pointwise to the reproducing kernel 𝕂_{λ}(z, w) of the direct Hilbert sum of all subspaces P_{m}(X_{λ}) with m ≥ 0, m_{k+1} = 0. But this direct Hilbert sum is by definition F_{λ} and the proof is complete.

The following consequence is a standard result for reproducing kernel spaces and can e.g. be found in [45, page 9].

Corollary 4.16. For every F ∈ F_{λ} and every z ∈ X_{λ} we have

|F (z)| ≤ I_{λ} (\frac{z}{2}, \frac{z}{2})^{\frac{1}{2}} ∥F ∥_{F_{λ}}.

4.5 Rings of differential operators and associated varieties

We recall the definition of the associated variety of an admissible representation for the example (ρ_{λ}, F_{λ}). Let (U_{k}(𝔤))_{k∈ℕ_{0}} denote the usual filtration of the universal enveloping algebra U(𝔤) and form the corresponding graded algebra gr U(𝔤) which is by the Poincare–Birkhoff–Witt theorem naturally isomorphic to the symmetric algebra S(𝔤_{ℂ}). The underlying (𝔤, K̃)-module X^{λ} = P(X_{λ}) of the representation ρ_{λ} carries an action of U(𝔤). For m ∈ ℕ_{0} further let X_{m}^{λ} be the subspace of polynomials in P(X_{λ}) of degree ≤ m. Then X_{0}^{λ} = ℂ1 is K̃-invariant and generates X^{λ} as a U(𝔤)-module. We further have

dρ_{λ}(U_{k}(𝔤))X_{m}^{λ} = X_{k+m}^{λ},  k, m ∈ ℕ_{0},

i.e. the filtrations (U_{k}(𝔤))_{k} and (X_{m}^{λ})_{m} are compatible. Thus the corresponding graded space

gr X^{λ} = ⨁_{m=0}^{∞} X_{m}^{λ}/X_{m−1}^{λ}

is a module over gr U(𝔤). Consider the annihilator ideal

J_{λ} := Ann_{gr U(𝔤)}(gr X^{λ}) ⊆ gr U(𝔤) ≅ S(𝔤_{ℂ}) ≅ ℂ[𝔤_{ℂ}^{∗}].

Then the associated variety V(ρ_{λ}) of ρ_{λ} is by definition the affine subvariety of 𝔤_{ℂ}^{∗} consisting of the common zeros of J_{λ}. Since 𝔨_{ℂ} lives in degree 1 in gr U(𝔤), but leaves each X_{m}^{λ} invariant, every element in V(ρ_{λ}) vanishes on 𝔨_{ℂ} and we can view V(ρ_{λ}) as a subset of 𝔭_{ℂ}^{∗}. Via the Killing form we identify 𝔭_{ℂ}^{∗} with 𝔭_{ℂ} and view V(ρ_{λ}) as a subset of 𝔭_{ℂ}. Then V(ρ_{λ}) is a K_{ℂ}-stable closed subvariety of 𝔭_{ℂ} consisting of nilpotent elements and hence the union of finitely many nilpotent K_{ℂ}-orbits (see [50, Corollary 5.23]). Recall from Section 1.5 the K_{ℂ}-orbits O_{k}^{K_{ℂ}} ⊆ 𝔭^{+} which are isomorphic to the L_{ℂ}-orbits X_{k} via the Cayley type transform C ∈ Int(𝔤_{ℂ}). The following result is due to A. Joseph [30, Theorem 7.14]:

Proposition 4.17. Let λ ∈ W and 0 ≤ k ≤ r such that X_{λ} = X_{k}. Then V(ρ_{λ}) = \overline{O_{k}^{K_{ℂ}}}.

Corollary 4.18. For λ ∈ W let k ∈ {0, …, r} such that X_{λ} = X_{k}. Then the Gelfand–Kirillov dimension of ρ_{λ} is k + k(2r − k − 1)\frac{d}{2}.

Proof. The Gelfand–Kirillov dimension of an irreducible unitary representation equals the dimension of its associated variety in 𝔭_{ℂ}^{∗} (combine [49, Corollary 4.7] and [50, Theorem 8.4]). Therefore the result follows from (1.9).

For an algebraic variety 𝕏 over ℂ denote by ℂ[𝕏] the ring of regular functions. Further let 𝔻(𝕏) be the ring of algebraic differential operators on 𝕏. This subring of End_{ℂ}(ℂ[𝕏]) can be defined inductively as follows: Let 𝔻_{0}(𝕏) := ℂ[𝕏] be the ring of multiplication operators and for m ∈ ℕ put

𝔻_{m}(𝕏) := {D ∈ End_{ℂ}(ℂ[𝕏]) : [D, f] ∈ 𝔻_{m−1}(𝕏) ∀ f ∈ ℂ[𝕏]}.

Then 𝔻(𝕏) = ⋃_{m∈ℕ_{0}} 𝔻_{m}(𝕏). Since the varieties \overline{X_{k}} are affine it follows that ℂ[\overline{X_{k}}] = P(X_{k}). The representation dρ_{λ} acts on P(X_{λ}) by differential operators and hence it induces a map

dρ_{λ} : U(𝔤) → 𝔻(\overline{X_{λ}}).

The following result is a qualitative version of [30, Theorem 4.5]:

Theorem 4.19. For λ = k \frac{d}{2} ∈ W_{disc} the map dρ_{λ} : U(𝔤) → 𝔻(\overline{X_{k}}) is surjective and induces an isomorphism U(𝔤)/J_{k} ≅ 𝔻(\overline{X_{k}}), where J_{k} = Ann_{U(𝔤)}(X^{λ}).

Proof. By [30, Theorem 4.5] the map dρ_{λ} is surjective onto the space of ℂendomorphisms of P(X_{λ}) = ℂ[\overline{X_{k}}] which are locally finite under the diagonal action of 𝔭^{−}. Now 𝔭^{−} acts by multiplication with coordinate functions and hence the condition for D ∈ End_{ℂ}(ℂ[\overline{X_{k}}]) to be locally finite under the action of 𝔭^{−} is equivalent to the existence of N ∈ ℕ such that the iterated commutator [[… [D, f_{1}(x)], …, f_{N−1}(x)], f_{N}(x)] = 0 for all f_{1}, …, f_{N} ∈ ℂ[\overline{X_{k}}]. This again is equivalent to D ∈ 𝔻(\overline{X_{k}}) and the proof is complete.

Remark 4.20. A quantitative version of Theorem 4.19 was obtained by Levasseur–Smith–Stafford [38] for the minimal orbit X_{1}, by Levasseur–Stafford [39] for classical 𝔤 and finally by Joseph [30] for the general case. However, their version is less explicit and does not provide a geometric construction of the unitary structure.

Corollary 4.21. For k = 1, …, r − 1 the ring 𝔻(\overline{X_{k}}) of algebraic differential operators on the affine variety \overline{X_{k}} is generated by the multiplications with regular functions in ℂ[\overline{X_{k}}] and the Bessel operators (u|B_{λ}), u ∈ V_{ℂ}, for λ = k \frac{d}{2}.

Proof. By Theorem 4.19 the ring 𝔻(\overline{X_{k}}) is generated by the constants and dρ_{λ}(𝔤). Note that in the decomposition 𝔤_{ℂ} = 𝔭^{−} ⊕ 𝔨_{ℂ} ⊕ 𝔭^{+} the subalgebra 𝔨_{ℂ} is generated by 𝔭^{+} and 𝔭^{−} and hence 𝔻(\overline{X_{k}}) is generated by the constants, dρ_{λ}(𝔭^{+}) = {(u|B_{λ}) : u ∈ V_{ℂ}} and dρ_{λ}(𝔭^{−}) = {(v|z) : v ∈ V_{ℂ}}.

Remark 4.22. Neither Theorem 4.19 nor Corollary 4.21 can hold for λ ∈ W_{cont} resp. k = r since in this case \overline{X_{λ}} = \overline{X_{r}} = V_{ℂ} and 𝔻(V_{ℂ}) = ℂ[x, \frac{∂}{∂x}] is a Weyl algebra (see [39, Lemma IV.1.5]).

5 The Segal–Bargmann transform

For every λ ∈ W we explicitly construct an intertwining operator, the Segal– Bargmann transform, between the Schrödinger model (π_{λ}, L^{2}(O_{λ}, dμ_{λ})) and the Fock model (ρ_{λ}, F_{λ}) in terms of its integral kernel.

5.1 Construction of the Segal–Bargmann transform

For each λ ∈ W the Segal–Bargmann transform 𝔹_{λ} is defined for ψ ∈ L^{2}(O_{λ}, dμ_{λ}) by

𝔹_{λ}ψ(z) := e^{− \frac{1}{2} tr(z)} ∫_{O_{λ}} I_{λ}(z, x)e^{− tr(x)}ψ(x) dμ_{λ}(x),  z ∈ V_{ℂ}.  (5.1)

Recall the space O(V_{ℂ}) of holomorphic functions on V_{ℂ}.

Proposition 5.1. For ψ ∈ L^{2}(O_{λ}, dμ_{λ}) the integral in (5.1) converges uniformly on bounded subsets and defines a function 𝔹_{λ}ψ ∈ O(V_{ℂ}). The Segal– Bargmann transform 𝔹_{λ} is a continuous linear operator

L^{2}(O_{λ}, dμ_{λ}) → O(V_{ℂ}).

Proof. Since the kernel function e^{− \frac{1}{2} tr(z)}I_{λ}(z, x)e^{− tr(x)} is analytic in z, it suffices to show that its L^{2}-norm in x has a uniform bound for |z| ≤ R, R > 0. By Lemma 3.1 there exists C > 0 such that

|I_{λ}(z, x)| ≤ C(1 + |z| · |x|)^{\frac{r(2n−1)}{4}} e^{2r\sqrt{|z|·|x|}}.

Then for x ∈ O_{λ}, z ∈ V_{ℂ} with |z| ≤ R, we find

|e^{− \frac{1}{2} tr(z)}I_{λ}(z, x)e^{− tr(x)}| ≤ C′(1 + R · |x|)^{\frac{r(2n−1)}{4}} e^{2r\sqrt{R·|x|}}e^{− tr(x)}
≤ C′(1 + R · |x|)^{\frac{r(2n−1)}{4}} e^{2r\sqrt{R}\sqrt{|x|}−|x|}

with C′ = C max_{|z|≤R} |e^{− \frac{1}{2} tr(z)}|. This is certainly L^{2} as a function of x ∈ O_{λ} with norm independent of z and the claim follows.

Next, we show that 𝔹_{λ} intertwines the action dπ_{λ} on L^{2}(O_{λ}, dμ_{λ}) with the action dρ_{λ} on F_{λ}. Recall the Cayley type transform C ∈ Int(𝔤_{ℂ}) introduced in Section 1.3.

Proposition 5.2. The following intertwining identity holds on L^{2}(O_{λ}, dμ_{λ})^{∞}:

𝔹_{λ} ∘ dπ_{λ}(X) = dπ_{λ}^{ℂ}(C(X)) ∘ 𝔹_{λ},  X ∈ 𝔤.  (5.2)

Proof. As remarked in Section 1.3 the subspace {(a, L(b), a) : a, b ∈ V } generates 𝔤 as a Lie algebra. Hence it suffices to prove (5.2) for the elements (a, ±2iL(a), a), a ∈ V. We show (5.2) for X = (a, −2iL(a), a), a ∈ V, the proof for (a, +2iL(a), a) works similarly. For X = (a, −2iL(a), a) we have C(X) = (a, 0, 0) and hence

( dπ_{λ}^{ℂ}(C(X)) ∘ 𝔹_{λ})ψ(z)
= i(a|z)e^{− \frac{1}{2} tr(z)} ∫_{O_{λ}} I_{λ}(z, x)e^{− tr(x)}ψ(x) dμ_{λ}(x).

By Proposition 3.9 we have zI_{λ}(z, x) = (B_{λ})_{x}I_{λ}(z, x). Further the Bessel operator B_{λ} is symmetric on L^{2}(O_{λ}, dμ_{λ}) and we obtain

= ie^{− \frac{1}{2} tr(z)} ∫_{O_{λ}} (a|B_{λ})_{x}I_{λ}(z, x)e^{− tr(x)}ψ(x) dμ_{λ}(x)
= ie^{− \frac{1}{2} tr(z)} ∫_{O_{λ}} I_{λ}(z, x)(a|B_{λ}) [e^{− tr(x)}ψ(x)] dμ_{λ}(x).

By the product rule (1.12) for the Bessel operator we obtain

B_{λ} [e^{− tr(x)}ψ(x)]
= B_{λ}e^{− tr(x)} · ψ(x) + 2P (\frac{∂}{∂x}e^{− tr(x)}, \frac{∂ψ}{∂x} ) x + e^{− tr(x)} · B_{λ}ψ(x)
= (x − λe)e^{− tr(x)}ψ(x) − 2e^{− tr(x)}x · \frac{∂ψ}{∂x} + e^{− tr(x)}B_{λ}ψ(x)

since

B_{λ}e^{− tr(x)} = (P (−e)x − λe)e^{− tr(x)} = (x − λe)e^{− tr(x)}  and
\frac{∂}{∂x}e^{− tr(x)} = −e · e^{− tr(x)}.

Hence we have

(a|B_{λ}) [e^{− tr(x)}ψ(x)] =e^{− tr(x)} [(a|x)ψ(x) − λ tr(a)ψ(x) − 2∂_{ax}ψ(x) + B_{λ}ψ(x)] .

Inserting this into our calculation above we find

( dπ_{λ}^{ℂ}(C(X)) ∘ 𝔹_{λ})ψ(z)
= 𝔹_{λ} ∘ (i(a|x) − 2i [∂_{L(a)x} + \frac{rλ}{2n} Tr(L(a))] + i(a|B_{λ})) ψ(z)
= (𝔹_{λ} ∘ dπ_{λ}(X))ψ(z).

To conclude that 𝔹_{λ} is an isomorphism L^{2}(O_{λ}, dμ_{λ}) → F_{λ}, we show that it maps the underlying (𝔤, 𝔨)-module W^{λ} ⊆ L^{2}(O_{λ}, dμ_{λ}) to the (𝔤, 𝔨)-module P(X_{λ}). In order to do so we show that the function

Ψ_{0} := 𝔹_{λ}ψ_{0} ∈ O(V_{ℂ})

is L_{ℂ}-invariant. In fact, the function ψ_{0} is K̃-equivariant by the character ξ_{λ} (see Section 2.1). By Proposition 5.2 the same has to be true for Ψ_{0}. But by Proposition 4.12 this implies that Ψ_{0} is invariant under η(K̃)^{#} = U. Now U is a real form of L_{ℂ} and the action of L_{ℂ} on Ψ_{0} is holomorphic, whence Ψ_{0} has to be L_{ℂ}-invariant. Therefore it has to be constant on every L_{ℂ}-orbit. Since Ψ_{0} is holomorphic on V_{ℂ} and V_{ℂ} decomposes into finitely many L_{ℂ}-orbits, it follows that Ψ_{0} is constant on V_{ℂ}. It remains to show that Ψ_{0} is non-zero.

Proposition 5.3. Ψ_{0}(0) = 1 and hence Ψ_{0} = 𝔹_{λ}ψ_{0} = 1.

Proof. Since I_{λ}(0, x) = 1 we have

Ψ_{0}(0) = ∫_{O_{λ}} e^{−2 tr(x)} dμ_{λ}(x) = ∥ψ_{0}∥_{L^{2}(O_{λ}, dμ_{λ})}^{2} = 1

as shown in Section 1.2.

Theorem 5.4. 𝔹_{λ} is a unitary isomorphism L^{2}(O_{λ}, dμ_{λ}) → F_{λ} intertwining the actions π_{λ} and ρ_{λ}.

Proof. Since W^{λ} = L^{2}(O_{λ}, dμ_{λ})_{𝔨} is generated by ψ_{0}, P(X_{λ}) = (F_{λ})_{𝔨} is generated by Ψ_{0} = 1, 𝔹_{λ}ψ_{0} = Ψ_{0} and 𝔹_{λ} intertwines the actions dπ_{λ} and dρ_{λ}, it has to map the irreducible (𝔤, 𝔨)-module W^{λ} into the irreducible (𝔤, 𝔨)-module P(X_{λ}) and is therefore, thanks to Schur’s Lemma, an isomorphism between the underlying (𝔤, 𝔨)-modules. It only remains to show that 𝔹_{λ} is isometric between W^{λ} and P(X_{λ}), then the statement follows since W^{λ} ⊆ L^{2}(O_{λ}, dμ_{λ}) and P(X_{λ}) ⊆ F_{λ} are dense. Both W^{λ} and P(X_{λ}) are irreducible infinitesimally unitary (𝔤, 𝔨)-modules and hence 𝔹_{λ} is a scalar multiple of a unitary operator. Since further

∥𝔹_{λ}ψ_{0}∥_{F_{λ}} = ∥1∥_{F_{λ}} = 1 = ∥ψ_{0}∥_{L^{2}(O_{λ}, dμ_{λ})}

the operator 𝔹_{λ} itself has to be unitary.

Remark 5.5. The Segal–Bargmann transform can also be obtained via a restriction principle (see [28, 46] for other instances of this principle). The formula R_{λ}F (x) = e^{− \frac{1}{2} tr(x)}F (x) defines an operator P(X_{λ}) → L^{2}(O_{λ}, dμ_{λ}) and hence we obtain a densely defined unbounded operator R_{λ} : F_{λ} → L^{2}(O_{λ}, dμ_{λ}). We consider its adjoint R_{λ}^{∗} : L^{2}(O_{λ}, dμ_{λ}) → F_{λ} as a densely defined unbounded operator. One can show that the Segal–Bargmann transform appears as the unitary part in the polar decomposition of the operator R_{λ}^{∗}:

R_{λ}^{∗} = 𝔹_{λ} ∘ \sqrt{R_{λ}R_{λ}^{∗}}.

For the case of the minimal discrete Wallach point this is done in [23, Proposition 5.5].

Corollary 5.6. The inverse Segal–Bargmann transform is given by

𝔹_{λ}^{−1}F (x) = \frac{e^{− tr(x)}}{c_{λ}} ∫_{X_{λ}} I_{λ}(x, z)e^{− \frac{1}{2} tr(\overline{z})}F (z)ω_{λ}(z) dν_{λ}(z), F ∈ F_{λ}.

Proof. Since the Segal–Bargmann transform is a unitary operator we have

⟨𝔹_{λ}^{−1}F, ψ⟩_{L^{2}(O_{λ}, dμ_{λ})} = ⟨F, 𝔹_{λ}ψ⟩_{F_{λ}} = \frac{1}{c_{λ}} ∫_{X_{λ}} F (z)\overline{𝔹_{λ}ψ(z)}ω_{λ}(z) dν_{λ}(z)

= \frac{1}{c_{λ}} ∫_{X_{λ}} ∫_{O_{λ}} F (z)e^{− \frac{1}{2} \overline{tr(z)}}\overline{I_{λ}(z, x)}e^{− tr(x)}\overline{ψ(x)} dμ_{λ}(x)ω_{λ}(z) dν_{λ}(z)
= \frac{1}{c_{λ}} ∫_{O_{λ}} (e^{− tr(x)} ∫_{X_{λ}} e^{− \frac{1}{2} tr(\overline{z})}I_{λ}(x, z)F (z)ω_{λ}(z) dν_{λ}(z)) \overline{ψ(x)} dμ_{λ}(x)

which implies the claim.

We now use the Segal–Bargmann transform to obtain an intrinsic description of the Fock space.

Theorem 5.7.

F_{λ} = {F |_{X_{λ}} : F ∈ O(V_{ℂ}), ∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) < ∞} .

Proof. The Segal–Bargmann transform is an isomorphism 𝔹_{λ} : L^{2}(O_{λ}, dμ_{λ}) → F_{λ} and hence by Proposition 5.1 every function in F_{λ} extends to a holomorphic function on V_{ℂ}. This shows the inclusion ⊆. For the other inclusion let F ∈ O(V_{ℂ}) such that

∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) < ∞.

We expand F into a power series which we can arrange as

F = ∑_{m≥0} p_{m} with p_{m} ∈ P_{m}(V_{ℂ})

by Theorem 1.6. Since p_{m}|_{X_{λ}} = 0 for m_{k+1} ≠ 0 by Corollary 1.9 we may assume F = ∑_{m≥0, m_{k+1}=0} p_{m} for the study of F |_{X_{λ}}. This series converges uniformly on compact subsets. We show that this series also converges in L^{2}(X_{λ}, ω_{λ} dν_{λ}). For R > 0 let

X_{λ}^{R} := {z ∈ X_{λ} : |z| ≤ R},
O_{λ}^{R} := {x ∈ O_{λ} : |x| ≤ R}.

Note that X_{λ}^{R} and O_{λ}^{R} are compact and hence integration over these sets commutes with taking the limit F = ∑_{m≥0, m_{k+1}=0} p_{m}. With (1.10) we find

∞ > ∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) = lim_{R→∞} ∫_{X_{λ}^{R}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z)

= lim_{R→∞} ∑_{\begin{matrix}m,n≥0, \\ m_{k+1}=n_{k+1}=0 \end{matrix}} ∫_{O_{λ}^{R}} ∫_{U} p_{m}(ux^{\frac{1}{2}} )\overline{p_{n}(ux^{\frac{1}{2}} )} duω_{λ}(x^{\frac{1}{2}} ) dμ_{λ}(x).

By Theorem 1.7 and Lemma 1.14 the integrals over U for m ≠ n vanish and we obtain

= lim_{R→∞} ∑_{m_{k+1}=0^{m≥0,}} ∫_{X_{λ}^{R}} |p_{m}(z)|^{2}ω_{λ}(z) dν_{λ}(z) = ∑_{m_{k+1}=0^{m≥0,}} ∥p_{m}∥_{L^{2}(X_{λ},ω_{λ} dν_{λ})}^{2}

which shows convergence in L^{2}(X_{λ}, ω_{λ} dν_{λ}). Since F_{λ} is the closure of the space of polynomials with respect to the norm of L^{2}(X_{λ}, ω_{λ} dν_{λ}) it is clear that F |_{X_{λ}} ∈ F_{λ} which shows the other inclusion.

This intrinsic description leads to the following conjecture which was proved in [23, Theorem 2.26] for the minimal orbit, i.e. λ ∈ W with X_{λ} = X_{1}:

Conjecture 5.8.

F_{λ} = {F ∈ O(X_{λ}) : ∫_{X_{λ}} |F (z)|^{2}ω_{λ}(z) dν_{λ}(z) < ∞} .

Recall the Laguerre functions ℓ_{m}^{λ} introduced in Section 1.8.

Proposition 5.9. Let λ ∈ W and k ∈ {0, …, r} such that O_{λ} = O_{k}. Then for every m ≥ 0 with m_{k+1} = 0 we have

𝔹_{λ}ℓ_{m}^{λ} = \frac{(−1)^{|m|}}{2^{|m|}} Φ_{m}.

Proof. ℓ_{m}^{λ} is the unique (up to scalar multiples) 𝔨^{𝔩}-invariant vector in the 𝔨type W_{m}^{λ} ⊆ L^{2}(O_{λ}, dμ_{λ}) whereas Φ_{m} is the unique (up to scalar multiples) 𝔨^{𝔩}-invariant vector in the 𝔨-type P_{m}(X_{λ}) ⊆ F_{λ}. Hence

𝔹_{λ}ℓ_{m}^{λ} = const ·Φ_{m}.

To find the constant we evaluate 𝔹_{λ}^{−1}Φ_{m}(x) and ℓ_{m}^{λ}(x) at x = 0. First observe that

ℓ_{m}^{λ}(0) = L_{m}^{λ}(0) = (λ)_{m}.

Next we calculate 𝔹_{λ}^{−1}Φ_{m}(0) using Corollary 5.6:

𝔹_{λ}^{−1}Φ_{m}(0) = \frac{1}{c_{λ}} ∫_{X_{λ}} e^{− \frac{1}{2} tr(\overline{z})}Φ_{m}(z)ω_{λ}(z) dν_{λ}(z).

By (1.17) the function e^{tr(z)} has the following expansion:

e^{tr(z)} = ∑_{m≥0} \frac{d_{m}}{(\frac{n}{r})_{m}} Φ_{m}(z).

Inserting this yields

𝔹_{λ}^{−1}Φ_{m}(0) = ∑_{n≥0} (−1)^{|n|} \frac{d_{n}}{2^{|n|}(\frac{n}{r})_{n}} · \frac{1}{c_{λ}} ∫_{X_{λ}} Φ_{m}(z)\overline{Φ_{n}(z)}ω_{λ}(z) dν_{λ}(z)

= ∑_{n≥0} (−1)^{|n|} \frac{d_{n}}{2^{|n|}(\frac{n}{r})_{n}} ⟨Φ_{m}, Φ_{n}⟩_{F_{λ}}

= (−1)^{|m|} \frac{d_{m}}{2^{|m|}(\frac{n}{r})_{m}} · \frac{4^{|m|}(\frac{n}{r})_{m}(λ)_{m}}{d_{m}}

= (−1)^{|m|}2^{|m|}(λ)_{m},

where we have used Proposition 4.2.

5.2 Intertwiner to the bounded symmetric domain model

The Fock space F_{λ} and the bounded symmetric domain model H_{λ}^{2}(D) have (as vector spaces) the same underlying (𝔤, K̃)-module, namely

⨁_{m≥0, m_{k+1}=0} P_{m}(V_{ℂ}),

where k ∈ {0, …, r} is such that X_{λ} = X_{k}. However, inner product and group action differ. In Remark 4.3 we already compared the norms in the two models. To gain a better understanding of the relation between the two models we investigate the intertwiner between them. For this we compose the inverse Segal–Bargmann transform 𝔹_{λ}^{−1} with the Laplace transform L_{λ} and the pullback of the Cayley transform γ_{λ} to obtain an intertwining operator between the Fock space picture and the realization on functions on the bounded symmetric domain. (For the definition of L_{λ} and γ_{λ} see Sections 2.2 and 2.3.) Let

A_{λ} := γ_{λ} ∘ L_{λ} ∘ 𝔹_{λ}^{−1} : F_{λ} → H_{λ}^{2}(D).

The operator A_{λ} intertwines the actions ρ_{λ} and π_{λ}^{D}.

Theorem 5.10. For F ∈ F_{λ} we have

A_{λ}F (z) = \frac{1}{c_{λ}} ∫_{X_{λ}} e^{− \frac{1}{2} (z|\overline{w})}F (w)ω_{λ}(w) dν_{λ}(w),  z ∈ D.

Geom. Quant. for Unit. Highest Weight Rep. 843

Proof. Using Lemma 3.8 we obtain

A_{λ}F (z) = γ_{λ} ∘ L_{λ} ∘ 𝔹_{λ}^{−1}F (z) = Δ(e − z)^{−λ}L_{λ} ∘ 𝔹_{λ}^{−1}F (c(z))

= Δ(e − z)^{−λ} ∫_{O_{λ}} e^{i(c(z)|u)}𝔹_{λ}^{−1}F (u) dμ_{λ}(u)
= \frac{Δ(e − z)^{−λ}}{c_{λ}} ∫_{O_{λ}} ∫_{X_{λ}} e^{i(c(z)|u)}e^{− tr(u)}I_{λ}(u, w)e^{− \frac{1}{2} tr(\overline{w})}F (w)
ω_{λ}(w) dν_{λ}(w) dμ_{λ}(u)
= \frac{Δ(e − z)^{−λ}}{c_{λ}} ∫_{X_{λ}} (∫_{O_{λ}} e^{−(u|e−ic(z))}I_{λ}(u, w) dμ_{λ}(u)) e^{− \frac{1}{2} tr(\overline{w})}F (w)
ω_{λ}(w) dν_{λ}(w)
= \frac{2^{rλ}Δ(e − z)^{−λ}}{c_{λ}} ∫_{X_{λ}} Δ(e − ic(z))^{−λ}e^{((e−ic(z))^{−1}|\overline{w})}e^{− \frac{1}{2} tr(\overline{w})}F (w)
ω_{λ}(w) dν_{λ}(w).

Now

e − ic(z) = e + (e + z)(e − z)^{−1} = 2(e − z)^{−1}

and therefore

Δ(e − ic(z))^{−λ} = 2^{−rλ}Δ(e − z)^{λ}.

This also implies that

((e − ic(z))^{−1}|\overline{w}) = \frac{1}{2} tr(\overline{w}) − \frac{1}{2}(z|\overline{w})

so that altogether we obtain

A_{λ}F (z) = \frac{1}{c_{λ}} ∫_{X_{λ}} e^{− \frac{1}{2} (z|\overline{w})}F (w)ω_{λ}(w) dν_{λ}(w).

Corollary 5.11. The operator A_{λ} acts on P_{m}(V_{ℂ}) by the scalar \frac{(−1)^{|m|}(λ)_{m}}{2^{|m|}}. Proof. Using the expansion (1.17) and Proposition 4.14 we obtain

e^{(z|\overline{w})} = ∑_{n≥0} (λ)_{n}𝕂_{λ}^{n}(z, w).

Hence for F ∈ P_{m}(V_{ℂ}) we find

A_{λ}F (z) = ∑_{n≥0} (λ)_{n}⟨F, 𝕂_{λ}^{n}(−, −\frac{1}{2}z)⟩_{F_{λ}} = (λ)_{m}F (−\frac{1}{2}z) = \frac{(−1)^{|m|}(λ)_{m}}{2^{|m|}} F (z).

6 The unitary inversion operator

Using the Segal–Bargmann transform we now calculate the integral kernel of the unitary inversion operator U_{λ}. The unitary inversion operator is essentially given by the action π_{λ}(j̃) of the inversion element (see [22, Section 3.3])

j̃ = expG̃ (\frac{π}{2} (e, 0, −e)) ∈ G̃.

More precisely, we put

U_{λ} := e^{−iπ \frac{rλ}{2}} π_{λ}(j̃).

The operator U_{λ} is unitary on L^{2}(O_{λ}, dμ_{λ}) of order 2, i.e. U_{λ}^{2} = id (see [22, Proposition 3.17 (1)]). We also study Whittaker vectors in the Schrödinger model. They can be derived from the explicit expression of the integral kernel of U_{λ}.

6.1 The integral kernel of U_{λ}

Recall the underlying (𝔤, 𝔨)-module W^{λ} = P(O_{λ})e^{− tr(x)} of the representation (π_{λ}, L^{2}(O_{λ}, dμ_{λ})).

Proposition 6.1. For every ψ ∈ W^{λ} the integral

T_{λ}ψ(z) := 2^{−rλ} ∫_{O_{λ}} J_{λ}(z, x)ψ(x) dμ_{λ}(x),  z ∈ V_{ℂ},

converges uniformly on compact subsets of V_{ℂ}. This defines a linear operator T_{λ} : W^{λ} → C(O_{λ}).

Proof. Let ψ(x) = p(x)e^{− tr(x)} ∈ P(O_{λ})e^{− tr(x)} = W^{λ}. Then |p(x)| ≤ C_{1}(1 + |x|)^{N} for some constants C_{1}, N > 0. Further, by Lemma 3.1 we have

|J_{λ}(z, x)| ≤ C_{2}(1 + |z| · |x|)^{\frac{r(2n−1)}{4}} e^{2r\sqrt{|z|·|x|}}  ∀ x ∈ O_{λ}, z ∈ V_{ℂ}

for a constant C_{2} > 0. Hence, for |z| ≤ R, R ≥ 1, we obtain

∫_{O_{λ}} |J_{λ}(z, x)ψ(x)| dμ_{λ}(x)
≤ C_{1}C_{2} ∫_{O_{λ}} (1 + R|x|)^{N+ \frac{r(2n−1)}{4}} e^{2r\sqrt{R}\sqrt{|x|}−|x|} dμ_{λ}(x) < ∞

and the proof is complete.

Proposition 6.2. The operator T_{λ} is a unitary isomorphism T_{λ} : W^{λ} → W^{λ} with T_{λ}ψ_{0} = ψ_{0} which intertwines the 𝔤-action dπ_{λ} with the 𝔤-action dπ_{λ} ∘ Ad(j).

Proof. By Proposition 3.6 the operator T_{λ} intertwines the Bessel operator B_{λ} with the coordinate multiplication −x and vice versa. Therefore we obtain

U_{λ} ∘ dπ_{λ}(X) = dπ_{λ}(Ad(j)X) ∘ U_{λ}  (6.1)

for X ∈ 𝔫⊕\overline{𝔫}. Since 𝔫 and \overline{𝔫} together generate 𝔤 as a Lie algebra the intertwining formula (6.1) holds for all X ∈ 𝔤. Further, for x ∈ O_{λ} ⊆ X_{λ} we find

T_{λ}ψ_{0}(x) = 2^{−rλ} ∫_{O_{λ}} J_{λ}(x, y)e^{− tr(y)} dμ_{λ}(y) = ∫_{O_{λ}} I_{λ}(−2x, y)e^{−2 tr(y)} dμ(y)
= e^{− tr(x)}𝔹_{λ}ψ_{0}(−2x) = e^{− tr(x)} = ψ_{0}(x).

Since W^{λ} = dπ_{λ}(U(𝔤))ψ_{0}, it follows that T_{λ}W^{λ} ⊆ dπ_{λ}(U(𝔤))T_{λ}ψ_{0} = W^{λ}. Now, since invariant Hermitian forms on the irreducible infinitesimally unitary (𝔤, 𝔨)-module W^{λ} are unique up to a scalar, we find that T_{λ} is a unitary isomorphism.

Theorem 6.3. U_{λ} = T_{λ}.

Proof. By the previous proposition U_{λ} ∘ T_{λ}^{−1} extends to a unitary isomorphism L^{2}(O_{λ}, dμ_{λ}) → L^{2}(O_{λ}, dμ_{λ}) which commutes with the 𝔤-action dπ_{λ}. Therefore, by Schur’s Lemma, U_{λ} is a scalar multiple of T_{λ}. Since U_{λ}ψ_{0} = ψ_{0} = T_{λ}ψ_{0} this gives the claim.

Remark 6.4. The group G is generated by the conformal inversion j and the maximal parabolic subgroup P := L^{∘} ⋉ N, where

L^{∘} = G ∩ Str(V).

Write P̃ for the preimage of P under the covering map G̃ → G. The restriction of π_{λ} to P̃ factors to P and is in the Schrödinger model quite simple. In fact

π_{λ}(n_{a})ψ(x) = e^{i(x|a)}ψ(x),  n_{a} ∈ N,
π_{λ}(g)ψ(x) = χ(g^{#})^{\frac{λ}{2}} ψ(g^{#}x),  g ∈ L^{∘},

and by Mackey theory this representation is even irreducible on L^{2}(O_{λ}, dμ_{λ}). Therefore, together with the action of j̃ in the Schrödinger model (see Theorem 6.3) this describes the complete group action π_{λ}. For the case of the rank 1 orbit O_{1} Theorem 6.3 was shown in [23, Theorem 4.3]. Earlier contributions to special cases are [33] for the case 𝔤 = 𝔰𝔬(2, k) and λ the minimal discrete Wallach point and [37] for the case 𝔤 = 𝔰𝔩(2, ℝ) and arbitrary λ ∈ W.

Remark 6.5. Since the functions x ↦ J_{λ}(x, y), y ∈ O_{λ}, are eigenfunctions of the Bessel operator, the inversion formula for U_{λ} gives an expansion of any function ψ ∈ L^{2}(O_{λ}, dμ_{λ}) into eigenfunctions of the Bessel operator:

ψ(x) = 2^{−rλ} ∫_{O_{λ}} J_{λ}(x, y) U_{λ}ψ(y) dμ_{λ}(y),  ψ ∈ L^{2}(O_{λ}, dμ_{λ}).

Compared to the action of j̃ in the Schrödinger model its action in the Fock model is extremely simple. Define (−1)^{∗} on F_{λ} by (−1)^{∗}F (z) = F (−z).

Proposition 6.6. 𝔹_{λ} ∘ U_{λ} = (−1)^{∗} ∘ 𝔹_{λ}.

Proof. We have dρ_{λ}(t(e, 0, −e)) = dπ_{λ}^{ℂ}(2it(0, 1, 0)) = 2it(∂_{z} + \frac{rλ}{2}). Therefore, we obtain

ρ_{λ}(e^{t(e,0,−e)})F (z) = e^{irλt}F (e^{2it}z).

For t = \frac{π}{2} we obtain the action of j̃ which is given by e^{iπ \frac{rλ}{2}} (−1)^{∗}.

This now allows us to compute the action of the unitary inversion operator on the Laguerre functions ℓ_{m}^{λ} introduced in Section 1.8.

Proposition 6.7.

U_{λ}ℓ_{m}^{λ} = (−1)^{|m|}ℓ_{m}^{λ}.

Proof. Since 𝔹_{λ}ℓ_{m}^{λ} = const ·Φ_{m} and Φ_{m}(−z) = (−1)^{|m|}Φ_{m}(z) the claim follows with Proposition 6.6.

6.2 Whittaker vectors

The integral kernel of the unitary inversion operator provides us with explicit Whittaker vectors for the representations π_{λ} on L^{2}(O_{λ}, dμ_{λ}) and hence with explicit embeddings into Whittaker models. For this let L^{2}(O_{λ}, dμ_{λ})^{∞} denote the space of smooth vectors of the representation π_{λ} endowed with the usual Frechet topology. Since π_{λ} leaves L^{2}(O_{λ}, dμ_{λ})^{∞} invariant the representation extends to the space L^{2}(O_{λ}, dμ_{λ})^{−∞} = (L^{2}(O_{λ}, dμ_{λ})^{∞})′ of distribution vectors by

⟨π_{λ}(g)u, φ⟩ := ⟨u, \overline{π_{λ}(g^{−1})}φ⟩,  g ∈ G̃,

for u ∈ L^{2}(O_{λ}, dμ_{λ})^{−∞} and φ ∈ L^{2}(O_{λ}, dμ_{λ})^{∞}. Here we use the complex conjugate \overline{U} of an operator U on L^{2}(O_{λ}, dμ_{λ})^{∞} which is defined by \overline{U}φ := \overline{U\overline{φ}} for φ ∈ L^{2}(O_{λ}, dμ_{λ})^{∞}. Further recall that W^{λ} = P(O_{λ})e^{− tr(x)} ⊆ L^{2}(O_{λ}, dμ_{λ}) is the underlying (𝔤, 𝔨)-module of π_{λ}. It does not carry a representation of the group G̃, but the representation dπ_{λ} of the Lie algebra 𝔤 restricts to W^{λ}. By duality dπ_{λ} extends to the algebraic dual (W^{λ})^{∗}:

⟨ dπ_{λ}(X)u, φ⟩ := −⟨u, \overline{dπ_{λ}(X)}φ⟩,  X ∈ 𝔤,

for u ∈ (W^{λ})^{∗} and φ ∈ W^{λ}. We have the following inclusions:

W^{λ} ⊆ L^{2}(O_{λ}, dμ_{λ})^{∞} ⊆ L^{2}(O_{λ}, dμ_{λ}) ⊆ L^{2}(O_{λ}, dμ_{λ})^{−∞} ⊆ (W^{λ})^{∗}.

Note that the unitary inversion operator U_{λ} leaves each of these spaces invariant.

Definition 6.8. A distribution u ∈ (W^{λ})^{∗} is an algebraic 𝔫-Whittaker vector of weight η ∈ 𝔫_{ℂ}^{∗} (resp. an algebraic \overline{𝔫}-Whittaker vector of weight η ∈ \overline{𝔫}_{ℂ}^{∗}) if

dπ_{λ}(X)u = η(X)u,  for all X ∈ 𝔫 (resp. X ∈ \overline{𝔫}).

If moreover u ∈ L^{2}(O_{λ}, dμ_{λ})^{−∞} we call u a smooth Whittaker vector.

For z ∈ V_{ℂ} let η_{z} ∈ 𝔫_{ℂ}^{∗} and \overline{η}_{z} ∈ \overline{𝔫}_{ℂ}^{∗} be defined by

η_{z}(u, 0, 0) := i(u|z),  u ∈ 𝔫_{ℂ},
\overline{η}_{z}(0, 0, v) := −i(v|z),  v ∈ \overline{𝔫}_{ℂ}.

Now fix λ ∈ W. In [40, Theorem 1] it is shown that there can only be nontrivial algebraic 𝔫-Whittaker vectors (resp. \overline{𝔫}-Whittaker vectors) of weight η ∈ 𝔫_{ℂ}^{∗} (resp. η ∈ \overline{𝔫}_{ℂ}^{∗}) for dπ_{λ} if η is contained in the associated variety of dπ_{λ} in 𝔤_{ℂ}^{∗}. The intersection of the associated variety of dπ_{λ} with 𝔫_{ℂ}^{∗} (resp. \overline{𝔫}_{ℂ}^{∗}) is equal to \overline{X_{λ}} (after identifying 𝔫_{ℂ}^{∗} and \overline{𝔫}_{ℂ}^{∗} with V_{ℂ}). Hence the existence of a non-trivial algebraic 𝔫-Whittaker vector (resp. \overline{𝔫}-Whittaker vector) of weight η_{z} (resp. \overline{η}_{z}) implies z ∈ \overline{X_{λ}}. We prove the converse of this statement:

Proposition 6.9. Let z ∈ \overline{X_{λ}}.

(1) The Dirac delta distribution δ_{λ,z} at z is contained in (W^{λ})^{∗} and defines an algebraic 𝔫-Whittaker vector of weight η_{z}.

(2) The distribution

ϕ_{λ,z}(x) := J_{λ}(z, x) dμ_{λ}(x),  x ∈ O_{λ},

is contained in (W^{λ})^{∗} and defines an algebraic \overline{𝔫}-Whittaker vector of weight \overline{η}_{z}.

(3) The algebraic Whittaker vectors δ_{λ,z} and ϕ_{λ,z} are related by

ϕ_{λ,z} = 2^{rλ}U_{λ}δ_{λ,z}.

Proof. (1) Since W^{λ} = P(O_{λ})e^{− tr(x)} and P(O_{λ}) ≅ ℂ[\overline{X_{λ}}] the point evaluation of every φ ∈ W^{λ} at z ∈ \overline{X_{λ}} is well-defined. Now dπ_{λ}(u, 0, 0) = i(u|x) and the Whittaker property follows.

(2) By Proposition 6.1 the function ϕ_{λ,z} belongs to (W^{λ})^{∗}. Since dπ_{λ}(0, 0, v) = i(v|B_{λ}) the Whittaker property follows from Proposition 3.6.

(3) By Theorem 6.3 we have for φ ∈ W^{λ}

⟨ϕ_{λ,z}, φ⟩ = ∫_{O_{λ}} J_{λ}(z, x)φ(x) dμ_{λ}(x) = 2^{rλ}U_{λ}φ(z)
= 2^{rλ}⟨δ_{λ,z}, U_{λ}φ⟩ = 2^{rλ}⟨U_{λ}δ_{λ,z}, φ⟩

since \overline{U_{λ}} = U_{λ}.

An interesting question is to determine the set of smooth Whittaker vectors for π_{λ}. We find that the algebraic 𝔫-Whittaker vector δ_{λ,z} (resp. the algebraic \overline{𝔫}-Whittaker vector ϕ_{λ,z}) is smooth if z ∈ \overline{O_{λ}}. For this we first prove estimates for δ_{λ,z} and ϕ_{λ,z} using the estimate for J_{λ}(z, w) by Nakahama [44] stated in Proposition 3.2.

Lemma 6.10. For each λ ∈ W and x ∈ \overline{O_{λ}} there exist elements X, Y ∈ U(𝔤) such that for all φ ∈ W^{λ} we have

|⟨δ_{λ,x}, φ⟩| ≤ ∥ dπ_{λ}(X)φ∥_{L^{2}(O_{λ}, dμ_{λ})},  (6.2)
|⟨ϕ_{λ,x}, φ⟩| ≤ ∥ dπ_{λ}(Y)φ∥_{L^{2}(O_{λ}, dμ_{λ})}.  (6.3)

Proof. We first prove (6.3). From Proposition 3.2 it follows that there exists constants C_{1} > 0 and k ∈ ℕ_{0} such that

|J_{λ}(x, y)| ≤ C_{1}(1 + |y|^{2})^{k}  ∀ y ∈ \overline{O_{λ}}.

Now there exists an N ∈ ℕ_{0} with

C_{2} := ∥(1 + | · |^{2})^{k−N}∥_{L^{2}(O_{λ}, dμ_{λ})} < ∞.

Hence we obtain for φ ∈ W^{λ}, using Hölder’s inequality:

|⟨ϕ_{λ,x}, φ⟩| ≤ C_{1} ∫_{O_{λ}} (1 + |y|^{2})^{k}|φ(y)| dμ_{λ}(y)
= C_{1} ∫_{O_{λ}} (1 + |y|^{2})^{k−N} · |(1 + |y|^{2})^{N}φ(y)| dμ_{λ}(y)
≤ C_{1}C_{2}∥(1 + | · |^{2})^{N}φ∥_{L^{2}(O_{λ}, dμ_{λ})}.

Since dπ_{λ}(U(𝔤)) contains all polynomials there exists X ∈ U(𝔤) such that

dπ_{λ}(X)φ(y) = C_{1}C_{2}(1 + |y|^{2})^{N}φ(y)

which shows (6.3). Now by Proposition 6.9 (3) we have

δ_{λ,x} = 2^{−rλ}U_{λ}ϕ_{λ,x}

and we obtain

|⟨δ_{λ,x}, φ⟩| = 2^{−rλ}|⟨U_{λ}ϕ_{λ,x}, φ⟩| = 2^{−rλ}|⟨ϕ_{λ,x}, U_{λ}φ⟩|
≤ 2^{−rλ}∥ dπ_{λ}(X)U_{λ}φ∥_{L^{2}(O_{λ}, dμ_{λ})}
= 2^{−rλ}∥ dπ_{λ}(X)π_{λ}(j̃)φ∥_{L^{2}(O_{λ}, dμ_{λ})}
= 2^{−rλ}∥π(j̃) dπ_{λ}(Ad(j̃^{−1})X)φ∥_{L^{2}(O_{λ}, dμ_{λ})}
= 2^{−rλ}∥ dπ_{λ}(Ad(j̃^{−1})X)φ∥_{L^{2}(O_{λ}, dμ_{λ})}.

Hence (6.2) follows with Y = 2^{−rλ} Ad(j̃^{−1})X.

Theorem 6.11. Let λ ∈ W. Then for every x ∈ \overline{O_{λ}} the algebraic Whittaker vectors δ_{λ,x} and ϕ_{λ,x} are smooth, i.e. δ_{λ,x}, ϕ_{λ,x} ∈ L^{2}(O_{λ}, dμ_{λ})^{−∞}.

Proof. Recall that for X ∈ U(𝔤) the map

L^{2}(O_{λ}, dμ_{λ}) → [0, ∞), φ ↦ ∥ dπ_{λ}(X)φ∥_{L^{2}(O_{λ}, dμ_{λ})}

is a continuous seminorm on L^{2}(O_{λ}, dμ_{λ})^{∞}. Since W^{λ} ⊆ L^{2}(O_{λ}, dμ_{λ})^{∞} is dense the claim now follows from (6.2) and (6.3)

Remark 6.12. For x ∈ O_{λ} it is easy to see that δ_{λ,x}, ϕ_{λ,x} ∈ L^{2}(O_{λ}, dμ_{λ})^{−∞}. In fact, by the Sobolev embedding theorem it follows that there is a continuous linear embedding L^{2}(O_{λ}, dμ_{λ})^{∞} ↪ C^{∞}(O_{λ}) and hence it is immediate that δ_{λ,x} ∈ L^{2}(O_{λ}, dμ_{λ})^{−∞}. Further ϕ_{λ,x} = 2^{rλ}U_{λ}δ_{λ,x} ∈ L^{2}(O_{λ}, dμ_{λ})^{−∞} as U_{λ} is an automorphism of L^{2}(O_{λ}, dμ_{λ})^{−∞}. However, it is a priori not clear that every function φ ∈ L^{2}(O_{λ}, dμ_{λ})^{∞} extends to \overline{O_{λ}} and that δ_{λ,x} is defined on L^{2}(O_{λ}, dμ_{λ})^{∞} for x ∈ ∂O_{λ}. In [37, Theorems 5.15, 5.18 & 5.19] the statements of Theorem 6.11 are shown for the special case V = ℝ, i.e. 𝔤 ≅ 𝔰𝔩(2, ℝ).

Remark 6.13. Note that since J_{λ}(0, y) = 1 for all y ∈ V we have ϕ_{λ,0} = 1 dμ_{λ}, the constant function on O_{λ} with value 1, and hence

U_{λ}δ_{λ,0} = 2^{rλ}1 dμ_{λ}.

This formula resembles the fact that the Euclidean Fourier transform maps the Dirac delta distribution at the origin to a constant function.

7 Application to branching problems

For 𝔤 = 𝔰𝔬(2, n) we consider the representation dπ_{λ}^{𝔰𝔬(2,n)} belonging to the minimal non-zero discrete Wallach point λ = \frac{d}{2} = \frac{n−2}{2}. We consider the restriction of dπ_{λ}^{𝔰𝔬(2,n)} to the symmetric subalgebra 𝔥 = 𝔰𝔬(2, m) ⊕ 𝔰𝔬(n − m) of 𝔤. Note that the pair (𝔤, 𝔥) is a symmetric pair of holomorphic type and hence the restriction of the lowest weight representation dπ_{λ}^{𝔰𝔬(2,n)} to 𝔥 is discretely decomposable with multiplicity one and the representations of 𝔥 appearing in the decomposition will again be lowest weight representations (see [32]). Consider the Jordan algebra V = ℝ^{1,n−1} which is the vector space ℝ × ℝ^{n−1} endowed with the multiplication

(x_{1}, x′) · (y_{1}, y′) := (x_{1}y_{1} + x′ · y′, x_{1}y′ + y_{1}x′)

for x_{1}, y_{1} ∈ ℝ, x′, y′ ∈ ℝ^{n−1}, where x′ · y′ denotes the standard inner product on ℝ^{n−1}. We have 𝔡𝔢𝔯(V) = 𝔰𝔬(n − 1), acting on the last n − 1 variables, and 𝔰𝔱𝔯(V) = 𝔰𝔬(1, n − 1) ⊕ ℝ. The conformal Lie algebra of V is given by 𝔤 = 𝔰𝔬(2, n). The subalgebra 𝔰𝔬(2, m), 1 ≤ m ≤ n, can be viewed as the conformal Lie algebra of the Jordan subalgebra U = {x ∈ V : x_{m+1} = … = x_{n} = 0} ≅ ℝ^{1,m−1}. Then 𝔰𝔬(2, m) consists of the elements (u, L(a)+D, v) with u, a, v ∈ U and D ∈ 𝔰𝔬(m − 1) acting on the coordinates x_{2}, …, x_{m}. The centralizer of 𝔰𝔬(2, m) in 𝔤 is 𝔰𝔬(n − m) acting linearly on the coordinates x_{m+1}, …, x_{n}. Hence 𝔰𝔬(n − m) ⊆ 𝔨^{𝔩} = 𝔰𝔬(n − 1). Finally let 𝔥 = 𝔰𝔬(2, m) ⊕ 𝔰𝔬(n − m) ⊆ 𝔤 be the corresponding subalgebra of 𝔤. To study the restriction dπ_{λ}^{𝔰𝔬(2,n)}|_{𝔥} we use the Fock model ( dρ_{λ}^{𝔰𝔬(2,n)}, F_{λ}) of the representation dπ_{λ}^{𝔰𝔬(2,n)}. In this case

X_{1}^{𝔰𝔬(2,n)} = {z ∈ ℂ^{n} : z_{1}^{2} − z_{2}^{2} − ⋯ − z_{n}^{2} = 0} \ {0},
X_{2}^{𝔰𝔬(2,n)} = {z ∈ ℂ^{n} : z_{1}^{2} − z_{2}^{2} − ⋯ − z_{n}^{2} ≠ 0},

realized in the ambient space V_{ℂ} = ℂ^{n}, and hence

P(X_{1}^{𝔰𝔬(2,n)}) = ℂ[Z_{1}, …, Z_{n}]/(Z_{1}^{2} − Z_{2}^{2} − ⋯ − Z_{n}^{2}),
P(X_{2}^{𝔰𝔬(2,n)}) = ℂ[Z_{1}, …, Z_{n}].

For arbitrary λ ∈ W the representation dρ_{λ}^{𝔰𝔬(2,n)} of 𝔤_{ℂ} acts by

dρ_{λ}(a, −2iL(a), a) = 2i ∑_{j=1}^{n} a_{j}z_{j},  dρ_{λ}(a, 0, −a) = 2i (∂_{az} + \frac{λ}{2} tr(a)) ,

dρ_{λ}(a, 2iL(a), a) = −2i ∑_{j=1}^{n} a_{j}B_{λ}^{n,j}, dρ_{λ}(0, D, 0) = −∂_{Dz},

where

B_{λ}^{n,j} = ε_{j}z_{j}□^{n} − 2(E^{n} + λ)\frac{∂}{∂z_{j}} ,  □^{n} = ∑_{j=1}^{n} ε_{j} \frac{∂^{2}}{∂z_{j}^{2}} ,

E^{n} = ∑_{j=1}^{n} z_{j} \frac{∂}{∂z_{j}} ,  ε_{j} = \begin{cases}+1 & for j = 1, \\ −1 & for 2 ≤ j ≤ n. \end{cases}

We first consider the action of the compact part 𝔰𝔬(n−m) which acts naturally on the coordinates z_{m+1}, …, z_{n}. Note that

ℂ[Z_{1}, …, Z_{n}] = ℂ[Z_{1}, …, Z_{m}] ⊗ ℂ[Z_{m+1}, …, Z_{n}].

Every polynomial in Z_{m+1}, …, Z_{n} can be written as a sum of spherical harmonics multiplied with powers of (Z_{m+1}^{2} + ⋯ + Z_{n}^{2}). In P(X_{1}^{𝔰𝔬(2,n)}) we have Z_{m+1}^{2} + ⋯ + Z_{n}^{2} = Z_{1}^{2} − Z_{2}^{2} − ⋯ − Z_{m}^{2} and hence

P(X_{1}^{𝔰𝔬(2,n)}) ≅ ℂ[Z_{1}, …, Z_{n}]/(Z_{1}^{2} − Z_{2}^{2} − ⋯ − Z_{n}^{2})

≅ ⨁_{k=0}^{∞} ℂ[Z_{1}, …, Z_{m}] ⊗ H^{k}(ℂ^{n−m}),

which gives the decomposition into irreducible 𝔰𝔬(n − m)-representations. Next we have to examine the action of 𝔰𝔬(2, m) on each of the summands. For this we use the notation V_{ℂ} = ℂ^{n} = ℂ^{m} ⊕ ℂ^{n−m} = U_{ℂ} ⊕ U_{ℂ}^{⊥}.

Lemma 7.1. For k ∈ ℕ_{0} and X ∈ 𝔰𝔬(2, m) we have

dρ_{λ}^{𝔰𝔬(2,n)}(X)|_{ℂ[Z_{1},...,Z_{m}]⊗H^{k}(ℂ^{n−m})} = dρ_{λ+k}^{𝔰𝔬(2,m)}(X) ⊗ id .

Proof. On ℂ[Z_{1}, …, Z_{m}] ⊗ H^{k}(ℂ^{n−m}) we check four exhaustive cases separately:

(1) X = (0, D, 0), D ∈ 𝔰𝔬(m − 1). The action is given by

dρ_{λ}^{𝔰𝔬(2,n)}(X) = −∂_{Dz} = dρ_{λ+k}^{𝔰𝔬(2,m)}(X) ⊗ id

since D only acts on the coordinates z_{2}, …, z_{m}.

(2) X = (a, −2iL(a), a), a ∈ U. Since a ∈ U ⊆ ℂ^{m} it is immediate that

dρ_{λ}^{𝔰𝔬(2,n)}(X) = 2i ∑_{j=1}^{m} a_{j}z_{j} = dρ_{λ+k}^{𝔰𝔬(2,m)}(X) ⊗ id .

(3) X = (a, 0, −a), a ∈ U. Assume first that a_{1} = 0. Then L(a) annihilates ℂ^{n−m} and hence

dρ_{λ}^{𝔰𝔬(2,n)}(X) = 2i∂_{az} = dρ_{λ+k}^{𝔰𝔬(2,m)}(X) ⊗ id .

For a = e we consider X = (e, 0, −e) which acts by

dρ_{λ}^{𝔰𝔬(2,n)}(X) = 2i(E^{n} + λ) = 2i(E^{m} + λ + k) = dρ_{λ+k}^{𝔰𝔬(2,m)}(X) ⊗ id

since E^{n−m} = ∑_{j=m+1}^{n} z_{j} \frac{∂}{∂z_{j}} acts on H^{k}(ℂ^{n−m}) by the scalar k.

(4) X = (a, 2iL(a), a), a ∈ U. It suffices to check the claimed formula for a = e_{j}, j = 1, …, m. We then have

dρ_{λ}^{𝔰𝔬(2,n)}(X) = −2i (ε_{j}z_{j}□^{n} − 2(E^{n} + λ)\frac{∂}{∂z_{j}} )

= −2i (ε_{j}z_{j}□^{m} − 2(E^{m} + λ + k)\frac{∂}{∂z_{j}} )

= dρ_{λ+k}^{𝔰𝔬(2,m)}(X) ⊗ id

since

□^{n}[f(z_{1}, …, z_{m})g(z_{m+1}, …, z_{n})] = □^{m}f(z_{1}, …, z_{m}) · g(z_{m+1}, …, z_{n})

+ f(z_{1}, …, z_{m}) · Δ^{n−m}g(z_{m+1}, …, z_{n}),

where Δ^{n−m} denotes the Laplacian on ℂ^{n−m} and E^{n−m} the Euler operator on ℂ^{n−m} which acts on H^{k}(ℂ^{n−m}) by the scalar k.

This shows the following theorem:

Theorem 7.2. For λ = \frac{n−2}{2} the minimal non-zero discrete Wallach point for 𝔰𝔬(2, n) we have

dπ_{λ}^{𝔰𝔬(2,n)} = ⨁_{k=0}^{∞} dπ_{λ+k}^{𝔰𝔬(2,m)} ⊠ H^{k}(ℝ^{n−m}).

For m < n we have λ + k = \frac{n+2k−2}{2} > \frac{m−2}{2} for k ∈ ℕ_{0} and hence the representations dπ_{λ+k}^{𝔰𝔬(2,m)} belong to Wallach points in the continuous part of the Wallach set. For 2k > 2m − n the representation dπ_{λ+k}^{𝔰𝔬(2,m)} are holomorphic discrete series. In particular, if 2m ≥ n then there occur representations in the branching law which are not holomorphic discrete series.

References

[1] D. Achab, Minimal representations of simple real Lie groups of Hermitian type - The Fock model, (2012), preprint, available at arXiv:1206.1737.

[2] D. Achab and J. Faraut, Analysis of the Brylinski-Kostant model for spherical minimal representations, Canad. J. Math. 64 (2012), no. 4, 721–754.

[3] G. E. Andrews, R. Askey, and R. Roy, Special functions, Encyclopedia of Mathematics and its Applications, vol. 71, Cambridge University Press, Cambridge, 1999.

[4] M. Aristidou, M. Davidson, and G. Ólafsson, Differential recursion relations for Laguerre functions on symmetric cones, Bull. Sci. Math. 130 (2006), no. 3, 246–263.

[5] S. Ben Saïd and B. Ørsted, Segal–Bargmann transforms associated with finite Coxeter groups, Math. Ann. 334 (2006), no. 2, 281–323.

[6] F. A. Berezin, Quantization in complex symmetric spaces, Izv. Akad. Nauk SSSR Ser. Mat. 39 (1975), no. 2, 363–402, 472.

[7] R. Brylinski and B. Kostant, Minimal representations, geometric quantization, and unitarity, Proc. Nat. Acad. Sci. U.S.A. 91 (1994), no. 13, 6026–6029.

[8] J.-L. Clerc, Fonctions K de Bessel pour les algèbres de Jordan, Harmonic analysis (Luxembourg, 1987), Lecture Notes in Math., vol. 1359, Springer, Berlin, 1988, pp. 122–134.

[9] M. Davidson, G. Ólafsson, and G. Zhang, Laplace and Segal–Bargmann transforms on Hermitian symmetric spaces and orthogonal polynomials, J. Funct. Anal. 204 (2003), no. 1, 157–195.

[10] H. Dib, Fonctions de Bessel sur une algèbre de Jordan, J. Math. Pures Appl. (9) 69 (1990), no. 4, 403–448.

[11] H. Ding and K. I. Gross, Operator-valued Bessel functions on Jordan algebras, J. Reine Angew. Math. 435 (1993), 157–196.

[12] H. Ding, K. I. Gross, R. A. Kunze, and D. St. P. Richards, Bessel functions on boundary orbits and singular holomorphic representations, The mathematical legacy of Harish-Chandra (Baltimore, MD, 1998), Proc. Sympos. Pure Math., vol. 68, Amer. Math. Soc., Providence, RI, 2000, pp. 223–254.

[13] T. Enright, R. Howe, and N. Wallach, A classification of unitary highest weight modules, Representation theory of reductive groups (Park City, Utah, 1982), Progr. Math., vol. 40, Birkhäuser Boston, Boston, MA, 1983, pp. 97–143.

[14] T. Enright and A. Joseph, An intrinsic analysis of unitarizable highest weight modules, Math. Ann. 288 (1990), no. 4, 571–594.

[15] J. Faraut and A. Korányi, Analysis on symmetric cones, The Clarendon Press, Oxford University Press, New York, 1994.

[16] J. Faraut and G. Travaglini, Bessel functions associated with representations of formally real Jordan algebras, J. Funct. Anal. 71 (1987), no. 1, 123–141.

[17] S. Gindikin and S. Kaneyuki, On the automorphism group of the generalized conformal structure of a symmetric R-space, Differential Geom. Appl. 8 (1998), no. 1, 21–33.

[18] I. S. Gradshteyn and I. M. Ryzhik, Table of integrals, series, and products, Academic Press, New York, 1965.

[19] M. Harris and H. P. Jakobsen, Singular holomorphic representations and singular modular forms, Math. Ann. 259 (1982), no. 2, 227–244.

[20] S. Helgason, Differential geometry and symmetric spaces, Pure and Applied Mathematics, Vol. XII, Academic Press, New York, 1962.

[21] C. S. Herz, Bessel functions of matrix argument, Ann. of Math. (2) 61 (1955), 474–523.

[22] J. Hilgert, T. Kobayashi, and J. Möllers, Minimal representations via Bessel operators, (2012), to appear in J. Math. Soc. Japan, available at arXiv:1106.3621.

[23] J. Hilgert, T. Kobayashi, J. Möllers, and B. Ørsted, Fock model and Segal– Bargmann transform for minimal representations of Hermitian Lie groups, J. Funct. Anal. 263 (2012), no. 11, 3492–3563.

[24] J. Hilgert and K.-H. Neeb, Vector valued Riesz distributions on Euclidian Jordan algebras, J. Geom. Anal. 11 (2001), no. 1, 43–75.

[25] J. Hilgert, K.-H. Neeb, and B. Ørsted, The geometry of nilpotent coadjoint orbits of convex type in Hermitian Lie algebras, J. Lie Theory 4 (1994), no. 2, 185–235.

[26] , Conal Heisenberg algebras and associated Hilbert spaces, J. Reine Angew. Math. 474 (1996), 67–112.

[27] , Unitary highest weight representations via the orbit method. I. The scalar case, Acta Appl. Math. 44 (1996), no. 1-2, 151–184, Representations of Lie groups, Lie algebras and their quantum analogues.

[28] J. Hilgert and G. Zhang, Segal–Bargmann and Weyl transforms on compact Lie groups, Monatsh. Math. 158 (2009), no. 3, 285–305.

[29] H. P. Jakobsen, Hermitian symmetric spaces and their unitary highest weight modules, J. Funct. Anal. 52 (1983), no. 3, 385–412.

[30] A. Joseph, Annihilators and associated varieties of unitary highest weight modules, Ann. Sci. École Norm. Sup. (4) 25 (1992), no. 1, 1–45.

[31] S. Kaneyuki, The Sylvester’s law of inertia in simple graded Lie algebras, J. Math. Soc. Japan 50 (1998), no. 3, 593–614.

[32] T. Kobayashi, Multiplicity-free theorems of the restrictions of unitary highest weight modules with respect to reductive symmetric pairs, Representation theory and automorphic forms, Progr. Math., vol. 255, Birkhäuser Boston, Boston, MA, 2008, pp. 45–109.

[33] T. Kobayashi and G. Mano, The inversion formula and holomorphic extension of the minimal representation of the conformal group, Harmonic analysis, group representations, automorphic forms and invariant theory, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ. Singap., vol. 12, World Sci. Publ., Hackensack, NJ, 2007, pp. 151–208.

[34] , The Schrödinger model for the minimal representation of the indefinite orthogonal group O(p, q), Memoirs of the AMS 213 (2011), no. 1000.

[35] T. Kobayashi and B. Ørsted, Analysis on the minimal representation of O(p, q). II. Branching laws, Adv. Math. 180 (2003), no. 2, 513–550.

[36] T. Kobayashi and Y. Oshima, Classification of symmetric pairs with discretely decomposable restrictions of (𝔤, k)-modules, (2012), preprint, available at arXiv:1202.5743.

[37] B. Kostant, On Laguerre polynomials, Bessel functions, Hankel transform and a series in the unitary dual of the simply-connected covering group of Sl(2, R), Represent. Theory 4 (2000), 181–224 (electronic).

[38] T. Levasseur, S. P. Smith, and J. T. Stafford, The minimal nilpotent orbit, the Joseph ideal, and differential operators, J. Algebra 116 (1988), no. 2, 480–501.

[39] T. Levasseur and J. T. Stafford, Rings of differential operators on classical rings of invariants, Mem. Amer. Math. Soc. 81 (1989), no. 412.

[40] H. Matumoto, Whittaker vectors and associated varieties, Invent. Math. 89 (1987), no. 1, 219–224.

[41] J. Möllers, Minimal representations of conformal groups and generalized Laguerre functions, Ph.D. thesis, University of Paderborn, 2010, available at arXiv:1009.4549.

[42] J. Möllers and Y. Oshima, Branching laws for minimal holomorphic representations, in preparation.

[43] J. Möllers and B. Schwarz, Bessel operators on Jordan pairs and L^{2}-models for small representations, in preparation.

[44] R. Nakahama, Integral formula and upper estimate of I- and J-Bessel functions on Jordan algebras, (2012), preprint, available at arXiv:1211.4702.

[45] K.-H. Neeb, Holomorphy and convexity in Lie theory, de Gruyter Expositions in Mathematics, vol. 28, Walter de Gruyter & Co., Berlin, 2000.

[46] G. Ólafsson and B. Ørsted, Generalizations of the Bargmann transform, Lie theory and its applications in physics (Clausthal, 1995), World Sci. Publ., River Edge, NJ, 1996, pp. 3–14.

[47] J. Sekiguchi, Remarks on real nilpotent orbits of a symmetric pair, J. Math. Soc. Japan 39 (1987), no. 1, 127–138.

[48] M. Vergne and H. Rossi, Analytic continuation of the holomorphic discrete series of a semi-simple Lie group, Acta Math. 136 (1976), no. 1-2, 1–59.

[49] D. A. Vogan, Jr., Gel′fand-Kirillov dimension for Harish-Chandra modules, Invent. Math. 48 (1978), no. 1, 75–98.

[50] , Associated varieties and unipotent representations, Harmonic analysis on reductive groups (Brunswick, ME, 1989), Progr. Math., vol. 101, Birkhäuser Boston, Boston, MA, 1991, pp. 315–388.

[51] N. R. Wallach, The analytic continuation of the discrete series. I, II, Trans. Amer. Math. Soc. 251 (1979), 1–17, 19–37.

Jan Möllers

Institut for Matematiske Fag Aarhus Universitet Ny Munkegade 118 8000 Aarhus C, Denmark moellers@imf.au.dk 856

Documenta Mathematica 18 (2013)