Seminar Sophus Lie 3 (1993) 27–34

Holomorphic extension of unitary representations

Karl-Hermann Neeb

Introduction

Let H be a complex Hilbert space and π: G → U(H) a continuous unitary representation of the Lie group G on H. In this note we will discuss the problem of extending π holomorphically to a complex manifold which carries the structure of a complex semigroup and which contains G as its group of units in its boundary.

Example 1. A classical example for such a construction arises when G is a compact Lie group and π is a finite dimensional unitary representation of G on a complex Hilbert space. Then there exists a universal complexification of G, i.e., an embedding of G into a complex Lie group G_{C} with the property that every Lie group homomorphism of G into a complex group extends to G_{C}. It follows in particular that every unitary representation π: G → U(n) ⊆ Gl(n,C) extends to a holomorphic representation of G_{C}.

We will see in the following that this example is untypical in the sense that one cannot hope in general that the complex manifold which serves as the domain of the holomorphic extension is a group.

The starting point of the modern theory of holomorphic extensions of unitary representations was Ol’shanskiĭ’s observation that, if W is a pointed generating invariant cone in a simple Lie algebra 𝔤, G a corresponding linear connected group, and G_{C} its universal complexification, then the set S_{W} = G exp(iW) is a closed subsemigroup of G_{C} ([14]). This theorem has been generalized by Hilgert and ’Olafsson to solvable groups ([5]) and the most general result of this type, due to Lawson ([7]), is that if G_{C} is a complex Lie group with an antiholomorphic involution inducing the complex conjugation on 𝔤_{C} = L(G_{C}), then the set S_{W} = G exp(iW) is a closed subsemigroup of G_{C}. The class of semigroups obtained by this construction is not sufficient for many applications in representation theory. For instance Howe’s oscillator semigroup (cf. [6]) is a 2-fold covering of such a semigroup, but it does not fit into any group.

In [9] we have shown that given a Lie algebra 𝔤, a generating invariant convex cone W ⊆ 𝔤, and a discrete central subgroup D of the simply connected group corresponding to the Lie algebra 𝔤 + i(W ∩ (−W)) which is invariant under complex conjugation, there exists a semigroup S = Γ(𝔤, W, D) called the Ol’shanskiĭ semigroup defined by this data. This semigroup is the quotient S̃/D, where S̃ is the universal covering semigroup of S (cf. [4, Ch. 3]) and D ≅ π_{1}(S) is a discrete central subgroup of S̃. Moreover, the semigroup S̃, also denoted Γ(𝔤, W) can be obtained as the universal covering semigroup of the subsemigroup ⟨exp(𝔤 + iW)⟩ of the simply connected complex Lie group G_{C} with Lie algebra 𝔤_{C}. We note that every Ol’shanskiĭ semigroup S contains a dense semigroup ideal int(S) which is a complex manifold and the restriction of the multiplication to int(S) is holomorphic.

These results provide the domains for the holomorphic extensions. We note that if W = 𝔤 is a Lie algebra, then Γ(𝔤, W) is the simply connected complex Lie group with Lie algebra 𝔤_{C}.

A holomorphic representation of a complex Ol’shanskiĭ semigroup S is a weakly continuous monoid morphism π: S → B(H) into the algebra of bounded operators on a Hilbert space H such that π is holomorphic on the interior int(S) of S. Now one can think of representations of S as holomorphic extensions of unitary representations of the subgroup U(S) = {s ∈ S: s^{∗}s = 1} of unitary elements in S.

I. Extension of general representations

One of the key examples of an Ol’shanskiĭ semigroup is the upper half plane C^{+} := {z ∈C: Im z ≥ 0}. With the parameters introduced above, we have C^{+} = Γ(IR, IR^{+}).

The following theorem is crucial for the whole theory of holomorphic extensions.

Theorem I.1. (Classification of the holomorphic representations of C^{+} ) If (π, H) is a holomorphic representation of C^{+} and A the self-adjoint operator on H with π(t) = e^{−itA} for all t ∈ IR, then A is bounded from above, and, conversely, if A is a self-adjoint operator bounded from above, then the unitary one-parameter group t ↦ e^{−itA} extends to a holomorphic representation of C^{+} with ∥e^{−izA}∥ = e^{Im zγ}, where γ = sup Spec(A).

Proof. [9] and [4, Ch. 9].

Let (π, H) be a holomorphic representation of G which extends holomorphically to the Ol’shanskiĭ semigroup S = Γ(𝔤, W, D). Then one important consequence of the preceding theorem is that all the self-adjoint operators idπ(X), X ∈ W must be bounded from above. A closer inspection of the situation shows that this boundedness is uniform in the sense that there exists a norm ∥ · ∥ on 𝔤 such that sup Spec (idπ(X)) ≤ ∥X∥. (cf. Lemma III.12 in [9]).

Now we consider the cone

W_{π} := {(X, t) ∈ 𝔤 ⊕ IR: sup Spec (idπ(X)) ≤ t}.

Let H^{∞} denote the space of smooth vectors for G in H, i.e., the set of all v ∈ H such that the mapping G → H, g ↦ π(g).v is smooth. Then H^{∞} lies in the common domain of all the operators idπ(X), X ∈ 𝔤 and therefore the condition sup Spec (idπ(X)) ≤ t means that ⟨idπ(X).v, v⟩ ≤ t∥v∥^{2} for all v ∈ H^{∞}. From this observation it follows easily that W_{π} is a generating closed convex cone in 𝔤^{♯} := 𝔤 ⊕ IR which is invariant under the adjoint action.

To get a better understanding for the meaning of W_{π}, let G^{♯} := G × S^{1} , where S^{1} := IR/ZZ. Then we can extend the unitary representation π to the representation

π^{♯}: G^{♯} → U(H), (g, t + ZZ) ↦ e^{it}π(g).

Now π^{♯} is a continuous unitary representation and

W_{π} = {X^{♯} ∈ 𝔤^{♯}: dπ^{♯}(X^{♯}) ≤ 0}.

Note that

ker dπ^{♯} = L(ker π^{♯}) = H(W_{π}) := W_{π} ∩ −W_{π}

([4, Lemma 9.14]).

With these remarks in mind, let (π, H) be a holomorphic representation of an Ol’shanskiĭ semigroup S = Γ(𝔤, W, D), considered as an extension of the restriction to G := U(S)_{0} . Suppose that π |_{G} has discrete kernel. Then either π^{♯} has discrete kernel or 𝔤 ⊕ ker dπ^{♯} = 𝔤^{♯}. In the first case the cone W_{π} is pointed and in the second case W_{π} ∩ 𝔤 is a pointed generating cone and (W_{π} ∩ 𝔤) ⊕ IR^{+} is pointed and generating in 𝔤^{♯}.

This observation has two consequences.

Proposition I.3. If (π, H) is a continuous unitary representation with discrete kernel which extends to an Ol’shanskiĭ semigroup S = Γ(𝔤, W, D), then 𝔤^{♯} contains a pointed generating invariant cone and 𝔥 := H(W) := W ∩ −W is a compact Lie algebra.

Proof. The first assertion follows from the discussion above and for the second assertion we refer to Lemma III.7 in [9].

We note that the condition that the Lie algebra 𝔤^{♯} contains a pointed generating invariant cone has significant consequences for the structure of 𝔤 and 𝔤^{♯} respectively. In the following we call a Lie algebra 𝔤 admissible if 𝔤^{♯} contains pointed generating invariant cones. For the structure theory of such Lie algebras we refer to Chapters II and III in [8].

Corollary I.4. If a continuous unitary representation (π, H) with discrete kernel extends to a holomorphic representation of a complex group G_{C}, then 𝔤 is a compact Lie algebra.

Proof. We apply Proposition I.3 with G_{C} = Γ(𝔤, 𝔤).

The following theorem is the main result on holomorphic extensions of general representations. We note that if H(W) is a compact Lie algebra, then the results from Section I in [9] show in particular that G̃ ≅ U(S̃) holds for the simply connected covering group of G. Hence π_{1}(G) which we consider as a subgroup of G̃ is a central subgroup of S̃ and therefore G ≅ U(S)_{0} holds for S = Γ(𝔤, W, π_{1}(G)). This means that G can be realized as the connected component of the group of unitary elements in S.

Theorem I.5. (The Holomorphic Extension Theorem) Let G be a connected Lie group and (π, H) a unitary representation of G. Then π extends to a holomorphic representation of the Ol’shanskiĭ semigroup S = Γ(𝔤, W, π_{1}(G)) if and only if H(W) is a compact Lie algebra and there exists a norm ∥ · ∥ on 𝔤 such that

sup Spec (idπ(X)) ≤ ∥X∥ ∀X ∈ W.

Proof. For the case where W is pointed and S is represented by contractions, this result is contained in [4, Ch. 9]. The proof in full generality, i.e., the extension to general representations and general Ol’shanskiĭ semigroups is contained in Section III of [9].

A holomorphic representation (π, H) of an Ol’shanskiĭ semigroup S is called a contraction representation if π(S) ⊆ C(H) := {A ∈ B(H): ∥A∥ ≤ 1}. We formulate the version of Theorem I.5 for contraction representations which actually is more a part of the proof than a corollary.

Corollary I.6. Let G be a connected Lie group and (π, H) a unitary representation of G. Then π extends to a holomorphic contraction representation of the Ol’shanskiĭ semigroup S = Γ(𝔤, W, π_{1}(G)) if and only if H(W) ⊆ L(ker π) and all the operators idπ(X), X ∈ W are negative.

In this form (for pointed cones and Hermitean simple groups) the holomorphic extension technique was used by Ol’shanskiĭ in [14]. The case of solvable groups is due to Hilgert and ’Olafsson ([5]). The case of simple Lie groups has also been considered in [15]. For the case of the metaplectic representation we refer to [6], [1], and [2].

II. Irreducible representations

From now on we assume that 𝔤 is a (CA) Lie algebra, i.e., the group of inner automorphisms of 𝔤 is closed in the group Aut(𝔤) of all automorphisms of 𝔤. As we have seen in [9], this condition is a rather natural one since it entails that every connected group G with L(G) = 𝔤 is a type I group.

We say that a subalgebra 𝔞 ⊆ 𝔤 is compactly embedded if the group generated by e^{ad 𝔞} has compact closure in Aut(𝔤). We assume that 𝔤 contains a compactly embedded Cartan algebra 𝔱. Associated to the Cartan subalgebra 𝔱_{C} in the complexification 𝔤_{C} is a root decomposition as follows ([4, Ch. 7]). For a linear functional λ ∈ 𝔱C^{∗} we set

𝔤C^{λ} := {X ∈ 𝔤_{C} : (∀Y ∈ 𝔱_{C})[Y, X] = λ(Y)X}

and

Δ := Δ(𝔤_{C}, 𝔱_{C}) := {λ ∈ 𝔱C^{∗} \ {0} : 𝔤C^{λ} ≠ {0}}.

Then

𝔤_{C} = 𝔱_{C} ⊕ ⨁_{λ∈Δ} 𝔤C^{λ},

λ(𝔱) ⊆ iIR for all λ ∈ Δ and σ(𝔤C^{λ}) = \overline{𝔤C^{λ}} = 𝔤C^{−λ}, where σ(X) = \overline{X} denotes complex conjugation on 𝔤_{C} with respect to 𝔤. Let 𝔨 ⊇ 𝔱 denote a maximal compactly embedded subalgebra. Then a root is said to be compact if 𝔤C^{λ} ⊆ 𝔨_{C}. We write Δ_{k} for the set of compact roots and Δ_{p} for the set of non-compact roots. The Lie algebra 𝔤 is said to have cone potential if [X_{α}, \overline{X}_{α}] ≠ 0 holds for all non-zero elements X_{α} in 𝔤C^{α} for all α ∈ Δ.

Definition II.1. A subset Δ^{+} ⊆ Δ is called a positive system if there exists X_{0} ∈ i𝔱 such that

Δ^{+} = {λ ∈ Δ : λ(X_{0}) > 0}.

A positive system is said to be 𝔨-adapted if

λ(X_{0}) > μ(X_{0}) ∀μ ∈ Δ_{k}, λ ∈ Δ_{p}^{+}.

Let Δ^{+} ⊆ Δ be a positive system of roots. For a subset M of a vector space V we write cone(M) for the smallest closed convex cone containing M and for a cone C in V the set C^{⋆} := {ν ∈ V^{∗}: ν(C) ⊆ IR^{+}} is called the dual cone.

We define the cone

C_{min} := C_{min}(Δ^{+}) := cone\{i[\overline{X}, X] : X ∈ 𝔤C^{λ}, λ ∈ Δ_{p}^{+}\} ⊆ 𝔱.

Definition II.2. Let Δ^{+} ⊆ Δ be a positive system.

(a) We set 𝔟 := 𝔟(Δ^{+}) := 𝔱_{C} ⊕ ⨁_{α∈Δ^{+}} 𝔤C^{α}.

(b) Let V be a 𝔤_{C}-module and v ∈ V . We say that v is a primitive element of V (with respect to Δ^{+} ) if v ≠ 0 and 𝔟.v ⊆C.v.

(c) For a 𝔤-module V and λ ∈ 𝔱C^{∗} we set

V^{λ} := {v ∈ V : (∀X ∈ 𝔱_{C})X.v = λ(X)v}.

This space is called the weight space of weight λ and λ is called a weight of V if V^{λ} ≠ {0}. We write P_{V} for the set of weights of V .

(d) A 𝔤_{C}-module V is called a highest weight module with highest weight λ (with respect to Δ^{+} ) if it is generated by a primitive element of weight λ.

(e) A highest weight module V is said to be unitarizable if there exists a unitary representation (π, H) of the simply connected Lie group G with L(G) = 𝔤 such that V is isomorphic to the 𝔤_{C}-module H^{K} of K -finite vectors in H (cf. [10, Sect. III]), where 𝔨 is a maximal compactly embedded subalgebra containing 𝔱 and K = exp 𝔨 is the corresponding subgroup of G. In this case (π, H) is called a highest weight representation of G.

For more details on highest weight representations we refer to Section II in [10].

Theorem II.3. Let S = Γ(𝔤, W, D) be an Ol’shanskiĭ semigroup, 𝔤 a (CA) Lie algebra containing a compactly embedded Cartan algebra, and (π, H) an irreducible holomorphic representation. Then the following assertions hold:

(i) H^{K} is an irreducible highest weight module of the Lie algebra 𝔤_{C}. (ii) For every s ∈ int(S) the operator π(s) is a trace class operator, i.e.,

π(int S) ⊆ B_{1}(H).

Proof. This is Theorem III.8 in [10], one of the main results of this paper.

We note that one can use the second part of Theorem II.3 to obtain a rather satisfactory desintegration and character theory for Ol’shanskiĭ semigroups (Section IV in [10]).

The best known examples for representations which fit into this theory are the irreducible representations of compact Lie groups, the holomorphic discrete series representations of simple Hermitean Lie groups, the metaplectic representation of the 2-fold cover H_{n} × Mp(n, IR) of H_{n} × Sp(n, IR), where H_{n} denotes the (2n + 1)-dimensional Heisenberg group, and the oscillator representation of the (2n + 2)-dimensional oscillator group. Other examples are the ladder representations of the subgroups of Mp(n, IR) obtained by restriction of tensor products of the metaplectic representations.

The classification

Theorem II.3 raises the problem to determine for a given Ol’shanskiĭ semigroup S the set Ŝ of all irreducible representations, i.e., the set of all those highest weight modules of 𝔤_{C} which are unitarizable. For the classification of all unitarizable highest weight representations, or at least for the reduction to the case of simple Hermitean Lie algebras which is due to Enright, Howe and Wallach ([3]), we refer to Section III in [11]. Modulo this classification, the following theorem yields a classification of the irreducible holomorphic representations of a given Ol’shanskiĭ semigroup S.

Theorem II.4. (Characterization of the extendable highest weight representations) Let 𝔤 be a Lie algebra with cone potential, G a corresponding connected Lie group, and (π, H) an irreducible highest weight representation of G with highest weight iω ∈ i int C_{min}^{⋆} with respect to the 𝔨-adapted positive system Δ^{+} . Further let W ⊆ 𝔤 be a generating invariant wedge with H(W) compact and S := Γ(𝔤, W, π_{1}(G)). Then π extends to a holomorphic representation of S if and only if

W ∩ 𝔱 ⊆ (iΔ_{p})^{⋆} = {X ∈ 𝔱: (∀α ∈ Δ_{p}^{+}) iα(X) ≥ 0}.

Proof. [11, Theorem I.26]. The proof of this result rests on such techniques as the moment mapping for unitary representations (cf. [13]) and the convexity theorem for coadjoint orbits (cf. [12]).

The Gelfand-Raïkov Theorem

We conclude this survey with two results on the existence of sufficiently many holomorphic representations for a given Ol’shanskiĭ semigroup S.

Theorem II.5. (Gelfand-Raïkov-Theorem for Ol’shanskiĭ semigroups) Let S = Γ(𝔤, W, D) be an Ol’shanskiĭ semigroup. Then the irreducible holomorphic representations of S separate the points of S if and only if H(W) is a compact Lie algebra and 𝔤 is admissible, i.e., if 𝔤 ⊕ IR contains pointed generating invariant cones. In this case S has an injective holomorphic representation. Proof. Theorem IV.12 in [11].

Theorem II.5. (Gelfand-Raïkov-Theorem for Contraction Representations) Let S = Γ (𝔤, W, π_{1}(G)) be an Ol’shanskiĭ semigroup with G = U(S)_{0} . Then S has an injective holomorphic contraction representation, and the irreducible holomorphic contraction representations separate the points. Proof. Theorem IV.12 in [11]

References

[1] Brunet, M., and P. Kramer, Complex extensions of the representation of the symplectic group associated with the canonical commutation relations, Reports on Math. Physics 17(1980), 205–215.

[2] Brunet, M., The metaplectic semigroup and related topics, Reports on Math. Physics 22(1985), 149–170.

[3] Enright, T. J., R. Howe, and N. Wallach, A classification of unitary highest weight modules, Proc. “Representation theory of reductive groups” (Park City, UT, 1982), pp. 97-149; Progr. Math. 40(1983), 97 –143.

[4] Hilgert, J., and K. – H. Neeb, “Basic theory of Lie semigroups and applications”, Lecture Notes in Math., to appear.

[5] Hilgert, J., and G. Olafsson, Analytic continuations of representations, the solvable case, Jap. Journal of Math., to appear.

[6] Howe, R., The Oscillator semigroup, in “The Mathematical Heritage of Hermann Weyl”, Proc. Symp. Pure Math. 48, R. O. Wells Ed., AMS Providence, 1988.

[7] Lawson, J.D., Polar and Ol’shanskiĭ decompositions, Seminar Sophus Lie 1:2(1991), 163–173.

[8] Neeb, K. – H., “Invariant subsemigroups of Lie groups”, Memoirs of the AMS, to appear.

[9] —, Holomorphic representation theory I, Preprint Nr. 1536, Technische Hochschule Darmstadt, Feb. 1993.

[10] —, Holomorphic representation theory II, Preprint Nr. 1537, Technische Hochschule Darmstadt, Feb. 1993.

[11]  —, Holomorphic representation theory III, submitted.

[12] —, Kähler structures and convexity properties of coadjoint orbits, submitted.

[13] —, On the convexity of the moment mapping for unitary highest weight representations, submitted.

[14] Ol’shanskiĭ, G. I., Invariant cones in Lie algebras, Lie semigroups, and the holomorphic discrete series, Funct. Anal. and Appl. 15 (1982), 275– 285.

[15] Stanton, R. J., Analytic Extension of the holomorphic discrete series, Amer. J. Math. 108(1986), 1411–1424.

Fachbereich Mathematik Technische Hochschule Darmstadt Schloßgartenstraße 7 D-6100 Darmstadt, Germany neeb@mathematik.th-darmstadt.de

Received April 5, 1993