Bayesian Density Estimation

Jayanta K. Ghosh

Abstract. This is a brief exposition of posterior consistency issues in Bayesian nonparametrics especially in the context of Bayesian Density estimation,

1991 Mathematics Subject Classification: 62A15,62G99. Keywords and Phrases: Dirichlet mixtures,density estimation

1 Introduction

We describe popular methods of Bayesian density estimation and explore sufficient conditions for the posterior given data to converge to a true underlying distribution P_{0} as the data size increases. One of the advantages of Bayesian density estimates is that,unlike classical frequentist methods,choice of the right amount of smoothing is not such a serious problem.

Section 2 provides a general background to infinite dimensional problems of inference such as Bayesian nonparametrics, semiparametrics and density estimation. Bayesian nonparametrics has been around for about twenty five years but the other two areas,specially the last, is of more recent vintage. Section 3 indicates in broad terms why different tools are needed for these three different problems and then Section 4 focuses on our main problem of interest ,namely,positive posterior consistency results for Bayesian density estimation.

2 Background

Let X_{1}, X_{2}, …, X_{n} be i.i.d. random variables with unknown common probability measure P on (R, B), where R is the real line and B the Borel σ− field. Typically P lies in some given set of probability measures P. In Bayesian analysis, a statistician puts a probability measure Π on P equipped with a suitable σ− field B_{P} and assumes that the unknown P is distributed over P according to Π and, given P, X_{1}, X_{2}, …, X_{n} are i.i.d. with common distribution P. This completely specifies the joint distribution of the random P and the random Xs. Hence, in principle one can calculate the conditional probability Π(B|X_{1}, X_{2}, …, X_{n}) of P lying in some subset B. This is the posterior in distinction with Π(B) which is the prior probability of B. Consistency of posterior to be defined below is a sort of partial validation of this method of analysis. We now define posterior consistency at P_{0}. Suppose unknown to the Bayesian statistician,X_{1}, X_{2}, …, X_{n} are i.i.d. ∼ P_{0}, where P_{0} is a given element of P and not random. Suppose that P is also equipped with a topology and the topology and B_{P} are compatible in the sense that the neighborhoods B of P_{0} are B_{P} measurable.

Definition: Π(. |X_{1}, X_{2}, …, X_{n}) is consistent at P_{0} if for all neighborhoods B of P_{0}, as n → ∞,

Π(B|X_{1}, X_{2}, …, X_{n}) → 1 a.s P_{0}

This property depends on both Π and P_{0}. It would be desirable to have this property at various P_{0}’s that seem plausible to the Bayesian who is using this posterior.

An old result of Doob shows that such a property holds for all but a π−null set of P_{0}’s. Unfortunately, this result is too weak to settle whether consistency holds for a particular P_{0}. It is well known that this property holds for a wide class of priors and all P_{0}’s if P is finite dimensional,e.g., when P is the set of all normal distributions N(μ, σ^{2}) with mean μ and variance σ^{2}, −∞ < μ < ∞, σ^{2} > 0. In contrast the answer is usually no when P is infinite dimensional as in density estimation.

There are three broad classes of infinite dimensional problems —(fully) nonparametric inference like making inference about an unknown distribution function, a semiparametric problem like estimating the point of symmetry of an unknown symmetrical distribution function, and density estimation. The set P is different for these three cases. In the first case,which is classical, P is the class of all probability measures on (R, B). In the third case and, in fact also in the second, we work instead with the set of probability measures P on (R, B) which have a density f with respect to the Lebesgue measure. In the first two problems the set P is equipped with the weak topology and the natural tools are the use of tail free priors or a theorem of Schwartz(1965). In the third case the natural topology is that induced by the L_{1} or the Hellinger metric. The natural tool is a new theorem that makes use of the notion of metric entropy or packing numbers for the space of densities in addition to one of Schwartz’s conditions.

3 Notations and other technicalities

3.1 Nonparametrics

We start with the nonparametric problem. Let P be the class of all probability measures on (R, B); P be equipped with the weak topology and B_{P} the corresponding Borel σ− field. Equivalently, B_{P} is the smallest σ− field which makes the evaluation maps P ↦ P (A) measurable for each A in B.

The most popular prior on (P, B_{P}) is the Dirichlet process due to Ferguson(1973,1974). It is specified by its finite dimensional distributions as follows. Let α be a finite non zero measure on (R, B). Let A_{1}, A_{2}, …, A_{k} form a measurable partition. Then P (A_{1}), P (A_{2}), …, P (A_{k}) have a finite dimensional Dirichlet distribution with parameters α(A_{1}), α(A_{2}), …, α(A_{k}). If α(A_{i}) > 0, i = 1, 2, …, k then this distribution has a density with respect (k−1) dimensional Lebesgue measure that has the form

\frac{Γ(R)}{∏_{1}^{k} Γ(α(A_{i}))} ∏_{1}^{k} p_{i}^{α(A_{i})−1}, 0 < p_{i}, ∑_{1}^{k} p_{i} = 1

If k = 2, one gets the beta distribution. Integrating out p_{1} one gets

E(P (A_{i})) = α(A_{i})/α(R) = ᾱ(A_{i}).  (1)

It can be shown that the posterior given X_{1}, X_{2}, …, X_{n} is again a Dirichlet with α + ∑_{1}^{n} δ_{X_{i}}, in place of α, where δ_{X_{i}} is the point mass at X_{i}. Using this fact and (1), one gets immediately,

E (P (A)|X_{1}, X_{2}, …, X_{n}) = \frac{α(R)}{α(R) + n}ᾱ(A) + \frac{n}{α(R) + n} (\frac{1}{n} ∑ δ_{X_{i}}(A)) (2)

which is a convex combination of the prior guess ᾱ(A) and the frequentist nonparametric maximum likelihood estimate P_{n}(A) = \frac{1}{n} ∑ δ_{X_{i}}(A). The weights reflect the Bayesian’s confidence in prior guess. One can elicit or choose ᾱ(.) and α(R) — and hence α(.)— from these considerations.

We denote the Dirichlet process by D_{α}.

Proposition. If Π is D_{α} and B is a weak neighborhood of true P_{0}, then Π (B|X_{1}, X_{2}, …, X_{n}) → 1 a.s. (P_{0}), i.e., posterior consistency holds for all P_{0}.

At the heart of this fact is the property of being tailfree,vide Ferguson(1974), which allows one to reduce an infinite dimensional problem to a finite dimensional problem and invoke posterior consistency for the latter. This idea as well as the introduction of Dirichlet for another infinite dimensional problem goes back to Freedman(1963).

3.2 Semiparametrics

We start with a famous example of Diaconis and Freedman(1986). Suppose we wish to make inference about θ and P_{θ}(.) = P (. − θ) where θ is real and P (.) is symmetric around zero. To put a prior distribution for P_{θ} one first chooses a P′ using a D_{α}, symmetrizes P′ to get P and independently chooses θ. Diaconis and Freedman(1986) show that the posterior for θ need not be consistent in the weak topology.

Various people have observed that semiparametrics should involve probability measures with densities but the Dirichlet assigns probability one to the set of discrete measures. However choosing priors on densities is not enough.

Ghosal, Ghosh and Ramamoorthi(1998) have pointed out that one may argue that the Diaconis–Freedman counter example occurs because of the breakdown of the tailfree property. They show that posterior consistency can be proved provided a condition used by Schwartz(1965) holds. Priors for which posterior consistency holds are exhibited in Ghosal, Ghosh and Ramamoorthi(1998).

The version of Schwartz’s(1965) theorem one has to use for this purpose is given below. We now work with P = the set of probability measures P having a density f with respect to Lebesgue measure. For two such probability measures P_{1}, P_{2}, with densities f_{1}, f_{2} the Kullback–Leibler number K(P_{1}, P_{2}) is defined as ∫_{R} f_{1} log \frac{f_{1}}{f_{2}} dx.

K(P_{1}, P_{2}) is always ≥ 0 and may be ∞. It is not a metric but measures the divergence between P_{1} and P_{2} with the extreme tail of the density playing an important role.

Theorem 1 Suppose P_{0} belongs to the Kullback–Leibler support of Π,i.e., for all δ > 0,

Π{K(P_{0}, P) < δ} > 0  (3)

Then Π (B|X_{1}, X_{2}, …, X_{n}) → 1 a.s. (P_{0}), for all weak neighborhoods B of P_{0}.

As Ghosal, Ghosh and Ramamoorthi(1998) show property(3)—unlike the tailfree property — continues to hold even with the addition of a finite dimensional parameter.

For later reference as well as completeness we record Schwartz’s(1965) theorem in its original form and an extension due to Barron(1988,1998).

Theorem 2 Let Π be a prior on P, and P_{0} ∈ B. Assume the following conditions:

1. Π(K(P_{0}, P) < δ) > 0 for all δ > 0;

2. There exists a uniformly consistent sequence of tests for testing H_{0} : P = P_{0} vs. H_{1} : P ∈ B^{c}, i.e., there exists a sequence of tests ϕ_{n}(X_{1}, X_{2}, …, X_{n}) such that as n → ∞,

E_{P_{0}}ϕ_{n}(X_{1}, X_{2}, …, X_{n}) → 0 and inf_{P ∈B^{c}} E_{P} ϕ_{n}(X_{1}, X_{2}, …, X_{n}) → 1.

Then Π(B|X_{1}, X_{2}, …, X_{n})→ 1 a.s. P_{0}.

Theorem 3 ((Barron(1988,1998))) Let Π be a prior on P, and P_{0} be in P and B be a neighborhood of P_{0}. Assume that Π(K(P_{0}, P) < δ) > 0 for all ϵ > 0. Then the following are equivalent.

1. There exists a β_{0} such that

P_{0}{Π(B^{c}|X_{1}, X_{2}, …, X_{n}) > e^{−nβ_{0}} infinitely often} = 0;

2. There exist subsets V_{n}, W_{n} of P, positive numbers c_{1}, c_{2}, β_{1}, β_{2} and a sequence of tests {ϕ_{n}(X_{1}, X_{2}, …, X_{n})} such that

(a) B^{c} = V_{n} ∪ W_{n},

(b) Π(W_{n}) ≤ C_{1}e^{−nβ_{1}},

(c) P_{0}{ϕ_{n}(X_{1}, X_{2}, …, X_{n}) > 0 infinitely often} = 0 and inf_{P ∈V_{n}} E_{P} ϕ_{n} ≥ 1 − c_{2}e^{−nβ_{2}}.

4 Density estimation

4.1 Dirichlet mixture of Normals

We illustrate with what seems to be currently the most popular and successful Bayesian method, first proposed by Lo(1984) and implemented in the early nineties via Markov Chain Monte Carlo(1994) by Escobar, Mueller and West (94).

Choose a random P′ ∼ D_{α}. Since P′ is discrete, as observed before,form a convolution with a normal density N(0, h). Let P = P′ ∗ N(0, h).

Since the smoothness of P depends on h and one does not know how much smoothness is right, put a prior(usually, inverse gamma)on h also. This completes the specification of a prior,which is often called a Dirichlet mixture of normal. It turns out that for MCMC to be feasible one needs α also to be normal. Simulations and heuristic calculations show that one can improve the rate of convergence by adding a location and scale parameter to α and by putting a prior on these parameters also. The following discussion can handle these refinements as well as general nonnormal α. However for the normal α,one can supplement the discussion below with non trivial heuristic argument that throws light on how convergence takes place. For lack of space the heuristic argument will not be given.

4.2 Posterior consistency for general priors

The basic theorem is the following which improves on an earlier result of Barron,Schervish and Wasserman(1997).

Let P_{0} ⊂ P. For δ > 0, the L_{1}− metric entropy of P_{0}, denoted by J(δ, P_{0}) is log a(δ), where a(δ) is the minimum over all k such that there exist P_{1}, P_{2}, ⋯, P_{k} in P with P_{′} ⊂ ∪_{1}^{k}{P : ∥P − P_{i}∥_{1} < δ}.

Theorem 4 (Ghosal,Ghosh and Ramamoorthi) Let Π be a prior on P. If P_{0} ∈ P and Π(K(P_{0}, P) < ϵ) > 0 for all ϵ > 0. If for each ϵ > 0 there is a δ < ϵ, c_{1}, c_{2} > 0, β < \frac{ϵ^{2}}{2} and also P_{n} such that

1. Π(P_{n}^{c}) < C_{1}e^{−nβ_{1}} for large n

2. J(δ, P_{n}) < nβ

then Π(B|X_{1}, X_{2}, …, X_{n})→ 1 a.s.P_{0}n for all L_{1}-neighborhoods B of P_{0}.

The proof of this theorem is based on the result of Barron recorded in Section 3. The first assumption is the condition assumed in Theorem 1 in Section 3 while the two remaining assumptions take care of conditions(2) and (3) of Barron’s Theorem.

4.3 Application to Dirichlet mixture of normals

One has to have two sets of tools to verify the two conditions in Theorem 4. The set or sieve P_{n} for verifying the condition is: fix a δ and β as in the theorem then

P_{n} = \{P = P′ ∗ N(0, h); P′[−\sqrt{n}, \sqrt{n}] > 1 − δ, h > \frac{c(δ, β)}{\sqrt{n}}\}

Various sufficient conditions which entail application of Theorem 4 are given in Ghosal, Ghosh and Ramamoorthi(97. For example if P_{0} is smooth unimodal with finite Shannon entropy and compact support, like the uniform on [a, b] then P_{0} belongs to the Kullback–Leibler support of the prior. For unbounded support the tails of P_{0} and ᾱ have to be compatible in a certain way.

4.4 Concluding Remarks

Theorem 4 can also be used to study posterior consistency for Gaussian process priors and Bayesian histograms (Barron(1988,1998) and Ghosh and Ramamoorthi(1998)).

One may also ask whether the Bayes estimate E(P |X_{1}, X_{2}, …, X_{n}) is consistent. It is easy to show that posterior consistency in the weak topology or the topology induced by L_{1} norm implies Bayes consistency.

One may also ask questions about rates of convergence and non-informative or default priors which attain a minimax rate of convergence for the posterior or Bayes estimates. This issue is currently under investigation by Ghosal,Ghosh and van der Vaart and by Wassserman and Shen.

A final important remark. In recent work Barron(1998) shows if we focus on the cumulative Kullback-Leibler predictive loss (also called the entropy loss) an elegant consistency theory can be built up using only Kullback-Leibler support.

References

Barron, A. R. (1986). On uniformly consistent tests and Bayes consistency. Unpublished manuscript.

Barron, A. R., Schervish, M. and Wasserman, L. (1996). The consistency of posterior distributions in non parametric problems. Preprint.

Barron, A. R. (1998). Information-theoretic characterizations of Bayes Performance and the Choice of Priors in Parametric and Nonparametric Problems.Bayesian Statistics 6, Editors J.M. Bernardo,J.O. Berger, A.P. Dawid and A.F.M. Smith Oxford University press

Diaconis, P. and Freedman, D. (1986a). On the consistency of Bayes estimates (with discussion). Ann. Statist. 14 1–67.

Ferguson, T. (1973). A Bayesian analysis of some nonparametric problems.Ann. Statist. 1 209-230.

Ferguson, T. (1974). Prior distribution on the spaces of probability measures. Ann. Statist. 2 615-629.

Freedman, D. (1963). On the asymptotic distribution of Bayes estimates in the discrete case I. Ann. Math. Statist. 34 1386–1403.

Ghosal, S., Ghosh, J. K. and Ramamoorthi, R. V. (1998). Consistent semiparametric estimation about a location parameter. Submitted.

Ghosal, S., Ghosh, J. K. and Ramamoorthi, R. V. (1997b). Posterior consistency of Dirichlet mixtures in density estimation. Technical report # WS-490, Vrije Universiteit, Amsterdam.

S.Ghosal, J.K. Ghosh and R.V. Ramamoorthi (1997) Consistency issues in Bayesian Nonparametrics . Asymptotics, Nonparametrics and Time series - a tribute to M.L. Puri Edited by S.Ghosh.Marcel Dekker

Ghosh, J. K. and Ramamoorthi R. V. (1998). Lecture notes on Bayesian asymptotics. Under preparation.

Lo, A. Y. (1984). On a class of Bayesian nonparametric estimates I: Density estimates. Ann. Statist. 12 351–357.

Schwartz, L. (1965). On Bayes procedures. Z. Wahrsch. Verw. Gebiete 4 10–26.

West, M., Muller, P. and Escobar, M. D. (1994). Hierarchical priors and mixture models, with applications in regression and density estimation. In Aspects of uncertainty: A Tribute to D. V. Lindley. 363–386.

Jayanta K. Ghosh

Indian Statistical Institute 203 B.T. Road Calcutta 700 035 India 244

Documenta Mathematica · Extra Volume ICM 1998 · III ·