Inequalities

We can use integrability to bound the probability of “tail events” (values far from the mean). These inequalities form the basis for proving laws of large numbers.

General case

Theorem: Generalized Chebyshev's Inequality

Statement

Let $X$ be a random variable and $g: \mathbb{R} \to [0, \infty)$ be a non-negative function. For any set $B \in \mathcal{B}$, let $L$ be the lower bound of $g$ on $B$:

$$L = \inf \{ g(x) : x \in B \}$$

Then:

$$\mathbb{P}(X \in B) \le \frac{\mathbb{E}[g(X)]}{L}$$

Proof

1. Construct a Lower Bound Function Define a new random variable $Y$ that is a scaled indicator of the set $B$:

   $$Y = L \cdot \mathbb{1}_{\{X \in B\}}$$

2. Compare with $g(X)$ We compare $Y$ and $g(X)$ pointwise:

   • If $\omega \in \{X \in B\}$, then $X(\omega) \in B$, so $g(X(\omega)) \ge L$ (by definition of $L$). Also $Y(\omega) = L$. Thus $Y \le g(X)$.
   • If $\omega \notin \{X \in B\}$, then $Y(\omega) = 0$. Since $g$ is non-negative, $Y \le g(X)$ holds trivialy.

   Therefore, $Y \le g(X)$ almost surely.

3. Monotonicity of Expectation Taking expectations on both sides:

   $$\mathbb{E}[Y] \le \mathbb{E}[g(X)]$$

   Calculating $\mathbb{E}[Y]$:

   $$\mathbb{E}[Y] = \mathbb{E}[L \cdot \mathbb{1}_{\{X \in B\}}] = L \cdot \mathbb{P}(X \in B)$$

   Substituting this back:

   $$L \cdot \mathbb{P}(X \in B) \le \mathbb{E}[g(X)] \implies \mathbb{P}(X \in B) \le \frac{\mathbb{E}[g(X)]}{L}$$

The whole argument in one picture: the step $L \cdot \mathbb{1}_B$ never rises above $g(x)$, touching it only at the boundary of $B$. Integrating this pointwise domination against the law of $X$ turns area into probability and gives $\mathbb{P}(X \in B) \le \mathbb{E}[g(X)] / L$. Markov’s inequality is the special case $g(x) = x$, $B = \{x \ge a\}$, where $L = a$.

Important special cases

Markov’s Inequality

Theorem: Markov's Inequality

Statement

If $X$ is a non-negative random variable ($X \ge 0$), then for any $a > 0$:

$$\mathbb{P}(X \ge a) \le \frac{\mathbb{E}[X]}{a}$$

Proof

This follows from the general theorem by setting:

1. $g(x) = x$ (valid since $X \ge 0$)
2. $B = [a, \infty)$
3. Then $L = \inf_{x \ge a} x = a$.

Chebyshev’s Inequality

Theorem: Chebyshev's Inequality

Statement

Let $X$ be a random variable with finite variance. Then for any $a > 0$:

$$\mathbb{P}(|X - \mu| \ge a) \le \frac{\text{Var}(X)}{a^2}$$

where $\mu = \mathbb{E}[X]$.

Proof

This follows by setting:

1. Centered variable: Consider $Y = X - \mu$.
2. $g(y) = y^2$ (always non-negative).
3. $B = \{ y : |y| \ge a \} = (-\infty, -a] \cup [a, \infty)$.
4. Then $L = \inf \{ y^2 : |y| \ge a \} = a^2$.
5. $\mathbb{E}[g(Y)] = \mathbb{E}[(X-\mu)^2] = \text{Var}(X)$.

Alternatively, for the raw second moment:

$$\mathbb{P}(|X| \ge a) \le \frac{\mathbb{E}[X^2]}{a^2}$$

Chebyshev is the general bound with $g(x) = (x - \mu)^2$ and $B = \{|x - \mu| \ge a\}$. The parabola $(x-\mu)^2/a^2$ sits above the indicator of the tail event, equal to it precisely at $\mu \pm a$. Integrating turns the squared deviation into variance: $\mathbb{P}(|X - \mu| \ge a) \le \mathrm{Var}(X)/a^2$.

Jensen’s Inequality

The previous inequalities bound tail probabilities. The next one is of a different kind: it relates the expectation of a convex transformation to the transformation of the expectation. It is the workhorse behind moment comparisons and the contraction properties of averaging operators.

For a two-point variable taking $x_1$ and $x_2$ with equal weight, $\mathbb{E}[X]$ is the midpoint and $\mathbb{E}[\varphi(X)]$ is the chord’s height there. Convexity keeps the chord above the curve, so $\varphi(\mathbb{E}[X])$ (on the curve) sits below $\mathbb{E}[\varphi(X)]$ (on the chord). The vertical gap is the slack in the inequality, and it closes only when $\varphi$ is affine on the range of $X$.

Theorem: Jensen's Inequality

Statement

Let $\varphi : \mathbb{R} \to \mathbb{R}$ be a convex function, and let $X$ be an integrable random variable ($\mathbb{E}|X| < \infty$). Then

$$\varphi\!\big(\mathbb{E}[X]\big) \;\le\; \mathbb{E}\!\big[\varphi(X)\big],$$

where the right side is well-defined in $(-\infty, +\infty]$.

If $\varphi$ is concave, the inequality reverses. (A familiar instance: $\mathbb{E}[\log X] \le \log \mathbb{E}[X]$.)

Proof

The proof rests on a single geometric fact about convex functions: at any point, the graph lies above one of its tangent lines.

1. A supporting line at the mean. Write $m = \mathbb{E}[X]$. A convex function on $\mathbb{R}$ has, at every point $m$, a left derivative $\varphi'_-(m)$ and a right derivative $\varphi'_+(m)$ with $\varphi'_-(m) \le \varphi'_+(m)$. For any slope $a \in [\varphi'_-(m), \varphi'_+(m)]$, the affine function $\ell(x) = \varphi(m) + a(x - m)$ is a supporting line:

   $$\varphi(x) \;\ge\; \varphi(m) + a(x - m) \qquad \text{for every } x \in \mathbb{R}.$$

2. Substitute $X$ and take expectations. The supporting-line bound holds pointwise with $x = X(\omega)$:

   $$\varphi(X) \;\ge\; \varphi(m) + a(X - m).$$

   By monotonicity and linearity of expectation,

   $$\mathbb{E}\!\big[\varphi(X)\big] \;\ge\; \varphi(m) + a\big(\mathbb{E}[X] - m\big).$$

3. The correction term vanishes. By construction $\mathbb{E}[X] = m$, so $a(\mathbb{E}[X] - m) = 0$, leaving

   $$\mathbb{E}\!\big[\varphi(X)\big] \;\ge\; \varphi(m) \;=\; \varphi\!\big(\mathbb{E}[X]\big).$$

A single supporting line suffices here because the mean $\mathbb{E}[X]$ is a fixed number, so we only need the line tangent at that one point. When the deterministic mean is replaced by the random variable $\mathbb{E}(X \mid \mathcal{G})$, no single line works for all outcomes at once, and the argument has to invoke a whole countable family of supporting lines simultaneously. That refinement is carried out in the conditional version of Jensen’s inequality.