Causality and directed graphical models

Lecture 1 — Causality and Graphical Models

Christophe Ambroise

M2 Data Science — Université d’Évry

Course overview

Six sessions

#	Lecture	Tutorial
1	Causality, DAGs, factorisation	Simpson, Gaussian Bayesian networks
2	d-separation, Markov equivalence	d-separation, sprinkler network
3	Markov random fields, Gaussian graphical models	Ising, graphical lasso
4	Interventions, SCMs, identification	adjustment, Mendelian randomisation
5	Structure learning: scores, PC	PC by hand, PC on cytometry data
6	LiNGAM, OT-LiNGAM	causal orders on simulated and RNA-editing data

References

• K. Murphy, Machine Learning: a Probabilistic Perspective (2012), ch. 10, 19, 26
• J. Pearl, M. Glymour, N. Jewell, Causal Inference in Statistics: a Primer (2016)
• J. Peters, D. Janzing, B. Schölkopf, Elements of Causal Inference (2017)
• B. Neal, Introduction to Causal Inference, course notes (2020)
• S. Shimizu, Statistical Causal Discovery: LiNGAM Approach (2022)

Why causality

Cause and effect

Cause: that which produces or modifies an effect.

• Aristotle (384-322 BC): knowledge of a thing = grasp of its why
• Hume (1711-1776): Causality stems from habit or mental expectation, built over repeated experience
• Science: temporal order + mechanism, e.g. \(\dfrac{dY}{dt} = f(X, Y, t)\)

Three statistical views of causality

• Dependence: \(P(Y \mid X = x) \neq P(Y)\)
• Prediction (Granger): past of \(X\) → better forecast of \(Y\)
• Intervention (Pearl): forced change of \(X\) → change in \(Y\)

Intervention view \(X\) causes \(Y\) when the distribution of \(Y\) under a forced value of \(X\) depends on that value.

\[ \exists\, x \neq x' : \quad P\big(Y \mid do(X = x)\big) \neq P\big(Y \mid do(X = x')\big) \]

Ladder of causation

Rung	Activity	Typical question	Quantity
1. Association	seeing	patients with \(X\): recovery?	\(P(y \mid x)\)
2. Intervention	doing	recovery if \(X\) imposed?	\(P(y \mid do(x))\)
3. Counterfactual	imagining	this patient, without \(X\)?	\(P(y_{x'} \mid x, y)\)

Each rung: more assumptions, more structure.

Pearl & Mackenzie, The Book of Why (2018).

Simpson’s paradox: kidney stones

Stone size	Treatment	Recovered	Total	Rate
Small	New	81	87	93 %
Small	Standard	234	270	87 %
Large	New	192	263	73 %
Large	Standard	55	80	69 %
All	New	273	350	78 %
All	Standard	289	350	83 %

New drug: better in each stratum, worse overall.

Charig et al. (1986), BMJ.

Mechanism of the reversal

• New drug: mostly large stones (263 / 350)
• Large stones: harder to treat, whatever the drug
• Aggregate rate = mixture with different weights

\[ \begin{aligned} P(R \mid T = \text{new}) &= \textstyle\sum_s P(R \mid T = \text{new}, s)\, P(s \mid T = \text{new}) \\ &\neq \textstyle\sum_s P(R \mid T = \text{new}, s)\, P(s) \end{aligned} \]

Causal graph of the paradox

• \(S\) (stone size): common cause of \(T\) and \(R\) → confounder
• Right comparison: within strata of \(S\)

Same table, another graph

• \(B\): post-treatment variable (e.g. blood pressure), mediator
• Stratification on \(B\): part of the effect removed → aggregate rate correct

Lesson Same numbers, opposite conclusions: the decision depends on the causal graph, not on the data alone.

Directed graphical models

Joint distribution: size problem

• Many correlated variables: genes, pixels, words
• Goal: compact representation of \(p(x_1, \dots, x_V)\)

Chain rule, any ordering of the variables:

\[ p(x_{1:V}) = p(x_1)\, p(x_2 \mid x_1)\, p(x_3 \mid x_1, x_2) \cdots p(x_V \mid x_{1:V-1}) \]

\(K\) states per variable → \(O(K^V)\) parameters.

Conditional independence

\(X\) is conditionally independent of \(Y\) given \(Z\) when, once \(Z\) is known, \(Y\) brings no further information on \(X\).

\[ X \perp Y \mid Z \iff p(x, y \mid z) = p(x \mid z)\, p(y \mid z) \quad \forall\, x, y, z \text{ with } p(z) > 0 \]

Conditional independence: card example

• Full deck: suit \(\perp\) value
• Diamond face cards removed: suit \(\not\perp\) value
• Given “face card”: independence restored

\[ \begin{aligned} P(\text{King} \mid F) &= 1/3, \quad P(\clubsuit \mid F) = 1/3 \\ P(\text{King}\clubsuit \mid F) &= 1/9 \end{aligned} \]

Markov chain: first simplification

Assumption: \(x_{t+1} \perp x_{1:t-1} \mid x_t\).

\[ p(x_{1:V}) = p(x_1) \prod_{t=2}^{V} p(x_t \mid x_{t-1}) \]

Parameters: \((K-1) + K(K-1)\) instead of \(K^V - 1\).

Graphical model

• Node = random variable
• Missing edge = conditional independence assumption
• Graph types: directed (DAG), undirected, mixed

Graphical model Joint distribution in factorised form, the factorisation being encoded by a graph.

Graph vocabulary

Term	Definition
parents \(pa(s)\)	\(\{t : t \to s\}\)
children \(ch(s)\)	\(\{t : s \to t\}\)
ancestors \(an(s)\)	nodes with a directed path to \(s\)
descendants \(de(s)\)	nodes reached from \(s\) by a directed path
non-descendants \(nd(s)\)	\(V \setminus (de(s) \cup \{s\})\)
root / leaf	no parent / no child

Directed acyclic graph

A DAG is a directed graph without directed cycle; its nodes then admit a topological order, parents before children.

\[ G \text{ acyclic} \iff \exists\, \pi \text{ permutation of } V : \quad s \to t \in E \Rightarrow \pi(s) < \pi(t) \]

• Order not unique in general
• Construction: repeated removal of roots

DAG factorisation

A distribution factorises according to a DAG \(G\) when each variable depends only on its parents.

\[ p(x_1, \dots, x_V) = \prod_{t=1}^{V} p\big(x_t \mid x_{pa(t)}\big) \]

Bayesian network DAG \(G\) + one conditional distribution \(p(x_t \mid x_{pa(t)})\) per node (CPD).

Ordered Markov property

Equivalent to the factorisation: given its parents, each node is independent of its other predecessors in a topological order.

\[ X_t \perp X_{pred(t) \setminus pa(t)} \mid X_{pa(t)} \]

Proof: chain rule in topological order, then simplification of each factor.

Parameter count: example

Binary variables:

Factor	Parameters
\(p(a)\), \(p(b)\)	\(1 + 1\)
\(p(c \mid a, b)\)	\(4\)
\(p(d \mid c)\), \(p(e \mid c)\)	\(2 + 2\)
DAG	10
full joint	\(2^5 - 1 = 31\)

Classical examples

Naive Bayes classifier

\[ p(y, x) = p(y) \prod_{j=1}^{p} p(x_j \mid y) \]

Features: independent given the class.

Markov chain and hidden Markov model

\[ p(z_{1:T}, x_{1:T}) = p(z_1) \prod_{t=2}^{T} p(z_t \mid z_{t-1}) \prod_{t=1}^{T} p(x_t \mid z_t) \]

• \(Z_t\): hidden state (gene region type, phoneme)
• \(X_t\): observation (nucleotide, acoustic signal)

Gaussian Bayesian network

Each variable is a linear function of its parents plus an independent Gaussian noise.

\[ p(x_t \mid x_{pa(t)}) = \mathcal N\big(x_t \;\big|\; \mu_t + \textstyle\sum_{s \in pa(t)} w_{ts}(x_s - \mu_s),\ \sigma_t^2\big) \]

Equivalent structural equations, \(z_t \sim \mathcal N(0, 1)\) independent:

\[ x_t = \mu_t + \textstyle\sum_{s \in pa(t)} w_{ts}(x_s - \mu_s) + \sigma_t z_t \]

Linear structural equation model

Matrix form, \(W = (w_{ts})\), \(S = \operatorname{diag}(\sigma_t)\), \(e = S z\):

\[ x - \mu = W (x - \mu) + e \]

• Topological order → \(W\) strictly lower triangular → \(I - W\) invertible
• \(x - \mu = (I - W)^{-1} e\): linear mixture of independent noises

Covariance of a Gaussian Bayesian network \[ \Sigma = (I - W)^{-1}\, S^2\, (I - W)^{-\top}, \qquad x \sim \mathcal N(\mu, \Sigma) \]

Simulation check

import numpy as np
rng = np.random.default_rng(0)
W = np.array([[0, 0, 0],      # x1 = e1
              [0.8, 0, 0],    # x2 = 0.8 x1 + e2
              [0, -0.5, 0]])  # x3 = -0.5 x2 + e3
S = np.diag([1.0, 0.6, 0.3])
U = np.linalg.inv(np.eye(3) - W)
Sigma = U @ S**2 @ U.T
X = (U @ S @ rng.standard_normal((3, 100_000))).T
print(np.round(Sigma, 3)); print(np.round(np.cov(X.T), 3))

[[ 1.    0.8  -0.4 ]
 [ 0.8   1.   -0.5 ]
 [-0.4  -0.5   0.34]]
[[ 1.     0.801 -0.401]
 [ 0.801  1.004 -0.502]
 [-0.401 -0.502  0.341]]

Two DAGs, one covariance

• \(X_2 = w X_1 + e_2\) and \(X_1 = w' X_2 + e_1'\): both Gaussian, same \(\Sigma\) for suitable \(w'\)
• Gaussian observational data: direction not recoverable

Identifiability DAG factorisation: a statistical model; the arrow direction is not always identifiable from \(p\) alone.