Home / Semester 4 / Behavioural Finance / The Challenge of the Behaviouralist: Robert Shiller Excess Volatility, Crash of 1987, and Adaptive Markets Hypothesis

Units:Unit 1 Unit 2 Unit 3 Unit 4 Unit 5

AKTU MBA Semester 4 • BMB FM 04 Unit 1: Foundations of Behavioral Finance Est. Reading Time: 10 Mins

The Challenge of the Behaviouralist: Robert Shiller Excess Volatility, Crash of 1987, and Adaptive Markets Hypothesis

Q: What is Robert Shiller's Excess Volatility argument?

Robert Shiller showed that stock market volatility far exceeds the volatility of discounted future dividends, proving market prices are driven by investor psychology.

Q: State the Shiller Variance Bound inequality mathematically.

The inequality is Var(P_t) <= Var(P_t^*), indicating that observed stock price variance cannot exceed the variance of perfect-foresight fundamental dividend values.

Q: Define the Adaptive Markets Hypothesis.

Proposed by Andrew Lo, AMH uses evolutionary biology principles to explain that market efficiency is dynamic and driven by human competition and adaptation.

MAuthored by mbanotes.in Academic Editorial Team

Verified Syllabus Alignment: AKTU MBA CoreTarget Score: 10/10 Marks (Section B/C)

⚡ 30-Second Revision Snapshot: The Behaviouralist Challenge AKTU MBA BMB FM 04 | Unit 1 Core Definition Empirical and theoretical cracks in EMH: Shiller’s excess volatility variance bounds, the 1987 market crash anomaly, limits to arbitrage, and Andrew Lo’s evolutionary synthesis. Shiller Volatility Bound Proving mathematically that stock market price volatility mathrmVar(P) drastically exceeds the variance of discounted subsequent dividends mathrmVar(P^*). Black Monday 1987 The US stock market fell 22.6% in a single day without any significant news, delivering a fatal blow to strict informational efficiency. Fama-Thaler Debate The theoretical duel between efficient market defenders (risk-based factor explanations) and behavioralists (psychological mispricing). Adaptive Markets Andrew Lo’s evolutionary synthesis: markets fluctuate between efficiency and inefficiency based on biological competition and adaptation.

Behavioralist Challenge Shiller Excess Volatility Equity Premium Puzzle and Adaptive Markets Hypothesis Infographic

Figure 1.3: Empirical Anomalies – Shiller Excess Volatility, Crash Contagion, Equity Premium Puzzle, and Paradigmatic Synthesis.

1. The Behavioralist Critique: Empirical Anomalies Shattering EMH

During the 1980s, the neoclassical assertion that market prices equal intrinsic fundamental value faced severe empirical challenges. Behavioral economists gathered undeniable statistical evidence of persistent market anomalies:

Excess Volatility: Market prices oscillate wildly between euphoric booms and depressive busts, far exceeding any rational revision in future corporate cash flows.
Predictable Long-Term Reversals: De Bondt and Thaler (1985) demonstrated that portfolios of past 3-year “loser” stocks systematically outperformed past “winner” stocks by 25% over the subsequent 3 years due to investor overreaction.
Siamese Twin Equities Disconnect: Royal Dutch and Shell traded at persistent 35% pricing disparities relative to their contractual 60:40 profit-sharing ratio, proving that arbitrage is incapable of enforcing the Law of One Price.

2. Robert Shiller’s Excess Volatility & Variance Bound Proof

In his seminal 1981 paper “Do Stock Prices Move Too Much to be Justified by Subsequent Changes in Dividends?”, Nobel laureate Robert J. Shiller formulated a rigorous mathematical test of the Present Value Dividend Discount Model.

Under EMH, the actual market price $P_t$ is the rational mathematical expectation of the ex-post fundamental price $P_t^*$, which represents the discounted present value of actual subsequent real dividends paid in the future:

\[ P_t = E_t[P_t^*] \implies P_t^* = P_t + u_t \]

Where $u_t$ is the rational forecast error. Crucially, under the rationality axiom of EMH, the forecast error $u_t$ must be uncorrelated with the forecast $P_t$ ($mathrm{Cov}(P_t, u_t) = 0$). Taking variances of both sides yields the celebrated Shiller Variance Bound:

\[ \mathrm{Var}(P_t^*) = \mathrm{Var}(P_t + u_t) = \mathrm{Var}(P_t) + \mathrm{Var}(u_t) \implies \mathrm{Var}(P_t) \le \mathrm{Var}(P_t^*) \]

The standard deviation of actual market stock prices must be strictly less than or equal to the standard deviation of fundamental present value dividends. However, when Shiller plotted 100 years of US equity data (S&P 500 from 1871 to 1979), he found that actual price volatility was 5 to 13 times higher than the fundamental dividend variance bound! This mathematically proved that market prices are driven by psychological sentiment, social contagion, and speculative fads.

3. The Great Synthesis: Andrew Lo’s Adaptive Markets Hypothesis

To reconcile the fierce intellectual warfare between Eugene Fama’s EMH and Richard Thaler’s Behavioral Finance, MIT professor Andrew Lo introduced the Adaptive Markets Hypothesis (AMH) in 2004. Grounded in evolutionary biology, sociobiology, and cognitive neuroscience, AMH posits that:

Investors are neither fully rational nor entirely irrational; they are biological organisms that deploy behavioral heuristics that adapt over time through competition, natural selection, and trial-and-error learning.
Market efficiency is not an absolute, static state (efficient vs inefficient). Instead, efficiency is a highly dynamic quality that varies across time, market sectors, and macroeconomic environments.
Profit opportunities (arbitrage alpha) continually emerge as market ecologies shift. As capital floods into an anomaly, the excess return is eroded until a new evolutionary shock (regime change) creates fresh inefficiencies.

4. Mathematical Model, Shiller Variance Bound & Worked Numerical Problem

We now formulate Shiller’s ex-post rational price model and calculate the variance bounds using a multi-period dividend discounting scenario.

Mathematical Formulation: Shiller Ex-Post Fundamental Price & Variance Bound

\[ P_t^* = \sum_{k=1}^T \frac{D_{t+k}}{(1 + r)^k} + \frac{P_{t+T}}{(1 + r)^T} \] \[ \mathrm{Var}(P_t) \le \mathrm{Var}(P_t^*) \iff \sigma(P_t) \le \sigma(P_t^*) \]

Where:

$P_t^*$: Ex-post rational fundamental stock price calculated with perfect dividend foresight.
$P_t$: Observed actual market price established by trading volume.
$D_{t+k}$: Real dividends realized in period $t+k$.
$r$: Constant real discount rate.
$\mathrm{Var}(P_t)$: Variance of observed market stock prices.
$\mathrm{Var}(P_t^*)$: Variance of ex-post fundamental discounted dividend stream.

Step-by-Step Worked Numerical Problem: Testing Shiller Volatility Bounds

Problem Statement: An empirical finance researcher analyzes a 4-year historical dataset for a blue-chip corporate equity index with a constant discount rate $r = 10.0\%$. The observed market prices ($P_t$) and actual subsequent dividend payments ($D_t$) are recorded below (with terminal price $P_4^* = \text{INR } 1,000$):

Year ($t$)	Observed Market Price ($P_t$)	Actual Dividend Paid ($D_t$)
Year 1	INR 1,250	INR 50
Year 2	INR 1,480	INR 55
Year 3	INR 850	INR 58
Year 4	INR 1,020	INR 60

Required:

Calculate the sample mean ($ar{P}$) and sample standard deviation $sigma(P_t)$ of observed market prices.
Calculate the ex-post rational fundamental prices ($P_t^*$) for Years 1, 2, and 3 using backward induction discounting: $P_t^* = rac{D_{t+1} + P_{t+1}^*}{1 + r}$.
Compute the sample standard deviation of fundamental prices $sigma(P_t^*)$.
Determine whether the Shiller Variance Bound ($sigma(P_t) le sigma(P_t^*)$) is satisfied or violated.

Step 1: Compute Sample Mean & Standard Deviation of Market Prices ($P_t$)

\[ \bar{P} = \frac{1250 + 1480 + 850 + 1020}{4} = \frac{4600}{4} = \text{INR } 1,150.00 \] \[ (1250 – 1150)^2 = 10,000; \quad (1480 – 1150)^2 = 108,900; \] \[ (850 – 1150)^2 = 90,000; \quad (1020 – 1150)^2 = 16,900 \] \[ \mathrm{Sum\ of\ Squared\ Deviations} = 10,000 + 108,900 + 90,000 + 16,900 = 225,800 \] \[ \mathrm{Var}(P_t) = \frac{225,800}{4 – 1} = \frac{225,800}{3} = 75,266.67 \implies \sigma(P_t) = \sqrt{75,266.67} = \text{INR } 274.35 \]

Step 2: Calculate Ex-Post Fundamental Prices ($P_t^*$) by Backward Induction

Given terminal fundamental price $P_4^* = \text{INR } 1,000$ and $r = 0.10$:

Year 3 ($P_3^*$): \[ P_3^* = \frac{D_4 + P_4^*}{1 + r} = \frac{60 + 1000}{1.10} = \frac{1060}{1.10} = \text{INR } 963.64 \]
Year 2 ($P_2^*$): \[ P_2^* = \frac{D_3 + P_3^*}{1 + r} = \frac{58 + 963.64}{1.10} = \frac{1021.64}{1.10} = \text{INR } 928.76 \]
Year 1 ($P_1^*$): \[ P_1^* = \frac{D_2 + P_2^*}{1 + r} = \frac{55 + 928.76}{1.10} = \frac{983.76}{1.10} = \text{INR } 894.33 \]

Fundamental Series ($P_t^*$): Year 1 = 894.33; Year 2 = 928.76; Year 3 = 963.64; Year 4 = 1,000.00.

Step 3: Compute Sample Standard Deviation of Fundamental Prices $sigma(P_t^*)$

\[ \bar{P}^* = \frac{894.33 + 928.76 + 963.64 + 1000.00}{4} = \frac{3786.73}{4} = \text{INR } 946.68 \] \[ (894.33 – 946.68)^2 = 2,740.50; \quad (928.76 – 946.68)^2 = 321.12; \] \[ (963.64 – 946.68)^2 = 287.64; \quad (1000.00 – 946.68)^2 = 2,843.06 \] \[ \mathrm{Sum\ of\ Squared\ Deviations} = 2,740.50 + 321.12 + 287.64 + 2,843.06 = 6,192.32 \] \[ \mathrm{Var}(P_t^*) = \frac{6,192.32}{3} = 2,064.11 \implies \sigma(P_t^*) = \sqrt{2,064.11} = \text{INR } 45.43 \]

Step 4: Shiller Volatility Bound Conclusion

\[ \sigma(P_t) = 274.35 \quad \text{vs} \quad \sigma(P_t^*) = 45.43 \implies \sigma(P_t) \approx 6.04 \times \sigma(P_t^*) \]

The observed stock price volatility is 6.04 times higher than the theoretical dividend variance bound. This decisive violation confirms Shiller’s excess volatility finding: asset market prices fluctuate far more than fundamental corporate cash flows warrant, proving behavioral pricing forces.

6. Corporate Case Vignette: Black Monday (October 19, 1987)

Corporate Case Vignette: Black Monday and Portfolio Insurance Cascades

On Monday, October 19, 1987, the Dow Jones Industrial Average collapsed by 508 points — a staggering single-day decline of 22.61%. Equivalent losses ripped through global financial centers in London, Tokyo, and Frankfurt. The crash destroyed over \$500 billion of market capitalization in six hours.

Under conventional neoclassical finance, an asset price drop of 22.6% requires a massive catastrophic collapse in fundamental future economic output or corporate dividend expectations. Yet on October 19, 1987, no economic news, regulatory upheaval, or geopolitical crisis occurred.

The Behavioral Mechanism: The crash was triggered by institutional herd behavior amplified by automated algorithmic stop-loss programs known as “Portfolio Insurance” (dynamic delta-hedging using S&P 500 futures contracts). As markets dipped, algorithms automatically executed massive sell orders in futures markets, driving index arbitrageurs to dump cash equities on the NYSE. The mechanical cascade created a catastrophic liquidity vacuum. Robert Shiller surveyed hundreds of institutional traders on the day of the crash; over 80% reported that their panic selling was motivated entirely by observing other participants selling, not by fundamental news. Black Monday proved that market prices can decouple violently from intrinsic economic value.

7. AKTU Exam Drill: 2-Mark & 10-Mark University Questions

AKTU Pattern Section A (2 Marks Each)

Short-Answer Conceptual Checkpoints

Q1: What is Robert Shiller’s ‘Excess Volatility’ argument?

Answer: Robert Shiller proved that the volatility of stock market prices is far too large to be explained by changes in subsequent real dividends, mathematically violating the variance bounds demanded by the Present Value Dividend Discount Model.

Q2: State the Shiller Variance Bound inequality mathematically.

Answer: $mathrm{Var}(P_t) le mathrm{Var}(P_t^*)$, stating that the variance of observed market prices must be less than or equal to the variance of the perfect foresight fundamental dividend present value.

Q3: Why was the Crash of October 19, 1987 a crisis for the EMH?

Answer: Because the US stock market dropped 22.6% in a single trading session without any material change in fundamental macroeconomic news or corporate earnings outlook, proving prices are driven by panic cascades.

Q4: Define the Adaptive Markets Hypothesis (AMH).

Answer: Proposed by Andrew Lo, AMH applies principles of evolutionary biology (competition, mutation, adaptation) to explain that market efficiency is dynamic and fluctuates across changing market environments.

Q5: What role did ‘Portfolio Insurance’ play in the 1987 stock market crash?

Answer: It was an automated synthetic dynamic hedging strategy that mechanically sold equity futures contracts as prices fell, creating a vicious selling cascade that drained market liquidity.

AKTU Pattern Section B & C (10 Marks Each)

In-Depth Analytical & Long-Answer Questions

Q6: Explain Robert Shiller’s Excess Volatility puzzle. Derive the variance bound inequality and explain why empirical tests refute neoclassical market efficiency.

Model Answer Outline:

Theoretical Framework: Formulate the Present Value Dividend Model $P_t = sum rac{E_t[D_{t+k}]}{(1+r)^k}$ and define the ex-post rational price $P_t^*$.
Mathematical Derivation of Variance Bound: Show that $P_t^* = P_t + u_t$. With rational expectations, $mathrm{Cov}(P_t, u_t) = 0$, leading directly to $mathrm{Var}(P_t^*) = mathrm{Var}(P_t) + mathrm{Var}(u_t) implies mathrm{Var}(P_t) le mathrm{Var}(P_t^*)$.
Empirical Findings: Contrast the theoretical line with Shiller’s historical S&P 500 dataset (1871-1979), demonstrating that $sigma(P_t)$ exceeds $sigma(P_t^*)$ by 5 to 13 times.
Worked Numerical Evidence: Present the 4-year backward induction numerical showing standard deviation of market price (INR 274.35) exceeding fundamental dividend standard deviation (INR 45.43) by 6x.
Policy & Investment Implications: Conclude that price swings reflect investor psychology, feedback loops, and animal spirits rather than rational discounted cash flows.

Q7: Compare and contrast Eugene Fama’s Efficient Market Hypothesis with Andrew Lo’s Adaptive Markets Hypothesis. How does evolutionary biology reconcile market efficiency with investor irrationality?