<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>Yanhong Wan</title>
<link>https://rainbowbow123.github.io/</link>
<atom:link href="https://rainbowbow123.github.io/index.xml" rel="self" type="application/rss+xml"/>
<description>Rainbow’s Storage</description>
<generator>quarto-1.9.37</generator>
<lastBuildDate>Sun, 03 May 2026 00:00:00 GMT</lastBuildDate>
<item>
  <title>Local Gaussian Approximation of the Posterior</title>
  <dc:creator>Yanhong Wan</dc:creator>
  <link>https://rainbowbow123.github.io/posts/local_gaussian/local_gaussian_approximation.html</link>
  <description><![CDATA[ 





<section id="local-gaussian-approximation-of-the-posterior" class="level2" data-number="1">
<h2 data-number="1" class="anchored" data-anchor-id="local-gaussian-approximation-of-the-posterior"><span class="header-section-number">1</span> Local Gaussian Approximation of the Posterior</h2>
<p>Consider the log posterior density of the latent state: <img src="https://latex.codecogs.com/png.latex?%0A%5Clog%20p(h%20%5Cmid%20Y,%20%5Ctheta)%20:=%20%5Cell(h).%0A"></p>
<p>To approximate the <em>local shape</em> of this function, take a second-order Taylor expansion around some point <img src="https://latex.codecogs.com/png.latex?h_0">:</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cell(h)%20%5Capprox%20%5Cell(h_0)%20+%20%5Cnabla%20%5Cell(h_0)'(h%20-%20h_0)%20+%20%5Ctfrac%7B1%7D%7B2%7D(h%20-%20h_0)'%20%5Cnabla%5E2%20%5Cell(h_0)(h%20-%20h_0).%0A"></p>
<p>Now choose <img src="https://latex.codecogs.com/png.latex?h_0%20=%20%5Chat%7Bh%7D">, where <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bh%7D"> is the mode of <img src="https://latex.codecogs.com/png.latex?%5Cell(h)">. By definition: <img src="https://latex.codecogs.com/png.latex?%0A%5Cnabla%20%5Cell(%5Chat%7Bh%7D)%20=%200,%0A"> so the linear term vanishes.</p>
<p>We are left with a quadratic approximation: <img src="https://latex.codecogs.com/png.latex?%0A%5Cell(h)%20%5Capprox%20%5Cell(%5Chat%7Bh%7D)%20+%20%5Ctfrac%7B1%7D%7B2%7D(h%20-%20%5Chat%7Bh%7D)'%20%5Cnabla%5E2%20%5Cell(%5Chat%7Bh%7D)(h%20-%20%5Chat%7Bh%7D).%0A"></p>
<p>Since <img src="https://latex.codecogs.com/png.latex?%5Chat%20h"> is a local mode, the Hessian is negative definite in a neighborhood of <img src="https://latex.codecogs.com/png.latex?%5Chat%20h">. Furthermore, rewrite the approximation as <img src="https://latex.codecogs.com/png.latex?%0A%5Cell(h)%20%5Capprox%20%5Cell(%5Chat%7Bh%7D)%20-%20%5Ctfrac%7B1%7D%7B2%7D(h%20-%20%5Chat%7Bh%7D)'%20%5Cbigl%5B-%5Cnabla%5E2%20%5Cell(%5Chat%7Bh%7D)%5Cbigr%5D(h%20-%20%5Chat%7Bh%7D).%0A"></p>
<p>Now, exponentiating both sides, we obtain the local kernel <img src="https://latex.codecogs.com/png.latex?%0Ap(h%20%5Cmid%20Y,%20%5Ctheta)%20%5Cpropto%0A%5Cexp%5Cleft(%0A-%5Ctfrac%7B1%7D%7B2%7D(h%20-%20%5Chat%7Bh%7D)'%20%5Cbigl%5B-%5Cnabla%5E2%20%5Cell(%5Chat%7Bh%7D)%5Cbigr%5D(h%20-%20%5Chat%7Bh%7D)%0A%5Cright).%0A"></p>
<p>This has the form of a Gaussian kernel. Defining <img src="https://latex.codecogs.com/png.latex?%0A%5CSigma%20=%20%5Cbigl%5B-%5Cnabla%5E2%20%5Cell(%5Chat%7Bh%7D)%5Cbigr%5D%5E%7B-1%7D,%0A"> we have the local Gaussian approximation to the posterior: <img src="https://latex.codecogs.com/png.latex?%0Ap(h%20%5Cmid%20Y,%20%5Ctheta)%20%5Capprox%20%5Cmathcal%7BN%7D(%5Chat%7Bh%7D,%20%5CSigma).%0A"></p>
<hr>
</section>
<section id="interpretation-curvature-and-uncertainty" class="level2" data-number="2">
<h2 data-number="2" class="anchored" data-anchor-id="interpretation-curvature-and-uncertainty"><span class="header-section-number">2</span> Interpretation: Curvature and Uncertainty</h2>
<ul>
<li>The mode <img src="https://latex.codecogs.com/png.latex?%5Chat%7Bh%7D"> is the most likely latent state given <img src="https://latex.codecogs.com/png.latex?Y"> and <img src="https://latex.codecogs.com/png.latex?%5Ctheta">.</li>
<li>The negative Hessian <img src="https://latex.codecogs.com/png.latex?-%5Cnabla%5E2%20%5Cell(%5Chat%7Bh%7D)"> captures the local curvature of the log posterior near the mode.</li>
<li>If the curvature is sharper in some direction, then the corresponding eigenvalue of the negative Hessian is larger, which implies a smaller variance in that direction.</li>
</ul>
<hr>
</section>
<section id="some-potential-limitations" class="level2" data-number="3">
<h2 data-number="3" class="anchored" data-anchor-id="some-potential-limitations"><span class="header-section-number">3</span> Some Potential Limitations</h2>
<p>This approximation only captures <strong>local geometry</strong>: it ignores skewness and heavy tails, and may fail when the posterior is multimodal.</p>


</section>

 ]]></description>
  <guid>https://rainbowbow123.github.io/posts/local_gaussian/local_gaussian_approximation.html</guid>
  <pubDate>Sun, 03 May 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Definitions from Hamilton Chapter3</title>
  <dc:creator>Yanhong Wan</dc:creator>
  <link>https://rainbowbow123.github.io/posts/Ergodicity_vs_Stationarity/</link>
  <description><![CDATA[ 





<p>Consider a time series denoted as <img src="https://latex.codecogs.com/png.latex?%5C%7BY_t%5C%7D_%7Bt=-%5Cinfty%7D%5E%7B%5Cinfty%7D">. An observed sequence <img src="https://latex.codecogs.com/png.latex?%5C%7By_t%5C%7D_%7Bt=-%5Cinfty%7D%5E%7B%5Cinfty%7D"> is <strong>one realization</strong> of the underlying stochastic process. In the finite case, the sequence <img src="https://latex.codecogs.com/png.latex?y_%7B1:T%7D"> is one realization of the joint distribution of <img src="https://latex.codecogs.com/png.latex?Y_%7B1:T%7D">. If we were to sample <img src="https://latex.codecogs.com/png.latex?n"> independent paths of <img src="https://latex.codecogs.com/png.latex?%5C%7BY_t%5C%7D_%7Bt=-%5Cinfty%7D%5E%7B%5Cinfty%7D">, the collection of random variables <img src="https://latex.codecogs.com/png.latex?%5C%7BY_t%5E%7B(1)%7D,%20Y_t%5E%7B(2)%7D,%20%5Cdots,%20Y_t%5E%7B(n)%7D%5C%7D"> at a specific time <img src="https://latex.codecogs.com/png.latex?t"> would constitute an i.i.d. random sample from the <strong>marginal distribution</strong> of <img src="https://latex.codecogs.com/png.latex?Y_t">.</p>
<p>To analyze if the time series is stationary, we look at the moments of <img src="https://latex.codecogs.com/png.latex?Y_t">. If <img src="https://latex.codecogs.com/png.latex?%5Cmathbf%7BE%7D%5BY_t%5D%20=%20%5Cmu"> and <img src="https://latex.codecogs.com/png.latex?%5Cgamma_j%20:=%20%5Ctext%7BCov%7D(Y_t,%20Y_%7Bt-j%7D)"> do not depend on time <img src="https://latex.codecogs.com/png.latex?t">, the process is said to be covariance-stationary (weakly-stationary).</p>
<p>However, covariance stationarity does not guarantee ergodicity for the mean, nor ergodicity for the distribution. A process is said to be <strong>ergodic for the mean</strong> if its time average converges in probability to the ensemble mean: <img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7BT%7D%20%5Csum_%7Bt=1%7D%5E%7BT%7D%20Y_t%20%5Cxrightarrow%7Bp%7D%20%5Cmathbf%7BE%7D%5BY_t%5D.%0A"> A sufficient condition for this is <img src="https://latex.codecogs.com/png.latex?%5Csum_%7Bj=0%7D%5E%7B%5Cinfty%7D%20%7C%5Cgamma_j%7C%20%3C%20%5Cinfty">. Furthermore, the process is <strong>ergodic for the autocovariance</strong> if: <img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B1%7D%7BT-j%7D%20%5Csum_%7Bt=j+1%7D%5E%7BT%7D%20(Y_t%20-%20%5Cmu)(Y_%7Bt-j%7D%20-%20%5Cmu)%20%5Cxrightarrow%7Bp%7D%20%5Cgamma_j%20%5Cquad%20%5Ctext%7Bfor%20all%20%7D%20j.%0A"></p>
<p>To understand: Stationarity ensures that within one realization of <img src="https://latex.codecogs.com/png.latex?%5C%7BY_t%5C%7D_%7Bt=-%5Cinfty%7D%5E%7B%5Cinfty%7D">, the marginal distribution for each random variable does not change with time; Ergodicity, however, ensures that for any realization of the sequence <img src="https://latex.codecogs.com/png.latex?%5C%7BY_t%5C%7D_%7Bt=-%5Cinfty%7D%5E%7B%5Cinfty%7D">, the empirical distribution converges to the <strong>same</strong> stationary distribution.</p>



 ]]></description>
  <category>Econometrics</category>
  <category>Time Series</category>
  <guid>https://rainbowbow123.github.io/posts/Ergodicity_vs_Stationarity/</guid>
  <pubDate>Fri, 03 Apr 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>A Guide to Distinguishing Tail Classes: Single Big Jumps</title>
  <dc:creator>Yanhong Wan</dc:creator>
  <link>https://rainbowbow123.github.io/posts/tail_behavior/</link>
  <description><![CDATA[ 





<section id="does-the-mgf-exist" class="level2">
<h2 class="anchored" data-anchor-id="does-the-mgf-exist">1. Does the MGF Exist?</h2>
<p>A fundamental distinction in probability theory and econometrics is whether a distribution possesses a <strong>moment generating function (MGF)</strong>.</p>
<p>Let</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AM_X(t)%20=%20E%5Be%5E%7BtX%7D%5D%0A"></p>
<p>denote the MGF of a random variable <img src="https://latex.codecogs.com/png.latex?X">.</p>
<p>A distribution <img src="https://latex.codecogs.com/png.latex?F"> is called <strong>heavy-tailed</strong> if</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AM_X(t)%20=%20%5Cinfty,%20%5Cqquad%20%5Cforall%20t%20%3E%200%0A"></p>
<p>In contrast, if there exists some <img src="https://latex.codecogs.com/png.latex?t%3E0"> such that</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AM_X(t)%20%3C%20%5Cinfty,%0A"></p>
<p>the distribution is said to be <strong>light-tailed</strong>.</p>
<p>This criterion acts as a natural “death line”: once the MGF ceases to exist, classical exponential large-deviation tools break down, and extreme outcomes become fundamentally more likely.</p>
<hr>
</section>
<section id="the-hierarchy-of-heavy-tails" class="level2">
<h2 class="anchored" data-anchor-id="the-hierarchy-of-heavy-tails">2. The Hierarchy of Heavy Tails</h2>
<p>Heavy-tailed distributions themselves form a structured hierarchy.<br>
A commonly used nesting is</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ctext%7BRegularly%20Varying%7D%0A%5Csubset%0A%5Ctext%7BSubexponential%7D%0A%5Csubset%0A%5Ctext%7BLong-tailed%7D%0A%5Csubset%0A%5Ctext%7BHeavy-tailed%7D.%0A"></p>
<p>Each step corresponds to progressively weaker assumptions about how slowly the tail decays.</p>
<hr>
<section id="long-tailed-distributions" class="level3">
<h3 class="anchored" data-anchor-id="long-tailed-distributions">2.1 Long-tailed distributions</h3>
<p>A distribution belongs to the <strong>long-tailed class</strong> if</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cbar%20F(x+y)%7D%7B%5Cbar%20F(x)%7D%20%5Cto%201%0A%5Cquad%20%5Ctext%7Bas%20%7D%20x%20%5Cto%20%5Cinfty%0A"></p>
<p>for every fixed <img src="https://latex.codecogs.com/png.latex?y">, where</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbar%20F(x)%20=%20P(X%3Ex)%0A"></p>
<p>is the survival function.</p>
<p>Intuitively, adding a fixed constant <img src="https://latex.codecogs.com/png.latex?y"> barely changes the tail probability when <img src="https://latex.codecogs.com/png.latex?x"> is large.</p>
<hr>
</section>
<section id="subexponential-distributions-and-the-single-big-jump" class="level3">
<h3 class="anchored" data-anchor-id="subexponential-distributions-and-the-single-big-jump">2.2 Subexponential distributions and the “Single Big Jump”</h3>
<p>A central subclass of heavy tails is the <strong>subexponential class</strong>, defined by</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Coverline%7BF%5E%7B*2%7D%7D(x)%20%5Csim%202%5Cbar%20F(x),%0A%5Cquad%20x%20%5Cto%20%5Cinfty.%0A"></p>
<p>Equivalently,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0AP(X_1%20+%20X_2%20%3E%20x)%20%5Csim%202P(X%20%3E%20x)%0A"></p>
<p>for independent copies <img src="https://latex.codecogs.com/png.latex?X_1,%20X_2">.</p>
<p>This property leads to the <strong>Single Big Jump Principle</strong>:</p>
<blockquote class="blockquote">
<p>When a sum of subexponential variables becomes extremely large, it is typically caused by <strong>one extreme observation</strong>, rather than by the accumulation of many moderate values.</p>
</blockquote>
<hr>
</section>
<section id="regular-variation-power-law-tails" class="level3">
<h3 class="anchored" data-anchor-id="regular-variation-power-law-tails">2.3 Regular variation (power-law tails)</h3>
<p>The most widely studied heavy tails are <strong>regularly varying tails</strong>, characterized by</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7B%5Cbar%20F(tx)%7D%7B%5Cbar%20F(x)%7D%20%5Cto%20t%5E%7B-%5Calpha%7D,%0A%5Cquad%20t%3E0.%0A"></p>
<p>Equivalently,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbar%20F(x)%20=%20x%5E%7B-%5Calpha%7D%20L(x),%0A"></p>
<p>where <img src="https://latex.codecogs.com/png.latex?L(x)"> is a <strong>slowly varying function</strong>.</p>
<p>Regular variation provides a tractable mathematical structure and plays a central role in extreme value theory and financial tail modeling.</p>
<hr>
</section>
</section>
<section id="light-tails-the-conspiracy-of-smalls" class="level2">
<h2 class="anchored" data-anchor-id="light-tails-the-conspiracy-of-smalls">3. Light Tails: The “Conspiracy of Smalls”</h2>
<p>Light-tailed distributions behave very differently from heavy-tailed ones.</p>
<p>For light tails, the probability of producing one extremely large observation is exponentially small. As a result, extreme sums typically arise through the <strong>combined effect of multiple moderately large values</strong>, rather than a single outlier.</p>
<p>For example, suppose</p>
<p><img src="https://latex.codecogs.com/png.latex?%0Af(x)%20%5Cpropto%20e%5E%7B-x%5E2%7D.%0A"></p>
<p>To produce a sum of <img src="https://latex.codecogs.com/png.latex?10">:</p>
<ul>
<li><strong>Single jump:</strong> <img src="https://latex.codecogs.com/png.latex?(10%20+%200)"> gives probability <img src="https://latex.codecogs.com/png.latex?%5Cpropto%20e%5E%7B-100%7D"></li>
<li><strong>Two moderate values:</strong> <img src="https://latex.codecogs.com/png.latex?(5%20+%205)"> gives probability <img src="https://latex.codecogs.com/png.latex?%5Cpropto%20e%5E%7B-50%7D"></li>
</ul>
<p>Since <img src="https://latex.codecogs.com/png.latex?e%5E%7B-50%7D%20%5Cgg%20e%5E%7B-100%7D">, the second scenario dominates.</p>
<p>This phenomenon explains why Gaussian-type distributions do <strong>not</strong> satisfy the subexponential property.</p>
<hr>
</section>
<section id="summary-of-tail-classes" class="level2">
<h2 class="anchored" data-anchor-id="summary-of-tail-classes">4. Summary of Tail Classes</h2>
<table class="caption-top table">
<colgroup>
<col style="width: 13%">
<col style="width: 50%">
<col style="width: 36%">
</colgroup>
<thead>
<tr class="header">
<th>Class</th>
<th>Mathematical Criterion</th>
<th>Typical Examples</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Light-tailed</td>
<td><img src="https://latex.codecogs.com/png.latex?M_X(t)%3C%5Cinfty"> for some <img src="https://latex.codecogs.com/png.latex?t%3E0"></td>
<td>Normal, Exponential</td>
</tr>
<tr class="even">
<td>Heavy-tailed</td>
<td><img src="https://latex.codecogs.com/png.latex?M_X(t)=%5Cinfty"> for all <img src="https://latex.codecogs.com/png.latex?t%3E0"></td>
<td>Lognormal, Pareto</td>
</tr>
<tr class="odd">
<td>Long-tailed</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cbar%20F(x+y)/%5Cbar%20F(x)%5Cto1"></td>
<td>Pareto, Lognormal</td>
</tr>
<tr class="even">
<td>Subexponential</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Coverline%7BF%5E%7B*2%7D%7D(x)/%5Cbar%20F(x)%5Cto2"></td>
<td>Pareto, Lognormal</td>
</tr>
<tr class="odd">
<td>Regularly varying</td>
<td><img src="https://latex.codecogs.com/png.latex?%5Cbar%20F(tx)/%5Cbar%20F(x)%5Cto%20t%5E%7B-%5Calpha%7D"></td>
<td>Pareto, Student-t</td>
</tr>
</tbody>
</table>
<hr>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">5. Conclusion</h2>
<p>Tail behavior determines how extreme events arise.</p>
<ul>
<li>In <strong>light-tailed environments</strong>, extreme outcomes result from a <em>conspiracy of many moderate deviations</em>.</li>
<li>In <strong>subexponential environments</strong>, extremes are dominated by a <em>single big jump</em>.</li>
</ul>
<p>Understanding which regime a model belongs to is crucial for applications ranging from risk management to econometric modeling of financial volatility.</p>


</section>

 ]]></description>
  <category>Econometrics</category>
  <category>Extreme Value Theory</category>
  <guid>https://rainbowbow123.github.io/posts/tail_behavior/</guid>
  <pubDate>Sun, 08 Mar 2026 00:00:00 GMT</pubDate>
  <media:content url="https://rainbowbow123.github.io/posts/tail_behavior/tail.png" medium="image" type="image/png" height="113" width="144"/>
</item>
<item>
  <title>Introduction to Machine Learning: Gradient Descent and Neural Networks</title>
  <dc:creator>Yanhong Wan</dc:creator>
  <link>https://rainbowbow123.github.io/posts/Neural_Network/</link>
  <description><![CDATA[ 





<p>To understand machine learning, one must have a basic understanding of optimization. In a learning problem, a loss function is specified to evaluate estimation accuracy, and the goal of optimization is to minimize this loss. Specifically, a loss function assigns a scalar value to model predictions as a quantitative measure of estimation error. Common examples include:</p>
<p>Mean Squared Error (MSE): <img src="https://latex.codecogs.com/png.latex?L_%7B%5Ctext%7BMSE%7D%7D%20=%20%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%20(%5Chat%20y_i%20-%20y_i)%5E2"></p>
<p>Mean Absolute Error (MAE): <img src="https://latex.codecogs.com/png.latex?L_%7B%5Ctext%7BMAE%7D%7D%20=%20%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%20%7C%5Chat%20y_i%20-%20y_i"></p>
<p>For binary classification, cross-entropy is commonly used: <img src="https://latex.codecogs.com/png.latex?L_%7B%5Ctext%7BCE%7D%7D%20=%20-%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%20%5Cleft%5By_i%5Clog%20%5Chat%20p_i%20+%20(1-y_i)%5Clog(1-%5Chat%20p_i)%5Cright%5D"></p>
<p>Below is an example of the loss function of MSE.</p>
<pre><code>import numpy as np 

y_true = np.array([10,20,30])
y_pred = np.array([12,18,27])

loss = np.mean((y_true-y_pred)**2 )
print(loss)</code></pre>
<p>Once a loss function is specified, the next question is how to minimize it. This is handled by optimization algorithms. We introduce the most fundamental algorithm - gradient descent here.</p>
<p>For a differentiable function, the gradient points in the direction of steepest increase. Minimization therefore proceeds by updating parameters in the opposite direction of the gradient. The step size of each update is controlled by the learning rate, whereas an excessively large learning rate may cause overshooting and divergence, and an excessively small learning rate leads to slow convergence.</p>
<p>Below is an example of gradient descent for a linear regression <img src="https://latex.codecogs.com/png.latex?%5Chat%20y%20=%20w%20x"> with loss <img src="https://latex.codecogs.com/png.latex?L%20=%20(y_%7B%5Ctext%7Btrue%7D%7D%20-%20wx)%5E2">. The gradient with respect to <img src="https://latex.codecogs.com/png.latex?w"> is <img src="https://latex.codecogs.com/png.latex?%7B%5Cpartial%20L%7D/%7B%5Cpartial%20w%7D%20=%20-2x%20(y_%7B%5Ctext%7Btrue%7D%7D%20-%20wx)">. Change the learning rate in the code below to see how it affects the optimization process. (It is generally good around 0.004. I have to change from Python to ObservableJS to make it interactive here.)</p>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code hidden" id="cb2" data-startfrom="46" data-source-offset="0" style="background: #f1f3f5;"><pre class="sourceCode js code-with-copy"><code class="sourceCode javascript" style="counter-reset: source-line 45;"><span id="cb2-46">viewof learning_rate <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> Inputs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">range</span>([<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.0001</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.05</span>]<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span> {<span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">step</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.0005</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">value</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.001</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">,</span> <span class="dt" style="color: #AD0000;
background-color: null;
font-style: inherit;">label</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">:</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"learning rate"</span>})</span></code></pre></div></div>
<div class="cell-output cell-output-display">
<div id="ojs-cell-1" data-nodetype="declaration">

</div>
</div>
</div>
<div class="cell">
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb3" data-startfrom="51" data-source-offset="-0" style="background: #f1f3f5;"><pre class="sourceCode js code-with-copy"><code class="sourceCode javascript" style="counter-reset: source-line 50;"><span id="cb3-51"><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Good Day Reader!🥰 Select your learning rate here👆"</span></span></code></pre></div></div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-1" data-nodetype="expression">

</div>
</div>
</div>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb4" data-startfrom="53" data-source-offset="-55" style="background: #f1f3f5;"><pre class="sourceCode js code-with-copy"><code class="sourceCode javascript" style="counter-reset: source-line 52;"><span id="cb4-53">x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">10</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-54">y_true <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">100</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-55">w0 <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-56"></span>
<span id="cb4-57">w <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> w0<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-58"></span>
<span id="cb4-59">outputs <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> {</span>
<span id="cb4-60">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">let</span> w <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> w0<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-61">  <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">let</span> results <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> []<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-62">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">for</span> (<span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">let</span> i <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">0</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span> i <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">&lt;</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">5</span><span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span> i<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">++</span>) {</span>
<span id="cb4-63">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">let</span> y_pred <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> w <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> x<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-64">    <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">let</span> gradient <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span> <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> x <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> (y_true <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> y_pred)<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-65">    w <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">=</span> w <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">-</span> learning_rate <span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> gradient<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-66">    results<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">push</span>(<span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">`iteration </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">${</span>i<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span><span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">1</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">: current prediction </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">${</span>y_pred<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">toFixed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">, next w </span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">${</span>w<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">toFixed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">`</span>)<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-67">  }</span>
<span id="cb4-68">  <span class="cf" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">return</span> results<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">;</span></span>
<span id="cb4-69">}</span>
<span id="cb4-70"></span>
<span id="cb4-71"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">html</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">`&lt;ul&gt;</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">${</span>outputs<span class="op" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">.</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">map</span>(item <span class="kw" style="color: #003B4F;
background-color: null;
font-weight: bold;
font-style: inherit;">=&gt;</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">html</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">`&lt;li&gt;</span><span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">${</span>item<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">&lt;/li&gt;`</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">}</span><span class="vs" style="color: #20794D;
background-color: null;
font-style: inherit;">&lt;/ul&gt;`</span></span></code></pre></div></div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-2" data-nodetype="declaration">

</div>
</div>
</div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-3" data-nodetype="declaration">

</div>
</div>
</div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-4" data-nodetype="declaration">

</div>
</div>
</div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-5" data-nodetype="declaration">

</div>
</div>
</div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-6" data-nodetype="declaration">

</div>
</div>
</div>
<div class="cell-output cell-output-display">
<div>
<div id="ojs-cell-2-7" data-nodetype="expression">

</div>
</div>
</div>
</div>
<p>So far, the model is a si mple linear regression. Next, we upgrade this regression model to a neural network. A neural network contains an input layer, one or more hidden layers, and an output layer. Each layer applies a linear transformation followed by an activation function.</p>
<p>For a generic layer,</p>
<p><img src="https://latex.codecogs.com/png.latex?z%20=%20W%20x%20+%20b,%5Cquad%20h%20=%20%5Cphi(z)">,</p>
<p>where <img src="https://latex.codecogs.com/png.latex?%5Cphi(%5Ccdot)"> denotes an activation function. Common choices include ReLU, sigmoid, and tanh. ReLU preserves positive inputs and truncates negative ones. Sigmoid and tanh map real-valued inputs into bounded intervals.</p>
<p>ReLU: <img src="https://latex.codecogs.com/png.latex?%5Cphi(z)%20=%20%5Cmax(0,%20z)">. Sigmoid:<img src="https://latex.codecogs.com/png.latex?%5Cphi(z)%20=%20%5Cfrac%7B1%7D%7B1%20+%20e%5E%7B-z%7D%7D">. Tanh:<img src="https://latex.codecogs.com/png.latex?%5Cphi(z)%20=%20%5Cfrac%7Be%5E%7Bz%7D%20-%20e%5E%7B-z%7D%7D%7Be%5E%7Bz%7D%20+%20e%5E%7B-z%7D%7D">. Softmax (for vector-valued input <img src="https://latex.codecogs.com/png.latex?z">):<img src="https://latex.codecogs.com/png.latex?%5Cphi(z_j)%20=%20%5Cfrac%7Be%5E%7Bz_j%7D%7D%7B%5Csum_k%20e%5E%7Bz_k%7D%7D">.</p>
<p>All these activation functions introduce nonlinearity。 Without activation functions, stacking multiple layers would still result in a linear model. Like in linear regression, we can also use gradient descent to optimize the parameters of a neural network. The gradients can be computed using backpropagation, which applies the chain rule of calculus to propagate gradients from the output layer back to the input layer. For example, we consider a neural network with 2 hidden layers. The input <img src="https://latex.codecogs.com/png.latex?x"> is transformed through the first hidden layer to produce <img src="https://latex.codecogs.com/png.latex?h_1">, then through the second hidden layer to produce <img src="https://latex.codecogs.com/png.latex?h_2">, and finally to the output <img src="https://latex.codecogs.com/png.latex?y">.</p>
<p>Forward:</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cbegin%7Baligned%7D%0Az_%7B1,i%7D%20&amp;=%20W_1%20x_i%20+%20b_1,%20%5C%5C%0Ah_%7B1,i%7D%20&amp;=%20%5Cmathrm%7BReLU%7D(z_%7B1,i%7D),%20%5C%5C%0Az_%7B2,i%7D%20&amp;=%20W_2%20h_%7B1,i%7D%20+%20b_2,%20%5C%5C%0Ah_%7B2,i%7D%20&amp;=%20%5Cmathrm%7BReLU%7D(z_%7B2,i%7D),%20%5C%5C%0A%5Chat%7By%7D_i%20&amp;=%20W_3%20h_%7B2,i%7D%20+%20b_3.%0A%5Cend%7Baligned%7D"></p>
<p>By the chain rule, backpropagation gives: <img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20W_1%7D%0A=%0A%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20%5Chat%20y%7D%0A%5Ccdot%0A%5Cfrac%7B%5Cpartial%20%5Chat%20y%7D%7B%5Cpartial%20h_2%7D%0A%5Ccdot%0A%5Cfrac%7B%5Cpartial%20h_2%7D%7B%5Cpartial%20z_2%7D%0A%5Ccdot%0A%5Cfrac%7B%5Cpartial%20z_2%7D%7B%5Cpartial%20h_1%7D%0A%5Ccdot%0A%5Cfrac%7B%5Cpartial%20h_1%7D%7B%5Cpartial%20z_1%7D%0A%5Ccdot%0A%5Cfrac%7B%5Cpartial%20z_1%7D%7B%5Cpartial%20W_1%7D.">The goal is still to minimize the MSE, <img src="https://latex.codecogs.com/png.latex?L%20=%20%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%20(%5Chat%20y_i%20-%20y_i)%5E2">. For this network, the loss is differentiable with respect to all parameters, so gradients can be written in closed form using the chain rule. Specifically, for ReLu, the derivative is defined in a piecewise manner,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cfrac%7Bd%7D%7Bdz%7D%5Cmathrm%7BReLU%7D(z)%20=%0A%5Cbegin%7Bcases%7D%0A1,%20&amp;%20z%20%3E%200,%20%5C%5C%0A0,%20&amp;%20z%20%3C%200,%0A%5Cend%7Bcases%7D%0A"></p>
<p>and a subgradient at ( z=0 ) is typically chosen by convention. This leads to the indicator term ( (z&gt;0) ) appearing in the gradients.</p>
<p>We analytically write out the gradients for all parameters below.</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20W_3%7D%0A=%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%20(%5Chat%20y_i%20-%20y_i)%5C,%20h_%7B2,i%7D%5ET">,</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20b_3%7D%0A=%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%20(%5Chat%20y_i%20-%20y_i)">,</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20W_2%7D%0A=%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%0A(%5Chat%20y_i%20-%20y_i)%5C,%0AW_3%5C,%0A%5Cmathbf%7B1%7D(z_%7B2,i%7D%3E0)%5C,%0Ah_%7B1,i%7D%5ET">,</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20b_2%7D%0A=%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%0A(%5Chat%20y_i%20-%20y_i)%5C,%0AW_3%5C,%0A%5Cmathbf%7B1%7D(z_%7B2,i%7D%3E0)">,</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20W_1%7D%0A=%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%0A(%5Chat%20y_i%20-%20y_i)%5C,%0AW_3%20W_2%5C,%0A%5Cmathbf%7B1%7D(z_%7B2,i%7D%3E0)%5C,%0A%5Cmathbf%7B1%7D(z_%7B1,i%7D%3E0)%5C,%0Ax_i%5ET">,</p>
<p><img src="https://latex.codecogs.com/png.latex?%5Cfrac%7B%5Cpartial%20L%7D%7B%5Cpartial%20b_1%7D%0A=%0A%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi=1%7D%5En%0A(%5Chat%20y_i%20-%20y_i)%5C,%0AW_3%20W_2%5C,%0A%5Cmathbf%7B1%7D(z_%7B2,i%7D%3E0)%5C,%0A%5Cmathbf%7B1%7D(z_%7B1,i%7D%3E0)">.</p>
<p>In the next post, I will use these analytically derived gradients to perform optimization for this network in Python.</p>



 ]]></description>
  <category>Optimization</category>
  <category>Machine Learning</category>
  <guid>https://rainbowbow123.github.io/posts/Neural_Network/</guid>
  <pubDate>Wed, 04 Feb 2026 00:00:00 GMT</pubDate>
  <media:content url="https://rainbowbow123.github.io/posts/Neural_Network/figures/neural_network.png" medium="image" type="image/png" height="96" width="144"/>
</item>
<item>
  <title>Intuition behind Analyzing Likelihood Shapes through Contours and Hessians</title>
  <dc:creator>Yanhong Wan</dc:creator>
  <link>https://rainbowbow123.github.io/posts/Likelihood_shape/</link>
  <description><![CDATA[ 





<p>When working with likelihood-based models, it is often useful to first form a rough picture of the likelihood surface before choosing an optimization or sampling method. Contour plots and Hessians provide two complementary views of this shape: contours show how likelihood varies across directions, while Hessians summarize local curvature at a point.</p>
<p>Throughout this post, contour plots refer to level sets of the likelihood function<br>
<img src="https://latex.codecogs.com/png.latex?f(y;%5Ctheta)"> as a function of the parameter vector <img src="https://latex.codecogs.com/png.latex?%5Ctheta">. For visualization, <img src="https://latex.codecogs.com/png.latex?%5Ctheta"> is restricted to two dimensions, even though the underlying parameter space is typically higher-dimensional.</p>
<p>A simple reference case is the round bowl. The contour lines are closed and nearly circular. Moving away from the center produces similar changes in the likelihood in all directions. No direction appears noticeably flatter than others.</p>
<p><img src="https://rainbowbow123.github.io/posts/Likelihood_shape/figures/round_bowl.png" class="img-fluid" style="width:65.0%"></p>
<p>When the parameters are highly correlated, the contour will be dense along the line <img src="https://latex.codecogs.com/png.latex?%5Ctheta_1%20%5Capprox%20%5Calpha%20%5Ctheta_2">. In this case, contours become elongated and rotated, forming an elliptical bowl. The likelihood still decreases in all directions, but at different rates depending on direction. Some directions are flatter than others, though none are close to flat.</p>
<p><img src="https://rainbowbow123.github.io/posts/Likelihood_shape/figures/elliptical_bowl.png" class="img-fluid" style="width:65.0%"></p>
<p>With stronger anisotropy, the contours stretch further and form a long, narrow corridor. The likelihood changes rapidly when moving across the corridor, while changes along the corridor are much slower. This type of shape is commonly referred to as a ridge.</p>
<p><img src="https://rainbowbow123.github.io/posts/Likelihood_shape/figures/near_ridge.png" class="img-fluid" style="width:65.0%"></p>
<p>These visual differences correspond to differences in local curvature, which are summarized by the Hessian matrix. The Hessian is the matrix of second derivatives of the log-likelihood with respect to the parameters, evaluated at a point. Its eigenvalues describe curvature along orthogonal principal directions.</p>
<p>For example, a Hessian of the form</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bpmatrix%7D%0A-50%20&amp;%2010%20%5C%5C%0A10%20&amp;%20-45%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p>has eigenvalues of comparable magnitude. Both principal directions exhibit similar curvature, which corresponds to the roughly circular contours above.</p>
<p>Increasing the off-diagonal terms produces stronger directional imbalance,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bpmatrix%7D%0A-50%20&amp;%2020%20%5C%5C%0A20%20&amp;%20-50%0A%5Cend%7Bpmatrix%7D.%0A"></p>
<p>Here, one direction is flatter than the other, consistent with elongated but still well-behaved contours.</p>
<p>In a ridge-like case, the imbalance becomes extreme. A matrix such as</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Cbegin%7Bpmatrix%7D%0A-50%20&amp;%2049%20%5C%5C%0A49%20&amp;%20-50%0A%5Cend%7Bpmatrix%7D%0A"></p>
<p>has eigenvalues that differ by orders of magnitude. For instance, one eigenvalue may be on the order of <img src="https://latex.codecogs.com/png.latex?100">, while the other is close to <img src="https://latex.codecogs.com/png.latex?0.1">. This indicates strong curvature in one direction and an almost flat direction in another, matching the narrow corridor seen in the contour plot.</p>
<p>The separation between eigenvalues is often summarized by the condition number,</p>
<p><img src="https://latex.codecogs.com/png.latex?%0A%5Ckappa%20=%20%5Cfrac%7B%7C%5Clambda_%7B%5Cmax%7D%7C%7D%7B%7C%5Clambda_%7B%5Cmin%7D%7C%7D.%0A"></p>
<p>Large values correspond to highly uneven curvature. By contrast, the determinant only reflects the product of curvatures across directions: it becomes small when at least one direction is nearly flat, but it does not distinguish a ridge from a uniformly rescaled surface.</p>
<p>Contour plots and Hessians provide a compact way to form a basic picture of likelihood shape. Contours show how likelihood changes across directions in parameter space, while Hessians quantify this behavior locally at a point. Over time, reading these two objects side by side becomes a practical way to anticipate how different numerical methods are likely to behave.</p>



 ]]></description>
  <category>Optimization</category>
  <category>Econometrics</category>
  <category>Intuition</category>
  <guid>https://rainbowbow123.github.io/posts/Likelihood_shape/</guid>
  <pubDate>Sun, 25 Jan 2026 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
