$\hat b$

nearest reachable output

$r$

residual, $\hat b\to b$

$A\T r=0$

$r\perp$ every column of $A$

$\operatorname{Col}(A)=\operatorname{span}(e_1)$ and $b=(2,1)$. Which candidate output is closest to $b$?

1. A$(0,0)$
2. B$(2,0)$✓
3. C$(3,0)$

Squared distances $d^2=\|b-y\|^2$: $5,\ 1,\ 2$. Ranking by $d$ or by $d^2$ is the same.

$r=b-\hat b=(0,1)$ is perpendicular to $e_1$.

$A=\begin{bmatrix}1&1\\0&0\end{bmatrix}$. Which inputs give the output $(2,0)$?

1. Aonly $(1,1)$
2. Bonly $(2,0)$
3. Call three✓

$Ax=(x_1+x_2,\,0)$, and all three have $x_1+x_2=2$.

These three are only examples: the whole line $x_1+x_2=2$ works. $A(1,-1)=0$, so sliding along $(1,-1)$ changes the input, not the output.

$A=\operatorname{diag}(1,\,0.01)$. Two output errors have equal length. Which grows more when we recover $x=A^{-1}y$?

×50 zoom: read the numbers, not the lengths.

1. Ahorizontal $\delta_1$
2. Bvertical $\delta_2$✓
3. Cequally

$A^{-1}=\operatorname{diag}(1,100)$: $\ 0.008\to0.008$, $\ 0.008\to0.8$.

What $A$ squeezes most, inversion stretches most.

$x\in\R^2$

the input displacement

$H=\begin{bmatrix}a&b\\b&c\end{bmatrix}$

real and symmetric, $H=H\T$

$Hx\in\R^2$

a vector

$q(x)\in\R$

one number: the height, or cost, of $x$

$H=H\T$

real $n\times n$; both tests assume it

$H\succ0$

positive definite (PD)

$H\succeq0$

positive semidefinite (PSD)

Every PD matrix is PSD. The trough is one PSD matrix that is not PD; the zero matrix is another.

Spectral theorem: real symmetric $H$ has real $\lambda_i$ and orthonormal eigenvectors $v_i$.

$Q=[v_1\cdots v_n]$

$Q\T Q=I$: rotation or reflection

$\Lambda$

diagonal $\lambda_i$

$z=Q\T x$

$z_i=v_i\T x$

$H\succeq0\iff$ all $\lambda_i\ge0$
$H\succ0\iff$ all $\lambda_i>0$

$A\in\R^{m\times n}$

any shape: tall, wide or square

$A\T A$

Gram matrix, $n\times n$: $(A\T A)\T=A\T A$

$\ge0$

for every $x\in\R^n$, so $A\T A\succeq0$

$A_1=\mathrm{diag}(1,2)$ and $A_2=\begin{bmatrix}1&1\\0&0\end{bmatrix}$ map the unit circle as shown. Which $A\T A$ is PD?

1. Aonly $A_1\T A_1$✓
2. Bonly $A_2\T A_2$
3. Cboth

$\|A_2x\|^2=(x_1+x_2)^2=0$ at $x=(1,-1)$. $A\T A\succ0$ exactly when $A$ has full column rank.

$\iff$

a squared length is 0 only for the zero vector

$\operatorname{Null}(A)$

all $x\in\R^n$ with $Ax=0$

why

$A\T Ax=0\Rightarrow x\T A\T Ax=0\Rightarrow Ax=0$; conversely $Ax=0\Rightarrow A\T Ax=0$

$4z_1^2+z_2^2=1$ is the level set $x\T Hx=1$ in principal coordinates. Which eigenvalue sets its long axis?

1. Athe larger
2. Bthe smaller✓
3. Cneither

On axis $i$, $\lambda_iz_i^2=1$, so the semiaxis is $1/\sqrt{\lambda_i}$ ($\lambda_i>0$):
larger $\lambda_i$, shorter axis.

A level set, not a circle's image (that comes with the SVD).

$H=Q\Lambda Q\T$

PSD: $\lambda_i\ge0$, $Q=[v_1\ v_2]$

$\Lambda^{1/2}$

$\mathrm{diag}(\sqrt{\lambda_i})$: real as $\lambda_i\ge0$

$H^{1/2}$

unique symmetric PSD: $H^{1/2}H^{1/2}=H$

$A$

$\approx\begin{bmatrix}2.61&0.42\\1.11&1.33\end{bmatrix}$, not symmetric

$v$

a unit input direction, turning

$A\T A$

its Gram matrix: symmetric PSD

Which $v$ makes $\|Av\|^2$ largest? Which makes it smallest?

$A$

$m\times n$: input $\R^n$ to output $\R^m$

$v_i\in\R^n$

input direction; the $v_i$ are orthonormal

$u_i\in\R^m$

output direction; the $u_i$ are orthonormal

$\sigma_i\ge0$

length gain: $\sigma_i=\|Av_i\|$, $i\le\min(m,n)$

$A=\mathrm{diag}(3,1)$. Which unit input has the longest output $Ax$?

1. Ahorizontal, $e_1$✓‖Ax‖ = 3
2. Bvertical, $e_2$‖Ax‖ = 1
3. Cdiagonal, $(e_1+e_2)/\sqrt2$‖Ax‖ = √5 ≈ 2.24

$\|Ax\|^2=9x_1^2+x_2^2$ with $x_1^2+x_2^2=1$: all of $x$ on $e_1$.

Does C’s output still point at 45°?

$U\T U=I_m$

orthonormal columns $u_i$: output $\R^m$

$V\T V=I_n$

orthonormal columns $v_i$: input $\R^n$

$\Sigma$

$\sigma_1\ge\sigma_2\ge\dots\ge0$ on the diagonal, else $0$

Every real matrix has a full SVD.

$A$ is $3\times2$, with full SVD $A=U\Sigma V\T$. What are the shapes of $(U,\Sigma,V\T)$?

1. A(3×3, 3×2, 2×2)✓product 3×2
2. B(3×2, 2×2, 2×3)product 3×3
3. C(2×2, 2×3, 3×3)product 2×3

$U$: square, output side $\R^3$. $V$: square, input side $\R^2$. $\Sigma$: the shape of $A$.

A $2\times2$ matrix maps the unit circle onto the segment from $-2e_1$ to $2e_1$. What are $(\sigma_1,\sigma_2)$?

1. A$(2,1)$would leave height 1
2. B$(2,0)$✓
3. C$(0,0)$image = one point

$\sigma_1=\|Av_1\|=2$ is half the length $4$: $\pm v_1\mapsto\pm2e_1$. And $Av_2=0$, so $\sigma_2=0$.

$\lambda_i$

$=\|Av_i\|^2\ge0$: eigenvalues of PSD $A\T A$

$v_i$

orthonormal eigenvectors: input axes

$\sigma_i=0$

$Av_i=0$: pick $u_i$ to finish the basis

$\delta_{ij}$

1 if $i=j$, else 0: orthonormal $u_i$ ($\sigma_i\sigma_j>0$)

Which statement about the SVD is true?

1. AIt exists only for square $A$every $A$ has one
2. BInput and output dimensions may differ✓
3. CSingular values can be negative$\sigma_i=\|Av_i\|\ge0$

$v_i\in\R^n$, $u_i\in\R^m$. An eigenvector needs one space: $m=n$.

Eigenvalues may be negative or complex; $\sigma_i$ is a length.

$U=V$

can be chosen exactly when $A$ is symmetric PSD

$D$

$\operatorname{diag}(\pm1,\pm1)$: flip $u_i$ and $v_i$ together

$v_{1,2}=(1,\pm1)/\sqrt2$. For $x=v_1+2v_2$, find $Ax$, $\sigma_i$, $u_i$.

1. A$3v_1-2v_2$✓
2. B$3v_1+2v_2$misses reversal
3. C$-3v_1+2v_2$wrong reversal

$Av_1=3v_1$, $Av_2=-v_2$: $\sigma=(3,1)$, $u_1=v_1$, $u_2=-v_2$. Thus $Ax=3u_1+2u_2$.

$r$

$\operatorname{rank}(A)$: how many $\sigma_i>0$

$U_r,V_r$

first $r$ columns: $m\times r$, $n\times r$

$\Sigma_r$

$r\times r$ diagonal, all $>0$

$U_rU_r\T$

$m\times m$, $\ne I_m$ if $r<m$

Thin (economy): $\min(m,n)$ columns, zeros kept.

Keep one term of $A=3u_1v_1\T+u_2v_2\T$:
$A_1=3u_1v_1\T$. Which input does $A_1$ send to $0$?

1. A$v_1$$\mapsto 3u_1$
2. B$v_2$✓
3. C$v_1+v_2$$\mapsto 3u_1$

$A_1x=3\,(v_1\T x)\,u_1$ reads only the $v_1$ coordinate, and $v_1\T v_2=0$: $\mathrm{Null}(A_1)=\mathrm{span}(v_2)$, a whole line.

$\mathrm{Row}(A)=\mathrm{span}(v_1,\dots,v_r)$, dim $r$

$\mathrm{Null}(A)=\mathrm{span}(v_{r+1},\dots,v_n)$, dim $n-r$

$\mathrm{Col}(A)=\mathrm{span}(u_1,\dots,u_r)$, dim $r$

$\mathrm{Null}(A\T)=\mathrm{span}(u_{r+1},\dots,u_m)$, dim $m-r$

A $3\times3$ matrix has singular values $5,\,2,\,0$. Which pair is (rank, nullity)?

1. A$(3,0)$a zero gain adds no rank
2. B$(2,1)$✓
3. C$(1,2)$2 is small, not zero

rank $=\#\{\sigma_i>0\}=2$; nullity $=n-\text{rank}=1$, $\mathrm{Null}(A)=\mathrm{span}(v_3)$.

$\|M\|_F^2$

$\sum_{i,j}M_{ij}^2$

$\|A-A_k\|_F^2$

$\sum_{i>k}\sigma_i^2$, the error

$k$

$0\le k\le r$; $A_r=A$

$B$

any matrix with $\operatorname{rank}\le k$

Eckart–Young

no proof; may not be unique if $\sigma_k=\sigma_{k+1}>0$

Next: each row of $A$ is one data sample.

$x_j\in\R^d$

sample $j$: one point in feature space

$N$

rows = samples; here $N=8$

$d$

columns = features; here $d=2$

Sixteen numbers, but a thin cloud. Could one number per sample nearly do?

$\mu\in\R^d$

mean sample, in feature space

$\mathbf 1\in\R^N$

all ones; $\mathbf 1\mu\T$ is $N\times d$, each row $\mu\T$

$X_c$

$N\times d$, row $j$ is $(x_j-\mu)\T$; columns sum to $0$

$\|Xv\|^2=\sum_j(x_j\T v)^2$ weighs raw vectors: $\|Xe_1\|^2=700.18$, $\|Xe_2\|^2=7$.

Centered, the vectors are nearly vertical, and so is $v_1$ of $X_c$.

Uncentered $X$: what does its first right singular vector $v_1$ mostly reflect?

1. Athe cloud’s offset from the origin✓
2. Bthe largest variation around the mean, guaranteed
3. Cthe number of samples $N$

$x_{c,j}\T$

row $j$ of $X_c$

$v\in\R^d$

unit direction

left side

variance of the $N$ projections (mean $0$): sum of squares over $N-1$

$C$

$d\times d$, symmetric, PSD; $N>1$

why

$X_c\T X_c=V\Sigma\T\Sigma V\T$

$v_i\in\R^d$

feature space: principal directions; $v_1$ maximizes $v\T Cv$ over unit $v$

$u_i\in\R^N$

sample space: $u_i=X_cv_i/\sigma_i$ if $\sigma_i>0$; one entry per sample

$\sigma_i^2$

$=\|X_cv_i\|^2$; variance along $v_i$ is $\sigma_i^2/(N-1)$

A new small example: centered data have singular values $\sigma_1=3$, $\sigma_2=1$. What share of the total variance does the first component keep?

1. A75%3/(3+1): σ not squared
2. B90%✓9/(9+1)
3. C50%counts components

Variance along $v_i$ is $\sigma_i^2/(N-1)$: square first, then share. $N-1$ cancels.

$V_k$

first $k$ columns of $V$, $d\times k$, orthonormal

$Z$

$N\times k$ scores, one row per sample

$\hat X$

$N\times d$; $ZV_k\T=X_cV_kV_k\T$: best rank-$k$ approximation of $X_c$

$\mu=(1,-1)$, $x=(3,-1)$, $v_1=(1,1)/\sqrt2$. Find $z$ and $\hat x$ in original coordinates; diagram: centered.

1. A$z=\sqrt2,\ \hat x=(2,0)$✓
2. B$z=2,\ \hat x=(3,1)$$(1,1)$ is not unit
3. C$z=1,\ \hat x=(2,-1)$off the line

$x_c=x-\mu=(2,0)$, $z=\sqrt2$;
$\hat x=\mu+zv_1=(2,0)$, $x-\hat x=(1,-1)\perp v_1$.

$\sigma_i$

singular values of $X_c$

$\text{kept}_k$

share in $k$ components; undefined if $X_c=0$

• Same samples, new units, new $v_1$.
• Center first. Standardizing is a modeling choice, not an automatic cure.
• Large variance need not mean task relevance.

$c=V\T x$

$x$ along $v_1,v_2$

$d=U\T b$

$b$ along $u_1,u_2$

$\sigma_i>0$

one per row: $c_i=d_i/\sigma_i$

$\sigma_i=0\ne d_i$

$0=d_i$: no solution

$\sigma_i=0=d_i$

$c_i$ free: take $0$

$A=\begin{bmatrix}3&0\\0&1\end{bmatrix}$, $b=(6,2)$. Which $x$ gives $Ax=b$?

1. A$x=(18,2)$Ax = (54, 2)
2. B$x=(2,2)$✓Ax = (6, 2)
3. C$x=(2,6)$Ax = (6, 6)

Divide by each gain: $x_1=6/3$, $x_2=2/1$.

$A\in\R^{m\times n}$

$A^+\in\R^{n\times m}$; rank $r$

$\sigma_i>0$

inverted; a zero gain is not: what is lost stays lost

$r=0$

$A^+=0$

$A=\begin{bmatrix}2&0\\0&0\end{bmatrix}$, $b=(4,3)$. What is $A^+b$?

1. A$(2,0)$✓Ax = (4, 0)
2. B$(2,3)$same residual, longer $\|x\|$
3. Cnone: no exact solutionA⁺ always exists

$A^+=\begin{bmatrix}\frac12&0\\0&0\end{bmatrix}$ leaves the zero gain alone. $A^+b$ exists even when $Ax=b$ has no solution.

$AA^+$

$m\times m$, projects onto $\operatorname{Col}(A)$

$A^+A$

$n\times n$, projects onto $\operatorname{Row}(A)$

$\ne I$

unless $r=m$, resp. $r=n$

Here $A=\begin{bmatrix}2&2\\1&1\end{bmatrix}$, rank 1.

First minimize the residual $\|b-Ax\|$; then take the smallest norm among tied minimizers.

$x$

any minimizer; $z\in{}$Null($A$)

$x^\star=A^+b$

in Row($A$), so $x^\star\perp z$

$\|x\|^2$

squared norm: least only at $z=0$

All three solve $x_1+x_2=2$. Which has the smallest norm $\|x\|$?

1. A$(2,0)$‖x‖² = 4
2. B$(1,1)$✓‖x‖² = 2
3. C$(0,2)$‖x‖² = 4

The first circle to touch the line is tangent to it, so the radius there is perpendicular to it: $x^\star\perp(1,-1)$, the null direction.

$\delta x\in\R^n$

input error (mm)

$\delta b\in\R^m$

output error (mm)

$A$ fixed

$x^\star+\delta x=A^+(b+\delta b)$: only $b$ moves

Both output errors have length $0.01$. Which gives the larger input error $\|\delta x\|$?

1. A$\delta b=(0.01,\,0)$$\|\delta x\|=0.01$
2. B$\delta b=(0,\,0.01)$✓$\|\delta x\|=1$
3. Cthe same for bothgains differ

Along $u_i$, $\|\delta x\|=\|\delta b\|/\sigma_i$: $0.01/1$ against $0.01/0.01$. Equal $\|\delta b\|$, unequal $\|\delta x\|$.

$\|A\|_2,r_\tau$

any $A$, $\tau\ge0$

$\kappa_2$

full rank; else $\infty$ by convention

$1/\sigma_{\min}$

absolute; $\kappa_2$ relative

scope

fixed invertible $A$; $Ax=b\ne0$, only $b$ moves

Conditioning is the problem’s, stability the algorithm’s.

Minimize $\|Ax-b\|^2+\lambda^2\|x\|^2$.

$c_i=v_i\T x$, $d_i=u_i\T b$, $\lambda>0$.

$\phi_i$

fit + penalty

Use $h_i=\sigma_i^2+\lambda^2>0$. Complete square:
$\phi_i=h_i(c_i-\sigma_i d_i/h_i)^2+\lambda^2d_i^2/h_i$.

$c_i^\star=\sigma_i d_i/h_i$; $x_\lambda=\sum_{i=1}^{r}c_i^\star v_i$.

Unique; gain $\le1/(2\lambda)$.

Fix $A,b$. As $\lambda$ grows $0\to3$, what do $\|x_\lambda\|$ and $\|r_\lambda\|$ do?

1. A$\|x_\lambda\|$ nonincreasing, $\|r_\lambda\|$ nondecreasing✓
2. B$\|x_\lambda\|$ increases, $\|r_\lambda\|$ always decreasesnorm falls
3. C$b$ stays exactly attainedresidual grows

No gain $\sigma_i/(\sigma_i^2+\lambda^2)$ grows with $\lambda$. Not always strictly: take $b=0$.

$A$, $r=n$

$\kappa_2=\sigma_1/\sigma_n$

$A\T A$

gains $\sigma_i^2$

$q$

joint vector: $n$ angles in radians here; a prismatic joint would contribute a length

$p$

tip position only: $m$ task coordinates, no orientation

$f$

forward kinematics: sines and cosines of $q$, not a matrix

Both shoulders turn $+5^\circ$, elbows fixed. The tip changes $\Delta p$ are…

1. Aalways identical≈0.172 vs ≈0.060 long
2. Bgenerally pose-dependent✓
3. Cdetermined only by joint countsame 2 joints

The shoulder swings the tip on a circle of radius $r$: $|\Delta p|\approx r\,\Delta q_1$ with $\Delta q_1=5^\circ\approx0.0873$ rad, not $5$.

$f'(q)$

local gain: output change per radian, at this $q$

$\Delta q$

step in radians; for small steps the model error shrinks like $\Delta q^2$

row $i$

output $p_i=f_i(q)$, $i=1,\dots,m$

column $j$

joint $q_j$, $j=1,\dots,n$; the other joints held fixed

$\Delta q\in\R^n$

a small joint change, in radians; $\Delta p\in\R^m$

An arm has 7 joints. The task is only its 3D tip position. What shape is $J$?

1. A$7\times3$ℝ³ → ℝ⁷
2. B$3\times7$✓ℝ⁷ → ℝ³
3. C$7\times7$ℝ⁷ → ℝ⁷

$J$ is $m\times n$: one row per task coordinate, one column per joint, so $J\Delta q$ takes $\Delta q\in\R^7$ to $\Delta p\in\R^3$.

$j_j\in\R^m$

column $j$: tip length per radian of joint $j$ alone

$\Delta q_j$

how far joint $j$ turns; the contributions add

$j_j(q)$

changes with the pose $q$

Drawn: $\dot p=J\dot q$ at $\dot q=(1,1)$ rad/s, $q=(30^\circ,60^\circ)$: velocities, not paths.

$J$

given; task axes $x$ right, $y$ up

Only joint 1 turns, by a small $\varepsilon>0$: $\Delta q=(\varepsilon,0)$. The tip mainly moves…

1. ArightΔq’s own axis
2. Bup✓
3. Cleftcolumn 2

Column 1, not row 1 $(0,-1)$: $\Delta p\approx\varepsilon j_1=(0,2\varepsilon)$

Straight arm, $q=(0,0)$: $J=\begin{bmatrix}0&0\\2&1\end{bmatrix}$. Which nonzero tip velocities $J\dot q$ exist?

1. Aonly horizontal, along the arm
2. Bonly vertical, across the arm✓
3. Cany direction in the plane

Rank $1<2$: a singular configuration; $\det J=l_1l_2\sin q_2=0\iff q_2=k\pi$.

A finite bend still moves the tip inward; a least-squares step still exists.

$v_i$

unit joint velocity: both joints move together

$u_i$

unit tip direction in task space

$\sigma_i$

local gain at this $q$: $1.62$ and $0.62$

$\dot q,\ \dot p$

joint and tip velocities; exact by the chain rule

$\Delta p\approx J\Delta q$

finite steps: first order only

The tip must move at the same speed $c$, along $u_1$ or $u_2$. Which needs the smaller joint speed $\|\dot q\|$?

1. Aalong $u_1$, the long axis✓
2. Balong $u_2$, the short axis
3. Cthe same for both

$J(\alpha v_i)=\alpha\sigma_iu_i$ has speed $c$ at $\alpha=c/\sigma_i$, when $\sigma_i>0$: more gain, less joint speed.

$\dot p_d$

target

$J^+$

best fit, min $\|\dot q\|$

$r$

$\dot p_d-J\dot q^\star$

The straight arm again. Which joint velocity gives $J\dot q=0$?

1. A$\dot q=(1,0)$Jq̇ = (0, 2)
2. B$\dot q=(1,-2)$✓Jq̇ = (0, 0)
3. C$\dot q=(1,2)$Jq̇ = (0, 4)

$1\cdot(0,2)+(-2)\cdot(0,1)=0$: the two column effects cancel exactly.

Exact now, not later: held for time $t$, $q=t(1,-2)$ puts the tip at $(2\cos t,\,0)$, drifting inward.

$J$

$2\times3$: two tip coordinates, three joints, rank 2

$z$

null motion, nullity $n-\mathrm{rank}\,J=1$: $z$ contributes exactly zero tip velocity

With $\lambda=0.1$ the damped step minimises $\|J\dot q-\dot p_{\rm d}\|^2+\lambda^2\|\dot q\|^2$. What does damping do?

1. Ahits every desired $\dot p_{\rm d}$ exactly
2. Baccepts task error to reduce joint speed✓
3. Cchanges the arm’s link lengths

The weak direction is mostly given up: its gain $\sigma_2/(\sigma_2^2+\lambda^2)$ stays below $1/(2\lambda)$.

$J$

$m\times n$ at this $q$

$JJ\T+\lambda^2I$

$m\times m$, positive definite

why

$(J\T J+\lambda^2I)J\T=J\T(JJ\T+\lambda^2I)$

*computed. Smaller steps reduce model error faster than target error; limits separate.

$H_{ij}$

row $i$: slope; column $j$: input

$f\in C^2$

continuous second partials near $x$

then

$H_{ij}=H_{ji}$: $H=H\T$, symmetric $n\times n$

$x$

fixed expansion point; $f\in C^2$

$\Delta x$

small step

$\nabla f$

$n\times1$, tilt: tangent plane

$H$

$n\times n$, bend: quadratic model

$f(x_1,x_2)=2x_1^2+\tfrac12x_2^2$. Which matrix is its Hessian?

1. A200$\frac12$$B$, the coefficients
2. B4001✓
3. C1004axes swapped

Differentiate twice: $4x_1$ then $4$; $x_2$ then $1$. No $x_1x_2$ term, so mixed $0$.

$a\,x_i^2$ gives $H_{ii}=2a$. Name the coefficients $B$: $f=x\T Bx=\tfrac12x\T Hx$, so $H=2B=\mathrm{diag}(4,1)$.

$v$

direction, unit: $\|v\|=1$, so directions compare fairly

$g$

height of $f$ along the line through $x$

$g''(0)$

second derivative along that line, not the surface's geometric curvature

why

$g'(t)=\nabla f(x+tv)\T v$; differentiate once more

here

$f=\tfrac12x\T Hx$, $H=\begin{bmatrix}3&1\\1&2\end{bmatrix}$, $x=0$

Along which unit $v$ is $g''(0)=v\T Hv$ largest?

1. A$v=e_1$✓= 4
2. B$v=e_2$= 1
3. C$v=\tfrac{1}{\sqrt2}(1,1)$, at $45^\circ$= 2.5

C gives $\tfrac12(4+1)$, an average. For unit $v$: $4v_1^2+v_2^2\le4$, with equality only at $v=\pm e_1$.

Local tests only. $f\in C^2$ on a convex open set: convex $\iff H(x)\succeq0$ at every $x$.

$(x,y)$

any point; both are $C^2$

at $(0,0)$

$\nabla f_\pm(0,0)=0$, $H_\pm(0,0)=\mathrm{diag}(2,0)$

Can this Hessian alone classify both points?

1. AYes: both are minimaf₋ falls along y
2. BYes: both are saddlesf₊ rises along y
3. CNo: higher-order terms matter✓

$f_+$: strict minimum with $H$ only PSD. PD is sufficient, not necessary.

$H$

Hessian

1224 $=A$, $b=(1,0)$: best output, shortest input, curvature?

Project onto $Ax=(1,2)s$.
$s=1/5$, $\hat b=(1,2)/5$, $r=(4,-2)/5$

$x_1+2x_2=1/5$; shortest $x=(1,2)/25$

$\lambda(H)=25,0$
Flat direction $(2,-1)$; $\min f=2/5$.

surfaces: b = 0

$f=\tfrac12\|Ax-b\|^2$. Choose a method; find $A\hat x,r,A^+b$, a flat direction and the graph shape.

1. AA bowl with one lowest pointAᵀA singular
2. BA trough, bottom: $x_1+x_2=2$✓
3. CA saddleAᵀA ⪰ 0

$Ax=(2,0)$, $r=(0,1)$: $\min f=1/2$.
$A^+b=(1,1)$; flat direction $(1,-1)$.

$V,\Sigma\T\Sigma$

SVD of $A$; diagonal $\sigma_i^2$, then zeros

$H_\lambda$

Hessian of $\tfrac12\|Ax-b\|^2+\tfrac{\lambda^2}2\|x\|^2$, $\lambda>0$

Damped LS as earlier, no halves: Hessian $2H_\lambda$, same minimiser.

Next lecture: systematic derivatives.

Linear model: $J=$ 2000, desired $\Delta p=(2,1)$.
Which set is right?

1. Arank 2, minimum-norm step $(1,1)$, $J\T J$ PD
2. Brank 1, minimum-norm LS step $(1,0)$, $J\T J$ PSD but not PD✓JᵀJ=diag(4,0)
3. Crank 1, minimum-norm step $(2,0)$, target exactly reachable

Why a best step if $r\ne0$? Why $\Delta q_2=0$?