Which point on the line is closest to $b$?

1. A$(0,0)$
2. B$(1,1)$
3. C$(2,2)$✓
4. D$(3,3)$

$C$: $1^2+1^2=2$, so $d=\sqrt2$.

At $C$, what angle does the orange error segment make with the line?

So far: representable? unique? — closest needs a distance.

$u,v,r\in\R^m$

$r$ = difference vector

$\|r\|_2$

a number $\ge0$; $=0$ iff $r=0$

$\|r\|_2^2$

a number too, no square root

Candidates $y=(t,t)$ on the same line, $d(t)=\|b-y\|_2$.

We minimise $d^2$ instead of $d$. Does the best $t$ move?

1. Ayes
2. Bno✓

Same argmin, different min: $\min d=\sqrt2$, $\min d^2=2$. Squaring is strictly increasing on $[0,\infty)$, so it keeps the order of distances.

$u,v\in\R^m$

real; $u\T v$ is a number, not a vector

Lecture 2

wrote $\langle u,v\rangle$

symmetry

each $u_iv_i=v_iu_i$

linear

$u'\in\R^m$ any other vector, $c_1,c_2$ scalars; slot 2 by symmetry

$=0$ iff $u=0$

each term is a square, so $0$ forces every $u_i=0$

same norm

the distance $\|r\|_2$, with $r$ for $u$

$\|u\|$

the Euclidean length, a number

unit vector

$\|u\|=1$; dividing by $\|u\|$ needs $u\neq0$

$\varphi\in[0,\pi]$

the angle between nonzero $u,v$; no angle if one is $0$, and then $u\T v=0$

why

a rotation $R$ keeps $u\T v$; turn both by $R$ with $Ru=\|u\|e_1$: $u\T v=\|u\|(Rv)_1$

sign

$\|u\|\|v\|>0$, so only $\cos\varphi$ can turn it

$s$

a number ($u\neq0$): the signed component of $v$ along the unit direction of $u$

$\|v\|\cos\varphi$

also needs $v\neq0$: no angle when $v=0$, though $s=0$

divisor

$\|u\|$, never $\|v\|$ — measured along $u$

$|s|$

the shadow's length; $s>0$ same side as $u$, $s<0$ opposite, $s=0$ perpendicular

Which have a positive dot product with $u=(1,0)$?

1. A$v_A=(2,1)$✓
2. B$v_B=(0,3)$
3. C$v_C=(-1,2)$
4. D$v_D=(1,-3)$✓

$v_D$: $u\T v_D=1>0$ with $\|v_D\|=\sqrt{10}$.  $v_B$: $\|v_B\|=3$, $u\T v_B=0$.
Sign follows $\varphi$: acute $+$, right $0$, obtuse $-$.

$\perp$

orthogonal: dot product zero

$0\T v=0$

so $0\perp$ every $v$ (no angle defined)

why

expand $(u+v)\T(u+v)$ by linearity

$2\,u\T v$

cross term: $0$ iff $u\perp v$

$S\subseteq\R^m$

a subspace; $y$ is every vector of it

$r\T y$

one number for each $y$ in $S$

a subspace

$0\in S^\perp$; $(c_1r_1+c_2r_2)\T y=0$ too, by linearity

edges

$\{0\}^\perp=\R^m$, $(\R^m)^\perp=\{0\}$: $r\T r=0\Rightarrow r=0$

$d_1=(1,0,0)$, $d_2=(0,1,0)$, $S=\text{span}\{d_1,d_2\}$, $r=(0,1,1)$

$r\perp d_1$. Which is true?

1. A$r\perp S$
2. B$r/\|r\|\perp S$
3. C$r\not\perp S$: $r\T d_2=1$✓
4. D$r\T d_1=0$ gives $r\perp S$

$y∈S$

$S=\text{span}\{d_1,…,d_k\}$, so $y=\sum c_id_i$

$k$ tests

settle every $y$: $r\T y$ is linear in $y$

span only

independence not needed

$\delta_{ij}$

$1$ if $i=j$ (unit), else $0$ (orthogonal)

Each pair is a basis. Sort:

1. Aindependent only
2. Borthogonal, not unit
3. Corthonormal

right angles

$v_i$

pairwise $\perp$, nonzero: $\|v_j\|^2>0$

all $j$

$c_j=0$: independent

fails

drop nonzero: $0\perp$ all

unit lengths

Not for independence; with $\perp$: $v=\sum c_iv_i⇒c_j=v_j\T v$.

For $Q=\begin{bmatrix}1&0\\0&1\\0&0\end{bmatrix}$, which are true?

1. A$Q\T Q=I_2$✓
2. B$QQ\T=I_3$QQᵀ = diag(1,1,0)
3. C$\|Qz\|=\|z\|$ for every $z\in\R^2$✓
4. D$Q\T v$ keeps all of any $v\in\R^3$v₃ is lost

Which $b'$ have $Q\T b'=Q\T b$?

$Q$ embeds

$Q$

$m×k$, columns $q_i$, $Q\T Q=I_k$; $S=\text{Col}(Q)$

$z∈ℝ^k$

$Qz∈ℝ^m$: lengths, angles kept

$Q\T$ extracts

$(Q\T b)_i=q_i\T b$: coordinates of the projection.

$Q\T$ loses $b$'s $S^⟂$ component.

$b$

the target in $\R^m$, off the line

$p=\alpha a$

a candidate; $\alpha$ is our only choice

$r$

what $p$ fails to explain

$a\T r\neq0$

$r$ keeps a part along the line: removable, so $p$ is not lowest

$a\T r=0$

nothing left along the line — why is that enough?

$a\neq0$, $b\in\R^m$. Candidate on span$\{a\}$: $p=\alpha a$

Ask for a perpendicular residual: $a\T(b-\alpha a)=0$

Expand: $a\T b-\alpha\,a\T a=0$ — one equation, one unknown

Substitute $\alpha$ back: the same $p$, now one map.

$a\T b$

$1\times m$ by $m\times1$: a number

$aa\T$

$m\times1$ by $1\times m$: $m\times m$, rank 1 if $a\neq0$, symmetric

$a=0$

divisor $0$: quotient undefined

Types: $\alpha$ number · $p$ vector · $\dfrac{aa\T}{a\T a}$ matrix

Alone, no arithmetic: which way does $r$ point from $p$ to $b$, and is $\alpha$ above or below 1?

Then in pairs: $\alpha$, $p$, $r$, check $r$.

Replace $a$ by $\lambda a$, $\lambda>0$. What happens to $a\T b$, $\alpha$, $p$?

1. A$a\T b$ ×λ, $\alpha$ ÷λ, $p$ same✓
2. Ball the same
3. C$a\T b$ ×λ, $\alpha$ ×λ, $p$ ×λ²
4. D$a\T b$ same, $\alpha$ ÷λ, $p$ same

$a\T b$ grew, $p$ did not. Why?

$a\T b=\|a\|s$ grows with the ruler; $p=\dfrac{\lambda^2aa\T}{\lambda^2a\T a}b$ needs only the line.

Unit $a$: $s=\alpha$, the coordinate of Lecture 2; else equal only if $a\T b=0$.

Why is a perpendicular residual enough to make $p$ the closest point?

Premise: $S$ any subspace, $p\in S$, $r=b-p\in S^\perp$. Take any other $y\in S$.

Yes. Take squared lengths on that right triangle.

Built for a line; a general $S$ still needs a construction.

An illustration of the converse: a residual still leaning along $S$.

$\tau$

the orthogonal projection of $b-y$ onto the line; step to $y+\tau\in S$

$b-y-\tau$

the perpendicular part, untouched

why

Pythagoras: $\tau\perp(b-y-\tau)$

At $\alpha=1$: $b-y=(2,0)$, $\tau=(1,1)$; at $y+\tau=p$: $4-2=2$, nothing left to improve.

$S⊆\R^m$ a subspace: each $b∈\R^m$ splits one way only, and that $p$ is closest.

unique

$p+r=p'+r'⇒p-p'=r'-r∈S∩S^\bot=\{0\}$ (⊥ itself)

closest

proved, for any $y∈S$

exists

construction follows, from an orthonormal basis $Q$

edges

$b∈S⇒r=0$; $b∈S^\bot⇒p=0$

Lecture 1

$S=$Col$(A)$: $b=\hat b+r$, $r∈$Null$(A\T)$, dims add

Both requirements, one at a time.

2.

$b-p$ ⊥ each $q_i$ ⇒ ⊥ $S$ (spanning test) ⇒ $p$ is the closest point

needs

$Q\T Q=I_k$; a general basis $A$ needs a correction factor, not plain $AA\T$

here

$r=(0,-3/5,4/5)$: $q_1\T r=0$, $q_2\T r=-12/25+12/25=0$

Same $S$ and $b$. Add $w=(0,-3,4)$, ⊥ $q_1$ and $q_2$. For $b+w=(2,-2,6)$:

1. Aprojection changes, residual same
2. Bprojection same, residual changes✓
3. Cboth stay the same
4. Dboth change

$Q\T w=0⇒QQ\T(b+w)=QQ\T b$; $r$ becomes $r+w$. Same for every $w∈S^\perp$ (Null$(A\T)$ if $S=$ Col$(A)$).
$τ∈S$: $QQ\T(b+τ)=p+τ$, $r$ same.

Orthogonal projector onto $S$: a matrix $P$ with $Pb∈S$ and $b-Pb∈S^\bot$ for every $b∈\R^m$ (the two requirements checked).

why

$QQ\T$ passes both; the closest point is unique

size

$Q$ is $m×k$: $P$ is $m×m$, rank $k$, symmetric

unique

any orthonormal basis of $S$ gives the same $P$

edges

$S=\{0\}$: $P=0$; $S=\R^m$: $P=I$

reason

$Pb∈S$, and the closest point of $S$ to a point of $S$ is itself

reads

$P(Pb)=Pb$ for every $b$: the second pass moves nothing

uses

$Q\T Q=I_k$, nothing else

$P^2=P$ alone means a projection, not necessarily orthogonal: the vote's $P_B$ is idempotent, yet $(1,-1)$, the direction it discards, is not ⊥ the axis.

$I-P$

orthogonal projector onto $S^\bot$: $Q\T(I-P)b=0$, $P(I-P)=0$

$b=p+r$

$Pp=p$, $Pr=0$

$λ$

1 on kept $p≠0$, 0 on discarded $r≠0$ (Lecture 2)

$λ=0$

$P$ singular: $P(b+w)=Pb$ for $w∈S^\bot$, so $Pb$ does not determine the $S^\bot$ part of $b$

edges

$P=0$: no $λ=1$; $P=I$: no $λ=0$

Model $y\approx\beta_0+\beta_1t$ at inputs $t_i$.

$A\in\R^{3\times2}$

$m=3$ readings, $n=2$ parameters

$x=(\beta_0,\beta_1)$

intercept and slope, the unknowns

$b$

the readings $y_i$, one per input $t_i$

$b\notin$ Col$(A)$

no exact $x$

$x\in\R^n$

the parameters $(\beta_0,\beta_1)$ we choose

$Ax,b,r\in\R^m$

prediction, readings, residual: measurement space

$f(x)$

one number, $\ge0$; any $A$, no rank assumption

$|r_i|$

the side of square $i$ — a side cannot be negative

$e=-r$

Lecture 5 writes $e=Ax-b$

$cf$, $c>0$

same minimiser, gradient $\times c$

$\min$

the smallest value $f$ reaches — one number (ours: p53)

$\arg\min$

the set of $x$ reaching it: one point or many

$\hat x$

any member of that set; any $A$, any rank

The best output $A\hat x$ is always unique; the best parameter vector $\hat x$ need not be.

From Lecture 1: for $G=A\T A$ with columns $a_1,a_2$, the entry is $G_{ij}=a_i\T a_j$.

$a_2$ is turned until perpendicular to $a_1$ and then doubled in length. What happens to $G_{11},G_{12},G_{22}$?

1. Aunchanged, $0$, $\times4$✓
2. B$1$, $0$, $1$
3. Cunchanged, $0$, $\times2$
4. D$0$, unchanged, $\times4$

$a_j$

column $j$ of $A\in\R^{m\times n}$ — any $A$, any rank

$G_{ij}$

row $i$, column $j$: column $i$ dotted with column $j$

$G\T=G$

symmetric, because $a_i\T a_j=a_j\T a_i$

$n$

the number of parameters, not of measurements

$G_{ii}$

squared length of column $i$

$G_{ij}$

lengths and angle at once (p8)

$\varphi$

between $a_i,a_j$, both $\neq0$

Large $|G_{ij}|$ alone does not imply near-parallel columns. $G=I$ exactly for orthonormal columns, singular exactly for dependent ones (p55).

The Gram matrix records every dot product between two columns of $A$.

Which equations does it give for the best-fit parameters?

For our three readings: $G=A\T A=\begin{bmatrix}3&3\\3&5\end{bmatrix}$

Necessary

If $\hat x$ minimises, $r=b-A\hat x\perp$ Col$(A)$ (p29)

$\perp$ Col$(A)$ $\Leftrightarrow$ $a_i\T r=0$ for every column (p14)

so $A\T r=0$: $r\in$ Null$(A\T)$ (p15)

row $i$

what column $i$ measures is matched; the leftover is invisible to it

$n\times n$

$n$ equations, $n$ unknowns, any $A$; solvable: take $Ax=\hat b$; $A\T(b-\hat b)=0$ (p29); only one? p54

Move $b$ by any $w\in$ Null$(A\T)=$ Col$(A)^\perp$ (p15): $A\T b$, and the equations, do not change (p32).

$A$ is $3\times2$, all ones; $b=(1,2,3)$. Move $x$ along $z=(1,-1)$. Which change?

1. Aparameters $x$✓
2. Bprediction $Ax$A(x+z) = Ax
3. Cresidual $r=b-Ax$Ax fixed
4. Dobjective $f=\|Ax-b\|^2$f = ‖r‖²

Steps leaving $Ax$ fixed are exactly Null$(A)=$ span$\{(1,-1)\}$: dependent columns leave least squares solvable — minimisers fill $(1,1)+$Null$(A)$, one $\hat b$.

$\hat x$ solves $A\T A\hat x=A\T b$. For every $A$, which is guaranteed? $\hat x$ is…

1. Aa global minimiser✓
2. Bflat here, but lower somewhere elsenever lower: p51
3. Cthe only minimiserexactly when Null(A)={0}

A, for every $A$: nothing gets below $\|r\|^2$. C is conditional, not false — it needs Null$(A)=\{0\}$ (p54), and p49 had a line of ties.

Sufficient

Take any other $\hat x+z$:

p49: $\hat x=(1,1)$, $z=(1,0)\notin$ Null$(A)$: $Az=(1,1,1)$, $f=2+3=5$.

$G=$3335, $A\T b=(5,6)$. Predict the signs of $r$, then find $\hat x,\hat b,r$.

$(-,+,-)$: the line runs below the middle point

$\beta_0=\frac76$, $\beta_1=\frac12$

$\hat b=(\frac76,\frac53,\frac{13}6)$

Why needn’t $r=0$?

$n$

number of parameters

One point or a family.

$A\T b=(7,10)$; $\hat x=(5/6,3/2)$
$r=(1,-2,1)/6$; $A\T r=0$. Rank unchanged: unique.

$Pb\in$ Col$(A)$, $A\T(b-Pb)=0$: the projection (p29) — $P$ depends only on Col$(A)$.

$A=Q$ orthonormal: $Q\T Q=I$, $P=QQ\T$ (p30). Otherwise the Gram factor corrects the basis.

any basis

cost

build $A\T A$, then solve

orthonormal basis

$Q\T b$

coordinates of $Pb=\sum_i q_i(q_i\T b)$: one each

Can we build such a $Q$ for the same Col$(A)$?

Shadows add up for perpendicular directions; unit length drops the denominators.
Old span · overlap · remainder · normalise.

$q_1=(1,0)$, $a_2=(1,1)$. Sketch each result: which removes exactly the part along $q_1$?

1. A$a_2-q_1(q_1\T a_2)$✓(0, 1)
2. B$a_2-q_1\|a_2\|$(1−√2, 1)
3. C$a_2/\|a_2\|$(1/√2, 1/√2)
4. D$a_2+q_1(q_1\T a_2)$(2, 1)

$q_1\T a_2$ is signed; the shadow’s length is $|q_1\T a_2|$.
$a_2'=(-1,1)$: $q_1\T a_2'=-1$, $a_2'-q_1(-1)=a_2'+q_1=(0,1)$.

$S_{j-1}$

span of the orthonormal $q_1,…,q_{j-1}$; $j=1$: empty sum, $S_0=\{0\}$

$\sum$

$=$ projection onto $S_{j-1}$ only because the $q_i$ are orthonormal

$u_j\in\R^m$

remainder; dividing needs $u_j\neq0$ ($j=1$: $u_1=a_1\neq0$)

$q_1=(1,1,1)/\sqrt3$, $u_2\in$ span$\{a_1,a_2\}$.
Predict its entries and the checks.

component$q_1\T a_2=(0+1+2)/\sqrt3=\sqrt3$

projection$q_1(q_1\T a_2)=(1,1,1)$

remainder$u_2=a_2-(1,1,1)=(-1,0,1)$

normalise$q_2=(-1,0,1)/\sqrt2$

check$q_1\T q_2=0$, $\|q_1\|=\|q_2\|=1$
$\sqrt3 q_1+\sqrt2 q_2=a_2$

Scope: $a_1,…,a_n$ independent — $A$ of full column rank.

$a_j$

column $j$ of $A\in\R^{m\times n}$

$q_i\T a_j$

coefficient on earlier $q_i$

Under that scope $\|u_j\|>0$ at every step, so each division is legal.

assume

$q_1,…,q_{j-1}$ already orthonormal — the induction

$i<j$

$q_i\T q_l=1$ at $l=i$, else $0$: one term survives

$u_j$

the remainder before normalising, in $\R^m$

$=q_i\T a_j-q_i\T a_j=0$ for every $i<j$.

$q_j=u_j/\|u_j\|$, so $q_i\T q_j=0$ too: scaling cannot change a zero.

$\|u_j\|$

$>0$ by the scope

span$\{q_1..q_j\}=$span$\{a_1..a_j\}$
⊇: this identity at each step $i\le j$
⊆: solve for $q_j$: from $a_j$ and earlier $q$’s (induction)

Kept: $q_i\T a_j$ and $\|u_j\|$ — column $j$ of $R$

Lower $a_3$ into the plane: which hits 0, $a_3$ or $u_3$?

$u_3=0$ means:

1. A$a_3\perp q_1,q_2$
2. B$a_3\in$ span$\{q_1,q_2\}$✓
3. C$a_3=0$
4. D$q_1,q_2$ invalid

No $q_3$ out of it: that column added no new direction, not that the space has none.

Rounding can hide exact dependence; a tiny $\|u_3\|/\|a_3\|$ ($a_3\neq0$) shows near-dependence, not exact dependence.

Why we could divide: $u_j=0$ would write $a_j$ from the earlier $q$’s, which independence forbids.

Without it, halting at $u_j=0$ is not basis extraction: $[e_1,2e_1,e_2]$ stops at $j=2$, missing $e_2$.

Skip it, carry on in the original order, index the kept $q$’s.
Pivoting instead reorders columns: $A\Pi=QR$; it is a different algorithm.

$r_{ij}$

$=q_i\T a_j$, $i<j$

$r_{jj}$

$=q_j\T a_j=\|u_j\|>0$: columns independent

$Q\T Q$

$=I_n$ (p19)

$R$

column $j$: $a_j$ in the $q$-basis

$x$

parameters, $\in\R^n$

$Rx$

$Ax$ in the $q$-basis, also $\in\R^n$

$Q$

keeps lengths (p19); $R$ need not

always

$R\T R=(QR)\T QR$

here

$G\neq I$: $R$ is not a rotation

$a_j$ uses only $q_1,\dots,q_j$ (p67). Why is $r_{ij}=q_i\T a_j=0$ for $i>j$?

1. AThe columns of $A$ are orthogonal
2. B$q_i\perp q_1,\dots,q_j$, so $\perp$ their span✓
3. COrthonormal $Q$ forces triangular $R$
4. DOnly when $A$ is square

$a_j\in\text{span}\{q_1,\dots,q_j\}$ (p64), $q_i\perp$ it: the zeros are structural.

Here $m\ge n$ and $A$ has full column rank.

thin

$Q$ is as tall as $A$ (p19)

shapes

$(m\times n)(n\times n)=(m\times n)$

$QQ\T$

$\neq I_m$ once $n<m$ (p18)

We assume full column rank so this construction cannot break down and $R$ is invertible.

$r_{jj}>0$

$r_{jj}=\|u_j\|$, and $u_j=0$ would put $a_j$ in the span of the earlier columns (p66)

$\neq0$

triangular; ours $\sqrt3\sqrt2=\sqrt6$

Rank $<n$: Gram–Schmidt stalls; any $R$ has an ${r_{jj}=0}$, so $\det R=0$ and the solve divides by zero.

Is $q_1$ an eigenvector of $A$?

Throughout $r=b-Ax$, so $QRx-b=-r$ and $\|QRx-b\|=\|r\|$: the same error, drawn from $b$ to the prediction.

Do not start from $QRx=b$: generally false.

check

$=QRx-QQ\T b-b+QQ\T b$

goal

$\min_x\|r\|^2$

The two pieces of $QRx-b$:

$=$

the span was kept

$\Rightarrow$

$\perp$ every $q_i$, so $\perp\text{Col}(A)$

$(I-QQ\T)b$

$=(I-P)b$, the projection's residual; drawn leg $-(I-P)b$

The same perpendicular-error decomposition, in the same colours.

The two parts are perpendicular (p74):

$+$

cross term $=0$: squares add (p11)

$z=Rx-Q\T b$

$\|Qz\|^2=z\T Q\T Qz=\|z\|^2$ (p19)

At the solution, which part must be zero?

second

no $x$ in it: unavoidable

first

$R$ invertible (p71): it can reach $0$

Coordinate part, reachable because $R$ is invertible: $\boxed{R\hat x=Q\T b}$, by back-substitution. Residual $0$ only if $b\in\text{Col}(A)$; here $\|r\|^2=1/6$, so $\|r\|=1/\sqrt6$.

Which unknown comes first, and why?

$\hat x$

$(β_0,β_1)$

$R$

$r_{jj}>0$

$β_1$: the bottom row has one unknown.

$\sqrt2β_1=1/\sqrt2⇒β_1=1/2$

$\sqrt3(β_0+1/2)=5/\sqrt3⇒β_0=7/6$

Same $\hat x$ as the normal equations: $R\hat x=Q\T b$.

What becomes hard when we infer parameters from measurements?

$0.2$ in the data moves the fitted $x$ by $2\sqrt2$, ≈14×: an amplification for these two data vectors in these parameter units, not $κ_2(A)$.

Near dependence = sensitivity (p79); exact ($a_2\parallel a_1$) = non-uniqueness (p54).

$A$

$m\times n$, $m\ge n$

$z$

parameter direction, $\|z\|_2=1$

The second lecture measured how a matrix stretches directions; the new object is the ratio.

min $>0$ $⇔$ Null$(A)=\{0\}$: no unit $z$ is flattened.

Rank lost: some unit $z$ has $Az=0$, so min $=0$ and $κ_2(A)=∞$ by convention.

Which is correct?

1. AQR cures the sensitivity
2. BForming $A\T A$ can hurt the numerics✓
3. CNormal equations fail even exactly
4. DEvery QR code is equally stable

Conditioning is the problem's, stability the algorithm's. In exact arithmetic the normal equations are right; in floating point classical Gram–Schmidt can lose orthogonality.

$R$

$\|Rz\|=\|QRz\|=\|Az\|$, $Q$ an isometry

$A\T A$

full column rank; proved next lecture

The triangular coefficient matrix has $A$'s stretch ratio; the normal equations square it.

Rules of thumb, not bounds: about $\log_{10}κ_2$ digits lost; sensitivity also depends on the residual, so $κ_2(A)$ alone does not settle it.

$e(θ)=f(θ)-p_{\rm target}\in\R^2$

$f$

tip position, nonlinear in $θ$

$θ=(θ_1,θ_2)$

joint angles in radians; $Δθ$ small

$J(θ)$

$∂f/∂θ$; column $j$: tip velocity per unit joint rate

$JΔθ$

a displacement, not a velocity

Near a singularity $J$ can be ill-conditioned; at a singularity it loses rank.

$\det J=l_1l_2\sinθ_2$, link lengths $l_1,l_2>0$

$JΔθ+e$

the model's error after the step

$J$

full column rank here

sign

$b-Ax=-(JΔθ+e)$

The predicted change must cancel $e$: the tip change we ask for is $-e$.

$J=QR⇒RΔθ=-Q\T e$, the fit's triangular solve

$q_i$

columns of $Q$, not joints

$m=n=2$

the step solves $JΔθ=-e$ exactly; $m\ge n$ is the general case

in this example

$e(θ)+JΔθ$

the model's error after the step; $0$ here

$e(θ+Δθ)$

the true error after moving

$E(Δθ)$

their difference: model discrepancy, $O(\|Δθ\|^2)$

Zero linearised error need not mean zero true error.

Linearise, step, measure, relinearise, repeat — each step small enough that the linear model stays accurate.

$\det J\neq0$: full rank, $R$ invertible

QR runs; step size depends on $e$'s direction

$\det J=0$: rank $1$, $r_{22}=0$

Ordinary solve fails, yet minimisers exist; no joint rate gives along-arm tip velocity

Every $A$ and $b$: one closest $\hat b$, one $r\perp\Col(A)$.

$\{x:Ax=\hat b\}=\hat x+\Null(A)$: one point exactly when the columns are independent.

1. $q=(0,1)$, $b=(3,2)$. Projection $p$? Residual $r$?
   ✓ $p=(0,2)$, $r=(3,0)$
2. Why does $A\T r=0$ not force $r=0$? One sentence of reasoning.
   ✓ $r\neq0$ can lie in $\text{Col}(A)^{\perp}$
3. Thin QR, full rank: which part of $\|QRx-b\|^2$ can no $x$ change?
   ✓ $\|(I-QQ\T)b\|^2$

Hand in before the answers.

Nearly dependent or rank-deficient columns: next lecture.