"Matrix Algebra Preliminary" (Introduction to Matrix ALGEBRA) course by Prof. A.k.kaw (University of South Florida) is designed and taught.
PDF format Learning note Download (academia.edu)
9th. Download the course handout (PDF)
Summary
- ill-conditional System
- A system of equations is considered to being ill-conditioned if a small change in the coefficient matrix or a small change in The right hand is side results in a large change in the solution vector.
- For example, the following system $$\begin{bmatrix} 1& 2\\ 2& 3.999\end{bmatrix}\begin{bmatrix}x\\ Y\end{bmatrix } = \begin{bmatrix}4\\ 7.999\end{bmatrix}$$ The solution is $$\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix} 1& 2 \ 2& 3.999\end{bmatrix} ^{-1}\cdot\begin{bmatrix}4\\ 7.999\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}$$ Make a small hand side vector of the equations $$\begin{bmatrix} 1& 2\\ 2& 3.999\end{bmatrix}\ begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4.001\\ 7.998\end{bmatrix}$$ gives $$\begin{bmatrix}x\\ y\end{ Bmatrix} = \begin{bmatrix} 1& 2\\ 2& 3.999\end{bmatrix} ^{-1}\cdot\begin{bmatrix}4.001\\ 7.998\end{bmatrix} = \ begin{bmatrix}-3.999\\ 4.000\end{bmatrix}$$ make a small change in the coefficient matrix of the equations $$\begin{bmatri X} 1.001& 2.001\\ 2.001& 3.998 \end{bmatrix} \begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7.999\end{ bmatrix}$$ gives $$\begin{bmatrix}x\\ Y\end{bmatrIX} = \begin{bmatrix} 1.001& 2.001\\ 2.001& 3.998\end{bmatrix} ^{-1}\cdot\begin{bmatrix}4\\ 7.999\end{bmatrix} = \begin{bmatrix} 6.989016\\ -1.497254\end{bmatrix}$$ We can see which a small change in the coefficient matrix or the righ T hand side resulted in a large change in the solution vector.
- well-conditional System
- A system of equations is considered to being well-conditioned if a small change in the coefficient matrix of a small change I n the right hand, side results in a small change in the solution vector.
- For example, the following system $$\begin{bmatrix} 1& 2\\ 2& 3\end{bmatrix}\begin{bmatrix}x\\ Y\end{bmatrix} = \ begin{bmatrix}4\\ 7\end{bmatrix}$$ The solution is $$\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix} 1& 2\\ 2& ; 3\end{bmatrix} ^{-1}\cdot\begin{bmatrix}4\\ 7\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}$$ make a small change in t He right hand side vector of the equations $$\begin{bmatrix} 1& 2\\ 2& 3\end{bmatrix} \begin{bmatrix}x\\ Y\END{BMA Trix} = \begin{bmatrix}4.001\\ 7.001\end{bmatrix}$$ gives $$\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix} 1& 2 \ \ 2& 3\end{bmatrix} ^{-1} \cdot \begin{bmatrix} 4.001\\ 7.001\end{bmatrix} = \begin{bmatrix}1.999\\ 1.001\end{ bmatrix}$$ make a small change in the coefficient matrix of the equations $$\begin{bmatrix} 1.001& 2.001\\ 2.001& 3.001 \end{bmatrix} \begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7\end{bmatrix}$$ gives $$\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix} 1.001& 2.001\\ 2.001& 3.001\end{bmatrix} ^{-1}\cdot\begin{bmatrix}4\\ 7\end{bmatrix} = \begin{bmatrix} 2.003\\ 0.997\end{bmatrix}$$ We can see which a small change in the coefficient matrix or the right hand side resulted in a small C Hange in the solution vector.
- Norm
- Just like the determinant, the norm of a matrix are a simple unique scalar number. For a $m \times n$ matrix $[a]$, the row sum norm of $[a]$ are defined as $$\| A\|_{\infty}=\max_{1\leq I\leq m}\sum_{j=1}^{n}|a_{ij}|$$ is, find the sum of the absolute value of the elements pf E Ach row of the matrix $[a]$. The maximum out of the $m $ such values is the row sum norm if the matrix $[a]$.
- For example, we have the following matrix $$[a] = \begin{bmatrix}10& -3& 5\\ -7& 2.099& -1\\ 0& 6& 5\end{bmatrix}$$ the row sum norm of $[a]$ is $$\| A\|_{\infty} = \max_{1\leq i\leq3} \sum_{j=1}^{3}|a_{ij}|$$ $$=\max[(10+7+0), (3+2.099+6), (5,-1,5)]$$ $$=\max[17, 11.099, 11] =17$$
- The relationship between the norm and the conditioning of the matrix
- Example of the ill-conditioned system. $$\begin{bmatrix} 1& 2\\ 2& 3.999\end{bmatrix}\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7.999\end {bmatrix}$$ which has the solution $$\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}$$ Denoting the Above system as $AX =b$, and hence we have $$\| x\|_{\infty}=2$$ $$\| b\|_{\infty}=7.999$$ Making A small change on the right hand side $$\begin{bmatrix} 1& 2\\ 2& 3.999\end{bmatrix}\b egin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4.001\\ 7.998\end{bmatrix}$$ gives $$\begin{bmatrix}x\\ y\end{ Bmatrix} = \begin{bmatrix}-3.999\\ 4.000\end{bmatrix}$$ denoting the above changed system as $AX ' =b ' $ and $$\delta x=x '-x= \begin{bmatrix}-3.999\\ 4.000\end{bmatrix}-\begin{bmatrix}2\\ 1\end{bmatrix} = \begin{bmatrix}-5.999\\ 3.000\end{ bmatrix}$$ $$\delta b=b '-B = \begin{bmatrix}4.001\\ 7.998\end{bmatrix}-\begin{bmatrix}4\\ 7.999\end{bmatrix} = \begin{ bmatrix}0.001\\ -0.001\end{bmatrix}$$ then $$\|\delta x\|_{\infty} = 5.999$$ $$\|\delta b\|_{\infty} = 0.001$$ The relative change in the norm of the solution vector is $${\|\delta x\|_ {\infty}\over \| X\|_{\infty}} = {5.999\over2}=2.9995$$ The relative change in the norm of the right hand side vector is $${\|\delta b\|_{\ Infty}\over \| B\|_{\infty}} = {0.001\over7.999}=1.25\times10^{-4}$$ That's, the small relative change of $1.25\times10^{-4}$ in the rig HT hand side vector norm results in a large relative change in the solution vector norm of $2.9995$. We can see the ratio of this and norms is $${\|\delta x\|_{\infty} \big/\| X\|_{\infty}\over \|\delta b\|_{\infty} \big/\| B\|_{\infty}} = 23993$$
- Example of the well-conditioned system. $$\begin{bmatrix} 1& 2\\ 2& 3\end{bmatrix}\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7\end{bmatrix }$$ which has the solution $$\begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}$$ Denoting the above Sy Stem as $AX =b$, and hence we have $$\| x\|_{\infty}=2$$ $$\| b\|_{\infty}=7$$ Making A small change on the right hand side $$\begin{bmatrix} 1& 2\\ 2& 3\END{BMATRIX}\BEGIN{BMA trix}x\\ Y\end{bmatrix} = \begin{bmatrix}4.001\\ 7.001\end{bmatrix}$$ gives $$\begin{bmatrix}x\\ y\end{bmatrix} = \ Begin{bmatrix}1.999\\ 1.001\end{bmatrix}$$ denoting the above changed system as $AX ' =b ' $ and $$\delta x=x '-x=\begin{bmatr Ix}1.999\\ 1.001\end{bmatrix}-\begin{bmatrix}2\\ 1\end{bmatrix} = \begin{bmatrix}-0.001\\ 0.001\end{bmatrix}$$ $$\ Delta b=b '-B = \begin{bmatrix}4.001\\ 7.001\end{bmatrix}-\begin{bmatrix}4\\ 7\end{bmatrix} = \begin{bmatrix}0.001\\ 0.001\end{bmatrix}$$ then $$\|\delta x\|_{\infty} = 0.001$$ $$\|\delTa b\|_{\infty} = 0.001$$ The relative change in the norm of the solution vector is $${\|\delta x\|_{\infty}\over \| X\|_{\infty}} = {0.001\over2}=5\times10{-4}$$ The relative change in the norm of the right hand side vector is $${\|\delta B\|_{\infty}\over \| B\|_{\infty}} = {0.001\over7} = 1.429 \times 10^{-4}$$ That's, the small relative change of $1.429\times10^{-4}$ in the R ight hand side vector norm results in a small relative change in the solution vector norm of $5\times10^{-4}$. We can see the ratio of this and norms is $${\|\delta x\|_{\infty} \big/\| X\|_{\infty}\over \|\delta b\|_{\infty} \big/\| B\|_{\infty}} = 3.5$$
- Properties of Norms
- $\| a\| \geq 0$
- $\|ka\| = |k|\| a\|$ where $k $ is a scalar.
- $\| A+b\|\leq \| a\| + \| b\|$
- $\| ab\| \leq \| a\|\cdot\| b\|$
- For a system $AX =b$, we have $${\|\delta x\|\over \| x\|} \leq \| a\|\| a^{-1}\| {\|\delta B\|\over \| b\|} $$ and $${\|\delta X\|\over \| X + \delta x\|} \leq \| a\|\| a^{-1}\| {\|\delta A\|\over \| a\|} $$ where $\| a\|\| A^{-1}\|$ is called the \textbf{condition number}, cond$ (A) $.
- significant Digits
- the possible relative error in the solution vector norm was no more then cond$ (A) \times\epsilon$, where $\epsilon$ is The machine epsilon which are $2.220446\times10^{-16}$ or $2^{-52}$ here (obtained by R code . Machine$double.eps on 64-bit PC, more details refer to Link1 and LINK2). Hence cond$ (A) \times \epsilon$ should give us the number of significant digits, $m $ that is at least correct in our Solu tion by finding out the largest value of $m $ for which cond$ (A) \times\epsilon$ are less than or equal to $0.5\times 10^{-m }$.
- How many significant digits can I trust in the solution of the following system of equations? $$\begin{bmatrix}1& 2 \ 2& 3\end{bmatrix} \begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7\end{bmatrix }$$ for $ $A =\begin{bmatrix}1& 2 \ \ 2& 3\end{bmatrix}$$ and $ $A ^{-1}= \begin{bmatrix}-3& 2 \ \ 2& -1\END{BMA Trix}$$ then $$\| A\|_{\infty}=5,\ \| A^{-1}\|_{\infty}=5\rightarrow \text{cond} (A) =\| a\|_{\infty}\| A^{-1}\|_{\infty} = 25$$ Thus $$\text{cond} (A) \times\epsilon \leq 0.5\times10^{-m}$$ $$\rightarrow 25\times\epsilon\ leq0.5\times10^{-m}$$ $$\rightarrow \log (25\times\epsilon) \leq \log (0.5\times10^{-m}) $$ $$\Rightarrow m\leq13.95459 $$ is, and digits is at least correct in the solution vector.
Selected problems
1. What factors does the adequacy of the solution of simultaneous linear equations in?
Solution:
The product of condition number cond$ (A) =\| a\|\| a^{-1}\|$ and Machine Epsilon $\epsilon$.
2. If a system of equations $[a][x]=[b]$ is ill-conditioned and then
A. $\det (a) =0$
B. cond$ (A) =1$
C. cond$ (A) $ is large
D. $\| a\|$ is large
Solution:
If the system is ill-conditioned and then the condition number cond$ (A) =\| a\|\| A^{-1}\|$ is large. The correct answer is C.
3. If cond$ (A) =10^{4}$ and $\epsilon=0.119\times10^{-6}$, then on $[a][x]=[b]$, at least how many significant digits is C Orrect in the solution?
solution:$$\text{cond} (A) \times\epsilon \leq 0.5\times10^{-m}$$ $$\rightarrow 10^{4}\times0.119\times10^{ -6} \leq 0.5\times10^{-m}$$ $$\rightarrow m\leq {\log (0.5)-\log (0.119\times10^{-2}) \over \log (Ten)} = 2.623423$$ Thus at l East 2 significant digits is correct in the solution.
4. Make a small the coefficient matrix of $$\begin{bmatrix}1& 2 \ 2& 3.999\end{bmatrix} \begin{bmatrix} x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7.999\end{bmatrix}$$ and find $${\|\delta x\|_{\infty} \big/\| X\|_{\infty}\over \|\delta a\|_{\infty} \big/\| a\|_{\infty}}$$
Solution:
The solution of the original system is $$ \begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}$ $Making A Small change in the coefficient matrix as $$\begin{bmatrix} 1.001& 2.001 \ 2.001& 4.000\end{bmatrix} \begin{bmatr ix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7.999\end{bmatrix}$$ and the solution is $$\begin{bmatrix}x\\ y\end{bmatrix} = \begin{bmatrix}5999\\ -2999\end{bmatrix}$$ Hence The row sum norms is $$\| x\| = 2,\ \|\delta x\|=5997,\ \| a\|=5.999,\ \|\delta a\|=0.002$$ Thus The ratio is $${\|\delta x\|_{\infty} \big/\| X\|_{\infty}\over \|\delta a\|_{\infty} \big/\| A\|_{\infty}} = {5997 \big/2\over 0.002 \big/5.999} = 8994001$$ It is a large number. Hence we can conclude that the this system is ill-conditioned. On the other hand, we can calculate the condition number of the coefficient matrix, note that $A ^{-1} = \begin{bmatrix}-39 99& $ \ 2000& -1000 \end{bmatrix}$, and hence $$\| a\|\| a^{-1}\|= 5.999\times5999=35988 $$ which is also a large nUmber.
5. Make a small the coefficient matrix of $$\begin{bmatrix}1& 2 \ 2& 3\end{bmatrix} \begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7\end{bmatrix}$$ and find $${\|\delta x\|_{\infty} \big/\| X\|_{\infty}\over \|\delta a\|_{\infty} \big/\| a\|_{\infty}}$$
Solution:
The solution of the original system is $$ \begin{bmatrix}x\\ Y\end{bmatrix} = \begin{bmatrix}2\\ 1\end{bmatrix}$ $Making A small change in the coefficient matrix as $$\begin{bmatrix} 1.001& 2.001 \ 2.001& 3.001\end{bmatrix} \BEGIN{BM atrix}x\\ Y\end{bmatrix} = \begin{bmatrix}4\\ 7\end{bmatrix}$$ and the solution is $$\begin{bmatrix}x\\ y\end{bmatrix} = \ begin{bmatrix}2.003\\ 0.997\end{bmatrix}$$ Hence The row sum norms is $$\| x\| = 2,\ \|\delta x\|=0.003,\ \| a\|=5,\ \|\delta a\|=0.002$$ Thus The ratio is $${\|\delta x\|_{\infty} \big/\| X\|_{\infty}\over \|\delta a\|_{\infty} \big/\| A\|_{\infty}} = {0.003 \big/2\over 0.002 \big/5} = 3.75$$ It is a small number. Hence we can conclude that the this system is well-conditioned. On the other hand, we can calculate the condition number of the coefficient matrix, note that $A ^{-1} = \begin{bmatrix}-3& Amp 2 \ 2& -1\end{bmatrix}$, and hence $$\| a\|\| A^{-1}\|= 5\times5=25$$ which is also a small number.
6. Prove $${\|\delta x\|\over \| x\|} \leq \| a\|\| a^{-1}\| {\|\delta B\|\over \| b\|} $$
Solution:
The key point is $\| xy\| \leq \| x\|\| y\|$. Let $AX =b$ and then if $B $ are changed to $B ' $, the $X $ are changed to $X ' $, such a $ $AX ' =b ' $$ Hence we have $ $AX =b,\ AX ' =b ' $$ $$\rightarrow \delta x=x '-x=a^ { -1}b '-a^{-1}b = A^{-1}\delta b$$ $$\rightarrow\|\delta X\|\leq \| A^{-1}\|\|\delta b\|$$ and $ $AX =b\rightarrow \| b\|=\| ax\| \leq \| a\|\| x\|$$ Multiply the above inequalities and obtain $$\|\delta x\|\| b\| \leq \| A^{-1}\|\|\delta b\|\| a\|\| x\|$$ $$\rightarrow {\|\delta x\|\over \| x\|} \leq \| a\|\| a^{-1}\| {\|\delta B\|\over \| b\|} $$
7. Prove $${\|\delta x\|\over \| X + \delta x\|} \leq \| a\|\| a^{-1}\| {\|\delta A\|\over \| a\|} $$
Solution:
Similar to the previous question, we have $ $AX =b,\ a ' x ' =b$$ $$\rightarrow ax=a ' x ' = (A+\delta A) (X+\delta X) =ax+a\delta x+\d Elta ax + \delta a\delta x$$ $$\rightarrow a\delta x+\delta AX + \delta A\delta X = [0]$$ $$\rightarrow \delta A (X+\Delta x) =-a\delta x $$ $$\rightarrow \delta x=-a^{-1}\delta A (X+\delta X) \leq \| A^{-1}\|\|\delta a\|\| X+\delta x\|$$ $$\rightarrow \| A\|\delta X\leq \| a\|\| A^{-1}\|\|\delta a\|\| X+\delta x\|$$ $$\rightarrow {\|\delta x\|\over \| X + \delta x\|} \leq \| a\|\| a^{-1}\| {\|\delta A\|\over \| a\|} $$
8. Prove that cond$ (A) \geq 1$.
solution:$$\text{cond} (A) = \| a\|\| a^{-1}\| \geq \| aa^{-1}\| = \| i\|=1$$
9. For $$[a] = \begin{bmatrix}10& -7& 0\\ -3& 2.099& 6\\ 5& -1& 5\end{bmatrix}$$ gives $$[A]^{-1} = \begin{bmatrix}-0.1099& -0.2333& 0.2799\\ -0.2999& -0.3332& 0.3999\\ 0.04995& 0.1666& 6.664\ times10^{-5}\end{bmatrix}$$ (A) What is the condition number of $[a]$?
(B) How many significant digits can we at least trust in the solution of $[a][x] = b$ if $\epsilon = 0.1192\times10^{-6}$?
Solution:
(a) cond$ (a) = \| a\|\| a^{-1}\| = 17\times1.033 = 17.561$
(B) $$\text{cond} (A) \times\epsilon \leq 0.5\times10^{-m}$$ $$\rightarrow 17.561\times0.1192\times10^{-6} \leq 0.5\ times10^{-m}$$ $$\rightarrow m \leq 5.378145$$ Hence 5 Significant digits can be trusted in the solution.
$$[a] = \begin{bmatrix}1& 2+\delta\\ 2-\delta& 1\end{bmatrix}$$ Based on the row sum norm and given that $ \delta\rightarrow0$, $\delta > 0$, what is the condition number of the matrix?
Solution:
Recall the inverse of the matrix $[m]=\begin{bmatrix}a &b\\ C &d \end{bmatrix}$ is $$\begin{bmatrix}{d\over\d ET (M)} & {-b\over \det (m)}\\ {-c\over\det (M)} &{a\over\det (M)} \end{bmatrix}$$ where $\det (m) = ad-bc$. Thus we have $ $A ^{-1} = \begin{bmatrix}{1\over-3+\delta^2}&-{2+\delta\over-3+\delta^2}\\{-2+\delta\over-3+\ delta^2}& {1\over-3+\delta^2}\end{bmatrix}$$ The row sum norms is $$\| a\| = \max (3+\delta, 3-\delta) = 3+\delta$$ and $$\| a^{-1}\| = \max\left ({3+\delta\over 3-\delta^2}, {3-\delta \over 3-\delta^2} \right) = {3+\delta\over 3-\delta^2}$$ Hence $$\text{ Cond} (A) = \| a\|\| a^{-1}\| = {(3+\delta) ^2\over3-\delta^2}$$
A.kaw Matrix Algebra Preliminary study note 9. Adequacy of Solutions
