Find the unit vectors that are mapped to vectors normal to…

Find the unit vectors that are mapped to vectors normal to themselves by the affine transformation $x'=8x-4y,\ y'=-6x+2y$.

Explanation of Method

Finding the unit vectors that an affine map sends to perpendicular vectors.

Step 1: Condition

Let $\vec v=\{a,b\}$ with $a^2+b^2=1$. A vector is transformed by the linear part only, so

\[ \vec v\,'=\{8a-4b,\ -6a+2b\}. \]

We require $\vec v\cdot\vec v\,'=0$: \[a(8a-4b)+b(-6a+2b)=8a^2-10ab+2b^2=0\ \Longrightarrow\ 4a^2-5ab+b^2=0.\]

Step 2: Solving

\[ 4a^2-5ab+b^2=(4a-b)(a-b)=0\ \Longrightarrow\ b=4a\quad\text{or}\quad b=a. \]

$b=4a$: $17a^2=1$, so $a=\pm\tfrac1{\sqrt{17}}$.
$b=a$: $2a^2=1$, so $a=\pm\tfrac1{\sqrt2}$.

\[\vec v=\pm\frac{1}{\sqrt{17}}\{1,4\}\qquad\text{or}\qquad \vec v=\pm\frac{1}{\sqrt2}\{1,1\}\]

Step 3: Check

$\vec v$	$\vec v\,'$	$\vec v\cdot\vec v\,'$
$\tfrac1{\sqrt{17}}\{1,4\}$	$\tfrac1{\sqrt{17}}\{-8,2\}$	$\tfrac{-8+8}{17}=0$
$\tfrac1{\sqrt2}\{1,1\}$	$\tfrac1{\sqrt2}\{4,-4\}$	$\tfrac{4-4}{2}=0$

The opposite vectors work as well, since $\vec v\mapsto-\vec v$ changes both factors of the dot product. In total there are four unit vectors: two directions, each with both orientations.

$\vec v_1$ $\mathcal A\vec v_1$ $\vec v_3$ $\mathcal A\vec v_3$

Unit vectors $\vec v_1=\tfrac1{\sqrt{17}}(1,4)$ and $\vec v_3=\tfrac1{\sqrt2}(1,1)$ (solid) and their images (dashed): each image is perpendicular to its vector. The opposite vectors $-\vec v_1,-\vec v_3$ behave the same way.

Result

\[\boxed{\,\vec v=\pm\frac{1}{\sqrt{17}}\{1,4\}\qquad\text{or}\qquad \vec v=\pm\frac{1}{\sqrt2}\{1,1\}\,}\]

Remark

In matrix form the condition is $\vec v^{\,T}A\vec v=0$, with only the symmetric part $\tfrac12(A+A^T)=\begin{pmatrix}8&-5\\-5&2\end{pmatrix}$ mattering. This form is indefinite (its determinant is $-9\lt 0$), so exactly two such directions exist.