Here are various notes on 3D rotations, including converting from axis-angle to vector and back.

(This page requires you to enable Javascript and use a reasonable processor since the equations are rendered locally.)

The Purpose of this Page

There are several different ways to look at rotations. A general rule is, that when there several ways of doing something, then none of the ways is enormously better than the others. Otherwise, we'd never talk about the poorer methods.
Rotation with an axis an angle is easy to understand.
Rotation by a matrix is easy to combine with other rotations and other transformations.
Rotation with quaternions is easy to animate, and also easy to combine several rotations, and is less sensitive to roundoff errors.
Rotation with Euler angles is liked by mechanical engineers, perhaps because gimbal rings work like that.
You can convert back and forth between methods.

Rotating a Point About a Given Axis

To rotate a point {$p$} by an angle {$\theta$} about an axis {$a$}, where {$|a|=1$}:

{$$\color{red}{\begin{eqnarray}p'& =& (a \cdot p) \, a + (p - (a \cdot p) a)\cos \theta+ (a \times p)\, \sin \theta\\ &=& p \cos \theta + (a \cdot p)\, a (1-\cos \theta) + (a \times p) \,\sin \theta\end{eqnarray}} $$}

That was derived as follows:

Decompose {$p$} into {$p_\|= (a\cdot p) \, a$}, a component parallel to {$a$}, and {$p_\bot = p - p_\|$}, a component perpendicular to {$a$}.
{$p_\|$} doesn't change when rotated: {$p_\|' = p_\|$}
{$p_\bot$} rotates in a plane perpendicular to {$a$}. {$p_\bot$} is like the x-axis, {$a$} is like the y-axis, and {$a\times p_\bot = a\times p$} is like the z-axis.
{$p_\bot' = p_\bot \cos\theta + a\times p \sin\theta$}
{$p'=p_\|'+p_\bot'$}

Example

Suppose that we want to rotate the point {$p=(3,2,1)$} by {$45^\circ$} about the axis {$a=(1,0,0)$}. Note that the length of {$a$} is {$1$}, but the length of {$p$} can be anything.

{$a\cdot p = 3$}, so {$p_\|= a\cdot p \, a = 3a = (3,0,0)$}.

{$p_\bot = p - p_\| = (3,2,1) - (3,0,0) = (0,2,1) $}

{$p_\|' = p_\| = (3,0,0)$}

{$ (0,2,1) $} rotates in a plane perpendicular to {$a$}, a plane defined by {$ (0,2,1) $} and {$ (1,0,0)\times(0,2,1)=(0,-1,2) $}

{$p_\bot' = \cos\theta (0,2,1) + \sin\theta (0,-1,2) = (0,1.4,.7) + (0,-.7,1.4) = (0,.7,2.1)$}

{$p' = (3,0,0) + (0,.7,2.1) = (3,.7,2.1)$}. Note that its length is the same as for {$p$}.

Example 2

Suppose we want to rotate the point {$p=(1,0,0)$} about the axis {$a=(1,1,1)$} by the angle {$120^\circ$}.

Since {$|a|=\sqrt{3} $}, we need to normalize it to {$\hat{a}=\left(\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}}\right)$}.
The component of {$p$} parallel to {$\hat{a}$} is {$p_\|=(p\cdot\hat{a}) \hat{a}=\frac{1}{\sqrt{3}}\left(\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}},\frac{1}{\sqrt{3}}\right)=\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$}.
It doesn't change when rotated.
The component of {$p$} perpendicular to {$\hat{a}$} is the rest of {$p$}, i.e., {$p_\bot=(1,0,0)-\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)=\left(\frac{2}{3},-\frac{1}{3},-\frac{1}{3}\right)$}
{$p_\bot$} rotates in a plane perpendicular to {$a$}, where {$p_\bot$} acts as the x-axis and the y-axis is played by {$\hat{a}\times p_\bot= \left(0,\frac{1}{\sqrt{3}},-\frac{1}{\sqrt{3}}\right)$}
Note that {$\cos120^\circ= -\frac{1}{2}$} and {$\sin120^\circ=\frac{\sqrt{3}}{2}$}
{$p_\bot'=-\frac{1}{2}\left(\frac{2}{3},-\frac{1}{3},-\frac{1}{3}\right)+\frac{\sqrt{3}}{2}\left(0,\frac{1}{\sqrt{3}},-\frac{1}{\sqrt{3}}\right)$} {$=\left(-\frac{1}{3},\frac{2}{3},-\frac{1}{3}\right)$}
{$p'=\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)+\left(-\frac{1}{3},\frac{2}{3},-\frac{1}{3}\right)=(0,1,0)$}
That makes sense. Rotating a point on the x-axis by {$120^\circ$} about (1,1,1) gives a point on the y-axis.

Converting the Vector Form To a Matrix Form

The problem is that the formula {$\color{blue}{p}' = (\color{red}{a} \cdot \color{blue}{p})\, \color{red}{a} + (\color{blue}{p} - (\color{red}{a} \cdot \color{blue}{p})\, \color{red}{a})\color{red}{\cos \theta}+ \color{red}{a} \times \color{blue}{p}\, \color{red}{\sin \theta} $} mixes up the rotation definition ({$ \color{red}{a} $} and {$\color{red}{\theta}$}) and the object ({$ \color{blue}{p} $}) throughout the formula. We want to separate the formula into a part depending on the rotation and another part depending on the object, which interact simply:

{$ \color{blue}{p}' = \color{red}{M_{a,\theta}}\cdot\color{blue}{p} $}

{$ \color{red}{M_{a,\theta}} $} means that {$ \color{red}{M} $} is a matrix that depends only on {$\color{red}{a,\theta} $}.

To convert this to a matrix form, {$p'=M p$}, where {$M$} is a function of {$a$} and {$\theta$}, use these. The problem they solve is as follows. Suppose we want to rotate an object by {$45^\circ$} about the axis {$(1,2,3)$}. We want some matrix {$M$}, which depends only on {$45^\circ$} and {$(1,2,3)$}. Then we will rotate the object by multiplying its points by {$M$}. In fact, {$M$} would be this matrix:

{$\left(\begin{array}{ccc} 0.07142857142 & - 0.6589265829 & 0.7488081981 \\ 0.9446408685 & 0.2857142857 & 0.1613101866 \\ - 0.3202367695 & 0.6958326704 & 0.6428571428 \end{array}\right)$}

Now, instead of {$45^\circ$} and {$(1,2,3)$}, think of a general angle, {$\theta$}, and axis, {$a$}.

We will work up to {$M$} in stages. First, is to find a matrix that depends only on {$a$} such that multiplying it by point {$p$} is the same as calculating {$a\cdot p\, a$}. Here it is.

{$$\left(\begin{array}{ccc} {a_1} ^2 & a_1 a_2 & a_1 a_3 \\ a_1 a_2 & {a_2} ^2 & a_2 a_3 \\ a_1 a_3 & a_2 a_3 & {a_3} ^2 \\ \end{array}\right) $$}

This matrix has the following property: {$$ (\color{red}{a \cdot p) \, a = \left(\begin{array}{ccc} {a_1} ^2 & a_1 a_2 & a_1 a_3 \\ a_1 a_2 & {a_2} ^2 & a_2 a_3 \\ a_1 a_3 & a_2 a_3 & {a_3} ^2 \end{array}\right) \left(\begin{array}{c} p_1 \\ p_2 \\ p_3 \end{array}\right)} $$}

Next, this matrix:

{$$\left(\begin{array}{ccc} 0 & - a_3 & a_2 \\ a_3 & 0 & - a_1 \\ - a_2 & a_1 & 0 \end{array}\right) $$}

has this useful property:

{$$ \color{red}{a \times p = \left(\begin{array}{ccc} 0 & - a_3 & a_2 \\ a_3 & 0 & - a_1 \\ - a_2 & a_1 & 0 \end{array}\right) \left(\begin{array}{c} p_1 \\ p_2 \\ p_3 \end{array}\right) }$$}

With them, we can find a matrix {$M$} to turn equation

{$$ p' = p \cos \theta + a \cdot p\, a (1- \cos \theta) + a \times p \sin \theta$$}

into a matrix multiplication, {$p' = Mp$}, where

{$$\color{red}{M= \left( \begin {array}{ccc} \cos\theta+ \left( 1-\cos\theta \right) {{a_1}}^{2}& \left( 1-\cos\theta \right) {a_1}\,{a_2}-\sin\theta{a_3}& \left( 1-\cos\theta \right) { a_1}\,{a_3}+\sin\theta{a_2}\\{} \left( 1-\cos\theta \right) { a_1}\,{a_2}+\sin\theta{a_3}&\cos\theta+ \left( 1-\cos\theta \right) {{a_2}}^{2}& \left( 1-\cos\theta \right) {a_2}\,{a_3}-\sin\theta{a_1}\\{} \left( 1-\cos\theta \right) {a_1}\,{a_3}-\sin\theta{a_2}& \left( 1-\cos\theta \right) { a_2}\,{a_3}+\sin\theta{a_1}&\cos\theta+ \left( 1-\cos\theta \right) {{a_3}}^{2}\end {array} \right) }$$}

It is complicated enough that you don't want to try memorizing it or copying it by hand. Even one widely-used text, Foley, got one sign wrong in one edition. I created the matrix with my Maple rotmat routines, and wrote it in LaTeX format.

Examples

If {$a=(1,2,3)$} then write {$(a\cdot p)\, a$} as a matrix depending on {$a$} times {$p$}. {$$\color{blue}{(a\cdot p)\, a = \begin{pmatrix} 1&2&3\\2&4&6\\3&6&9\end{pmatrix} p} $$}
Write {$a\times p$} ditto. {$$\color{blue}{a\times p = \begin{pmatrix}0&-3&2\\3&0&-1\\-2&1&0\end{pmatrix}p} $$}

Telling If a Matrix is a Rotation Matrix

Since a rotation preserves distances, and therefore preserves angles, then, if {$r_i$} is the i-th row, and {$c_i$} is the i-th column, then

{$$ \color{red}{r_i \cdot r_j = c_i \cdot c_j = \delta_{ij}} $$}

{$\delta_{ij}$} is defined as 1 if {$i=j$} and 0 otherwise.

Finally, {$$\color{red}{\det(M) = 1} $$}

Here is an explanation of these tests for whether a matrix, {$M$}, is a rotation.

{$\det(M)=1$} since the determinant is how much the volume of a small cube scales.
The length of the first column, when treating that as a vector, is one, since that is the vector (1,0,0) after the transformation. Ditto the other columns.
The columns are perpendicular to each other since the transformed axes must still be perpendicular.

Examples

The following matrix {$$ \begin{pmatrix} 1&1&0\\0&1&0\\0&1&1\end{pmatrix} $$} is not a rotation matrix because the 2nd column has length {$\sqrt{3}\ne1$}.
The following matrix {$$ \begin{pmatrix} 1&.6&0\\0&.8&0\\0&1&1\end{pmatrix} $$} is not a rotation matrix because the first 2 columns are not perpendicular.
The following matrix {$$ \begin{pmatrix} 1&0&0\\0&1&0\\0&0&-1\end{pmatrix} $$} is not a rotation matrix because its determinant is -1.

Finding the Axis and Angle of a Rotation Matrix

The eigenvalues are {$$\color{red}{1} \,\text{and}\, \color{red}{\cos\theta\pm i \sin \theta} $$}
If {$\theta = 180^\circ$}, then the eigenvalues will be 1, -1, -1.
The axis is the eigenvector corresponding to the eigenvalue of 1.
You can get the angle, apart from some ambiguity, from the other eigenvalues.
There is a complication because if you invert the axis and also negate the angle of rotation, you get the same rotation. This means that you can get two possible axis/angle combos from one matrix.
Here is another way to find the axis, if {$\theta$} is not {$0^\circ$} or {$180^\circ$}. If {$M=m_{ij}$} is the matrix and {$a$} is the normalized axis, then {$$ 2 a \sin\theta = (m_{32}-m_{23}, m_{13}-m_{31},m_{21}-m_{12})$$} Therefore, {$$\color{red}{ (m_{32}-m_{23}, m_{13}-m_{31},m_{21}-m_{12}) } $$} is an unnormalized axis, and its length is {$\color{red}{2\sin\theta}$} .
Here is another way to find the angle, up to an ambiguous sign. The trace of a matrix, which is the sum of the diagonal elements, is independent of the coordinate system used, so long as that is orthonormal. This implies that the trace is the sum of the eigenvalues. Here that is
{$1+\cos \theta + i \sin \theta + \cos \theta - \sin \theta = 1 + 2 \cos \theta $}
Therefore, {$$\color{red}{\theta = \arccos \left({\frac{m_{11}+m_{22}+m_{33}-1}{2}}\right)}$$} Note that this gives 2 possibilities.

Paper on Rotation Methods

See my paper summarizing rotation methods, Efficient Rotation of an Object, IEEE Transactions on Computing, Nov. 1983. It compares Euler angles, quaternions, and other methods. It shows how to optimize animation rotations using quaternions.

Maple example of 3D rotation

Here's an example of using Maple to combine two 90° rotations, one about the X-axis, the other about the Y-axis. The code is here.

      > read `rot_maple.code`;
        [dotprod]
        [crossprod]
	Warning: new definition for   norm
        [norm]
        [scalarmul]
	proc(expr) ... end

	rotate :=
    	proc(a,t,p)
    	local a2;
        a2 := normalize(a);
        evalm(cos(t)*p+(1-cos(t))*dotprod(a2,p)*a2+sin(t)*crossprod(a2,p))
    	end

	normalize := proc(a) local l; l := evalf(norm(a,2)); evalm(a/l) end

        [ 1  0  0 ]
        [         ]
        ii := [ 0  1  0 ]
        [         ]
        [ 0  0  1 ]

	rotmat := proc(a,t)
        local a2;
        a2 := normalize(a);
        evalm(cos(t)*ii+extprod(a2,a2)*(1-cos(t))+sin(t)*crossmat(a2))
        end

	rotmat2 := proc()
        local a,a1,a2,a3,c,s;
        a[1] := a1;
        a[2] := a2;
        a[3] := a3;
        evalm(c*ii+extprod(a,a)*(1-c)+s*crossmat(a))
        end

	extprod := proc(a,b)
        local m,i,j;
        m := array(1 .. 3,1 .. 3);
        for i to 3 do  for j to 3 do  m[i,j] := a[i]*b[j] od od;
        op(m)
        end

	crossmat :=
  	proc(a)
  	local m;
      	m := array(1 .. 3,1 .. 3,[[0,-a[3],a[2] ],[a[3],0,-a[1] ],[-a[2],a[1],0] ]);
      	op(m)
  	end
	----------------------------------------------------------------
	Find the rotation matrix for an axis of (1,0,0) and an angle of Pi/2:

      > r1:=evalf(rotmat([1,0,0],Pi/2));
        [ 1.   0   0  ]
        [             ]
        r1 := [  0   0  -1. ]
        [             ]
        [  0  1.   0  ]
	----------------------------------------------------------------
      > r2:=evalf(rotmat([0,1,0],Pi/2));
        [  0    0  1. ]
        [             ]
        r2 := [  0   1.   0 ]
        [             ]
        [ -1.   0   0 ]
	----------------------------------------------------------------
      > r12:=multiply(r1,r2);
        [  0   0  1. ]
        [            ]
        r12 := [ 1.   0   0 ]
        [            ]
        [  0  1.   0 ]
	----------------------------------------------------------------
      > r21:=multiply(r2,r1);
        [  0   1.   0  ]
        [              ]
        r21 := [  0    0  -1. ]
        [              ]
        [ -1.   0   0  ]
	----------------------------------------------------------------
      > eigenvects(r12);

  	[ - .4999999997 - .8660254037 I, 1,
      	{ [  - .7382716585 + .3487429165 I, .06711560405 - .8137334705 I,
        .6711560548 + .4649905518 I ] }                               ],
      	[ - .4999999997 + .8660254037 I, 1,
        { [  - .7382716585 - .3487429165 I, .06711560405 + .8137334705 I,
        .6711560548 - .4649905518 I ] }                               ],
      	[1.000000000, 1, { [ -.5773502687, -.5773502691, -.5773502691 ] }]

	This represents a 120 degree rotation about the axis 
	[ -.5773502687, -.5773502691, -.5773502691 ] .

	----------------------------------------------------------------
      > eigenvects(r21);

   	[ - .4999999997 - .8660254037 I, 1,
       	{ [ .7382716585 - .3487429165 I,  - .6711560548 - .4649905518 I,
        .06711560405 - .8137334705 I ] }                             ],
       	[ - .4999999997 + .8660254037 I, 1,
        { [ .7382716585 + .3487429165 I,  - .6711560548 + .4649905518 I,
        .06711560405 + .8137334705 I ] }                             ],
       	[1.000000000, 1, { [ .5773502687, .5773502691, -.5773502691 ] }]

	This represents a 120 degree rotation about the axis 
	[ .5773502687, .5773502691, -.5773502691 ], which is different, so the
	order does matter.

4D Rotations

Rotations in 4d and other dimensions.

Other 3D Rotation Methods

Quaternions
Euler angles - very bad for animation, suffers gimbal lock. This is a problem where you cannot always achieve a smooth path by smoothly varying the parameters. Here's the idea in a simpler domain: traveling on the surface of the earth, and using latitude ({$\phi$}) and longitude ({$\lambda$}) as your coordinates.
Assume that your path {$p$} crosses the north pole with constant or smoothly varying velocity, so {$d^2p/dt^2$} is finite. {$\phi$} increases smoothly to {$90^o$} and then suddenly starts decreasing. I.e., {$d^2\phi/dt^2$} is discontinuous. {$\lambda$} is worse. {$p$} is curved, {$d\lambda/dt$} increases to infinity as {$\phi\rightarrow90^o$}.
You might think this is an artifact of latitude and longitude; that maybe you can find a better 2D coordinate system on the surface of a sphere. However, you cannot. The only way to have coordinates that always vary smoothly when your path does is to add an extra coordinate, and use {$(x,y,z)$} with the restriction that {$x^2+y^2+z^2=R^2$}.
3D rotations have this same problem, but in a 3D space rather than 2D.

Degrees of Freedom and Number of Coefficients

A 3D rotation has 3 degrees of freedom.
Representing a rotation with Euler angles uses 3 coefficients, but Euler angles have problems.
Matrices use 9 coefficients. When you multiply rotation matrices, roundoff error means that the result is not exactly a rotation. Projecting the matrix to the closest rotation matrix is a little messy.
Quaternions use 4 coefficients, which is the sweet spot for number of coefficients. Fewer coefficients means less chance for trouble. Projecting a non-rotation quaternion to a close rotation quaternion is easy.