Machine Learning Interviews Book
Last modified on November 07, 2021

1. [E] What’s the geometric interpretation of the dot product of two vectors?
   The dot product \(\mathbf{x} \cdot \mathbf{y}\) is the length of the projection of \(\mathbf{x}\) onto the unit vector \(\mathbf{\hat{y}}\); algebraically this is equal to \(\lvert\lvert \mathbf{x} \rvert\rvert \cos(\theta)\), where \(\theta\) is the angle between the vectors. Intuitively, the more the two vectors point in the same direction, the higher the dot product will be, as can be seen from the graphic below (where \(DP\) shows the value of the dot product between \(a\) and \(b\).) When the vectors are orthogonal, the dot product is zero.
   

   
   

2. [E] Given a vector \(u\), find vector \(v\) of unit length such that the dot product of \(u\) and \(v\) is maximum.
   This would simply be the unit vector in the direction of \(u\); namely, \(v = \frac{u}{\lvert\lvert u \rvert\rvert}\).
   

1. [E] Given two vectors \(a=[3,2,1]\) and \(b=[−1,0,1]\), calculate the outer product \(a^T b\).
   \(a^T b = \begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 1 \end{bmatrix} = \begin{bmatrix} -3 & 0 & 3 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{bmatrix}\)
   

   We could also calculate this using numpy, like so:
   

   import numpy as np
   a = np.array([3,2,1])
   b = np.array([-1,0,1])
   print(np.outer(a, b))
   

   [[-3  0  3]
    [-2  0  2]
    [-1  0  1]]
   

2. [M] Give an example of how the outer product can be useful in ML.
   Potentially…in some sort of collaborative filtering system, you’d want an outer product between \(m\) user vectors and \(n\) item vectors, to produce a \(m \times n\) similarity matrix?
   

   Many applications seem to use the Kronecker product (generalization of the outer product to tensors) – don’t quite understand them yet, but here are a few:
   

   • Optimizing Neural Networks with Kronecker-factored Approximate Curvature
     
   • Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters
     

   There has to be something simpler that I’m missing.
   

[E] What does it mean for two vectors to be linearly independent?
A set of vectors is linearly independent if there is no way to nontrivially combine them into the zero vector. What this means is that there should e no way for the vectors to “cancel each other out,” so to speak; there are no “redundant” vectors.

For only two vectors, this occurs iff (1) neither of them is the zero vector (otherwise the other vector could easily be canceled out by a scalar multiple of \(0\)), and (2) they are not scalar multiples of each other (otherwise they could simply cancel each other out by multiplying one by a scalar such that it points in the opposite direction as the other.)

1. [E] What’s a norm? What is \(L_0\), \(L_1\), \(L_2\), \(L_{norm}\)?
   Can be thought of as a vector’s distance from the origin. This becomes relevant in machine learning, when we want to objectively measure and minimize a scalar distance between the ground truth labels and model predictions.
   

   So first, we get the vector distance between the ground truth and prediction, which we can call \(x\). Then, we pass is through one of these norms to obtain a scalar distance:
   

   \(L_0\) “norm” is simply the total number of nonzero elements in a vector. If you consider \(0^0 = 0\), then
   \[L_0(x) = \left|x_{1}\right|^{0}+\left|x_{2}\right|^{0}+\cdots+\left|x_{n}\right|^{0}\]
   \(L_1\) norm: add up the absolute values of the elements.
   \[L_1(x) = \left|x_{1}\right|^{1}+\left|x_{2}\right|^{1}+\cdots+\left|x_{n}\right|^{1}\]
   \(L_2\) norm: add up the squares of the elements.
   \[L_2(x) = \left|x_{1}\right|^{2}+\left|x_{2}\right|^{2}+\cdots+\left|x_{n}\right|^{2}\]
   …and so on.
   \(L_\infty\) norm is the equal to the maximum absolute value of a component of \(x\). That is:
   \[L_\infty = \max_i |x_i|\]
   

   I can’t find anything on \(L_{norm}\), so maybe that was a typo?
   

2. [M] How do norm and metric differ? Given a norm, make a metric. Given a metric, can we make a norm?
   My understanding is that a norm is simply a mathematical operation on a vector, whereas a (loss) metric is measuring the distance between two things. Given a norm, let’s take the \(L_2\) norm, we can easily make a metric, by plugging in a distance vector between ground truth and prediction, as shown in the previous question, and summing the squares of the components. A norm takes in a single variable (i.e. a vector,) whereas a metric takes in two (i.e. ground truth vector and predictions vector.)
   

1. [M] Find \(x\) such that \(Ax = b\).
   
2. [E] When does this have a unique solution?
   
3. [M] Why is it when \(A\) has more columns than rows, \(Ax = b\) has multiple solutions?
   
4. [M] Given a matrix \(A\) with no inverse, how would you solve the equation \(Ax = b\)? What is the pseudoinverse and how to calculate it?
   

1. [E] What does the derivative represent?
   
2. [M] What’s the difference between derivative, gradient, Jacobian?