Supervised learning uses data with both covariates and response values, while unsupervised does not use response values.

Parametric models make assumptions regarding the shape and/or properties of \(f\), while non-parametric models do not. Parametric models can make better predictions with little data, but the model assumptions might be wrong. Non-parametric models might give you insight in the data when nothing is known a priori, but often requires more data.

Regression models continuous data, while classification models categorical data, i.e. class membership.

\[\begin{equation*} \mathrm{MSE} = \frac{1}{n} \sum \limits_{i = 1}^{n} (\hat{f}(x_i) - y_i)^2 \end{equation*}\]

Note that this only makes sense for continuous data.

\[\begin{equation*} E[Y - \hat{f}(x_0)]^2 \end{equation*}\]

It is helpful to remember the following properties:

• \(\mathrm{Var}(Y) = E[Y^2] - E[Y]^2\)
• \(Y\) and \(f(x_0)\) can be considered independent.
• \(E[Y] = f(x_0)\).

Using this, we can derive the expression:

\[\begin{align*} E[(Y - \hat{f}(x_0))^2] &= E[Y^{2} + \hat{f}(x_0)^2 - 2Y\hat{f}(x_0)] \\ &= E[Y^{2}] + E[\hat{f}(x_0)^2] - E[2Y\hat{f}(x_0)] \\ &= \mathrm{Var}(Y) + E[Y]^2 + \mathrm{Var}(\hat{f}(x_0)) + E[\hat{f}(x_0)]^2 - 2E[Y]E[\hat{f}(x_0)] \\ &= \mathrm{Var}(\epsilon) + f(x_0)^2 + \mathrm{Var}(\hat{f}(x_0)) + E[\hat{f}(x_0)]^2 - 2f(x_0)E[\hat{f}(x_0)] \\ &= \mathrm{Var}(\epsilon) + \mathrm{Var}(\hat{f}(x_0)) + (f(x_0) - E[\hat{f}(x_0)])^2 \\ &= \text{irreducible error} + \text{model variance} + \text{bias}^2 \end{align*}\]

The interpretation is as follows:

• Irreducible error is intrinsic to the data and can not be removed, regardless of model choice.
• Model variance is a measure of how much the model predictions are dependent on the training data. It is the expected deviation around the mean \(x_0\).
• The bias is a measure of expected square euclidean distance from the actual true mean. It measures how close the model is to the true values.

• Logistic regression: Increasing the number of regression coefficients decreases the bias.
• KNN: Decreasing \(K\) decreases the bias.
• Single hidden-layer neural net: Increasing the number of hidden nodes decreases the bias.
• Lasso regression: Decreasing the penalty parameter \(\lambda\) decreases the bias.
• Polynomial interpolation: Increasing the polynomial degree decreases the bias.

Predicting the grade of a student based on earlier grades, time investment, class size, study programme, and so on. The response variable is therefore categorical and ordinal, \(y \in \mathcal{C} = \{A, B, C, D, E, F\}\).

\[\begin{equation*} \frac{1}{n} \sum \limits_{i = 1}^{n} I(\hat{f}(x_i) \neq y_i) \end{equation*}\]

Assume that you are able to derive the conditional class probability \(p_k(x) = P(Y = k \mid X = x)\), given classes \(k \in \mathcal{C}\). The Bayes classifier assigns the class of greatest conditional probability for the given predictor:

\[\begin{equation*} \hat{f}(x_0) = \underset{k \in \mathcal{C}}{\operatorname{argmax}}~p_k(x_0) \end{equation*}\]

PS: For a two-class problem this implies a cut-off probability of \(0.5\).

\[\begin{equation*} 1 - E_X\left[ \underset{k \in \mathcal{C}}{\operatorname{max}} p_k(x) \right] \end{equation*}\]

The conditional distribution \(P(Y = j \mid X)\) is seldom known.

\[\begin{equation*} P(Y = j \mid X = x_0) = \frac{1}{K} \sum \limits_{i \in \mathcal{N}_0} I(y_i = j) \end{equation*}\]

Where \(\mathcal{N}_0\) is the set of the \(K\) closest neighbours to \(x_0\), i.e. \(|\mathcal{N}_0| = K\). The predictor becomes the “majority vote” of the \(K\) closest neighbours.

The KNN-algorithm excels at data sets which contains lots of observations and few covariates, i.e. \(p << n\). When the dimensionality increases, the distances between the observations “blow up” and the model suffers as a consequence because it can no longer be considered local.

A flexible model is able to adapts its “shape” according to training data quite easily. An inflexible model is more rigid, it only allows certain “shapes” to be modelled.

• Linear regression is a relatively inflexible model, as it assumes a linear correlation between the covariates and the continuous response.
• KNN regression and classification is a flexible model, as it makes no assumptions upon the shape of the data. It is simply a “majority vote” of the neighbours.

This often comes down to a compromise between the adaptability and interpretability of \(f\).

• An underfitted model makes wrong predictions upon its own training data, and will therefore probably not perform well on additional testing data. It is too rigid in order to capture the underlying structure of the data.
• An overfitted model fits its training data too closely. It has therefore weighted random noise too much, and will perform poorly on additional testing data.

For two independent variables, we have:

\[\begin{equation*} E[\mathbf{X} + \mathbf{Y}] = E[\mathbf{X}] + E[\mathbf{Y}] \end{equation*}\]

This relation is also applicable for dependent variables because it is a linear operator.

\[\begin{equation*} E[\mathbf{AXB}] = \mathbf{A} E[\mathbf{X}] \mathbf{B} \end{equation*}\]

\[\begin{equation*} \mathrm{Cov}(X_i, X_j) := E[(X_i - \mu_i)(X_j - \mu_j)] = E[X_i X_j] - \mu_i \mu_j \end{equation*}\]

\[\begin{align*} \Sigma &= \mathrm{Cov}(\mathbf{X}) \\ &:= E[(X - \mu)(X - \mu)^T] \\ &= E[XX^T] - \mu \mu^T \end{align*}\]

This matrix is real, symmetric, and positive semidefinite.

We define the matrix element wise as:

\[\begin{equation*} \rho_{ij} = \frac{\sigma_{ij}}{\sqrt{2 \sigma_i \sigma_j}} \end{equation*}\]

Or in matrix-form:

\[\begin{equation*} \rho = (\mathbf{V}^{1/2})^{-1} \Sigma (\mathbf{V}^{1/2})^{-1} \end{equation*}\]

Where:

\[\begin{equation*} \mathbf{V}^{1/2} := \text{diag}(\sqrt{\sigma_{11}}, \sqrt{\sigma_{22}}, ...) \end{equation*}\]

\[\begin{align*} E[\mathbf{Z}] &= E[\mathbf{AX}] = \mathbf{A} E[\mathbf{X}] = \mathbf{A} \mu \\ \mathrm{Cov}(\mathbf{Z}) &= \mathrm{Cov}(\mathbf{AX}) = \mathbf{A} \mathrm{Cov}(\mathbf{X}) \mathbf{A}^T = \mathbf{A \Sigma A^T} \end{align*}\]

B

A

B

A

D

C

Multivariate case:

\[\begin{equation*} f(\mathbf{x}) = \frac{1}{(2\pi)^{p/2} |\Sigma|^{1/2}} \mathrm{exp}\left\{ - \frac{1}{2} (\mathbf{x} - \mathbf{\mu})^T \Sigma^{-1} (\mathbf{x} - \mathbf{\mu}) \right\} \end{equation*}\]

They form ellipsoids.

True.

All \(p\) components are independent of each other.

\[\begin{equation*} A \Sigma B^T = 0 \Longleftrightarrow \text{independence} \end{equation*}\]

The conditional is also multivariat normally distributed:

\[\begin{equation*} \mathbf{X}_2 \mid (\mathbf{X}_1 = \mathbf{x}_1) \sim \mathcal{N}_{p_2}\left( \mu_2 + \Sigma_{21} \Sigma_{11}^{-1}(\mathbf{x}_1 - \mu_1), \Sigma_{22} - \Sigma_{21} \Sigma_{11}^{-1} \Sigma_{12} \right) \end{equation*}\]

• Center at \(\mathbf{\mu}\).
• The axes point in the direction of the eigenvectors of \(\Sigma\).
• The magnitudes are \(\pm \sqrt{b \lambda}\), where \(\lambda\) are the eigenvalues of \(\Sigma\).
• \((\mathbf{x}-\mathbf{\mu})^T\Sigma^{-1}(\mathbf{x}-\mathbf{\mu})\) is distributed as \(\chi_p^2\). Solve \(b = {\chi^2_p}(\alpha)\) for \(\alpha\). Then the volume has a probability of \(1 - \alpha\).

A

D

B

C

B

C

This is a linear regression model. It is supervised and parametric.

This linear regression model assumes the following relationship:

\[\begin{align*} Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \end{align*}\]

The terms are:

• \(Y\) is here the circumference of the tree in millimeters
• The intercept \(\beta_0\) is the width of the tree at 1968/12/31.
• The regression coefficient \(\beta_1\) is the growth rate of the trees in millimeters per day.
• \(x\) are the number of days since 1968/12/31.
• \(\epsilon\) is a random error term, also called measurement noise.

PS: If all the trees were planted at 1968/12/31, we should remove the intercept from the model. How do we do this?

In linear regression, the following assumptions are made:

• The observation pairs \((Y_i, x_i)\) are independent of each other.
• The error terms have mean zero, i.e. \(E[\epsilon_i] = 0\) and have constant variance \(\mathrm{Var}(\epsilon_i) = \sigma^2\).

Additionally, in normal linear regression, we assume:

• The error terms are normally distributed, \(\epsilon_i \sim \mathcal{N}(0, \sigma^2)\).

A residual is the difference between the predicted value and the observed value.

\[\begin{equation*} e_i := Y_i - \hat{Y}_i \end{equation*}\]

The blue line gives the estimated model:

\[\begin{equation*} \hat{Y_i} = \hat{\beta}_0 + \hat{\beta}_1 x_i \end{equation*}\]

The dots are the observed values, the color is dependent on which tree that has been measured. We have five such trees. The red lines show the residuals, \(e_i\).

The residual sum of squares (RSS) is the sum of all the squared residuals:

\[\begin{align*} \mathrm{RSS} &:= \sum \limits_{i = 1}^{n} e_i^2 \\ &= \sum \limits_{i = 1}^{n} (Y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2 \end{align*}\]

The regression coefficients are chosen such that the RSS is minimized:

\[\begin{equation*} \hat{\beta}_0, \hat{\beta}_1 = \underset{(\hat{\beta}_0, \hat{\beta}_1) \in \mathbb{R}^2}{\operatorname{argmax}} RSS \end{equation*}\]

In practice, this yields the following formulas:

\[\begin{align*} \hat{\beta}_0 &= \bar{Y} - \hat{\beta}_1 \bar{x} \\ \hat{\beta}_1 &= \frac{ \sum \limits_{i = 1}^{n} (x_i - \bar{x}) (Y_i - \bar{Y})}{ \sum \limits_{i = 1}^{n} (x_i - \bar{x})^2} \end{align*}\]

\[\begin{align*} \mathrm{Var}(\hat{\beta}_0) &= \sigma^2 \left[ \frac{1}{n} + \frac{\bar{x}^2}{ \sum \limits_{i = 1}^{n} (x_i - \bar{x})^2} \right] \\ \mathrm{Var}(\hat{\beta}_1) &= \frac{\sigma^2}{ \sum \limits_{i = 1}^{n} (x_i - \bar{x})^2} \end{align*}\]

We prefer small variances in the regression coefficient, since this makes the predictor for \(\beta\) better. We observe the larger the sample size, \(n \rightarrow \infty\), then the variance goes to zero, which is good.

No, this is not the same. The formula is:

\[\begin{equation*} \mathrm{RSE} := \sqrt{\frac{\mathrm{RSS}}{n - 2}} \end{equation*}\]

We can see that RSE is scaled according to the number of observations, while RSS is not. RSE is used as an estimator for the error terms \(\epsilon\), using the residuals \(e\). This is the so-called restricted maximum likelihood estimator, and is unbiased.

If we assume \(\epsilon \sim \mathcal{N}(0, \sigma^2)\), then:

\[\begin{equation*} \frac{\mathrm{RSE}^2 (n - 2)}{\sigma^2} = \frac{ \sum \limits_{i = 1}^{n} (Y_i - \hat{Y}_i)^2}{\sigma^2} \sim \chi_{n - 2}^2 \end{equation*}\]

One of the assumptions of linear regression is that \(Y_i\) and \(\epsilon_i\) should be independent of each other. This plot can be used if this does not hold, i.e. if there is a trend between the residuals and the fitted values. Ideally, we would like to see a symmetric distribution above and below the zero line, and constant variance.

This plot can be used to check the normality assumption of \(\epsilon\), using the standardized residuals as an estimator for the error terms. Ideally we would like to see:

• A completely straight line, indicating a bell curve shape.
• A line passing through \((0, 0)\), indicating zero mean.
• A slope equal to one.

The last point regarding the slope only applies for standardized residuals.

Since \(Y\) is considered to be normally distributed conditioned on the predictors, \(\hat{\beta}\) as a linear combination of \(Y\) should also be normally distributed. It is an unbiased estimator, so \(E[\hat{\beta}_j] = \beta_j\).

\[\begin{equation*} \hat{\beta}_j \sim \mathcal{N}(\beta_j, \mathrm{Var}(\hat{\beta}_j)) \end{equation*}\]

Since \(\hat{\beta}_j\) is normally distributed with an unknown standard deviation, we end up with a \(T\)-statistic with \(n - 2\) degrees of freedom:

\[\begin{equation*} \frac{\hat{\beta}_j - \beta_j}{\widehat{\mathrm{SE}}(\hat{\beta}_j)} \sim T_{n - 2} \end{equation*}\]

Using the test statistic from the last question, a \(1 - \alpha\) confidence interval can be constructed:

\[\begin{equation*} \hat{\beta}_j \pm t_{\alpha / 2,~n - 2} \cdot \widehat{\mathrm{SE}}(\hat{\beta}_j) \end{equation*}\]

A regression coefficient is considered to be significant if we are reasonable sure of the fact that it is not zero, i.e. that the respective covariate actually is correlated with the response. The formal hypothesis test becomes:

\[\begin{align*} H_0: \beta_j = 0 & vs. & H_1: \beta_j \neq = 0 \end{align*}\]

This is a two sided hypothesis test, and we can test it by using a confidence interval. If a \(1 - \alpha\) confidence interval does not contain \(0\), we can conclude with \(1 - \alpha\) confidence that the regression coefficient is significant. The \(p\)-value gives the probability of observing the data we have in a world where the null-hypothesis is true. If the \(p\)-value is small enough, it strengthens our belief in the regression coefficient being significant.

The total sum of squares is defined as:

\[\begin{equation*} \mathrm{TSS} := \sum \limits_{i = 1}^{n} (Y_i - \bar{Y})^2 \end{equation*}\]

It is proportional to the estimated variance of the response variable.

The \(R^2\) statistic is defined as:

\[\begin{align*} R^2 &:= \frac{\mathrm{TSS} - \mathrm{RSS}}{\mathrm{TSS}} \\ = \frac{\mathrm{RSS}}{\mathrm{TSS}} \end{align*}\]

We have \(R^2 \in [0, 1]\), and it can be interpreted as the fraction of the variance in the data explained by the model.

This shows summary statistics for \(e_i\). Here we would like the median to be \(0\), and the other statistics to be symmetric.

This shows the estimators for \(\beta_0\) and \(\beta_1\). The model predicts a tree to start with a 17 millimeter circumference at 1968/12/31, and the circumference to grow 0.1 millimeters each day.

This shows the estimators \(\widehat{\mathrm{SE}}(\beta_0)\) and \(\widehat{\mathrm{SE}}(\beta_1)\).

This shows the calculated \(T\)-statistic to check if the regression coefficients are significant or not. The respective \(p\)-values are also shown.

This shows the calculated value of RSE, with \(n - 2\) degrees of freedom.

Calculated values for \(R^2\) and \(R^2_{\text{adj}}\), the closer to 1, the better. The latter accounts for overfitting.

Shows the \(F\)-statistic for the entire model, checks if the model is better than just a null-model (which only takes the mean into account).

The underlying distributional assumption is

\[ \log(\text{FEV}) = Y = f(x) + \epsilon = x^T \beta + \epsilon = \beta_0 + \beta_{\text{age}}x_{\text{age}} + \beta_{\text{height}}x_{\text{height}} + \beta_{\text{male}}x_{\text{male}} + \beta_{\text{smoke}}x_{\text{smoke}} + \epsilon \]

Where we assume \(\epsilon \sim N_{n}(0, \sigma^2 I)\). The fitted model is \(\hat{Y} = x^T \hat{\beta}\).

How does the model’s FEV prediction change from \(\text{FEV}_{\text{before}}\) to \(\text{FEV}_{\text{after}}\) in such a case?

\[ \text{FEV}_{after} = e^{x^T \hat\beta} = e^{\hat{\beta}_0 + x_1 \hat{\beta}_1 + ... + (x_j + 1) \hat\beta_j + ... \hat{\beta}_p x_p} = e^{\hat \beta_j} e^{x^T \hat \beta} = e^{\hat \beta_j} \text{FEV}_{before} \]

The exponential of the coefficient estimates represents the multiplicative change of the model prediction as the respective covariate changes.

\[\begin{align*} \hat{\sigma}^2 &= \frac{\sum_{i = 1}^n e_i^2}{n - p - 1} = \frac{\text{RSS}}{n - p - 1} \\ e_i := Y_i - \hat{Y}_i = Y_i - x_i^T \hat{\beta} \end{align*}\]

The estimation of the standard error of the parameter estimation for \(\beta_j\) can be expressed as

\[ \hat{\mathrm{SE}}(\hat{\beta}_i) = \sqrt{[(X^TX)^{-1}]_{[i,i]}} ~ \hat{\sigma} := \sqrt{c_{jj}} ~ \hat{\sigma}, \]

where \(c_{jj}\) is the j’th diagonal entry of the matrix \((X^T X)^{-1}\).

\[ \mathrm{RSS} = \sum_{i = 1}^{n} e_i^2 = \sum_{i = 1}^{n} (Y_i - \hat{Y_i})^2 = \sum_{i = 1}^{n} (Y_i - x_i^T \hat{\beta})^2 = (Y - X\hat{\beta})^T (Y - X\hat{\beta}) \]

Using the \(T\)-statistic, you should end up with:

\[\begin{equation*} [{\bf x}_0^T \hat{\beta}-t_{\alpha/2,n-p-1}\hat{\sigma}\sqrt{1+{\bf x}_0^T({\bf X}^T{\bf X})^{-1}{\bf x}_0},~{\bf x}_0^T \hat{\beta}+t_{\alpha/2,n-p-1}\hat{\sigma}\sqrt{1+{\bf x}_0^T({\bf X}^T{\bf X})^{-1}{\bf x}_0}], \end{equation*}\]

\[\begin{align*} P(Y = k \mid \mathbf{X} = \mathbf{x}) &= \frac{P(Y = k \cap X = \mathbf{x})}{f(\mathbf{x})} \\ &= \frac{\pi_k f_k(\mathbf{x})}{ \sum \limits_{l = 1}^{K} \pi_l} \end{align*}\]

Where we have denoted:

• The prior class membership probability for class \(k\) as \(\pi_k := P(Y = k)\)
• The prior predictor pdf of \(\mathbf{x}\) as \(f(\mathbf{x})\).

The Bayes classifier minimizes the expected 0/1-loss.

The decision boundary is located at \(x = 25\).

• The diagnostic paradigm is concerned with estimating the posterior class membership probability \(P(Y = j \mid X = x)\).
• The sampling paradigm is concerned with estimating the class conditional probabilities \(\pi_k\) and pdf of the predictor variables for each class, \(f_k(\mathbf{x})\).

Linear discriminant analysis is part of the sampling paradigm, while KNN-classification is part of the diagnostic paradigm.

• We assume that the class conditional probabilities \(f_k(\mathbf{x})\) are normally distributed: \(\mathbf{x}~|~( Y = k) \sim \mathcal{N}(\mu_k, \sigma_k^2)\).
• Additionally, we assume that all the class conditional probabilities have equal variance: \(\sigma_k = \sigma\).

The linear discriminant score, \(\delta_k(x)\), can be maximized instead of using the complicated posterior probability derived with Bayes formula. They admit the same maxima.

Univariate case:

\[\begin{equation*} \delta_k(x) = x \frac{\mu_k}{\sigma_k^2} - \frac{\mu_k^2}{2\sigma^2} + \log{(\pi_k)} \end{equation*}\]

Multivariate case:

\[\begin{equation*} \delta_k(\mathbf{x}) = \mathbf{x}^T \Sigma^{-1} \mu_k - \frac{1}{2} \mu_k^T \Sigma^{-1} \mu_k + \log{(\pi_k)} \end{equation*}\]

\[\begin{equation*} \hat{\pi}_k := \frac{n_k}{n} \end{equation*}\]

\[\begin{equation*} \hat{\mu}_k := \frac{1}{n_k} \sum \limits_{i: y_i = k} x_i \end{equation*}\]

Univariate case:

\[\begin{equation*} \hat{\sigma}_k^2 := \frac{1}{n_k - 1} \sum \limits_{i: y_i = k} (x_i - \hat{\mu}_k)^2 \end{equation*}\]

Multivariate case:

\[\begin{equation*} \hat{\Sigma}_k := \frac{1}{n_k - 1} \sum \limits_{i: y_i = k} (\mathbf{x}_i - \hat{\mathbf{\mu}}_k)(\mathbf{x}_i - \hat{\mathbf{\mu}}_k)^T \end{equation*}\]

Univariate version:

\[\begin{align*} \hat{\sigma}^2 &:= \frac{1}{n - K} \sum \limits_{k = 1}^{K} \sum \limits_{i: y_i = k} (x_i - \hat{\mu}_k)^2 \\ &= \sum \limits_{k = 1}^{K} \frac{n_k - 1}{n - K} \hat{\sigma}_k^2 \end{align*}\]

Multivariate version:

\[\begin{equation*} \hat{\sigma}^2 := \sum \limits_{k = 1}^{K} \frac{n_k - 1}{n - K} \hat{\Sigma}_k \end{equation*}\]

When we want to evaluate a classification model we are interested in both decreasing false positives and false negatives. A confusion matrix is really useful for this purpose. Write the predicted classes on the top, the actual classes on the left, and finally fill inn the matrix accordingly. The grater the values across the diagonal, the better (for two-class problems).

See this image:

\[\begin{equation*} \hat{P}(Y = k \mid X = x) = \frac{e^{\hat{\delta}_k(x)}}{ \sum \limits_{l = 1}^{K} e^{\hat{\delta}_l(x)}} \end{equation*}\]

Quadratic discriminant analysis does not assume the variances to be equal for all the class conditional distributions, and therefore estimates separate values for \(|\Sigma_k|\).

The decision boundaries are quadratic functions of \(x\).

LDA estimates \(p(p+1)/2\) parameters, while QDA estimates \(Kp(p+1)/2\).

Since quadratic discriminant analysis estimates \(K\) times more parameters than LDA, it is much more flexible. With that comes the cost of additional variance in the model. We should therefore prefer LDA when we have few observations compared to the number of covariates, as the estimates for the covariances becomes worse. With lots of data, i.e. \(n >> p\), we might prefer QDA over LDA.

\[\begin{equation*} \delta_k(\mathbf{x}) := - \frac{1}{2} (\mathbf{x} - \mu_k)^T \Sigma_k^{-1} (\mathbf{x} - \mu_k) - \frac{1}{2} \log{|\Sigma_k|} + \log{(\pi_k)} \end{equation*}\]

Define \(\mathcal{N}_x\) to be the set of the \(K\) closest points to the point \(x = (x_1, x_2)\). The KNN istimator \(\hat{y}(x) \in \{0, 1\}\) is determined by taking a “majority vote” of \(x\)’s \(N\) closest neighbors. The KNN estimate of the posterior class probability becomes

\[\begin{gather*} \hat{P}(Y = y | X = x) = \frac{1}{K} \sum_{i \in \mathcal{N}_x} I(y_i = y) = \begin{cases} \frac{1}{K} \sum_{i \in \mathcal{N}_x} y_i,~~~~~~~~\text{if } y = 1 \\ 1 - \frac{1}{K} \sum_{i \in \mathcal{N}_x} y_i,~\text{if } y = 0 \end{cases} \end{gather*}\]

And the Bayes classifier can be used to construct the estimator \(\hat{y}(x)\)

\[\begin{gather*} \hat{y}(x) = \begin{cases} 1, ~~~~~\text{if } \hat{P}(Y = 1 | X = x) \geq 0.5 \\ 0, ~~~~~\text{otherwise} \end{cases} \end{gather*}\]

The training error rate is given by

\[\frac{1}{n_{train}}\sum_{i \in J_{train}}I(y_i \neq \hat{y}(x_i)),\] where \(J_{train}\) is the set of \(n_{train}\) indices defining the training set. Similarly the test error rate is given by

\[\frac{1}{n_{\mathrm{test}}}\sum_{i \in J_{\text{test}}}I(y_i \neq \hat{y}(x_i))\]

Instead of choosing the \(K\) that results in the smallest CV error rate, \(\hat{\mathrm{K}} := \mathrm{argmin}~\mathrm{CV}(\mathrm{K})\), we instead choose the greatest \(K\) which is within one standard deviation of \(\hat{\mathrm{K}}\). More exactly,

\[ \mathrm{K} = \underset{\mathrm{K^*}}{\mathrm{max}} ~ \{K^* ~ | ~ \mathrm{CV}(\mathrm{K^*}) \leq \mathrm{CV}(\hat{\mathrm{K}}) + \mathrm{SE}(\hat{\mathrm{K}})\} \]

This way, we are able to simplify our model within acceptable bounds from the minimal error.

\[\begin{gather*} \text{sensitivity} := \frac{\text{Number of true positives}}{\text{Number of positives}} = \frac{\mathrm{TP}}{\mathrm{P}} \\ \text{specificity} := \frac{\text{Number of true negatives}}{\text{Number of negatives}} = \frac{\mathrm{TN}}{\mathrm{N}} \end{gather*}\]

Thus, these extreme thresholds (probability cut-off) must therefore imply the following

\[\begin{gather*} \text{threshold} \gets 0 \implies \begin{cases} \text{sensitivity} = 1 \\ \text{specificity} = 0 \end{cases} \\ \text{threshold} \gets 1 \implies \begin{cases} \text{sensitivity} = 0 \\ \text{specificity} = 1 \end{cases} \end{gather*}\]

The \(\mathrm{AUC} \in [0, 1]\) is the area under the \(\mathrm{ROC}\) curve.

The best case scenario is when the \(\mathrm{ROC}\) curve “hugs” the upper left corner, resulting in \(\mathrm{AUC} = 1\).

Random guessing can be thought of as drawing samples \(u \sim \mathcal{U}(0, 1)\) and comparing it to the threshold probability in order to make a prediction. Since \(P(u < p) = p\), the resulting specificity-sensitivity graph becomes a straight line (on average), and the \(\mathrm{AUC}\) becomes \(0.5\).

A random guess classifier corresponds to a posterior distribution \(\mathrm{P}(Y = 1 | X = x)\sim\mathcal{U}(0,1)\). This in turn means that \(\mathrm{TP}\sim \mathcal{Bin}(P,1-t)\), \(t\) being the threshold, and \(\mathrm{TN}\sim \mathcal{Bin}(N,t)\). Thus

\[ \text{E}[\text{sensitivity}] = \text{E}\left[\frac{TP}{P}\right] = 1-t = 1-\text{E}\left[\frac{TN}{N}\right] = 1 - \text{E}[\text{specificity}] \]

Which means that the ROC curve will be expected to lie on diagonal line in a ROC plot, which will result in \(\text{E}[\text{AUC}] = 0.5\)

• Subset selection: We identify and select a suitable subset of the \(p\) available predictors.
• Dimension reduction: We project the \(p\) predictors into an \(M\)-dimensional subspace where \(M < p\).

The difference is therefore if the projection is orthogonal to the axes or not.

No, pay attention to the sum indices in the following ridge expression:

\[\begin{equation*} \mathrm{RSS} + \lambda \sum \limits_{i = 1}^{p} \beta_j^2 \end{equation*}\]

We don’t want to penalize the average response in the data set.

B: Only lasso regression shrinks \(\beta_j\) to \(0\) for finite penalties.

The ridge regression estimator is given by \[ \widetilde{\boldsymbol \beta}=({\bf X}^T{\bf X}+\lambda {\bf I})^{-1}{\bf X}^T{\bf Y} = \textbf{M}\textbf{Y} \] Where \(\lambda\) is called the regularization parameter and \(\textbf{M} = ({\bf X}^T{\bf X}+\lambda {\bf I})^{-1}{\bf X}^T\) is a \((p+1)\times n\) matrix.

The Akaike Information Criterion (AIC) is defined as

\[ \text{AIC} := -2 \cdot \ell(\hat{\beta}_M, \tilde{\sigma}^2) +2(\lvert M\rvert +1), \]

where \(\hat{\beta}_M\) is the estimated coefficient vector for a model with \(M\) regression parameters, and \(\ell(\cdot)\) is the logarithm of the maximum likelihood for that model. How you calculate the maximum likelihood depends on what type of regression you perform. We use \(|M| + 1\), because we have to add the estimated variance of the error terms \(\epsilon_i\). For linear regression, \(AIC\) becomes:

\[ \text{AIC} = \frac{1}{n\hat{\sigma}^2} (\text{RSS} + 2d\hat{\sigma}^2) \]

When we do model selection, i.e. picking between different candidate models, we can pick the model with the lowest value for AIC.

Both methods are intended to evaluate the predictive power of a model while still accounting for the fact that the model might be overfitted (and punishing accordingly).

AIC achieves this by using RSS as the measure of fit while adding a complexity term in order to adjust for overfitting. The entirety of the data is used for the measure.

Cross-validation with mean squared test error MSE does not require such an additional complexity term. It rather splits the data into training and validation sets, trains the model on the training set, and calculates the MSE with the validation test. That way it implicitly guards against overfitting since the validation set is not part of the model training procedure.

Here are some advantages/disadvantages:

• CV requires you to split your data set into training and testing subsets. This results in smaller sample sizes for fitting the models, and can result in an overestimation of errors.
• CV test MSE can have high variability for smaller sample sizes, since it is dependent on which observations are included/excluded.

A greedy approach means that the apparently best choice is taken at any one time, not considering the possibility of a suboptimal choice may be taken in order to achieve an overall better result further down the road. Once a decision has been made, no backtracking is performed, and the choice is “locked in”.