Machine Learning

For single dimension data, linear regression is defined as \(w = A^{-1}b\), where A is \(xx^\mathsf{T}\), and b is \(yx\). Since the matrix may not be invertible, we take the pseudo inverse. Since I added a column of ones to the A matrix, \(w_0\) holds the y intercept, and since it is only a single dimension, \(w_1\) holds the slope of the line.

The linear regression code may be run on the iris dataset with the following:

from portfolio.linear_regression import LinearRegression

x_sepal_width = x[:, 1]

regression_plot = Plot(x_sepal_width, y)
regression_plot.add_view(LinearRegression)
regression_plot.embed()

Decision stumps are a type of weak learner. They consist of a decision tree with only a single node. My implementation only accepts a binary split of classes labeled -1 and 1, although in theory a decision stump can split based on threshold into any number of classes.

I couldn't figure out how to translate the pseudocode for the efficient \(O(dm)\) implementation, so this is the naïve \(O(dm^2)\) version. I also have a somewhat fuzzy understanding of the book's explanation, but this is equivalent as far as I can tell.

Since the decision stump needs a threshold \(\theta\) in order to classify everything on each side as a different class, we need to define a set of steps. Given that two points may be arbitrarily close, I used the \(x\) values from the data to ensure that if there was a single optimal \(\theta\) it would be selected.

The other part of the decision stump is the dimension of the data against which it will classify. This implementation considers all given dimensions and selects the most suitable one.

The last part is the error function. The error function I used is a count of the differences between the predicted class and the expected class.

For each of the combinations of dimensions and possible thresholds, the decision stump checks the error function. The one with the lowest error is selected as the dimension to classify against and the \(\theta\) to use as a threshold.

Shown here is the decision boundary, along with the values plotted for the chosen dimension, with the label on the y axis. As you can see, these are not seperable, but the optimal boundary was still found.

from portfolio.decision_stumps import DecisionStumps

x_sepal_width = x[:, 1]

regression_plot = Plot(x, y)
regression_plot.add_view(DecisionStumps)
regression_plot.embed()

K nearest neighbors classifier simply classifies a sample based upon the closest training data.

At least in this implementation, although also in general to a lesser extent, KNN tends to be very computationally intensive for large datasets, since it must compute the distance to all points in order to find the best.

from portfolio.knn import KNN

y = df.iloc[50:, 4].values
y = np.where(y == 'Iris-versicolor', -1, 1)
X = df.iloc[50:, [0, 3]].values
np.c_[y, X]

knn = KNN(X, y)

knn.predict([[0, 0]]).item()

knn.predict([[3, 8]]).item()

Given the same data as the SVM example, we can see that KNN is able to classify this data entirely as opposed to the lossy linear classifier. This isn't necessarily an argument, since it doesn't take into account the possibility of overfitting, which KNN is always affected by.

Given certain datasets, including this one, the performance is very good.

fig = plt.figure()
visualize.plot_decision_regions(fig.add_subplot(), X, y, knn)
Plot.embed(fig)