Preliminaries

import pandas as pd
from sklearn import neighbors
import numpy as np
%matplotlib inline  
import seaborn

Create Dataset

Here we create three variables, test_1 and test_2 are our independent variables, ‘outcome’ is our dependent variable. We will use this data to train our learner.

training_data = pd.DataFrame()

training_data['test_1'] = [0.3051,0.4949,0.6974,0.3769,0.2231,0.341,0.4436,0.5897,0.6308,0.5]
training_data['test_2'] = [0.5846,0.2654,0.2615,0.4538,0.4615,0.8308,0.4962,0.3269,0.5346,0.6731]
training_data['outcome'] = ['win','win','win','win','win','loss','loss','loss','loss','loss']

training_data.head()

Plot the data

This is not necessary, but because we only have three variables, we can plot the training dataset. The X and Y axes are the independent variables, while the colors of the points are their classes.

seaborn.lmplot('test_1', 'test_2', data=training_data, fit_reg=False,hue="outcome", scatter_kws={"marker": "D","s": 100})

<seaborn.axisgrid.FacetGrid at 0x11008aeb8>

Convert Data Into np.arrays

The scikit-learn library requires the data be formatted as a numpy array. Here are doing that reformatting.

X = training_data.as_matrix(columns=['test_1', 'test_2'])
y = np.array(training_data['outcome'])

Train The Learner

This is our big moment. We train a KNN learner using the parameters that an observation’s neighborhood is its three closest neighors. weights = 'uniform' can be thought of as the voting system used. For example, uniform means that all neighbors get an equally weighted “vote” about an observation’s class while weights = 'distance' would tell the learner to weigh each observation’s “vote” by its distance from the observation we are classifying.

clf = neighbors.KNeighborsClassifier(3, weights = 'uniform')
trained_model = clf.fit(X, y)

View The Model’s Score

How good is our trained model compared to our training data?

trained_model.score(X, y)

0.80000000000000004

Our model is 80% accurate!

Note: that in any real world example we’d want to compare the trained model to some holdout test data. But since this is a toy example I used the training data.

Apply The Learner To A New Data Point

Now that we have trained our model, we can predict the class any new observation, $y_{test}$. Let us do that now!

/* Create a new observation with the value of the first independent variable, 'test_1', as .4 
   and the second independent variable, test_1', as .6 */
x_test = np.array([[.4,.6]])

/* Apply the learner to the new, unclassified observation. */
trained_model.predict(x_test)

array(['loss'], dtype=object)

Huzzah! We can see that the learner has predicted that the new observation’s class is loss.

We can even look at the probabilities the learner assigned to each class:

trained_model.predict_proba(x_test)

array([[ 0.66666667,  0.33333333]])

According to this result, the model predicted that the observation was loss with a ~67% probability and win with a ~33% probability. Because the observation had a greater probability of being loss, it predicted that class for the observation.

Notes

• The choice of K has major affects on the classifer created.
• The greater the K, more linear (high bias and low variance) the decision boundary.
• There are a variety of ways to measure distance, two popular being simple euclidean distance and cosine similarity.