Facial Expression Recognition - PCA + KNN

Kaggle - Challenges in Representation Learning: Facial Expression Recognition Challenge

https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data

The data consists of 48x48 pixel grayscale images of faces. The faces have been automatically registered so that the face is more or less centered and occupies about the same amount of space in each image. The task is to categorize each face based on the emotion shown in the facial expression in to one of seven categories (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral).

train.csv contains two columns, "emotion" and "pixels". The "emotion" column contains a numeric code ranging from 0 to 6, inclusive, for the emotion that is present in the image. The "pixels" column contains a string surrounded in quotes for each image. The contents of this string a space-separated pixel values in row major order. test.csv contains only the "pixels" column and your task is to predict the emotion column.

The training set consists of 28,709 examples. The public test set used for the leaderboard consists of 3,589 examples. The final test set, which was used to determine the winner of the competition, consists of another 3,589 examples.

This dataset was prepared by Pierre-Luc Carrier and Aaron Courville, as part of an ongoing research project. They have graciously provided the workshop organizers with a preliminary version of their dataset to use for this contest.

In [2]:
%matplotlib inline
In [3]:
import graphlab as gl
import numpy as np
import matplotlib.pyplot as plt
from timeit import default_timer as timer
from scipy.stats import mode
In [4]:
from sklearn.decomposition import PCA
This non-commercial license of GraphLab Create for academic use is assigned to rsridha2@uci.edu and will expire on December 03, 2017.
[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: /tmp/graphlab_server_1509334214.log

Get data

In [40]:
trainfile = 'train.txt'
testfile = 'test.txt'
pri_testfile = 'pri_test.txt'
In [41]:
# Read training data and labels into lists
trainlabels = []
traindata = []
with open(trainfile, 'r') as f:
    for line in f:
        currfile, currlabel = line.split()
        trainlabels.append(currlabel)
        currfile = plt.imread(currfile)
        currdata = np.reshape(currfile, (48*48))
        traindata.append(currdata)
In [42]:
# Read public test data and labels into lists
testlabels = []
testdata = []
with open(testfile, 'r') as f:
    for line in f:
        currfile, currlabel = line.split()
        testlabels.append(currlabel)
        currfile = plt.imread(currfile)
        currdata = np.reshape(currfile, (48*48))
        testdata.append(currdata)
In [43]:
# Read private test data and labels into lists
pri_testlabels = []
pri_testdata = []
with open(pri_testfile, 'r') as f:
    for line in f:
        currfile, currlabel = line.split()
        pri_testlabels.append(currlabel)
        currfile = plt.imread(currfile)
        currdata = np.reshape(currfile, (48*48))
        pri_testdata.append(currdata)
In [8]:
# Create training dictionary containing data row wise
finaltrain = {}
for i in range(48*48):
    varname = "X"+str(i)
    column = []
    for j in range(len(traindata)):        
        column.append(traindata[j][i])
    finaltrain[varname] = column
# finaltrain['label'] = trainlabels
In [9]:
# Create public test dictionary containing data row wise
finaltest = {}
for i in range(48*48):
    varname = "X"+str(i)
    column = []
    for j in range(len(testdata)):        
        column.append(testdata[j][i])
    finaltest[varname] = column
# finaltest['label'] = testlabels
In [10]:
# Create private test dictionary containing data row wise
final_pritest = {}
for i in range(48*48):
    varname = "X"+str(i)
    column = []
    for j in range(len(pri_testdata)):        
        column.append(pri_testdata[j][i])
    final_pritest[varname] = column
# finaltest['label'] = testlabels
In [3]:
# Create SFrames (GraphLab's efficient data structure) for each dataset
sf_train = gl.SFrame(finaltrain) 
sf_test = gl.SFrame(finaltest)   
sf_pri_test = gl.SFrame(final_pritest)   
In [2]:
# Save the SFrames
sf_train.save('SFrame/sf_train')
sf_test.save('SFrame/sf_test')
sf_pri_test.save('SFrame/sf_pri_test')
In [10]:
# Load to verify
sf_train = gl.load_sframe('SFrame/sf_train')
sf_test = gl.load_sframe('SFrame/sf_test')
sf_pri_test = gl.load_sframe('SFrame/sf_pri_test')
In [ ]:
# Delete objects that are not required
# del traindata, testdata, currfile, currlabel

Standardize data

In [11]:
import os
start = timer()

# Standardize training and test data
# Use training mean and std dev for test data - normalize test data first 
# (or else train will get changed)
for c in sf_train.column_names():
    sf_test[c] = (sf_test[c] - sf_train[c].mean()) / sf_train[c].std()
    sf_pri_test[c] = (sf_pri_test[c] - sf_train[c].mean()) / sf_train[c].std()
    sf_train[c] = (sf_train[c] - sf_train[c].mean()) / sf_train[c].std()

end = timer()
print "Time elapsed = ", (end - start), "seconds"
os.system("say 'Your program has finished'")
Time elapsed =  1148.68046403 seconds
Out[11]:
0
In [13]:
# Verify means and std devs
print sf_pri_test['X1'].mean()
print sf_test['X1'].mean()
print sf_train['X1'].mean()
print sf_test['X1'].std()
print sf_train['X1'].std()
print sf_pri_test['X1'].std()
-0.020830041082
-0.0113799719475
7.0733349733e-16
0.992623105366
1.0
1.0030817717
In [6]:
sf_train.save('SFrame/sf_train_standardized')
sf_test.save('SFrame/sf_test_standardized')
sf_pri_test.save('SFrame/sf_pri_test_standardized')
In [15]:
print sf_train.shape
print sf_test.shape
print sf_pri_test.shape
(28707, 2304)
(3589, 2304)
(3589, 2304)

Convert SFrames to numpy

In [16]:
start = timer()
sf_train_numpy = sf_train.to_numpy()
sf_test_numpy = sf_test.to_numpy()
sf_pri_test_numpy = sf_pri_test.to_numpy()
print sf_train_numpy.shape    # Verify shape
print sf_test_numpy.shape     # Verify shape
print sf_pri_test_numpy.shape # Verify shape
end = timer()
print "Time elapsed = ", (end - start), "seconds"
os.system("say 'Your program has finished'")
(3589, 2304)
Time elapsed =  19.0210530758 seconds
Out[16]:
0
In [17]:
# Save the numpy versions of the objects
np.save('SFrame/nptrain', sf_train_numpy)
np.save('SFrame/nptest', sf_test_numpy)
np.save('SFrame/np_pritest', sf_pri_test_numpy)
In [7]:
sf_train_numpy = np.load('SFrame/nptrain.npy')
sf_test_numpy = np.load('SFrame/nptest.npy')
sf_pri_test_numpy = np.load('SFrame/np_pritest.npy')
In [ ]:
 

Perform Principal Component Analysis (PCA) to reduce data dimensionality

In [8]:
num_comp = 120 # How many components
pca = PCA(n_components = num_comp, svd_solver='full')
pca.fit(sf_train_numpy)
Out[8]:
PCA(copy=True, iterated_power='auto', n_components=120, random_state=None,
  svd_solver='full', tol=0.0, whiten=False)
In [9]:
# Percentage of variance in the data explained by the components
print sum(pca.explained_variance_ratio_)
0.90396197548

50 components ==> 83.4%
100 components ==> 89%
120 components ==> 90.4%
160 components ==> 92.2%
500 components ==> 97.4%

In [10]:
# Reduce dimensionality of data using the principal components
dimred_train = pca.transform(sf_train_numpy)
dimred_test = pca.transform(sf_test_numpy)
dimred_pri_test = pca.transform(sf_pri_test_numpy)
print dimred_train.shape
print dimred_test.shape
print dimred_pri_test.shape
(28707, 120)
(3589, 120)
(3589, 120)

t-SNE

In [12]:
from sklearn.manifold import TSNE
test = TSNE(2)
test.fit(dimred_test)
Out[12]:
TSNE(angle=0.5, early_exaggeration=12.0, init='random', learning_rate=200.0,
   method='barnes_hut', metric='euclidean', min_grad_norm=1e-07,
   n_components=2, n_iter=1000, n_iter_without_progress=300,
   perplexity=30.0, random_state=None, verbose=0)
In [44]:
plt.scatter(test.embedding_[:, 0], test.embedding_[:, 1], c=testlabels)
Out[44]:
<matplotlib.collections.PathCollection at 0x152d4fb50>

K-NN (K-nearest neighbor)

In [35]:
# Convert PCA transformed data into SFrames 
dimred_sf_test = gl.SFrame(map(gl.SArray, dimred_test.T))
dimred_sf_train = gl.SFrame(map(gl.SArray, dimred_train.T))
dimred_sf_pri_test = gl.SFrame(map(gl.SArray, dimred_pri_test.T))
In [39]:
# Verify shape
print dimred_sf_test.shape, dimred_sf_train.shape, dimred_sf_pri_test.shape
(3589, 120) (28707, 120) (3589, 120)
In [40]:
# Create knn model
# For numeric data, the 'distance' function options are euclidean, manhattan, cosine, 
# and transformed_dot_product
model = gl.nearest_neighbors.create(dimred_sf_train, distance='cosine')
Starting brute force nearest neighbors model training.
In [41]:
model.summary()
Class                          : NearestNeighborsModel

Attributes
----------
Method                         : brute_force
Number of distance components  : 1
Number of examples             : 28707
Number of feature columns      : 120
Number of unpacked features    : 120
Total training time (seconds)  : 0.502

In [42]:
# Test knn model on a few training data points
knn = model.query(dimred_sf_train[:5], k=5)
knn.head()
Starting pairwise querying.
+--------------+---------+-------------+--------------+
| Query points | # Pairs | % Complete. | Elapsed Time |
+--------------+---------+-------------+--------------+
| 0            | 10      | 0.00696694  | 60.337ms     |
| Done         |         | 100         | 177.45ms     |
+--------------+---------+-------------+--------------+
Out[42]:
query_label reference_label distance rank
0 0 -2.22044604925e-16 1
0 3438 0.0585730370704 2
0 2485 0.0589124940746 3
0 356 0.100776882576 4
0 8126 0.134507986458 5
1 1 0.0 1
1 25204 0.0337793624353 2
1 2482 0.0346130123721 3
1 573 0.0355195292887 4
1 1690 0.0355195292887 5
[10 rows x 4 columns]
In [384]:
# Test knn model on a few test data points
knn = model.query(dimred_sf_test[10:11], k=15, verbose=False)
knn.print_rows(15)
+-------------+-----------------+----------------+------+
| query_label | reference_label |    distance    | rank |
+-------------+-----------------+----------------+------+
|      0      |      26921      | 0.44166597085  |  1   |
|      0      |      11805      | 0.493119586584 |  2   |
|      0      |      16385      | 0.496122870611 |  3   |
|      0      |      10459      | 0.496363117623 |  4   |
|      0      |      12159      | 0.503393274362 |  5   |
|      0      |       8298      | 0.510277341191 |  6   |
|      0      |       6015      | 0.512401771528 |  7   |
|      0      |       5612      | 0.514036655732 |  8   |
|      0      |       6281      | 0.516311980706 |  9   |
|      0      |       6784      | 0.52304626813  |  10  |
|      0      |      18619      | 0.523253347917 |  11  |
|      0      |      27849      | 0.525117730473 |  12  |
|      0      |      23578      | 0.527634004893 |  13  |
|      0      |      21019      | 0.528061781413 |  14  |
|      0      |      10668      | 0.528152437309 |  15  |
+-------------+-----------------+----------------+------+
[15 rows x 4 columns]

In [ ]:
# Predict on entire public and private test data

start = timer()
testpred = []
pri_testpred = []
num_nbr = 11

for i in range(len(dimred_sf_test)):
    print i,
    # Public test
    curr_preds = []
    knn = model.query(dimred_sf_test[i:(i+1)], k=num_nbr, verbose=False)
    curr_preds = np.array(knn['reference_label'])    
    mode_preds = mode((np.take(trainlabels, curr_preds)))[0][0]
    testpred.append(mode_preds)
    
    # Private test
    curr_preds = []
    knn = model.query(dimred_sf_pri_test[i:(i+1)], k=num_nbr, verbose=False)
    curr_preds = np.array(knn['reference_label'])    
    mode_preds = mode((np.take(trainlabels, curr_preds)))[0][0]
    pri_testpred.append(mode_preds)
    
end = timer()

print "Time elapsed = ", (end - start), "seconds"
os.system("say 'Your program has finished'")    
In [52]:
# Verify shapes
print len(dimred_sf_test), len(testpred), len(testlabels), len(dimred_sf_pri_test), len(pri_testlabels), \
len(pri_testpred)
3589 3589 3589 3589 3589 3589
In [54]:
# Accuracy of predictions
print np.sum(np.array(testpred) == np.array(testlabels))/ float(len(testlabels))
print np.sum(np.array(pri_testpred) == np.array(pri_testlabels))/ float(len(pri_testlabels))
0.371412649763
0.367511841739

Results
PCA with 100 components; knn with cosine similarity; 5 neighbors; Test accuracy = 36.8626%
PCA with 100 components; knn with cosine similarity; 10 neighbors; Test accuracy = 36.5%
PCA with 500 components; knn with cosine similarity; 5 neighbors; Test accuracy = 36.08%
PCA with 50 components; knn with cosine similarity; 11 neighbors; Test accuracy = 36.47%
PCA with 50 components; knn with euclidean; 7 neighbors; Test accuracy = 34.44%
PCA with 50 components; knn with euclidean; 11 neighbors; Test accuracy = 34.74%
PCA with 50 components; knn with manhattan; 15 neighbors; Test accuracy = 36.388%
PCA with 120 components; knn with cosine similarity; 11 neighbors; Public Test accuracy = 37.14%; Private Test accuracy = 36.75%
PCA with 160 components; knn with cosine similarity; 11 neighbors; Test accuracy = 35.63%
PCA with 160 components; knn with cosine similarity; 5 neighbors; Test accuracy = 36.42%

In [55]:
# Save the predictions
test_37percent_120cos11 = testpred
pritest_37percent_120cos11 = pri_testpred
np.save('SFrame/test_37percent_120cos11', test_37percent_120cos11)
np.save('SFrame/pritest_37percent_120cos11', pritest_37percent_120cos11)
In [ ]: