Student research opportunities

Comparison of Classification-Based Loss Functions

Project Code: CECS_873

This project is available at the following levels:
CS single semester, Honours, Summer Scholar
Please note that this project is only for undergraduate students.

Keywords:

Classification, Optimization, Machine Learning

Supervisor:

Dr Justin Domke

Outline:

Perhaps the most basic problem in machine learning is binary classification: given a set of "features" of an entity, predict the class of that entity. For example, given someone's height, age, and weight, predict if they are male.

Classifiers are trained by taking a dataset of features and output classes, and fitting a classifier function that would do a good job on that dataset. (For example, an algorithm might discover that predicting male if .72*height + .24*weight - .05*age > .2 works well.) However, there isn't general agreement of how exactly we should measure what a "good job" is, since our goal isn't to do a good job on the training data, but on future "test" data. The obvious technique of simply measuring how many of the data would be correctly classified turns out to be very computationally challenging, and sometimes generalizes poorly to future data.

Different classifiers often use different loss functions. For example, support vector machines use the "hinge" loss, logistic regression uses the "logistic" loss, and boosting uses the exponential loss. It isn't obvious on theoretical grounds which of these will be best, since it fundamentally depends on properties of the datasets, which are only knowable experimentally.

Goals of this project

In this project, you will implement standard machine learning methods, and compare the results of using different loss functions on a variety of datasets. Your goal will be to understand what properties of different loss functions lead to better performance in different situations.

Requirements/Prerequisites

A successful student will have most of the following: working knowledge of basic linear algebra, optimization and statistics, good programming skills in some (any!) language, and some experience with numerical computing in a language such as Matlab, Octave, Mathematica, R, Python/NumPy, IDL, Fortran, or C++/Eigen.