On Combining Metric and Kernel Learning
National Framework
Funding by the Swiss National Science Foundation.
Description
Kernel and metric learning have become very active research fields in machine learning over the last
years. Although they have developed in a distinct manner they share common elements. One of
the most popular approaches to kernel learning is learning a linear combination of a set of kernels,
usually identified as Multiple Kernel Learning (MKL). On the metric learning side, the most prominent
approach is learning a Mahalanobis distance in some feature space. Both methods in fact learn linear
feature transformations; in the case of MKL the learned transformation has a specific structure.
Many of the metric learning methods are kernelized which raises the issue of which kernel should one use
for a given problem, nevertheless there is no work so far that tries to combine metric learning with MKL.
On the other hand since MKL is learning a special form of linear transformation over the concatenation
feature space one could use metric learning techniques in order to learn such linear transformations
or more general forms of them. On the same time, and despite the increasing popularity of metric
learning methods, there exist so far no methods that scale well with increasing problem sizes,
i.e. large feature space dimensionality and large number of instances, and on the same time retain a
good generalization performance. In this project we address the issues briefly described above by
exploring two research directions.
In the first research direction we will link metric and kernel learning methods, by transferring ideas
and tools from one domain to the other and vice versa. Here we will tackle two tasks. In the one task
we will combine metric learning with MKL, i.e. learning metrics over kernels learned by MKL and we will explore
different metric parametrizations which will lead to different learning problems. In the other task
we will work in the opposite direction and exploit metric learning ideas in the context of kernel
learning. More precisely we will learn linear transformations of the feature space induced by some
kernel and we will experiment with different objective functions in order to learn the linear transformations.
In the second research direction we will propose metric learning methods that scale well with large
datasets. At first we will explore the use of stochastic gradient descent methods in order to improve
the scalability of metric learning. Subsequently we will go to the extreme case of metric learning and
we will learn linear transformations of rank one in order to make metric learning possible for very large
datasets.
Relevant Publications
Jun Wang, Huyen Do, Adam Woznica, Alexandros Kalousis: Metric Learning with Multiple Kernels
In Proceedings of the 25th Neural Information Processing Conference, NIPS, 2011. Draft version,
pdf.
Huyen Do, Alexandros Kalousis, Jun Wang, Adam Woznica: A metric learning perspective of SVM: on the relation of LMNN and SVM.
In Proceedings of the 15th AI-STATS conference 2012.
pdf.
Jun Wang, Adam Woznica, Alexandros Kalousis: Learning neighborhoods for metric learning.
In Proceedings of the ECML/PKDD conference 2012.
pdf.
Jun Wang, Adam Woznica, Alexandros Kalousis: Parametric Local Metric Learning for Nearest Neighbor Classification.
Accepted for publication in NIPS-2012.