Multi-modal Human-computer Interaction Attila Fazekas Attila.Fazekas@inf.unideb.hu SSIP 2008, July 2008 Hungary and Debrecen July,2008 SSIP’09 Multi-modal Human-computer Interaction - Debrecen – Big Church July,2008 SSIP’09 Multi-modal Human-computer Interaction - University of Debrecen July,2008 SSIP’09 Multi-modal Human-computer Interaction - Coming Soon July,2008 SSIP’09 ¬ Summer School on Image Processing 2009 Coming Soon July,2008 SSIP’09 ¬ Summer School on Image Processing 2009 ¬ Coming Soon July,2008 SSIP’09 ¬ Summer School on Image Processing 2009 ¬ ¬ http:\\www.inf.unideb.hu\˜ssip Multi-modal Human-computer Interaction - Road Map July,2008 SSIP’09 ¬ Multi-modal interactions and systems (main categories, examples, benefits) ¬ Turk-2 – Multi-modal chess player ¬ Face detection, facial gestures recognition ¬ Experimental results ¬ Examples Multi-modal Human-computer Interaction - Defining Multi-Modal Interaction July,2008 SSIP’09 ¬ There are two views on multi-modal interaction: MMHCI Turk Face detection Face gesture Examples SVM Defining Multi-Modal Interaction July,2008 SSIP’09 ¬ There are two views on multi-modal interaction: « The first focuses on the human side: perception and control There the word modality refers to human input and output channels MMHCI Turk Face detection Face gesture Examples SVM Examples July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 42 Examples July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 43 Examples July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 44 Examples July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 45 Support Vector Machine July,2008 SSIP’09 ¬ Statistical learning from examples aims at selecting from a given set of functions {fα(x) | α ∈ Λ}, the one which predicts best the correct response MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 46 Support Vector Machine July,2008 SSIP’09 ¬ This selection is based on the observation of l pairs that build the training set: (x1, y1), , (xl , yl ), xi ∈ Rm, yi ∈ {+1, −1} which contains input vectors xi and the associated ground ”truth” given by an external supervisor MMHCI Turk Face detection Face gesture Examples SVM Support Vector Machine July,2008 SSIP’09 ¬ This selection is based on the observation of l pairs that build the training set: (x1, y1), , (xl , yl ), xi ∈ Rm, yi ∈ {+1, −1} which contains input vectors xi and the associated ground ”truth” given by an external supervisor MMHCI Turk Face detection ¬ Let the response of the learning machine fα(x) belongs to a set of indicator functions Face gesture Examples SVM Multi-modal Human-computer Interaction - 47 Support Vector Machine July,2008 SSIP’09 ¬ If we define the loss-function: L(y, fα(x)) = 0, if y = fα(x), 1, if y = fα(x) The expected value of the loss is given by: R(α) = MMHCI Turk L(y, fα(x))p(x, y)dxdy, where p(x, y) is the joint probability density function of random variables x and y Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 48 Support Vector Machine July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM ¬ We would like to find the function fα0 (x) which minimizes the risk function R(α) Support Vector Machine July,2008 SSIP’09 ¬ We would like to find the function fα0 (x) which minimizes the risk function R(α) ¬ The basic idea of SVM to construct the optimal separating hyperplane MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 49 Support Vector Machine July,2008 SSIP’09 ¬ Suppose that the training data can be separated by a hyperplane, fα(x) = αT x + b = 0, such that: yi(αT xi + b) ≥ 1, i = 1, 2, , l where α is the normal to the hyperplane MMHCI Turk Face detection Face gesture Examples SVM Support Vector Machine July,2008 SSIP’09 ¬ Suppose that the training data can be separated by a hyperplane, fα(x) = αT x + b = 0, such that: yi(αT xi + b) ≥ 1, i = 1, 2, , l where α is the normal to the hyperplane MMHCI Turk ¬ For the linearly separable case, SVM simply seeks for the separating hyperplane with the largest margin Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 50 Support Vector Machine July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM ¬ For linearly nonseparable data, by mapping the input vectors, which are the elements of the training set, into a high-dimensional feature space through so-called kernel function Support Vector Machine July,2008 SSIP’09 ¬ For linearly nonseparable data, by mapping the input vectors, which are the elements of the training set, into a high-dimensional feature space through so-called kernel function ¬ We construct the optimal separating hyperplane in the feature space to get a binary decision MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 51 July,2008 SSIP’09 Thank you for your attention! Multi-modal Human-computer Interaction - 52 ... Multi-modal Human-computer Interaction - Debrecen – Big Church July,2008 SSIP’09 Multi-modal Human-computer Interaction - University of Debrecen July,2008 SSIP’09 Multi-modal Human-computer Interaction. .. Experimental results ¬ Examples Multi-modal Human-computer Interaction - Defining Multi-Modal Interaction July,2008 SSIP’09 ¬ There are two views on multi-modal interaction: MMHCI Turk Face detection... gesture Examples SVM Multi-modal Human-computer Interaction - 17 Turk-2 July,2008 SSIP’09 MMHCI Turk Face detection Face gesture Examples SVM Multi-modal Human-computer Interaction - 18 System Components