Expectation of hinge loss under Gaussian distribution

The standard form of SVM is \begin{align*} \min_{\boldsymbol{w}} \ \ \ \frac{\lambda}{2} \|\boldsymbol{w}\|^2 + \frac{1}{m} \sum_{i=1}^M \ Max \{0, 1-y_i (\boldsymbol{w}^\top \boldsymbol{x}_i + b) \} \end{align* where $\boldsymbol{x}_i \in \mathbb{r}^d$, $y _i \in \{1,-1\}$. The first is the regularization term that controls the complexity of the model, and the second is the hinge loss term that measures the error of the model.

　　Now suppose that each sample $ (\boldsymbol{x}_i, y_i) $ is derived from a Gaussian distribution $\mathcal{n} (\boldsymbol{x}_i, \boldsymbol{\sigma}_i) $, where the covariance matrix $\boldsymbol{\sigma}_i \in \mathbb{s}_{++}^{d}$ depicts the uncertainty of the $\boldsymbol{x}_i$ position, then the new problem can be re-formalized as \begin{align} \label{equ : Svm_gaussian_1} \min_{\boldsymbol{w}} \ \ \ \frac{\lambda}{2} \|\boldsymbol{w}\|^2 + \frac{1}{m} \sum_{i=1}^M \int_{\ma Thbb{r}^d} \max \{0, 1-y_i (\boldsymbol{w}^\top \boldsymbol{x} + b) \} p_i (\boldsymbol{x}) \mbox{d} \boldsymbol{x}\end{a Lign} \begin{align*} p_i (\boldsymbol{x}) = \frac{1}{(2 \pi) ^{D/2} |\BOLDSYMBOL{\SIGMA}_I|^{1/2}} \exp \left (-\frac{1 }{2} (\boldsymbol{x}-\boldsymbol{x}_i) ^\top \boldsymbol{\sigma}_i^{-1} (\boldsymbol{x}-\boldsymbol{x}_i) \right) \ end{align*}, the hinge loss of the original single sample, now becomes the expectation under the Gaussian distribution. Note Type (\ref{equ:svm_gaussian_1}) can be overridden for \begin{align*} \min_{\boldsymbol{w}} \ \ \ \frac{\lambda}{2} \|\boldsymbol{w}\|^2 + \ FRAC{1}{M} \sum_{i=1}^m \int_{\omega_i} (1-y_i \boldsymbol{w}^\top \boldsymbol{x}-y_i b) p_i (\boldsymbol{x}) \mbox{d} \boldSYMBOL{X} \end{align*} where $\omega_i = \{\boldsymbol{x} | y_i (\boldsymbol{w}^\top \boldsymbol{x} + b) \leq 1 \}$. So the focus is as follows: \begin{align*} L (\boldsymbol{g}, H, \boldsymbol{\mu}, \boldsymbol{\sigma}) = \int_{\omega} \frac{\ Boldsymbol{g}^\top \boldsymbol{x} + H} {(2 \pi) ^{D/2} |\BOLDSYMBOL{\SIGMA}|^{1/2}} \exp \left (-\frac{1}{2} (\boldsymbol {x}-\boldsymbol{\mu}) ^\top \boldsymbol{\sigma}^{-1} (\boldsymbol{x}-\boldsymbol{\mu}) \right) \mbox{d} \boldsymbol{ X} \end{align*}, where $\omega = \{\boldsymbol{x} | \boldsymbol{g}^\top \boldsymbol{x} + H \geq 0 \}$, note that there is a corresponding $\boldsymbol{g} =-y_i \boldsymbol{w}$, $h = 1-y_i B$,$\boldsymbol{\mu} = \boldsymbol{x}_i$,$\boldsymbol{\sigma} = \boldsymbol{\Sigma}_i $

　　Since $\boldsymbol{\sigma}$ is a positive definite matrix, we can do eigenvalue decomposition $\boldsymbol{\sigma} = \boldsymbol{u} \boldsymbol{d} \boldsymbol{u}^\top$, Each column of the orthogonal matrix $\boldsymbol{u}$ is a $\boldsymbol{\sigma}$ eigenvector, and the $\boldsymbol{d}$ is a diagonal matrix composed of corresponding eigenvalues, so $\boldsymbol{\sigma}^{ -1} = \boldsymbol{u} \boldsymbol{d}^{-1} \boldsymbol{u}^\top$. Kee $\boldsymbol{z} = \boldsymbol{u}^\top \boldsymbol{u}$ and $\boldsymbol{g}_1 = \boldsymbol{u}^\top \boldsymbol{g}$, easy to know there \ begin{align*} \boldsymbol{g}^\top \boldsymbol{u} & = \boldsymbol{g}^\top (\boldsymbol{u} \boldsymbol{U}^\top) \ Boldsymbol{u} = (\boldsymbol{u}^\top \boldsymbol{g}) ^\top \boldsymbol{u}^\top \boldsymbol{u} = \boldsymbol{g}_1^\top \ Boldsymbol{z} \ \ \boldsymbol{u}^\top \boldsymbol{\sigma}^{-1} \boldsymbol{u} & = \boldsymbol{u}^\top \boldsymbol{u } \boldsymbol{d}^{-1} \boldsymbol{u}^\top \boldsymbol{u} = \boldsymbol{z}^\top \boldsymbol{d}^{-1} \boldsymbol{z} \ \ \ Mbox{d} \boldsymbol{u} & = |\boldsymbol{u}| \mbox{d} \boldsymbol{z} = \mbox{d} \boldsymbol{z} \end{align*} so \begin{align*} L (\bOldsymbol{g}, H, \boldsymbol{\mu}, \boldsymbol{\sigma}) = \int_{\omega_2} \frac{\boldsymbol{g}_1^\top \boldsymbol{z} + \boldsymbol{g}^\top \boldsymbol{\mu} + H} {(2 \pi) ^{D/2} |\BOLDSYMBOL{\SIGMA}|^{1/2}} \exp \left (-\frac{1}{2} \ Boldsymbol{z}^\top \boldsymbol{d}^{-1} \boldsymbol{z} \right) \mbox{d} \boldsymbol{z} \end{align*} where $\omega_2 = \{\ Boldsymbol{z} | \boldsymbol{g}_1^\top \boldsymbol{z} + \boldsymbol{g}^\top \boldsymbol{\mu} + H \geq 0 \}$.

kee $\boldsymbol{v} = \boldsymbol{d}^{-1/2} \boldsymbol{z}$ and $\boldsymbol{g}_2 = \BOLDSYMBOL{D}^{1/2} \boldsymbol{g} _1$, easy to know there

　　Note that $\boldsymbol{g}_2$ is a $d$-dimensional vector, the existence of $d-1$ vector and $\boldsymbol{g}_2/\|\boldsymbol{g}_2\|$ together form an orthogonal matrix $\boldsymbol{b}$, it is advisable to set $\boldsymbol{b}$ $j$ is listed as $\boldsymbol{g}_2/\|\boldsymbol{g}_2\|$, so $\boldsymbol{b}^\top \boldsymbol{g}_2 = \|\ boldsymbol{g}_2\| \boldsymbol{e}_j$, where $\boldsymbol{e}_j$ is the unit column vector that $j$ is $0$ for the remainder of the $1$ dimension. Kee $\boldsymbol{m} = \boldsymbol{b}^\top \boldsymbol{v}$, easy to know \begin{align*} \boldsymbol{g}_2^\top \boldsymbol{v} & = (\|\boldsymbol{g}_2\| \boldsymbol{b} \boldsymbol{e}_j) ^\top \boldsymbol{v} = \|\boldsymbol{g}_2\| \boldsymbol{e}_j^\top \boldsymbol{b}^\top \boldsymbol{v} = \|\boldsymbol{g}_2\| \boldsymbol{e}_j^\top \boldsymbol{m} = \|\boldsymbol{g}_2\| M_j \ \boldsymbol{v}^\top \boldsymbol{v} & = \boldsymbol{v}^\top \boldsymbol{b} \boldsymbol{b}^\top \boldsymbol{v} = \boldsymbol{m}^\top \boldsymbol{m} \ \ \mbox{d} \boldsymbol{v} & = |\boldsymbol{b}| \mbox{d} \boldsymbol{m} = \mbox{d} \boldsymbol{m}\end{align*} where $m_j$ is the $\boldsymbol{m}$ dimension, $j$} L (\ Boldsymbol{g}, H, \boldSYMBOL{\MU}, \boldsymbol{\sigma}) = \int_{\omega_4} \frac{\|\boldsymbol{g}_2\| m_j + \boldsymbol{g}^\top \boldsymbol{\ MU} + H} {(2 \pi) ^{D/2}} \exp \left (-\frac{1}{2} \boldsymbol{m}^\top \boldsymbol{m} \right) \mbox{d} \boldsymbol{m} \end{ align*} where $\omega_4 = \{\boldsymbol{m} | \|\boldsymbol{g}_2\| M_j + \boldsymbol{g}^\top \boldsymbol{\mu} + H \geq 0 \}$.

　　The (\ref{equ:svm_gaussian_5}) and type (\ref{equ:svm_gaussian_6}) surrogate (\ref{equ:svm_gaussian_4}) can be \begin{align*} L (\ Boldsymbol{g}, H, \boldsymbol{\mu}, \boldsymbol{\sigma}) = \frac{\|\boldsymbol{g}_2\|} {\sqrt{2 \pi}} \exp \left (-\frac{1}{2} c^2 \right) + \frac{\boldsymbol{g}^\top \boldsymbol{\mu} + h}{2} \left (1-\mbox{e RF} \left (\frac{c}{\sqrt{2}} \right) \right) \end{align*} Note \begin{align*} \|\boldsymbol{g}_2\|^2 & = (\boldsymbol{ D}^{1/2} \boldsymbol{g}_1) ^\top \boldsymbol{d}^{1/2} \boldsymbol{g}_1 = \boldsymbol{g}_1^\top \boldsymbol{D} \ Boldsymbol{g}_1 = (\boldsymbol{u}^\top \boldsymbol{g}) ^\top \boldsymbol{d} \boldsymbol{u}^\top \boldsymbol{g} = \ Boldsymbol{g}^\top \boldsymbol{\sigma} \boldsymbol{g} = \boldsymbol{w}^\top \boldsymbol{\sigma}_i \boldsymbol{w} \ C & Amp =-\frac{\boldsymbol{g}^\top \boldsymbol{\mu} + h}{\|\boldsymbol{g}_2\|} = \frac{y_i (\boldsymbol{w}^\top \boldsymbol{ X}_i + b)-1}{\sqrt{\boldsymbol{w}^\top \boldsymbol{\sigma}_i \boldsymbol{w}}}\\ \boldsymbol{g}^\toP \BOLDSYMBOL{\MU} + H & = 1-y_i (\boldsymbol{w}^\top \boldsymbol{x}_i + b) \end{align*} All generations are finally available in the sample $ (\boldsymbol{x}_i , Y_i, \boldsymbol{\sigma}_i) $ on the expected loss of \begin{align*} L (\boldsymbol{w}, B, \boldsymbol{x}_i, y_i, \boldsymbol{\sigma}_i ) = \sqrt{\frac{\boldsymbol{w}^\top \boldsymbol{\sigma}_i \boldsymbol{w}}{2 \pi}} \exp \left (-\frac{(\boldsymbol{w}^\ Top \boldsymbol{x}_i + b-y_i) ^2}{2\boldsymbol{w}^\top \boldsymbol{\sigma}_i \boldsymbol{w}} \right) + \frac{1-y_i (\bol Dsymbol{w}^\top \boldsymbol{x}_i + b)}{2} \left (1-\mbox{erf} \left (\frac{y_i (\boldsymbol{w}^\top \boldsymbol{x}_i + b) -1}{\sqrt{2\boldsymbol{w}^\top \boldsymbol{\sigma}_i \boldsymbol{w}}} \right) \right) \end{align*}

This loss is the original hinge loss of the non-negative combination, so it is convex, its gradient is not difficult to calculate, so the standard convex optimization method directly can be directly used to solve.

Expectation of hinge loss under Gaussian distribution