LR review LR is the first step in machine learning, summing up, Linear Regression and logistic Regression are GLM, after the set of logistic, the output becomes a probability, loss function and Likelihood function inversion is a similar thing, can be used as the goal of optimization. But I feel likelihood function from the probability of statistics, more theoretical support it. Loss function directly to the residual squared sum, intuitive is also quite reasonable; Of course, for the logistic regression, the probability of the residual error is not very good, the more important reason is the non-convex function, difficult to solve.
Coursera ml of the programming of the document is really good, very meticulous, resulting in the video did not see how to directly put the topic down to do off. Hands or is very important, than to see the theory can have a deeper understanding. After the programming problem, the most profound impression is w0 processing, regularization of the time to exclude it. This thing is called bias, or called intercept term what. Because of this thing, it is quite troublesome to deal with it. To give a simple case y= W1 * x + w0, here w0 is a intercept, adjusting the line does not pass through the origin. From this point of view, W0 really should not be regularization, the value is how much.
Note that often the coefficientW0 is omitted from the RegularizerBecause its
Inclusion causes the results to depend on the choice of origin for the target variable
(Hastie et al. , 2001), or it may is included but with its own regularization coefficient
(We shall discuss this topic in the more detail in section 5.5.1).
Prml mentioned two points. Should be well understood, meaning that you can choose to linearly transform y, or choose a regularization factor for w0 alone.
Another question is whether the LR model we refer to at work refers to linear Regression or Logistic Regression? Generally refers to the latter, mainly used to do Ctr (PV/CTR) estimates and other work, the results can be directly as a score to use, is actually a 2 classification of the problem, click, or do not click. LR interesting point is that if the "occurrence ratio" (odd, the probability of occurrence divided by the probability of not occurring) to take the natural logarithm, the result w*x, is linear. (You can look at Hangyuan Li's "Statistical learning Method")
Another problem is the regular item, and the regular item is a thing that prevents over fitting. What is the L1 in the mouth of others, L2? L1 easy to produce sparse solution what's the matter? In terms of probability statistics, this regular term belongs to the Bayesian regularization method ("We introduce a Gaussian priori probability into the parameter to calculate the map (max Posteriori) estimate (rather than the maximum likelihood estimate)", in ufldl see this sentence, and then Google a bit, to find this reply to answer HTTP. www.zhihu.com/question/20700829 can see). Later read Scikit-learn document, found some new term, frighten my chrysanthemum a tight, thought there are many algorithms to learn. It turned out to be just a vest. L2 regularization, Ridge L1 regularization, Lassomix L1 and L2, elastic Net
LR is a linear model. Linearity is a good feature, but it is also a relatively strong assumption. But linearity is for weight, not for input, that is to say Y=w *x^2 + B is also linear. In practice, the LR model is used to fit the nonlinearity, which is usually the combination of discretization and feature. Discretization: For example, two students college entrance examination, a test 98, a test 95, from the score is a difference, but from the ability does not have a particularly big difference, too thin but inaccurate. We can disperse fractions into a, B, C, D four levels and then encode 4 binary features, such as A=0001,b=0 0 1 0. Or age 18, for example.+ What, in reality is the way to grade, only need the level, the specific value is not people's attention point. Feature combination: Numeric features are combined with ID features. For example, the query category mens =0,1, women's =1,0. User purchasing power =x, User level =y, ask a Cartesian product, you can get 4 characteristics. < purchasing power, Menswear > < purchasing power, women's category > < user level, men's category > < user level, women's category >. By this kind of fitting nonlinearity, it is actually the "and" relation that complements the characteristic.
Start SVM The second hurdle is SVM. However, SVM is difficult, in fact, it is difficult to solve the mathematical process. If you first learned convex optimization (session notes have a lite version of Cs229-cvxopt.pdf and cs229-cvxopt2.pdf, very good. Only from the content itself, is to look at these two look at the SVM, but in fact, I feel so dull things more ugly, may first look at SVM, was inside of the dual toss half-dead time, only interested to see the convex optimization, two data best fork see it) This course, estimated a second kill. But to learn an algorithm, it is important to understand the idea of the algorithm. I began to adhere to the mathematical derivation and solution, the result of stepping on the pit. The idea of SVM is actually very simple, that is, to find the nearest point from the boundary, and then choose from which can make the largest interval. Compare LR. The SVM focus is more on the classification boundaries (more precisely, those support vectors), whereas LR considers all the samples. It reminds me of a look-up edge filter in Photoshop. Think about the classification, are we more concerned about the boundaries of the place? Are we concerned about the point too far from the border?in the mathematical language to describe, is the SVM loss function is Higne loss Http://en.wikipedia.org/wiki/Hinge_loss , SVM is really that simple! L (Y) = max (0, 1-t*y). A magical max operation, throw away the point too far from the border (loss=0, drop it directly). So cs229 courseware, SVM see my egg pain chrysanthemum tight, coursera course incredibly simply draw a picture.
Although from the picture is similar, but from the calculation, in order to remove those far away from the boundary point, but also with Lagrange duality, make a bit of trouble.
Alternatively, we can understand SVM from a different perspective. The SVM-trained model is also WTX + B, where
We can understand that for each sample x (i), we assign a like to indicate whether X (i) is not a support vector, and through constant iterations, those points away from the boundary slowlythe like becomes 0. At the same time, we can also find that WTX is actually the span of the support vector (linear generation space). In addition, compared to LR, suppose xi∈r2, we have learned the model only w0,w1,w2 three parameters, and the number of SVM parameter is the number of alpha support vector, can be far more than 2, forming a complex model than LR.
Comparison of LR, regularization term c=1/λ
Model results, LR output W,SVM w=∑alphas_i * Yi * XI, if you want to use the kernel function, you have to take the XI band, not just save W. Briefly summarize:
- SVM focuses on the boundary, discarding points too far from the boundary. Input accuracy is low, output is generally only output classification-1 or 1
- LR considers all points, output probability (finer)
Hands-on practice SVMI bought a "machine learning combat" before, I also think with the code inside to learn to achieve the SVM is still relatively good. Classification of the chariot has a good summary, with the "Machine learning actual Combat" example, the code implementation process, some formulas to be simple deduction, inside pointing, suddenly on the brain hole big Open, the beginning has not been clear.
"Classified Chariot SVM" This series finally finished, the first: open the question Http://t.cn/RAl1Qou; second: linear classification Http://t.cn/RAl1QoT; third: Lagrange dual Http://t.cn/RAl1Qom ; The four words: nuclear function Http://t.cn/RAl1QoE; V: smohttp://t.cn/ral1qon; words: Programming Http://t.cn/RAl1QoQ; Appendix: Http://t.cn/RAl1QoR
if you want to implement SVM, Lagrange duality, kkt what is still have to be honest to understand the line. I used to feel not fully understand, look at the pseudo-code to implement the SVM itself is not much meaning. And you move .Handwriting, or some of the experience."machine learning Combat" with the implementation of Python, do the Coursera curriculum programming work, feel with octave write a bit more smooth, the code is more close to the formula.
function MODEL=SVM (x, Y, C, Max_iter, Kernel) y (y==0) = -1;%m: # of sample%n: # of feature dimension[m,n] = size (x); alpha s = zeros (m,1); b=0;toler=1e-3; k = zeros (m); for i = 1:m for j = i:m K (i,j) = Kernel (x (i,:) ', X (J,:) '); K (j,i) = K (i,j); %the matrix is symmetric endenditer = 0;while iter < max_iteralphapairschanged = 0;for i = 1:MFXI = f (Alphas, Y, X, B, I, K); ei = fxi-y (i); if (Y (i) * Ei <-toler & Alphas (i) < C) ... | | (Y (i) * Ei > Toler & Alphas (i) > 0) j = randselectj (i, m); #j = mod (i + 1, m) + 1;FXJ = f (Alphas, Y, X , B, J, K); Ej = Fxj-y (j); alpha_i_old = Alphas (i); alpha_j_old = Alphas (j); if (Y (i) ~= y (j)) L=max (0, Alphas (j)-alphas (i)); H=min (c, C + Alphas (j)-alphas (i)); Elsel=max (0, Alphas (j) + Alphas (i)-C); H=min (C, Alphas (j) + Alphas (i)), endif L = = H fprintf ("l==h!\n"), Continue;endeta = 2 * k (i,j)-K (i,i)-K (J, J); if (ET A >= 0) fprintf ("eta>=0\n"); Continue;endalphas (j)-= Y (j) * (Ei-ej)/eta;tmp_j= Clipalpha (Alphas (j), H, L); Alphas (j) = Tmp_j; if (ABS (Alphas (j)-Alpha_j_old) < 0.00001) Continue;endalphas (i) + = y (j) * Y (i) * (Alpha_j_old-alphas (j)); B1 = B-ei-y (i) * (Alphas (i)-alpha_i_old) * k (i, i) ...-y (j) * (Alphas (j)-Alpha_j_old) * k (I, j); b2 = B-e J-y (i) * (Alphas (i)-alpha_i_old) * k (i, j) ...-y (j) * (Alphas (j)-Alpha_j_old) * k (J, J); if (0 < AL PHAs (i)) & (C > Alphas (i)) b = B1;elseif (0 < Alphas (j)) & (C > Alphas (j)) b = B2;elseb = (b1+b 2)/2;endalphapairschanged + = 1;fprintf ("iter:%d, i:%d, pairs changed%d\n", ITER, I, alphapairschanged); if exist (' OCTAVE _version ') fflush (stdout); end end%end of IfEnd%end of for if alphapairschanged = = 0 iter + = 1;else iter = 0; End fprintf ("Iteration Number:%d\n", ITER); end idx = Alphas > 0;model.idx=idx;model. x= X (idx,:); model.y= y (idx); model.kernelfunction = kernel;model.b= b;model.alphas= alphas (idx); MODEL.W= ((alphas.*y) ' *x '; end
Experience:
| Svm |
Lr |
model also contains the original x, Y (of course, as long as the support vector, if not kernel, it can also be directly to the last W)
|
requires only W |
| Y takes the value { -1,1}, of course, just for convenience of calculation. But the implementation of the time or step on the pit, did not find any random data inside Y incredibly {0,1}. Printing debugging information just dawned. |
Y value {0,1}
|
I initially felt that SVM B was because it could use kernel. Is that kernel SVM-specific? No. So why doesn't LR need kernel? "Dragon Star machine learning" Video said that the solution is not high efficiency. For the time being also superficial understanding.
cs229 Stanford Machine Learning Note (ii)
