Linear Functions

< Linear Regression Theory >

 

Regression (Examples)

Exam score Prediction (Linear Regression)

 

Classification

Pass / Fail (Binary Classification)

Letter Grades (Multi-level Classification)

 

Linear Model : H(x) = Wx = b

Which model is the best among the given three?

Statistics: Linear Regression

Cost Function (or Loss Function)

How to fit the line to Training data

The difference between model calue and real meaurements

 

m : The number of training data

- Cost Function Minimization

 

Model :

Cost Function :

 

- How to Minimize this Function? → Gradient Descent Method

Angle → Differentiation

&alpha; : Learning rate

Stochastic vs Batch Gradient Descent

- Learning Rates

Too large: Overshooting

Too small: takes too long, stops in the middle

 

- How can we determine the learning rates?

Try several learning rates

Observe the cost function

Check it goes down in a resonable rate

 

- Cost Function Minimization

Gradient Descent Method is only good for convex functions.

Unconvex Fn

- Multi-Variable Linear Regression

Model :

Cost :

Model:

Make x values a vector : (1 - by n) row Vector

w : n by 1 Column Vector

 

Transpose: XWt + b

 

= Linear Combination

---

< Binary Classification >

 

- Binary Classification Examples

Spam Detection: Spam [1] or Ham[0]

Facebook Feed: Show[1] or Hide[0]

 Facebook learns with your like-articles; and shows your favors.

Credeit card Fraudulent Transaction Detection: Fraud [1] or Legitmate[0]

Tumor Image Detection in Radiology: Malignat [1] or Benign [0]

 

- Binary Classification Basic Idea

Step 1) Linear regression with:

Step 2) Logistic / Sigmoid function (sig(t)) based on the result of Step 1.

Sigmoid Function

 

Linear Regression Model:

Gradient Descent Method can be used because cost(W, b) is convex (local minimum is global minimum).

 

Binary Classification Model:

Gradient Descent Method can not be used because Cost(W, b) is non-convex.

New Cost Function required.

---

< Softmax Classification > = (Multinomial Classification)

 

- Cost Function : Cross-Entropy