Primers • Partial Derivative of the Cost Function for Logistic Regression

• The partial derivative of the logistic regression cost function with respect to \(\theta\) is:

• Let’s begin with the cost function used for logistic regression, which is the average of the log loss across all training examples, as given below:

  \[J(\theta)=-\frac{1}{m} \sum_{i=1}^{m} y^{(i)} \log \left(h_{\theta}\left(x^{(i)}\right)\right)+\left(1-y^{(i)}\right) \log \left(1-h_{\theta}\left(x^{(i)}\right)\right)\]
  • where the logs are natural logarithms and \(h_{\theta}(x)\) is defined as:
  \[\begin{array}{l} h_{\theta}(x)=g(\theta^{T} x) \\ g(z)=\frac{1}{1+e^{-z}} \end{array}\]

• We use the notation:

• Since our original cost function is the form of:

• Now,

• Plugging in the two simplified expressions above in our original cost function, we obtain:

• which can be simplified to:

  \[\boxed{J(\theta)=-\frac{1}{m} \sum_{i=1}^{m}\left[y_{(i)} \theta x^{(i)}-\theta x^{(i)}-\log \left(1+e^{-\theta x^{(i)}}\right)\right]=-\frac{1}{m} \sum_{i=1}^{m}\left[y_{(i)} \theta x^{(i)}-\log \left(1+e^{\theta x^{(i)}}\right)\right]}\]
  • where the second equality follows from:
  \[-\theta x^{(i)}-\log \left(1+e^{-\theta x^{(i)}}\right)=-\left[\log e^{\theta x^{(i)}}+\log \left(1+e^{-\theta x^{(i)}}\right)\right]=-\log \left(1+e^{\theta x^{(i)}}\right)\] \[\text { because, } \log (x)+\log (y)=\log (x y)\]

• Now, all you need is to compute the partial derivative of the boxed equation above w.r.t. \(\theta_{j}\), using the following:

• Finally, plugging in the two components above in the expression for \(\frac{\partial J(\theta)}{\partial \theta_j}\), we obtain the end result:

References

• Derivative of cost function for Logistic Regression
• Properties of logarithms