• Prove that the derivative of the Rectified Linear Unit (ReLU) with respect to the input \(z\) is:
\[\frac{dReLU(z)}{dx}=\left\{\begin{array}{ll} 0 & \text { if z < 0} \\ 1 & \text { if z > 0} \end{array}\right.\]

  • Recall that the RelU performs zero-thresholding of the input, i.e., the input cannot be lower than \(0\). In other words, it acts as a “gate-keeper” (or a switch) and only propagates forward non-negative inputs, while zeroing out other inputs.
\[ReLU(z)=\left\{\begin{array}{ll} 0 & \text { if z < 0} \\ z & \text { if z > 0} \end{array}\right.\]
  • Put simply,
\[ReLU(z) = max(0,z)\]
  • So the output of a ReLU is either \(z\) or \(0\), depending on whether the input is non-negative or negative respectively. Note that the ReLU is not defined at \(0\), so there must be a convention to set it either at \(0\) or \(1\) in this case.

  • As such, the derivative of ReLU with respect to the input \(z\) is:

\[\boxed{\frac{dReLU(z)}{dz}=\left\{\begin{array}{ll} 0 & \text { if z < 0} \\ 1 & \text { if z > 0} \end{array}\right.}\]
  • Intuitively, the derivative of the ReLU indicates that the error either fully propagates to the previous layer (owing to the \(1\)) in case if the input to the ReLU is non-negative, or is completely stopped (owing to the \(0\)) if the input to ReLU is negative.