• Prove that the derivative of the Rectified Linear Unit (ReLU) with respect to the input \(z\) is:
\[\frac{dReLU(z)}{dx}=\left\{\begin{array}{ll} 0 & \text { if z < 0} \\ 1 & \text { if z > 0} \end{array}\right.\]

  • Recall that the RelU performs zero-thresholding of the input, i.e., the input cannot be lower than 0. In other words, it acts as a “gate-keeper” (or a switch) and only propagates forward non-negative inputs, while zeroing out other inputs.
\[ReLU(z)=\left\{\begin{array}{ll} 0 & \text { if z < 0} \\ z & \text { if z > 0} \end{array}\right.\]
  • Put simply,
\[ReLU(z) = max(0,z)\]

  • So the output of a ReLU is either \(z\) or 0, depending on whether the input is non-negative or negative respectively. Note that the ReLU is not defined at 0, so there must be a convention to set it either at 0 or 1 in this case.

  • As such, the derivative of ReLU with respect to the input \(z\) is:

\[\boxed{\frac{dReLU(z)}{dz}=\left\{\begin{array}{ll} 0 & \text { if z < 0} \\ 1 & \text { if z > 0} \end{array}\right.}\]
  • Intuitively, the derivative of the ReLU indicates that the error either fully propagates to the previous layer (owing to the 1) in case if the input to the ReLU is non-negative, or is completely stopped (owing to the 0) if the input to ReLU is negative.