When training a neural network using the back-propagation algorithm where you update after looking at a single training item (stochastic or online training), you must supply a value for the learning rate. The learning rate controls how much each weight and bias is changed in each iteration of training.