Why do Neural Networks Have Bias Nodes?

See this great Stack Overflow answer.

I've also reproduced the whole thing below, and cleaned it up a bit.


I think that biases are almost always helpful. In effect, a bias value allows you to shift the activation function to the left or right, which may be critical for successful learning.

It might help to look at a simple example. Consider this 1-input, 1-output network that has no bias:

simple network

The output of the network is computed by multiplying the input $$x$$ by the weight ($$w_0$$) and passing the result through some kind of activation function (e.g. a sigmoid function.)

Here is the function that this network computes, for various values of $$w_0$$:

network output, given different w0 weights

Changing the weight $$w_0$$ essentially changes the "steepness" of the sigmoid. That's useful, but what if you wanted the network to output 0 when $$x$$ is 2? Just changing the steepness of the sigmoid won't really work -- you want to be able to shift the entire curve to the right.

That's exactly what the bias allows you to do. If we add a bias to that network, like so:

simple network with a bias

...then the output of the network becomes sig$$(w_0x + w_1 \cdot 1.0)$$. Here is what the output of the network looks like for various values of $$w_1$$.

network output, given different w1 weights

Having a weight of -5 for $$w_1$$ shifts the curve to the right, which allows us to have a network that outputs 0 when $$x$$ is 2.

Related Posts

Some ChatGPT Sessions

2016 ML thoughts

My biggest takeaway from Redwood Research REMIX

finite, actual infinity, potential infinity

Actions and Flows

PSA: reward is part of the habit loop too

a kernel of lie theory

The hyperfinite timeline

Slick hyperfinite Ramsey theory proof

Gaoxing Guy