Section 06

Drawing a boundary

Turning scores into yes/no decisions

The last section left us with a gap. Cross-entropy needs a probability for each class, but a plain weighted sum of inputs (the only thing our model knows how to compute) can land anywhere from minus infinity to plus infinity. A score of −7.2 or +340 isn’t a probability. This section is about the small, clever function that bridges the two, and what it does to the geometry of a classification problem.

From a number to a probability

Start with the raw output of a linear model, the score (often called the logit): $z = w_1 x_1 + w_2 x_2 + b$ . It’s an unbounded real number: big and positive when the model leans toward “yes,” big and negative for “no,” near zero when it’s unsure. We want to squash that whole range into the interval $(0, 1)$ so it reads as a probability.

The sigmoid function (also called the logistic function) does exactly that:

\sigma(z) = \frac{1}{1 + e^{-z}}

Feed it a huge positive score and it returns something just under 1; a huge negative score gives something just above 0; a score of 0 gives exactly 0.5. Its S-shape is gentle in the middle and flattens at the ends: confident scores map to confident probabilities, and once you’re already sure, being more sure barely moves the output. That output is precisely the probability cross-entropy was waiting for.

The score also draws a line

Here’s the geometric payoff. In a space of two features, the set of points where the score is exactly zero (where $w_1 x_1 + w_2 x_2 + b = 0$ , so the probability is exactly 0.5) is a straight line. On one side the score is positive and the model predicts “yes”; on the other it’s negative and predicts “no.” That line is the decision boundary, and the weights set its angle while the bias sets its offset.

So a linear classifier’s whole personality is one straight cut through feature space. The sigmoid didn’t add any bending power — the boundary is still a line — it only converted the linear score into a probability-shaped number so we can measure and penalize confidence.

From score to probability to boundary

The sigmoid (left) squashes a raw score into a probability. The same weights place a straight decision boundary (right); the threshold decides where it falls.

probability = sigmoid(w₁·score + b)

two classes + straight boundary

w₁ (weight on x₁ · steepness) = 1.20w₂ (weight on x₂) = 1.20b (bias · shift) = 0.00decision threshold = 0.50

correct 41 / 44accuracy 93%boundary at score = 0.00

Raise the weights and the sigmoid steepens — the model grows more decisive, its probabilities snapping toward 0 or 1. The bias slides the whole curve left or right. In the 2-D view those same numbers set the angle and position of one straight boundary; the threshold slides it perpendicular to itself, trading false alarms against misses. White-ringed dots are the ones currently on the wrong side.

Notice the threshold. We called the boundary “probability = 0.5,” but nothing forces 0.5. Slide the threshold and the line slides with it: demand 0.9 confidence before you cry “fraud” and you’ll raise fewer false alarms but miss more real cases; drop it to 0.2 and you catch more fraud at the cost of crying wolf. The weights fix the direction of the cut; the threshold picks where along it you’re willing to commit.

More than two classes: softmax

Sigmoid handles a yes/no question. For “which of $K$ classes?” (which item in a long list comes next) we need $K$ scores turned into $K$ nonnegative numbers that sum to 1. That’s the softmax function: exponentiate every score to make it positive, then divide by the total so the whole set can be treated as one probability distribution. A two-class softmax and a sigmoid are closely related ways to express the same yes/no choice.

One caution: softmax gives numbers shaped like probabilities, but that does not automatically mean a score of 0.9 is right 90% of the time. When predicted probabilities match real-world frequencies that way, the model is called calibrated. Calibration is useful, but it is a separate property from simply summing to 1.

We can now score, squash, and classify: turn raw inputs into a probability and a boundary, and measure how wrong that probability was. What we still can’t do is improve it automatically. We have a loss; which way do we turn each knob to make it smaller? Answering that needs one idea from calculus (the slope of the loss with respect to a knob) and that’s where we go next.