The Most Important Memory Rule
Memorize this:
Hidden layers → ReLU
Binary classification → Sigmoid
Multiclass classification → Softmax
Multilabel classification → Sigmoid
Regression → Linear
An even easier memory sentence is:
ReLU inside, Sigmoid for yes/no, Softmax for one-of-many, Sigmoid for many-of-many, Linear for numbers.
1. ReLU: The Default Choice for Hidden Layers
ReLU stands for:
Rectified Linear Unit
In TensorFlow:
tf.keras.layers.Dense(
128,
activation="relu"
)
or:
tf.keras.layers.Conv2D(
32,
(3, 3),
activation="relu"
)
Conceptually:
ReLU(x) = max(0, x)
Examples:
Input Output
-10 → 0
-5 → 0
-1 → 0
0 → 0
2 → 2
10 → 10
ReLU removes negative activations but keeps positive values unchanged.
Why ReLU is commonly used
ReLU is popular because it:
- introduces non-linearity;
- is computationally simple;
- works well in deep neural networks;
- reduces some vanishing-gradient problems;
- works especially well with CNNs and Dense layers.
Typical usage
model = tf.keras.Sequential([
tf.keras.layers.Dense(
128,
activation="relu"
),
tf.keras.layers.Dense(
64,
activation="relu"
)
])
Memory tip
Hidden layer? Start with ReLU.
2. Sigmoid for Binary Classification
Suppose you want to predict:
Cat or Dog
There are only two possible outcomes.
Use:
tf.keras.layers.Dense(
1,
activation="sigmoid"
)
Sigmoid produces a value between:
0 and 1
For example:
0.04
0.27
0.52
0.88
0.99
You can interpret the value using a threshold:
if prediction >= 0.5:
print("Dog")
else:
print("Cat")
Typical output:
0.08 → Cat
0.93 → Dog
Binary Classification Examples
Sigmoid is commonly used for:
Cat vs Dog
Horse vs Human
Spam vs Not Spam
Fraud vs Legitimate
Disease vs No Disease
Positive vs Negative
Sarcastic vs Not Sarcastic
Approved vs Rejected
Example model:
model = tf.keras.Sequential([
tf.keras.layers.Dense(
128,
activation="relu"
),
tf.keras.layers.Dense(
1,
activation="sigmoid"
)
])
Compile with:
model.compile(
optimizer="adam",
loss="binary_crossentropy",
metrics=["accuracy"]
)
Memory tip
Yes or No → Sigmoid
or:
Two classes → one sigmoid output
3. Softmax for Multiclass Classification
Now suppose you want to classify an image as:
Cat
Dog
Horse
Human
Only one answer should be correct.
Use:
tf.keras.layers.Dense(
4,
activation="softmax"
)
Softmax generates one probability for every class.
Example:
Cat 0.03
Dog 0.07
Horse 0.86
Human 0.04
The probabilities normally add up to:
1.00
The highest probability becomes the prediction.
predicted_class = np.argmax(predictions)
In this example:
Horse = 86%
So the model predicts:
Horse
Multiclass Examples
MNIST
Digits:
0
1
2
3
4
5
6
7
8
9
There are ten possible classes.
Use:
tf.keras.layers.Dense(
10,
activation="softmax"
)
Fashion MNIST
There are ten clothing classes.
Use:
tf.keras.layers.Dense(
10,
activation="softmax"
)
Cat, Dog, Horse, Human
Use:
tf.keras.layers.Dense(
4,
activation="softmax"
)
Typical loss:
loss="sparse_categorical_crossentropy"
Memory tip
Many choices, choose ONE → Softmax
4. Why Vocabulary Prediction Uses Softmax
Consider:
tf.keras.layers.Dense(
vocab_size,
activation="softmax"
)
This is very common in NLP.
Suppose:
vocab_size = 10000
The model is trying to predict one token from 10,000 possible tokens.
Imagine the input:
I love machine
The model might generate probabilities like:
learning 0.87
pizza 0.02
car 0.01
computer 0.06
science 0.04
...
The most likely next word is:
learning
So:
I love machine learning
Why Softmax?
Because the model is selecting:
one token from many vocabulary tokens
That is exactly a multiclass classification problem.
Memory tip
One next word from the vocabulary → Softmax
5. Multilabel Classification Uses Sigmoid
Multilabel classification is different from multiclass classification.
Suppose an image contains:
Person
Dog
Tree
Car
Several labels can be correct at the same time.
Therefore:
Person = Yes
Dog = Yes
Tree = Yes
Car = No
Use:
tf.keras.layers.Dense(
4,
activation="sigmoid"
)
Example output:
Person 0.96
Dog 0.91
Tree 0.83
Car 0.14
Using a threshold of:
0.5
the model predicts:
Person
Dog
Tree
The scores do not need to sum to 1.
Softmax vs Sigmoid for Multiple Classes
This difference is extremely important.
Softmax
Use when only one class is correct:
Cat OR Dog OR Horse OR Human
Dense(
4,
activation="softmax"
)
Example:
Cat 0.05
Dog 0.05
Horse 0.85
Human 0.05
Only one winner.
Sigmoid
Use when several classes can be correct:
Person AND Dog AND Tree
Dense(
4,
activation="sigmoid"
)
Example:
Person 0.95
Dog 0.92
Tree 0.88
Car 0.10
Three labels can be active simultaneously.
Memory trick
Softmax = competition
Sigmoid = independent decisions
6. Linear Activation for Regression
Regression predicts a numerical value rather than a class.
Examples:
House price
Temperature
Salary
Sales
Distance
Age
Revenue
Demand
Stock quantity
Use:
tf.keras.layers.Dense(1)
This automatically uses a linear activation.
It is equivalent to:
tf.keras.layers.Dense(
1,
activation="linear"
)
Example:
Input:
Bedrooms = 4
Area = 2500 sq ft
Location = Dhaka
Prediction:
15,700,000
There is no need for the prediction to be restricted between 0 and 1.
Typical losses include:
loss="mse"
or:
loss="mae"
Memory tip
Predict a number → Linear
7. Multiple Regression Outputs
Suppose the model predicts:
Temperature
Humidity
Wind Speed
You can use:
tf.keras.layers.Dense(3)
The model could return:
Temperature = 31.5
Humidity = 78.2
Wind Speed = 15.6
Again:
Numerical outputs → Linear
8. Tanh Activation
Tanh produces values between:
-1 and +1
Example:
tf.keras.layers.Dense(
64,
activation="tanh"
)
Conceptually:
Large negative input → close to -1
Zero → 0
Large positive input → close to +1
Tanh is commonly associated with:
- recurrent neural networks;
- LSTM internals;
- outputs requiring both positive and negative values;
- normalized data around zero.
Sigmoid vs Tanh
Sigmoid:
0 to 1
Tanh:
-1 to +1
Memory:
Tanh is like a zero-centered sigmoid.
For modern Dense and CNN hidden layers, ReLU is usually a more common default.
9. Leaky ReLU
Standard ReLU does this:
Negative → 0
If a neuron repeatedly receives negative values, it may stop contributing meaningfully.
This is sometimes called the:
dying ReLU problem
Leaky ReLU allows a small negative value.
Example:
tf.keras.layers.LeakyReLU(
negative_slope=0.01
)
Instead of:
-10 → 0
it might produce:
-10 → -0.1
Example model:
model = tf.keras.Sequential([
tf.keras.layers.Dense(128),
tf.keras.layers.LeakyReLU(),
tf.keras.layers.Dense(
1,
activation="sigmoid"
)
])
Memory tip
Leaky ReLU = ReLU that keeps a small negative signal alive.
10. Special Case: Regression Between 0 and 1
Sometimes you are predicting a numerical value, but the answer must stay between:
0 and 1
For example:
Normalized quality score
Risk score
Percentage expressed as 0–1
You could use:
tf.keras.layers.Dense(
1,
activation="sigmoid"
)
Example:
0.72
11. Special Case: Regression Between -1 and +1
If the prediction is explicitly constrained between:
-1 and +1
you can use:
tf.keras.layers.Dense(
1,
activation="tanh"
)
However, for unrestricted regression, use:
Dense(1)
Activation Functions for Common AI Tasks
Cat vs Dog
Dense(
1,
activation="sigmoid"
)
Why?
Binary classification
Horse vs Human
Dense(
1,
activation="sigmoid"
)
Cat, Dog, Horse, Human
Dense(
4,
activation="softmax"
)
Why?
One class from four possibilities
MNIST Digit Recognition
Dense(
10,
activation="softmax"
)
Positive vs Negative Sentiment
Dense(
1,
activation="sigmoid"
)
Positive, Neutral, Negative Sentiment
Dense(
3,
activation="softmax"
)
Sarcasm Detection
Dense(
1,
activation="sigmoid"
)
because:
Sarcastic
Not sarcastic
Next-Word Prediction
Dense(
vocab_size,
activation="softmax"
)
because one token is selected from the vocabulary.
Multiple Objects Present
Suppose the model only needs to report whether an image contains:
Person
Dog
Car
Tree
and multiple objects can be present.
Use:
Dense(
4,
activation="sigmoid"
)
House Price Prediction
Dense(1)
because it is regression.
Activation and Loss Function Pairing
It is useful to memorize activation functions together with their common losses.
| Prediction problem | Output | Activation | Loss |
|---|---|---|---|
| Binary classification | Dense(1) |
Sigmoid | binary_crossentropy |
| Multiclass | Dense(N) |
Softmax | sparse_categorical_crossentropy |
| Multiclass one-hot labels | Dense(N) |
Softmax | categorical_crossentropy |
| Multilabel | Dense(N) |
Sigmoid | binary_crossentropy |
| Regression | Dense(1) |
Linear | mse / mae |
| Multiple regression | Dense(N) |
Linear | mse / mae |
| Next-token prediction | Dense(vocab_size) |
Softmax | sparse_categorical_crossentropy |
Sparse Categorical vs Categorical Crossentropy
Both are normally paired with Softmax.
Use:
sparse_categorical_crossentropy
when your labels look like:
0
1
2
3
Example:
label = 2
Use:
categorical_crossentropy
when your labels are one-hot encoded:
[0, 0, 1, 0]
The activation remains:
softmax
Complete Binary Classification Example
model = tf.keras.Sequential([
tf.keras.layers.Input(
shape=(150, 150, 3)
),
tf.keras.layers.Rescaling(
1.0 / 255
),
tf.keras.layers.Conv2D(
32,
3,
activation="relu"
),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Conv2D(
64,
3,
activation="relu"
),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(
128,
activation="relu"
),
tf.keras.layers.Dense(
1,
activation="sigmoid"
)
])
model.compile(
optimizer="adam",
loss="binary_crossentropy",
metrics=["accuracy"]
)
The pattern is:
Hidden → ReLU
Output → Sigmoid
Complete Multiclass Example
model = tf.keras.Sequential([
tf.keras.layers.Input(
shape=(150, 150, 3)
),
tf.keras.layers.Rescaling(
1.0 / 255
),
tf.keras.layers.Conv2D(
32,
3,
activation="relu"
),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Conv2D(
64,
3,
activation="relu"
),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(
128,
activation="relu"
),
tf.keras.layers.Dense(
4,
activation="softmax"
)
])
model.compile(
optimizer="adam",
loss="sparse_categorical_crossentropy",
metrics=["accuracy"]
)
For:
Cat
Dog
Horse
Human
the pattern is:
Hidden → ReLU
Output → Softmax
Complete Multilabel Example
Suppose an image can contain:
Person
Dog
Horse
Car
Use:
model = tf.keras.Sequential([
tf.keras.layers.Dense(
128,
activation="relu"
),
tf.keras.layers.Dense(
4,
activation="sigmoid"
)
])
model.compile(
optimizer="adam",
loss="binary_crossentropy",
metrics=["accuracy"]
)
The pattern is:
Hidden → ReLU
Output → Sigmoid
Complete Regression Example
model = tf.keras.Sequential([
tf.keras.layers.Dense(
128,
activation="relu"
),
tf.keras.layers.Dense(
64,
activation="relu"
),
tf.keras.layers.Dense(1)
])
model.compile(
optimizer="adam",
loss="mse",
metrics=["mae"]
)
The pattern is:
Hidden → ReLU
Output → Linear
The Decision Tree
When choosing your output activation, ask:
What am I predicting?
|
|
+--- A number?
| |
| +--- YES → Linear
|
+--- A category?
|
+--- Only two possible outcomes?
| |
| +--- YES → Sigmoid
|
+--- More than two categories?
|
+--- Only ONE can be correct?
| |
| +--- YES → Softmax
|
+--- Several can be correct?
|
+--- YES → Sigmoid
Quick Cheat Sheet
====================================================
ACTIVATION CHEAT SHEET
====================================================
HIDDEN LAYERS
----------------------------------------------------
Dense / CNN hidden layers → ReLU
OUTPUT LAYER
----------------------------------------------------
Yes / No → Sigmoid
Two classes → Sigmoid
One class from many → Softmax
Next word from vocabulary → Softmax
Multiple labels simultaneously → Sigmoid
Predict number → Linear
Predict value 0–1 → Sigmoid
Predict value -1–1 → Tanh
====================================================
Five Rules to Memorize
Rule 1
Hidden layer
↓
ReLU
Rule 2
Binary / Yes-No
↓
Sigmoid
Rule 3
Many classes
but only ONE correct
↓
Softmax
Rule 4
Many classes
and MANY can be correct
↓
Sigmoid
Rule 5
Predict numerical value
↓
Linear
Final Memory Trick
If you remember only one thing from this article, remember:
ReLU inside, Sigmoid for yes/no, Softmax for one-of-many, Sigmoid for many-of-many, Linear for numbers.
For example:
tf.keras.layers.Dense(
vocab_size,
activation="softmax"
)
uses Softmax because the model is selecting:
ONE next token
from:
MANY vocabulary tokens
Therefore:
One from many → Softmax
Once you learn to identify the type of prediction problem, selecting the output activation function becomes much easier.
There are 0 comments